Jsoup使用HTTP POST方法发送带有负载的请求

Question

我正在尝试使用jsoup进行此HTTP请求，如此处所示：

http://api.decarta.com/v1/[KEY]/batch?requestType=geocode

以下是我编写的代码：

String postUrl = postURLPrefix + apiKey + "/batch?requestType=geocode";
String response = Jsoup.connect(postUrl).timeout(60000).ignoreContentType(true)
        .header("Content-Type", "application/json;charset=UTF-8")
        .method(Connection.Method.POST)
        .data("payload", jsonPayload.toString())
        .execute()
        .body();

jsonPayload.toString() 的输出结果如下：

{
  "payload": [
    "146 Adkins Street,Pretoria,Pretoria,Gauteng",
    "484 Hilda Street,Pretoria,Pretoria,Gauteng",
    "268 Von Willich Street,Centurion,Centurion,Gauteng",
    ...
  ]
}

这是一个完全有效的JSON。

然而，每次使用jsoup返回HTTP状态码400（格式错误）。
那么，如何使用jsoup发送正确的HTTPPOST和JSON数据，如果可能的话？（请注意，这是数据负载，不是URL中的普通键值对。）

- rahulserver

Answer 1

请使用最新的JSOUP库。

如果您正在使用maven，则将下面的条目添加到pom.xml中。

<dependency>
        <groupId>org.jsoup</groupId>
        <artifactId>jsoup</artifactId>
        <version>1.10.2</version>
</dependency>

以下代码将解决您的问题。

String postUrl=postURLPrefix+apiKey+"/batch?requestType=geocode";
            System.out.println(postUrl);
            String response= Jsoup.connect(postUrl).timeout(60000).ignoreContentType(true)
                    .method(Connection.Method.POST)
                    .requestBody("payload",jsonPayload.toString())
                    .execute()
                    .body();

- Santosh Hegde

---

根据最新的1.11.3版本，我们需要使用“.method(Method.POST)”和“.requestBody(jsonPayload.toString())”。 - Veeresh Devireddy

.header("Accept", "application/json") .header("Content-Type", "application/json")同样也是必需的！ - VISWESWARAN NAGASIVAM

Answer 2

你需要发布原始数据。该功能已实现，但尚未添加。请查看此拉取请求https://github.com/jhy/jsoup/pull/318。你真的需要使用jsoup吗？我的意思是，你可以使用HttpURLConnection（这是jsoup在底层使用的）来发出请求，然后将响应作为字符串传递给jsoup。

这里有一个例子HttpURLConnection（从www.mkyong.com中获取，并简化和添加了json /原始数据）。

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;

public class Main {

    public static void main(String[] args) {

        try {

            String url = "http://www.google.com";

            URL obj = new URL(url);
            HttpURLConnection conn = (HttpURLConnection) obj.openConnection();
            conn.setReadTimeout(5000);
            conn.addRequestProperty("Accept-Language", "en-US,en;q=0.8");
            conn.addRequestProperty("User-Agent", "Mozilla");
            conn.addRequestProperty("Referer", "google.com");

            conn.setDoOutput(true);

            OutputStreamWriter w = new OutputStreamWriter(conn.getOutputStream(), "UTF-8");

            w.write("SOME_JSON_STRING_HERE");
            w.close();

            System.out.println("Request URL ... " + url);

            int status = conn.getResponseCode();

            System.out.println("Response Code ... " + status);

            BufferedReader in = new BufferedReader(new InputStreamReader(
                    conn.getInputStream()));
            String inputLine;
            StringBuffer html = new StringBuffer();

            while ((inputLine = in.readLine()) != null) {
                html.append(inputLine);
            }

            in.close();
            conn.disconnect();

            System.out.println("URL Content... \n" + html.toString());
            System.out.println("Done");

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

- Alkis Kalogeris

---

这只是一个例子。它可以和应该被清理干净。错误处理不存在。我故意保留它这样，以便尽可能简单。 - Alkis Kalogeris

非常棒的最小化答案，可以在不使用JSoup等库的情况下本地实现此操作。我第一次尝试时就完美地运行了。+1 - Muhammad Abdul-Rahim

Answer 3

示例

    private fun token(email: String, password: String): String {
        val url = "https://asdf.com/tokens"

        val headers = mapOf(
            "Accept" to "application/json",
            "Content-Type" to "application/json",
            "Authorization" to "null"
        )
        val json = mapOf(
            "auth_type" to "CREDENTIAL",
            "credential_type_payload" to mapOf(
                "email" to email,
                "password" to password,
            ),
        ).toJson()

        return CrawlUtil.post(url, headers, json)["token"] as String
    }

---

爬虫工具

import org.jsoup.Connection
import org.jsoup.Jsoup

object CrawlUtil {
    private const val TIMEOUT = 5 * 60 * 1000
    private const val MAX_BODY_SIZE = 0
    private const val IGNORE_CONTENT_TYPE = true

    private fun connection(
        url: String,
        headers: Map<String, String>,
        json: String,
        data: Map<String, String>? = emptyMap()
    ): Connection {
        var connection = Jsoup.connect(url)

        headers.forEach {
            connection = connection.header(it.key, it.value)
        }

        data!!.forEach {
            connection = connection.data(it.key, it.value)
        }

        return connection
            .timeout(TIMEOUT)
            .maxBodySize(MAX_BODY_SIZE)
            .ignoreContentType(IGNORE_CONTENT_TYPE)
            .requestBody(json)
    }

    fun get(
        url: String,
        headers: Map<String, String>,
        json: String,
        data: Map<String, String>? = emptyMap()
    ): Map<String, Any> {
        return connection(url, headers, json, data)
            .get()
            .text()
            .toObject()
    }

    fun post(
        url: String,
        headers: Map<String, String>,
        json: String,
        data: Map<String, String>? = emptyMap()
    ): Map<String, Any> {
        return connection(url, headers, json, data)
            .post()
            .text()
            .toObject()
    }
}

ExtensionUtil

inline fun <reified T> String.toObject(): T {
    return JacksonUtil.toObject(this)
}

JacksonUtil

import com.fasterxml.jackson.core.type.TypeReference
import com.fasterxml.jackson.databind.DeserializationFeature
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializationFeature
import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule
import com.fasterxml.jackson.datatype.jsr310.deser.LocalDateDeserializer
import com.fasterxml.jackson.datatype.jsr310.deser.LocalDateTimeDeserializer
import com.fasterxml.jackson.module.afterburner.AfterburnerModule
import com.fasterxml.jackson.module.kotlin.KotlinModule
import java.time.LocalDate
import java.time.LocalDateTime
import java.time.format.DateTimeFormatter

object JacksonUtil {
    val objectMapper: ObjectMapper = ObjectMapper()
        .registerModule(KotlinModule.Builder().build())
        .registerModule(JavaTimeModule())
        .registerModule(AfterburnerModule())
        .configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)
        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
        .configure(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS, false)
        .registerModule(
            JavaTimeModule().apply {
                addDeserializer(LocalDate::class.java, LocalDateDeserializer(DateTimeFormatter.ISO_DATE))
                addDeserializer(
                    LocalDateTime::class.java,
                    LocalDateTimeDeserializer(DateTimeFormatter.ISO_DATE_TIME)
                )
            }
        )

    fun toJson(obj: Any?): String {
        if (obj == null) {
            return ""
        }

        return objectMapper.writeValueAsString(obj)
    }

    fun merged(a: MutableMap<String, Any>, b: Any): Map<String, Any> {
        return objectMapper
            .readerForUpdating(a)
            .readValue(toJson(b))
    }

    inline fun <reified T> toObject(s: String): T {
        return objectMapper.readValue(s, object : TypeReference<T>() {})
    }
}

https://seunggabi.tistory.com/entry/Jsoup-GET-POST-crawling

- seunggabi

Answer 4

这是我使用 jsoup v1.14.3 和 Kotlin（同样适用于Java）完成的方法。
它将两个参数 name 和 age 作为 JSON 对象的一部分发送（假设服务器接受该对象）：

val document = Jsoup.connect("the/target/url")
    .userAgent("Mozilla")
    .header("content-type", "application/json")
    .header("accept", "application/json")
    .requestBody("""{"name": "Katy", "age": 43}""")
    .ignoreContentType(true)
    .post()

你也可以以url-encoded format的格式发送参数（假设服务器接受该格式）：

val document = Jsoup.connect("the/target/url")
    .userAgent("Mozilla")
    .header("content-type", "application/x-www-form-urlencoded")
    .header("accept", "application/json")
    .data("name", "Katy")
    .data("age", "43")
    // OR
    //.data(mapOf("name" to "Katy", "age" to "43"))
    .ignoreContentType(true)
    .post()

- Mahozad