如何解决Jsoup错误：无法找到请求目标的有效认证路径

Question

我正在尝试解析以下URL的HTML：

https://www.smuc.ac.kr/mbs/smuc/jsp/board/list.jsp?boardId=6993&id=smuc_040100000000

我得到了以下错误：

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1491)
    ... 15 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
    ... 21 more

这是我的代码：

public class MainActivity extends AppCompatActivity {
    private ListView listView;
    private TextView textView;
    public ArrayList<String> arrayList = new ArrayList<String>();
    private ArrayAdapter<String> arrayAdapter;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        listView = (ListView) findViewById(R.id.listView);
        new Insert().execute();
        arrayAdapter = new ArrayAdapter<String>(MainActivity.this, R.layout.list_ok, R.id.text, arrayList );
       }

    class Insert extends AsyncTask<String, Void, String> {
        @Override
        protected String doInBackground(String... params) {
            try {
               // Connection.Response res = Jsoup.connect("https://www.smuc.ac.kr/mbs/smuc/index.jsp")
               //         .method(Connection.Method.POST)
               //         .execute();
                Document document = Jsoup.connect("https://www.smuc.ac.kr/mbs/smuc/jsp/board/list.jsp?boardId=6993&id=smuc_040100000000").get();
                Elements elements = document.select(".tit");
                arrayList.clear();
                for (Element element : elements) {
                    arrayList.add(element.text());
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
            return null;
        }
        @Override
        protected void onPostExecute(String result){
            listView.setAdapter(arrayAdapter);
        }

    }

}

- Behzod Muslimov

Answer 1

所选答案在最新版本的JSoup中将无法工作，因为validateTLSCertificates已被弃用并删除。我创建了以下帮助类:

public class SSLHelper {

    static public Connection getConnection(String url){
        return Jsoup.connect(url).sslSocketFactory(SSLHelper.socketFactory());
    }

    static private SSLSocketFactory socketFactory() {
        TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager() {
            public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                return new X509Certificate[0];
            }

            public void checkClientTrusted(X509Certificate[] certs, String authType) {
            }

            public void checkServerTrusted(X509Certificate[] certs, String authType) {
            }
        }};

        try {
            SSLContext sslContext = SSLContext.getInstance("SSL");
            sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
            SSLSocketFactory result = sslContext.getSocketFactory();

            return result;
        } catch (NoSuchAlgorithmException | KeyManagementException e) {
            throw new RuntimeException("Failed to create a SSL socket factory", e);
        }
    }
}

然后我只需按照以下方式调用：

Document doc = SSLHelper.getConnection(url).userAgent(USER_AGENT).get();

(*) - https://dzone.com/articles/how-setup-custom - 对解决方案有所帮助

- checklist

---

效果很好。我还为此编写了 Kotlin 版本：https://gist.github.com/sezabass/6a95deee8656571ce7046f3adc2ba573 - Cesar Castro

也对我有用！ - Jefferson Bien-Aime

Answer 2

注意：JSoup在1.12.1版本中已经弃用并删除了validateTLSCertificates方法。请参见此答案以获取替代解决方案。

---

在 JSoup 1.12.1 版本之前，可以按以下方式忽略 TLS 验证：

Document doc = Jsoup.connect("URL").timeout(10000).validateTLSCertificates(false).get();

既然读取页面也需要一段时间，那么请将超时时间增加为timeout(10000)。

- flavio.donze

Answer 3

根据George的评论，我会将这个作为答案发布。
我做了一个Kotlin版本的清单解决方案，如下：

package crawlers

import java.security.KeyManagementException

import java.security.NoSuchAlgorithmException

import java.security.SecureRandom
import java.security.cert.X509Certificate
import javax.net.ssl.SSLContext

import javax.net.ssl.SSLSocketFactory
import javax.net.ssl.TrustManager
import javax.net.ssl.X509TrustManager

class SSLHelperKotlin {

    companion object {

        @JvmStatic
        fun socketFactory(): SSLSocketFactory {
            val trustAllCerts = arrayOf<TrustManager>(
                    object : X509TrustManager {
                        override fun getAcceptedIssuers() = arrayOf<X509Certificate>()
                        override fun checkClientTrusted(certs: Array<X509Certificate>, authType: String) {}
                        override fun checkServerTrusted(certs: Array<X509Certificate>, authType: String) {}
                    }
            )

            return try {
                val sslContext: SSLContext = SSLContext.getInstance("SSL")
                sslContext.init(null, trustAllCerts, SecureRandom())
                sslContext.socketFactory
            } catch (e: NoSuchAlgorithmException) {
                throw RuntimeException("Failed to create a SSL socket factory", e)
            } catch (e: KeyManagementException) {
                throw RuntimeException("Failed to create a SSL socket factory", e)
            }
        }
    }
}

它也可以在这个代码片段上找到。

- Cesar Castro