Odtnhj

I'm using Apache HttpClient in a web crawler that is only for crawling public data.

I'd like it to be able to crawl sites with invalid certificates, no matter how invalid.

My crawler won't be passing in any usernames, passwords, etc and no sensitive data is being sent or received.

For this use case, I'd crawl the http version of a site if it exists, but sometimes it doesn't of course.

How can this be done with Apache's HttpClient?

I tried a few suggestions like this one, but they still fail for some invalid certs, for example:

failed for url:https://dh480.badssl.com/, reason:java.lang.RuntimeException: Could not generate DH keypair
failed for url:https://null.badssl.com/, reason:Received fatal alert: handshake_failure
failed for url:https://rc4-md5.badssl.com/, reason:Received fatal alert: handshake_failure
failed for url:https://rc4.badssl.com/, reason:Received fatal alert: handshake_failure
failed for url:https://superfish.badssl.com/, reason:Connection reset

Note that I've tried this with my $JAVA_HOME/jre/lib/security/java.security file's jdk.tls.disabledAlgorithms set to nothing, to ensure this wasn't an issue, and I still get failures like the above.

The short answer to your question, which is to specifically trust all certs, would be to use the TrustAllStrategy and do something like this:

SSLContextBuilder sslContextBuilder = new SSLContextBuilder();
sslContextBuilder.loadTrustMaterial(null, new TrustAllStrategy());
SSLConnectionSocketFactory socketFactory = new SSLConnectionSocketFactory(
 sslContextBuilder.build());
CloseableHttpClient httpclient = HttpClients.custom().setSSLSocketFactory(
 socketFactory).build();

However... an invalid cert may not be your main issue. A handshake_failure can occur for a number of reasons but in my experience it's usually due to a SSL/TLS version mismatch or cipher suite negotiation failure. This doesn't mean the ssl cert is "bad", it's just a mismatch between the server and client. You can see exactly where the handshake is failing using a tool like Wireshark (more on that)

While Wireshark can be great to see where it's failing, it won't help you come up with a solution. Whenever I've gone about debugging handshake_failures in the past I've found this tool particularly helpful: https://testssl.sh/

You can point that script at any of your failing websites to learn more about what protocols are available on that target and what your client needs to support in order to establish a successful handshake. It will also print information about the certificate.

For example (showing only two sections of the output of testssl.sh):

./testssl.sh www.google.com
....
 Testing protocols (via sockets except TLS 1.2, SPDY+HTTP2) 

 SSLv2 not offered (OK)
 SSLv3 not offered (OK)
 TLS 1 offered
 TLS 1.1 offered
 TLS 1.2 offered (OK)
 ....
Server Certificate #1
 Signature Algorithm SHA256 with RSA
 Server key size RSA 2048 bits
 Common Name (CN) "www.google.com"
 subjectAltName (SAN) "www.google.com" 
 Issuer "Google Internet Authority G3" ("Google Trust Services" from "US")
 Trust (hostname) Ok via SAN and CN (works w/o SNI)
 Chain of trust "/etc/*.pem" cannot be found / not readable
 Certificate Expiration expires < 60 days (58) (2018-10-30 06:14 --> 2019-01-22 06:14 -0700)
 ....
 Testing all 102 locally available ciphers against the server, ordered by encryption strength 
(Your /usr/bin/openssl cannot show DH/ECDH bits)

Hexcode Cipher Suite Name (OpenSSL) KeyExch. Encryption Bits
------------------------------------------------------------------------
xc030 ECDHE-RSA-AES256-GCM-SHA384 ECDH AESGCM 256 
xc02c ECDHE-ECDSA-AES256-GCM-SHA384 ECDH AESGCM 256 
xc014 ECDHE-RSA-AES256-SHA ECDH AES 256 
xc00a ECDHE-ECDSA-AES256-SHA ECDH AES 256 
x9d AES256-GCM-SHA384 RSA AESGCM 256 
x35 AES256-SHA RSA AES 256 
xc02f ECDHE-RSA-AES128-GCM-SHA256 ECDH AESGCM 128 
xc02b ECDHE-ECDSA-AES128-GCM-SHA256 ECDH AESGCM 128 
xc013 ECDHE-RSA-AES128-SHA ECDH AES 128 
xc009 ECDHE-ECDSA-AES128-SHA ECDH AES 128 
x9c AES128-GCM-SHA256 RSA AESGCM 128 
x2f AES128-SHA RSA AES 128 
x0a DES-CBC3-SHA RSA 3DES 168 

So using this output we can see that if your client only supported SSLv3, the handshake would fail because that protocol isn't supported by the server. The protocol offering is unlikely the problem but you can double check what your java client supports by getting the list of enabled protocols. You can provide an overridden implementation of the SSLConnectionSocketFactory from above code snippet to get the list of enabled/supported protocols and cipher suites as follows (SSLSocket):

class MySSLConnectionSocketFactory extends SSLConnectionSocketFactory 
 @Override
 protected void prepareSocket(SSLSocket socket) throws IOException 
 System.out.println("Supported Ciphers" + Arrays.toString(socket.getSupportedCipherSuites()));
 System.out.println("Supported Protocols" + Arrays.toString(socket.getSupportedProtocols()));
 System.out.println("Enabled Ciphers" + Arrays.toString(socket.getEnabledCipherSuites()));
 System.out.println("Enabled Protocols" + Arrays.toString(socket.getEnabledProtocols()));
 

I often encounter handshake_failure when there is a cipher suite negotiation failure. To avoid this error, your client's list of supported cipher suites must contain at least one match to a cipher suite from the server's list of supported cipher suites.

If the server requires AES256 based cipher suites you probably need the java cryptographic extensions (JCE). These libraries are nation restricted so they may not be available to someone outside the US.

More on cryptography restrictions, if you're interested: https://crypto.stackexchange.com/questions/20524/why-there-are-limitations-on-using-encryption-with-keys-beyond-certain-length

I think that the post you are referring is very close to what it needs to be done. Have you tried something like:

HttpClientBuilder clientBuilder = HttpClientBuilder.create();
SSLContextBuilder sslContextBuilder = SSLContextBuilder.create();
sslContextBuilder.setSecureRandom(new java.security.SecureRandom());
try 
 sslContextBuilder.loadTrustMaterial(new TrustStrategy() 
 @Override
 public boolean isTrusted(X509Certificate arg0, String arg1) throws CertificateException 
 return true;
 
 );
 clientBuilder.setSSLContext(sslContextBuilder.build());
 catch (Throwable t) 
 Logger.getLogger(getClass().getName()).log(Level.SEVERE, "Can't set ssl context", t);

CloseableHttpClient apacheHttpClient = clientBuilder.build();

I have not tried this code but hopefully it could work.

Cheers

If you are fine to use other open source libraries like netty then worth trying below:

SslProvider provider = SslProvider.JDK; // If you are not concerned about http2 / http1.1 then JDK provider will be enough
SSLContext sslCtx = SslContextBuilder.forClient()
 .sslProvider(provider)
 .trustManager(InsecureTrustManagerFactory.INSTANCE) // This will trust all certs
 ... // Any other required parameters used for ssl context.e.g. protocols , ciphers etc.
 .build();

I have used below version of netty for trusting any certificates with above code:

<dependency>
 <groupId>io.netty</groupId>
 <artifactId>netty-all</artifactId>
 <version>4.1.29.Final</version>
</dependency>

I think @nmorenor answer is pretty close to the mark. What I would have done in addition is explicitly enabling SSLv3 (HttpClient automatically disables it by default due to security concerns) and disabling host name verification.

SSLContext sslContext = SSLContexts.custom()
 .loadTrustMaterial((chain, authType) -> true)
 .build();

CloseableHttpClient client = HttpClients.custom()
 .setSSLSocketFactory(new SSLConnectionSocketFactory(sslContext,
 new String"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2",
 null,
 NoopHostnameVerifier.INSTANCE))
 .build();

You can do it with core jdk too, but iirc, httpclient also allows you to set the SSL Socket Factory too.

The factory defines and uses a ssl context that you construst with a trust manager. That manager would simply not verify the cert chain, as shown in above post.

You also need a hostnameverifier instance that would also choose to ignore the potential mismatch of cert hostname with the url's host (or ip). Otherwise, it would still fail even if the cert signer is blindly trusted.

I used to convert many client stack to 'accept self-signed' and it's quite easy in most stack. The worse cases is when the 3rd party lib doesn't allow choosing a ssl socket factory instance but only its clasname. In that case, I use a ThreadLocalSSLSocketFactory which doesn't own any actual factory but simply looks up the threadlocal to find one that the upper stackframes (that you can control) would have prepared. This only works if the 3rd party lib is not doing the work on distinct thread of course. I know http client can be told to use a specific ssl socket factory so this is easy.

Also take the time to read the JSSE doc, it is totally worth the time it takes to read.