Crystal: Potential bug with HTTP::Client SSL requests

[barisbalic]

A month or so ago I tried writing a small API client with Crystal, I immediately got "unexpected end of http request" exceptions. I tried looking into it but within a few nights the problem had subsided and I was getting responses back, so I know there is some server-side configuration causing the issue.

That issue has now resurfaced (for the time being) and I have tried to track it down, but I am out of my depth. I have descended through the calls made by the client, all the way down to this LibSSL.read call, which returns -1.

It's possible to recreate the error with the following snippet:

require "http"

client = HTTP::Client.new URI.parse("https://getmondo.co.uk")
resp = client.get("/")
puts resp.body

cURL, wget and ruby clients all work fine.

I am happy to do anything I can to help, so please let me know.

[jhass]

It looks like we don't add the SNI extension.

[asterite]

I was able to reproduce it too. I get this error (I'll later add it to the code):

error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error

I found this, which mentions SNI at some point.

(just to add some info, I don't know what we need to do to fix this)

[asterite]

I used this to get the error message:

err = LibSSL.err_get_error
str = LibSSL.err_error_string(err, nil)
::puts String.new(str)

[asterite]

And this in LibSSL:

lib LibSSL
  fun ssl_get_error = SSL_get_error(ssl : SSL, ret : Int) : Int
  fun err_error_string = ERR_error_string(e : LibC::ULong, buf : LibC::Char*) : LibC::Char*
end

[jhass]

I'm currently writing proper error handling into SSL::Socket and will look into SNI afterwards.

[asterite]

I'd really like someone to try this on a mac, though, just in case:

require "http"

client = HTTP::Client.new URI.parse("https://getmondo.co.uk")
resp = client.get("/")
puts resp.body

[barisbalic]

I am using OSX, and I'll try it as soon as I get home!

[barisbalic]

Just tested on OSX, result is: SSL_connect: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (OpenSSL::SSL::Error)

[asterite]

Same here.

@jhass Tomorrow I'll try to send you a wireshark log, see if it helps

[asterite]

@bararchy Just to be sure, what's your openssl version?

$ openssl version

[jhass]

Already got one from @barisbalic, the major difference is that it looks like for some reason on OS X it uses TLS 1.0 by default, while on Linux it does use TLS 1.2.

[jhass]

Though the server supports TLS 1.0 just fine, the ciphersuites send in the Client hello vastly differ though:

Broken (crystal on OS X):
Working (from curl on the same system):
Working (crystal on any Linux really):
Supported by the server:

[jhass]

Here's what Ruby sets explicitly https://github.com/ruby/ruby/blob/trunk/ext/openssl/lib/openssl/ssl.rb#L23-L55

Should we go with something like this? What do other languages do here? @ysbaddaden do you think Mozilla's server list makes sense for a client configuration too (I'm unsure)?

[ysbaddaden]

I reproduced on Mavericks using the system's libssl (ie. /usr/lib/libssl.0.9.8.dylib) but I can't reproduce this when I link against OpenSSL 1.0.2h_1 installed with homebrew (--link-flags="-L/usr/local/opt/openssl/lib"). This may related to OpenSSL 0.9.8 (which is no longer found on linux, even debian oldstable has 1.0.1, maybe RHEL?).

As you suggested @jhass, I passed a context with specified ciphers (ie. HTTP::SSL_CIPHERS) and this fixed the problem for me. So let's do it :-)

[asterite]

Works for me! \o/

Awesome job @jhass and @ysbaddaden ❤ 💙💛

[barisbalic]

Works for me too, big :heart: to everyone involved.

[satb]

I get the same error too on the latest version 0.23.1
On a Mac 10.11.6
Trying this in the playground.
It looks like something in particular with the paypal sandbox api. Tried another domain but that connects fine.

require "http/client" headers = HTTP::Headers{"Authorization" => ""} response = HTTP::Client.post_form("https://api.sandbox.paypal.com/v1/oauth2/token", "grant_type=client_credentials", headers)

Getting this error. It connects fine with another http client (postman, curl etc)

SSL_connect: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure CLOSE

---

Update:

For the benefit of others, after I updated the OS version to High Sierra and reinstalled openssl and exported the PKG_CONFIG_PATH and recompiled and ran, it worked fine.