Crystal: Issues verifying certificate connecting with HTTPS

[theodorton]

Crystal version

~/code/nabobil/autotoll · (master±)
⟩ crystal -v
Crystal 0.23.1 (2017-10-12) LLVM 4.0.1

How to reproduce

# test.cr
require "http/client"

client = HTTP::Client.new("maps.googleapis.com", tls: true)
client.get "/"

Then run crystal run test.cr.

Expected outcome

No errors raised, or some redirect.

Actual outcome

SSL_connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (OpenSSL::SSL::Error)
0x10d990e45: *CallStack::unwind:Array(Pointer(Void)) at ??
0x10d990de1: *CallStack#initialize:Array(Pointer(Void)) at ??
0x10d990db8: *CallStack::new:CallStack at ??
0x10d98d615: *raise<OpenSSL::SSL::Error>:NoReturn at ??
0x10da33571: *OpenSSL::SSL::Socket::Client#initialize<TCPSocket, OpenSSL::SSL::Context::Client, Bool, String>:Nil at ??
0x10da333ec: *OpenSSL::SSL::Socket::Client::new:context:sync_close:hostname<TCPSocket, OpenSSL::SSL::Context::Client, Bool, String>:OpenSSL::SSL::Socket::Client at ??
0x10da29c91: *HTTP::Client#socket:(OpenSSL::SSL::Socket+ | TCPSocket+) at ??
0x10da299eb: *HTTP::Client#exec_internal_single<HTTP::Request>:(HTTP::Client::Response | Nil) at ??
0x10da26f7c: *HTTP::Client#exec_internal<HTTP::Request>:HTTP::Client::Response at ??
0x10da26e72: *HTTP::Client#exec<HTTP::Request>:HTTP::Client::Response at ??
0x10da26bdd: *HTTP::Client#exec<String, String, Nil, Nil>:HTTP::Client::Response at ??
0x10da26bb7: *HTTP::Client#get<String>:HTTP::Client::Response at ??
0x10d97bcba: __crystal_main at ??
0x10d98c678: main at ??

Notes

• This also happens when running the code within the official docker image for 0.23.1.
• I've managed to get it to pass randomly, but I believe there is some cipher requirement and that the certificates differ somewhat between the servers behind this domain
• When running openssl s_client -connect maps.googleapis.com:443 -tls1_2 I get:

CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = *.googleapis.com
verify return:1
---
...
...

• It works if I setup a similar example in Ruby:

require 'net/http'
require 'uri'

uri = URI('https://maps.googleapis.com/')

Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
  request = Net::HTTP::Get.new uri
  response = http.request request # Net::HTTPResponse object
  puts response
end

• In other words, I don't believe this is an issue with my openssl configuration.

[Sija]

Duplicate of #3477.

See also https://github.com/composer/composer/issues/3346 and relevant question on SO.

[theodorton]

I'm not sure, I'm able to connect with TLS to github.com:

HTTP::Client.get("https://github.com/") # This works
HTTP::Client.get("https://maps.googleapis.com/") # This doesn't

I belive there might be something with the allowed cipher suites as SSLLabs Test results for maps.googleapis.com show that the root certificate is insecure.

[asterite]

I face this problem with some URLs. I think it depends on the version of openssl. Ruby ships with one version of openssl and that's very good: reproducible builds and behavior no matter what your system is. In Crystal we dynamically link to the openssl that's present in the system and it leads to these random and hard to reproduce problems (because we have to have your version of openssl in our machines).

I wish we could just ship Crystal with one version of openssl, one version of llvm, one version of libpcre, one version of libxml2, etc., so that everything works exactly the same everywhere...

[RX14]

@asterite And then what if openssl has a security bug? If we distribute our own libraries we are now distro maintainers. We have to follow the security bugs closely with a quick response, and possibly backport patches, and deal with a whole bunch of things we don'y have time to do. This is the distro maintainer's job, not ours, it's already done for us.

I've never had an openssl problem with crystal on archlinux, i've only seem people have problems on osx and on debian-based (read: ancient versions of openssl) distros.

[theodorton]

I don't have any strong opinions on whether to bundle openssl with the compiler or not, but I think this doing VERIFY_NONE as a workaround (which is mentioned in the linked issues) here unsatisfactory.

I tried to use OpenSSL::SSL::Context::Client#add_x509_verify_flags(OpenSSL::SSL::X509VerifyFlags:: TRUSTED_FIRST) which was mentioned as a fix in a Stack Overflow issue I found yesterday. I'm unable to find it again now, but the problem I ran into was that OpenSSL::SSL::X509VerifyFlags was not defined. Seems like there's a compiler flag for that constant that wasn't set.

It seems highly related, but I'm not sure? I'm building Crystal via Homebrew. Could it be that it's missing some SSL/crypto lib while compiling? Or that the compiled bottle comes without this support?

Edit: I'm trying to install from homebrew with --build-from-source to see if that resolves the issue.
Update: No, it doesn't help. I'm still getting undefined constant OpenSSL::OPENSSL_102 errors when attempting to use X509VerifyFlags.

[RX14]

It appears to me that openssl's API is a mess, perhaps we should implement the libtls API using openssl (in C, how openssl wants us to), and then use that as the base for crystal's TLS module and remove OpenSSL.

[theodorton]

The workaround here is:

context = OpenSSL::SSL::Context::Client.insecure
client = HTTP::Client.new("maps.googleapis.com", tls: context)

I'm not sure if that's a very good workaround though, as if I point the client to expired.badssl.com, I'm not getting any errors.

[ysbaddaden]

Hence the insecure: you basically accept whatever OpenSSL supports —I'm not even sure it verifies certificates. Would you be willing to take the Client.insecure and tweak it until you find the culprit in defaults?

[theodorton]

@ysbaddaden Yes, we're ok with Client.insecure for now, but it doesn't seem to verify certificates. I think that should probably be noted in the documentation. Currently the docs say:

Use this only if undoing the defaults that new sets is too much hassle.

Not sure if I can spend more time at the moment looking into the defaults. I'm not very fluent in TLS/SSL terminology so I'd just have to guess what the different options represent. The failures were pretty consistent with that hostname, even when I ran it in the Docker image for 0.23.1.

I still think this is highly relevant:

I'm still getting undefined constant OpenSSL::OPENSSL_102 errors when attempting to use X509VerifyFlags.

As I believe using the OpenSSL::SSL::X509VerifyFlags:: TRUSTED_FIRST might work.

[vladfaust]

Any updates? I'm basically unable to connect to any of Google's APIs :confused:

require "http/client"
client = HTTP::Client.new(URI.parse("https://google.com"))
p client.get("/")

It raises the same certificate verify failed (OpenSSL::SSL::Error).

[will]

Just chiming into say that I'm having this problem on heroku with google oauth.

Also since the oauth::client doesn't take a tls: context, the workaround of not verifying doesn't work, unless you monkey patch

[waj]

I just sent a PR with a potential fix. Please, let me know if it works for you and also provide some comments on the solution if possible.

[codenoid]

Crystal 0.24.1 (2017-12-22)

LLVM: 4.0.0
Default target: x86_64-unknown-linux-gnu

still error when accessing maps.google.com

SSL_connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (OpenSSL::SSL::Error)
  from /usr/share/crystal/src/openssl/ssl/socket.cr:34:9 in 'initialize'
  from OpenSSL::SSL::Socket::Client::new:context:sync_close:hostname<TCPSocket, OpenSSL::SSL::Context::Client, Bool, String>:OpenSSL::SSL::Socket::Client
  from /usr/share/crystal/src/http/client.cr:657:5 in 'socket'
  from /usr/share/crystal/src/http/client.cr:500:5 in 'exec_internal_single'
  from /usr/share/crystal/src/http/client.cr:486:5 in 'exec_internal'
  from /usr/share/crystal/src/http/client.cr:482:5 in 'exec'
  from /usr/share/crystal/src/http/client.cr:585:5 in 'exec'
  from /usr/share/crystal/src/http/client.cr:612:7 in 'exec'
  from /usr/share/crystal/src/http/client.cr:329:3 in 'get'
  from src/facebook.cr:32:5 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:11:3 in '_crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:112:5 in 'main_user_code'
  from /usr/share/crystal/src/crystal/main.cr:101:7 in 'main'
  from /usr/share/crystal/src/crystal/main.cr:135:3 in 'main'
  from __libc_start_main
  from _start
  from ???

oh wait, i'll try @waj PR

Update fixed with @waj PR (#5601)

Update

Python Script :

import requests

headers = {
    'Host': 'maps.googleapis.com',
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://google-developers.appspot.com/maps/documentation/utils/geocoder/embed',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Cache-Control': 'max-age=0',
}

params = (
    ('', ''),
    ('address', 'Kalibata City'),
)

response = requests.get('https://maps.googleapis.com/maps/api/geocode/json', headers=headers, params=params)

print response.text

""" result (200) :
{
   "results" : [
      {
         "address_components" : [
            {
               "long_name" : "Kalibata City",
               "short_name" : "Kalibata City",
               "types" : [ "premise" ]
            },
            {
               "long_name" : "1",
               "short_name" : "1",
               "types" : [ "street_number" ]
            },
            {
               "long_name" : "Jalan Raya Kalibata",
               "short_name" : "Jl. Raya Kalibata",
               "types" : [ "route" ]
            },
            {
               "long_name" : "Rawajati",
               "short_name" : "Rawajati",
               "types" : [ "administrative_area_level_4", "political" ]
            },
            {
               "long_name" : "Pancoran",
               "short_name" : "Pancoran",
               "types" : [ "administrative_area_level_3", "political" ]
            },
            {
               "long_name" : "Kota Jakarta Selatan",
               "short_name" : "Kota Jakarta Selatan",
               "types" : [ "administrative_area_level_2", "political" ]
            },
            {
               "long_name" : "Daerah Khusus Ibukota Jakarta",
               "short_name" : "Daerah Khusus Ibukota Jakarta",
               "types" : [ "administrative_area_level_1", "political" ]
            },
            {
               "long_name" : "Indonesia",
               "short_name" : "ID",
               "types" : [ "country", "political" ]
            },
            {
               "long_name" : "12750",
               "short_name" : "12750",
               "types" : [ "postal_code" ]
            }
         ],
         "formatted_address" : "Kalibata City, Jl. Raya Kalibata No.1, Rawajati, Pancoran, Kota Jakarta Selatan, Daerah Khusus Ibukota Jakarta 12750, Indonesia",
         "geometry" : {
            "bounds" : {
               "northeast" : {
                  "lat" : -6.2557049,
                  "lng" : 106.8546168
               },
               "southwest" : {
                  "lat" : -6.258145,
                  "lng" : 106.8493882
               }
            },
            "location" : {
               "lat" : -6.2572684,
               "lng" : 106.8521429
            },
            "location_type" : "GEOMETRIC_CENTER",
            "viewport" : {
               "northeast" : {
                  "lat" : -6.255575969708497,
                  "lng" : 106.8546168
               },
               "southwest" : {
                  "lat" : -6.258273930291502,
                  "lng" : 106.8493882
               }
            }
         },
         "place_id" : "ChIJJ1jytbLzaS4RcqOnV26PRPs",
         "types" : [ "premise" ]
      }
   ],
   "status" : "OK"
}
"""

Crystal :

require "http/client"
a = HTTP::Client.get("https://maps.googleapis.com/maps/api/geocode/json?&address=Kalibata%20City", headers: HTTP::Headers{"User-Agent" => "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0", "Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language" => "en-US,en;q=0.5", "Accept-Encoding" => "gzip, deflate, br", "Referer" => "https://google-developers.appspot.com/maps/documentation/utils/geocoder/embed", "Connection" => "keep-alive", "Upgrade-Insecure-Requests" => "1", "Cache-Control" => "max-age=0"})
puts a.body

# result (200) :
# VM@+:Y^
QwGL����zۥ;O&zU!dDp}E?t-EF1#Sat
PHq
V@@D1DoR6N.j)z76+]Ig(a<n32LX.;xt(Mh"g)[
@! xI`*TD$!+y/,-yϙ<3xL^0*gV~[s8
MWKjwcr4kb*:Qm."ߒn8
9*{SQP3TVR7Eh_E
#    Du5{ZGԡE;i,YFk4j_<hT   U)8>jixRm1y`<X&xJDC˱<s49+ii8#onimD}S6B
$Q9q<X}
#                                                                            YПv.~ @G3X̕a}k 

• lolno

[codenoid]

@waj @ysbaddaden @RX14 the endpoint has not been verified ?

UPDATE : when maps.googleapis.com returning 400 status code, the response.body is not encrypted (just like usually response body / html page that showing a error)

Replace %20 with a space :

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (Bad Request)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>That’s an error.</ins>
  <p>Your client has issued a malformed or illegal request.  <ins>That’s all we know.</ins>

[RX14]

@codenoid it's because you specify Accept-Encoding in the request headers. You're telling crystal you accept the gzip encoding, and so crystal gives you back your gzip encoding.

Please don't copy your browser's headers into crystal code, just

require "http"

HTTP::Client.get("https://maps.googleapis.com/maps/api/geocode/json?&address=Kalibata%20City")

is enough and you get a nice response by default.

[codenoid]

wat, why it's work ? because when i try with simple GET, i got 400 :/ aarrrg

UPDATE

at 02:30 and i;m tired, i don't encode the url

UPDATE

at 08:59, i try @RX14 code, and it's give me a wanted response, so fixed by sleep :+1:

UPDATE

I sleep at 05:00

UPDATE

I react :hooray: to @waj comment

[sam0x17]

I am also experiencing this issue here in the context of a docker image running in lambda via up.sh: https://github.com/Sija/raven.cr/issues/47#issuecomment-520280743