Deno: error trying to connect: tls handshake eof

[enjikaka]

For some reason doing a GET on https://copernicus.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2018_WM/MapServer/0?f=json
crashes Deno with:

error: Uncaught Http: error sending request for url (https://copernicus.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2018_WM/MapServer/0?f=json): 
error trying to connect: tls handshake eof

Works fine with curl and client side JS. Something wrong with SSL management in Deno? Calling on HTTP doesn't work either since Deno upgrades to HTTPS.

Steps to reproduce

Source:

const layerInfoURL = 'https://copernicus.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2018_WM/MapServer/0?f=json';
const json = await fetch(layerInfoURL).then(r => r.json());

console.log(json);

or

deno run --allow-net https://deno.land/x/gh:enjikaka:terrain-server/poor_api.ts

Version

deno 1.1.1
v8 8.5.104
typescript 3.9.2
macOS 10.15.5 (19F101)
also fails in docker on hayd/alpine-deno:1.1.1

[zinthose]

I'm having the same issue. ATM, I'm dead in the water as this is a blocking issue for me to adopt Deno.

I have a POST request to a remote REST API that uses a standard GoDaddy Cert so it's not self signed like I'm seeing in other issues. From my understanding the issue's root is with the TLS method used but I'm unclear how to change or correct it.

I can successfully make the call using Node or GO. The Go code used for reference is below with certain details changed due to security and confidentiality issues.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "strings"
)

func main() {

    url := "https://api.test.com/webapi/api/session"
    method := "POST"

    payload := strings.NewReader("{\"UserId\":\"[email protected]\",\"Password\":\"P@$$w0rd\",\"SourceSystem\":\"BUDDY\"}")

    client := &http.Client{}
    req, err := http.NewRequest(method, url, payload)

    if err != nil {
        fmt.Println(err)
    }
    req.Header.Add("Content-Type", "application/json;charset=UTF-8")

    res, err := client.Do(req)
    defer res.Body.Close()
    body, err := ioutil.ReadAll(res.Body)

    fmt.Println(string(body))
}

[enjikaka]

I temporarily solved it on my end by proxying through another cert. https://github.com/Freeboard/thingproxy

[zinthose]

I did try adding cors and origin to my request headers using soxa but to no effect.
The response headers from the server I'm attempting to connect to is below if some insight can be gleamed from it that I'm not seeing:

Access-Control-Allow-Origin: *
Cache-Control: no-cache
Content-Length: 461
Content-Type: application/json; charset=utf-8
Date: Thu, 25 Jun 2020 12:52:13 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

@enjikaka I'll look into thingproxy as a temporary solution.

[enjikaka]

@zinthose CORS doesn't matter here. It's the HTTPS management in Deno and/or the rust lib it uses that doesn't handle specific certs correctly. With that thingproxy it'll use another cert that Deno doesn't error on, which is a temporary hack around the issue. :)

[enjikaka]

To compare here's curl -v with the plain server vs proxied response:

curl -v https://copernicus.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2018_WM/MapServer/0\?f\=json

*   Trying 87.54.7.100...
* TCP_NODELAY set
* Connected to copernicus.discomap.eea.europa.eu (87.54.7.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.discomap.eea.europa.eu
*  start date: Apr 20 00:00:00 2020 GMT
*  expire date: Jul 23 00:00:00 2022 GMT
*  subjectAltName: host "copernicus.discomap.eea.europa.eu" matched cert's "*.discomap.eea.europa.eu"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
> GET /arcgis/rest/services/Corine/CLC2018_WM/MapServer/0?f=json HTTP/1.1
> Host: copernicus.discomap.eea.europa.eu
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 14388
< Date: Thu, 25 Jun 2020 13:25:46 GMT
< Content-Type: application/json;charset=UTF-8
< ETag: 34ce7b3
< Server: Microsoft-IIS/10.0
< Server:
< Cache-Control: private, must-revalidate, max-age=0
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
<

curl -v https://thingproxy.freeboard.io/fetch/https://copernicus.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2018_WM/MapServer/0/\?f\=json

*   Trying 52.5.163.236...
* TCP_NODELAY set
* Connected to thingproxy.freeboard.io (52.5.163.236) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.freeboard.io
*  start date: Jun 15 00:00:00 2020 GMT
*  expire date: Jul 15 12:00:00 2021 GMT
*  subjectAltName: host "thingproxy.freeboard.io" matched cert's "*.freeboard.io"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7feb5e00e800)
> GET /fetch/https://copernicus.discomap.eea.europa.eu/arcgis/rest/services/Corine/CLC2018_WM/MapServer/0/?f=json HTTP/2
> Host: thingproxy.freeboard.io
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< content-type: application/json;charset=UTF-8
< content-length: 14388
< x-aspnet-version: 4.0.30319
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< vary: origin
< cache-control: private, must-revalidate, max-age=0
< server: Microsoft-IIS/10.0
< etag: 34ce7b3
< date: Thu, 25 Jun 2020 13:29:03 GMT
<

I think ALPN, server did not agree to a protocol is the issue here?

[zinthose]

I did a DIFF compare with the results below. It dose appear the protocol is the issue.

[zinthose]

FYI: I created a thingproxy docker image for the time being as an alternative workaround for others.

[zinthose]

_I know it seems silly to put so much effort into a workaround, but here you go anyway._ 🤪

I created a module, zinthose/thingproxy-deno , that can replace the fetch api to automatically forward requests through a thingproxy server.

⚠️ The module is fairly simple but may still have bugs. ⚠️

Example

const response = await phetch("https://postman-echo.com/get?foo1=bar1&oo2=bar2"); 
console.log(response.status); 
// e.g. 200 console.log(response.statusText); 
// e.g. "OK" const jsonData = await response.json();

[jacobgc]

Looking at the rustls library page, it looks to only support AESGCM not AES.

The europa.eu site only supports weak ciphers that rustls doesn't support.

Not really sure where to go from here / suggest. @ry got any ideas?

[enjikaka]

It's quite a showstopper for using Deno if you can't consume certain third-party APIs... Will be bad for adoption.

Is rustls planning support? Otherwise I guess an alternative needs to be evaluated.

[jacobgc]

I agree completely. In an ideal world though these sites would upgrade their SSL certs to more modern ciphers. However this isn't going to happen. I dont think rustls has any plans to add in AES without GCM however I'll raise an issue and see what I get back.

[jacobgc]

Got a reply at https://github.com/ctz/rustls/issues/381 The host runs an old version of IIS and thus has old certificates that just aren't supported. Not sure what else to suggest. They need to move with the times. For now, I'd suggest keeping with the proxy setup you've got going.

[Spoonbender]

As long as Deno uses rustls as its TLS library exclusively, it inherits the constraints imposed by rustls. Rustls is quite opinionated about security.

An alternative would be to enable Deno to use another TLS crate, like native-tls (that uses schannel on Windows and OpenSSL on Linux), which supports a very wide range of ciphersuites and certificates and protocols - including those that have known vulnerabilities and weaknesses. Perhaps with some "feel bad" command line flag, like deno run --allow-weak-and-broken-ciphersuites.
Then again, since Deno is "secure by default", maybe Deno should be opinionated against supporting such connections, and this issue should be closed as "won't fix".

[jacobgc]

The connections are still secure, just not as secure as more modern cipher suites. I do like the idea of using a wider supported SSL library like native-tls. Not sure exactly if you could block specific cipher suites though.

[Spoonbender]

@jacobgc the native-tls crate won't let us control the exact ciphersuites, but it does enable controlling the min and max TLS protocol version, trusted root certificate, and whether to accept or reject invalid certificates.

[jacobgc]

@Spoonbender Certainly a much better solution than what Deno currently has.

[Nexxkinn]

Hello, any progress over this case?

[lucacasonato]

No.