Deno: https connection times out

[timonson]

I have a problem running an https server and I can't solve it on my own. I tried various things which include:

• switching linux distros
• using different ssl certificates
• changing server configurations
• different domains
• comparing to http server
• and more

The problem is that after an indeterminable time, but mostly about 20 hours, the server stops working in the sense that requests just seem to timeout: curl: (28) Failed to connect to xyz.com port 443: Connection timed out. Before this time period, everything works prefectly. But the deno process is still running and the port is still used:

I don't use an nginx proxy server; just deno directly on a linux distro.
I execute deno with sudo and I also tried to start deno with pm2. In all cases it stops working after a couple of hours.

The server runs on AWS Lightsail and I can rule out that a huge amount of requests is causing this (maybe 10 requests per hour).

I can also say with certainty that this does not happen with a deno http server nor with a node https server, which I both tested at least once for two days.

There was no big CPU or memory load which I could see in the monitoring tools. I only respond with hello world to requests at the moment.
If necessary I would be happy to give ssh access to somebody who knows how to debug this issue.

Thanks a lot!

[lucacasonato]

I have spun up a system on GCP (e2-medium, 2 vCPUs, 4 GB memory) with this code:

import { listenAndServeTLS } from "https://deno.land/[email protected]/http/server.ts";

const body = "Hello HTTPS";
const options = {
  port: 443,
  certFile: "/etc/letsencrypt/live/denohttpstest.lcas.dev/fullchain.pem",
  keyFile: "/etc/letsencrypt/live/denohttpstest.lcas.dev/privkey.pem",
};
listenAndServeTLS(options, (req) => {
  req.respond({ body });
});

I have also set up uptime checks to see if it goes down. Let's see if I can reproduce this.

[timonson]

I am using a 512 MB RAM, 1 vCPU, 20 GB SSD machine. But I could not see high resources usage. Otherwise just dm me and I send you my ssh key.

[lucacasonato]

I can confirm this bug. My test server went down exactly 5 hours (on the minute) after starting it. The entire server log:

lucacasonato@deno-https-test:~$ sudo /home/lucacasonato/.deno/bin/deno run -A server.ts
TLS alert received: Message {
    typ: Alert,
    version: TLSv1_3,
    payload: Alert(
        AlertMessagePayload {
            level: Fatal,
            description: CertificateUnknown,
        },
    ),
}
TLS alert received: Message {
    typ: Alert,
    version: TLSv1_3,
    payload: Alert(
        AlertMessagePayload {
            level: Fatal,
            description: CertificateUnknown,
        },
    ),
}
Illegal SNI hostname received [51, 52, 46, 49, 50, 50, 46, 49, 55, 48, 46, 49, 56, 51]
Sending fatal alert DecodeError
Sending fatal alert ProtocolVersion
Illegal SNI hostname received [51, 52, 46, 49, 50, 50, 46, 49, 55, 48, 46, 49, 56, 51]
Sending fatal alert DecodeError
error: Uncaught ConnectionReset: Connection reset by peer (os error 104)
    at processResponse (core.js:226:13)
    at Object.jsonOpAsync (core.js:244:12)
    at async TLSListener.accept (deno:cli/rt/40_tls.js:42:19)
    at async Server.acceptConnAndIterateHttpRequests (server.ts:212:14)
    at async MuxAsyncIterator.callIteratorNext (mux_async_iterator.ts:30:31)
lucacasonato@deno-https-test:~$ 

I am going to rerun the test, but add more metrics to the log so we can tell what happens when.

[rvdende]

I've been battling this one as well.. and its a bit out of my league at the moment. Hoping @lucacasonato you can help! Thank you

I don't know if the Illegal SNI hostname has anything to do with the ConnectionReset though opine might hide that from my log.

[rvdende]

more searching found this: https://en.it1352.com/article/6b9299e70af448e3a86b63562103ae4e.html

Solution
I believe the problem is that you are not fully reading the data sent from the client, so the client never has a chance to transition to reading the response. When it tries to write more data, it notices the socket has been closed and fails.

[lucacasonato]

The server crashed again after a little over 7 hours. Log of server with -Ldebug here: https://0x0.st/ikSM.txt

[rvdende]

In the mean time I am using nginx to reverse proxy https to http instead. That seems to have solved the immediate need for this fix

[rvdende]

12 hours later server crashed on http instead of https, related issue https://github.com/denoland/deno/issues/6319 Uncaught BrokenPipe: Broken pipe (os error 32)

This might mean that Deno is unable to run a web api reliably at this moment?

[tveronezi]

@lucacasonato One of the last lines in this log is...

DEBUG RS - deno::tsc:757 - compiled filename: "/root/.cache/deno/gen/https/deno.land/37e30d67ded9172ae43b9aaedbaeb1788b75e43c942e5ec21f0cc717b20eb0aa.js"

This is the tsc stating it compiled the same file as the initial server.ts. Why did deno compiles that file twice? Actually it also compiled https://deno.land/[email protected]/async/delay.ts before crashing. Maybe that's why we have a broken pipe: deno recompiled the server and its dependencies and didnt properly closed pending connections.

Anyway, the deno::tsc:757 - compiled filename: log message is coming from the previous version of deno. https://github.com/denoland/deno/blob/v1.4.6/cli/tsc.rs#L757

@timonson @lucacasonato @rvdende

Maybe try again with the version 1.5?

[rvdende]

Strange that it would compile so far into the process, what could be triggering this?
Even if deno is running in watch mode it should cleanly restart the process.

[ry]

A guess is that we encounter some TLS error but don't propagate it properly.

We should try setting up a TLS server and sending it some garbage data - does it crash? If so, we can probably easily fix it.

[timonson]

@tveronezi I started a new https server with deno 1.5 after I read your post. It crashed after about 9 hrs again. If you need my Ldebug file, let me know, because I saved it. On the other hand, my http sever keeps running. Only https is crashing.

[mrkurt]

For what it's worth, this seems to happen with the HTTP server and no TLS (just like @rvdende saw). We're running about 2k requests per second through the http server code and it seems like there are a whole bunch of uncatchable errors. It is difficult to keep running. ;)

This PR is getting some of them, but the connection reset error we got still slipped through: https://github.com/denoland/deno/pull/8365

Actually, seems like req.respond is a promise you _can_ handle these on.

[bartlomieju]

I believe this issue will be fixed in https://github.com/denoland/deno/pull/8365

[bartlomieju]

I'm gonna provisionally close this issue because #8365 has landed. Let's reopen if the problem keeps happening.