Node-mysql2: Cannot connect to Amazon RDS from within Docker container

[jsumners]

I have been banging my head against this for hours and I just do not understand what is going on here. I'm hoping whomever reads this has some idea I haven't had yet. Basically, within the attached zip file is a very simple project that attempts a connection to an Amazon RDS instance, issues a select 1 = 1, and logs the results. If this project is run normally then the desired result is logged (1 = 1). But if this project is run via a Docker container (as shown in the included run.sh script) then the following will be logged:

{ Error: EE certificate key too weak
    at TLSSocket.<anonymous> (/app/node_modules/mysql/lib/Connection.js:320:48)
    at TLSSocket.emit (events.js:198:13)
    at TLSSocket._finishInit (_tls_wrap.js:636:8)
    --------------------
    at Protocol._enqueue (/app/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at Protocol.handshake (/app/node_modules/mysql/lib/protocol/Protocol.js:51:23)
    at Connection.connect (/app/node_modules/mysql/lib/Connection.js:119:18)
    at Object.<anonymous> (/app/mysql.js:29:6)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12) code: 'HANDSHAKE_SSL_ERROR', fatal: true }

mysql-wtf.zip

[sidorares]

Which node version are you using @jsumners ?

[sidorares]

Also, in theory what are you doing with .end() should work ( it just adds COM_QUIT command to command queue ) but could you try to have conn.end(); inside .query() callback and see if that makes any difference?

conn.connect();
conn.query('select 1 = 1', (err, results) => {
  console.log(err || results);
  conn.end();
}); 

[jsumners]

As indicated be the run script: Node 12.13.0.

[sidorares]

I see. Let me spawn test rds quickly and check if I can reproduce

[sidorares]

Can confirm I'm able to reproduce this.

Looks like this is similar to what is described here: https://stackoverflow.com/questions/57456325/https-createserver-error-ssl-ctx-use-certificate-ca-md-too-weak

If you are using Node.js>=10.0.0, it will raise the exception if certs are encrypted by sha1 or md5.

Looking at the certificates in "Amazon RDS" bundle I can see that first 10 are signed with sha1 and rest with sha256

const profiles = require('mysql2/lib/constants/ssl_profiles.js')['Amazon RDS'];
const fs = require('fs');

for (var i = 0; i < profiles.ca.length; ++i) {
    fs.writeFileSync(`cas/ca${i}.pem`, profiles.ca[i]);
}

$ openssl x509 -in cas/ca10.pem -text -noout | grep 'Signature Algorithm'
    Signature Algorithm: sha1WithRSAEncryption
    Signature Algorithm: sha1WithRSAEncryption
$ openssl x509 -in cas/ca11.pem -text -noout | grep 'Signature Algorithm'
    Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption

For combined bundle ssl reports sha1:

$ wget https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem
$ openssl x509 -in rds-combined-ca-bundle.pem -text -noout | grep "Signature Al"
    Signature Algorithm: sha1WithRSAEncryption

I'll need to investigate in two directions: 1) is there an easy way to relax handling of Error: CA signature digest algorithm too weak error on node side 2) contact Amazon and ask if they have plans to update all certs to have sha256 signature ( or maybe there is a way to re-pack them without involwing Amazon?). If you can help with anything above that would be great @jsumners

[sidorares]

The following code works for me:

const mysql = require('mysql2');
const rdsProfile = require('mysql2/lib/constants/ssl_profiles.js')['Amazon RDS'];
const conn = mysql.createConnection({
    user: process.env.MYSQL_USER,
    password: process.env.MYSQL_PASS,
    host: process.env.MYSQL_HOST,
    database: process.env.MYSQL_DB,
    ssl: {
        rejectUnauthorized: false,
        ca: rdsProfile.ca
     }
});

conn.connect();
conn.query('select 1 = 1', (err, results) => console.log(err || results));
conn.end();

Unfortunately rejectUnauthorized: false will allow more than just sha1 signature, I think it'll eat other errors too - see https://nodejs.org/api/tls.html#tls_new_tls_tlssocket_socket_options ( this is how it's used in the driver - https://github.com/sidorares/node-mysql2/blob/f4e7c962052a1c7f314930bb223f4f24fad4c087/lib/connection.js#L307-L310 )

[sidorares]

Interesting that the error I'm getting is different from what you reported. Maybe we need to test exact image tag - can you try with 12.13.0-buster-slim?

[jsumners]

Hmm. The note about 10.x being affected as well lends credence to why I see the same error if I use the centos:latest image and Node via yum install nodejs.

Using rejectUnauthorized: false isn't such a great idea because it will open the door for a MITM attack (I am fairly sure).

I can investigate some more with this new information tomorrow. But I doubt I'll get further than you already have (thank you). Let's tag the only person I know of working on AWS stuff and see what they say. @trivikr can you provide any insight here?

[jsumners]

FYI, both:

• https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem
• https://s3.amazonaws.com/rds-downloads/rds-ca-2019-us-east-1.pem

Yield Signature Algorithm: sha256WithRSAEncryption.

So I'll try constructing a new pem set to pass in via the connection options tomorrow and see what that does.

[jsumners]

Here's the issue:

The RDS server is presenting the 2015 certificates. When the connection is being upgraded, the client verifies the certificates presented by the server. Since the server is presenting the 2015 cert, which is signed with SHA1, the client rejects the _server_ certificate. So the solution is to update the RDS instance to the 2019 certificates.

[jsumners]

One more note: it works just fine under node:12-stretch.

[sidorares]

Can you point to a docs on how to update db instance certs? Can you do that from aws console?

[jsumners]

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html

I do wish I knew why Debian Stretch works. But it’s a mystery I will have to live with.