Meshcentral: certUrl not being hashed

[RussellAult]

Setup

Debian Buster, nodejs v10.15.2, npm v6.14.2

1. $ sudo mkdir /opt/meshcentral && cd /opt/meshcentral (per "MeshCentral2InstallGuide.pdf")
2. $ sudo npm install meshcentral (installs 0.5.0-m)
3. $ sudo npm install [email protected] (meshcentral tries to install this on first run, but can't because it doesn't have the permissions to do so)
4. $ sudo mkdir meshcentral-data && sudo mkdir meshcentral-files
5. $ sudo chown -R meshcentral:meshcentral meshcenral-*
6. $ sudo vim meshcentral-data/config.json as (sanitized, adapted from "MeshCentral2UserGuide.pdf"):
   { "settings": { "Cert": "meshcentral.EXAMPLE.COM", "Port": 8443, "AliasPort": 443, "AgentPong": 240, "TlsOffload": "172.24.9.65" }, "domains": { "": { "CertUrl": "https://meshcentral.EXAMPLE.COM/" } } }
7. Setup reverse proxy on 172.24.9.65 to proxy connections for meshcentral.EXAMPLE.COM to 172.24.9.178:8443
8. $ sudo -u meshcentral node node_modules/meshcentral

Expected results

1. A console message to the effect of Loaded RSA web certificate at https://meshcentral.EXAMPLE.COM
2. Agents can connect to meshcentral

Actual results

1. No console message to the effect of Loaded RSA web certificate at https://meshcentral.EXAMPLE.COM
2. When an agent tries to connect to meshcentral, instead of connecting, a console message on the server is produced:
   Agent bad web cert hash (Agent:b2877b7525 != Server:b2e93e2c1d or e86251c74e), holding connection (172.24.9.65:36495). Agent reported web cert hash:b2877b7525556c323142782f7bf32de210daf8055d5898a47bde2921ab4b7524c70bf8dcc725878d29b9c347bdcffdf9.

Further troubleshooting steps taken

1. Changed certUrl in config.json to a non-existent URI, which resulted in Failed to load web certificate at: "https://172.24.9.165:443/", host: "meshcentral.EXAMPLE.COM", so config.json is being read correctly and meshcentral is looking for the certificate; it's just not doing anything with the certificate it finds

[Ylianst]

Hi. First, thank you so much for posting such a detailed report. I need to give you some GitHub award for best posted issue.

As an added test, can you try something like this:

"CertUrl": "https://172.24.9.65:443/"

That is, put the private IP address (instead of DNS name) in the CertUrl where MeshCentral can make a TLS connection and get the certificate that the MeshAgents would also see when they connect. Of course, a private IP address to the reverse proxy is expected in this case.

You should see a message like

Loading from "https://172.24.9.65:443/", host: "meshcentral.EXAMPLE.COM"

Maybe you did this test already, but let me know.

[RussellAult]

Hi Ylian! I should have mentioned that under troubleshooting steps, but yes I did try it. Either way, on start I get four lines (and only four lines) of console messages:
$ sudo -u meshcentral node node_modules/meshcentral MeshCentral HTTP redirection server running on port 1024. MeshCentral v0.5.0-m, Hybrid (LAN + WAN) mode. MeshCentral Intel(R) AMT server running on meshcentral.EXAMPLE.COM:4433. MeshCentral HTTP server running on port 8443, alias port 443.

The redirection server is showing port 1024 because, with the reverse proxy, I'm not planning to give meshcentral access to well-known ports. (As it stands, I'm planning to not use meshcentral's redirection service at all; my reverse proxy is already doing that for every domain pointed at it anyway.)

I should also mention (although, if I understand the process correctly, it shouldn't matter) that the reverse proxy I'm using is HAProxy (which I've been using for years); when I get this up and running I'm hoping to contribute the necessary settings to use HAProxy with meshcentral for the benefit of other users.

Thanks!

[RussellAult]

Okay, a bit more data I can contribute. I think the issue has to do with obj.loadCertificate starting at line 198 of certoperations.js.

First, I tried something I hadn't tried, which was setting certUrl to an https server with a valid TLS certificate, but not one that is valid for cert. This produced no console message, either.

It appears that tls.connect is working correctly, but that (and this may be a function of the reverse proxy I'm using) the this.end; in the callback doesn't actually cause the socket to get to the 'end' status.

Incidentally, I might have uncovered an unrelated bug, because it seems like the current obj.loadCertificate isn't checking whether the cert at certUrl is actually valid for cert.

[jiri-novacek]

I have the same issue with my HAproxy (on pfSense) setup. I didn't have time (and skill) for making such a beautiful report like RussellAult did, and my googling didn't help me much. After reading about same issue on reddit I adapted workaround, where I scripted copying certificate after every LetsEncrypt renewal (through ACME client also on pfSense).
Sorry for being mostly another "+1", I just wanted to confirm there is some issue between MeshCentral's cert fetching and HAproxy. I am open to any testing if necessary, looking forward to ditching my ugly workaround.

[Ylianst]

Ok, going to look into this...

[Ylianst]

I am installing HAProxy. Does anyone has a sample config.json for HAProxy that would work for MeshCentral? (feel free to remove any private data from the config file). Posting it here would be helpful. I want to add HAProxy to my regular testing.

[RussellAult]

Here's roughly what I'm using (sanitized); this is for HAProxy 1.8.19 running on Debian Buster (in this configuration only the global section should be platform-dependent):

````
$ cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon

# https://ssl-config.mozilla.org/#server=haproxy&server-version=1.8.19&config=intermediate&openssl-version=1.1.1d
# generated 2019-10-29
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

# curl https://ssl-config.mozilla.org/ffdhe2048.txt > /etc/haproxy/dhparam.pem

ssl-dh-param-file /etc/haproxy/dhparam.pem

# Maximum connections
maxconn 250

defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
timeout connect 300s
timeout client 300s
timeout client-fin 300s
timeout server 300s
timeout tunnel 1h

frontend http
bind *:80
redirect scheme https code 301 if !{ ssl_fc }

frontend https
bind *:443 ssl crt /etc/haproxy/certs/EXAMPLE.COM.pem
http-request add-header X-Forwarded-Proto https
acl acmepath path_beg /.well-known/acme-challenge/
acl meshcentralhost hdr(host) -i meshcentral.EXAMPLE.COM
use_backend acme if acmepath
use_backend meshcentral if meshcentralhost

backend acme
server certbot localhost:54321

backend meshcentral
http-request add-header X-Forwarded-Host %[req.hdr(Host)]
server meshcentral MESHCENTRAL-SERVER:8443
````

A couple things to note about this configuration:

• HAProxy is expecting to see ssl-dh-param-file at /etc/haproxy/dhparam.pem; it can be downloaded using the commented instructions in the .cfg file
• I've left in the code for handling acme challenges
• HAproxy is expecting see a combined TLS certificate and key at /etc/haproxy/certs/EXAMPLE.COM.pem; while some acme clients will do this for you, simply cating the certificate and the key together into a single .pem will also do the trick

Also, I should mention that, despite trying "IgnoreAgentHashCheck": true, I haven't actually been able to get any of the interactive stuff within meshcentral working yet (the browser interface just says Setup... then goes back to saying Disconnected). All of the agent information is populating, though. I'm not sure yet if this is a reverse proxy issue or something else.

[Ylianst]

Just published MeshCentral v0.5.0-n with some fixes with CertURL and added more debugging messages. You can do:

node meshcentral --debug cert

The --debug cert will display added messages. For example:

node meshcentral --debug cert
MeshCentral HTTP redirection server running on port 80.
CERT: loadCertificate() - Loading certificate from google.com:443, Hostname: central.mesh.meshcentral.com...
MeshCentral v0.5.0-n, Hybrid (LAN + WAN) mode.
CERT: loadCertificate() - TLS connected, got certificate.
Loaded web certificate from "https://google.com", host: "central.mesh.meshcentral.com"
  SHA384 cert hash: fd9c14eead0...
  SHA384 key hash: 89d965...
MeshCentral Intel(R) AMT server running on central.mesh.meshcentral.com:4433.
MeshCentral HTTPS server running on central.mesh.meshcentral.com:443.

If you can give this version and try and let me know what happens. I did modify loadCertificate() to address some issues.

[RussellAult]

The changes you made seem to have solved the problem! meshcentral is now pulling and hashing the certificate correctly. Now I just need to figure out why none of the interactive stuff (except the agent console) is working. Were you able to get meshcentral working behind HAProxy?

Thanks!

[Ylianst]

My developer machine runs Windows and I have not found a recent version of HAProxy that runs on Windows (1.7 is the last one I can find). So, I have not tried HAProxy yet.