Meshcentral: Server peering fails

[treicadani]

Hi Ylian,

I have created a pair of servers to test the server peering feature, and in config.json i've added for both servers the same config and changing the serverid accordingly to the servers list.

  "MongoDb": "mongodb://user:[email protected]:27017/meshcentral?authSource=admin",
  "MongoDbChangeStream": true,
    "peers": {
      "serverId": "mcapp01",
      "servers": {`
        "mcapp01": { "url": "ws://1.1.1.2:4430/" },
        "mcapp02": { "url": "ws://1.1.1.3:4430/" }
     }
  },

When I start the first server everything is ok, but when I start the second server Meshcentral starts to throw errors like the one below

Loaded web certificate from "https://meshcentral.example.com:443/", host: "meshcentral.example.com"
SHA384 cert hash: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SHA384 key hash: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
MeshCentral Intel(R) AMT server running on meshcentral.example.com:4433.
MeshCentral HTTP server running on port 4430, alias port 443.
ERR: TypeError: obj.ws._socket.getPeerCertificate is not a function
at WebSocket.<anonymous> (/node_modules/meshcentral/multiserver.js:78:106)
at WebSocket.emit (events.js:198:13)
at WebSocket.setSocket (/node_modules/ws/lib/websocket.js:170:10)
at ClientRequest.req.on (/node_modules/ws/lib/websocket.js:665:15)
at ClientRequest.emit (events.js:198:13)
at Socket.socketOnData (_http_client.js:484:11)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
{ Error: Command failed: /usr/bin/node /usr/local/lib/node_modules/pm2/lib/ProcessContainerFork.js --launch 12510
TypeError: obj.ws._socket.getPeerCertificate is not a function
at WebSocket.<anonymous> (/node_modules/meshcentral/multiserver.js:78:106)
at WebSocket.emit (events.js:198:13)
at WebSocket.setSocket (/node_modules/ws/lib/websocket.js:170:10)
at ClientRequest.req.on (/node_modules/ws/lib/websocket.js:665:15)
at ClientRequest.emit (events.js:198:13)
at Socket.socketOnData (_http_client.js:484:11)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at ChildProcess.exithandler (child_process.js:294:12)
at ChildProcess.emit (events.js:203:15)
at maybeClose (internal/child_process.js:982:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
killed: false,
code: 1,
signal: null,
cmd: '/usr/bin/node /usr/local/lib/node_modules/pm2/lib/ProcessContainerFork.js --launch 12510' }
ERROR: MeshCentral failed with critical error, check MeshErrors.txt. Restarting in 5 seconds...
`

[Ylianst]

Ok, I will be checking this.

[Ylianst]

Oh, I just noticed that you are using "ws://" instead of "wss://" for peering connections.

"MongoDb": "mongodb://user:[email protected]:27017/meshcentral?authSource=admin",
"MongoDbChangeStream": true,
"peers": {
  "serverId": "mcapp01",
  "servers": {
    "mcapp01": { "url": "ws://1.1.1.2:4430/" },
    "mcapp02": { "url": "ws://1.1.1.3:4430/" }
  }
}

I had not expected this, but I presume your behind a reverse proxy and don't need to worry about authenticating the servers. I will fix this.

[Ylianst]

Just did a bunch of improvements to peering system in MeshCentral v0.6.87. Update and let me know if it works. The new version supports "ws://" peering connections and many other fixes.

[treicadani]

Yes MC is behind a reverse proxy so I followed the procedure stated in the user guide to use ws:// instead of wss://.

I'm updating the servers now and I will update later my results, thank you for fast action!

[treicadani]

UPDATE:

I have installed the latest version of MC and the servers seems to peer now without any errors not sure if a successful connection between the two was made as there is no message to say servers in peer or something similar would be really helpful.

I've noticed something odd happening, after I've started the 0.6.87 some groups and machines goes missing from the database.

If I import the database again from version 0.6.84 then i start the version I have all the machines and groups.
But when i start 0.6.87 I am missing 4 groups and the machines in those groups.
Then if I start 0.6.84 again the groups and machines are still missing, I have to import the database again, but when I start 0.6.87 again they goes missing.

[Ylianst]

Oh right, I removed the server peering connection messages. You can see them again by adding --debug peer to the command line when running the server.

Your new problem sort of blows my mind, I am not sure what would cause some groups to just go missing (?!?!?!). If you have a test setup and don't mind. Send me the database import file in a private mail and I will import it myself and debug it. My contact info is here. Thanks.

[treicadani]

Hi Ylian,

I have sent you an email with some more details.

[treicadani]

So, the server peering in 0.7.10 works now, but some machines when i click to view desktop stuck on "Setup..." and in the trace i get

08:46:29 - PEER: FTunnel disconnect server2
08:46:29 - PEER: FTunnel2: Soft disconnect
08:46:29 - PEER: FTunnel1: Soft disconnect
08:46:29 - PEER: FTunnel disconnect server2
08:46:28 - DISPATCH: DispatchEvent, *
08:45:59 - PEER: FTunnel server2: Connected
08:45:59 - PEER: FTunnel server2: Start connect to ws://[xxx.xxx.xxx.xxx]:4430/meshrelay.ashx?browser=1&p=2&nodeid=node//[NODE_ID]
08:45:59 - RELAY: Relay holding: a4qe7e5ajd (xxx.xxx.xxx.xxx) Authenticated
08:45:59 - COOKIE: Decoded AESGCM cookie: {"userid":"user//[USERNAME]","domainid":"","ip":"xxx.xxx.xxx.xxx","time":1606985051000,"dtime":108793}
08:45:59 - WEBREQUEST: (xxx.xxx.xxx.xxx) /meshrelay.ashx/.websocket?browser=1&p=2&nodeid=node//[NODE_ID]