Cylc-flow: ØMQ security: encryption for WFS comms

[sadielbartholomew]

Describe exactly what you would like to see in an upcoming release

At present, communications with the WFS via ZeroMQ are not encrypted. We currently use JSON Web Tokens (via the python-jose implementation):

• the JWT verifies the connection;
• the WFS checks that the JWT was signed using the correct key (previously referred to as the passphrase)

However, the contents (the "payload") of the JWT are not encrypted:

Do note that for signed tokens this information, though protected against tampering, is readable by anyone. Do not put secret information in the payload or header elements of a JWT unless it is encrypted.

JWTs were added as a placeholder, but going forward towards a deployable release we need to add genuine encryption.

Additional context

Note the "Address encryption" item in #2978 largely covers the above, but the statement there that encryption is included was recently found to not be true (comments & function names in the network/authentication.py are therefore currently misleading too).

[For the Issue tracking search (too many aliases!): ZeroMQ, ZMQ, ØMQ, 0MQ]

Thanks to @dpmatthews for raising this.

[sadielbartholomew]

Proposal for a solution: CurveZMQ

Motivation

Whilst there are alternatives to solve this, most simply perhaps to replace the placeholder JWT implementation with a JWE one, we propose to replace with CurveZMQ, which involves generating public/private key pairs for both client and server. This is because it:

• is the ZeroMQ built-in method for encryption, hence standard;
• offers many additional security benefits.

Suggested plan

The main concern is that this approach may have too much overhead. We would like to code up a minimal POC, & profile it to ensure it is not too sluggish to be practicable. For that initial implementation we would:

• generate both key pairs as part of the WFS startup;
• for remote tasks, install two files for the client private key & the server public key, instead of one for the current passphrase;
• use CurveZMQ to provide secure client-server communication via the above steps & also verify the connection to the same level as present with the placeholder JWT code.

If from subsequent testing the overhead is not too heavy, we can go ahead & merge, then we:

• may also be able to use separate client keys for each job;
• would look to improve the WFS authentication (not just the communications to & from it) along the lines proposed here.

Otherwise, we could continue using CurveZMQ to encrypt the communication & then add a separate lighter-weight mechanism (such as JWT) to do the authentication (as per the current proposal in cylc/cylc-admin#41).

[kinow]

Using the ZeroMQ built-in method initially, and profiling it sounds like a good plan :+1: +1 from me

[hjoliver]

Sounds fine to me.

I think we should keep unencrypted as an option if it has a significant performance cost.

would look to improve the WFS authentication (not just the communications to & from it) along the lines proposed here.

The CurveZMQ site blurb says "CurveZMQ is an authentication and encryption protocol for ZeroMQ" - so I'm not sure I understand your comment here??

[oliver-sanders]

I think we should keep unencrypted as an option if it has a significant performance cost.

I think the Python overheads vastly outstrip the ZMQ overheads to this shouldn't be a concern.

[oliver-sanders]

Sounds good.

FYI the outstanding ZMQ matters left-over after the Py3 upgrade were gathered in #2978

ZMQ has recently(ish) acquired "curve" based auth, might this be better

[sadielbartholomew]

Thanks all for the thoughts. @oliver-sanders:

FYI the outstanding ZMQ matters left-over after the Py3 upgrade were gathered in #2978

Yes the 'Additional context' section in the opening comment noted that. Thanks for registering under there. We agreed on Friday that this merits a separate Issue, though. I did forget to link this in on that Issue, sorry, so thanks for doing so.

To clarify, @hjoliver (with apologies, in retrospect 'CLI' was a keyword that was missing):

would look to improve the WFS authentication (not just the communications to & from it) along the lines proposed here.

I'm not sure I understand your comment here??

There are two general cases we are distinguishing with regards to the WFS security (encryption &/or auth) potential for CurveZMQ, communications for the:

1) for the network layer; &
2) between the command line (interactive, jobs) & WFS, i.e. client-server comms.

So the suggestion is that if the basic implementation for case 1 provides profiling results that suggest it could be effective & performant enough, we consider using it (or at least trying it out) for case 2. Does that make sense?

[kinow]

I think we should keep unencrypted as an option if it has a significant performance cost.

I think the Python overheads vastly outstrip the ZMQ overheads to this shouldn't be a concern.

If it's simple enough to measure the performance of both cases, with and without encryption, then maybe this could have a toggle flag in the workflow configuration, and then users could decide to enable/disable it if it's too expensive.

[dpmatthews]

I like to refer to https://developer.ibm.com/tutorials/se-distributed-apps-zeromq-part2/ when trying to understand what we're trying to implement here (can't see it referenced above).