Materialize: Secure the network interface

[umanwizard]

This is much less complete than full RBAC (which is tracked in https://github.com/MaterializeInc/materialize/issues/677), but it will already handle the access/security needs of a significant fraction of users that aren't well served now by our "house with no doors" model. Thanks @krishmanoh2 for reporting.

[benesch]

This is proposing a specific solution to a problem that has not been well motivated. Specifically, what class of user does not need RBAC but needs the ability to password protect their Materialize instance?

The only class of user I can think of is someone who wants to expose their Materialize instance to the world, or at least within a part of their internal network that is not trusted. But Materialize is _not_ secure if you expose port 6875. Full stop. There are several services exposed on the port: HTTP, pgwire, and inter-node traffic. Even if we secure pgwire, you can still exfiltrate data over HTTP. Even if we add authentication to HTTP, you'll _still_ be able to exfiltrate data by pretending to be another Materialize node and sending raw peek requests to the worker threads. This is absolutely a problem in many deployment models, but that's what #677 is tracking.

Even if we _do_ make Materialize secure enough to expose to the world, password authentication is a pretty weak way to do so. I'd much prefer to use TLS client authentication rather than passwords.

Anyway, even this response is putting the cart before the horse. We need to gather more data from prospects. What specifically are their requirements?

[umanwizard]

Right -- we would need to address those issues in order to make this task useful -- if authentication is turned on, you also need to lock down everything else that can access 6875. I'm not suggesting have a password that only locks down the psql interface.

That said, I think it might be worth doing; I can easily imagine deployments where it's not feasible to "push the authentication into the network layer", i.e. have the set of trusted machines be exactly those which can route packets to 6875. For example, a setup where all of a company's assets are on the same VPN, but not all of them are trusted to interact with Materialize.

Anyway, these issues (gating access to HTTP, etc.) will _still_ be needed to implement full RABC, so I see this issue as a subset of that.

Assigning to Arjun and Krishna to help further define the product requirements.

[rjnn]

i believe the ask here is to perform the "step zero" prerequisites that would also be needed in any case for RBAC - i.e. there is a single role, its name is "admin", and it is allowed to do everything. "Admin" is identified by a single authentication mechanism - @umanwizard assumes a root password, which I believe is plausible but premature - and without this authentication mechanism, nothing else should be accessible.

As a result, other accesses (e.g. other nodes that issue peeks, HTTP UI) need to have some other form of authentication (CockroachDB's certificate strategy is one possible mechanism).

[benesch]

What I'm getting at is that I think it's an error to tie this to RBAC at all. What I think is being asked for here is the following: make it secure to expose Materialize over the network. Whether "the network" refers to the internet at large or a untrusted swath of an internal network is immaterial, I think.

Is that correct? If so I think the correct solution is to ensure that enabling the TLS support that we have today also turns on a requirement for client authentication. I don't think we even need to talk about a default "admin" role, or anything like that.

[benesch]

Whether "the network" refers to the internet at large or a untrusted swath of an internal network is immaterial, I think.

Actually, I take this back. It would be helpful to have more information about why users want this. E.g., do they still want unauthenticated access to the Prometheus metrics? Does that need to be behind a toggle? You probably don't want unauthenticated metric access if exposing to the internet, but maybe you do if exposing to a corporate network.

[rjnn]

That seems correct.

[krishmanoh2]

As end-user, would prefer to control read-write access to materialize with an authentication method that is easy to implement and standalone.

[benesch]

As end-user, would prefer to control read-write access to materialize with an authentication method that is easy to implement and standalone.

Could you expand on what authentication methods you consider easy to implement and standalone?

[krishmanoh2]

Password authentication - Setting root password with alter user or equivalent which end-user would be able to automate implementation as needed.

[benesch]

We're still talking about a specific solution (a root account with a password), but still no one has explicitly stated the problem that we are trying to solve. I wrote up a list of questions that I think will make things more clear.

• What class of user needs this feature?
• Why does the existing authentication strategy not work for these users? (The existing strategy is controlling access to Materialize by controlling access to the host/port on which Materialize is listening.)
• Are these users planning to expose Materialize within a corporate network, or to the internet at large?
• Which of the following interfaces do the users want to be controlled by the new authentication mechanism?
  
  
  
  • Web UI
  
  
  • Prometheus metrics
  
  
  • SQL interface
  
  
• Can these users be coaxed into using mutual TLS rather than passwords? Passwords are less secure than mTLS.
• If these users cannot use mutual TLS, why not?

[krishmanoh2]

What class of user needs this feature?
Any user deploying materialize would benefit from this.

Why does the existing authentication strategy not work for these users? (The existing strategy is controlling access to Materialize by controlling access to the host/port on which Materialize is listening.)
The existing method is not strictly authentication, but enforcing restrictions on access via the network. It requires a network admin to be aware and then take action to enable this method. It also does not prevent a user from logging into materialize if they have access to the network. In reality, having both would be the ideal choice.

Are these users planning to expose Materialize within a corporate network, or to the internet at large?
This can be corporate or internet or both. We cannot control how an user will expose materialize.

Which of the following interfaces do the users want to be controlled by the new authentication mechanism?
Web UI
Prometheus metrics
SQL interface

At the very minimum, the SQL interface. The rest currently are mostly read-only and do not offer the ability to make/see changes to the data (please correct if mistaken).

Can these users be coaxed into using mutual TLS rather than passwords? Passwords are less secure than mTLS.
Mutual TLS would require some bit of prepwork. This can be an option, passwords would be the simplest method.

If these users cannot use mutual TLS, why not?
Mutual TLS requires the client and materialized to be configured correctly. In comparison to other traditional products, this could be viewed as a hassle and users may not use this feature. On the other hand, we would want to encourage strongly the users to set the root password.

[benesch]

Are these users planning to expose Materialize within a corporate network, or to the internet at large?
This can be corporate or internet or both. We cannot control how an user will expose materialize.

Which of the following interfaces do the users want to be controlled by the new authentication mechanism?
Web UI
Prometheus metrics
SQL interface

At the very minimum, the SQL interface. The rest currently are mostly read-only and do not offer the ability to make/see changes to the data (please correct if mistaken).

The web UI now permits issuing arbitrary SQL queries, so it's as dangerous as the SQL interface. Metrics are less sensitive, but still expose information like the names of the available sources.

The reason I ask about exposing to corporate networks vs. the internet at large is because I think it's probably safe to expose /metrics on a corporate network, but not the internet at large. The latter will probably require some form of authentication on /metrics. I agree that we can't control how users will deploy Materialize. I'm just trying to get a sense for what we need to build and when.

Can these users be coaxed into using mutual TLS rather than passwords? Passwords are less secure than mTLS.
Mutual TLS would require some bit of prepwork. This can be an option, passwords would be the simplest method.

If these users cannot use mutual TLS, why not?
Mutual TLS requires the client and materialized to be configured correctly. In comparison to other traditional products, this could be viewed as a hassle and users may not use this feature. On the other hand, we would want to encourage strongly the users to set the root password.

Yeah, I agree mTLS is more of a hassle, but it's also more secure. My proposal is that we build basic mTLS to start, and add password auth as part of RBAC when required. Password authentication is harder engineering wise, in addition to being less secure, because we'll need some sort of login screen for the admin UI (unless we're willing to live with the ugliness of HTTP basic auth).

[benesch]

Ok, here's the plan.

• Support a minimal form of CREATE ROLE and DROP ROLE. The goal is not to actually support RBAC (#677), but to lay _just enough_ groundwork for RBAC in the future. Only users with the LOGIN SUPERUSER option will be allowed.
• Add a new --tls-ca option which configures the TLS certificate authority that Materialize will trust. By default it uses the system's CA.
• Add a new --tls-mode=<disable|allow|require|verify-ca|verify-full> option which controls whether TLS is allowed and whether client certificates are checked.
  
  
  
  • disable means that TLS connections are rejected. This is the default if no --tls-cert is specified.
  
  
  • allow means that TLS is permitted at the client's option.
  
  
  • require means that non-TLS connections are rejected, but no client certificate verification is performed.
  
  
  • verify-ca means that not-TLS connections are rejected and the client's certificate must be signed by the CA known to the server.
  
  
  • verify-full means that non-TLS connections are rejected, the client's certificate must be signed by the CA known to the server, and the common name field in the certificate must match the name of the connecting user. This is the default if --tls-cert is specified.
  
  

This leaves the door open to also supporting additional authentication methods, like passwords, e.g. via an --auth-method=password that would apply after the TLS certificate verification.

[cirego]

Are there other alternatives that were considered and dropped? I'm curious because I never want to touch a CA file again and I don't know that our customers would be happy doing this either. At Flowmill, we started out using a custom certificate authority and very quickly moved over to using API keys.

Is the CA because that's what the postgres client supports?

[benesch]

So the only alternative that works with pgwire is password-based authentication. I imagine we'll get stuck supporting that eventually, but it's much easier to mess up security with password-based schemes. For example, you have to be certain to enable TLS encryption but not verification to avoid sending cleartext passwords. The idea here is to build the most secure possible scheme first, and then think about weaker security models in response to user complaints.

This approach worked well for Cockroach, which hid most of the commands required to generate the necessary certs behind a cockroach cert command: https://www.cockroachlabs.com/docs/v20.1/cockroach-cert

Can you say more about your pain points running a CA?

[benesch]

Also cc @benLogN

[cirego]

Thanks for the pointer to the cockroach document!

That Cockroach document is very well written. I think the main pain point is called out, but also glossed over, when they say:

After that, you'll keep the key safe and secret; you will not upload it to your nodes.

Operating a CA / keeping the CA safe is a major pain point for people. How do I keep it backed up / share it with my colleagues? Do I become the point person for generating new certificates for other developers on my team or can they self serve their own? How am I distributing the CA to other machines and how is that more secure than API keys that can be rotated? How do I rotate certificates if one has been exposed?

At the risk of sounding reductionist, this scheme with creating your own CA and distributing client certificates seems like a fancy method for sharing really long secrets (passwords). I don't know that it's stronger or better than API keys.

To the other point about which to deliver first, why not deliver the simpler feature first? I'm concerned that asking customers to manage their own CA will cause too much friction in the POC phase, especially if they would have been okay with password-based authentication.

[cirego]

How am I distributing the CA to other machines and how is that _more_ secure than API keys that can be rotated? How do I rotate certificates if one has been exposed?

To follow up on this point for a moment, Cockroach's guide for rotating security certificates seems very similar to the advice you would give someone for when to rotate passwords:

You may need to rotate the node, client, or CA certificates in the following scenarios:

• The node, client, or CA certificates are expiring soon.
• Your organization's compliance policy requires periodical certificate rotation.
• The key (for a node, client, or CA) is compromised.
• You need to modify the contents of a certificate, for example, to add another DNS name or the IP address of a load balancer through which a node can be reached. In this case, you would need to rotate only the node certificates.

[benesch]

To the other point about which to deliver first, why not deliver the simpler feature first?

TLS authentication is the easier feature to implement. There are a couple reasons for that:

• We need to support TLS encryption no matter what. The only alternative to TLS encryption is GSSAPI encryption, and that's definitely lower on the priority list. (PostgreSQL went thirty years before it supported GSSAPI encryption.) So we're already pulling in OpenSSL and hooking it up in all the right places; adding a few more checks to verify the certificates against a CA is quite straightforward.

• We have three separate interfaces to secure: HTTP, pgwire, and "comm" (Materialize's internal communication layer). TLS client authentication is uniform across the protocols, but password authentication requires a protocol-specific handshake: HTTP requires the WWW-Authenticate/Authorization headers; pgwire requires a special password flow; comm is under our control but would require some sort of password notification.

  • In particular, we'd need to write quite a bit of SASL/SCRAM related glue code to get password authentication working with pgwire. Here's the PostgreSQL implementation, for reference: https://github.com/postgres/postgres/blob/master/src/backend/libpq/auth-scram.c

• Not using mTLS to authenticate internode communication means you'll need to supply a "cluster secret" to every node on startup, which you'll have to generate yourself, and the security of your cluster will depend on you generating an appropriately secure cluster secret.

I definitely hear what you're saying about CAs being irritating to deploy in small and medium-sized organizations. The OpenSSL commands required are arcane, and if you're just gonna share the CA around that's not any better than just sharing passwords around. But I feel much more confident about our ability to deliver a secure mTLS-based authentication scheme in the short term than I do about our ability to deliver a secure password-based authentication scheme.

For context, the motivating use case here isn't a specific customer but our self-serve cloud console, where we're perfectly happy to take responsibility for generating TLS certificates for our customers.

[benesch]

Also, to be clear, "short term" is like, by the end of the week, and I suspect we'll start building out password support within a few months, too. I figure that could alleviate some concerns!

[cirego]

For context, the motivating use case here isn't a specific customer but our self-serve cloud console, where we're perfectly happy to take responsibility for generating TLS certificates for our customers.

If we're managing the CA, then I withdraw any objections. Especially if TLS makes it easier to implement / integrate all of the existing connection methods.

[benesch]

Your points are well taken though! Both you and @krishmanoh2 are 👎🏽 on recommending that customers use TLS CAs in their own deployments, so I am mentally bracing for doing password authentication a bit sooner than anticipated.

[benesch]

Support for TLS authentication has landed. I'll file a follow-up issue about password authentication.