Cockroach: security: need a license for HTTP access #42567

[knz]

tldr: the security fix for #42567 has been backported to 2.1, 19.1 and 19.2, however it also breaks legitimate use cases. We need to find a workaround.

(CockroachCloud users are not affected.)

Background

Issue #42567 describes how previous versions of CockroachDB did not authorize accesses to the status endpoints including e.g. /admin/_v1/logs so any non-root user could access it.

This security vulnerability has been patched by requiring all such access to be authenticated as an admin user.

Description of the problem

tldr: without an Enterprise license, the only admin user is root and root is not authorized to use the HTTP endpoints. Therefore, core users without a valid license cannot use advanced HTTP monitoring any more. This is a regression.

Details:

• Only users with a valid password are allowed to log in to the privileged HTTP endpoints, using their password to create a valid HTTP auth cookie via /login (This is a product feature.) CockroachDB does not yet support client authentication using client TLS certificates.

• Meanwhile, root is prevented from having a password (This is a current product feature, discussed in #43847)

• Without an enterprise license, only root is an admin user: An enterprise license is required to make other users members of the admin role.

• This does not affect CockroachCloud customers because all SQL users created with the CC console are admins.

Directions for workaround and solutions

Any of the following product directions would un-block HTTP monitoring on clusters without a license:

1. create a cockroach CLI command that creates/obtains a web cookie for a given user. The CLI command would use a client cert to obtain this cookie. Prototype here: #43872

2. document how to manually create an auth cookie for the root user in system.web_sessions using a custom client program using SQL (possibly providing a Python script)

3. pre-define another user than root that is also an admin in every cluster (e.g. create an operator user using a cluster migration), without requiring a license. This was proposed by @israellot in https://github.com/cockroachdb/cockroach/issues/43847#issuecomment-572806638

4. enabling root to have a password #43847; this in turn would enable a root client to obtain an auth cookie via the /login endpoint then use that to perform further monitoring.

5. enabling authentication using TLS certs in addition to web cookies, so that a client can perform HTTP monitoring using the regular root client cert.

6. open a separate HTTP port for insecure (non-authenticated) status/debug endpoints, and let the DBA/sysadm manage security at the network level (i.e. with firewalling)

| Solution | Complexity | Suitable for backport in 19.2/19.1/2.1 |
|----------|------------|----------------------------------------|
| 1 - new CLI command #43872 | small/medium | yes |
| 2 - document manual cookie creation | medium | yes
| 3 - predefined operator user | small | unsure/yes |
| 4 - root password #43847 #43893 | small | unsure/no |
| 5 - http auth via client TLS certs | medium | no |
| 6 - separate insecure http port | small | unsure/no |

(we can add more solution design points to this list)

[knz]

@rolandcrosby @aaron-crl can you have a look at the above

[knz]

There is a workaround:

1. obtain an evaluation license
2. create an "operator" user and grant it the admin role
3. let the evaluation license expire

Then it is still possible to continue to use the "operator" user.

[RoachietheSupportRoach]

Zendesk ticket #4375 has been linked to this issue.

[knz]

Created a doc PR to update the past release notes: https://github.com/cockroachdb/docs/pull/6321

[aaron-crl]

I recommend against solution 6 - "separate insecure http port" as it may:
(i) Enable credential theft
(ii) Expose sensitive information

Otherwise, I think the other solutions are addressed in relevant tickets or present no addition security concerns.

[wzrdtales]

There is a workaround:

1. obtain an evaluation license

2. create an "operator" user and grant it the admin role

3. let the evaluation license expire

Then it is still possible to continue to use the "operator" user.

Easier one posted here before #43691:

A workaround would be:

insert into role_members (role, member, "isAdmin") VALUES ('admin', 'username', true);

And then start through all nodes.