Teleport: AC-10 - Concurrent Session Control

[benarent]

What happened:
Customers have a requirement to meet https://nvd.nist.gov/800-53/Rev4/control/AC-10, "Concurrent Session Control"

Organizations may define the maximum number of concurrent sessions for information system accounts globally, by account type (e.g., privileged user, non-privileged user, domain, specific application), by account, or a combination. For example, organizations may limit the number of concurrent sessions for system administrators or individuals working in particularly sensitive domains or mission-critical applications. This control addresses concurrent sessions for information system accounts and does not address concurrent sessions by single users via multiple system accounts.

What you expected to happen:
Teleport Admins can set limits on number of sessions per user via RBAC as max_concurrent_sessions this would allow for customers to set session requirements per role.

e.g. set a policy is 3 sessions for privileged users and 2 for non-privileged

Proposal:

For our customers currently undergoing FedRAMP, they required control over AC-10 Concurrent Session Control.

For example, organizations may limit the number of concurrent sessions for system administrators or individuals working in particularly sensitive domains or mission-critical applications. This control addresses concurrent sessions for information system accounts and does not address concurrent sessions by single users via multiple system accounts.

Advanced Session Control - Public

[klizhentas]

This setting should be enforced on a cluster though, so that's a technical difficulty, also there is a timeline of 90-180 days to deliver this.

[klizhentas]

@fspmarshall some work done here:

https://github.com/gravitational/teleport/commit/1dfc4bb9224fac3f483e3cb42a0a1cccb7d938b0

[fspmarshall]

Note to myself and any interested party:

I've been debating the model for deciding the max_concurrent_sessions value for an individual user given the parameters laid out above. My current opinion is that max_concurrent_sessions should be defined at the role level, and be part of the options block (as opposed to an allow/deny blocks). Furthermore, I think that the value of max_concurrent_sessions for a specific user should be determined as the minimum of the specified values.

I can elaborate on my reasoning, but TLDR is that I think this is the model that is least susceptible to human error, and has the fewest specific downsides. It is also the least expressive option, but I don't see a way to make it more expressive without compromising obviousness or inviting user error.

EDIT: One unresolved issue here is whether the value of max_concurrent_sessions should be subject to local or global authority... if the maximum is 3 at the time a user's certificate is issued, and then the maximum changes to 4, should the value enforced be 3 or 4? The semaphore primitive built by @klizhentas assumes a global maximum, and a global maximum is more intuitive in some ways, but that is in conflict with the current model by which other parameters (e.g. max_session_ttl) are handled, which treats the certificate as the source of truth. Furthermore, enforcing a global requirement introduces complexities around how to handle cases where the max_concurrent_sessions value decreases.

[pmoust]

@klizhentas @fspmarshall any updates on this? max_concurrent_sessions on a per user level is a compliance requirement that comes up a lot.

[fspmarshall]

@pmoust This feature is schedule for the next major release, 4.3.

[fspmarshall]

@benarent, @russjones, @klizhentas: I'm partway through implementation of a Concurrent Session Control MVP. Below is a summary of the in-progress design, and major outstanding questions. Thoughts/feedback appreciated:

Configuration

Roles must be able to specify some kind of concurrent_sessions value, which constrains the number of concurrent sessions a user may hold. If no value is specified, no limit should be applied and the number of concurrent sessions should not be tracked (in the interest of HA/performance).

There are two possible strategies for configuring the concurrent session limit:

1. Define a max_concurrent_sessions value under a role's options block. The maximum allowed for a given user would therefore be the calculated as the minimum of the defined max_concurrent_sessions values.

2. Define a concurrent_sessions value under a role's allow/deny blocks. The maximum allowed for a given user would therefore be either the maximum of the limits defined under allow, or the minimum defined under deny, whichever is smaller.

Strategy 1 has the advantage of simplicity, and is consistent with how max_session_ttl is handled, making it more intuitive for existing users. Strategy 2 has the advantage of flexibility and better supports additive composition, whereby a more privileged user may have their permissions
defined in part by holding one or more less privileged roles (e.g. users with role admin might also hold role dev).

I'm still pretty split on the advantages/disadvantages of the above strategies. My current prototype uses strategy 1, but that is subject to change.

Session Tracking

The concept of a Session within CSC will differ slightly from what Teleport's audit log treats as a Session. The Teleport audit log treats interactive terminals or execs as individual sessions, which means that one successfully authenticated connection to a node may produce multiple distinct sessions from the point of view of the audit log. CSC will instead treat an authenticated connection to a node as a "session", meaning that a multi-exec will count as a single session. This strategy is both simpler and less error-prone.

There is also the issue of Trusted Clusters. Separate clusters should be as isolated as possible. Once role-mapping occurs, a remote user should be indistinguishable from a local user for all practical purposes. Therefore, if a teleport session is started by a user with CSC enabled, Teleport should always increment the semaphore in its own auth server.

This leads us to the question of where we should be tracking sessions (i.e. which component of Teleport should increment the semaphore?):

1. Concurrent sessions are tracked at the Proxy:

   • Pros: Forwarding proxies in root clusters can include connections to leaf clusters as part of a user's concurrent session count, effectively allowing for the concurrent session limit of the root to function as a global limit. Also, this strategy will include connections to OpenSSH nodes in the count.

   • Cons: Direct dialing of Teleport Nodes will effectively circumvent concurrent session limits.

2. Concurrent sessions are tracked at the Node:

   • Pros: Direct dialing of Teleport Nodes will be correctly counted, and management of leases is a bit more decentralized.

   • Cons: Implementing any form of global limit will be more difficult, if that feature ends up being a requirement. Also, since nodes are far more numerous than Auth/Proxy servers, reducing their complexity and sensitivity to version differences is probably always desirable.

I am currently leaning toward option 1. There is also the possibility of simply making the tracking location configurable, but I am hoping to avoid that if possible.

Semaphores

In the interest of HA and simplicity, existing sessions will not be terminated if the maximum number of concurrent sessions for a user is reduced while the session is active. By explicitly forgoing this, we can safely allow for intermittent periods of bad Auth Server connectivity, or even full Auth Server rotation, without interrupting ongoing SSH sessions.

Semaphores will be lazily initialized, and the caller attempting to claim a lease will provide the maximum resource count, above which their lease will be denied. Because of the distributed nature of Teleport, different actors may have a different view of what a user's maximum allowed concurrent sessions is. Because all of the user's other capabilities will be judged based on the perspective of the evaluating entity, I think it is most consistent to have the semaphore lease claim evaluated based on the same "view" that the rest of the capabilities use. This does not, however, preclude future additions such as semaphore poisoning which could allow full termination of a all of a user's sessions via starvation (and would maintain backwards-compatibility with this API).

[klizhentas]

@fspmarshall thanks for the thoughtful write-up, a couple of thougts:

Configuration

Regarding configuration, I would go with option 1. As you noted, it is consistent with max_session_ttl and is simple. If there are two roles with two values, the lowest value will be picked from the set.

I would also add global ClusterConfig field with the same name, so administrators can set default value in the config file. This will be helpful for fedramp deployments that can set this by default.

Session Tracking

When recording on the proxy, the question is what if a user opens a tcp-ip tunnel and opens another 5 concurrent sessions? Do we assume that session is any atomic interaction with the server or simply a connection. The description is pretty vague, but in face of uncertainty I suggest we pick the strictest interpretation then. This will force us to put the semaphore tracking on the nodes then.

Semaphores

In the interest of HA and simplicity, existing sessions will not be terminated if the maximum number of concurrent sessions for a user is reduced while the session is active

I agree, I think this is OK. If someone wants to block/revoke user access, they should be using blocking API that should be instantaneous

[fspmarshall]

@klizhentas Thanks for the feedback!

When recording on the proxy, the question is what if a user opens a tcp-ip tunnel and opens another 5 concurrent sessions? Do we assume that session is any atomic interaction with the server or simply a connection. The description is pretty vague, but in face of uncertainty I suggest we pick the strictest interpretation then. This will force us to put the semaphore tracking on the nodes then.

When talking about counting at the proxy, I was talking about counting outbound connections. In the case of a user making multiple tcp-ip requests against the proxy, each one would count as a separate session.

That being said, I see your point. Tracking at the node is more strict, and much less sensitive to network configuration. I'll go with that.

[russjones]

@fspmarshall @klizhentas I was talking about this with @benarent in the morning, since this control will mainly be for people going through FedRAMP, recording at the node is reasonable. That means it won't work for people that are recording sessions at the proxy with OpenSSH nodes, but that's fine.

@fspmarshall This may be out of scope for the MVP, but do you have an idea if we could support Kubernetes as well?

[benarent]

Hey @fspmarshall Nice write up. I did a quick review with RJ, since Sasha covered a lot of parts already. I think the two other product thoughts is how we want to show this in the logs and in tsh.

tsh:

We should provide a user friendly error in the case of a user reaching maximum concurrent sessions.

$ tsh ssh foo@bar
An error occurred (UnauthorizedOperation). You have reached maximum sessions.

Events:

Do we want to keep an internal count of sessions? Such as sid_count=2, or simply send an event if max sessions has been reached. If so, is this a failed session.start event? Or a unique event?

Teleport UI:

Do we want to add this limit to the UI? The node list, audit log can provide some sensitive data about a system. I'm unsure if this is needed, this could be something we ask for feedback once release?

[fspmarshall]

do you have an idea if we could support Kubernetes as well?

My knowledge of our kubernetes integration is pretty rudimentary, but my initial impression is that tracking interactive execs and port-forwards individually would be pretty easy since they are long-lived and have persistent state within the proxy.

Deciding how to handle other requests is a bit trickier. A few options come to mind:

1. Acquire and release semaphore for every HTTP request (doesn't seem at all scalable).
2. Cache leases in memory for a short time s.t. successive requests that hit the same proxy count as part of the same "session" (likely to produce inconsistent behavior, especially if there are more proxies than max_concurrent_sessions).
3. Make semaphore leases named s.t. multiple proxies may "share" a single lease. Combined with caching (2), this would have the effect of treating all successive kubernetes API requests as part of a single ephemeral session (this option has a certain elegance, but it is also the most complicated and least granular).
4. Track execs and port-forwards individually; ignore everything else (not ideal, but this is what the audit log does).

Leaning a bit toward option 3 at the moment, though I think its best to get a working MVP for ssh connections and then go from there.

---

Do we want to keep an internal count of sessions? Such as sid_count=2, or simply send an event if max sessions has been reached. If so, is this a failed session.start event? Or a unique event?

Should be easy to add a method to show the current count tracked by the semaphore, so we can display/log that info however. In terms of events, the MVP as described above doesn't quite line up with how session.start/session.end events are generated. The proposed design will increment/decrement on open/close of the authenticated ssh connection, which may end up containing more than one session.start/session.end event due to the fact that a single authenticated connection may carry multiple exec or interactive terminal sessions (e.g. when using ControlMaster).

I made the decision to start with connection-scoped tracking because I feel that it is simpler, easier to maintain, and maps more directly to the true scope of a user's permissioned actions against a node. I'm open to the idea that consistency is more important tho... we already have an established nomenclature around what a Session is, and having the concurrent session counter not keep parity with session.* events may be too confusing.

[awly]

Could you walk us through this example:

• cluster has 2 proxies
• session limit for user alice is 4
• alice dials 2 sessions through each proxy successfully (total of 4 connections)
• alice attempts to dial a 5th session through one of the proxies and gets rejected

In this scenario, how are session counts incremented and what processes/messages are involved.

We should also consider what happens when a proxy crashes and fails to notify anyone about decremented sessions.

[fspmarshall]

walk us through this example...

@awly is this in relation to the hypothetical discussion about kubernetes API sessions? If so, depends on which of the proposed approaches we go with (if any). If you're asking for an example of how normal SSH control would work, it would go like this:

Alice dials her target nodes (hitting same node or proxy doesn't change anything here, only the number of separate connections to teleport nodes matters). Each time, when her connections hit the target node the node attempts to acquire a semaphore lease from the auth server. This acquisition attempt is performed after the connection has been authenticated (gotta have access to user certs), but before it starts serving the ssh connection (i.e. before exec,env,forward-agent,etc...). For alice's fifth connection, lease acquisition fails and an error is returned. For her first four connections, the lease is successfully acquired and work continues as normal except that a goroutine in charge of periodic lease renewal is spawned. If lease expires without successful renewal (e.g. due to being unable to contact auth server for some time), the connection is terminated prematurely. When connection terminates normally, lease is explicitly released.

what happens when a proxy crashes and fails to notify anyone about decremented sessions.

Per feedback here and here, we're opting to track at the node level instead of the proxy for normal SSH connections. Regardless of location, however, the semaphore system will require periodic renewal of leases, so effectively a failure to decrement will resolve itself after some timeout. I'm just using a modified version of sasha's semaphore work (1dfc4bb), which already handles this pretty nicely.

[russjones]

Leaning a bit toward option 3 at the moment, though I think its best to get a working MVP for ssh connections and then go from there.

Sounds good to me.

[russjones]

Best: 10
Worst: 15