Vault: Certmonger / SCEP support

[james-masson]

I'm looking at using Vault to issue/manage/renew SSL certificates on large numbers of UNIX systems.

There are already established OS client-side utilities for managing this - what are your thoughts on supporting:

• SCEP - https://en.wikipedia.org/wiki/Simple_Certificate_Enrollment_Protocol

and/or

• Certmonger - https://fedorahosted.org/certmonger/

It looks like Vault is mostly targeting Application SSL certs, rather than OS level.

[jefferai]

Vault is actually targeting _any_ SSL certs. Currently it allows issuing for server/client/code signing uses; for most OS needs the client usage is what you're looking for. In 0.4 (as currently planned) it will gain the ability to handle CSRs and CA usages, which will make it able to do pretty much anything you could want a CA to do while allowing for the strong and flexible policy controls currently baked into Vault.

My guess right now is that SCEP won't be supported, at least not without someone working with us to implement it. Supporting custom protocols is not generally easy to do in Vault; instead Vault tries to present a fairly unified API even while supporting a large array of needs. One possibility might be for an intermediary that knows how to talk to Vault and speak SCEP to be an in-between.

Certmonger is a client utility, so the likely support scenario there is for it to gain the knowledge to talk to Vault.

[james-masson]

Application certs vs OS level certs - I was more commenting about the current state of certificate management for Applications vs. OS - I realise a properly issued cert could be used for either.

OS/Network hardware level cert management is a mature area. Systems and Network people have tools like certmonger and/or SCEP based alternatives, used with a local CA that supports those options - or the Windows people use the inbuilt CA with AD managed issuing policies. Enterprise PKI cert management is more than just what APIs Vault has.

Application developers who are new to PKI ( probably because they've never seen Enterprise-wide PKI ) are typically going to be more comfortable with coding their app specifically for Vault.

Hence my comment that Vault was more aimed at Application level, rather than OS/Network level - "build an API, and wait", rather than join an existing mature ecosystem.

That said, I understand that Vault is new, and that adding a Certmonger back-end, or a standalone SCEP<>Vault bridge is perfectly possible. In fact, had the 0.4 release been available now, I'd probably be coding a Certmonger back-end for Vault myself.

What I'm trying to gauge is your level of engagement with large scale Infrastructure PKI installs. "We have an API", is very different from "Here's Vault (re)issuing certs to thousands of Linux boxes & Cisco devices, here's how to switch from your existing back-end API-driven CA to Vault".

[jefferai]

Until recently I was at a very large company (6k employees) with > 150k machines. While I can't go into details, for most of those machines, I will say that I am a fan of the exact kind of model that Vault is designed to make difficult, on purpose, to encourage non-shared secrets.

For the rest -- by which I mean the daily machines for engineers and sales and management and so on -- for most of them (most machines were OSX and Linux), certificate rotation and installation was handled by a set of shell scripts that simply generated CSR and curl'd them up to the CA; when it was approved, you'd run a different script that would curl them down to your machine and install them into the appropriate location/keychain. This is obviously a workflow that is easy to adapt to Vault.

At the place I was at before then (3.5k employees), you simply created your CSR, went to a web page, pasted it in, waited until it was approved, went back to that page, and retrieved the cert. They could have used a good API.

My point, and the reason I don't have a good answer for you with respect to "your level of engagement with large scale infrastructure PKI installs", is because I know (in addition to the methods described above that I have had personal experience with) of lots of ways that large scale infrastructures are designed and deployed. The specific technologies in the mature ecosystem you describe are not ubiquitous, and at this point we think it serves our customers best to enhance Vault's core capabilities so that it can serve as many needs as possible, rather than focus on any specific client tool.

I don't agree with the idea that providing an API means that we're only application-focused or that we don't understand large scale infrastructure or enterprises (this is HashiCorp we're talking about...). It just means that it is currently up to a system integrator to figure out how Vault fits into their workflow...and in many ways a simple JSON API that can be used from any language or a large number of CLI tools is easier to fit into a workflow than something like SCEP. This does mean that people with existing setups and workflows would need to put in effort to switch, but that's the normal way of things.

FWIW, 0.3 is nearly out, and I am hoping for 0.4 to be out around the end of November.

[james-masson]

Thanks for letting me know your approximate timelines. Likely we'll re-evaluate how our certificate distribution architecture works when 0.4 is released, if we see shortcomings in the one in progress right now.

From the experiences you describe, it sounds like where we differ is Build vs Off-the-shelf in client-side cert management. I definitely didn't mean to imply that Vault couldn't be used to backend large OS level PKI installs - more, is the Vault philosophy & cert issuance concept something you are actively targeting at large scale system wide PKI deployments.

I really like the Vault concepts of very short-lived, frequently rotated secrets (including cert lifetimes) - almost Kerberos ticket style - however, the problem then focuses on client-side management of these certificates. SSL PKI's need for CSRs, signing, transport to a remote CA, the built-in lifetiming, the slight "magic black box" view many people have of SSL PKI - all makes this a little more complicated than password management.

eg. A microservice based application, communicating internally over TLS between multiple different subcomponents, on a large fleet of VMs.

• Using the Vault model to frequently re-issue VM client certificates every 24 hours. ( Aggressive, I know - just to show the problem - but aligned with Vault's frequent key rotation style)
• local cert management detects that client cert is within an hour of expiry
• generates CSR
• Submits CSR to CA
• CA issues cert
• local cert management detects cert issuance, downloads cert
• local cert management has list of local apps requiring the client cert
• lcm injects new cert in all the correct formats into local apps
• lcm triggers reload of apps in question
• Apps carry on functioning

Given the issues I've seen working with clients having "unexpected cert expiry" emergencies ;-) - or not even being able to diagnose the fault - even with cert lifetimes of 1+ years - Cert lifetiming and replacement is not done well currently, in many environments. Imagine the chaos that could be caused by 24 hour cert lifetimes, in environments like these!

The operations I've listed above are fairly simple on their own, and easily scriptable. However, people still get them very wrong, or don't think about them at all. Hence giving people easy off-the-shelf integration points would be a big win in reliability & hence for Vault and security in general in my opinion.

This is an area that Microsoft does have an advantage in, due to the integrated nature of their environments. Automatic cert re-issuance and local install is just a few check-boxes away.... But I doubt many people will use Vault for Windows OS cert issuance.

While I would definitely like to see short lifetime PKI made easy by Vault, without robust client-side support on UNIX, I think you'll see a lot of the existing local cert management bad practice bubble to the surface. Maybe that's a good thing, as it does need to be fixed ;-) I want your idea to succeed, I know the Vault API is great - I just worry the take-up will be less than it deserves because of client-side problems.

Thanks for the healthy debate on this, I've got everything I need.

[jefferai]

Likely we'll re-evaluate how our certificate distribution architecture works when 0.4 is released, if we see shortcomings in the one in progress right now.

FYI, the additions to the PKI backend are being coded right now. Expect a delay with HashiConf, but I imagine within two weeks the enhancements will have landed. You may want to watch #388

almost Kerberos ticket style

Kerberos was indeed a major influence. One of the issues with Kerberos is needing to build in client support everywhere else; Vault uses a similar model but eases it by having APIs that can be used from CLIs up to any language you like.

I mostly agree with your view of the issues with rapid certificate rotation. There are definitely a lot of steps, although I think a lot of the issues come down to delegation of responsibility. For instance, for OS certs, a cron job approach means that nothing has to notice that a certificate is expiring...simply download a new one every 24 hours (perhaps with a certificate lifetime of 28 hours, for safety).

Also, one key to the way Vault handles certificate issuance now, and will handle CSRs shortly, is that the cert will either come back, or not, when the CSR is submitted. You don't need to spin and wait to figure out whether it's been approved and is available. Any time you need that kind of logic, things become much more complicated, so I'm happy that Vault doesn't require this. It does mean that Vault does not allow manual approval of certificates, but the role definitions are quite flexible, and generally for manual approval someone will be looking at a set of policies anyways.

[ixe013]

I really like Vault's single executable model... Maybe such protocol additions could be delivered as external add-ons to the Vault core?

For this particular feature, a SCEP-to-Vault API bridge comes to mind. Some front end would receive requests in SCEP format and use Vault's PKI backend API to service them.

Best part is that it can be implemented completely independently of Vault, in any language. Running on a different host would also avoid giving access directly to Vault to machines with a different level of trust.

[kedare]

I confirm this would be very interesting.
In our case we are replacing ADCS (that we hate and doesn't make sense as the last majority of our infra is Linux and OSX) and the SCEP one of the most important missing features for us.

The objective would be to automate delivery of identity certificate delivery for users and laptops joining the domains (Basically that ADCS gives natively)

Probably we will keep ADCS as intermediate CA just for this usage (with vault as Root CA and intermediate CA for all the other usages)