Envoy: HSM support

[gfrankliu]

What's the plan for HSM support?

[mattklein123]

For what exactly? Loading keys? There is no plan currently. cc @PiotrSikora

[gfrankliu]

Yes, this is for private keys.
Do you have other ways to securely store keys for envoy to load, or is the local filesystem the only way we have put the keys?

[mattklein123]

Right now all we have is filesystem support. The v2 API will allow inline cert delivery from the management server. There are no plans that I know of right now to explicitly add HSM support directly into envoy.

[gfrankliu]

What about Google KMS? Any plans?

[dnoe]

No concrete plans but you're asking at the right time. We (Google) have been discussing various certificate management solutions internally and I'd certainly be interested to hear how you'd like to use KMS with Envoy.

[gfrankliu]

I am currently using nginx with HSM since HSM seems to be industrial standard and is available on-prem and most cloud providers. Just started evaluating envoy and hence the HSM support questions. Without HSM, the next choice is some cloud specific solutions (e.g.: Google KMS). Putting keys on the server filesystem seems least secure.

[mattklein123]

In exciting news it looks like @floatingsidewalk over at MSFT would like to take this on. AFAIK, boringssl already supports callouts for key use during handshake. I think this work is mainly about utilizing that support in Envoy and developing the right interfaces/abstractions for plugins.

I would loosely suggest that we offer loadable "HSM" modules that can be plugged in. E.g., a static module could use a local HW HSM, another could call an external API, etc.

@floatingsidewalk please work with @ggreenway and @PiotrSikora on the design. I would suggest that we do a design proposal for this feature in this issue prior to starting development. Thank you!

[ipuustin]

This can probably be done nicely with the BoringSSL Private Key Methods once https://github.com/envoyproxy/envoy/pull/6326 lands. Looks like https://github.com/tpm2-software/tpm2-tools will be the right lib for a reference implementation. I'll try to get myself a TPM module next...

[ipuustin]

After some more investigation it seems that we might as well do a generic PKCS#11 private key method provider. This would give us the most flexibility regarding the actual backend (just load the SO/DLL implementing the PKCS#11 API) and also be pretty cross-platform. We can also use something like SoftHSM for testing.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[salrashid123]

@ipuustin Is there an example somewhere on how to use the "BoringSSL Private Key Methods"?

From what i can tell, i need to configure a provider but unsure what to do

---

WHat i'm trying to do is have envoy run such that its tls private key is actually backed by the TPM.

I do know how to do this with plain openssl:

for examle, if i use tpm2-tools, i can confgure openssl and curl to recognize TPM-based certificates

$ openssl engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
(tpm2tss) TPM2-TSS engine for OpenSSL

$ curl --engine list
Build-time engines: 
rdrand 
dynamic 
tpm2tss

I can generate a private key that wraped by the TPM:

$ tpm2tss-genkey -a rsa rsa.tss

$ more rsa.tss 
-----BEGIN TSS2 PRIVATE KEY-----
redacted
-----END TSS2 PRIVATE KEY-----

Then finally use that key to startup a simple openssl listener:

$ openssl req -new -x509 -engine tpm2tss -key rsa.tss -keyform engine -out rsa.crt
$ openssl s_server -cert rsa.crt -key rsa.tss -keyform engine -engine tpm2tss -accept 8443

What i'd like to do is run envoy usign that rsa.tss, rsa.crt certificate

[ipuustin]

@salrashid123 : I've been experimenting with a PKCS#11 provider (which can use TPM as the backend, see https://github.com/tpm2-software/tpm2-pkcs11), but been busy with other stuff lately. I know Santosh has done a TPM private key provider here: https://github.com/aruba/envoy-extn-pkm-provider . You could maybe check if you could use that directly or use that as starting point for your own work?

[salrashid123]

thanks, i'll take a look at the extn-pkm-provider. fwiw, i endedup trying openssl EVP (tpm2-tss-engine ) but thats not whats directly applicable now. if you needed a reference for that, i wrote it up here

[dwmw2]

Regarding the use of hardware tokens, and even various different file formats: please see http://david.woodhou.se/draft-woodhouse-cert-best-practice.html

In particular, this stuff should Just Work when the file that the user passes as the key just happens to be a TPM2-wrapped key file. Without any nonsense about configuring "engines" and the like. Likewise if the filename provided isn't a filename at all, but is a PKCS#11 URI (RFC7512).

[iamktm]

Is it possible to support higher level abstractions such as the Windows Certificate Store (via CNG), macOS keychain (via CryptoTokenKit), etc., in addition to PKCS#11? By doing this, users could get easier access to the cryptographic keys and certs currently present in their local stores by making use of the drivers they already have installed. Perhaps this deserves its own issue...

[meringu]

AWS has announced ACM support in for their Nitro instances: https://aws.amazon.com/about-aws/whats-new/2020/10/announcing-aws-certificate-manager-for-nitro-enclaves/

Is this issue what's needed for TLS and mTLS certificates from ACM in Envoy? This would be a big win for us in reducing certificate management overhead.

[PiotrSikora]

@meringu it looks that ACM it's using PKCS#11, so yeah.