Aws-app-mesh-roadmap: Support SPIFFE/SPIRE for mTLS

[paavan98pm]

Tell us about your request
What do you want us to build?
SPIFFE standard integration for bootstrapping and issues identities to services to establish and accept connections with other SPIFFE-compliant systems.

Which integration(s) is this request for?
App Mesh

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Assigning identities to workloads and establish end-to-end encryption between services using the standard identities.

What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.
Enables integration across services that use the SPIFFE identity standard. SPIRE has support for AWS through a Secrets Manager plugin (https://github.com/spiffe/spire/blob/master/doc/plugin_server_upstreamca_awssecret.md)

Are you currently working around this issue?
How are you currently solving this problem?

Additional context
Anything else we should know?

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[bcelenza]

Hey @paavan98pm, out of curiosity, where would you be running your workloads? At the moment it looks like node attestation for Fargate is not present (https://github.com/spiffe/spire/issues/558), but Spire should support EKS, ECS on EC2, and plain EC2 today.

[rodrigodiez]

We would also love to see this implemented. In our case workloads will be running on ECS Fargate

[evan2645]

👋 just popping in to raise my hand, happy to help out on this one if I can.

[efe-selcuk]

@evan2645 :wave: A bit of lack of updates here. Here's what we're leaning towards.

The two things we need are:

1. Configure expected Subject Alternative Names, since the certs are user-provided. This is something we need for mTLS (#34) anyway.
2. Configure envoy to pull certs and validation contexts from secrets discovery service via unix domain socket. Envoy can then be pointed to SPIRE.

The configuration is slightly tricky as SPIRE is vending the certs, and we want to make sure SANs are honored. We should be able to do combined contexts like in this spiffe example.

There's certainly a bit of a disconnect between configuring SPIRE to vend certs but also configuring validation via App Mesh. I think at least being able to vend via SPIRE unblocks customers. But as always we're looking for feedback.

[evan2645]

There's certainly a bit of a disconnect between configuring SPIRE to vend certs but also configuring validation via App Mesh. I think at least being able to vend via SPIRE unblocks customers. But as always we're looking for feedback.

Not sure if you're aware of this, or if it will help, but we do have a small utility that can automatically register k8s workloads in SPIRE. The end result is that you can codify the SPIFFE ID in k8s... so maybe there is a path forward there to having only a single place where you define a name.

Right now, it acts as an admission controller, and reads annotations/labels on your podspec. The community is currently working on a new mode that uses a SPIFFE CRD to define names. Open to other ideas on how this could be improved as well.

I don't know much about App Mesh, so sorry if this is off the mark... I guess my above thought could be summarized as "maybe there is a happy path towards naming source-of-truth if folks use the registration utility".

https://github.com/spiffe/spire/tree/master/support/k8s/k8s-workload-registrar

[efe-selcuk]

Thanks for that. I didn't know about it, though we haven't yet dug into this.

Envoy (within App Mesh) still receives the vast majority of its configuration from us, and some of that configuration includes TLS bits: https://docs.aws.amazon.com/app-mesh/latest/userguide/tls.html

External SDS is of course not the only mechanism for retrieving certs, but it would be the first addition to the API where Envoy is getting config both from us and somewhere else (validation context through SPIRE). Our API today is declarative, and we need to make sure there isn't confusion in these mixed scenarios while also providing a good security posture. For example, If I specify SANs in App Mesh, that probably should take precedence over SPIRE, but how do we make sure customers can reconcile those?

A tool along the lines of the registrar might be the appropriate integration point within kubernetes. Though naturally we have to consider our non-k8s customers too.

Hopefully that clarifies a bit.

[sekka1]

I am excited to see this feature. Is there any timeline on when this feature would land?

[efe-selcuk]

@sekka1 We generally don't comment on release dates and don't have a timeline to share right now.

Can you share any information on how you plan to use SPIFFE/SPIRE with App Mesh? For example, what you'd expect for SAN validation between configuring SPIRE vs configuring a virtual node.

[efe-selcuk]

Folks, we've published a proposal for mutual TLS in which we're looking to bundle support for an SDS provider (like SPIRE). You can find it here: https://github.com/aws/aws-app-mesh-roadmap/issues/34#issuecomment-694468392

As of right now, we're still exploring the experience of using SPIRE with App Mesh on AWS-managed compute platforms like ECS and Fargate. There has already been discussion earlier in this thread about this. In the short term, we're evaluating SPIRE on EKS via the App Mesh controller and I'm hoping to report back on that before long.

[efe-selcuk]

Hi all. We just released the mutual TLS feature into preview. https://github.com/aws/aws-app-mesh-roadmap/issues/34#issuecomment-732445868

This includes supporting SPIRE on Kubernetes via the App Mesh Controller. Please do try it out and provide feedback.

Amazon ECS is currently not supported. As mentioned in my previous comment, this is still in exploration.

[efe-selcuk]

Hey everyone. SPIRE support is now generally available in all App Mesh regions via the App Mesh Controller for Kubernetes release v1.3.0:

• https://aws.amazon.com/about-aws/whats-new/2021/02/announcing-aws-app-mesh-controller-for-kubernetes-version-1-3-0-with-mtls-support/
• https://github.com/aws/aws-app-mesh-controller-for-k8s/releases/tag/v1.3.0

Big thanks to @achevuru and @fawadkhaliq