Linkerd2: Allow trustAnchorsPem ca cert value to be read from Kubernetes Secrets

[rnsv]

Feature Request

Support for trustAnchorsPem ca cert value in linkerd-config to be read from an existing kubernetes secrets

What problem are you trying to solve?

Automating the deployment of linkerd configuration with an existing tls certificate

How should the problem be solved?

Allow alternate way to provide trustAnchorsPem cert. One possible solution is to provide a key trustAnchorsPemSecret: my-namespace/secret-name in identityContext

What do you want to happen? Add any considered drawbacks.

Any alternatives you've considered?

No alternatives

Is there another way to solve this problem that isn't as good a solution?

None

How would users interact with this feature?

User Creates a tls certificate using external provider like cert-manager. The secret name will then is referred in linkerd-config global -> identityContext -> trustAnchorsPemSecret

[grampelberg]

Could you explain a little more about why you need this for only the trust anchors? As it is designed currently, it gets picked up automatically from the identity issuer cert.

[zaharidichev]

@rnsv currently if you install Linkerd (as of edge-19.10.5 ) with --external-issuer, the issuer cert, key and the anchors will be fetched from a secret named linkerd-identity-issuer. Isn't that what you are aiming for? You can read more about it right here: https://linkerd.io/2/tasks/use_external_certs/

[rnsv]

--identity-external-issuer options configures identity to use tls.key and tls.crt from the linkerd-identity-issuer secret.

• The challenge is with proxy-injector. Proxy injector currently reads the Root CA cert (stored in identityContext.trustAnchorsPem) from linkerd-config which is needed for LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS. This means, I need to get the ca.crt value from identity-issuer secret and store in linkerd-config before starting the proxy-injector webhook https://github.com/linkerd/linkerd2/blob/36478312b201808be8535c6e9b96afc364f15078/pkg/inject/inject.go#L531-L533 My question is whether this value can automatically be read from the linkerd-identity-issuer

• It seems like identity is also using linkerd-config to get AnchorPem. https://github.com/linkerd/linkerd2/blob/36478312b201808be8535c6e9b96afc364f15078/controller/cmd/identity/main.go#L87-L89

Any reason why this value is not read from linkerd-identity-issuer secret.

[olix0r]

Any reason why this value is not read from linkerd-identity-issuer secret.

The proxy-injector service account has no business being able to read identity secrets.

[rnsv]

@olix0r I think, right now, when you run linkerd with --identity-external-issuer, it does exactly that - which is reading the linkerd-identity-issuer secret and populating LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS with the ca.crt. Am I missing something?
My question was more on whether this logic could be moved to the server side

[grampelberg]

To provide a little more context, the identity private key is in that secret. Anyone with access to that secret can create arbitrary certificates that are valid anywhere. This isn't an issue as part of the install process because the installer needs superuser access already. It is super important that no one else gets access to that secret.

It'd be possible to have another configmap that only contains the trust root and can be populated by other tooling. That interface just feels pretty fragile and doesn't match up with how folks are using things like cert-manager right now.

[ihcsim]

@rnsv It sounds like you are trying to install both cert-manager and Linkerd in one automated step? With cert-manager in the mix, if possible, I recommend separating your installation process into (at least) two steps:

1. Install cert-manager, create your trust root and the Linkerd's identity issuer secret per our docs. There is no point of installing Linkerd if this step fails. This will avoid Linkerd installation errors caused by cert-manager issues, making debugging a lot easier.
2. Install Linkerd by using the linkerd install command. This command will automate the _get the ca.crt value from identity-issuer secret and store in linkerd-config before starting the proxy-injector webhook_ step for you. By default, this command will only output the Linkerd control plane YAML, which you can customize using tools like Kustomize. It will save you the extra work of trying to figure out the details of the Linkerd installation process.

[vijaygos]

@ihcsim How would this work with helm? With helm we are mandated to provide the identityTrustAnchorsPEM. It would be a lot easier if we can specify the trust Anchor as an existing K8s secret.

[Luke-Vear]

Do we expect the helm value of IdentityContext.TrustAnchorsPem never to change? The documents describe expiry on the trust-anchor, would that not make the config become stale?

[grampelberg]

@Luke-Vear it can change. It just requires rolling the entire data plane when you change it.

[erwbgy]

@rnsv It sounds like you are trying to install both cert-manager and Linkerd in one automated step? With cert-manager in the mix, if possible, I recommend separating your installation process into (at least) two steps:

1. Install cert-manager, create your trust root and the Linkerd's identity issuer secret per our docs. There is no point of installing Linkerd if this step fails. This will avoid Linkerd installation errors caused by cert-manager issues, making debugging a lot easier.
2. Install Linkerd by using the linkerd install command. This command will automate the _get the ca.crt value from identity-issuer secret and store in linkerd-config before starting the proxy-injector webhook_ step for you. By default, this command will only output the Linkerd control plane YAML, which you can customize using tools like Kustomize. It will save you the extra work of trying to figure out the details of the Linkerd installation process.

@ihcsim Please can this comment be added to the Linkerd docs. It is spot on and would have saved me several hours trying to get Linkerd, cert-manager and kustomize to work together.

[cpretzer]

@erwbgy we'd love a PR if you have time to help out :smile:

[rnsv]

In my current installation, I have to manage multiple Kubernetes clusters and I use flux v1 (gitops) for cluster add-on automation. In this setup, I am able to deploy all addons by configuring flux to fetch from a centralized git repo. I use certmanager for all certificate management. I would like to leverage gitops for linkerd installation as well - with all the yaml files managed via git repo and certmanager responsible for certificates. Using a 2 step process with linkerd install might not be an ideal solution in this setup.
I am able to complete this automation with few workarounds (all cert related) but I thought it would be nice if linkerd natively supports reading certs from secrets instead of storing it in linkerd-config (as this info is already available in linkerd-identity-issuer secret)

    data:
      values: |
        global:
          identityTrustAnchorsPEM: "why duplicate this value in here when it is available in linkerd-identity-issuer"
        identity:
          issuer:
            scheme: kubernetes.io/tls
            tls:
              keyPEM:
              crtPEM: "why duplicate this value in here when it is available in linkerd-identity-issuer"