Pulumi-kubernetes: Creating a Secret adds the clear text secret value to the annotation

[ringods]

I’m creating a Kubernetes Secret with Pulumi, and the secret information is added to the Secret in clear text as an annotation.

My code snippet:

const privatePullCredentials = new kubernetes.core.v1.Secret('dockerprivatepull', {
    type: "kubernetes.io/dockerconfigjson",
    metadata: {
        namespace: namespace.metadata.name
    },
    stringData: {
        ".dockerconfigjson": config
            .requireSecret("docker-hub-token")
            .apply(value => {
                return JSON.stringify({
                    auths: {
                        "https://index.docker.io/v1/": {
                            auth: value
                        }
                    }
                })
            })
    }
});

After running Pulumi, I check what was created using kubectl and got this:

apiVersion: v1
data:
  .dockerconfigjson: <redacted>
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{"pulumi.com/autonamed":"true"},"labels":{"app.kubernetes.io/managed-by":"pulumi"},"name":"dockerprivatepull-s2nimzmf","namespace":"apps"},"stringData":{".dockerconfigjson":"{\"auths\":{\"https://index.docker.io/v1/\":{\"auth\":\"<clear text secret here!!!>"}}}"},"type":"kubernetes.io/dockerconfigjson"}
    pulumi.com/autonamed: "true"
  creationTimestamp: "2020-01-28T09:43:00Z"
  labels:
    app.kubernetes.io/managed-by: pulumi
  name: dockerprivatepull-s2nimzmf
  namespace: apps
  resourceVersion: "935549"
  selfLink: /api/v1/namespaces/osimis/secrets/dockerprivatepull-s2nimzmf
  uid: c19731a4-45e9-414a-8a04-fb92ce5f05bd
type: kubernetes.io/dockerconfigjson

The secret value is in clear text in the annotation!!!

When I create the Secret manually via the CLI, I don't get this annotation:

$ kubectl create secret docker-registry regcred --docker-server='https://index.docker.io/v1/' --docker-username=ringods --docker-password=<clear text secret here> --docker-email='[email protected]' --namespace=apps
$ kubectl get secret regcred --namespace=apps --output=yaml
apiVersion: v1
data:
  .dockerconfigjson: <redacted>
kind: Secret
metadata:
  creationTimestamp: "2020-01-28T11:33:08Z"
  name: regcred
  namespace: apps
  resourceVersion: "944760"
  selfLink: /api/v1/namespaces/osimis/secrets/regcred
  uid: b9689f62-e57b-4b7b-a119-df18226d43c8
type: kubernetes.io/dockerconfigjson

So what is Pulumi doing wrong here?

Context:

• https://pulumi-community.slack.com/archives/CRFURDVQB/p1580204949047000
• @pulumi v1.8.1
• @pulumi/kubernetes v1.4.5

[theneva]

I'm pretty sure this is caused by the secret being created and updated using (the same approach as) kubectl apply (which adds a last-applied-configuration annotation), rather than creating with create and updating with replace (which do not add the annotation, as the secret is new every time).

[lblackstone]

Related to https://github.com/pulumi/pulumi-kubernetes/issues/956 and https://github.com/pulumi/pulumi-kubernetes/issues/915

[lblackstone]

I'm working on a fix for this already in https://github.com/pulumi/pulumi-kubernetes/pull/961, but in the meantime, you should be able to wrap the string output in a pulumi.secret() to get the behavior you were expecting (all references to the secret encrypted in the state, including the annotation).

[theneva]

Since we don't set the annotation explicitly (but it is rather set by Kubernetes itself) I'm not sure what to wrap anything in pulumi.secret that will help here.

Maybe we could set the annotation to something else to prevent Kubernetes from setting it at all?

[lblackstone]

@ringods Actually, I just reread the issue, and I'm unclear if you're reporting on the Pulumi state, or the actual Secret in k8s. If it's the latter, any secret values are always readable if you pull them via kubectl or similar; that's just the way k8s works

All values in the data field are base64 encoded, but that's effectively equivalent to being in plaintext.

[lblackstone]

Since we don't set the annotation explicitly (but it is rather set by Kubernetes itself) I'm not sure what to wrap anything in pulumi.secret that will help here.

If the input (either stringData or data) is a pulumi.secret, then the provider will automatically encrypt the annotation for you.

Again, this doesn't have any affect on how this appears in k8s. Any user who can access a Secret resource on the cluster can directly read the secret values contained within. The annotation doesn't change that.

[theneva]

Unless I'm mistaken (currently on phone) describing a secret lists annotations while you need to read (get) it to see its value, so there is a functional difference in terms of permission management

[lblackstone]

Unless I'm mistaken (currently on phone) describing a secret lists annotations while you need to read (get) it to see its value, so there is a functional difference in terms of permission management

I just checked this locally with kubectl, and it's not showing me the last-applied-configuration annotation. In the process of checking k8s docs on RBAC/Secrets to make sure this annotation isn't changing the roles required to view the secret value.

$ kubectl describe secrets secret-07dtkk6i
Name:         secret-07dtkk6i
Namespace:    default
Labels:       app.kubernetes.io/managed-by=pulumi
Annotations:  pulumi.com/autonamed: true

Type:  Opaque

Data
====
password:  11 bytes

Edit:
The kubectl describe command also uses the get RBAC role, so having the secret value in that annotation shouldn't change the scope. You'll also notice that they explicitly skip showing the last-applied-configuration annotation.

[theneva]

Oh okay, thanks for checking that out! Sorry :)

[lblackstone]

Alright, after some further testing, I discovered that kubectl's behavior differs depending on which command you use to create the Secret. The annotation does not appear on Secrets created with kubectl create <args>, but does for Secrets created from a manifest using kubectl apply -f <manifest>

$ kubectl create secret generic prod-db-secret --from-literal=username=produser
secret/prod-db-secret created

$ kubectl get secrets prod-db-secret -o yaml
apiVersion: v1
data:
  username: cHJvZHVzZXI=
kind: Secret
metadata:
  creationTimestamp: "2020-01-28T20:03:14Z"
  name: prod-db-secret
  namespace: default
  resourceVersion: "1815341"
  selfLink: /api/v1/namespaces/default/secrets/prod-db-secret
  uid: a4b4ba92-153b-473c-b5e6-163269fbb0b2
type: Opaque

$ kubectl apply -f secret.yaml
secret/mysecret created

$ kubectl get secrets mysecret -o yaml
apiVersion: v1
data:
  password: c3VwZXJzZWNyZXQ=
  username: cHJvZHVzZXI=
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"mysecret","namespace":"default"},"stringData":{"password":"supersecret","username":"produser"},"type":"Opaque"}
  creationTimestamp: "2020-01-28T19:57:38Z"
  name: mysecret
  namespace: default
  resourceVersion: "1814927"
  selfLink: /api/v1/namespaces/default/secrets/mysecret
  uid: cd39f91c-615e-4dd3-838c-9e5eab02afa1
type: Opaque

Since we are using this annotation to support client-side diffing, we'll keep the current behavior, and will create that annotation for any Secrets managed by Pulumi.

[EronWright]

To elaborate on the above remark by @lblackstone, the kubectl.kubernetes.io/last-applied-configuration annotation is applied by kubectl when using the _declarative management technique_, i.e. when using apply, versus the _imperative management technique_, i.e. create. From the docs:

The annotation contains the contents of the object configuration file that was used to create the object.

See Kubernetes Object Management for more details.