Pulumi-kubernetes: Secret created with pulumi has last-applied-configuration leaking secret

[nesl247]

Problem description

Creating a Secret with pulumi is creating the last-applied-configuration annotation. This contains the raw values of the secret.

Suggestions for a fix

According to my research

• https://github.com/kubernetes/kubernetes/issues/23564
• https://github.com/kubernetes/kubernetes/issues/29923

kubectl apply should not be used with secrets. I am not sure if this is what pulumi is using or not, but it seems like a possibility.

[nesl247]

I guess this is a repeat of #965. However, I really do think this should be changed, regardless of if people who have access to see the annotation would have access to the values or not. The difference is when doing an get -o yaml, you will see the values on screen. If for some reason (potentially screen sharing), you do not want this, it will happen regardless.

[confiq]

Creating a Secret with pulumi is creating the last-applied-configuration annotation.

I'm not sure that pulumi that creates kubectl.kubernetes.io/last-applied-configuration, it's how k8s manages the declarative config as explained in #965

[nesl247]

@confiq I'm aware. I wasn't stating that pulumi is creating it, just that creating a secret with pulumi is creating the annotation.

[jaxxstorm]

This is a tricky problem as mentioned in #965 and I'm open to ideas about solutions.

For those not following, this is a downstream consideration with kubectl apply and has been set as wontfix/by design.

We have a few non ideal solutions:

• Don't use kubectl apply style logic for applying secrets, which isn't ideal because we lose the declarative management technique we get with it
• Use a magic transformation to remove the annotation.

It's worth noting that describe and get -o yaml will dump "plaintext" data to stdout, but of course it's base64 encoded, so much harder to be worried about in a screen share environment.

Considering this is set as "by design" upstream I'm inclined to close it, but happy to take feedback/considerations.

[nesl247]

I don't think I saw an explanation as to what we lose by switching away from the declarative method?

[jaxxstorm]

I don't actually know how we would be able to switch away from the declarative object configuration. The Kubernetes imperative management has no knowledge of the previous state so it would be a very dramatic change in how we operate on the cluster.

As an example:

If we use kubectl create secret style logic to create a secret, the initial secret created would not have the annotation in plaintext. However, if/when that secret is updated, pulumi would need to then import the state and manage the update itself. Using kubectl apply style logic means we don't have to manage and maintain this within the app, we can defer to the upstream logic.

[nesl247]

Why would pulumi need to import the state?

Create secret -> secret stored in statefile -> update secret -> check diff against statefile -> diff detected -> replace secret

[jaxxstorm]

check diff against statefile

Sorry if it wasn't clear, that's largely what I meant by saying "import the state".
This is a large amount of logic to implement for a single resource that isn't being fixed upstream.

[nesl247]

The thing is it wouldn't just benefit Secrets. Look at #1048 which has an issue due to last-applied-configuration.

[jaxxstorm]

That's totally understandable. It's not clear how we work around this upstream limitation at this time, due to how we actually perform the apply logic.

[farvour]

The entire concept of the last-applied-configuration is such a k8s screw-up IMO. I don't even know where to begin. There's really no reason why this value cannot simply be a hash or checksum of sorts, and the actual content logged to the API servers somewhere if desired. This type of problem extends way beyond Pulumi. It's almost like they designed this really cool declarative config system... and forgot about Secrets, then decided to tell users "Well you should really not use apply with secrets..."

Excuses.. excuses... there's plenty of ways this situation could have been avoided/protected from snooping. If the real values needed to exist then the values could simply be encrypted using the API servers certificates some how.

Ah well.... pipe dreams I guess... since k8s doesn't yet have an access level that separates "get metadata" vs "get entire object" it's kind of a non-issue, but the day that granularity comes [if it does] this issue needs to be revisited by the k8s core team again. It's like 6 year old issue... not sure why hashing the value 'breaks' any of this but I digress.

[GauravJain98]

Issue is there but you can remove the annonate using

kubectl annotate secret <secret-name> kubectl.kubernetes.io/last-applied-configuration- -n <namespace>

[farvour]

Issue is there but you can remove the annonate using

kubectl annotate secret <secret-name> kubectl.kubernetes.io/last-applied-configuration- -n <namespace>

This is a sufficient workaround for CI/CD processes, but would be best to not have it at all. The other problem with last applied is it limits configmaps to half of the actual available capacity. Although, one can argue having very large configmaps is probably not a great idea, it would be ideal if Pulumi could support some sort of patch capability instead of relying on apply-only.