Pulumi-kubernetes: Kubernetes resources created with Helm and imported to Pulumi don't do proper diffs

Created on 13 Dec 2019 · 9Comments · Source: pulumi/pulumi-kubernetes

When attempting to import existing Kubernetes resources managed by Helm to Pulumi, Pulumi doesn't seem to properly diff the k8s resources. Below are steps to reproduce the issue. I've also deployed the stack using just Pulumi with the exact same programs (i.e., skip helm install and just run pulumi up) and the diffs are managed correctly. It seems the issue is with importing only.

Program

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

const chart = new k8s.helm.v2.Chart("google", {
  path: "./google",
  transformations: [addImportForHelmChart()],
});

function addImportForHelmChart() {
  return (obj: any, opts: pulumi.CustomResourceOptions): void => {
    if (obj != null && obj.metadata != null && obj.metadata.name) {
      opts.import = obj.metadata.namespace
        ? `${obj.metadata.namespace}/${obj.metadata.name}`
        : obj.metadata.name;
    }
  };
}

Helm Chart

// src/google/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: google
  namespace: "{{ .Release.Namespace }}"
spec:
  type: ExternalName
  externalName: www.google.com

Pulumi Preview

Previewing update (dev):
+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:dev::pulumi-helm-import-issue::pulumi:pulumi:Stack::pulumi-helm-import-issue-dev]
    + kubernetes:helm.sh/v2:Chart: (create)
        [urn=urn:pulumi:dev::pulumi-helm-import-issue::kubernetes:helm.sh/v2:Chart::google]
warning: inputs to import do not match the existing resource; importing this resource will fail
        = kubernetes:core/v1:Service: (import)
            [id=google]
            [urn=urn:pulumi:dev::pulumi-helm-import-issue::kubernetes:helm.sh/v2:Chart$kubernetes:core/v1:Service::google]
            [provider=urn:pulumi:dev::pulumi-helm-import-issue::pulumi:providers:kubernetes::default_1_4_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
          + spec: {
              + externalName: "www.google.com"
              + type        : "ExternalName"
            }
Resources:
    + 2 to create
    = 1 to import
    3 changes

Steps To Reproduce

# install Node.js
# install Pulumi
# install minikube
git clone https://github.com/migueloller/pulumi-helm-import-issue
cd pulumi-helm-import-issue
npm i
helm install src/google
pulumi preview --diff

dry-run-diff kinbug

Source

migueloller

👍3

Most helpful comment

After trying many different ways to import these resources to Pulumi, the following was the only way I was able to make it work:

Run helm get manifest $RELEASE_NAME > manifest.yaml to get all the manifests as applied by Helm in the last release.
Add missing namespaces to all manifests that need it by running cat manifest.yaml | add-ns.py > manifest-with-namespaces.yaml

Here's the add-ns.py:

#!/usr/bin/env python3

import yaml
import sys

# Run `kubectl api-resources --namespaced` to get all namespaced resources in your cluster.
namespaced_resource_kinds = ['Service']

for manifest in yaml.load_all(sys.stdin):
    if manifest and manifest['kind'] in namespaced_resource_kinds:
        if 'metadata' in manifest:
            if 'namespace' not in manifest['metadata'] or manifest['metadata']['namespace'] == '':
                manifest['metadata']['namespace'] = sys.argv[1]
        else:
            manifest['metadata'] = {"namespace": sys.argv[1]}
    print('---')
    print(yaml.dump(manifest))

Use kubectl apply -f manifest-with-namespaces.yaml to force application of kubectl.kubernetes.io/last-applied-configuration annotations. Unfortunately, I wasn't able to use kubectl apply set-last-applied --create-annotation due to this issue.
Run pulumi up.

migueloller on 13 Dec 2019

❤2

All 9 comments

To narrow down the issue a bit more, I've attempted to replicate the issue without Helm in the branch no-helm. In this case, there is no issue and Pulumi works as expected.

Program

import * as k8s from "@pulumi/kubernetes";

const service = new k8s.core.v1.Service(
  "google",
  {
    metadata: { name: "google", annotations: {} },
    spec: { type: "ExternalName", externalName: "www.google.com" }
  },
  { import: "default/google" }
);

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: google
spec:
  type: ExternalName
  externalName: www.google.com

Pulumi Preview

Previewing update (dev):
+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:dev::pulumi-helm-import-issue::pulumi:pulumi:Stack::pulumi-helm-import-issue-dev]
    = kubernetes:core/v1:Service: (import)
        [id=default/google]
        [urn=urn:pulumi:dev::pulumi-helm-import-issue::kubernetes:core/v1:Service::google]
        [provider=urn:pulumi:dev::pulumi-helm-import-issue::pulumi:providers:kubernetes::default_1_4_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
        apiVersion: "v1"
        kind      : "Service"
        metadata  : {
            name       : "google"
            namespace  : "default"
        }
        spec      : {
            externalName: "www.google.com"
            type        : "ExternalName"
        }
Resources:             
    + 1 to create
    = 1 to import
    2 changes

Steps To Reproduce

# install Node.js
# install Pulumi
# install minikube
git clone https://github.com/migueloller/pulumi-helm-import-issue
cd pulumi-helm-import-issue
git checkout no-helm
npm i
kubectl apply -f src/templates
pulumi preview --diff

migueloller on 13 Dec 2019

Repro Repo URL: https://github.com/migueloller/pulumi-helm-import-issue

cc: @lukehoban

migueloller on 13 Dec 2019

Another extremely interesting phenomenon is that if the service is created in the no-helm branch using kubectl apply and then one checks out to master and runs pulumi preview --diff, the result is the following:

Previewing update (dev):
+ pulumi:pulumi:Stack: (create)
    [urn=urn:pulumi:dev::pulumi-helm-import-issue::pulumi:pulumi:Stack::pulumi-helm-import-issue-dev]
    + kubernetes:helm.sh/v2:Chart: (create)
        [urn=urn:pulumi:dev::pulumi-helm-import-issue::kubernetes:helm.sh/v2:Chart::google]
warning: inputs to import do not match the existing resource; importing this resource will fail
        = kubernetes:core/v1:Service: (import)
            [id=google]
            [urn=urn:pulumi:dev::pulumi-helm-import-issue::kubernetes:helm.sh/v2:Chart$kubernetes:core/v1:Service::google]
            [provider=urn:pulumi:dev::pulumi-helm-import-issue::pulumi:providers:kubernetes::default_1_4_0::04da6b54-80e4-46f7-96ec-b56ff0331ba9]
          ~ metadata: {
              - annotations: {}
            }
Resources:
    + 2 to create
    = 1 to import
    3 changes

This means that somehow, even though the k8s resources are the same when using helm install and kubectl apply, Pulumi is able to import the latter properly but not the former. I've requested the service resources using kubectl get svc google after having applied them both with Helm and with kubectl and the resources have a single difference, the kubectl.kubernetes.io/last-applied-configuration annotation.

Does this mean that Pulumi's import is relying on the kubectl.kubernetes.io/last-applied-configuration annotation somehow to determine whether the import resource matches the one declared in the program?

For reference, here are the two resources:

Installed With Helm

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-12-13T19:08:04Z"
  name: google
  namespace: default
  resourceVersion: "88903"
  selfLink: /api/v1/namespaces/default/services/google
  uid: 6bbeac08-3489-4a6d-a642-b4de915c3a0a
spec:
  externalName: www.google.com
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

Installed With `kubectl`

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"google","namespace":"default"},"spec":{"externalName":"www.google.com","type":"ExternalName"}}
  creationTimestamp: "2019-12-13T19:05:23Z"
  name: google
  namespace: default
  resourceVersion: "88705"
  selfLink: /api/v1/namespaces/default/services/google
  uid: 041c6738-5b39-469f-b738-b6a5cd043087
spec:
  externalName: www.google.com
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

migueloller on 13 Dec 2019

To determine if Pulumi does rely on kubectl.kubernetes.io/last-applied-configuration I tried pulumi preview --diff with the service deployed with kubcetl before and after manually running kubectl annotate svc google kubectl.kubernetes.io/last-applied-configuration- to remove the annotation.

And unsurprisingly, the pulumi preview --diff shows similar behavior as when the Helm chart is used. I will investigate if there is a way to have Helm apply that annotation and report back if that's a way to get around this issue for now.

migueloller on 13 Dec 2019

Ok, after spending some time looking at various alternatives, the most reliable way to get the configuration applied by helm, i.e., the semantic equivalent to kubectl.kubernetes.io/last-applied-configuration is running helm get manifest $RELEASE_NAME.

I understand that Pulumi only supports the import property for CustomResource but k8s.helm.v2.Chart is a ComponentResource. Would it be possible to support something like import for ComponentResource?

If supporting import for ComponentResource isn't an option, could the behavior of import in k8s resources be improved by checking if the resource corresponds to a Helm release, and if so, retrieving the manifest and using that when kubectl.kubernetes.io/last-applied-configuration is not present? Perhaps the resource itself could be used instead of the kubectl.kubernetes.io/last-applied-configuration annotation?

migueloller on 13 Dec 2019

❤2

After trying many different ways to import these resources to Pulumi, the following was the only way I was able to make it work:

Run helm get manifest $RELEASE_NAME > manifest.yaml to get all the manifests as applied by Helm in the last release.
Add missing namespaces to all manifests that need it by running cat manifest.yaml | add-ns.py > manifest-with-namespaces.yaml

Here's the add-ns.py:

#!/usr/bin/env python3

import yaml
import sys

# Run `kubectl api-resources --namespaced` to get all namespaced resources in your cluster.
namespaced_resource_kinds = ['Service']

for manifest in yaml.load_all(sys.stdin):
    if manifest and manifest['kind'] in namespaced_resource_kinds:
        if 'metadata' in manifest:
            if 'namespace' not in manifest['metadata'] or manifest['metadata']['namespace'] == '':
                manifest['metadata']['namespace'] = sys.argv[1]
        else:
            manifest['metadata'] = {"namespace": sys.argv[1]}
    print('---')
    print(yaml.dump(manifest))

Use kubectl apply -f manifest-with-namespaces.yaml to force application of kubectl.kubernetes.io/last-applied-configuration annotations. Unfortunately, I wasn't able to use kubectl apply set-last-applied --create-annotation due to this issue.
Run pulumi up.

migueloller on 13 Dec 2019

❤2

Pulumi's k8s provider does indeed use the last-applied-configuration annotation to calculate diffs (client-side diff behavior). There was some work to move to server-side diffing that would avoid these problems, but it's not complete.

Here are a couple related issues:
https://github.com/pulumi/pulumi-kubernetes/issues/694
https://github.com/pulumi/pulumi-kubernetes/issues/641#issuecomment-518433228

Although server-side diff isn't a stable feature yet, you might be able to get import working by setting the enableDryRun flag on the provider for the Chart:

const provider = new k8s.Provider("foo", {enableDryRun: true});

lblackstone on 17 Dec 2019

I'm going to move this issue to the k8s provider repo because the issue is with the provider's diff behavior and the lack of the last-applied-configuration annotation on these resources.

lblackstone on 17 Dec 2019

I'm going to move this issue to the k8s provider repo because the issue is with the provider's diff behavior and the lack of the last-applied-configuration annotation on these resources.

Sounds good. Thanks for the info on enableDryRun!

migueloller on 17 Dec 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

helm.v3.Chart doesn't render charts with `"helm.sh/hook": pre-install,pre-upgrade` in annotations

zarkone · 10Comments

Creating a Secret adds the clear text secret value to the annotation

ringods · 11Comments

Updating a configmap's data with enableDryRun fails with "configsmap already exists"

pierlucg-xs · 5Comments

Update fails when no changes are really required (kube stripping empty array?)

oliverholliday · 9Comments

Invalid stack state stored when a K8s cluster is not responding during `refresh`?

dotansimha · 4Comments

Pulumi-kubernetes: Kubernetes resources created with Helm and imported to Pulumi don't do proper diffs

Program

Helm Chart

Pulumi Preview

Steps To Reproduce

Most helpful comment

All 9 comments

Program

service.yaml

Pulumi Preview

Steps To Reproduce

Installed With Helm

Installed With kubectl

Related issues

Installed With `kubectl`