Pulumi-kubernetes: Chart namespace transformation breaks update logic

Created on 21 Feb 2019 · 12Comments · Source: pulumi/pulumi-kubernetes

I‘m installing prometheus-operator and prometheus-adapter helm charts (https://github.com/helm/charts/tree/master/stable) via pulumi. A clean installation works good, but if i destroy these resources via Pulumi and create them again, it's complaining about some APIs already exist.

Here is my code:

//Deploy the Prometheus Operator to the cluster
const prometheusOperator = new k8s.helm.v2.Chart("prometheus-operator", {
  repo: "stable",
  version: "3.0.0",
  namespace: "kube-system",
  chart: "prometheus-operator",
  transformations: [addKubeSystemNamespace],
  values: {
    alertmanager: {
      enabled: false,
    },
    grafana: {
      enabled: false,
    },
    defaultRules: {
      create: false,
    },
    coreDns: {
      enabled: false,
    },
    kubeEtcd: {
      enabled: false,
    },
    kubeScheduler: {
      enabled: false,
    },
    kubeControllerManager: {
      enabled: false,
    },
    "kube-state-metrics": {
      enabled: false,
      podSecurityPolicy: {
        enabled: false,
      },
    },
    prometheus: {
      service: {
        type: "LoadBalancer",
      },
    },
  },
}, {providers: {kubernetes: cluster.provider}});

// Deploy the Prometheus Adapter to the cluster
const prometheusAdapter = new k8s.helm.v2.Chart("prometheus-adapter", {
  repo: "stable",
  version: "",
  namespace: "kube-system",
  chart: "prometheus-adapter",
  transformations: [addKubeSystemNamespace],
  values: {
    prometheus: {
      url: "http://prometheus-operator-prometheus.kube-system.svc"
    },
  },
}, {providers: {kubernetes: cluster.provider}});

And the error I'm getting everytime re-installing:

Diagnostics:
  kubernetes:apiextensions.k8s.io:CustomResourceDefinition (kube-system/prometheuses.monitoring.coreos.com):
    error: Plan apply failed: customresourcedefinitions.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" already exists

  kubernetes:apiextensions.k8s.io:CustomResourceDefinition (kube-system/alertmanagers.monitoring.coreos.com):
    error: Plan apply failed: customresourcedefinitions.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" already exists

kubernetes:apiregistration:APIService (kube-system/v1beta1.custom.metrics.k8s.io):
    error: Plan apply failed: apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io" already exists

And if you do kubectl api-versions all these APIs are still there not being deleted.
It seems to me like Pulumi doesn't clean up APIs created completely when destroying resources. I don't have any issue when using kubectl to install/re-install.

pulumi-eks verion: 0.16.6
eks kubernetes version: 1.11.5

aresdk kinbug prioritP1

Source

Lavender-Ding

Most helpful comment

Here's a working version of the program:

import * as k8s from "@pulumi/kubernetes";

function addKubeSystemNamespace(o: any) {
    if (o !== undefined) {
        if (o.metadata !== undefined) {
            o.metadata.namespace = "kube-system";
        } else {
            o.metadata = {namespace: "kube-system"};
        }
    }
}

//Deploy the Prometheus Operator to the cluster
const prometheusOperator = new k8s.helm.v2.Chart("prometheus-operator", {
    repo: "stable",
    version: "3.0.0",
    namespace: "kube-system",
    chart: "prometheus-operator",
    transformations: [addKubeSystemNamespace],
    values: {
        alertmanager: {
            enabled: false,
        },
        grafana: {
            enabled: false,
        },
        defaultRules: {
            create: false,
        },
        coreDns: {
            enabled: false,
        },
        kubeEtcd: {
            enabled: false,
        },
        kubeScheduler: {
            enabled: false,
        },
        kubeControllerManager: {
            enabled: false,
        },
        "kube-state-metrics": {
            enabled: false,
            podSecurityPolicy: {
                enabled: false,
            },
        },
        prometheus: {
            service: {
                type: "LoadBalancer",
            },
        },
        prometheusOperator: {
            createCustomResource: false,
        },
    },
});

// Deploy the Prometheus Adapter to the cluster
const prometheusAdapter = new k8s.helm.v2.Chart("prometheus-adapter", {
    repo: "stable",
    version: "v0.4.1",
    namespace: "kube-system",
    chart: "prometheus-adapter",
    transformations: [addKubeSystemNamespace],
    values: {
        prometheus: {
            url: "http://prometheus-operator-prometheus.kube-system.svc"
        },
    },
});

Note the value:

        prometheusOperator: {
            createCustomResource: false,
        },

I don't think this is an issue with our provider, so I'm closing this issue. Feel free to reopen if you're still having problems.

lblackstone on 26 Mar 2019

❤3

All 12 comments

@lblackstone not sure if you have the bandwidth to look at this, but FYI. LMK if we need to load balance

hausdorff on 22 Feb 2019

@hausdorff Yeah, I won't have time to look at this in M21. You can grab it, or we can punt to M22.

lblackstone on 22 Feb 2019

Spent a few minutes trying to reproduce, and it appears to be a bug related to the transformation on the charts (setting kube-system namespace). The diff logic isn't properly handling the non-namespaced resources (CRDs, ClusterRoleBindings) since the namespace is being set in the transformation.

lblackstone on 25 Feb 2019

I'm new to this, but I feel this is related:

Running the following in a non-default namespace fails:

const sealedSecrets = new k8s.helm.v2.Chart(
    'secrets',
    {
        namespace: 'kube-system', // <--- this is the problem
        repo: 'stable',
        chart: 'sealed-secrets',
    }
);

With:

 panic: secrets "sealed-secrets-key" is forbidden: User "system:serviceaccount:default:secrets-sealed-secrets" cannot get resource "secrets" in API group "" in the namespace "default"
goroutine 1 [running]:
main.main()
    /home/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/controller/main.go:215 +0xc5

However, this works fine if namespace is left out.

Also the chart works just fine if deploying it via heml CLI.

moltar on 11 Mar 2019

👍1

@moltar we'll look at this ASAP

hausdorff on 12 Mar 2019

+1 as this is now blocking our infra automation for production EKS clusters. Is there a workaround available?

eshamay on 25 Mar 2019

@moltar I was unable to reproduce that failure on 0.21.0 or later.

$ pulumi up --skip-preview
Updating (pulumi-k8s-test-dev):

     Type                                                            Name                                 Status
 +   pulumi:pulumi:Stack                                             pulumi-k8s-test-pulumi-k8s-test-dev  created
 +   └─ kubernetes:helm.sh:Chart                                     secrets                              created
 +      ├─ kubernetes:rbac.authorization.k8s.io:ClusterRoleBinding   secrets-sealed-secrets               created
 +      ├─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition  sealedsecrets.bitnami.com            created
 +      ├─ kubernetes:core:ServiceAccount                            secrets-sealed-secrets               created
 +      ├─ kubernetes:rbac.authorization.k8s.io:Role                 sealed-secrets-key-admin             created
 +      ├─ kubernetes:rbac.authorization.k8s.io:ClusterRole          secrets-unsealer                     created
 +      ├─ kubernetes:rbac.authorization.k8s.io:RoleBinding          secrets-sealed-secrets               created
 +      ├─ kubernetes:apps:Deployment                                secrets-sealed-secrets               created
 +      └─ kubernetes:core:Service                                   secrets-sealed-secrets               created

Resources:
    + 10 created

Duration: 16s

Are you still getting this error?

lblackstone on 26 Mar 2019

I was able to reproduce the original error with the following program using version 0.21.1 of the provider:

import * as k8s from "@pulumi/kubernetes";

function addKubeSystemNamespace(o: any) {
    if (o !== undefined) {
        if (o.metadata !== undefined) {
            o.metadata.namespace = "kube-system";
        } else {
            o.metadata = {namespace: "kube-system"};
        }
    }
}

//Deploy the Prometheus Operator to the cluster
const prometheusOperator = new k8s.helm.v2.Chart("prometheus-operator", {
    repo: "stable",
    version: "3.0.0",
    namespace: "kube-system",
    chart: "prometheus-operator",
    transformations: [addKubeSystemNamespace],
    values: {
        alertmanager: {
            enabled: false,
        },
        grafana: {
            enabled: false,
        },
        defaultRules: {
            create: false,
        },
        coreDns: {
            enabled: false,
        },
        kubeEtcd: {
            enabled: false,
        },
        kubeScheduler: {
            enabled: false,
        },
        kubeControllerManager: {
            enabled: false,
        },
        "kube-state-metrics": {
            enabled: false,
            podSecurityPolicy: {
                enabled: false,
            },
        },
        prometheus: {
            service: {
                type: "LoadBalancer",
            },
        },
    },
});

// Deploy the Prometheus Adapter to the cluster
const prometheusAdapter = new k8s.helm.v2.Chart("prometheus-adapter", {
    repo: "stable",
    version: "v0.4.1",
    namespace: "kube-system",
    chart: "prometheus-adapter",
    transformations: [addKubeSystemNamespace],
    values: {
        prometheus: {
            url: "http://prometheus-operator-prometheus.kube-system.svc"
        },
    },
});

Updating (pulumi-k8s-test-dev):

     Type                                                            Name                                               Status                  Info
     pulumi:pulumi:Stack                                             pulumi-k8s-test-pulumi-k8s-test-dev
     └─ kubernetes:helm.sh:Chart                                     prometheus-operator
 +      ├─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition  kube-system/servicemonitors.monitoring.coreos.com  **creating failed**     1 error
 +      ├─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition  kube-system/prometheusrules.monitoring.coreos.com  **creating failed**     1 error
 +      ├─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition  kube-system/alertmanagers.monitoring.coreos.com    **creating failed**     1 error
 +      └─ kubernetes:apiextensions.k8s.io:CustomResourceDefinition  kube-system/prometheuses.monitoring.coreos.com     **creating failed**     1 error

Diagnostics:
  kubernetes:apiextensions.k8s.io:CustomResourceDefinition (kube-system/alertmanagers.monitoring.coreos.com):
    error: Plan apply failed: customresourcedefinitions.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" already exists

  kubernetes:apiextensions.k8s.io:CustomResourceDefinition (kube-system/prometheuses.monitoring.coreos.com):
    error: Plan apply failed: customresourcedefinitions.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" already exists

  kubernetes:apiextensions.k8s.io:CustomResourceDefinition (kube-system/prometheusrules.monitoring.coreos.com):
    error: Plan apply failed: customresourcedefinitions.apiextensions.k8s.io "prometheusrules.monitoring.coreos.com" already exists

  kubernetes:apiextensions.k8s.io:CustomResourceDefinition (kube-system/servicemonitors.monitoring.coreos.com):
    error: Plan apply failed: customresourcedefinitions.apiextensions.k8s.io "servicemonitors.monitoring.coreos.com" already exists

lblackstone on 26 Mar 2019

After further investigation, the actual problem is that the prometheus operator deployment is also creating the CRDs. I'm checking to see if there's a way to disable that behavior in the helm chart.

level=info ts=2019-03-26T19:59:48.6290404Z caller=operator.go:628 component=alertmanageroperator msg="CRD created" crd=Alertmanager
level=info ts=2019-03-26T19:59:48.6571449Z caller=operator.go:1453 component=prometheusoperator msg="CRD created" crd=Prometheus
level=info ts=2019-03-26T19:59:48.6917202Z caller=operator.go:1453 component=prometheusoperator msg="CRD created" crd=ServiceMonitor
level=info ts=2019-03-26T19:59:48.7070801Z caller=operator.go:1453 component=prometheusoperator msg="CRD created" crd=PrometheusRule

lblackstone on 26 Mar 2019

Here's a working version of the program:

import * as k8s from "@pulumi/kubernetes";

function addKubeSystemNamespace(o: any) {
    if (o !== undefined) {
        if (o.metadata !== undefined) {
            o.metadata.namespace = "kube-system";
        } else {
            o.metadata = {namespace: "kube-system"};
        }
    }
}

//Deploy the Prometheus Operator to the cluster
const prometheusOperator = new k8s.helm.v2.Chart("prometheus-operator", {
    repo: "stable",
    version: "3.0.0",
    namespace: "kube-system",
    chart: "prometheus-operator",
    transformations: [addKubeSystemNamespace],
    values: {
        alertmanager: {
            enabled: false,
        },
        grafana: {
            enabled: false,
        },
        defaultRules: {
            create: false,
        },
        coreDns: {
            enabled: false,
        },
        kubeEtcd: {
            enabled: false,
        },
        kubeScheduler: {
            enabled: false,
        },
        kubeControllerManager: {
            enabled: false,
        },
        "kube-state-metrics": {
            enabled: false,
            podSecurityPolicy: {
                enabled: false,
            },
        },
        prometheus: {
            service: {
                type: "LoadBalancer",
            },
        },
        prometheusOperator: {
            createCustomResource: false,
        },
    },
});

// Deploy the Prometheus Adapter to the cluster
const prometheusAdapter = new k8s.helm.v2.Chart("prometheus-adapter", {
    repo: "stable",
    version: "v0.4.1",
    namespace: "kube-system",
    chart: "prometheus-adapter",
    transformations: [addKubeSystemNamespace],
    values: {
        prometheus: {
            url: "http://prometheus-operator-prometheus.kube-system.svc"
        },
    },
});

Note the value:

        prometheusOperator: {
            createCustomResource: false,
        },

I don't think this is an issue with our provider, so I'm closing this issue. Feel free to reopen if you're still having problems.

lblackstone on 26 Mar 2019

❤3

See also the docs, which explain the Helm bug that causes them to do this: https://github.com/helm/charts/tree/master/stable/prometheus-operator/#helm-fails-to-create-crds

hausdorff on 26 Mar 2019

Thank you for looking into this, your solution, and the explanation regarding the Helm bug. It makes sense!

moltar on 27 Mar 2019

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Await logic doesn't handle networking/v1beta1:Ingress correctly

phil-lgr · 8Comments

helm.v3.Chart doesn't render charts with `"helm.sh/hook": pre-install,pre-upgrade` in annotations

zarkone · 10Comments

Creating a Secret adds the clear text secret value to the annotation

ringods · 11Comments

Importing custom resource fails with unknown resource type error

HagaiBarel · 5Comments

[dotnet] Unit tests with Helm.V3.Chart throw NullReferenceException

vipentti · 9Comments