Aws-cdk: [aws-eks] Unable to continue with install: existing resource conflict when creating multiple Helm charts

[arjanschaaf]

When adding multiple Helm charts to an EKS cluster I get an error. The error is probably triggered because the helm install actions are performed in parallel: I install the same Helm chart twice with a different configuration and both installs start by trying to create the same Kubernetes CustomResourceDefinition's.
I can't find any documentation or examples on how to force a sequential installation of these Helm charts using the CDK.

Reproduction Steps

1. Add two Helm charts to an EKS cluster in the CDK with the following snippet:

    // install external facing NGINX Ingress Controller
    cluster.addChart('external-ingress-controller', {
      chart: 'nginx-ingress',
      namespace: 'ingress-controller',
      release: 'external-ingress-controller',
      repository: 'https://helm.nginx.com/stable',
      values: {
        "controller": {
          "config": {
            "proxy-body-size": "512m"
          },
          "metrics": {
            "enabled": true,
            "service": {
              "type": "NodePort"
            }
          },
          "replicaCount": 1,
          "service": {
            "externalTrafficPolicy": "Local",
            "type": "LoadBalancer"
          }
        }
      }
    });
    // install internal facing NGINX Ingress Controller
    cluster.addChart('internal-ingress-controller', {
      chart: 'nginx-ingress',
      namespace: 'ingress-controller',
      release: 'internal-ingress-controller',
      repository: 'https://helm.nginx.com/stable',
      values: {
        "controller": {
          "config": {
            "proxy-body-size": "512m"
          },
          "ingressClass": "internal-nginx",
          "metrics": {
            "enabled": true,
            "service": {
              "type": "NodePort"
            }
          },
          "replicaCount": 1,
          "service": {
            "annotations": {
              "service.beta.kubernetes.io/aws-load-balancer-internal": "0.0.0.0/0"
            },
            "externalTrafficPolicy": "Local",
            "type": "LoadBalancer"
          }
        }
      }
    });

Error Log

  4/11 | 11:50:20 AM | CREATE_FAILED        | Custom::AWSCDK-EKS-HelmChart                | eks-cloud-infra/chart-internal-ingress-controller/Resource/Default (ekscloudinfrachartinternalingresscontroller6054650E) Failed to create resource. Error: b'Release "internal-ingress-controller" does not exist. Installing it now.\nError: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: CustomResourceDefinition, namespace: , name: virtualservers.k8s.nginx.org\n'
    at invokeUserFunction (/var/task/framework.js:85:19)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    new CustomResource (~/git/cloud-infrastructure/node_modules/@aws-cdk/aws-cloudformation/lib/custom-resource.ts:163:21)
    \_ new HelmChart (~/git/cloud-infrastructure/node_modules/@aws-cdk/aws-eks/lib/helm-chart.ts:83:5)
    \_ Cluster.addChart (~/git/cloud-infrastructure/node_modules/@aws-cdk/aws-eks/lib/cluster.ts:650:12)
    \_ new CloudInfrastructureStack (~/git/cloud-infrastructure/lib/cloud-infrastructure-stack.ts:114:13)
    \_ Object.<anonymous> (~/git/cloud-infrastructure/bin/cloud-infrastructure.ts:7:1)
    \_ Module._compile (internal/modules/cjs/loader.js:1123:30)
    \_ Module.m._compile (~/git/cloud-infrastructure/node_modules/ts-node/src/index.ts:839:23)
    \_ Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    \_ Object.require.extensions.<computed> [as .ts] (~/git/cloud-infrastructure/node_modules/ts-node/src/index.ts:842:12)
    \_ Module.load (internal/modules/cjs/loader.js:972:32)
    \_ Function.Module._load (internal/modules/cjs/loader.js:872:14)
    \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    \_ main (~/git/cloud-infrastructure/node_modules/ts-node/src/bin.ts:227:14)
    \_ Object.<anonymous> (~/git/cloud-infrastructure/node_modules/ts-node/src/bin.ts:512:3)
    \_ Module._compile (internal/modules/cjs/loader.js:1123:30)
    \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    \_ Module.load (internal/modules/cjs/loader.js:972:32)
    \_ Function.Module._load (internal/modules/cjs/loader.js:872:14)
    \_ Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    \_ /usr/local/lib/node_modules/npm/node_modules/libnpx/index.js:268:14

Environment

• **CLI Version : aws-cli/2.0.5
• **Framework Version: cdk 1.34.1
• **OS : OSX
• **Language : Typescript

Other

---

This is :bug: Bug Report

[arjanschaaf]

I worked around this by declaring a dependency between these two actions:

    const externalIngressController = new eks.HelmChart(this, 'external-ingress-controller', {
      chart: 'nginx-ingress',
      cluster: cluster,
      namespace: 'ingress-controller',
      release: 'external-ingress-controller',
      repository: 'https://kubernetes-charts.storage.googleapis.com',
      values: {
        "controller": {
          "config": {
            "proxy-body-size": "512m"
          },
          "metrics": {
            "enabled": true,
            "service": {
              "type": "NodePort"
            }
          },
          "replicaCount": 1,
          "service": {
            "externalTrafficPolicy": "Local",
            "type": "LoadBalancer"
          }
        },
        wait: true
      }
    });

    const internalIngressController = new eks.HelmChart(this, 'internal-ingress-controller', {
      chart: 'nginx-ingress',
      cluster: cluster,
      namespace: 'ingress-controller',
      release: 'internal-ingress-controller',
      repository: 'https://kubernetes-charts.storage.googleapis.com',
      values: {
        "controller": {
          "config": {
            "proxy-body-size": "512m"
          },
          "ingressClass": "internal-nginx",
          "metrics": {
            "enabled": true,
            "service": {
              "type": "NodePort"
            }
          },
          "replicaCount": 1,
          "service": {
            "annotations": {
              "service.beta.kubernetes.io/aws-load-balancer-internal": "0.0.0.0/0"
            },
            "externalTrafficPolicy": "Local",
            "type": "LoadBalancer"
          }
        },
        wait: true
      }
    });
    // declare a dependency between the two helm charts
    internalIngressController.node.addDependency(externalIngressController);

I think it would be very helpful to document how to add dependencies using the CDK, it is hard to find documentation on this topic right now

[eladb]

We will add some documentation about dependencies as part of #4291