Pulumi-kubernetes: Support dependsOn depending on all chart resources

[nesl247]

Problem description

Currently dependsOn only waits until Pulumi registers the chart, not until it's fully available.

See: https://github.com/pulumi/pulumi/issues/1743 and https://github.com/pulumi/pulumi-kubernetes/issues/814 (hack)

[pgavlin]

This is a consequence of https://github.com/pulumi/pulumi/issues/3282.

[0x53A]

this seems to be similar for k8s.yaml.ConfigGroup:

// Set up TLS
// -------------------------------------------------------------------------------------------------------------------------------------------------------

const certManagerCustomResources = new k8s.yaml.ConfigGroup(`${prefix}-k8-tls-custom-resources`, {
    files: ["https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml"]
}, { provider: k8sProvider });



const certmanagerNamespace = new k8s.core.v1.Namespace(`${prefix}-k8-certmanager-namespace`, {
    metadata: {
        name: "cert-manager",
        labels: { "certmanager.k8s.io/disable-validation": "true" }
    }
  }, { provider: k8sProvider, dependsOn: [certManagerCustomResources] });



const certManagerDeployment = new k8s.helm.v2.Chart(`cert-manager`, {
    repo: "jetstack",
    chart: "cert-manager",
    namespace: certmanagerNamespace.metadata.name,
    version: "v0.9.0",
    values: {
        nodeSelector: { "beta.kubernetes.io/os": "linux" },
        cainjector: { nodeSelector: { "beta.kubernetes.io/os": "linux" } },
        webhook: { nodeSelector: { "beta.kubernetes.io/os": "linux" } },
    }
}, {
    providers: { kubernetes: k8sProvider },
    dependsOn: [ certmanagerNamespace, certManagerCustomResources ]
});


const clusterIssuerYaml = `apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: ${letsencryptName}
  namespace: ingress-basic
spec:
  acme:
    server: ${acmeServer}
    email: ${letsEncryptEmail}
    privateKeySecretRef:
      name: ${letsencryptName}
    http01: {}
    solvers:
    - http01:
        ingress:
          podTemplate:
            spec:
              nodeSelector:
                "beta.kubernetes.io/os": "linux"`;


const clusterIssuerDeployment = new k8s.yaml.ConfigGroup(`${prefix}-k8-certmanager-cluster-issuer`, { yaml: [ clusterIssuerYaml ]}, { provider: k8sProvider, dependsOn: [ certManagerDeployment ] });

The namespace depends on the CRD ConfigGroup, and everything else obviously depends on the namespace, so it should only continue after the CRDs have been created.

As you can see it tried to create the cluster issuer before the CRDs have been created.

This makes the whole thing really fragile.

[ulrichSchreiner]

hi,
any work on this? i have exactly the same problem with the cert-manager manifests and dependent resssources.

[mitchellmaler]

I think I just ran into this issue as well using .NET. I need to deploy the rancher helm release before I create a rancher2.bootstrap resource. I have the bootstrap resource depending on the helm release but it seems to be ignored due to it always trying to run first which ends with an error since rancher isn't installed yet.

c# var rancherHelmRelease = new Pulumi.Kubernetes.Helm.V3.Chart("rancher", new ChartArgs { Repo = "rancher-latest", Chart = "rancher", Version = config.Require("rancherVersion"), Namespace = cattleSystemNamespace.Metadata.Apply(x => x.Name), Values = { {"ingress.tls.source", "external"}, {"tls", "external"}, {"hostname", config.Require("rancherHost")}, {"debug", config.RequireBoolean("rancherEnableDebug")}, {"replicas", 3} } }, new ComponentResourceOptions { Provider = kubernetesProvider, DependsOn = { rkeCluster, cattleSystemNamespace } }); var rancherBootstrapProvider = new Pulumi.Rancher2.Provider("rancherBootstrapProvider", new Pulumi.Rancher2.ProviderArgs { ApiUrl = $"https://{config.Require("rancherHost")}", Bootstrap = true }); // Initially setup server the first time creating admin account var rancherBootstrap = new Pulumi.Rancher2.Bootstrap("rancherBootstrap", new BootstrapArgs { Password = commonEnvSecret.Data.Apply(x => (string)x["rancher_server_admin_password"]), Telemetry = false }, new CustomResourceOptions { Provider = rancherBootstrapProvider, DependsOn = { rancherHelmRelease } });

[lblackstone]

This is a known problem for ComponentResource classes, which are used in the implementation for Helm and YAML support.

We're working to resolve the underlying issue so that you can depend on the top-level resource as you'd expect, but for now, you can work around the problem by depending on one or more sub-resources from the ComponentResource.

[mitchellmaler]

So how can I return the list of resources or even the resource from the component so that another resource can depend on it? It does not let me return Outputs that are not "String, Boolean, Int32, Double, Nullable<...>, ImmutableArray<...> and ImmutableDictionary

The Helm resource is also not customizable since it is a library so how can I get the list of resources it creates as a property so I can add it for depends on?

[mitchellmaler]

Was able to solve the issue with the components to creating a class property that holds a reference to all the created resources.

@mikhailshilkov Can a new property be created on the Pulumi.Kubernetes.Helm.V3.Chart class that tracks all resources and returns them? Looks like CollectionComponentResource has a Resources property that tracks it and a method GetResource that returns a single resource if you know what you are looking for. Would be great if we could reference the resources property or a new method to returns all resources.

[mikhailshilkov]

Sounds reasonable. @lblackstone have we considered exposing child resources from a Helm chart?

[lblackstone]

have we considered exposing child resources from a Helm chart?

I haven't tried that, but it sounds like it could be a reasonable solution here. I'm open to trying it.

If that works out, it would be worth seeing if we could make that solution more general in the ComponentResource class.

[jmgilman]

I'm slightly confused on how one would depend on a resource created by the chart. I would of thought something like this would have worked:

crd = self.chart.get_resource(group_version_kind='apiextensions.k8s.io/v1/CustomResourceDefinition',
                                      name='clusterissuers.cert-manager.io',
                                      namespace=self.props.namespace_str).apply(lambda res: res)

...
opts=pulumi.ResourceOptions(parent=self, depends_on=[crd]))
...



md5-1a1a92957cdee4f55d38c81f5f4d2457



Exception: 'depends_on' was passed a value that was not a Resource.



md5-a24d61a40d7d18c19fddd480d6032fce



self.crds = yaml.ConfigFile('cert-manager-crds',
                                    'https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.crds.yaml',
                                    opts=pulumi.ResourceOptions(parent=self))



md5-3a87830cb76f2104a25206456a95f93c



...
opts=pulumi.ResourceOptions(parent=self, depends_on=[self.crds]))
...

I would really be interested in a workaround here - at this point continuing with Pulumi is basically impossible in my use-case due to the cascading failure that can occur when a higher-level resource fails because of dependencies not being ready.

[manast]

@jmgilman I am suffering your pain too. Seems like a ticking bomb, using some resource whose dependencies will not be correctly tracked, has high probability to break not only the deployment but in my experience the stack as well, which in production is really a no-go.
If we could at least get a list of which resources not to use so that we can avoid them completely, otherwise this is like a recipe for disaster sooner or later...

[mitchellmaler]

@lblackstone Is there any workaround for this issue? We deploy many helm charts and it is causing a lot of headaches.
I think just having a list of Resources exported from the Chart release would fix this issue. We could then depend on that list of a single item from that list.

[lblackstone]

@mitchellmaler I've opened #1364 to investigate this -- that approach seems to be working for TypeScript, but we need to decide on the best way to do this, and make sure it works for all languages.

[mitchellmaler]

@lblackstone Great news. Really like the idea of being able to wait for the whole chart to be created.

[lblackstone]

Although TypeScript is working now, the other languages will require changes in the Pulumi SDK to support (the dependsOn opt needs to support Inputs rather than just the raw Resource). Since this might be a breaking change, it's going to require more investigation/discussion.

https://github.com/pulumi/pulumi/issues/5642 is related.

[harryttd]

I'm not sure if this is related exactly to this issue:

We are using gh actions to run pulumi up. We have local Helm charts on our computers. One is a dep of the other. We reference the charts in the path prop when instantiating a new chart via new k8s.helm.v2.Chart. We tried deploying both charts, but pulumi fails with:
```

• pulumi:pulumi:Stack tqinfra-dev creating Error: found in Chart.yaml, but missing in charts/ directory: authenticated-rpc-tezos-chain
  
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating Error: Error: Command failed: helm template mainnet_bcd/tezos-k8s/charts/tezos --name-template bcd-main --values mainnet_bcd/tezos-k8s/charts/tezos/values.yaml --values /tmp/tmp-2059gBUNjUvO4U4E.yaml --namespace tezos --include-crds
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating Error: found in Chart.yaml, but missing in charts/ directory: authenticated-rpc-tezos-chain
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at /home/runner/work/tqinfra/tqinfra/node_modules/@pulumi/helm/v2/helm.ts:250:23
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at /home/runner/work/tqinfra/tqinfra/node_modules/@pulumi/pulumi/output.js:249:35
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at Generator.next ()
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at /home/runner/work/tqinfra/tqinfra/node_modules/@pulumi/pulumi/output.js:21:71
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at new Promise ()
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at __awaiter (/home/runner/work/tqinfra/tqinfra/node_modules/@pulumi/pulumi/output.js:17:12)
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at applyHelperAsync (/home/runner/work/tqinfra/tqinfra/node_modules/@pulumi/pulumi/output.js:228:12)
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at /home/runner/work/tqinfra/tqinfra/node_modules/@pulumi/pulumi/output.js:182:65
  
  
  • pulumi:pulumi:Stack tqinfra-dev creating at processTicksAndRejections (internal/process/task_queues.js:93:5)
    
    ```
  
  

The problem may be that pulumi is not running helm dependency update on the parent chart.