Pulumi-kubernetes: Duplicate resource URN with helm when using same release name on multiple clusters

[regisbelson]

I create several kubernetes clusters in the same stack which all have some core helm charts installed (traefik, external-dns...). I want the release name to be the same on all clusters but the helm chart resource only uses the release name as the name of the resource.
With any other resource, I can provide the resource name so I could just use `${provider}/${region}/${cluster}/${releasename}` to solve this. It's not possible right now since I can only provide the release name.

[lblackstone]

Related to #353

@kurlzor Would it be possible to split the k8s clusters into different stacks? Interested to hear your use case here.

[SamuelMcAravey]

I am facing a similar situation, so I'll give my use case as well.

Our intended design is to have a k8s cluster in the East US region and the West US region for production purposes, with load balancing and whatnot in front of the clusters. The two clusters should not be independently managed in my mind as they both make up the single production environment.

In Pulumi we have an array that defines information about each region. We have a plain k8s yaml file that needs to be applied to both clusters via kubernetes.yaml.ConfigFile. The behavior we see is identical to what @kurlzor is seeing with the Helm charts with the error: Duplicate resource URN.

If having multiple stacks are the only way to manage this, then I feel like there needs to be some guidance around how to create and manage complex environments like this within pulumi.

[regisbelson]

@lblackstone We only have 3 projects: the _main_ one with everything that's used to directly support our production infrastructure, another one for our data team which has its own infrastructure, and another one for internal stuff.
We only have 2 stacks on the main/data projects: staging and production. We don't need staging for our internal stuff so there's only one stack for this project.
A stack describes the whole infrastructure for the given environment, so we have several kubernetes clusters definitions since we have kubernetes clusters in different regions.

I agree completely with what @SamuelMcAravey said:

The two clusters should not be independently managed in my mind as they both make up the single production environment.

Also, we're on the Team plan, so having more stacks costs more money.

[lblackstone]

That makes sense, thanks for the feedback!

We'll definitely get this fixed soon; I just wanted to make sure we understood the use case. We have mostly used multiple stacks internally, but one of our primary goals is to provide flexibility so you can choose a solution that makes sense for you.

[hausdorff]

The tricky part about this is that any change to the URN spec here will cause resources in existing stacks to be replaced. Ideally we'd like the provider to be part of the URN, but we'd need to provide some kind of an upgrade strategy for people with existing Helm deployments. @lukehoban @joeduffy thoughts?

[lblackstone]

After further investigation, this isn't a trivial thing to solve. For the original Helm issue the current workarounds are:

1. Choose a unique releaseName for the Helm chart
2. Use a different stack for each k8s cluster

I've opened #536 to track the related YAML issue.

Sorry for the inconvenience this causes in the meantime, and thanks for your patience as we work through a fix.

[hausdorff]

I've gone ahead and opened https://github.com/pulumi/pulumi/issues/2647 to track the broader question of how to name resources allocated to different first-class providers. I'm not sure when we'll decide what to do here -- this seems like it would be an invasive change, and we probably want to be purposeful about it.

[lukehoban]

@lblackstone/@hausdorff Could we add an optional name-prefix to the chart and configfile constructors? That would allow optionally qualifying the URNs of the resources construed by the kubernetes yaml. By making it optional, this would not force changes for existing resources. Would that work?

[lblackstone]

@lukehoban That would work, but I don't think it's a good solution. It should be possible to infer that these resources are different because the attached provider is different. I strongly prefer finding a solution that doesn't require the user to to disambiguate the name for each targeted cluster.

[lukehoban]

It should be possible to infer that these resources are different because the attached provider is different.

This is a very general design thing in Pulumi. As a general rule, any function which creates resources on behalf of the user needs to take a name input that it uses as part of the name of the generated resources. It is in general not possible to use the properties of a resource to uniquely construct a name. This is true for normal properties, and just as true for the provider property.

I strongly prefer finding a solution that doesn't require the user to to disambiguate the name for each targeted cluster.

I just want to emphasize that asking the user to provide a name is an extremely broad design requirement that's pretty fundamental to API design in Pulumi. Here's an example in a totally different part of our world:

bucket.onObjectCreated("somename", async (ev) => console.log(ev));

It would be much "nicer" if the user didn't have to pass "somename" here. And the same arguments made in this thread could lead to ways you might try to make that work ("why not just embed the identitity of the bucket plus a hash of the contents of the function into the name or URN structure"). This in general doesn't work. Among other things, two objects with identical properties could be created but be legitimately different. But even without that, identity must mean something across changes to resources.

I may still be missing something here - but my general read of this is that there is a reasonably clear cut answer within existing Pulumi API design patterns, and the alternative is a fundamental change to the notion of identity in Pulumi generally - which would be a major breaking change for all users, would complicate the URN format, is not aligned directionally with how we want users to think about URNs/identity, and doesn't even seem to be a "correct" solution in this specific case due to providers not being 1:1 with "clusters".

[lblackstone]

As a general rule, any function which creates resources on behalf of the user needs to take a name input that it uses as part of the name of the generated resources.

Both the Chart and ConfigFile types already include the name parameter, but it is not used for the underlying resources. While we could change the SDK to include that in the name of the child resources, that has the unfortunate side effect of requiring a replacement for any resources created with the existing SDK.

After talking with @lukehoban offline, it sounds like the most reasonable suggestion is an optional prefix parameter. The default behavior would remain unchanged, but this parameter could be used to disambiguate if needed. It's not pretty, but it has the smallest impact on existing resources, and doesn't require breaking changes to the URN format. We might be able to do away with this extra parameter in the future with URN aliasing.

[gobengo]

Could you also allow passing an optional releaseName parameter to helm.Chart, and use that if provided instead of the name first param. i.e. let me name my pulumi resources and my helm release (which often implies my k8s resource names) separately.

To me it's too much magic to make it the same as the pulumi resource name. For example, the pulumi resource names allow '/' and the helm chart uses that to prefix resulting pulumi resource names like {namespace}/{k8sResourceName}.

At the same time, the convention in helm community seems to preprending the helm release name to to all k8s resource names templated by the chart. But most k8s resource names follow this naming convention:

By convention, the names of Kubernetes resources should be up to maximum length of 253 characters and consist of lower case alphanumeric characters, -, and ., but certain resources have more specific restrictions.

i.e. they don't allow '/'. Def services don't, because I hit an error when doing the below.

A scenario to consider for changes here is this program

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
// https://github.com/pulumi/pulumi-kubernetes/issues/217#issuecomment-459105809
const addNamespaceTransformation = (namespaceName: string) => (o: any) => {
    if (o !== undefined) {
        if (o.metadata !== undefined) {
            o.metadata.namespace = namespaceName;
        } else {
            o.metadata = {namespace: namespaceName}
        }
    }
}
const numNamespaces = 2
for (const namespaceIndex of (new Array(numNamespaces).fill(0).map((_, index) => index))) {
    const namespaceName = `pulumi-wp-${namespaceIndex}`
    const namespace = new k8s.core.v1.Namespace(namespaceName, {
        metadata: {
            name: namespaceName
        }
    })
    const wordpressChart = new k8s.helm.v2.Chart(`${namespaceName}-wordpress`, {
        namespace: pulumi.output(namespace).apply(namespace => namespace.metadata.name),
        repo: "stable",
        version: "2.1.3",
        chart: "wordpress",
        transformations: [addNamespaceTransformation(namespaceName)]
    }); 
}

Note the first parameter I pass to helm.Chart for name.
If I don't include nameSpaceName in there, the pulumi resource for the chart will have a duplicate name.
If it were ${namespaceName}/wordpress, using the '/' like the internals of helm.Chart must do, then it become the release name, goes into the wordpress k8s resource names, which then k8s will reject because '/' is not allowed.
So I used a hyphen. But even still, including ns string in there it also gets set to the helm release name, which then goes in pretty much all the k8s resource names, and then it gets duplicated again in the pulumi resource names for those k8s resources templated by the helm chart.

     Type                                         Name                                                  Plan
     pulumi:pulumi:Stack                          wordpress-dev
 +   ├─ kubernetes:helm.sh:Chart                  pulumi-wp-0-wordpress                                 create
 +   │  ├─ kubernetes:core:ConfigMap              pulumi-wp-0/pulumi-wp-0-wordpress-mariadb             create
 +   │  ├─ kubernetes:core:Secret                 pulumi-wp-0/pulumi-wp-0-wordpress-mariadb             create
 +   │  ├─ kubernetes:core:ConfigMap              pulumi-wp-0/pulumi-wp-0-wordpress-mariadb-tests       create
 +   │  ├─ kubernetes:core:Secret                 pulumi-wp-0/pulumi-wp-0-wordpress-wordpress           create
 +   │  ├─ kubernetes:core:PersistentVolumeClaim  pulumi-wp-0/pulumi-wp-0-wordpress-wordpress           create
 +   │  ├─ kubernetes:core:Service                pulumi-wp-0/pulumi-wp-0-wordpress-mariadb             create
 +   │  ├─ kubernetes:core:Service                pulumi-wp-0/pulumi-wp-0-wordpress-wordpress           create
 +   │  ├─ kubernetes:core:Pod                    pulumi-wp-0/pulumi-wp-0-wordpress-mariadb-test-ssfgf  create
 +   │  ├─ kubernetes:core:Pod                    pulumi-wp-0/pulumi-wp-0-wordpress-credentials-test    create
 +   │  ├─ kubernetes:extensions:Deployment       pulumi-wp-0/pulumi-wp-0-wordpress-wordpress           create
 +   │  └─ kubernetes:apps:StatefulSet            pulumi-wp-0/pulumi-wp-0-wordpress-mariadb             create
 +   ├─ kubernetes:helm.sh:Chart                  pulumi-wp-1-wordpress                                 create
 +   │  ├─ kubernetes:core:Service                pulumi-wp-1/pulumi-wp-1-wordpress-mariadb             create
 +   │  ├─ kubernetes:core:Secret                 pulumi-wp-1/pulumi-wp-1-wordpress-mariadb             create
 +   │  ├─ kubernetes:core:Secret                 pulumi-wp-1/pulumi-wp-1-wordpress-wordpress           create
 +   │  ├─ kubernetes:extensions:Deployment       pulumi-wp-1/pulumi-wp-1-wordpress-wordpress           create
 +   │  ├─ kubernetes:core:PersistentVolumeClaim  pulumi-wp-1/pulumi-wp-1-wordpress-wordpress           create
 +   │  ├─ kubernetes:core:Service                pulumi-wp-1/pulumi-wp-1-wordpress-wordpress           create
 +   │  ├─ kubernetes:core:ConfigMap              pulumi-wp-1/pulumi-wp-1-wordpress-mariadb-tests       create
 +   │  ├─ kubernetes:core:Pod                    pulumi-wp-1/pulumi-wp-1-wordpress-credentials-test    create
 +   │  ├─ kubernetes:core:Pod                    pulumi-wp-1/pulumi-wp-1-wordpress-mariadb-test-buwaj  create
 +   │  ├─ kubernetes:apps:StatefulSet            pulumi-wp-1/pulumi-wp-1-wordpress-mariadb             create
 +   │  └─ kubernetes:core:ConfigMap              pulumi-wp-1/pulumi-wp-1-wordpress-mariadb             create

Kinda gross, right? I need a pretty long name for the Chart resource to guarantee stack-wide uniqueness. But then it also gets put in the resulting k8s objects. When I had a longer namespaceName prefix while initially devving, this even resulted in the StatefulSet (maybe its pvc?) having a name that was longer than 65 characters so k8s failed to create it fully and pulumi up failed!

What I think I want is something like: (note top two Chart options):

const numNamespaces = 2
for (const namespaceIndex of (new Array(numNamespaces).fill(0).map((_, index) => index))) {
    const namespaceName = `pulumi-wp-${namespaceIndex}`
    const namespace = new k8s.core.v1.Namespace(namespaceName, {
        metadata: {
            name: namespaceName
        }
    })
    const wordpressChart = new k8s.helm.v2.Chart(`${namespaceName}/wordpress`, {
                release: {
                  name: 'wordpress'
                },
                // makes pulumi name for k8s resource templated by helm chart
                templatedResourceNamer: (k8sManifest) => `${namespaceName}/${k8sManifest.metadata.name}`,
        namespace: pulumi.output(namespace).apply(namespace => namespace.metadata.name),
        repo: "stable",
        version: "2.1.3",
        chart: "wordpress",
        transformations: [addNamespaceTransformation(namespaceName)]
    }); 
}

This opt-in config would give me:

• pretty consistent pulumi resource names between the helm chart and the k8s objects templated by it (i.e. with a namespace/ prefix)
• opt-in so URNs wouldn't change by default
• ability to make separate naming decisions about my helm.Chart pulumi resource (must be unique per stack), the pulumi resource names of the resulting k8s objects, and the helm release names that usually affect my k8s object names (which must be unique per k8s namespace, and release names conventionally so)

Hope this makes sense. Either way hope the sample pulumi program is something yall can use to test out how new naming options would be used.

(as a small aside, it would be rad if the helm.Chart transformations option supported resolving inputs or other async logic so I could rely on namespace autonaming, but that's a whole separate thing. I realize my use of it here is a workaround to be fixed soon.)