Pulumi-kubernetes: Can't install Prometheus Operator via Helm3: Duplicate resource URN

Created on 29 Jul 2020  ·  23Comments  ·  Source: pulumi/pulumi-kubernetes

Problem description

Using Pulumi 2.7.1, I've been trying to install the prometheus-operator Helm chart from the stable repo, version 9.2.2.

I'm using the following Pulumi definition to do it:

public static void DeployPrometheus(CustomResourceOptions k8sCustomResourceOptions)
{
    var appName = "prometheus";
    var appLabels = new InputMap<string>
    {
        {"app", appName}
    };

    /**
     * Enable Prometheus data storage on a persistent volume claim
     */
    // See https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv for more details on provisioning
    var prometheusVolumeClaim = new Pulumi.Kubernetes.Core.V1.PersistentVolumeClaim("promtheus-volume-claim", new PersistentVolumeClaimArgs()
    {
        Metadata = new ObjectMetaArgs
        {
            Name = $"prometheus",
        },
        ApiVersion = "v1",
        Spec = new PersistentVolumeClaimSpecArgs
        {
            AccessModes = "ReadWriteOnce", // can only be mounted and written to by a single 
            StorageClassName = "default", // standard disk,
            Resources = new ResourceRequirementsArgs
            {
                Requests =
                {
                    // TODO: make storage amounts configurable
                    { "storage", "15Gi"} // 15Gb of storage
                }
            }
        }
    }, k8sCustomResourceOptions);

    // need to work around CRD bug here https://github.com/pulumi/pulumi-kubernetes/issues/1023
    //var configGroup = new ConfigGroup("prometheus-crds", new ConfigGroupArgs
    //{
    //    Files = new[]{ "https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml",
    //    "https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml",
    //    "https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml",
    //    "https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml",
    //    "https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml",
    //    "https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml"},
    //}, new ComponentResourceOptions()
    //{
    //    Provider = k8sCustomResourceOptions.Provider
    //});

    var prometheusChartValues = prometheusVolumeClaim.Metadata.Apply(f => new Dictionary<string, object>()
    {
        /* define Kubelet metrics scraping */
        ["kubelet"] = new Dictionary<string, object>
        {
            ["enabled"] = true,
            ["serviceMonitor"] = new Dictionary<string, object>
            {
                ["https"] = false
            },
        },
        ["kubeControllerManager"] = new Dictionary<string, object>
        {
            ["enabled"] = false
        },
        ["kubeScheduler"] = new Dictionary<string, object>
        {
            ["enabled"] = false
        },
        ["kubeEtcd"] = new Dictionary<string, object>
        {
            ["enabled"] = false
        },
        ["kubeProxy"] = new Dictionary<string, object>
        {
            ["enabled"] = false
        },
        ["prometheusOperator"] = new 
        {
            createCustomResource = false
        },
        ["prometheus"] = new Dictionary<string, object>
        {
            ["prometheusSpec"] = new Dictionary<string, object>
            {
                ["volumes"] = new[]
                {
                    new Dictionary<string, object>
                    {
                        ["name"] = "prometheus-storage",
                        ["persistentVolumeClaim"] = new Dictionary<string, object>{
                            ["claimName"] = f.Name
                        },
                    }
                },
                ["volumeMounts"] = new[] {
                    new Dictionary<string, object>
                    {
                        ["name"] = "prometheus-storage",
                        ["mountPath"] = "/prometheus/"
                    }
                }
            }
        },
    });

    var objectNames = new ConcurrentBag<string>();

    ImmutableDictionary<string, object> OmitCustomResource(ImmutableDictionary<string, object> obj, CustomResourceOptions opts)
    {
        var metadata = (ImmutableDictionary<string, object>)obj["metadata"];
        var resourceName = $"{obj["kind"]}.{metadata["name"]}";

        if (objectNames.Contains(resourceName))
        {
            Console.WriteLine($"Deduplicated [{resourceName}]");
            return new Dictionary<string, object>
            {
                ["apiVersion"] = "v1",
                ["kind"] = "List",
                ["items"] = new Dictionary<string, object>(),
            }.ToImmutableDictionary();
        }
        objectNames.Add(resourceName);
        Console.WriteLine(resourceName);
        return obj;
    }

    var prometheusOperatorChart = new Chart("pm", new Pulumi.Kubernetes.Helm.ChartArgs
    {
        Repo = "stable",
        Chart = "prometheus-operator",
        Version = "9.2.2",
        Namespace = "monitoring",
        Values = prometheusChartValues,
        Transformations = { OmitCustomResource }
    }, new ComponentResourceOptions()
    {
        Provider = k8sCustomResourceOptions.Provider,
        //DependsOn = configGroup
    });
}

I'm using the transformation mechanism for the Helm chart to attempt to resolve the error

Errors & Logs

I get a similar error to this one each time I attempt to run this chart installation:

 error: Duplicate resource URN 'urn:pulumi:dev::sdkbin-resources::kubernetes:helm.sh/v2:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/pm-prometheus-operator-alertmanager'; try giving it a unique name

It's always one of the custom resource definitions - it's not always the same one that throws the error though. This time it was the alertmanager CRD that caused the error - other times it's been the prometheus CRD or the node-exporter.

I attempted to fix this issue by adding the OmitCustomResource filtering code to my application to omit any duplicates - I also added some logging to see what those filtered / unfiltered outputs were:

  1. CustomResourceDefinition.alertmanagers.monitoring.coreos.com
  2. CustomResourceDefinition.podmonitors.monitoring.coreos.com
  3. CustomResourceDefinition.prometheuses.monitoring.coreos.com
  4. CustomResourceDefinition.prometheusrules.monitoring.coreos.com
  5. CustomResourceDefinition.servicemonitors.monitoring.coreos.com
  6. CustomResourceDefinition.thanosrulers.monitoring.coreos.com
  7. PodSecurityPolicy.pm-grafana
  8. PodSecurityPolicy.pm-grafana-test
  9. PodSecurityPolicy.pm-kube-state-metrics
  10. PodSecurityPolicy.pm-prometheus-node-exporter
  11. PodSecurityPolicy.pm-prometheus-operator-alertmanager
  12. PodSecurityPolicy.pm-prometheus-operator-operator
  13. PodSecurityPolicy.pm-prometheus-operator-prometheus
  14. ServiceAccount.pm-grafana
  15. ServiceAccount.pm-grafana-test
  16. ServiceAccount.pm-kube-state-metrics
  17. ServiceAccount.pm-prometheus-node-exporter
  18. ServiceAccount.pm-prometheus-operator-alertmanager
  19. ServiceAccount.pm-prometheus-operator-operator
  20. ServiceAccount.pm-prometheus-operator-prometheus
  21. Secret.pm-grafana
  22. Secret.alertmanager-pm-prometheus-operator-alertmanager
  23. ConfigMap.pm-grafana-config-dashboards
  24. ConfigMap.pm-grafana
  25. ConfigMap.pm-grafana-test
  26. ConfigMap.pm-prometheus-operator-grafana-datasource
  27. ConfigMap.pm-prometheus-operator-apiserver
  28. ConfigMap.pm-prometheus-operator-cluster-total
  29. ConfigMap.pm-prometheus-operator-k8s-coredns
  30. ConfigMap.pm-prometheus-operator-k8s-resources-cluster
  31. ConfigMap.pm-prometheus-operator-k8s-resources-namespace
  32. ConfigMap.pm-prometheus-operator-k8s-resources-node
  33. ConfigMap.pm-prometheus-operator-k8s-resources-pod
  34. ConfigMap.pm-prometheus-operator-k8s-resources-workload
  35. ConfigMap.pm-prometheus-operator-k8s-resources-workloads-namespace
  36. ConfigMap.pm-prometheus-operator-kubelet
  37. ConfigMap.pm-prometheus-operator-namespace-by-pod
  38. ConfigMap.pm-prometheus-operator-namespace-by-workload
  39. ConfigMap.pm-prometheus-operator-node-cluster-rsrc-use
  40. ConfigMap.pm-prometheus-operator-node-rsrc-use
  41. ConfigMap.pm-prometheus-operator-nodes
  42. ConfigMap.pm-prometheus-operator-persistentvolumesusage
  43. ConfigMap.pm-prometheus-operator-pod-total
  44. ConfigMap.pm-prometheus-operator-prometheus
  45. ConfigMap.pm-prometheus-operator-statefulset
  46. ConfigMap.pm-prometheus-operator-workload-total
  47. ClusterRole.pm-grafana-clusterrole
  48. ClusterRole.pm-kube-state-metrics
  49. ClusterRole.psp-pm-kube-state-metrics
  50. ClusterRole.psp-pm-prometheus-node-exporter
  51. ClusterRole.pm-prometheus-operator-operator
  52. ClusterRole.pm-prometheus-operator-operator-psp
  53. ClusterRole.pm-prometheus-operator-prometheus
  54. ClusterRole.pm-prometheus-operator-prometheus-psp
  55. ClusterRoleBinding.pm-grafana-clusterrolebinding
  56. ClusterRoleBinding.pm-kube-state-metrics
  57. ClusterRoleBinding.psp-pm-kube-state-metrics
  58. ClusterRoleBinding.psp-pm-prometheus-node-exporter
  59. ClusterRoleBinding.pm-prometheus-operator-operator
  60. ClusterRoleBinding.pm-prometheus-operator-operator-psp
  61. ClusterRoleBinding.pm-prometheus-operator-prometheus
  62. ClusterRoleBinding.pm-prometheus-operator-prometheus-psp
  63. Role.pm-grafana
  64. Role.pm-grafana-test
  65. Role.pm-prometheus-operator-alertmanager
  66. RoleBinding.pm-grafana
  67. RoleBinding.pm-grafana-test
  68. RoleBinding.pm-prometheus-operator-alertmanager
  69. Service.pm-grafana
  70. Service.pm-kube-state-metrics
  71. Service.pm-prometheus-node-exporter
  72. Service.pm-prometheus-operator-alertmanager
  73. Service.pm-prometheus-operator-coredns
  74. Service.pm-prometheus-operator-operator
  75. Service.pm-prometheus-operator-prometheus
  76. DaemonSet.pm-prometheus-node-exporter
  77. Deployment.pm-grafana
  78. Deployment.pm-kube-state-metrics
  79. Deployment.pm-prometheus-operator-operator
  80. Alertmanager.pm-prometheus-operator-alertmanager
  81. MutatingWebhookConfiguration.pm-prometheus-operator-admission
  82. Prometheus.pm-prometheus-operator-prometheus
  83. PrometheusRule.pm-prometheus-operator-alertmanager.rules
  84. PrometheusRule.pm-prometheus-operator-general.rules
  85. PrometheusRule.pm-prometheus-operator-k8s.rules
  86. PrometheusRule.pm-prometheus-operator-kube-apiserver-availability.rules
  87. PrometheusRule.pm-prometheus-operator-kube-apiserver-slos
  88. PrometheusRule.pm-prometheus-operator-kube-apiserver.rules
  89. PrometheusRule.pm-prometheus-operator-kube-prometheus-general.rules
  90. PrometheusRule.pm-prometheus-operator-kube-prometheus-node-recording.rules
  91. PrometheusRule.pm-prometheus-operator-kube-state-metrics
  92. PrometheusRule.pm-prometheus-operator-kubelet.rules
  93. PrometheusRule.pm-prometheus-operator-kubernetes-apps
  94. PrometheusRule.pm-prometheus-operator-kubernetes-resources
  95. PrometheusRule.pm-prometheus-operator-kubernetes-storage
  96. PrometheusRule.pm-prometheus-operator-kubernetes-system-apiserver
  97. PrometheusRule.pm-prometheus-operator-kubernetes-system-kubelet
  98. PrometheusRule.pm-prometheus-operator-kubernetes-system
  99. PrometheusRule.pm-prometheus-operator-node-exporter.rules
  100. PrometheusRule.pm-prometheus-operator-node-exporter
  101. PrometheusRule.pm-prometheus-operator-node-network
  102. PrometheusRule.pm-prometheus-operator-node.rules
  103. PrometheusRule.pm-prometheus-operator-prometheus-operator
  104. PrometheusRule.pm-prometheus-operator-prometheus
  105. ServiceMonitor.pm-prometheus-operator-alertmanager
  106. ServiceMonitor.pm-prometheus-operator-coredns
  107. ServiceMonitor.pm-prometheus-operator-apiserver
  108. ServiceMonitor.pm-prometheus-operator-kube-state-metrics
  109. ServiceMonitor.pm-prometheus-operator-kubelet
  110. ServiceMonitor.pm-prometheus-operator-node-exporter
  111. ServiceMonitor.pm-prometheus-operator-grafana
  112. ServiceMonitor.pm-prometheus-operator-operator
  113. ServiceMonitor.pm-prometheus-operator-prometheus
  114. ValidatingWebhookConfiguration.pm-prometheus-operator-admission
  115. PodSecurityPolicy.pm-prometheus-operator-admission
  116. ServiceAccount.pm-prometheus-operator-admission
  117. ClusterRole.pm-prometheus-operator-admission
  118. ClusterRoleBinding.pm-prometheus-operator-admission
  119. Role.pm-prometheus-operator-admission
  120. RoleBinding.pm-prometheus-operator-admission
  121. Pod.pm-grafana-test
  122. Job.pm-prometheus-operator-admission-create
  123. Job.pm-prometheus-operator-admission-patch

As you can see, there were no duplicates included in this list - nor did my logging code for finding duplicates catch any.

Affected product version(s)

2.7.1

Reproducing the issue

The code I included above will reproduce it.

Suggestions for a fix

I've tried looking at other similar issues in this repo, such as https://github.com/pulumi/pulumi-kubernetes/issues/1023 - it's not clear to me what the issue is here but I suspect it has something to do with the prometheus-operator chart pulling in other charts that all create the same custom resources under the covers. If my transform could also apply to those dependent charts that would be wonderful.

Or is my issue something else?

arehelm kinenhancement
Source
picture Aaronontheweb

Most helpful comment

The problem here isn't with the Kubernetes CustomResourceDefinition, but with the URN generation of Pulumi.

Specifically in @Aaronontheweb case, there is an AlertManager generated by the helm chart called monitoring/pm-prometheus-operator-alertmanager. But there's also a ServiceMonitor called monitoring/pm-prometheus-operator-alertmanager.

Because these are not standard Kubernetes resource kinds, Pulumi generates a URN for a CustomResourceDefinition type (is this the fallback type for unknown resource kinds?). The result is 2 resources with URN

urn:pulumi:dev::sdkbin-resources::kubernetes:helm.sh/v2:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/pm-prometheus-operator-alertmanager

Kubernetes has no issue with resources named the same, as long as they are not of the same Kind. From the helm chart perspective there is no issue. I would consider it a bug/missing feature in Pulumi. It would be nice if Pulumi just takes the Kubernetes resource Kind in the URN, also for unknown kinds.

I worked around it by applying this transform

Transformations = {
    (obj, opt) =>
    {
         // Transform the name with suffix to prevent URN collisions
         obj = AddNameSuffixForKind(
         kind: "ServiceMonitor",
         suffix: "service-monitor",
         obj,
         opt);

         // Transform the name with suffix to prevent URN collisions
         obj = AddNameSuffixForKind(
         kind: "AlertManager",
         suffix: "alert-manager",
         obj,
         opt);

        return obj;
    }
...
static ImmutableDictionary<string, object> AddNameSuffixForKind(
            string kind,
            string suffix,
            ImmutableDictionary<string, object> obj,
            CustomResourceOptions opts)
        {
            if ((string)obj["kind"] == kind)
            {
                var metaData = (ImmutableDictionary<string, object>)obj["metadata"];
                var name = metaData["name"];
                metaData = metaData.SetItem("name", $"{name}-{suffix}");
                obj = obj.SetItem("metadata", metaData);

                return obj;
            }

            return obj;
        }
picture Freakazoid182 on 13 Aug 2020
👍4 ❤1

All 23 comments

This error usually means you have two resources with the same type, namespace and name. Almost always this is a bug in the Helm chart. In particular, it looks like you're creating the monitoring/pm-prometheus-operator-alertmanager CRD twice.

Alex

Theory is the first term in the Taylor series of practice. -- Thomas M Cover (1992)

On Wed, Jul 29, 2020 at 3:30 PM, Aaron Stannard < [email protected] > wrote:

Problem description

Using Pulumi 2.7.1, I've been trying to install the prometheus-operator
Helm chart from the stable repo (
https://github.com/helm/charts/tree/master/stable/prometheus-operator ) ,
version 9.2.2.

I'm using the following Pulumi definition to do it:

public static void DeployPrometheus ( CustomResourceOptions k8sCustomResourceOptions
)
{
var appName = " prometheus " ;
var appLabels = new InputMap < string >
{
{ " app " , appName }
};

/* *
 * Enable Prometheus data storage on a persistent volume claim
 */
// See https:/ / docs. microsoft. com/ en-us/ azure/ aks/ azure-disks-dynamic-pv

( https://docs.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv ) for
more details on provisioning
var prometheusVolumeClaim = new Pulumi. Kubernetes. Core. V1. PersistentVolumeClaim
( " promtheus-volume-claim " , new PersistentVolumeClaimArgs ()
{

Metadata = new ObjectMetaArgs
{
Name = $" prometheus " ,
},
ApiVersion = " v1 " ,
Spec = new PersistentVolumeClaimSpecArgs
{
AccessModes = " ReadWriteOnce " , // can only be mounted and
written to by a single
StorageClassName = " default " , // standard disk,

Resources = new ResourceRequirementsArgs
{
Requests =
{
// TODO: make storage amounts configurable

{ " storage " , " 15Gi " } // 15Gb of storage
}

}
}
}, k8sCustomResourceOptions );

// need to work around CRD bug here https:/ / github. com/ pulumi/ pulumi-kubernetes/

issues/ 1023 ( https://github.com/pulumi/pulumi-kubernetes/issues/1023 )

// var configGroup = new ConfigGroup("prometheus-crds", new ConfigGroupArgs

// {
//    Files = new[]{ " https:/ / raw. githubusercontent. com/ coreos/ prometheus-operator/

release-0. 38/ example/ prometheus-operator-crd/ monitoring. coreos. com_alertmanagers.
yaml (
https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml
) ",
// " https:/ / raw. githubusercontent. com/ coreos/ prometheus-operator/
release-0. 38/ example/ prometheus-operator-crd/ monitoring. coreos. com_podmonitors.
yaml (
https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml
) ",
// " https:/ / raw. githubusercontent. com/ coreos/ prometheus-operator/
release-0. 38/ example/ prometheus-operator-crd/ monitoring. coreos. com_prometheuses.
yaml (
https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml
) ",
// " https:/ / raw. githubusercontent. com/ coreos/ prometheus-operator/
release-0. 38/ example/ prometheus-operator-crd/ monitoring. coreos. com_prometheusrules.
yaml (
https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
) ",
// " https:/ / raw. githubusercontent. com/ coreos/ prometheus-operator/
release-0. 38/ example/ prometheus-operator-crd/ monitoring. coreos. com_servicemonitors.
yaml (
https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml
) ",
// " https:/ / raw. githubusercontent. com/ coreos/ prometheus-operator/
release-0. 38/ example/ prometheus-operator-crd/ monitoring. coreos. com_thanosrulers.
yaml (
https://raw.githubusercontent.com/coreos/prometheus-operator/release-0.38/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml
) "},
// }, new ComponentResourceOptions()
// {
// Provider = k8sCustomResourceOptions.Provider
// });

var prometheusChartValues = prometheusVolumeClaim. Metadata. Apply ( f

=> new Dictionary < string , object >()
{
/* define Kubelet metrics scraping */
[ " kubelet " ] = new Dictionary < string , object >
{

[ " enabled " ] = true ,
[ " serviceMonitor " ] = new Dictionary < string , object >

{
[ " https " ] = false
},
},
[ " kubeControllerManager " ] = new Dictionary < string , object >

{
[ " enabled " ] = false
},
[ " kubeScheduler " ] = new Dictionary < string , object >

{
[ " enabled " ] = false
},
[ " kubeEtcd " ] = new Dictionary < string , object >
{

[ " enabled " ] = false
},
[ " kubeProxy " ] = new Dictionary < string , object >
{

[ " enabled " ] = false
},
[ " prometheusOperator " ] = new
{
createCustomResource = false
},
[ " prometheus " ] = new Dictionary < string , object >
{

[ " prometheusSpec " ] = new Dictionary < string , object >
{

[ " volumes " ] = new []
{
new Dictionary < string , object >

{
[ " name " ] = " prometheus-storage " ,

[ " persistentVolumeClaim " ] = new Dictionary < string , object >{

[ " claimName " ] = f. Name
},

}
},
[ " volumeMounts " ] = new [] {
new Dictionary < string , object >

{
[ " name " ] = " prometheus-storage " ,

[ " mountPath " ] = " /prometheus/ "
}

}
}
},
});

var objectNames = new ConcurrentBag < string >();

ImmutableDictionary < string , object > OmitCustomResource ( ImmutableDictionary

< string , object > obj , CustomResourceOptions opts )
{
var metadata = ( ImmutableDictionary < string , object >) obj [ " metadata
" ];
var resourceName = $" { obj [ " kind " ]}.{ metadata [ " name " ]}
" ;

    if ( objectNames. Contains ( resourceName ))
    {
        Console. WriteLine ( $" Deduplicated [{ resourceName }] " );

return new Dictionary < string , object >
{
[ " apiVersion " ] = " v1 " ,
[ " kind " ] = " List " ,
[ " items " ] = new Dictionary < string , object >(),

}. ToImmutableDictionary ();
}
objectNames. Add ( resourceName );
Console. WriteLine ( resourceName );
return obj ;
}

var prometheusOperatorChart = new Chart ( " pm " , new Pulumi. Kubernetes

. Helm. ChartArgs
{
Repo = " stable " ,
Chart = " prometheus-operator " ,
Version = " 9.2.2 " ,
Namespace = " monitoring " ,
Values = prometheusChartValues ,
Transformations = { OmitCustomResource }
}, new ComponentResourceOptions ()
{
Provider = k8sCustomResourceOptions. Provider ,
// DependsOn = configGroup
});
}

I'm using the transformation mechanism for the Helm chart to attempt to
resolve the error

Errors & Logs

I get a similar error to this one each time I attempt to run this chart
installation:

error: Duplicate resource URN
'urn:pulumi:dev::sdkbin-resources::kubernetes: helm. sh/ v2:Chart$kubernetes:apiextensions.
k8s. io/ v1:CustomResourceDefinition::monitoring/ pm-prometheus-operator-alertmanager
(
http://helm.sh/v2:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/pm-prometheus-operator-alertmanager
) '; try giving it a unique name

It's always one of the custom resource definitions - it's not always the
same one that throws the error though. This time it was the alertmanager CRD
that caused the error - other times it's been the prometheus CRD or the node-exporter
.

I attempted to fix this issue by adding the OmitCustomResource filtering
code to my application to omit any duplicates - I also added some logging
to see what those filtered / unfiltered outputs were:

  • CustomResourceDefinition. alertmanagers. monitoring. coreos. com (
    http://customresourcedefinition.alertmanagers.monitoring.coreos.com/ )
  • CustomResourceDefinition. podmonitors. monitoring. coreos. com (
    http://customresourcedefinition.podmonitors.monitoring.coreos.com/ )
  • CustomResourceDefinition. prometheuses. monitoring. coreos. com (
    http://customresourcedefinition.prometheuses.monitoring.coreos.com/ )
  • CustomResourceDefinition. prometheusrules. monitoring. coreos. com (
    http://customresourcedefinition.prometheusrules.monitoring.coreos.com/ )
  • CustomResourceDefinition. servicemonitors. monitoring. coreos. com (
    http://customresourcedefinition.servicemonitors.monitoring.coreos.com/ )
  • CustomResourceDefinition. thanosrulers. monitoring. coreos. com (
    http://customresourcedefinition.thanosrulers.monitoring.coreos.com/ )
  • PodSecurityPolicy.pm-grafana
  • PodSecurityPolicy.pm-grafana-test
  • PodSecurityPolicy.pm-kube-state-metrics
  • PodSecurityPolicy.pm-prometheus-node-exporter
  • PodSecurityPolicy.pm-prometheus-operator-alertmanager
  • PodSecurityPolicy.pm-prometheus-operator-operator
  • PodSecurityPolicy.pm-prometheus-operator-prometheus
  • ServiceAccount.pm-grafana
  • ServiceAccount.pm-grafana-test
  • ServiceAccount.pm-kube-state-metrics
  • ServiceAccount.pm-prometheus-node-exporter
  • ServiceAccount.pm-prometheus-operator-alertmanager
  • ServiceAccount.pm-prometheus-operator-operator
  • ServiceAccount.pm-prometheus-operator-prometheus
  • Secret.pm-grafana
  • Secret.alertmanager-pm-prometheus-operator-alertmanager
  • ConfigMap.pm-grafana-config-dashboards
  • ConfigMap.pm-grafana
  • ConfigMap.pm-grafana-test
  • ConfigMap.pm-prometheus-operator-grafana-datasource
  • ConfigMap.pm-prometheus-operator-apiserver
  • ConfigMap.pm-prometheus-operator-cluster-total
  • ConfigMap.pm-prometheus-operator-k8s-coredns
  • ConfigMap.pm-prometheus-operator-k8s-resources-cluster
  • ConfigMap.pm-prometheus-operator-k8s-resources-namespace
  • ConfigMap.pm-prometheus-operator-k8s-resources-node
  • ConfigMap.pm-prometheus-operator-k8s-resources-pod
  • ConfigMap.pm-prometheus-operator-k8s-resources-workload
  • ConfigMap.pm-prometheus-operator-k8s-resources-workloads-namespace
  • ConfigMap.pm-prometheus-operator-kubelet
  • ConfigMap.pm-prometheus-operator-namespace-by-pod
  • ConfigMap.pm-prometheus-operator-namespace-by-workload
  • ConfigMap.pm-prometheus-operator-node-cluster-rsrc-use
  • ConfigMap.pm-prometheus-operator-node-rsrc-use
  • ConfigMap.pm-prometheus-operator-nodes
  • ConfigMap.pm-prometheus-operator-persistentvolumesusage
  • ConfigMap.pm-prometheus-operator-pod-total
  • ConfigMap.pm-prometheus-operator-prometheus
  • ConfigMap.pm-prometheus-operator-statefulset
  • ConfigMap.pm-prometheus-operator-workload-total
  • ClusterRole.pm-grafana-clusterrole
  • ClusterRole.pm-kube-state-metrics
  • ClusterRole.psp-pm-kube-state-metrics
  • ClusterRole.psp-pm-prometheus-node-exporter
  • ClusterRole.pm-prometheus-operator-operator
  • ClusterRole.pm-prometheus-operator-operator-psp
  • ClusterRole.pm-prometheus-operator-prometheus
  • ClusterRole.pm-prometheus-operator-prometheus-psp
  • ClusterRoleBinding.pm-grafana-clusterrolebinding
  • ClusterRoleBinding.pm-kube-state-metrics
  • ClusterRoleBinding.psp-pm-kube-state-metrics
  • ClusterRoleBinding.psp-pm-prometheus-node-exporter
  • ClusterRoleBinding.pm-prometheus-operator-operator
  • ClusterRoleBinding.pm-prometheus-operator-operator-psp
  • ClusterRoleBinding.pm-prometheus-operator-prometheus
  • ClusterRoleBinding.pm-prometheus-operator-prometheus-psp
  • Role.pm-grafana
  • Role.pm-grafana-test
  • Role.pm-prometheus-operator-alertmanager
  • RoleBinding.pm-grafana
  • RoleBinding.pm-grafana-test
  • RoleBinding.pm-prometheus-operator-alertmanager
  • Service.pm-grafana
  • Service.pm-kube-state-metrics
  • Service.pm-prometheus-node-exporter
  • Service.pm-prometheus-operator-alertmanager
  • Service.pm-prometheus-operator-coredns
  • Service.pm-prometheus-operator-operator
  • Service.pm-prometheus-operator-prometheus
  • DaemonSet.pm-prometheus-node-exporter
  • Deployment.pm-grafana
  • Deployment.pm-kube-state-metrics
  • Deployment.pm-prometheus-operator-operator
  • Alertmanager.pm-prometheus-operator-alertmanager
  • MutatingWebhookConfiguration.pm-prometheus-operator-admission
  • Prometheus.pm-prometheus-operator-prometheus
  • PrometheusRule.pm-prometheus-operator-alertmanager.rules
  • PrometheusRule.pm-prometheus-operator-general.rules
  • PrometheusRule.pm-prometheus-operator-k8s.rules
  • PrometheusRule.pm-prometheus-operator-kube-apiserver-availability.rules
  • PrometheusRule.pm-prometheus-operator-kube-apiserver-slos
  • PrometheusRule.pm-prometheus-operator-kube-apiserver.rules
  • PrometheusRule.pm-prometheus-operator-kube-prometheus-general.rules

  • PrometheusRule.pm-prometheus-operator-kube-prometheus-node-recording.rules

  • PrometheusRule.pm-prometheus-operator-kube-state-metrics

  • PrometheusRule.pm-prometheus-operator-kubelet.rules
  • PrometheusRule.pm-prometheus-operator-kubernetes-apps
  • PrometheusRule.pm-prometheus-operator-kubernetes-resources
  • PrometheusRule.pm-prometheus-operator-kubernetes-storage
  • PrometheusRule.pm-prometheus-operator-kubernetes-system-apiserver
  • PrometheusRule.pm-prometheus-operator-kubernetes-system-kubelet
  • PrometheusRule.pm-prometheus-operator-kubernetes-system
  • PrometheusRule.pm-prometheus-operator-node-exporter.rules
  • PrometheusRule.pm-prometheus-operator-node-exporter
  • PrometheusRule.pm-prometheus-operator-node-network
  • PrometheusRule.pm-prometheus-operator-node.rules
  • PrometheusRule.pm-prometheus-operator-prometheus-operator
  • PrometheusRule.pm-prometheus-operator-prometheus
  • ServiceMonitor.pm-prometheus-operator-alertmanager
  • ServiceMonitor.pm-prometheus-operator-coredns
  • ServiceMonitor.pm-prometheus-operator-apiserver
  • ServiceMonitor.pm-prometheus-operator-kube-state-metrics
  • ServiceMonitor.pm-prometheus-operator-kubelet
  • ServiceMonitor.pm-prometheus-operator-node-exporter
  • ServiceMonitor.pm-prometheus-operator-grafana
  • ServiceMonitor.pm-prometheus-operator-operator
  • ServiceMonitor.pm-prometheus-operator-prometheus
  • ValidatingWebhookConfiguration.pm-prometheus-operator-admission
  • PodSecurityPolicy.pm-prometheus-operator-admission
  • ServiceAccount.pm-prometheus-operator-admission
  • ClusterRole.pm-prometheus-operator-admission
  • ClusterRoleBinding.pm-prometheus-operator-admission
  • Role.pm-prometheus-operator-admission
  • RoleBinding.pm-prometheus-operator-admission
  • Pod.pm-grafana-test
  • Job.pm-prometheus-operator-admission-create
  • Job.pm-prometheus-operator-admission-patch

As you can see, there were no duplicates included in this list - nor did
my logging code for finding duplicates catch any.

Affected product version(s)

2.7.1

Reproducing the issue

The code I included above will reproduce it.

Suggestions for a fix

I've tried looking at other similar issues in this repo, such as #1023 (
https://github.com/pulumi/pulumi-kubernetes/issues/1023 ) - it's not clear
to me what the issue is here but I suspect it has something to do with the
prometheus-operator chart pulling in other charts that all create the same
custom resources under the covers. If my transform could also apply to
those dependent charts (
https://github.com/helm/charts/blob/master/stable/prometheus-operator/requirements.yaml
) that would be wonderful.

Or is my issue something else?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub (
https://github.com/pulumi/pulumi-kubernetes/issues/1225 ) , or unsubscribe
(
https://github.com/notifications/unsubscribe-auth/AAKYBBEOFSDYTWISUFSVOL3R6CIHLANCNFSM4PMCYDFQ
).

hausdorff picture hausdorff on 29 Jul 2020

@hausdorff I figured it was an issue with the chart - specifically, an issue with the chart and one or more of its dependencies both attempting to create the same dependent resource type. Is there a way I can improve my code to work around this problem?

Aaronontheweb picture Aaronontheweb on 30 Jul 2020

As a hack you could use the transforms API in the ChartOpts to find one of those CRD definitions and replace it with an empty v1.List. Or undefined may work—I forget.

There's probably a flag somewhere that's causing it to get double-generated though.

Alex

Theory is the first term in the Taylor series of practice. -- Thomas M Cover (1992)

On Wed, Jul 29, 2020 at 4:25 PM, Aaron Stannard < [email protected] > wrote:

@ hausdorff ( https://github.com/hausdorff ) I figured it was an issue with
the chart - specifically, an issue with the chart and one or more of its
dependencies both attempting to create the same dependent resource type.
Is there a way I can improve my code to work around this problem?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (
https://github.com/pulumi/pulumi-kubernetes/issues/1225#issuecomment-665962144
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAKYBBF7JXPEGDB6WL5EQMLR6COV5ANCNFSM4PMCYDFQ
).

hausdorff picture hausdorff on 30 Jul 2020

I have come across similar issue when trying to install KNative serving from KNative docs specifically when installing CRD definitions and core components

kmr0877 picture kmr0877 on 30 Jul 2020

Yes. This is extremely common. Most Charts have lots of bugs exactly like this.

Alex

Theory is the first term in the Taylor series of practice. -- Thomas M Cover (1992)

On Wed, Jul 29, 2020 at 6:28 PM, Mohan < [email protected] > wrote:

I have come across similar issue when trying to install KNative serving
from KNative docs (
https://knative.dev/docs/install/any-kubernetes-cluster/ ) specifically
when installing CRD definitions and core components

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (
https://github.com/pulumi/pulumi-kubernetes/issues/1225#issuecomment-666000850
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAKYBBBRMC6565OCPHAVKDLR6C5ERANCNFSM4PMCYDFQ
).

hausdorff picture hausdorff on 30 Jul 2020

Yes. This is extremely common. Most Charts have lots of bugs exactly like this.

If that's the case then it severely limits Pulumi's usefulness if we can't install most Helm charts.

Aaronontheweb picture Aaronontheweb on 31 Jul 2020
👍1

Yes. This is extremely common. Most Charts have lots of bugs exactly like this.

If that's the case then it severely limits Pulumi's usefulness if we can't install most Helm charts.

FWIW - I do not consider this "a bug in the charts" exactly. Since the charts do work correctly via the Helm CLI, it is a bug in Pulumi's support for deploying Helm charts that we do not handle this smoothly as well - even if it is "poor design on behalf of the chart author".

We should definitely look into what we can do to fix/improve Pulumi's support for charts like these (especially as the majority of the most used charts have this sort of "issue").

lukehoban picture lukehoban on 31 Jul 2020
👍2

@lblackstone and I are having a discussion on how best to support this. It seems this was introduced by #1102 and we do need to figure out a way to either
A) let the user opt-out of installing the CRDs, because they might be installed out of band by another process
B) detect if the crd is in the api and omit them if automatically.

The workaround at the moment is to use a transformation to drop the resource, something a bit like this:

transformations: [
          (obj: any, opts: pulumi.CustomResourceOptions) => {
            if (obj.kind === "CustomResourceDefinition" && (obj.metadata.name === "postgresqls.acid.zalan.do" || obj.metadata.name === "operatorconfigurations.acid.zalan.do")) {
              obj.apiVersion = "v1"
                obj.kind = "List"
            }
          },
        ],
jaxxstorm picture jaxxstorm on 31 Jul 2020
👍1

The workaround at the moment is to use a transformation to drop the resource

This is documented for each SDK here:

https://www.pulumi.com/docs/reference/pkg/kubernetes/helm/v3/chart/#chart-with-transformations

lblackstone picture lblackstone on 31 Jul 2020

So adding this transformation to my Helm chart did not work around the problem:

static ImmutableDictionary<string, object> OmitCustomResource(ImmutableDictionary<string, object> obj, CustomResourceOptions opts)
            {
                var metadata = (ImmutableDictionary<string, object>)obj["metadata"];
                var resourceName = $"{obj["kind"]}.{metadata["name"]}";

                if (string.Equals(obj["kind"], "CustomResourceDefinition"))
                {
                    return new Dictionary<string, object>
                    {
                        ["apiVersion"] = "v1",
                        ["kind"] = "List",
                        ["items"] = new Dictionary<string, object>(),
                    }.ToImmutableDictionary();
                }
                return obj;
            }

Any other ideas for how I can work around this?

Aaronontheweb picture Aaronontheweb on 11 Aug 2020

It looks like line 132-135 of ChartBase.cs is forcing the inclusion of crds with the --include-crds flag. https://github.com/pulumi/pulumi-kubernetes/blob/9069b3a1c780df9e6ac0d7ef6b6fe46a23acef72/sdk/dotnet/Helm/ChartBase.cs#L134

Shouldn't this become an option so the user can decide to either use the --include-crds or --skip-crds flag?

ekeij picture ekeij on 12 Aug 2020

Shouldn't this become an option so the user can decide to either use the --include-crds or --skip-crds flag?

Yes - I believe @jaxxstorm is working on a plan for how best to expose this option.

lukehoban picture lukehoban on 12 Aug 2020
👍1

The short term fix is to provide the option in the helm options to not install CRDs, similar to the helm flag. It was my mistake to assume this would always be needed.

In the long term, I think @lblackstone is considering some ideas to make this more seamless to users. Considering this is becoming a pressing issue, I'm going to author the short term fix for this and get this out ASAP

jaxxstorm picture jaxxstorm on 12 Aug 2020

The problem here isn't with the Kubernetes CustomResourceDefinition, but with the URN generation of Pulumi.

Specifically in @Aaronontheweb case, there is an AlertManager generated by the helm chart called monitoring/pm-prometheus-operator-alertmanager. But there's also a ServiceMonitor called monitoring/pm-prometheus-operator-alertmanager.

Because these are not standard Kubernetes resource kinds, Pulumi generates a URN for a CustomResourceDefinition type (is this the fallback type for unknown resource kinds?). The result is 2 resources with URN

urn:pulumi:dev::sdkbin-resources::kubernetes:helm.sh/v2:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/pm-prometheus-operator-alertmanager

Kubernetes has no issue with resources named the same, as long as they are not of the same Kind. From the helm chart perspective there is no issue. I would consider it a bug/missing feature in Pulumi. It would be nice if Pulumi just takes the Kubernetes resource Kind in the URN, also for unknown kinds.

I worked around it by applying this transform

Transformations = {
    (obj, opt) =>
    {
         // Transform the name with suffix to prevent URN collisions
         obj = AddNameSuffixForKind(
         kind: "ServiceMonitor",
         suffix: "service-monitor",
         obj,
         opt);

         // Transform the name with suffix to prevent URN collisions
         obj = AddNameSuffixForKind(
         kind: "AlertManager",
         suffix: "alert-manager",
         obj,
         opt);

        return obj;
    }
...
static ImmutableDictionary<string, object> AddNameSuffixForKind(
            string kind,
            string suffix,
            ImmutableDictionary<string, object> obj,
            CustomResourceOptions opts)
        {
            if ((string)obj["kind"] == kind)
            {
                var metaData = (ImmutableDictionary<string, object>)obj["metadata"];
                var name = metaData["name"];
                metaData = metaData.SetItem("name", $"{name}-{suffix}");
                obj = obj.SetItem("metadata", metaData);

                return obj;
            }

            return obj;
        }
Freakazoid182 picture Freakazoid182 on 13 Aug 2020
👍4 ❤1

I've got this problem with cert-manager's CRDs. I'm deploying multiple clusters in my pulumi stack and cert-manager to each cluster.

Pulumi throws a duplicate URN error because the helm chart uses a hardcoded name for it's CRD "certificaterequests.cert-manager.io".

Unfortunately I can't work around this issue by using "resourcePrefix" because then my resource names start to exceed 53 characters (also it makes the naming very silly with a double {name}-{name} prefix).

I also can't work around this problem by using a resource transformation to rename the CRD because the CRD name must be spec.names.plural+"."+spec.group as enforced by the k8s api.

I also can't work around this problem by using k8s.yaml.ConfigFile or k8s.yaml.parse because they also map resource names to URNs.

I think pulumi needs to do something to better handle URN generation for helm charts.

Perhaps an urnPrefix would work?

Place1 picture Place1 on 1 Sep 2020
👍4

I'm currently working on https://github.com/pulumi/pulumi-kubernetes/issues/920, and then I'll see if I can rework the URNs to fix this problem (with aliases so that they remain compatible).

lblackstone picture lblackstone on 2 Sep 2020

@lblackstone and I are having a discussion on how best to support this. It seems this was introduced by #1102 and we do need to figure out a way to either
A) let the user opt-out of installing the CRDs, because they might be installed out of band by another process
B) detect if the crd is in the api and omit them if automatically.

The workaround at the moment is to use a transformation to drop the resource, something a bit like this:

transformations: [
          (obj: any, opts: pulumi.CustomResourceOptions) => {
            if (obj.kind === "CustomResourceDefinition" && (obj.metadata.name === "postgresqls.acid.zalan.do" || obj.metadata.name === "operatorconfigurations.acid.zalan.do")) {
              obj.apiVersion = "v1"
                obj.kind = "List"
            }
          },
        ],

Would be nice to at least have the flag --skip-crds exposed on the _v3.Chart_ object, for now this workaround works for us. 👍

vyrwu picture vyrwu on 14 Dec 2020
👍1

If I understand correctly - there are at least two issues being tracked here:

  1. URNs for customer resources can frequently conflict as noted in https://github.com/pulumi/pulumi-kubernetes/issues/1225#issuecomment-684237439 and https://github.com/pulumi/pulumi-kubernetes/issues/1225#issuecomment-673455979.
  2. Issues around whether CRDs are skipped or not in Helm charts - and specifically the desire to offer users control over it - which is tracked in https://github.com/pulumi/pulumi-kubernetes/issues/1195.

I'll assume this issue is tracking (1) - improving URNs and/or offering more configurability for URNs via urnPrefix or similar. This has some relationship to https://github.com/pulumi/pulumi-kubernetes/issues/1472 which also proposes changes to URNs - though is a separate issue.

lukehoban picture lukehoban on 2 May 2021

@lukehoban Correct

lblackstone picture lblackstone on 3 May 2021

Has this been fixed? I see the correct resource type in URNs, even if the resource is custom. In any case, I have no problem installing prometheus from https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack

almson picture almson on 26 May 2021

1195 is done, so it's possible to skip the CRD creation now. The other URN issue (#1225) is still open.

lblackstone picture lblackstone on 26 May 2021

@lblackstone The above-referenced comment says

Because these are not standard Kubernetes resource kinds, Pulumi generates a URN for a CustomResourceDefinition type (is this the fallback type for unknown resource kinds?). The result is 2 resources with URN
pulumi:dev::sdkbin-resources::kubernetes:helm.sh/v2:Chart$kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition::monitoring/pm-prometheus-operator-alertmanager

I see that this is not necessarily true. The Helm chart that I use creates resources with urns

urn:pulumi:dev::swkube::sw:KubernetesMonitoring$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v2:Chart$kubernetes:monitoring.coreos.com/v1:ServiceMonitor::monitoring/prometheus-operator-kube-state-metrics
urn:pulumi:dev::swkube::sw:KubernetesMonitoring$kubernetes:core/v1:Namespace$kubernetes:helm.sh/v3:Chart$kubernetes:monitoring.coreos.com/v1:PrometheusRule::monitoring/prometheus-operator-kube-state-metrics

As you can see, the URNs use the correct custom resource types (they don't have CustomResourceDefinition in their URNs) and don't conflict. So, the problem seems more nuanced than just "urns don't include custom resources."

almson picture almson on 26 May 2021

@Place1 You can prefix the URN by declaring CustomResource classes of different types. See https://github.com/pulumi/pulumi-kubernetes/issues/1472#issuecomment-849017562 This should fix your issue of declaring multiple clusters.

almson picture almson on 26 May 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pierlucg-xs picture pierlucg-xs  ·  5Comments
phil-lgr picture phil-lgr  ·  8Comments
mazamats picture mazamats  ·  7Comments
clstokes picture clstokes  ·  4Comments
oliverholliday picture oliverholliday  ·  9Comments