Terraform-provider-kubernetes: Ability to force create resources

[dpiddockcmp]

Hi there,

I'm working with an open source module that creates an AWS EKS cluster and associated resources. On master we're experimenting with using this provider instead of a local-exec to kubectl.

I'm specifically working with the new EKS Node Groups resources. This creates the kubernetes node instances so users don't have to worry about them.

As part of EKS's authentication system it uses a ConfigMap, aws-auth, to configure how to convert from AWS-land users (IAM), into k8s RBAC. Using this provider to manage the aws-auth configmap mostly works. However when adding a node group, AWS creates or modifies the aws-auth configmap on the user's behalf. When creating the cluster from scratch this breaks on the kubernetes provider which requires resources to not exist.

Terraform Version

Terraform v0.12.18

• provider.kubernetes v1.10.0

Affected Resource(s)

Please list the resources as a list, for example:

• kubernetes_config_map
• probably every resource that creates something

Terraform Configuration Files

resource "kubernetes_config_map" "example" {
  metadata {
    name = "my-config"
  }

  data = {
    example = "example"
  }
}

Debug Output

Panic Output

Expected Behavior

I would love for the provider to have an option to ignore the current resource's state and apply the configuration specified in Terraform config.

Actual Behavior

kubernetes_config_map.example: Creating...

Error: configmaps "my-config" already exists

  on main.tf line 1, in resource "kubernetes_config_map" "example":
   1: resource "kubernetes_config_map" "example" {

Steps to Reproduce

1. kubectl create cm my-config --from-literal=key=value
2. terraform apply

Important Factoids

References

[peimanja]

Same issue here.

[hazcod]

Possibly related to #775

[jwhiting]

@hazcod I don't believe this is related to #775 - the key difference is that in this case, the k8s config map resource is created by AWS, outside of terraform's purview, so when we want to set a new value for the config map, it fails because terraform is not expecting it to exist.

This issue is about being able to say to terraform in effect "yes, I know you don't expect this resource to exist, I know it was created outside of any prior terraform apply, please ignore that and replace it with the desired state anyway."

This pertains to being able to setup an EKS cluster purely with terraform, without resorting to kubectl for the final step of installing the aws-auth configmap (see https://learn.hashicorp.com/terraform/aws/eks-intro for that final non-terraform step)

I a big fan of the notion of not using kubectl for any writes or changes to a k8s cluster (just use it to debug/inspect) and rather using terraform for all change management, and I think on EKS this issue impedes that goal.

(Perhaps the bigger issue here is, how can terraform reliably control state on managed cluster like EKS, in which AWS is making its own changes?)

@dpiddockcmp do you have a workaround currently in your module that you could share?

[dpiddockcmp]

@dpiddockcmp do you have a workaround currently in your module that you could share?

Yes, I came up with a solution. Key requirement was to remove the race condition between Terraform kubernetes provider and AWS's managed node groups system.

I created a null_data_source that passes through the cluster_name and has implicit dependency on other vital resources. The managed_node_group then uses the cluster_name output from the null_data_source. Ordering is safely enforced during cluster creation and deletion 🙂

You can see my solution here

However that does not mean this issue is no longer required. It would be useful to have terraform control over default resources installed by AWS in the cluster, like the aws-node DaemonSet and coredns Deployment. AWS does not manage their upgrade cycle. Currently we do this with kubectl and kustomize yamls.

[aareet]

Relates to #801

[kadzu]

Don't make sense... creating a configuration on IaC and can't be update!..
Instead, I used local-exec and sh script with ENV vars.. (workaround)

[alexsomesan]

I'm wondering if this can be accomplished by just using the import command?

Using the example provided in the issue description, here's how it did that:

~/test-719 » kubectl create cm my-config --from-literal=key=value                                                                                                                             alex@alex-macbook
configmap/my-config created
------------------------------------------------------------
~/test-719 » terraform import kubernetes_config_map.example default/my-config                                                                                                                 alex@alex-macbook
kubernetes_config_map.example: Importing from ID "default/my-config"...
kubernetes_config_map.example: Import prepared!
  Prepared kubernetes_config_map for import
kubernetes_config_map.example: Refreshing state... [id=default/my-config]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

------------------------------------------------------------
~/test-719 » terraform plan                                                                                                                                                                   alex@alex-macbook
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

kubernetes_config_map.example: Refreshing state... [id=default/my-config]

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # kubernetes_config_map.example will be updated in-place
  ~ resource "kubernetes_config_map" "example" {
        binary_data = {}
      ~ data        = {
          + "example" = "example"
          - "key"     = "value" -> null
        }
        id          = "default/my-config"

        metadata {
            annotations      = {}
            generation       = 0
            labels           = {}
            name             = "my-config"
            namespace        = "default"
            resource_version = "18062"
            self_link        = "/api/v1/namespaces/default/configmaps/my-config"
            uid              = "c930b9e6-ab3c-42c8-8f9e-29bc9010e491"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

------------------------------------------------------------
~/test-719 » terraform apply -auto-approve                                                                                                                                                    alex@alex-macbook
kubernetes_config_map.example: Refreshing state... [id=default/my-config]
kubernetes_config_map.example: Modifying... [id=default/my-config]
kubernetes_config_map.example: Modifications complete after 0s [id=default/my-config]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
------------------------------------------------------------
~/test-719 » kubectl get cm my-config -oyaml                                                                                                                                                  alex@alex-macbook
apiVersion: v1
data:
  example: example
kind: ConfigMap
metadata:
  creationTimestamp: "2020-04-08T14:52:11Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data: {}
    manager: kubectl
    operation: Update
    time: "2020-04-08T14:52:11Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        f:example: {}
    manager: HashiCorp
    operation: Update
    time: "2020-04-08T14:52:42Z"
  name: my-config
  namespace: default
  resourceVersion: "18129"
  selfLink: /api/v1/namespaces/default/configmaps/my-config
  uid: c930b9e6-ab3c-42c8-8f9e-29bc9010e491
------------------------------------------------------------

[mreyes-netquest]

Hi @dpiddockcmp,

I'm confusing with the scenarios that you are describing. Because as far as I understood the output that you have included in this issue is related to "adding node group* (to existing stack)" but there is a second escenario, which is to "create a terraform eks stack with kubernetes provider and node groups from scratch". I think that I am in the second one and my output reports this error:

module.eks.kubernetes_config_map.aws_auth[0]: Creating...
Error: Unauthorized
  on .terraform/modules/eks/terraform-aws-eks-11.1.0/aws_auth.tf line 62, in resource "kubernetes_config_map" "aws_auth":
  62: resource "kubernetes_config_map" "aws_auth" {

This error above is happening to me when _terrafom apply_ from scratch, but if I apply again, the error dissapears.

module.eks.kubernetes_config_map.aws_auth[0]: Creating...
module.eks.kubernetes_config_map.aws_auth[0]: Creation complete after 1s [id=kube-system/aws-auth]

Looks like a race condition but I'm not sure about that. Am I going in the right way?

Thank you,

[alexsomesan]

@mreyes-netquest What you are seeing is indeed a race condition, if you are indeed creating the EKS resource in that same apply. This is a use-case we don't support and we're constantly advising against due to limitations in Terraform itself.
It would not be solved by "force creating" as it is described in the original issue, so your report is not related to this particular issue. Try separating the cluster creation in a different apply and see if that solves your problem. If not, please open a new GH issue.