Terraform-provider-kubernetes: 'doc is missing path: “/spec/volumes/0”' error when launching Replica Set with IAM serviceaccount

[awesometown]

I am trying to use the Kubernetes provider to create a serviceaccount and a deployment that uses the service account. My cluster is provisioned using EKS in AWS and I am trying to attach an IAM role to the service account using the eks.amazonaws.com/role-arn annotation. When I run my script, Terraform creates the serviceaccount and then eventually times-out waiting for the deployment to be created. Using kubectl to describe the associated replicaset shows the error Error creating: Internal error occurred: Internal error occurred: jsonpatch add operation does not apply: doc is missing path: "/spec/volumes/0"

I have tried all combinations of creating one or both of the serviceaccount and deployment with terraform or kubectl. It appears that if either or both resources are created with Terraform, I will get the error above. If both resources are created with equivalent yaml and kubectl apply, everything is fine.

If I remove the eks.amazonaws.com/role-arn annotation from the terraform and apply, the deployment also succeeds. The issue seems to specifically be the combination of the eks annotated service account and the Terraform Kubernetes provider.

Terraform Version

Terraform v0.12.13

Affected Resource(s)

• kubernetes_service_account
• kubernetes_deployment

Terraform Configuration Files

provider "kubernetes" {
  config_path = PATH_TO_KUBECONFIG
}

resource "kubernetes_service_account" "this" {
  metadata {
    name = "example"
    namespace = "default"
    annotations = {
      "eks.amazonaws.com/role-arn" = IAM_ROLE_ARN
    }
  }
}

resource "kubernetes_deployment" "example" {
  metadata {
    name = "example"
    labels = {
      test = "MyExampleApp"
    }
  }

  spec {
    replicas = 3

    selector {
      match_labels = {
        test = "MyExampleApp"
      }
    }

    template {
      metadata {
        labels = {
          test = "MyExampleApp"
        }
      }

      spec {
        service_account_name = "example"
        container {
          image = "nginx:1.7.8"
          name  = "example"
        }
      }
    }
  }
}

Debug Output

https://gist.github.com/awesometown/e11d77894f6b767d580fc96d0adea25e

Expected Behavior

ReplicaSet should have created successfully and launched pods.

Actual Behavior

ReplicaSet failed to create. The ReplicaSet shows the error Error creating: Internal error occurred: Internal error occurred: jsonpatch add operation does not apply: doc is missing path: "/spec/volumes/0"

Steps to Reproduce

1. Use EKS to provision a cluster in AWS
2. Create an IAM role that can be attached to a serviceaccount
3. terraform apply using the tf file above (substituting in the ARN of the created role)

Terraform will time out waiting for the deployment to be created. You can run kubectl describe replicaset ... to see the error mentioned above.

[jfelten]

can confirm I get the same issue. I've tried using both the terraform kubernetes_service_account and also running a kubectl apply -f script directly null_resource localexec. Both have the same issue

[stoksc]

It's probably because your pod has no volumes more than likely. The amazon-eks-pod-identity-webhook tries to patch a nonexistent /spec/volumes/0 when it tries to mutate your pod to mount the creds. If this defaulted to true, this wouldn't be an issue though. Answer from this post.

[awesometown]

I confirmed that setting automount_service_account_token = true on the Deployment addressed the problem, so I guess this is not a bug per se so much as a confusing default. My workflow was basically to just do a direct port of an existing manifest to what I thought was equivalent terraform, but I needed to add this setting.

Is changing the default an option (seems like that would result in less surprises), or is that too breaking a change at this point?

[stoksc]

I agree it would be less surprising default in this case, I wonder if the terraform-provider-kubernetes peeps set it that way for a reason.

I think makes sense for amazon-eks-pod-identity-webhook to either patch the the volume on (I honestly don't know if that's possible or the extent of what is possible with patches, but I feel like it's probably an option) or throw an error _other_ than just a failed patch deploying the replicaset.

[stoksc]

there was a pr into amazon-eks-pod-identity-webhook that closed this issue on their side.

[awesometown]

Yes, that looks like it makes this issue moot.

[stevie-]

We still hit this issue today (eks-pod-id.. fix probably didn't help), needed to enable automount_service_account_token.

[jrhouston]

Duplicate of https://github.com/terraform-providers/terraform-provider-kubernetes/issues/263, this issue will be tracked there.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!