Terraform-provider-kubernetes: unexpected state 'Succeeded', wanted target 'Running'. last error: %!s(<nil>)

[yaman]

I have a pod and running it through terraform kubernetes gives the following error;

$ terraform apply -var-file="$TFVARS_FILE" -auto-approve $TERRAFORM_FOLDER
data.google_container_cluster.skydome_cluster: Refreshing state...
module.flink.kubernetes_config_map.signal_flink_hadoop_config: Refreshing state... [id=qa/signal-flink-hadoop-config]
module.flink.kubernetes_config_map.signal_flink_flink_config: Refreshing state... [id=qa/signal-flink-flink-config]
module.flink.kubernetes_service.signal_flink_jobmanager: Refreshing state... [id=qa/signal-flink-jobmanager]
module.flink.kubernetes_deployment.signal_flink_taskmanager: Refreshing state... [id=qa/signal-flink-taskmanager]
module.flink.kubernetes_deployment.signal_flink_jobmanager: Refreshing state... [id=qa/signal-flink-jobmanager]
module.cassandra.data.helm_repository.incubator: Refreshing state...
module.kafka.data.helm_repository.incubator: Refreshing state...
module.cassandra.helm_release.signal_cassandra: Refreshing state... [id=signal-cassandra]
module.kafka.helm_release.signal_kafka: Refreshing state... [id=signal-kafka]
module.cassandra.helm_release.signal_cassandra: Creating...
module.cassandra.helm_release.signal_cassandra: Still creating... [10s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [20s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [30s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [40s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [50s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [1m0s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [1m10s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [1m20s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [1m30s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [1m40s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [1m50s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [2m0s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [2m10s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [2m20s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [2m30s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [2m40s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [2m50s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [3m0s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [3m10s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [3m20s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [3m30s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [3m40s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [3m50s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [4m0s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [4m10s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [4m20s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [4m30s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [4m40s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [4m50s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [5m0s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [5m10s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [5m20s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [5m30s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [5m40s elapsed]
module.cassandra.helm_release.signal_cassandra: Still creating... [5m50s elapsed]
module.cassandra.helm_release.signal_cassandra: Creation complete after 5m53s [id=signal-cassandra]
module.cassandra.kubernetes_pod.cassandra_migrate: Creating...
module.cassandra.kubernetes_pod.cassandra_migrate: Still creating... [10s elapsed]

Error: unexpected state 'Succeeded', wanted target 'Running'. last error: %!s(<nil>)

  on terraform/cassandra/migration.tf line 1, in resource "kubernetes_pod" "cassandra_migrate":
   1: resource "kubernetes_pod" "cassandra_migrate" {

Terraform Version

• Terraform v0.12.6
• terraform-providers/helm 0.10.2
• terraform-providers/google 2.13.0
• terraform-providers/kubernetes 1.9.0

Affected Resource(s)

• terraform kubernetes provider, kubernetes_pod

Terraform Configuration Files

resource "kubernetes_pod" "cassandra_migrate" {
  metadata {
    name      = "cassandra-migrate"
    namespace = "${var.namespace}"
  }

  spec {
    container {
      name    = "cassandra-migrate"
      image   = "${var.image_registry_url}/${var.migration_image_name}:${var.migration_image_version}"
      command = ["cassandra-migrate"]
      args    = ["-H", "${var.cassandra_name}", "-c", "${var.migration_yaml_file}", "migrate"]
    }

    restart_policy = "Never"

    image_pull_secrets {
      name = "${var.image_pull_secret}"
    }
  }

  depends_on = ["helm_release.signal_cassandra"]
}

Expected Behavior

Pod creation was not an issue before, but I can not say anything about what might have caused this issue. It should run successfully and return success.

Actual Behavior

Error: unexpected state 'Succeeded', wanted target 'Running'. last error: %!s(<nil>)

Steps to Reproduce

1. terraform apply

Important Factoids

Running at GKE 1.13.7-gke.8

[maheshmadpathi]

I'm also facing the same issue while creating AWS EKS cluster, any hints?

[pdecat]

I believe short lived pods should rather be run using kubernetes_job instead of kubernetes_pod.

[StyleT]

Same issue here

[garbetjie]

I'm encountering this too. I've created a pull request to (hopefully) resolve this: https://github.com/hashicorp/terraform-provider-kubernetes/pull/1031