Terraform: Terraform does not wait for local provisioner in null resource

[r-moiseev]

I use kubespray to create kubernetes cluster (using null-resource) that generates config for kubectl. Then I use kubernetes provider with specified config.

Problem is I have error connecting to cluster when I run apply for first time. On second run it is all good. I have to ideas:

1. Kubernetes resources does not wait for generate_kubeconfig resource. It is strange because all my k8s resources depends on it.
2. provider rads config file before it is ready. In this case question is how to read config only after it is finally ready?

Terraform Version

Terraform v0.11.7
+ provider.azurerm v1.3.2
+ provider.helm (unversioned)
+ provider.kubernetes v1.1.0
+ provider.null v1.0.0
+ provider.template v1.0.0

Terraform Configuration Files

resource "null_resource" "kubespray" {
  provisioner "local-exec" {
    command = "ansible-playbook -i ./inventory/hosts.ini ./kubespray/cluster.yml --become"
  }

  triggers {
    template = "${null_resource.inventories.id}-${null_resource.ssh_keyscan.id}"
  }
}

resource "null_resource" "generate_kubeconfig" {
  provisioner "local-exec" {
    command = "sed -i 's!server: https://.*:6443!server: https://${azurerm_public_ip.master_ip.*.ip_address[0]}:6443!' ${path.module}/inventory//artifacts/admin.conf && sleep 10"
  }

  depends_on = ["null_resource.kubespray"]
}
...
provider "kubernetes" {
  config_path = "${path.module}/inventory/artifacts/admin.conf"
}

Expected Behavior

Provider reads kubernetes config when it is ready

Actual Behavior

See on top of post

[adamlis]

Facing same issue. I need to run local-exec as one of last resources in a module. Then expected behavior is that module call would not finish before local-exec finishes.

I've tried to use 'id' attribute from null_resource and put it into module outputs. I expected 'id' parameter of null_resource would not appear until local-exec is finished - but unfortunately attribute appears immediately.

In my case local-exec is ansible call. And module is responsible for infrastructure creation in real provider and then executing ansible to provision VMs. Then next module creates next infrastructure pieces and calls their ansible for provisioning. Dependency between modules can be achieved by passing to 2nd module output from 1st module, which is generated in 1st module in it's last resource. This approach does not work when local-exec is in use.

[adamlis]

BTW: Funny thing is, that there is an opposite request in the history: #4141 In that issue discussion is about Azure then, but seems no one says that mentioned behavior (waiting for local-exec to finish) is wrong. But there is Windows cmd used.

So maybe it's just a matter of calling interpreter some special way?

[adamlis]

I've made attempt with https://www.terraform.io/docs/providers/external/data_source.html - which executes external command and offers it's output as data. So - I've written fancy shell script, that loops waiting for my real file to be executed (actually real parameters) to be present. And I expected data source would become ready when command gets executed.

The problem is, on 'plan' step it also tries to execute my fancy script. So it loops to infinity. I tried to use some 'terraform.env' build-in variable to not loop when terraform command would be 'plan', but no success yet.

But maybe it would be some solution for you. One problem is, that dependency needs to be build in into fancy script - so e.g. my fancy script is waiting for a disk file to start execution. But it's just about placing 'local-file' resource.

[apparentlymart]

Hi @r-moiseev and @adamlis! Sorry this isn't working as expected.

Providers in Terraform are special in that they need to be configured even for the _plan_ phase, and therefore they can't depend on anything that hasn't been created yet at that phase. This situation and an idea for how to address it in a future major release are discussed in #4149.

As I noted there, the two common workarounds today are:

• On the first run of Terraform against an empty state, run with -target=null_resource.generate_kubeconfig to force Terraform to work only on that resource at first. After that resource has created, you can then run Terraform normally where it'll be able to find that file.
• Split the configuration into multiple separate configurations that each work on one "layer" of the problem. In @r-moiseev's case, that might be one configuration that deals with the AzureRM provider and writes the Kubernetes config somewhere and then a second configuration that deals only with Kubernetes, reading that configuration. You'd then apply the AzureRM-related configuration first and the Kubernetes configuration second to bootstrap.

Something like discussed in #4149 is our long-term plan to address this, but at this time our focus (that is, the focus of the Terraform team at HashiCorp) is on configuration language changes for the next major release, so unfortunately we are not making progress against #4149 at this time.

Since this ultimately covers the same change as #4149, I'm going to close this just to consolidate the discussion over there.

[ahmedig]

I don't think this should be closed tbh. This is a pain that lots of people are experiencing.
When we add a null resource that runs soms script, and other resources are depending on this null resource, the expected behaviour should be that Terraform waits for the null resource to complete.

Any plans to fix that? or this is not in your backlog at all?

[ahmedig]

After endless trials, I think this solution works!

resource "firstResource" "bla"
{

}

resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "./Do-SomethingThatTakesALongTime.ps1"
    interpreter = ["PowerShell"]
 }
  triggers = {
    "before" = "${firstResource.bla.id}"
 }
}

resource "SecondResource" "bla2" {

  depends_on = ["null_resource.delay"]
}

[brunoscota]

After endless trials, I think this solution works!

resource "firstResource" "bla"
{

}

resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "./Do-SomethingThatTakesALongTime.ps1"
    interpreter = ["PowerShell"]
 }
  triggers = {
    "before" = "${firstResource.bla.id}"
 }
}

resource "SecondResource" "bla2" {

  depends_on = ["null_resource.delay"]
}

Thank you, I had the similar issue and solved with this workaround

[rcarcasses]

After endless trials, I think this solution works!

resource "firstResource" "bla"
{

}

resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "./Do-SomethingThatTakesALongTime.ps1"
    interpreter = ["PowerShell"]
 }
  triggers = {
    "before" = "${firstResource.bla.id}"
 }
}

resource "SecondResource" "bla2" {

  depends_on = ["null_resource.delay"]
}

Unfortunately didn't work for me:

...
Provider "firstResource" not available for installation.
...

am I missing something?

[ptphuc]

After endless trials, I think this solution works!

resource "firstResource" "bla"
{

}

resource "null_resource" "delay" {
  provisioner "local-exec" {
    command = "./Do-SomethingThatTakesALongTime.ps1"
    interpreter = ["PowerShell"]
 }
  triggers = {
    "before" = "${firstResource.bla.id}"
 }
}

resource "SecondResource" "bla2" {

  depends_on = ["null_resource.delay"]
}

Unfortunately didn't work for me:

...
Provider "firstResource" not available for installation.
...

am I missing something?

The template means for actual resource type like "aws_instance", "null_resource" supported by Terraform put in order in .tf file
There is no resource named "firstResource" I believe

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.