Terraform: remote-exec provisioner: apt-get install failed occasionally

Created on 15 Nov 2017  ·  9Comments  ·  Source: hashicorp/terraform

Hi there, I found an issue when I tried to create EC2 instance with remote-exec provisioner.

Terraform Version

  • Terraform v0.10.8
  • terraform-provider-aws_v1.2.0_x4

Terraform Configuration Files

resource "aws_instance" "foo" {
  provisioner "remote-exec" {
...
    scripts = ["bar.sh"]
  }

bar.sh

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y make

Debug Output

module.aws_instance.foo (remote-exec): + sudo apt-get update -qq
module.aws_instance.foo: Still creating... (30s elapsed)
module.aws_instance.foo: Still creating... (40s elapsed)
module.aws_instance.foo (remote-exec): + sudo apt-get upgrade -y
module.aws_instance.foo (remote-exec): Reading package lists... 0%
module.aws_instance.foo (remote-exec): Reading package lists... 0%
module.aws_instance.foo (remote-exec): Reading package lists... 1%
module.aws_instance.foo (remote-exec): Reading package lists... 36%
module.aws_instance.foo (remote-exec): Reading package lists... 36%
module.aws_instance.foo (remote-exec): Reading package lists... 45%
module.aws_instance.foo (remote-exec): Reading package lists... 45%
module.aws_instance.foo (remote-exec): Reading package lists... 92%
module.aws_instance.foo (remote-exec): Reading package lists... 92%
module.aws_instance.foo (remote-exec): Reading package lists... 96%
module.aws_instance.foo (remote-exec): Reading package lists... 96%
module.aws_instance.foo (remote-exec): Reading package lists... Done
module.aws_instance.foo (remote-exec): Building dependency tree... 0%
module.aws_instance.foo (remote-exec): Building dependency tree... 0%
module.aws_instance.foo (remote-exec): Building dependency tree... 50%
module.aws_instance.foo (remote-exec): Building dependency tree... 50%
module.aws_instance.foo (remote-exec): Building dependency tree
module.aws_instance.foo (remote-exec): Reading state information... 0%
module.aws_instance.foo (remote-exec): Reading state information... 8%
module.aws_instance.foo (remote-exec): Reading state information... Done
module.aws_instance.foo (remote-exec): Calculating upgrade... Done
module.aws_instance.foo (remote-exec): The following packages will be upgraded:
module.aws_instance.foo (remote-exec):   libssl1.0.0 openssl perl perl-base perl-modules tzdata
module.aws_instance.foo (remote-exec): 6 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
module.aws_instance.foo (remote-exec): Need to get 7,991 kB of archives.
module.aws_instance.foo (remote-exec): After this operation, 2,048 B of additional disk space will be used.
module.aws_instance.foo (remote-exec): 0% [Working]
module.aws_instance.foo (remote-exec): 0% [Waiting for headers]
module.aws_instance.foo (remote-exec): Get:1 http://security.ubuntu.com/ubuntu/ trusty-security/main perl amd64 5.18.2-2ubuntu1.3 [2,656 kB]
module.aws_instance.foo (remote-exec): 0% [1 perl 14.2 kB/2,656 kB 1%]
module.aws_instance.foo (remote-exec): 1% [1 perl 43.2 kB/2,656 kB 2%]
module.aws_instance.foo (remote-exec): 3% [1 perl 210 kB/2,656 kB 8%]
module.aws_instance.foo (remote-exec): 10% [1 perl 826 kB/2,656 kB 31%]
module.aws_instance.foo (remote-exec): 20% [1 perl 1,616 kB/2,656 kB 61%]
module.aws_instance.foo (remote-exec): 30% [1 perl 2,407 kB/2,656 kB 91%]
module.aws_instance.foo (remote-exec): 33% [Working]
module.aws_instance.foo (remote-exec): Get:2 http://security.ubuntu.com/ubuntu/ trusty-security/main perl-base amd64 5.18.2-2ubuntu1.3 [1,149 kB]
module.aws_instance.foo (remote-exec): 34% [2 perl-base 27.3 kB/1,149 kB 2%]
module.aws_instance.foo (remote-exec): 43% [2 perl-base 790 kB/1,149 kB 69%]
module.aws_instance.foo (remote-exec): 48% [Working]
module.aws_instance.foo (remote-exec): Get:3 http://security.ubuntu.com/ubuntu/ trusty-security/main perl-modules all 5.18.2-2ubuntu1.3 [2,699 kB]
module.aws_instance.foo (remote-exec): 48% [3 perl-modules 22.9 kB/2,699 kB 1%]
module.aws_instance.foo: Still creating... (50s elapsed)
module.aws_instance.foo (remote-exec): 58% [3 perl-modules 790 kB/2,699 kB 29%]
module.aws_instance.foo (remote-exec): 67% [3 perl-modules 1,581 kB/2,699 kB 59%]
module.aws_instance.foo (remote-exec): 77% [3 perl-modules 2,373 kB/2,699 kB 88%]
module.aws_instance.foo (remote-exec): 81% [Working]
module.aws_instance.foo (remote-exec): Get:4 http://security.ubuntu.com/ubuntu/ trusty-security/main libssl1.0.0 amd64 1.0.1f-1ubuntu2.23 [830 kB]
module.aws_instance.foo (remote-exec): 82% [4 libssl1.0.0 17.1 kB/830 kB 2%]                             1,061 kB/s 1s
module.aws_instance.foo (remote-exec): 91% [4 libssl1.0.0 790 kB/830 kB 95%]                             1,061 kB/s 0s
module.aws_instance.foo (remote-exec): 92% [Working]                                                     1,061 kB/s 0s
module.aws_instance.foo (remote-exec): Get:5 http://security.ubuntu.com/ubuntu/ trusty-security/main tzdata all 2017c-0ubuntu0.14.04 [166 kB]
module.aws_instance.foo (remote-exec): 92% [5 tzdata 3,981 B/166 kB 2%]                                  1,061 kB/s 0s
module.aws_instance.foo (remote-exec): 94% [Working]                                                     1,061 kB/s 0s
module.aws_instance.foo (remote-exec): Get:6 http://security.ubuntu.com/ubuntu/ trusty-security/main openssl amd64 1.0.1f-1ubuntu2.23 [490 kB]
module.aws_instance.foo (remote-exec): 94% [6 openssl 28.7 kB/490 kB 6%]                                 1,061 kB/s 0s
module.aws_instance.foo (remote-exec): 100% [Working]                                                    1,061 kB/s 0s
module.aws_instance.foo (remote-exec): Fetched 7,991 kB in 7s (1,067 kB/s)
module.aws_instance.foo (remote-exec): Preconfiguring packages ...
module.aws_instance.foo (remote-exec): (Reading database ...
module.aws_instance.foo (remote-exec): (Reading database ... 5%
module.aws_instance.foo (remote-exec): (Reading database ... 10%
module.aws_instance.foo (remote-exec): (Reading database ... 15%
module.aws_instance.foo (remote-exec): (Reading database ... 20%
module.aws_instance.foo (remote-exec): (Reading database ... 25%
module.aws_instance.foo (remote-exec): (Reading database ... 30%
module.aws_instance.foo (remote-exec): (Reading database ... 35%
module.aws_instance.foo (remote-exec): (Reading database ... 40%
module.aws_instance.foo (remote-exec): (Reading database ... 45%
module.aws_instance.foo (remote-exec): (Reading database ... 50%
module.aws_instance.foo (remote-exec): (Reading database ... 55%
module.aws_instance.foo (remote-exec): (Reading database ... 60%
module.aws_instance.foo (remote-exec): (Reading database ... 65%
module.aws_instance.foo (remote-exec): (Reading database ... 70%
module.aws_instance.foo (remote-exec): (Reading database ... 75%
module.aws_instance.foo (remote-exec): (Reading database ... 80%
module.aws_instance.foo (remote-exec): (Reading database ... 85%
module.aws_instance.foo (remote-exec): (Reading database ... 90%
module.aws_instance.foo (remote-exec): (Reading database ... 95%
module.aws_instance.foo (remote-exec): (Reading database ... 100%
module.aws_instance.foo (remote-exec): (Reading database ... 51283 files and directories currently installed.)
module.aws_instance.foo (remote-exec): Preparing to unpack .../perl_5.18.2-2ubuntu1.3_amd64.deb ...
module.aws_instance.foo (remote-exec): Unpacking perl (5.18.2-2ubuntu1.3) over (5.18.2-2ubuntu1.1) ...
module.aws_instance.foo (remote-exec): Preparing to unpack .../perl-base_5.18.2-2ubuntu1.3_amd64.deb ...
module.aws_instance.foo (remote-exec): Unpacking perl-base (5.18.2-2ubuntu1.3) over (5.18.2-2ubuntu1.1) ...
module.aws_instance.foo (remote-exec): Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
module.aws_instance.foo (remote-exec): Setting up perl-base (5.18.2-2ubuntu1.3) ...
module.aws_instance.foo (remote-exec): (Reading database ...
module.aws_instance.foo (remote-exec): (Reading database ... 5%
module.aws_instance.foo (remote-exec): (Reading database ... 10%
module.aws_instance.foo (remote-exec): (Reading database ... 15%
module.aws_instance.foo (remote-exec): (Reading database ... 20%
module.aws_instance.foo (remote-exec): (Reading database ... 25%
module.aws_instance.foo (remote-exec): (Reading database ... 30%
module.aws_instance.foo (remote-exec): (Reading database ... 35%
module.aws_instance.foo (remote-exec): (Reading database ... 40%
module.aws_instance.foo (remote-exec): (Reading database ... 45%
module.aws_instance.foo (remote-exec): (Reading database ... 50%
module.aws_instance.foo (remote-exec): (Reading database ... 55%
module.aws_instance.foo (remote-exec): (Reading database ... 60%
module.aws_instance.foo (remote-exec): (Reading database ... 65%
module.aws_instance.foo (remote-exec): (Reading database ... 70%
module.aws_instance.foo (remote-exec): (Reading database ... 75%
module.aws_instance.foo (remote-exec): (Reading database ... 80%
module.aws_instance.foo (remote-exec): (Reading database ... 85%
module.aws_instance.foo (remote-exec): (Reading database ... 90%
module.aws_instance.foo (remote-exec): (Reading database ... 95%
module.aws_instance.foo (remote-exec): (Reading database ... 100%
module.aws_instance.foo (remote-exec): (Reading database ... 51283 files and directories currently installed.)
module.aws_instance.foo (remote-exec): Preparing to unpack .../perl-modules_5.18.2-2ubuntu1.3_all.deb ...
module.aws_instance.foo (remote-exec): Unpacking perl-modules (5.18.2-2ubuntu1.3) over (5.18.2-2ubuntu1.1) ...
module.aws_instance.foo (remote-exec): Preparing to unpack .../libssl1.0.0_1.0.1f-1ubuntu2.23_amd64.deb ...
module.aws_instance.foo (remote-exec): Unpacking libssl1.0.0:amd64 (1.0.1f-1ubuntu2.23) over (1.0.1f-1ubuntu2.22) ...
module.aws_instance.foo (remote-exec): Preparing to unpack .../tzdata_2017c-0ubuntu0.14.04_all.deb ...
module.aws_instance.foo (remote-exec): Unpacking tzdata (2017c-0ubuntu0.14.04) over (2016j-0ubuntu0.14.04) ...
module.aws_instance.foo (remote-exec): Setting up tzdata (2017c-0ubuntu0.14.04) ...

module.aws_instance.foo (remote-exec): Current default time zone: 'Etc/UTC'
module.aws_instance.foo (remote-exec): Local time is now:      Wed Nov 15 02:35:03 UTC 2017.
module.aws_instance.foo (remote-exec): Universal Time is now:  Wed Nov 15 02:35:03 UTC 2017.
module.aws_instance.foo (remote-exec): Run 'dpkg-reconfigure tzdata' if you wish to change it.

module.aws_instance.foo (remote-exec): (Reading database ...
module.aws_instance.foo (remote-exec): (Reading database ... 5%
module.aws_instance.foo (remote-exec): (Reading database ... 10%
module.aws_instance.foo (remote-exec): (Reading database ... 15%
module.aws_instance.foo (remote-exec): (Reading database ... 20%
module.aws_instance.foo (remote-exec): (Reading database ... 25%
module.aws_instance.foo (remote-exec): (Reading database ... 30%
module.aws_instance.foo (remote-exec): (Reading database ... 35%
module.aws_instance.foo (remote-exec): (Reading database ... 40%
module.aws_instance.foo (remote-exec): (Reading database ... 45%
module.aws_instance.foo (remote-exec): (Reading database ... 50%
module.aws_instance.foo (remote-exec): (Reading database ... 55%
module.aws_instance.foo (remote-exec): (Reading database ... 60%
module.aws_instance.foo (remote-exec): (Reading database ... 65%
module.aws_instance.foo (remote-exec): (Reading database ... 70%
module.aws_instance.foo (remote-exec): (Reading database ... 75%
module.aws_instance.foo (remote-exec): (Reading database ... 80%
module.aws_instance.foo (remote-exec): (Reading database ... 85%
module.aws_instance.foo (remote-exec): (Reading database ... 90%
module.aws_instance.foo (remote-exec): (Reading database ... 95%
module.aws_instance.foo (remote-exec): (Reading database ... 100%
module.aws_instance.foo (remote-exec): (Reading database ... 51283 files and directories currently installed.)
module.aws_instance.foo (remote-exec): Preparing to unpack .../openssl_1.0.1f-1ubuntu2.23_amd64.deb ...
module.aws_instance.foo (remote-exec): Unpacking openssl (1.0.1f-1ubuntu2.23) over (1.0.1f-1ubuntu2.22) ...
module.aws_instance.foo (remote-exec): Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
module.aws_instance.foo (remote-exec): Setting up libssl1.0.0:amd64 (1.0.1f-1ubuntu2.23) ...
module.aws_instance.foo (remote-exec): Setting up openssl (1.0.1f-1ubuntu2.23) ...
module.aws_instance.foo (remote-exec): Setting up perl-modules (5.18.2-2ubuntu1.3) ...
module.aws_instance.foo (remote-exec): Setting up perl (5.18.2-2ubuntu1.3) ...
module.aws_instance.foo (remote-exec): Processing triggers for libc-bin (2.19-0ubuntu6.13) ...
module.aws_instance.foo (remote-exec): + sudo apt-get install -y make
module.aws_instance.foo (remote-exec): Reading package lists... 0%
module.aws_instance.foo (remote-exec): Reading package lists... 100%
module.aws_instance.foo (remote-exec): Reading package lists... Done
module.aws_instance.foo (remote-exec): Building dependency tree... 0%
module.aws_instance.foo (remote-exec): Building dependency tree... 0%
module.aws_instance.foo (remote-exec): Building dependency tree... 50%
module.aws_instance.foo (remote-exec): Building dependency tree... 50%
module.aws_instance.foo (remote-exec): Building dependency tree
module.aws_instance.foo (remote-exec): Reading state information... 0%
module.aws_instance.foo (remote-exec): Reading state information... 8%
module.aws_instance.foo (remote-exec): Reading state information... Done
module.aws_instance.foo (remote-exec): Package make is not available, but is referred to by another package.
module.aws_instance.foo (remote-exec): This may mean that the package is missing, has been obsoleted, or
module.aws_instance.foo (remote-exec): is only available from another source

module.aws_instance.foo (remote-exec): E: Package 'make' has no installation candidate

Error: Error applying plan:

1 error(s) occurred:

* module.aws_instance.foo: 1 error(s) occurred:

* Script exited with non-zero exit status: 100

The provisioning failed, but I can ssh-login and install make by sudo apt-get update && sudo apt-get install -y make after above. 🤔

Crash Output

Expected Behavior

Actual Behavior

Steps to Reproduce

  1. terraform apply

Important Factoids

References

provisioneremote-exec question
Source
picture koooge

Most helpful comment

I have the same on terraform 0.11.11
error executing "/tmp/terraform_68527046.sh": Process exited with status 100

picture jlhinojal-vl on 29 Jan 2019
👍7 😕1

All 9 comments

I have the same issue.

terraform version
Terraform v0.11.3
ibayer picture ibayer on 1 Mar 2018

A workaround is to issue the update command twice.

apt-get update
apt-get update

This worked for me but other workaround are discussed here:
https://github.com/hashicorp/terraform/issues/1025#issuecomment-84139959

ibayer picture ibayer on 1 Mar 2018

Same here

$ terraform version
Terraform v0.11.8
grebois picture grebois on 20 Sep 2018

I found the same issue on packer. and it seems to be caused by cloud-init. packer says following code can solve it

while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done

https://github.com/hashicorp/packer/issues/2639

koooge picture koooge on 5 Dec 2018
👍1

I have the same on terraform 0.11.11
error executing "/tmp/terraform_68527046.sh": Process exited with status 100

jlhinojal-vl picture jlhinojal-vl on 29 Jan 2019
👍7 😕1

I have the same problem. Adding 'Waiting for cloud-init' (suggested by @koooge) as the first item in remote-exec solves the problem on Digitalocean's droplet.

tanin47 picture tanin47 on 7 Jun 2019

I run into the same error. It seams that every line in script or scripts are run asynchronous !?

This gave me an error:

  provisioner "remote-exec" {
    script = "./bootstrap.sh"

    connection {
      agent       = false
      type        = "ssh"    
      user        = "ubuntu"
      private_key = "${file(var.private_key_path)}"
      host = "${aws_instance.main.public_ip}"
    }
  }

But this works:

  provisioner "file" {
    source      = "bootstrap.sh"
    destination = "/tmp/bootstrap.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "chmod +x /tmp/bootstrap.sh",
      "/tmp/bootstrap.sh",
    ]
  }

  connection {                                                                                                                                                                                              
    agent       = false  
    type        = "ssh"  
    user        = "ubuntu"  
    private_key = "${file(var.private_key_path)}"  
    host        = "${aws_instance.main.public_ip}"
  }
delijati picture delijati on 21 Jun 2019

Hello! :robot:

We use GitHub issues for tracking bugs and enhancements rather than for questions. While we can sometimes help with certain simple problems here, it's better to use the community forum where there are more people ready to help. The GitHub issues here are monitored only by our few core maintainers.

Since this issue doesn't represent a specific bug or feature request, I'm going to close it. Please do feel free to ask your question in the community forum. Thanks!

hashibot picture hashibot on 13 Aug 2019

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashibot[bot] picture hashibot[bot] on 13 Sep 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pawelsawicz picture pawelsawicz  ·  3Comments
larstobi picture larstobi  ·  3Comments
darron picture darron  ·  3Comments
sprokopiak picture sprokopiak  ·  3Comments
allomov picture allomov  ·  3Comments