Packer: redis example fails to build

Created on 28 Jun 2013 · 23Comments · Source: hashicorp/packer

with error: "Unable to locate package redis-server"

example.json has these contents:

{
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "REDACTED",
    "secret_key": "REDACTED",
    "region": "us-east-1",
    "source_ami": "ami-de0d9eb7",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-example {{.CreateTime}}"
  }],
  "provisioners": [{
    "type": "shell",
    "inline": [
      "sudo apt-get update",
      "sudo apt-get install -y redis-server"
    ]
  }]
}

Running the commands from the getting started guide:

$ packer validate example.json
Template validated successfully.
$ packer build example.json
amazon-ebs output will be in this color.

==> amazon-ebs: Creating temporary keypair for this instance...
==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing SSH access on the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Waiting for instance to become ready...
==> amazon-ebs: Connecting to the instance via SSH...
==> amazon-ebs: Provisioning with shell script: /var/folders/gp/gn1043nx4fn5k3db55x6w9l40000gn/T/packer-shell151021730
Ign http://security.ubuntu.com precise-security InRelease
Ign http://archive.ubuntu.com precise InRelease
Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B]
Ign http://archive.ubuntu.com precise-updates InRelease
Get:2 http://security.ubuntu.com precise-security Release [49.6 kB]
Hit http://archive.ubuntu.com precise Release.gpgse 0 B
Get:3 http://archive.ubuntu.com precise-updates Release.gpg [198 B]
Hit http://archive.ubuntu.com precise Release0%] [2 Rel
Get:4 http://archive.ubuntu.com precise-updates Release [49.6 kB]
Get:5 http://security.ubuntu.com precise-security/main amd64 Packages [295 kB]
Hit http://archive.ubuntu.com precise/main amd64 Packages
Hit http://archive.ubuntu.com precise/restricted amd64 Packages
Hit http://archive.ubuntu.com precise/universe amd64 Packages
Hit http://archive.ubuntu.com precise/multiverse amd64 Packages
Hit http://archive.ubuntu.com precise/main i386 Packages
Hit http://archive.ubuntu.com precise/restricted i386 Packages
Hit http://archive.ubuntu.com precise/universe i386 Packages
    amazon-ebs: Get:6 http://security.ubuntu.com precise-security/restricted amd64 Packages [4627 B]
Hit http://archive.ubuntu.com precise/multiverse i386 Packages
Get:7 http://security.ubuntu.com precise-security/universe amd64 Packages [77.1 kB]
Hit http://archive.ubuntu.com precise/main TranslationIndex
Hit http://archive.ubuntu.com precise/multiverse TranslationIndex
Get:8 http://security.ubuntu.com precise-security/multiverse amd64 Packages [2182 B]
Get:9 http://security.ubuntu.com precise-security/main i386 Packages [309 kB]
Hit http://archive.ubuntu.com precise/restricted TranslationIndex
Hit http://archive.ubuntu.com precise/universe TranslationIndex
Get:10 http://archive.ubuntu.com precise-updates/main amd64 Packages [644 kB]
Get:11 http://security.ubuntu.com precise-security/restricted i386 Packages [4620 B]
Get:12 http://security.ubuntu.com precise-security/universe i386 Packages [79.8 kB]
Get:13 http://security.ubuntu.com precise-security/multiverse i386 Packages [2367 B]
Get:14 http://security.ubuntu.com precise-security/main TranslationIndex [74 B]
Get:15 http://security.ubuntu.com precise-security/multiverse TranslationIndex [71 B]
Get:16 http://security.ubuntu.com precise-security/restricted TranslationIndex [72 B]
Get:17 http://security.ubuntu.com precise-security/universe TranslationIndex [73 B]
Get:18 http://security.ubuntu.com precise-security/main Translation-en [143 kB]
Hit http://security.ubuntu.com precise-security/multiverse Translation-en
Get:19 http://security.ubuntu.com precise-security/restricted Translation-en [1253 B]
Get:20 http://security.ubuntu.com precise-security/universe Translation-en [49.1 kB]
Get:21 http://archive.ubuntu.com precise-updates/restricted amd64 Packages [10.1 kB]
Get:22 http://archive.ubuntu.com precise-updates/universe amd64 Packages [207 kB]
Get:23 http://archive.ubuntu.com precise-updates/multiverse amd64 Packages [13.6 kB]
Get:24 http://archive.ubuntu.com precise-updates/main i386 Packages [662 kB]
Get:25 http://archive.ubuntu.com precise-updates/restricted i386 Packages [10.0 kB]
Get:26 http://archive.ubuntu.com precise-updates/universe i386 Packages [211 kB]
Get:27 http://archive.ubuntu.com precise-updates/multiverse i386 Packages [13.8 kB]
Get:28 http://archive.ubuntu.com precise-updates/main TranslationIndex [3564 B]
Get:29 http://archive.ubuntu.com precise-updates/multiverse TranslationIndex [2605 B]
Get:30 http://archive.ubuntu.com precise-updates/restricted TranslationIndex [2461 B]
Get:31 http://archive.ubuntu.com precise-updates/universe TranslationIndex [2850 B]
Hit http://archive.ubuntu.com precise/main Translation-en
Hit http://archive.ubuntu.com precise/multiverse Translation-en
Hit http://archive.ubuntu.com precise/restricted Translation-en
Hit http://archive.ubuntu.com precise/universe Translation-en
Get:32 http://archive.ubuntu.com precise-updates/main Translation-en [296 kB]
Get:33 http://archive.ubuntu.com precise-updates/multiverse Translation-en [7834 B]
Get:34 http://archive.ubuntu.com precise-updates/restricted Translation-en [2432 B]
Get:35 http://archive.ubuntu.com precise-updates/universe Translation-en [122 kB]
Fetched 3280 kB in 8s (387 kB/s)       bzip2 0 B] [35 T
Reading package lists... Donege lists... 0%
Reading package lists... Donege lists... 0%
Building dependency tree... 66%ency tree... 0%
Reading state information... Doneormation... 0%
    amazon-ebs: E: Unable to locate package redis-server
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Script exited with non-zero exit status: 100

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Script exited with non-zero exit status: 100

==> Builds finished but no artifacts were created.
$

need-more-info

Source

englishm

Most helpful comment

This was biting me, too, and I think I might know what's going on. On a typical Ubuntu EC2 instance, the top of /etc/apt/sources.list says:

## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.

Indeed apt-get update _should_ be using a mirror like http://us-east-1.ec2.archive.ubuntu.com. The normal sequence of events is:

Instance boots up and the ssh service starts.
The cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
sudo apt-get update runs against the _new_ mirror.
sudo apt-get install -y redis-server runs against the _new_ mirror and everything works.

However, apt-get update is still using http://archive.ubuntu.com when the error happens, which makes me think the problematic sequence of events is:

Instance boots up and the ssh service starts.
sudo apt-get update starts running against the _old_ mirror.
Meanwhile, the cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
sudo apt-get install -y redis-server runs against the _new_ mirror, but errors out because apt-get update has never been run against this particular mirror.

The workaround I'm using is:

    "provisioners": [{
        "type": "shell",
        "inline": [
            "sleep 30",
            "sudo apt-get update",
            "sudo apt-get install -y redis-server"
        ]
    }]

I assume Packer considers the system "ready" as soon as sshd is listening? Ideally it would check for some kind of indication that the system has finished booting before it runs any commands, but I don't know a good way to do this on Ubuntu, let alone the general case of any Linux distribution. Which is why I'm using a sleep statement for now...

grosskur on 20 Jul 2013

👍4

All 23 comments

I haven't yet checked what packages are available from the source AMI, but this seems pretty odd.
From another Ubuntu VM I have running right now:

vagrant@redacted:~$ sudo apt-get install -y redis-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  redis-server
0 upgraded, 1 newly installed, 0 to remove and 34 not upgraded.
Need to get 204 kB of archives.
After this operation, 523 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/universe redis-server amd64 2:2.2.12-1build1 [204 kB]
Fetched 204 kB in 0s (297 kB/s)
Selecting previously unselected package redis-server.
(Reading database ... 51084 files and directories currently installed.)
Unpacking redis-server (from .../redis-server_2%3a2.2.12-1build1_amd64.deb) ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db ...
Setting up redis-server (2:2.2.12-1build1) ...
Starting redis-server: redis-server.
vagrant@vwhitfoot:~$ uname -a
Linux vwhitfoot 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
vagrant@redacted:~$

englishm on 28 Jun 2013

Ugh this is the weirdest thing. I've seen this happen too and simply don't know what causes it. Why... why does this happen?

mitchellh on 28 Jun 2013

I think this is actually a timing issue with apt. I added another line (cat /etc/apt/sources.list) to the inline shell provisioning steps and now redis-server _sometimes_ gets installed. Running into other intermittent errors now. Possibly related to #42?

englishm on 29 Jun 2013

In hindsight, I should have had had PACKER_LOG=1 while testing this, but here are some examples of errors I still get _intermittently_ with 0.1.1:

https://gist.github.com/englishm/e12fff3aa975d956e5c7

englishm on 29 Jun 2013

Yeah it is intermittent for me too. I wonder if you put a sleep 1 in there if this completely solves the problem? But why is there a timing issue? I execute the commands in serial and wait for an exit status from the prior one before executing the next one. :(

mitchellh on 29 Jun 2013

Yeah, I was wondering how related it might be to https://github.com/mitchellh/packer/issues/42 as well. It does seem to be some kind of race condition. Out of curiosity is it any more stable on an m1.small? If there was ever a box prone to AWS weirdness it's the t1.micro

MattSurabian on 29 Jun 2013

I'm headed out for vacation today and can't check this right now.
I'll have limited Internet connectivity while I'm gone though and I'll
update this if I have time/bandwidth to retest at all.

On Sat, Jun 29, 2013 at 10:23 AM, Matthew Surabian <[email protected]

wrote:

Yeah, I was wondering how related it might be to #42https://github.com/mitchellh/packer/issues/42as well. It does seem to be some kind of race condition. Out of curiosity
is it any more stable on an m1.small? If there was ever a box prone to AWS
weirdness it's the t1.micro

—
Reply to this email directly or view it on GitHubhttps://github.com/mitchellh/packer/issues/41#issuecomment-20230622
.

Mike English
Atomic Object | http://www.atomicobject.com
mike.[email protected]
[Ph] +1.616.776.6020

englishm on 29 Jun 2013

@mitchellh I am confused, I thought Packer was creating a single shell script for the whole provisioner? How could this be caused by a race within Packer when the whole script is executed in one command?

Instead, could it be an issue with the VM image itself? I saw it happen often with the AMI from the documentation, but not with Raring (ami-e995e380).

priteau on 29 Jun 2013

@priteau It does create a shell script, it can't be a race within Packer. That comment was just speculation.

@MattSurabian Unfortunately, it is unrelated to #42. Packer makes a single shell script from your inline calls, so it can't be a race within Packer itself.

mitchellh on 29 Jun 2013

I'm running through the docs myself, and I got the same error. I added the sleep, and got the same.

    "provisioners": [{
        "type": "shell",
        "inline": [
            "sudo apt-get update",
            "sleep 1",
            "sudo apt-get install -y redis-server"
        ]
    }]

I also manually launched the demo AMI, and sudo apt-get update; sudo apt-get install -y redis-server worked fine. How do I see the script that packer is executing on the server?

Edit: I modified provisioner.go to log the script, and it's pretty much what's in the JSON file:

2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Opening /tmp/packer-shell394007326 for reading
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Read script: sudo apt-get update
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: sudo apt-get install -y redis-server
2013/07/01 21:39:55 /home/eric/src/go/src/github.com/ericlathrop/packer/bin/packer-provisioner-shell: 2013/07/01 21:39:55 Uploading /tmp/packer-shell394007326 => /tmp/script.sh

ericlathrop on 2 Jul 2013

👍1

@ericlathrop You can see here what Packer does: https://github.com/mitchellh/packer/blob/06108b4f96cb899c9bf9c40808170e0ea8143c8e/provisioner/shell/provisioner.go#L127-L150

Basically it just takes each inline command and puts them in a script one line at a time. It then saves it. Line 140 is particularly where things happen.

mitchellh on 2 Jul 2013

This was biting me, too, and I think I might know what's going on. On a typical Ubuntu EC2 instance, the top of /etc/apt/sources.list says:

## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.

Indeed apt-get update _should_ be using a mirror like http://us-east-1.ec2.archive.ubuntu.com. The normal sequence of events is:

Instance boots up and the ssh service starts.
The cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
sudo apt-get update runs against the _new_ mirror.
sudo apt-get install -y redis-server runs against the _new_ mirror and everything works.

However, apt-get update is still using http://archive.ubuntu.com when the error happens, which makes me think the problematic sequence of events is:

Instance boots up and the ssh service starts.
sudo apt-get update starts running against the _old_ mirror.
Meanwhile, the cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.
sudo apt-get install -y redis-server runs against the _new_ mirror, but errors out because apt-get update has never been run against this particular mirror.

The workaround I'm using is:

    "provisioners": [{
        "type": "shell",
        "inline": [
            "sleep 30",
            "sudo apt-get update",
            "sudo apt-get install -y redis-server"
        ]
    }]

grosskur on 20 Jul 2013

👍4

@grosskur your detective work is excellent! We can check if cloudinit has finished by checking for the existence of /var/lib/cloud/instance/boot-finished.

priteau on 20 Jul 2013

I've updated the getting started guide to get it working. Thanks!

mitchellh on 20 Jul 2013

Thanks, @grosskur, @mitchellh!

Would be nice to also see a more robust "instance readiness" check like @priteau suggests.
Maybe as a config option?

englishm on 20 Jul 2013

@englishm Maybe, but I think it fits into the provisioner's role.

mitchellh on 21 Jul 2013

@mitchellh fair enough.

englishm on 21 Jul 2013

I am unable to find documentation of the above in the docs anymore, and this seems to still be a problem.

davclark on 28 Aug 2014

@davclark It is still present in http://www.packer.io/intro/getting-started/provision.html, and also described in the shell provisioner documentation at the very end of http://www.packer.io/docs/provisioners/shell.html.

priteau on 29 Aug 2014

OK - I see the sleep lines. I was looking for something more cloud-init specific that would look for the existence of a file (which is what I'm doing). I suppose sleep 30 is a good general solution, though.

Thanks @priteau!

davclark on 30 Aug 2014

Some thoughts on waiting for cloud-init more effectively than a hardcoded sleep:

Sufficiently new cloud-init versions (0.7.5 and newer, I believe) have json status files in /var/lib/cloud/data/ and /run/cloud-init that are documented at http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/status.txt
All cloud-init versions (or at least 0.6.3 and newer which shipped with Ubuntu 12.04) update /var/lib/cloud/instance/boot-finished. That file is persistent across boots, though, so it only works once. That might be ok for packer, but could lead to weirdness if it's left lying around in some upstream AMI. Documented in https://bugs.launchpad.net/cloud-init/+bug/1258113
uvtool (which appears to be an Ubuntu-focused packer-like image creation tool) uses the following code to wait for cloud-init to finish: http://bazaar.launchpad.net/~uvtool-dev/uvtool/trunk/view/head:/remote-wait.sh

mlococo on 2 Jun 2015

This was biting me, too, and I think I might know what's going on. On a typical Ubuntu EC2 instance, the top of /etc/apt/sources.list says:
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
Indeed apt-get update _should_ be using a mirror like http://us-east-1.ec2.archive.ubuntu.com. The normal sequence of events is:

Instance boots up and the ssh service starts.

The cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.

sudo apt-get update runs against the _new_ mirror.

sudo apt-get install -y redis-server runs against the _new_ mirror and everything works.

However, apt-get update is still using http://archive.ubuntu.com when the error happens, which makes me think the problematic sequence of events is:

Instance boots up and the ssh service starts.

sudo apt-get update starts running against the _old_ mirror.

Meanwhile, the cloud-init service starts and overwrites /etc/apt/sources.list with an EC2-specific mirror.

sudo apt-get install -y redis-server runs against the _new_ mirror, but errors out because apt-get update has never been run against this particular mirror.

The workaround I'm using is:
    "provisioners": [{
        "type": "shell",
        "inline": [
            "sleep 30",
            "sudo apt-get update",
            "sudo apt-get install -y redis-server"
        ]
    }]
I assume Packer considers the system "ready" as soon as sshd is listening? Ideally it would check for some kind of indication that the system has finished booting before it runs any commands, but I don't know a good way to do this on Ubuntu, let alone the general case of any Linux distribution. Which is why I'm using a sleep statement for now...

This was very helpful, thank you very much. We had the same issue while installing expect on Ubuntu using packer.

arungkrishna on 6 Nov 2019

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.