Packer: Shell provisioner Unexpected Failures

Created on 2 Oct 2019  ·  6Comments  ·  Source: hashicorp/packer

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

Can install packages on Ubuntu 18.04 with apt-get running commands on AWS EC2 host, but the same installation fails with Packer running Ubuntu 18.04 with the shell provisioner.

Reproduction Steps

I launched an AWS EC2 instance from latest Ubuntu 18.04 AMI:

ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 (ami-06f2f779464715dc5)

then I attempt to install java and maven:

# bootstrap.sh
$ uname -a
Linux ip-10-200-63-175 4.15.0-1044-aws #46-Ubuntu SMP Thu Jul 4 13:38:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ sudo apt-get update
$ sudo apt-get install -y openjdk-8-jre-headless maven
.
.
.
The following additional packages will be installed:
  ca-certificates-java fontconfig-config fonts-dejavu-core java-common libaopalliance-java libapache-pom-java
  libatinject-jsr330-api-java libavahi-client3 libavahi-common-data libavahi-common3 libcdi-api-java
  libcommons-cli-java libcommons-io-java libcommons-lang3-java libcommons-parent-java libcups2 libfontconfig1
  libgeronimo-annotation-1.3-spec-java libgeronimo-interceptor-3.0-spec-java libguava-java libguice-java
  libhawtjni-runtime-java libjansi-java libjansi-native-java libjpeg-turbo8 libjpeg8 libjsr305-java liblcms2-2
  libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java libmaven3-core-java libnspr4 libnss3
  libpcsclite1 libplexus-cipher-java libplexus-classworlds-java libplexus-component-annotations-java
  libplexus-interpolation-java libplexus-sec-dispatcher-java libplexus-utils2-java libsisu-inject-java
  libsisu-plexus-java libslf4j-java libwagon-file-java libwagon-http-shaded-java libwagon-provider-api-java libxi6
  libxrender1 libxtst6 x11-common
Suggested packages:
  default-jre libaopalliance-java-doc libatinject-jsr330-api-java-doc libservlet3.1-java libcommons-io-java-doc
  libcommons-lang3-java-doc cups-common libasm-java libcglib-java libjsr305-java-doc liblcms2-utils
  libmaven-shared-utils-java-doc liblogback-java pcscd libplexus-cipher-java-doc libplexus-classworlds-java-doc
  libplexus-interpolation-java-doc libplexus-sec-dispatcher-java-doc libplexus-utils2-java-doc junit4 testng
  libcommons-logging-java liblog4j1.2-java libnss-mdns fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho
  fonts-wqy-microhei fonts-wqy-zenhei fonts-indic
The following NEW packages will be installed:
  ca-certificates-java fontconfig-config fonts-dejavu-core java-common libaopalliance-java libapache-pom-java
  libatinject-jsr330-api-java libavahi-client3 libavahi-common-data libavahi-common3 libcdi-api-java
  libcommons-cli-java libcommons-io-java libcommons-lang3-java libcommons-parent-java libcups2 libfontconfig1
  libgeronimo-annotation-1.3-spec-java libgeronimo-interceptor-3.0-spec-java libguava-java libguice-java
  libhawtjni-runtime-java libjansi-java libjansi-native-java libjpeg-turbo8 libjpeg8 libjsr305-java liblcms2-2
  libmaven-parent-java libmaven-resolver-java libmaven-shared-utils-java libmaven3-core-java libnspr4 libnss3
  libpcsclite1 libplexus-cipher-java libplexus-classworlds-java libplexus-component-annotations-java
  libplexus-interpolation-java libplexus-sec-dispatcher-java libplexus-utils2-java libsisu-inject-java
  libsisu-plexus-java libslf4j-java libwagon-file-java libwagon-http-shaded-java libwagon-provider-api-java libxi6
  libxrender1 libxtst6 maven openjdk-8-jre-headless x11-common
0 upgraded, 53 newly installed, 0 to remove and 68 not upgraded.
.
.
.
$ java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_222, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "4.15.0-1044-aws", arch: "amd64", family: "unix"

This package installation succeeds.

now doing the same with packer configuration: packer.json:

``` packer.json
{
"builders": [{
"type": "amazon-ebs",
"profile": "de-central",
"region": "us-west-2",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*",
"root-device-type": "ebs"
},
"owners": ["099720109477"],
"most_recent": true
},
"instance_type": "t2.micro",
"ssh_username": "ubuntu",
"ami_name": "java-maven-image {{timestamp}}"
}],
"provisioners": [{
"type": "shell",
"script": "bootstrap.sh"
}]
}

this fails with the following error:

$ packer build packer.json
.
.
.
==> amazon-ebs: Provisioning with shell script: bootstrap.sh
amazon-ebs: Linux ip-172-31-30-251 4.15.0-1050-aws #52-Ubuntu SMP Thu Sep 12 19:56:00 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
.
.
.
amazon-ebs: Some packages could not be installed. This may mean that you have
amazon-ebs: requested an impossible situation or if you are using the unstable
amazon-ebs: distribution that some required packages have not yet been created
amazon-ebs: or been moved out of Incoming.
amazon-ebs: The following information may help to resolve the situation:
amazon-ebs:
amazon-ebs: The following packages have unmet dependencies:
amazon-ebs: maven : Depends: libjansi-java but it is not installable
amazon-ebs: Depends: libmaven3-core-java (= 3.6.0-1~18.04.1) but it is not going to be installed
amazon-ebs: Depends: libwagon-file-java but it is not installable
amazon-ebs: Depends: libwagon-http-shaded-java but it is not installable
amazon-ebs: openjdk-8-jre-headless : Depends: ca-certificates-java but it is not installable
amazon-ebs: Depends: libjpeg8 (>= 8c) but it is not installable
amazon-ebs: Depends: libfontconfig1 but it is not installable
amazon-ebs: Depends: libnss3 (>= 2:3.17.1) but it is not going to be installed
amazon-ebs: Depends: libpcsclite1 (>= 1.3.0) but it is not installable
amazon-ebs: Depends: libxi6 but it is not installable
amazon-ebs: Depends: libxrender1 but it is not installable
amazon-ebs: Depends: libxtst6 but it is not installable
==> amazon-ebs: E: Unable to correct problems, you have held broken packages.
.
.
.

What is going on with Packer? Why would it perform differently than running the package installation commands on EC2?

### Packer version

packer --version
1.4.3
```

Operating system and Environment details

MacOS

Darwin localhost 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64

provisioneshell question
Source
picture aidan-melen

Most helpful comment

you nailed it @SwampDragons. I just needed to wait for cloud-init. He is my modified bootstrap.sh that I am passing to the shell provisioner in Packer:

# https://www.packer.io/docs/other/debugging.html#issues-installing-ubuntu-packages
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done

sudo apt-get update
sudo apt-get install openjdk-8-jre-headless maven -y
picture aidan-melen on 24 Oct 2019
❤1 🎉1 👍1

All 6 comments

Can you share bootstrap.sh please? I assume it contains 

uname -a
sudo apt-get update
sudo apt-get install -y openjdk-8-jre-headless maven

But I'd like to have the exact file.

SwampDragons picture SwampDragons on 2 Oct 2019

Those are the first lines of the script other than some environment variables that are unrelated. The failure actually occurs on this line with sudo apt-get install -y openjdk-8-jre-headless maven because it is unable to install the dependencies with the shell provisioner. I don't know why it fails in packer. My current work around is to spin up the latest ubuntu 18.04 manually, manually run the bootstrap.sh, and manually creating an AMI and terminating the instance -- you know what packer is suppose to do.

aidan-melen picture aidan-melen on 2 Oct 2019

I'd still like the exact file so I can see what flags and what shebang you're using.

SwampDragons picture SwampDragons on 3 Oct 2019

Closing since I never heard back and I can't reproduce with the given information. The installation completes successfully for me.

That said, I have seen this kind of failure happening due to race conditions in the AWS setup scripts; sometimes having a brief sleep before performing the update and the installation can help eliminate failures.

https://www.packer.io/docs/other/debugging.html#issues-installing-ubuntu-packages

SwampDragons picture SwampDragons on 18 Oct 2019

you nailed it @SwampDragons. I just needed to wait for cloud-init. He is my modified bootstrap.sh that I am passing to the shell provisioner in Packer:

# https://www.packer.io/docs/other/debugging.html#issues-installing-ubuntu-packages
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done

sudo apt-get update
sudo apt-get install openjdk-8-jre-headless maven -y
aidan-melen picture aidan-melen on 24 Oct 2019
❤1 🎉1 👍1

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashibot[bot] picture hashibot[bot] on 23 Jan 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jesse-c picture jesse-c  ·  3Comments
tleyden picture tleyden  ·  3Comments
sourav82 picture sourav82  ·  3Comments
s4mur4i picture s4mur4i  ·  3Comments
DanielBo picture DanielBo  ·  3Comments