Packer: docker file upload failure

[eugenetaranov]

Hello,

I am using Packer v0.8.6 on linux 64bit, docker 1.7

First thing in my recipe is to put file into docker container but fails immediately, please look into error attached:
https://gist.github.com/eugenetaranov/f7a9c7f30fc3c2952fb3

Let me know if you need any other info.

Thanks

[cbednarski]

@eugenetaranov Thanks for the report. Can you share a minimal packer template that reproduces this problem? This may be a duplicate of https://github.com/mitchellh/packer/issues/2581

Also, we have recently fixed a bunch of issues in this part of packer so this may be fixed in packer master -- if you feel comfortable building packer yourself you can try the latest to see if it is fixed.

[eugenetaranov]

Sure, here you are:

{
  "builders": [
    {
      "type": "docker",
      "image": "docker.io/centos",
      "commit": "true"
    }
  ],

 "provisioners": [
    {
      "type": "file",
      "source": "vaultpasswd.txt",
      "destination": "/tmp/vaultpasswd.txt"
    },
    {
      "type": "shell",
      "inline": ["yum install -y http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm", "yum --enablerepo=epel install ansible -y"]
    },
    {
      "type": "ansible-local",
      "playbook_file": "/home/centos/addnow/playbooks/local_app.yml",
      "playbook_dir": "/home/centos/addnow/playbooks",
      "extra_arguments": "--vault-password-file /tmp/vaultpasswd.txt"
    },
    {
      "type": "shell",
      "inline": ["rm -f /tmp/vaultpasswd.txt"]
    }
  ]

}

[sidcarter]

I was about to open an issue for the same. I'm using:

# ./packer -v
0.8.6
# docker -v
Docker version 1.7.1, build 446ad9b/1.7.1
# cat /etc/centos-release
CentOS Linux release 7.1.1503 (Core)

And here's my template:

# cat docker-heketi.json
{
    "builders":[{
        "type": "docker",
        "image": "centos:centos7",
        "pull": true,
        "export_path": "docker-heketi-image.tar"
    }],
    "provisioners":[
        {
            "type": "shell",
            "scripts": [
                "provision/scripts/prep-heketi.sh"
            ]
        }
    ]
}

And here's the errors:

2015/10/15 04:50:45 [INFO] 699 bytes written for 'uploadData'
2015/10/15 04:50:45 packer-builder-docker: 2015/10/15 04:50:45 Executing in container cca9b12c3eb3b6df74bf469fe75c3759cd6b4d2fef33811407d68bc8a93349fa: "(command cp /packer-files/upload026000179 /tmp/script_7943.sh) >/packer-files/cmd079914230 2>&1; echo $? >/packer-files/cmd079914230-exit"
2015/10/15 04:50:49 packer-provisioner-shell: 2015/10/15 04:50:49 Retryable error: Error uploading script: Upload failed with non-zero exit status: 1
2015/10/15 04:50:51 packer-provisioner-shell: 2015/10/15 04:50:51 [INFO] 699 bytes written for 'uploadData'
2015/10/15 04:50:51 [INFO] 699 bytes written for 'uploadData'
2015/10/15 04:50:51 packer-builder-docker: 2015/10/15 04:50:51 Executing in container cca9b12c3eb3b6df74bf469fe75c3759cd6b4d2fef33811407d68bc8a93349fa: "(command cp /packer-files/upload858505693 /tmp/script_7943.sh) >/packer-files/cmd072223640 2>&1; echo $? >/packer-files/cmd072223640-exit"
2015/10/15 04:50:55 packer-provisioner-shell: 2015/10/15 04:50:55 Retryable error: Error uploading script: Upload failed with non-zero exit status: 1

Will try packer from master and see if that helps.

[eugenetaranov]

@cbednarski my config is without "pty" set in builder, although tested with "pty" set to to true and false - still fails

[eugenetaranov]

@cbednarski had a chance to test v0.8.7.dev, worked like a charm :)

[cbednarski]

had a chance to test v0.8.7.dev, worked like a charm :)

@eugenetaranov Happy to hear that! We're still working out some kinks with the version on master but this fix will be included in the next release.

@sidcarter If you are are still running into this problem on master please let me know, and also include the output from packer version so I can see which commit you've built from.

[sidcarter]

@cbednarski Built from master:

# packer version
Packer v0.8.7.dev (82a7f6ef9718740e272b21a4a3fce720b0c4fb77)
# PACKER_LOG=1 packer build docker-heketi.json
2015/10/16 05:15:15 [INFO] Packer version: 0.8.7 dev 82a7f6ef9718740e272b21a4a3fce720b0c4fb77
2015/10/16 05:15:15 Packer Target OS/Arch: linux amd64
2015/10/16 05:15:15 Built with Go Version: go1.4.2

Issue persists:

2015/10/16 05:17:00 ui: ==> docker: Starting docker container...
    docker: 812e9d9d677f: Already exists
2015/10/16 05:17:00 ui:     docker: Run command: docker run -v /tmp/packer-docker375116298:/packer-files -d -i -t centos:latest /bin/bash
2015/10/16 05:17:00 packer-builder-docker: 2015/10/16 05:17:00 Starting container with args: [run -v /tmp/packer-docker375116298:/packer-files -d -i -t centos:latest /bin/bash]
    docker: 4234bfdd88f8: Already exists
    docker: ce20c473cd8a: Already exists
2015/10/16 05:17:00 packer-builder-docker: 2015/10/16 05:17:00 Waiting for container to finish starting
    docker: Digest: sha256:3aaab9f1297db9b013063c781cfe901e2aa6e7e334c1d1f4df12f25ce356f2e5
    docker: Status: Image is up to date for docker.io/centos:latest
==> docker: Starting docker container...
    docker: Run command: docker run -v /tmp/packer-docker375116298:/packer-files -d -i -t centos:latest /bin/bash
2015/10/16 05:17:01 ui:     docker: Container ID: 0564ca0661c479ff1e59507b3f5a29819ca09cf83857f126082739cc91ddcff4
    docker: Container ID: 0564ca0661c479ff1e59507b3f5a29819ca09cf83857f126082739cc91ddcff4
2015/10/16 05:17:01 packer-builder-docker: 2015/10/16 05:17:01 Running the provision hook
2015/10/16 05:17:01 ui: ==> docker: Provisioning with shell script: provision/scripts/prep-heketi.sh
2015/10/16 05:17:01 packer-provisioner-shell: 2015/10/16 05:17:01 Opening provision/scripts/prep-heketi.sh for reading
==> docker: Provisioning with shell script: provision/scripts/prep-heketi.sh
2015/10/16 05:17:01 packer-provisioner-shell: 2015/10/16 05:17:01 [INFO] 690 bytes written for 'uploadData'
2015/10/16 05:17:01 [INFO] 690 bytes written for 'uploadData'
2015/10/16 05:17:01 packer-builder-docker: 2015/10/16 05:17:01 Executing in container 0564ca0661c479ff1e59507b3f5a29819ca09cf83857f126082739cc91ddcff4: "(command cp /packer-files/upload009242081 /tmp/script_7943.sh) >/packer-files/cmd784927692 2>&1; echo $? >/packer-files/cmd784927692-exit"
2015/10/16 05:17:05 packer-provisioner-shell: 2015/10/16 05:17:05 Retryable error: Error uploading script: Upload failed with non-zero exit status: 1
2015/10/16 05:17:07 packer-provisioner-shell: 2015/10/16 05:17:07 [INFO] 690 bytes written for 'uploadData'

[jhoblitt]

I'm seeing this issue with packer 0.8.6 and current master on centos 7 with docker 1.8.2. Oddly, I do not see it with packer 0.8.6 on fedora 22 with docker 1.8.2-fc22.

$ ../packer --version
0.8.7
$ ../packer version
Packer v0.8.7.dev (0660c703ad7cc93a81a1038be2812d5d61cb459f)

$ docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Package Version: docker-1.8.2-7.el7.centos.x86_64
 Go version:   go1.4.2
 Git commit:   bb472f0/1.8.2
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Package Version: 
 Go version:   go1.4.2
 Git commit:   bb472f0/1.8.2
 Built:        
 OS/Arch:      linux/amd64

...
==> docker-centos-7: Provisioning with shell script: scripts/build.sh


==> docker-centos-7: Killing the container: 29ba7b4053bfc64d0dc09dc41314da25a915065377a1c4fcb54d8e8215fcc38f
Build 'docker-centos-7' errored: Retryable error: Error uploading script: Upload failed with non-zero exit status: 1

[wangjunwei87]

packer 0.8.6, hang when upload dir to container.

[tylermauthe]

Packer 0.10.1, Docker 1.9.1, CentOS 7 -- same issue:

2016/05/25 13:18:48 packer: 2016/05/25 13:18:48 Opening ./bootstrap/puppet.sh for reading
2016/05/25 13:18:48 packer: 2016/05/25 13:18:48 [INFO] 153 bytes written for 'uploadData'
2016/05/25 13:18:48 [INFO] 153 bytes written for 'uploadData'
2016/05/25 13:18:48 packer: 2016/05/25 13:18:48 Executing in container 779505d1192957799d6ef54bcc41954d71be9db3c7bfdd29e7b8149e0a1b7c1a: "(command cp /packer-files/upload526191701 /tmp/script_1659.sh) >/packer-files/cmd293740464 2>&1; echo $? >/packer-files/cmd293740464-exit"
2016/05/25 13:18:52 packer: 2016/05/25 13:18:52 Retryable error: Error uploading script: Upload failed with non-zero exit status: 2

Here's my git repo: https://github.com/tylermauthe/wintermute

[chrono]

Running into the same issue. Packer 0.10.2, Docker server+client 1.12.2. Ubuntu 15.10.

==> docker: Upload failed: Upload failed with non-zero exit status: 2

I noticed that packer instructed docker to mount /home/ubuntu/.packer.d/tmp/packer-docker465479121:/packer-files.

Packer then ran command cp /packer-files/upload625208819 destination_file_name. At that point, the directory /home/ubuntu/.packer.d/tmp/packer-docker465479121 was empty.

Packer was at: ==> docker: Pausing before cleanup of step 'StepRun'. Press enter to continue.)

It seems the initial "provisioning" of the docker tmp directory content failed.

[chrono]

I was able to build a minimal packerfile that can reproduce the issue:

{
  "builders": [
    {
      "type": "docker",
      "image": "ubuntu:trusty",
      "export_path": "foo.tar",
      "run_command": ["-u", "nobody", "-d", "-i", "-t", "{{.Image}}", "/bin/bash"]
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "test.txt",
      "destination": "/tmp/test.txt"
    }
  ]
}

runs as follows:

docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Pulling Docker image: ubuntu:trusty
    docker: trusty: Pulling from library/ubuntu
    docker: Digest: sha256:147913621d9cdea08853f6ba9116c2e27a3ceffecf3b492983ae97c3d643fbbe
    docker: Status: Image is up to date for ubuntu:trusty
==> docker: Starting docker container...
    docker: Run command: docker run -v /home/ubuntu/.packer.d/tmp/packer-docker624883109:/packer-files -u nobody -d -i -t ubuntu:trusty /bin/bash
    docker: Container ID: 968b8f8ba4fcde2ea5e2d4f5236d9f5a0ae9ebea27af510fdc753cd3cfef8a27
==> docker: Uploading test.txt => /tmp/test.txt
==> docker: Upload failed: Upload failed with non-zero exit status: 2
==> docker: Killing the container: 968b8f8ba4fcde2ea5e2d4f5236d9f5a0ae9ebea27af510fdc753cd3cfef8a27
Build 'docker' errored: Upload failed with non-zero exit status: 2

==> Some builds didn't complete successfully and had errors:
--> docker: Upload failed with non-zero exit status: 2

==> Builds finished but no artifacts were created.

Meanwhile -- proof that nobody can write to /tmp/test.txt:

$ docker run --rm -u nobody ubuntu:trusty bash -c 'touch /tmp/test.txt && echo success'
success

More debugging shows the file permissions:

2016/10/19 23:01:03 ui: ==> docker: Uploading test.txt => /tmp/test.txt
[pid  9901] stat("test.txt", {st_dev=makedev(202, 1), st_ino=395333, st_mode=S_IFREG|0664, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=0, st_size=0, st_atime=2016/10/19-22:45:19, st_mtime=2016/10/19-22:44:39, st_ctime=2016/10/19-22:44:39}) = 0
[pid  9901] openat(AT_FDCWD, "test.txt", O_RDONLY|O_CLOEXEC) = 3
[pid  9903] openat(AT_FDCWD, "/home/ubuntu/.packer.d/tmp/packer-docker563916886/upload924968381", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3
[pid  9903] openat(AT_FDCWD, "/home/ubuntu/.packer.d/tmp/packer-docker563916886/cmd736322552", O_RDWR|O_CREAT|O_EXCL|O_CLOEXEC, 0600) = 3

Creating files that have a mask of 0600. If the user inside the docker container does not happen to be root and does not share the uid with the host, they will not be able to read the upload file.

The temporary file -- to be shared with the docker container via volume mount is created via ioutil.TempFile: https://github.com/mitchellh/packer/blob/d52a83a901e99794167252071382be1a7a75d861/builder/docker/communicator.go#L63-L96

ioutil.TempFile creates tmpfiles with permissions of 0600: https://golang.org/src/io/ioutil/tempfile.go?s=1285:1342#L37

[chrono]

I just verified that my permission-related upload issues are going to be resolved with #3922

[mwhooker]

Can anyone verify that this is still an issue? unable to reproduce with the above json

docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Pulling Docker image: ubuntu:xenial
    docker: xenial: Pulling from library/ubuntu
    docker: Digest: sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950
    docker: Status: Image is up to date for ubuntu:xenial
==> docker: Starting docker container...
    docker: Run command: docker run -v /Users/mwhooker/.packer.d/tmp/packer-docker001346228:/packer-files -u nobody -d -i -t ubuntu:xenial /bin/bash
    docker: Container ID: 1667b0f0117bf94594622fa033212843db96170e3af3e02fa629cdbfee03fb04
==> docker: Uploading files/foo.txt => /tmp/test.txt
==> docker: Exporting the container
==> docker: Killing the container: 1667b0f0117bf94594622fa033212843db96170e3af3e02fa629cdbfee03fb04
Build 'docker' finished.

==> Builds finished. The artifacts of successful builds are:
--> docker: Exported Docker file: foo.tar

[vtolstov]

@mwhooker i'm confirm:

{
  "builders": [
      {
        "type": "docker",
        "image": "fedora:25",
        "commit": true
      }
  ],
  "provisioners": [
{
      "type": "file",
      "source": "scripts/packer.sh",
      "destination": "/tmp/test.txt"
    }
]
}

[vtolstov]

hm, @chrono please check dmesg, as i remember docker have some issues like this:
SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)
in my case

[mwhooker]

@vtolstov I can't reproduce with that json, either. Can you give your host OS and docker version?

[vtolstov]

@mwhooker docker-1.12.6 as i know it broken and i need 1.13...

[mwhooker]

hmm, I'm on 1.12.5. I wonder if this is only broken on linux or something

[chrono]

Re-checked and still problematic, at minimum on:

• Ubuntu trusty 20161213 / kernel 3.13.0-105-generic (ami-5ac2cd4d)
• Docker version 1.13.0, build 49bf474
• Packer 0.12.2

A pretty fool-proof way to repeat for anybody with an AWS account:

Launch ami-5ac2cd4d (recent-ish / 20161213 official trusty cloudimg) in AWS us-east-1 on t2.micro with the following userdata:

#!/bin/bash

curl -fsSL https://yum.dockerproject.org/gpg | apt-key add -
add-apt-repository \
       "deb https://apt.dockerproject.org/repo/ \
       ubuntu-$(lsb_release -cs) \
       main"
apt-get update
apt-get -y install docker-engine unzip
export HOME=/root
cd $HOME
wget https://releases.hashicorp.com/packer/0.12.2/packer_0.12.2_linux_amd64.zip
unzip packer_0.12.2_linux_amd64.zip
echo "hi" > test.txt
cat <<EOF > build.json
{
  "builders": [
    {
      "type": "docker",
      "image": "ubuntu:trusty",
      "export_path": "foo.tar",
      "run_command": ["-u", "nobody", "-d", "-i", "-t", "{{.Image}}", "/bin/bash"]
    }
  ],
  "provisioners": [
    {
      "type": "file",
      "source": "test.txt",
      "destination": "/tmp/test.txt"
    }
  ]
}
EOF
uname -a
lsb_release -a
docker --version
./packer --version
./packer build build.json

Now follow either the console output via AWS console (screenshot attached at end) or ssh in and tail /var/log/cloud-init-output.log:

Linux ip-172-30-5-77 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:        14.04
Codename:       trusty
Docker version 1.13.0, build 49bf474
0.12.2
docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Pulling Docker image: ubuntu:trusty
    docker: trusty: Pulling from library/ubuntu
    docker: c60055a51d74: Pulling fs layer
    docker: 755da0cdb7d2: Pulling fs layer
    docker: 969d017f67e6: Pulling fs layer
    docker: 37c9a9113595: Pulling fs layer
    docker: a3d9f8479786: Pulling fs layer
    docker: 37c9a9113595: Waiting
    docker: a3d9f8479786: Waiting
    docker: 755da0cdb7d2: Verifying Checksum
    docker: 755da0cdb7d2: Download complete
    docker: 969d017f67e6: Verifying Checksum
    docker: 969d017f67e6: Download complete
    docker: 37c9a9113595: Verifying Checksum
    docker: 37c9a9113595: Download complete
    docker: a3d9f8479786: Verifying Checksum
    docker: a3d9f8479786: Download complete
    docker: c60055a51d74: Verifying Checksum
    docker: c60055a51d74: Download complete
    docker: c60055a51d74: Pull complete
    docker: 755da0cdb7d2: Pull complete
    docker: 969d017f67e6: Pull complete
    docker: 37c9a9113595: Pull complete
    docker: a3d9f8479786: Pull complete
    docker: Digest: sha256:8f5f12335124c1b78e4cf2f8860d395f75ba279bae70a3c18dd470e910e38ec5
    docker: Status: Downloaded newer image for ubuntu:trusty
==> docker: Starting docker container...
    docker: Run command: docker run -v /root/.packer.d/tmp/packer-docker621318098:/packer-files -u nobody -d -i -t ubuntu:trusty /bin/bash
    docker: Container ID: d01033574b199c6173cf1e269d159b37bd96faeb2e6955497e17bea40331681f
==> docker: Uploading test.txt => /tmp/test.txt
==> docker: Upload failed: Upload failed with non-zero exit status: 1
==> docker: Killing the container: d01033574b199c6173cf1e269d159b37bd96faeb2e6955497e17bea40331681f
Build 'docker' errored: Upload failed with non-zero exit status: 1

==> Some builds didn't complete successfully and had errors:
--> docker: Upload failed with non-zero exit status: 1

==> Builds finished but no artifacts were created.

@vtolstov No SELinux involved.

[mwhooker]

I think this will be resolved by #4186. After that gets merged, please open a new issue if you continue to see this

[jhoblitt]

This issue was closed 6 months ago but this problem is still present in the latest release (1.0.3) and #4186 hasn't been merged. Could we reopen this issue?

[chiefy]

docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Starting docker container...
    docker: Run command: docker run -v /Users/najewicc/.packer.d/tmp/packer-docker232980080:/packer-files -d -i -t public/centos:7.3.1611 /bin/bash
    docker: Container ID: 306f07742b73a901da034a8fdac1775ded5a1d715580bf68dd2e2d5031759c65
==> docker: Uploading ./files/centos7/ => /
==> docker: Killing the container: 306f07742b73a901da034a8fdac1775ded5a1d715580bf68dd2e2d5031759c65
Build 'docker' errored: Upload failed with non-zero exit status: 1

==> Some builds didn't complete successfully and had errors:
--> docker: Upload failed with non-zero exit status: 1

Also running into this:

❯ docker version
Client:
 Version:      17.06.0-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:31:53 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.0-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   02c1d87
 Built:        Fri Jun 23 21:51:55 2017
 OS/Arch:      linux/amd64
 Experimental: true

packer 1.0.3

using centos:7.3.1611

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.