Lots of details tracked here.
macOS 10.13.3
$ packer --version
1.2.1
$ vboxmanage --version
5.2.8r121009
Debug1.log
The above log as a gist
Debug2.log
The above log as a gist
"keep_input_artifact": true
HFS+ filesystem
output-virtualbox-iso/
$ ls -l
total 21220296
-rw-r--r-- 1 tanner staff 10864777216 Mar 5 18:46 WindowsServer2016Docker-disk001.vmdk
-rwx------ 1 tanner staff 10412 Mar 5 18:23 WindowsServer2016Docker.ovf
```shell
$ ls -l *.box
-rw-r--r-- 1 tanner staff 10729788371 Mar 5 18:56 windows_2016_docker_virtualbox.box
```shell
$ tar -tvf windows_2016_docker_virtualbox.box
-rw-r--r-- 0 tanner staff 2286 Mar 5 18:51 Vagrantfile
-rw-r--r-- 0 tanner staff 0 Mar 5 18:51 WindowsServer2016Docker-disk001.vmdk
-rw-r--r-- 0 tanner staff 10412 Mar 5 18:51 box.ovf
-rw-r--r-- 0 tanner staff 26 Mar 5 18:51 metadata.json
The tar file is created internally, could we be hitting a file size limit in packer (or golang)?
The packer build file: windows_2016_docker.json.
@StefanScherer
Oh, I can repro the problem on macOS 10.13.3 and APFS, Packer 1.2.1, VirtualBox 5.2.8
$ packer build --only=virtualbox-iso windows_2016_docker.json
$ ls -l windows_2016_docker_virtualbox.box
-rw-r--r-- 1 stefan staff 10146394489 Mar 8 12:16 windows_2016_docker_virtualbox.box
$ tar tvf windows_2016_docker_virtualbox.box
-rw-r--r-- 0 stefan staff 2286 Mar 8 12:16 Vagrantfile
-rw-r--r-- 0 stefan staff 0 Mar 8 12:16 WindowsServer2016Docker-disk001.vmdk
-rw-r--r-- 0 stefan staff 10411 Mar 8 12:16 box.ovf
-rw-r--r-- 0 stefan staff 26 Mar 8 12:16 metadata.json
I have retried with an older Packer version 1.1.3, still macOS 10.13.3 and APFS, VirtualBox 5.2.8 and the box file looks fine afterwards:
$ ls -l *box
-rw-r--r-- 1 stefan staff 10817278351 Mar 11 14:28 windows_2016_docker_virtualbox.box
$ tar tvf windows_2016_docker_virtualbox.box
-rw-r--r-- 0 501 20 2286 Mar 11 14:27 Vagrantfile
-rw-r--r-- 0 501 20 10952051712 Mar 11 14:27 WindowsServer2016Docker-disk001.vmdk
-rw-r--r-- 0 501 20 10412 Mar 11 14:27 box.ovf
-rw-r--r-- 0 501 20 26 Mar 11 14:27 metadata.json
~/code/packer-windows on my*
What might be the difference between Packer 1.1.3 and Packer 1.2.1?
Is the func DirToBox significantly different in 1.2.1?
Sorry, not very good with git, so I don't know how to pull up that code like you did with
A git blame shows no changes since packer 1.1.1 in packer/post-processor/vagrant/util.go.
Packer 1.1.3 is compiled with Go 1.9, Packer 1.2.1 is compiled with Go 1.10.
It really looks like a problem with Golang 1.10.
I've extracted the packer/post-processor/vagrant/util.go file into a small main.go calling the DirToBox
function without the packer.Ui.
Compiling that code with Golang 1.9 the created tar file looks good.
Compiling that code with Golang 1.10 the tar file shows a zero byte file.
The problem also occurs with compression level = 0, so it's independent from the pgzip package.
The sample source dirtobox.go
, the Dockerfile
and build.sh
script to build the Darwin binary can be found in this gist: https://gist.github.com/StefanScherer/c55fd0ae752687c806f8d5fb40cadc78
I have created a sample in
directory with a big file, compiled the dirtobox
binary for Darwin and run the test
$ ./build.sh
$ ls -l in
total 21509960
-rw-r--r-- 1 stefan staff 11012556800 Mar 12 15:11 abigfile.tar
-rw-r--r-- 1 stefan staff 6 Mar 12 15:08 asmallfile.txt
$ ./dirtobox
2018/03/12 15:35:02 Turning dir into box: in => out.tar
2018/03/12 15:35:02 Skipping directory 'in' for box 'out.tar'
2018/03/12 15:35:02 Box add: 'in/abigfile.tar' to 'out.tar'
Compressing: %s abigfile.tar
2018/03/12 15:35:32 Box add: 'in/asmallfile.txt' to 'out.tar'
Compressing: %s asmallfile.txt
$ ls -l out.tar
-rw-r--r-- 1 stefan staff 11012560384 Mar 12 15:27 out.tar
$ tar tvf out.tar
-rw-r--r-- 0 stefan staff 0 Mar 12 15:11 abigfile.tar
-rw-r--r-- 0 stefan staff 6 Mar 12 15:08 asmallfile.txt
Changing the Dockerfile
back to golang:1.9
FROM golang:1.9
WORKDIR /go/src/app
COPY . .
RUN go get -d -v ./...
RUN GOOS=darwin go build dirtobox.go
and running ./build.sh
and ./dirtobox
again produces a correct tar file:
$ ls -l out.tar
-rw-r--r-- 1 stefan staff 11012559360 Mar 12 15:37 out.tar
$ tar tvf out.tar
-rw-r--r-- 0 501 20 11012556800 Mar 12 15:11 abigfile.tar
-rw-r--r-- 0 501 20 6 Mar 12 15:08 asmallfile.txt
It seems to happen with files > 8GByte.
Time to open an issue over at golang ?
Excellent triaging @StefanScherer
I ran into a very similar issue with Vagrant today - see HERE.
Long and short the tar utility embedded with Vagrant 2.0.3 is having problems unpacking archives with large files - when using bsdtar to extract the files from the box tar archive I end up with a very slimline vmdk of zero bytes! Smaller boxes don't cause any issues.
Note that the system tar has no issues extracting the files from the same box/tar archive...
$ /opt/vagrant/embedded/bin/bsdtar --version
bsdtar 3.2.2 - libarchive 3.2.2 zlib/1.2.11 liblzma/5.2.3
$ tar --version
tar (GNU tar) 1.29
@StefanScherer - Are you sure it's not the tar version you are using that's causing the issues?
@StefanScherer Just for reference:
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.11.6
BuildVersion: 15G19009
$ go version
go version go1.10 darwin/amd64
$ packer version
Packer v1.2.2-dev...
Thanks @DanHam for the heads-up.
There may be a problem with bsdtar on macOS.
The problem in Packer seems to be with Golang 1.10. It obviously creates different tar files between Golang 1.9 and 1.10.
I used another Go binary from https://github.com/mholt/archiver/releases to extract the tar files created with the dirtobox sample code above compiled with Golang 1.9 and 1.10.
The archiver can extract the file from the tar file created with the dirtobox Golang 1.9 binary.
And the archiver just wrote an empty file extracting the tar file created with the dirtobox Golang 1.10 binary.
No tar / bsdtar binary was included in that test.
A test with the embedded bsdtar of my Vagrant 2.0.2 also can extract big files
$ /opt/vagrant/embedded/bin/bsdtar tvf out-go1.9.tar
-rw------- 0 501 20 10737418240 Mar 23 23:02 10Gigfile
$ /opt/vagrant/embedded/bin/bsdtar xvf out-go1.9.tar
x 10Gigfile
$ ls -l 10Gigfile
-rw------- 1 stefan staff 10737418240 Mar 23 23:02 10Gigfile
$ /opt/vagrant/embedded/bin/bsdtar tvf out-go1.10.tar
-rw------- 0 stefan staff 0 Mar 23 23:02 10Gigfile
$ /opt/vagrant/embedded/bin/bsdtar xvf out-go1.10.tar
x 10Gigfile
$ ls -l 10Gigfile
-rw------- 1 stefan staff 0 Mar 23 23:02 10Gigfile
@StefanScherer Ah... OK. Thought we had that one explained away! Good luck tracking down the bug...
I've been having the issue recently, but am unable to make any headway with reverting to Packer 1.1.3. I don't suppose you have any other ideas?
(Issue is documented in https://github.com/Parallels/vagrant-parallels/issues/319, as I thought it was a plugin issue!)
So investigating some more;
I reverted to Packer 1.1.2 on OSX 10.12, and that managed to produce a stable Box I could use.
Testing with Packer 1.1.2 on OSX 10.13 and that fails for some reason? The harddrive says it's 0
:
$ tar -tvf sdk-mac.box
-rw-r--r-- 0 user group 180 27 Mar 12:57 Vagrantfile
-rw-r--r-- 0 user group 25 27 Mar 12:57 metadata.json
-rw-r--r-- 0 user group 65536 27 Mar 12:56 sdk-mac.pvm/NVRAM.dat
-rw-r--r-- 0 user group 7224 27 Mar 12:56 sdk-mac.pvm/VmInfo.pvi
-rw-r--r-- 0 user group 26381 27 Mar 12:56 sdk-mac.pvm/config.pvs
-rw-r--r-- 0 user group 1710 27 Mar 12:56 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r-- 0 user group 0 27 Mar 12:56 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r-- 0 user group 0 27 Mar 12:57 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r-- 0 user group 8816 27 Mar 12:57 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh
So definitely looks like it's a packer issue somewhere... how do you go about debugging this, as I really need to get vagrant working for these machines!
FYI. There is a new packer and golang from homebrew today for Sierra. And new packer for High Sierra.
@basictheprogram Thanks for the update!
I've updated them both to the latest Packer available on Homebrew (1.2.2)
I'm not sure what else could be causing this issue apart from the OS difference?
What's weird is that running $ tar .box
shows that for both OSs think that the hard disk has a size of 0
... but doing vagrant up
works fine for the Sierra one. (They are both running Vagrant 2.0.3).
Sierra:
$ tar tvf sdk-mac.box
-rw-r--r-- 0 user group 180 27 Mar 15:25 Vagrantfile
-rw-r--r-- 0 user group 25 27 Mar 15:25 metadata.json
-rw-r--r-- 0 user group 65536 27 Mar 15:23 sdk-mac.pvm/NVRAM.dat
-rw-r--r-- 0 user group 7224 27 Mar 15:23 sdk-mac.pvm/VmInfo.pvi
-rw-r--r-- 0 user group 26385 27 Mar 15:23 sdk-mac.pvm/config.pvs
-rw-r--r-- 0 user group 1710 27 Mar 15:23 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r-- 0 user group 0 27 Mar 15:23 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r-- 0 user group 0 27 Mar 15:25 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r-- 0 user group 6784 27 Mar 15:25 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh
High Sierra:
$ tar tvf sdk-mac.box
-rw-r--r-- 0 user group 180 27 Mar 15:29 Vagrantfile
-rw-r--r-- 0 user group 25 27 Mar 15:29 metadata.json
-rw-r--r-- 0 user group 65536 27 Mar 15:29 sdk-mac.pvm/NVRAM.dat
-rw-r--r-- 0 user group 7224 27 Mar 15:29 sdk-mac.pvm/VmInfo.pvi
-rw-r--r-- 0 user group 26381 27 Mar 15:29 sdk-mac.pvm/config.pvs
-rw-r--r-- 0 user group 1710 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r-- 0 user group 0 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r-- 0 user group 0 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r-- 0 user group 7728 27 Mar 15:29 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh
On Sierra upgrading packer installed go.
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.12.6
BuildVersion: 16G1212
$ go version
go version go1.10 darwin/amd64
On High Sierra upgrading packer did not install go
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.13.3
BuildVersion: 17D102
$ which go
$ ls -l /usr/local/bin/go*
ls: /usr/local/bin/go*: No such file or directory
I manually installed go and things are working for me now.
I need to confirm all of this on other installations.
Packer 1.2.2 binary for macOS (https://releases.hashicorp.com/packer/1.2.2/packer_1.2.2_darwin_amd64.zip) is compiled with Golang 1.8.1. I assume it should work again.
Homebrew Packer 1.2.2 is compiled with Golang 1.10
Man, why do they have to compile binaries when binaries are already deployed by HashiCorp?
Awesome investigating, folks. It sounds to me like this is a bug in Homebrew rather than a bug in Packer -- would you all agree? Should I close this ticket?
I think the bug is in Golang and the packer is compiled with a buggy Golang. Hashicorp should compile their packer with Golang 1.10.
Sounds like the buggy binary is Homebrew's and the HashiCorp binary works?
Someone should test the HashiCorp 1.2.2 binary. Can test it tomorrow.
I'm gonna recompile and re-release the HashiCorp 1.2.2. binary on golang 1.10 now. We should have that on latest golang.
Hm, but then it probably will be broken as 1.2.1.
Here's the golang-1.8-compiled binary, for those folks who need a workaround:
packer_1.2.2_darwin_amd64.zip . Releasing it compiled with 1.8 was an accident, and just because it fixes this bug doesn't mean it doesn't introduce others.
@basictheprogram My systems already had go
installed when I did the check:
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.12.6
BuildVersion: 16G1212
$ go version
go version go1.10 darwin/amd64
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.13.3
BuildVersion: 17D102
$ go version
go version go1.10 darwin/amd64
I just tested the 1.8 compiled binary of Packer 1.2.2 @SwampDragons posted above:
Sierra: vagrant box fails (to find hard drive)
High Sierra: the vagrant box failed again (same issue)
Testing the default Packer 1.2.2 compiled with 1.10:
Sierra: vagrant box fails (to find hard drive)
High Sierra: the vagrant box failed again (same issue)
The .pvm
file is 16GB it tries to compress, and the output box is 9GB...
I'm beginning to think it has something to do with my system now.
Can anyone confirm 1.2.2 is working, and if so, what setup?
(Weird note; I said earlier that I had it working on Sierra with Brew's Packer, it's now not working...)
@SwampDragons as expected:
docker run -it -v $(pwd):/test -w /test ubuntu tar tvf windows_2016_docker_virtualbox-brew-packer-1.2.2-golang-1.10.box
.@SwampDragons I've been using go 1.10 (installed via MacPorts) to compile my Packer binaries. As stated above, I can create large boxes with out issue...
@SurferL My set up:
$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.11.6
BuildVersion: 15G19009
$ go version
go version go1.10 darwin/amd64
$ packer version
Packer v1.2.2-dev...
Unpacking the box (using GNU tar) from my Windows 2016 build gives me:
.
โโโ [ 340] vmware_desktop
โโโ [ 983] Vagrantfile
โโโ [ 30] metadata.json
โโโ [ 8684] windows2016-vmware-iso.nvram
โโโ [13725794304] windows2016-vmware-iso.vmdk
โโโ [ 0] windows2016-vmware-iso.vmsd
โโโ [ 3515] windows2016-vmware-iso.vmx
โโโ [ 277] windows2016-vmware-iso.vmxf
1 directory, 7 files
This was why I was so convinced (at first) that we were seeing this issue rather than an issue with Packer. However, the work @StefanScherer has done seems to suggest the problem lies elsewhere...
EDIT: Just seen @StefanScherer 's comment above... so this could be the same issue albeit with some additional info provided by Stefan?
This might have been fixed in https://golang.org/doc/devel/release.html#go1.10.minor
I can see a fix for archive/zip in 1.10.1. I've tried with a stripped down sample code, but there is still the problem. It also occurs with a Linux Golang binary. The macOS tar (bsdtar) cannot read the file size correctly.
I've created https://github.com/golang/go/issues/24599
Here is packer built with go 1.10.1 for darwin. Would appreciate any help in testing if this solves the problem.
Thanks @StefanScherer for filing the upstream bug. There's a ton of useful information in it
What can you do?
Go1.10 gives the user finer control over the output format. You can see tar.Header.Format to tar.FormatGNU to work around the libarchive bug.
That seems like it should be it. If anyone would like to submit a PR, please do, otherwise we will add the workaround for the next point release. Compiling packer with an older version of go should be a viable workaround for now.
Will do after some tests.
Afraid I'm still getting the same error with Packer built with go 1.10.1 for Darwin which @mwhooker provided... the .box
generated at the end contains an empty hard disk supposedly for box size of 12GB
$ tar -tvf sdk-mac.box
-rw-r--r-- 0 user group 180 3 Apr 11:26 Vagrantfile
-rw-r--r-- 0 user group 25 3 Apr 11:26 metadata.json
-rw-r--r-- 0 user group 65536 3 Apr 11:25 sdk-mac.pvm/NVRAM.dat
-rw-r--r-- 0 user group 7224 3 Apr 11:25 sdk-mac.pvm/VmInfo.pvi
-rw-r--r-- 0 user group 26381 3 Apr 11:25 sdk-mac.pvm/config.pvs
-rw-r--r-- 0 user group 1710 3 Apr 11:25 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r-- 0 user group 0 3 Apr 11:25 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r-- 0 user group 0 3 Apr 11:26 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r-- 0 user group 8336 3 Apr 11:26 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh
(and running vagrant up
fails with the hard disk error)
I'm also still getting the (vagrant up
hard disk) error using Packer 1.1.3
, so I'm not sure if it's something with my system?
Even though the hard disk seems to be initialised here?
$ tar -tvf sdk-mac.box
-rw-r--r-- 0 501 20 180 3 Apr 12:24 Vagrantfile
-rw-r--r-- 0 501 20 25 3 Apr 12:24 metadata.json
-rw-r--r-- 0 501 20 65536 3 Apr 12:23 sdk-mac.pvm/NVRAM.dat
-rw-r--r-- 0 501 20 7224 3 Apr 12:23 sdk-mac.pvm/VmInfo.pvi
-rw-r--r-- 0 501 20 26381 3 Apr 12:23 sdk-mac.pvm/config.pvs
-rw-r--r-- 0 501 20 1710 3 Apr 12:23 sdk-mac.pvm/harddisk.hdd/DiskDescriptor.xml
-rw-r--r-- 0 501 20 0 3 Apr 12:23 sdk-mac.pvm/harddisk.hdd/harddisk.hdd
-rw-r--r-- 0 501 20 19503513600 3 Apr 12:24 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds
-rw-r--r-- 0 501 20 7472 3 Apr 12:24 sdk-mac.pvm/harddisk.hdd/harddisk.hdd.drh
Have you tried packer from brew?
@basictheprogram Yeah, I've tried the latest 1.2.2, as well as going back to 1.1.3 with Brew but they both didn't work with the same issues above
@SurferL Are you using vagrant box add...
? I'm pretty sure that Vagrants embedded version of tar will cause these issues irrespective of the fix that went in to Packer. See my comment here.
Long and short you may need to manually unpack the box into the right location in the .vagrant.d... directory using GNU tar rather than bsdtar until the next version of Vagrant is released.
@DanHam Thanks for pointing that out - I was not using vagrant box add
, and this made my build work with Packer 1.1.3
!
Just some more info:
Using Packer 1.2.2
from brew (and using vagrant box add
did not work), nor did using Packer built with go 1.10.1 for Darwin (and using vagrant box add
)...
I'm going to lock this issue because it has been closed for _30 days_ โณ. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Most helpful comment
A git blame shows no changes since packer 1.1.1 in packer/post-processor/vagrant/util.go.
Packer 1.1.3 is compiled with Go 1.9, Packer 1.2.1 is compiled with Go 1.10.
It really looks like a problem with Golang 1.10.
I've extracted the packer/post-processor/vagrant/util.go file into a small main.go calling the
DirToBox
function without the packer.Ui.Compiling that code with Golang 1.9 the created tar file looks good.
Compiling that code with Golang 1.10 the tar file shows a zero byte file.
The problem also occurs with compression level = 0, so it's independent from the pgzip package.
The sample source
dirtobox.go
, theDockerfile
andbuild.sh
script to build the Darwin binary can be found in this gist: https://gist.github.com/StefanScherer/c55fd0ae752687c806f8d5fb40cadc78I have created a sample
in
directory with a big file, compiled thedirtobox
binary for Darwin and run the testChanging the
Dockerfile
back to golang:1.9and running
./build.sh
and./dirtobox
again produces a correct tar file: