Packer: Add support for building from an existing Vagrant box

[aseigneurin]

It's quite complex to build a Vagrant box from an ISO file. You need to know the boot sequence and the preseed. Both of these are hardly readable.

Sometimes, we just need to build a box out of an existing Vagrant box. It would be great if Packer had a pre-processor that would take, in input, a Vagrant box, and would then feed the "virtualbox-ovf" builder. Alternatively, this could be another builder.

That would add the ability to provision a pre-built Vagrant box, then repackage it as a Vagrant box.

Ideally, the Vagrant box in input could be taken from the local context or from a URL (e.g. a box from http://www.vagrantbox.es/).

Does it make sense?

[apetresc]

+1

[jorisvddonk]

For Virtualbox, you can currently achieve this manually by downloading a .box, extracting its contents (they're just tarballs, optionally gzipped) and then pointing the source_path of the virtualbox-ovf builder to the box.ovf file that was extracted from the tarball.

I agree though that having actual support for Vagrant boxes inside Packer without requiring above workaround would be really useful.

[ryanbreen]

+1. This would remove a lot of friction from the following workflow: take an existing Vagrant box, update contents, and update deployed box so other users can benefit.

[vampirical]

+1. Would definitely be convenient to have.

[andreisavu]

+1

[riedi]

+1 - i was also looking for that feature - just want to make small changes to an existing base box.

[tchemineau]

+1
Agree with @aseigneurin

[ckulla]

+1

[CpuID]

+1

Nathan Sullivan
Sent from a mobile device

On 25 Jul 2014, at 1:29 am, Christoph Kulla [email protected] wrote:

+1

—
Reply to this email directly or view it on GitHub.

[michaelmior]

+1 Seems like it would be fairly straightforward to do this for VirtualBox boxes as @jorisvddonk suggests. My guess would be most people who want this are aiming to do it for the VirtualBox Vagrant provider anyway.

[peterfox]

+1 this would make it easier as a lot of the time I just want to take a box and then add simple things like chef etc. rather than do a full install of the OS from an ISO

[Jim-Duke]

+1

There are issues in that a Vagrant Box is not necessarily intended for the VirtualBox provider and may not contain an OVF/OVA file. But for all those who want this workflow that wouldn't be a problem.

[mattiasholmqvist]

+1

[kuboj]

+1

[haad]

+1

Ability to skip installation process would be great.

[d3netxer]

+1

[vladistan]

+1

[lantrix]

+1

Consideration to the pre-processor would be what type of vagrant box provider is in use as well; I'm guessing the two most common are VirtualBox and VMWare

[dev-head]

+1 this should happen

[jonyeezs]

+1

I'm sitting here waiting for a windows update....this takes ages.

I know i should do this during off-peak but sometimes you just need to build a new box. And would be lovely if i don't have to do a windows update from an installation.

[ajmath]

+1

[mwpeterson]

I started converting an existing Vagrant workflow to Packer so we can also build AMIs. I was very surprised to discover that Packer doesn't support building directly from virtualboxes on Atlas. :(

:+1:

[amontalban]

+1

I have a FreeBSD vagrant box that I build from ISO that would be great if I can use it as base box for other Packer generated boxes that uses FreeBSD as base so I can have better build times.

[libertyy]

+1

[dragon788]

I may have to take a look at what it would take to manually script this (I don't know any Go). Basically according to the box specification there is a metadata.json file in the .box that explains which provider it was built for, this would allow you to error out if you are trying to "convert" a box to another provider (definitely harder to support). This could support at least Virtualbox and VMware (desktop) fairly easily, I haven't played with the Docker support in Vagrant to see how their boxes are constructed.

[BrendanThompson]

@dragon788 yeah see, that's exactly what I'm looking to do. Is to convert from the default virtualbox provider to vmware_fusion

[dragon788]

I've converted that way, sometimes its better to simply rebuild the box if
possible. You have to do some crazy tricks like importing the box into
Virtualbox, exporting it as an ova/ovf but using the OLD 0.9 version, then
using the VMware OVF Tool to convert it over to VMware (and you have to
tell it to ignore warnings) and then you import it into VMware and if you
are lucky, you can then use that VMDK and feed it into Packer to build a
new "box".

On Wed, Jul 8, 2015 at 1:50 AM, Brendan Thompson [email protected]
wrote:

@dragon788 https://github.com/dragon788 yeah see, that's exactly what
I'm looking to do. Is to convert from the default virtualbox provider to
vmware_fusion

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/869#issuecomment-119459486.

[amontalban]

Hey guys,

I have created a couple of tools to manage boxes, mainly to download a box from a repository and use it as starting point for Packer. This download script allows you to decompress the box and then use the .ovf file inside the box as starting point for the Packer image you are generating.

Take a look at it at https://github.com/amontalban/vagrant-tools and let me know what do you think.

Thanks!

[davidkarlsen]

+1. Today I build my boxes from base-boxes + my provisioning. I do this with vagrant up && halt && package. I could keep my workflow if moving over to packer (which I want in order to have one consolidated packer build/template for docker, vbox ++)

[dbaba]

+1

[russroy]

+1

[rraheja]

+1

[d3viant0ne]

+1 - I have no idea how this doesn't already exist. Seems like a pretty obvious use case.

[spinx]

+1

[ashwanthkumar]

+1

[ghost]

+1 for all of the reasons above and it would allow one to build incrementally similar to the docker model.

[iAnomaly]

+1 This would be a huge help especially considering the OVA/OVFs in Canonical's provided Cloud images are not readable by VirtualBox (at least for Trusty LTS) and therefore unusable by Packer and Vagrant. Known issue: #1726

The OVF/VMDK extracted from their provided Vagrant boxes are usable however.

A pre-processor for Packer to download and extract Vagrant boxes (or more generically: download and extract TARs) would allow us to close this gap.

[JackChance]

+1 I had assumed this already existed.

[newtonlabs]

+1

[chizou]

+1

[tisba]

I would be really interested, if someone from Hashicorp could comment on that. I mean, is this something that is worth thinking about? Feels weird that this is not available yet and no one "official" has commented on that issue for almost two years...

[drewglass]

+1

[dave-newson]

+1 This is also a missing step in my workflow. I want to use a CoreOS AMI and a CoreOS .box and have Packer instigate provisioning for both. The inability to handle the .box is a pain point that requires an ugly script back before Packer in order to get the .ovm out of the .box.

I'm assuming nobody from Hashi has looked at this because it's outside the remit of both the ISO and the OVF drivers, and would require a third driver which is actually specific to vagrant.
But then why would Hashi want to support interop between their own products?

I don't know Go either, but we're all developers, so let's fix this.
I'm thinking we'd need to clone the OVF driver, then before the Import step:
https://github.com/mitchellh/packer/blob/master/builder/virtualbox/ovf/builder.go#L76
You'd integrate something similar to the DownloadStep in the ISO driver:
https://github.com/mitchellh/packer/blob/master/builder/virtualbox/iso/builder.go#L230
You'd also need to bring across the checks from higher up in the ISO driver, too:
https://github.com/mitchellh/packer/blob/master/builder/virtualbox/iso/builder.go#L162

That's the easy stuff, but you'd also need to:

1. Extract the downloaded .box file (tarball)
2. Optionally: Open the metadata.json file, check the provider is virtualbox
3. look for the .ovf (though it's normally "box.ovf", right?)
4. Continue the import process using the newly discovered ovf.

Actually looking at it, it's not that hard. I just don't know how to do it in Go.

[provonet]

+1

[jeqo]

+1

[lifeart]

+1

[tybritten]

+1

[kaluzki]

+1

[Tankanow]

+1

[cburyta]

+1

[rauchrob]

+1

[laupow]

On the bandwagon...+1

[luis-silva]

+1

[athak]

+1

[dragon788]

For those folks requesting this functionality, are you intending to convert
an existing VirtualBox (or VMware) box into an updated VirtualBox (or
VMware) box, or are you intending to convert from a VirtualBox box over to
a VMware box or vice versa?
On Dec 1, 2015 2:04 PM, "Atha Kouroussis" [email protected] wrote:

+1

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/869#issuecomment-161079965.

[athak]

I'm trying to use an existing box as a base for other boxes. Not interested in conversion from Virtualbox to VMware. Currently working around this by keeping artifacts from the base box build but not ideal.

[vincer]

What I'm looking for is to be able to build upon previously built VirtualBox boxes to create new VirtualBox boxes.

With Amazon AMIs, I can have packer definition A start with a base Ubuntu AMI as a source image, then packer definition B start with the AMI build in A, then definition C start with AMI from build B.

With VirtualBox (in Atlas that only supports ISO-based builds) I have to start from the base Ubuntu ISO image from A, B, and C. i.e. B can't use the Virtualbox box generated by A as a "source box".

[wangjunwei87]

+1

[wangjunwei87]

Currently, we can specify the path of local vagrant box file to packer builder source_path.
e.g:

        {
            "type": "virtualbox-ovf",
            "source_path": "/Users/xxx/.vagrant.d/boxes/bento-VAGRANTSLASH-centos-6.7/2.2.2/virtualbox/box.ovf",
            "ssh_username": "vagrant",
            "ssh_password": "vagrant",
            "headless": true,
            "shutdown_command": "echo 'packer' | sudo -S shutdown -P now"
        }

[ravenbyron]

+1

[Maxwolf]

+1

[barabaka]

+1

[RyanBowlby-Reflektion]

+1

I want to use the common/generic bento/centos-7.2 as the starting point in my packer template. Alternatively, once I have a private vagrant in atlas it would be ideal to use the last one as the source for subsequent packer builds. Being as my provisioner is idempotent there's no reason to build the box from an ISO each time.

[sigmonsays]

+1

[dineshbhat]

Currently, we can specify the path of local vagrant box file to packer builder source_path.
This worked pretty for local builds.

+1

[BetaGrumm]

+1

[atonse]

+1 Is anyone working on this? If not, are you soliciting contributions?

[dragon788]

With local/host shell commands coming soon in Packer this will be trivial.
On Mar 2, 2016 12:33 PM, "Ashish Tonse" [email protected] wrote:

+1 Is anyone working on this? If not, are you soliciting contributions?

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/869#issuecomment-191362198.

[smolinari]

I too was wondering, can a premade box be used in Packer to continue to build a Vagrant box off of?

:+1: to making it happen. Also because, it would speed up the process considerably.

As to "local/host shell commands", couldn't it be possible just to have something like

  "builders": [
    {
      "type": "vagrantbox",
      "sourcebox": "hashicorp/precise64",
      "virtualization": "Virtualbox"
    }
  ]

and let Packer do the rest with Atlas, Vagrant and Virtualbox?

Not so sure about the virtualization bit, if Packer can't tell what is available on its own, but I hope you get the gist. :smile:

Scott

[dragon788]

Scott, minor note, you can't have spaces in a json key name. If you had a
local script that unpacked the .box (really just a .tar file) into a
directory structure, that you can parse and reference to do a future build.
This is where the local shell commands will come in handy, otherwise you
need an extra stage before Packer to do the extraction.

On Sat, Mar 5, 2016 at 2:28 AM, Scott [email protected] wrote:

I too was wondering, can a premade box be used in Packer to continue to
build a Vagrant box off of?

[image: :+1:] to making it happen. Also because, it would speed up the
process considerably.

As to "local/host shell commands", couldn't it be possible just to have
something like

"builders": [
{
"type": "vagrantbox",
"source box": "hashicorp/precise64"
"virtualization": "Virtualbox"
}

and let Packer do the rest with Atlas, Vagrant and Virtualbox?

Not so sure about the virtualization bit, if Packer can't tell what is
available on its own, but I hope you get the gist. [image: :smile:]

Scott

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/869#issuecomment-192607673.

[smolinari]

Corrected! LOL!

It still seems Packer should be "smart" enough to do the unpacking, extraction or whatever else is necessary, itself.

Scott

[stefco]

+1

[minimum-hsu]

+1

[paulszego]

+1

[benpriestman]

+1

[gamename]

+1

[elconas]

+1

[ozzyogkush]

+1 this would make life a lot easier!

[bcg62]

+1 need this.

[simonguest]

+1 - this would greatly help speed up builds

[charleswisniewski]

+1 would be very helpful

[BSick7]

+1 super helpful, but even with the workaround, publish an article on how to reach the exact vagrant box images

[ymenager]

+1 would be quite helpful

[raven]

+1
Going from vmware_basebox -> vmware_extra_provisioning box. Doing this all from packer would be awesome. I work with OS X images for iOS building, and repackaging is something you do pretty often to save time.

[w32-blaster]

+1

[conatus]

This would be much appreciated.

[denova]

+1

[oryband]

Hi HashiCorp devs! any chance to get this going? this issue has been open for so long now!

[smolinari]

I've found a much better system. Docker! Bye bye Hashicorp.

Scott

[dragon788]

Vagrant also does Docker, so you don't need to leave . ;)

On Sep 13, 2016 7:36 AM, "Scott" [email protected] wrote:

I've found a much better system. Docker! Bye bye Hashicorp.

Scott

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mitchellh/packer/issues/869#issuecomment-246667118,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdxXvgJ_ek2tvfiHWdtJFYA3vfckH6iks5qppi8gaJpZM4BeIMi
.

[smolinari]

You're joking, right? Edit: I am now using Docker for Windows. Will never look back. Sorry.

Scott

[minimum-hsu]

+1

[james-s-nduka]

Would like to see this feature added +1

[djvdorp]

This is exactly what I am currently missing, so +1

[jb11211]

+1 - Also wanted to say a big Thank You to Hashicorp for giving me such a helpful tool for free. Amazing

[Adelrisk]

+1 the workaround is really brittle.

[andrey-moor]

+1

[Raffaello]

:100:
:+1:
+1

especially it will be handy to try replicating has close as possible some cloud provider images with a full virtualized VM and not a container that it might use even different kernels..

[oberones]

+1

I'm surprised this isn't a thing already. Might be time to hack on some Go

[themalkolm]

I hacked a custom builder packer-builder-vagrant that works for me. It simply finds the existing vagrant box (or tries to download it) and updates source_path in builder definition. Works fine with packer 0.12 + vagrant 1.9.0 + VirtualBox 5.1.8 (tried both mac & windows). It requires you to slightly update your packer template. README should give you a hint how it works. Feel free to try.

https://github.com/themalkolm/packer-builder-vagrant.

[arnaud-dezandee]

+1

[znerd]

+1

[tvnpraveen]

+1

This works as suggested with packer 0.12.2. Here is the builder config I am using

"builders": [ { "type": "virtualbox-ovf", "source_path": "{{userovf_path}}", "headless": true, "boot_wait": "2m", "ssh_username": "vagrant", "ssh_password": "vagrant", "ssh_wait_timeout": "6h", "shutdown_command": "shutdown /s /t 10 /f /d p:4:1 /c \"Packer Shutdown\"", "floppy_files": [ "{{userautounattend}}", "scripts/microsoft-updates.bat", "scripts/win-updates.ps1", "scripts/openssh.ps1" ] }, "variables": { "ovf_path": "./vbox/box.ovf", "autounattend": "answer_files/2012_r2/Autounattend.xml" }

[sebastianreloaded]

Using the unpacked box from https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-vagrant.box doesn't work with packer/virtualbox for me. It always hangs after “non blocking pool is initialized”. Anybody has the same problem?
To clarify, using the box with vagrant works fine.

[srggavrilov]

@sebastianreloaded try to add serial device. And you probably also need to change first NIC type to NAT:

        "vboxmanage": [
          [ "modifyvm", "{{ .Name }}", "--uart1", "0x3F8", "4" ],
          [ "modifyvm", "{{ .Name }}", "--uartmode1", "disconnected" ],
          [ "modifyvm", "{{ .Name }}", "--nic1", "nat" ]
        ]

Anyway, I'd recommend to stick with bento boxes, since after you're successfully booted you'll have to solve login issue (ubuntu uses random password for its default 'ubuntu' user and it is being changed from version to version) and who knows what else. I dropped this way after 2 days of fighting with ubuntu images.

[cornfeedhobo]

@themalkolm thank you very much for attempting to solve this. Unfortunately your solution is highly targeted at the VirtualBox provider, and doesn't support things like variables in the builder.

[darkn3rd]

It seems obvious that many are surprised a Hashicorp technology doesn't support Hashicorp formats in the scope of Vagrant boxes. With others, I am also interested in support of a pipeline similar to AMI builder, to use an existing image and modify it.

Most would prefer not to build a base box each time only to add some small component, such as a software version bump. It forces the workflow to be locked to an _golden image_ pattern, where you get all or nothing, when you only want to change a subset of the whole. This ultimately adds friction toward implementing _immutable production_.

[gvenzl]

+1

This functionality would make the cross usage of Vagrant and Packer a lot better!

[arnaud-dezandee]

Here is a bash script to use a vagrant box as source for virtualbox builder:

build.sh

#!/bin/bash
set -e

vagrant box add --provider virtualbox ubuntu/xenial64 2> /dev/null || true
BOX_VERSION="$(ls -1 ~/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-xenial64 | grep ^20 | sort -r | head -1)"
BOX_FOLDER="$(realpath ~/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-xenial64/${BOX_VERSION}/virtualbox)"
BOX_CHECKSUM="$(sed -n 's/.*ovf.*= \(.*\)/\1/p' ${BOX_FOLDER}/ubuntu-xenial-16.04-cloudimg.mf)"
BOX_BASE_MAC="$(sed -n 's/.*base_mac = "\(.*\)"/\1/p' ${BOX_FOLDER}/Vagrantfile)"

packer build \
  -var "box_folder=${BOX_FOLDER}" \
  -var "box_checksum=${BOX_CHECKSUM}" \
  -var "box_base_mac=${BOX_BASE_MAC}" \
  "template.json"

template.json

{
  "builders": [
    {
      "type": "virtualbox-ovf",
      "source_path": "{{ user `box_folder` }}/box.ovf",
      "checksum": "{{ user `box_checksum` }}",
      "checksum_type": "sha256",
      "guest_additions_mode": "disable",
      "headless": true,
      "ssh_pty": true,
      "ssh_username": "vagrant",
      "ssh_private_key_file": "~/.vagrant.d/insecure_private_key",
      "shutdown_command": "sudo shutdown -P now",
      "output_directory": "/tmp/packer_output",
      "target_path": "/tmp/packer_cache",
      "skip_export": true,
      "vboxmanage": [
        [ "modifyvm", "{{ .Name }}", "--ostype", "Ubuntu_64" ],
        [ "modifyvm", "{{ .Name }}", "--cpus", "2" ],
        [ "modifyvm", "{{ .Name }}", "--memory", "2048" ],
        [ "modifyvm", "{{ .Name }}", "--vrde", "off" ],
        [ "modifyvm", "{{ .Name }}", "--nic1", "nat" ],
        [ "modifyvm", "{{ .Name }}", "--uart1", "0x3F8", "4" ],
        [ "modifyvm", "{{ .Name }}", "--uartmode1", "file", "/tmp/packer_output/console.log" ],
        [ "modifyvm", "{{ .Name }}", "--macaddress1", "{{ user `box_base_mac` }}" ]
      ]
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": ["echo foo"]
    }
  ]
}

[helloIAmPau]

+1

[chirangaalwis]

+1 for introducing this feature. Will simplify a lot of work.

[jamesla]

+1

[mat-mcloughlin]

+1

[alaa]

+1

[lqueryvg]

I've +1'd this request.
But until this feature is available, what's wrong with @davidkarlsen suggestion ?
I.e. use existing vagrant functionality:

1. vagrant up (from base .box)
2. vagrant provision (add your customisations here)
3. vagrant halt
4. vagrant package -> create a new .box

Are there any gotchas with this approach ?

[chirangaalwis]

@lqueryvg you may want to look into WSO2's product Vagrant box repository [1], which was recently initiated. In this repository, the approach which you have pointed out has been used.

When working on building Vagrant boxes for WSO2 products, we encountered the same issue discussed here and we came up with the same solution. Yet hopefully, this feature would be introduced sooner rather than later.

[filipelenfers]

+1

[rockraft7]

+1

[hdlim15]

+1 @KFoxder

[grv87]

@cornfeedhobo, what did you mean saying that @themalkolm's solution doesn't support variables?
He uses variables in tests, and everything looks OK.

[jarrettj]

+1

[koaps]

+1

[Swader]

5 years after the issue was first opened, let me throw in a +1 as well.

[SwampDragons]

I've implemented an integration for the virtualbox-ovf builder already and for the vmware-ovf one, but I'm not wild about the complexity it adds to the code, and I think this would work better as a standalone builder -- so I'll be working on that next, with the Vagrant team.

I'd love volunteers to test builds once I've got something working.

[koaps]

I'd love volunteers to test builds once I've got something working.

I'm game

[Swader]

Also willing to test while in progress, feel free to ping me whenever

[SwampDragons]

I have a rough builder created here

So far thanks to some hardcoding it only works on OSX (I've attached a binary to that PR) but I intend to fix the pathing for linux and windows tomorrow.

I'd love some testing and feedback. Docs changes are included in the PR, and there's a usage example in the PR comments.

[SwampDragons]

Windows and linux builds are now available for the vagrant builder.

[sudomateo]

+1 for this. Also voicing the need/want for this to work via libvirt and not only VirtualBox and VMware.

[SwampDragons]

@sudomateo the new vagrant builder will call out directly to vagrant rather than requiring users to set up the underlying builder; you will have the opportunity to set whichever vagrant provider suits your needs.

[nhojpatrick]

packer build \ -var "box_folder=${BOX_FOLDER}" \ -var "box_checksum=${BOX_CHECKSUM}" \ -var "box_password=${BOX_PASSWORD}" \ -var "box_base_mac=${BOX_BASE_MAC}" \ "templates.json"

As this is still open as I just want to take official vagrant boxes, execute some ansible playbooks, then create a new vagrant box or personal usage...

@Adezandee Do you have an updated version as I've just tried with packer v1.3.4 and get this;

Failed to parse template: 14 error(s) occurred:

* Unknown root level key in template: 'checksum'
* Unknown root level key in template: 'checksum_type'
* Unknown root level key in template: 'guest_additions_mode'
* Unknown root level key in template: 'headless'
* Unknown root level key in template: 'output_directory'
* Unknown root level key in template: 'shutdown_command'
* Unknown root level key in template: 'skip_export'
* Unknown root level key in template: 'source_path'
* Unknown root level key in template: 'ssh_password'
* Unknown root level key in template: 'ssh_pty'
* Unknown root level key in template: 'ssh_username'
* Unknown root level key in template: 'target_path'
* Unknown root level key in template: 'type'
* Unknown root level key in template: 'vboxmanage'

[arnaud-dezandee]

@nhojpatrick my original comment was a partial packer file. I just updated it with a complete working example.

[SwampDragons]

@nhojpatrick that functionality will be added when #7221 is merged :)

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.