Packer: [RFC] Deprecate Shell-Local postprocessor

[SwampDragons]

The shell-local post-processor will never be as seamless or full-featured as running shell scripts locally after the build is done. The main reason for having this post-processor is being able to access packer variables in the post-processing script, but this is easily handled by using the manifest post-processor (https://www.packer.io/docs/post-processors/manifest.html). Rather than spending energy on a redundant feature, let's remove it and make sure that all use cases that the shell-local post-processor handled have workarounds.

TODO:

• [x] Add deprecation warning to v 1.1.0
• [ ] Write blog post demonstrating how to migrate off of it
• [ ] better document manifest pp [#5413]
• [ ] Remove processor (or leave it, but document that it is unsupported?)

[Phoenix09]

manifest doesn't work for my use case.
I'm using vagrancy to host my own vagrant boxes.

I'm currently using shell-local to upload the box using curl, I could easily enough write it in PHP, but the output filename for the vagrant box isn't available in the manifest and neither are my user variables, which makes this somewhat difficult to do.

[vtolstov]

Please, not touch shell-local. Its very simple post-processor and i can take it maintanership to me.

[mwhooker]

@vtolstov can you give us some use cases where a script that runs after packer wouldn't suffice? We don't think it fits well with the packer model, but happy to discuss it more here

[vtolstov]

script is useful because you can access some packer variables inside processor.
Personally for me - i'm use it to move built images to destination dir.
Also it useful before checksum post-processor, also it can be used to do many thinkg without need to do bash stuff.
I'm use packer inside docker (via drone ci) for building cloud images. Sometimes i need to run checkers after build complited and it very easy with post processor.

shell local brings you ability to do any stuff without writing packer plugins in go.
In case of windows i thins that it easy to modify processor to support windows env.

[Rowern]

I also use the shell-local post-processor to provision some terraform variables.
Mainly, I parse the manifest to create different .auto.tfvars to update the latest available AMI to my terraform.
I can always remove the shell-local post-processor and just use my script. But for me its simpler to have it inside packer and not add some intelligence outside of it.

[PaulAngus]

@mwhooker Isn't using shell scripts outside of Packer kinda missing the point of Packer. I could run Ansible outside of packer, and I can create EC2 or Google Compute instances outside of Packer, but I want to run one thing that brings it all together/one command.

I use shell-scripts to run OVFTOOL to create an OVA file from the vmware workstation output and also create XenServer VHD files from QEMU output.
.. And then push them to our FTP host.

[mwhooker]

@PaulAngus The thinking goes that packer should do one thing and do it well (see the tao of hashicorp). We'll never be able to provide as good an experience as native tools for post-processing, so we should just let you use the proper tools, but empower you to use them effectively.

Additionally, packer is for creating artifacts. Running arbitrary code after the artifact has been created isn't a core part of that. I think packer has done an okay job of being a single entry point to run your entire build, but we're at the point where supporting that workflow is going to be counter productive. It's our opinion that running a script after packer isn't a large burden, and the benefits are worth-while, since it will free us up to focus on other things.

What I'm trying to collect are use-cases where a script that runs after packer isn't tenable. We're still open to community comments, so the more concrete information we get, the better the end product will be.

Hope that helps!

[PaulAngus]

@mwhooker I get you, I want to create appliances for XenServer, VMware workstation, VMware ESXi and disk images for KVM (as it doesn't really do appliances).
There aren't any post processors that will give me OVAs as the final result. It's actually something that i think should natively be there....
so i'm working with shell scripts driving vhd-util and ovf-tool and will probably need a bit of VBoxManage

[cornfeedhobo]

Yeah, this choice makes me a sad panda, and IMHO is being communicated/handled poorly. 1 minor version to deprecate a widely used post-processor?

Further, there is extremely poor documentation about manifest. I have no idea what it outputs. Will that change?
There isn't even a sample output in the documentation. Is every variable that was available through shell-local written out by manifest?

@mwhooker I love the tao of hashicorp and can get on board with your logic here (I already use Makefiles), but you guys consistently fall short on verbose documentation and expect users to fill in the gaps. I should not have to build a box, knowing it will be a waste of my time, just to see what manifest does and glue all this stuff together. Please review your timeline and replacement documentation.

[Phoenix09]

@cornfeedhobo here's an example
manifest.txt

although, I've done a lot of builds for vagrant and only once has it included the box output name (every build had a vagrant output)
so for me, it's completely useless

[mwhooker]

Hi folks,

Thanks everyone for your great feedback. It's obvious we moved from the RFC to the implementation phase too quickly. Clearly we need to provide a better path forward before going forward with this. We will leave the shell-local post-processor around for now, but we will probably not be spending too much developer time on it.

I've created a ticket at #5413 to enhance the manifest pp documentation, and if/when we go forward with this, we will make a blog post highlighting our reasoning and how to migrate

[vtolstov]

I don't think that manifest can replace shell-local.

[kquinsland]

Here's my post processor block:

  "post-processors": [
    {
      "type": "manifest",
      "output": "manifest.json",
      "strip_path": true
    },
    {
     "type": "shell-local",
      "only": ["amazon-ebs"],
      "script": "./scripts/share-ami.sh"
    }
  ]

the share-ami.sh script just shares the AMI w/ a few other AWS accounts that need to consume the built AMI. I get the AMI-ID from the manifest.json file, using jq. If you take away my shell-local how will i replace this?

[mwhooker]

@kquinsland I think your invocation of packer would turn into packer packer.json && ./scripts/share-ami.sh and share-ami would quit if it didn't find an AMI id in the manifest file

[kquinsland]

@mwhooker We have some tooling that sits on top of / around packer that make that type of chaining a bit difficult; which is why i currently run the script this way. But, i've been hoping to get a lot of the current tooling re-done in a much simpler way... so this sounds like the perfect argument to do so. thanks :).

[mwhooker]

@kquinsland I'd be really happy to hear your experiences. Just to make sure you saw my note, we're not targeting this for any specific release until we get more feedback from the community. We might just keep things the way they are, but any anecdotes you want to share removing the post-processor will help us understand what's left to be done

[kquinsland]

@mwhooker the tooling we've got right now is designed to set up packer dependencies based on the type of artifact (AMI and docker, primarily) and where the packer job is running (local, ec2 or code build).

a simple use cases look like this:

$ ./build_tool --type AMI --job application_host --app someCodebase --other variables...

in this case, build tool will do some checks to make sure someCodebase has an appropriate AMI to start with, that it's available, determine which branch it should use if it's not specified by the user, which AWS credentials to use to spin up the AWS instance and which IAM profile to assign to the box as it's being built, which SNS topics to send messages to for success / fail, ... etc. This tool also does some sanity checking on the packer logs depending on if this is a local test build or a code-build thing.... this checking is what caught the depreciation warning ;)

We'll still keep the majority of the process, it's just that the tooling has grown "organically" over time through my organization; not structured or planned. I've been angling to break a lot of the glomed on code / process into many smaller pieces for a while now. Even if hashi changes nothing, it's not ideal for me and the developers that i support to have a tool chain that cant easily accommodate the changes we would need to make _if_ hashi did change this

[rbclark]

In my case I use the shell-local processor to run ovftool to build an OVA with the results of my build. Removal of this feature is not impossible to workaround however it is not ideal. Currently packer handles our whole build process and with this change it would no longer be able to do so.

[brettjacobson]

@rbclark I've had good luck with https://github.com/drscream/packer-post-processor-ovftool

[alexhexabeam]

We use the shell-local post processor to modify an output OVF file built with virtualbox to create multiple OVAs for different versions of ESX as well as virtualbox. We make copies of the OVF and VMDK files, use sed to change the hardware in the OVF, then tar it up into an OVA. This saves a massive amount of time and compute over rebuilding it with different builders, not to mention not needing any VMWare infrastructure for the actual build.

While this could be done with a separate script, that would mean increasing the complexity of our script to manage the invocation of packer and/or adding dependencies to parse the manifest file (jq?). This is all doable, but the additional dependencies might make building our OVAs beyond the capabilities of our less technical users.

[sakers]

I use shell-local for assigning permissions and tags to AWS AMI images and setting Terraform variables. Without shell-local my simple:

packer build banana.json

which currently does everything it's supposed to do without any further complications. My users can easily remember such a command, but:

packer build banana.json && ./image_permissions.sh

is more difficult for them to deal with. In fact they might forget to do the 2nd part altogether and that'll waste quite a lot of time.

[nkk1]

I use shell-local for uploading box files to artifactory vargrant cloud.. the "vagrant cloud" post processor doesn't work with artifactory

[SwampDragons]

Closing; clearly enough people care enough about this that it's worth keeping around for now.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.