Packer: Rewrite shell-local post-processor

Created on 1 Jul 2016 · 15Comments · Source: hashicorp/packer

Post-processors in Packer operate on an artifact from the build phase of a packer run. The artifact may be a local virtual machine image, an identifier from a remote API, or a loose collection of files from artifice.

Today the shell-local post-processor operates on files, not artifacts. The result of this design is that for builds that produce remote artifacts, like AWS, the shell-local post-processor doesn't work. Also, if the artifact has multiple files this results in multiple invocations of the script.

Overall this behavior is very confusing and most people expect the shell-local post-processor to operate more like a lifecycle hook that runs at the end of the build.

There is more context in #3669 but I have had this conversation with 5 different people this week alone, so clearly this is broken and we need to fix it!

enhancement post-processoshell-local

Source

cbednarski

Most helpful comment

Just stumbled over this while surfing and now I finally realize why it seems like shell-local provisioners are simply not run (no local artifact). I doubt I'm alone when I'm thinking that the shell-local provisioner should be run once per build, regardless of what artifacts have been produced.

Jalle19 on 3 Jul 2016

👍3

All 15 comments

So what you want to change?

vtolstov on 2 Jul 2016

👍1

Jalle19 on 3 Jul 2016

👍3

@cbednarski do you know if any work has been started on this? I'd be happy to get a PR going if only for discussion 😁

meatballhat on 3 Aug 2016

I doubt I'm alone when I'm thinking that the shell-local provisioner should be run once per build, regardless of what artifacts have been produced.

I'm having to create fake artifacts for using with shell-local. Here's how it looks:

{
  "provisioners": [
<skipped main provisioners>
    {
      "type": "shell",
      "inline": ["date > /tmp/dummy.txt"]
    },
    {
      "type": "file",
      "source": "/tmp/dummy.txt",
      "destination": "dummy.txt",
      "direction": "download"
    }
  ],
  "post-processors": [
<skipped main post-processors>
    [
      {
        "type": "artifice",
        "files": [
          "dummy.txt"
        ]
      },
      {
        "type": "shell-local",
        "script": "script/last_post_processor_running_only_once.sh"
      }
    ]
  ]
}

This workaround was suggested by Hashicorp support and works well so far.

Would definitely love to see it working without fake artifacts.

iroller on 3 Aug 2016

👍2

@iroller samesies, which is how I landed here 😺 https://github.com/travis-ci/packer-templates/blob/6969956e44f9d86724cdfe3b455b0cd74941f665/ci-connie.yml#L103-L122

The fake artifacts are one part. Keeping secret env vars secret is another. Overall, I'd be happy if post-processor/shell-local behaved _mostly_ like provisioner/shell-local, including having same/similar config fields.

meatballhat on 3 Aug 2016

I want to take this, but need more info about design, what i need to pass to the script? @Cbeck527 @rickard-von-essen ?

vtolstov on 16 Aug 2016

@cbednarski agreed that it would be more universally useful to execute once rather than once per file (or artifact). It's also unfortunate that it's focused specifically on files. If we're talking about useful changes to the behavior, how about supplying a JSON array of all structured artifact data to the script via stdin?

jeremy-asher on 19 Aug 2016

@jeremy-asher I think that creates a significant limitation since most applications you might want to run afterward will not expect that. However, you can accomplish something very similar using the manifest post-processor (see master for a reference).

Sorry about the radio silence in general. I will try to update with a spec soon so contributors can take a stab at this.

cbednarski on 19 Aug 2016

🎉1

Yeah, bit of an extreme idea. The manifest post-process definitely does the trick! Nice!

jeremy-asher on 19 Aug 2016

jotting some notes

if you want to run scripts against all of the generated artifacts, use the manifest and then shell post-processor and read the manifest file
we'll rewrite post-processor so that it runs once per builder. This is how the other post-processors work. can use filter to run only for specific builders

mwhooker on 17 Nov 2016

@mwhooker @cbednarski what args we need to pass to shell script in shell-local post processor ?
In previous version we pass artifact file, but now as i understand we no need to pass anything? Or i'm wrong?

vtolstov on 21 Nov 2016

right, no arguments.

mwhooker on 21 Nov 2016

2016-11-21 21:33 GMT+03:00 Matthew Hooker [email protected]:

right, no arguments.

So if you have no objections, please check my pr about shell-local
post-processor

Vasiliy Tolstov,
e-mail: v.[email protected]

vtolstov on 22 Nov 2016

closed via #4189

mwhooker on 8 Dec 2016

Awesome!

On Dec 8, 2016 21:41, "Matthew Hooker" notifications@github.com wrote:

Closed #3671 https://github.com/mitchellh/packer/issues/3671.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mitchellh/packer/issues/3671#event-887461962, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAiCgyfyKODIS1WlCTGN1lRuyDh31qKcks5rGGt1gaJpZM4JDYqX
.