Packer: Packer file upload shouldn't be part of validation

[brikis98]

Packer v0.12.2

When you use the Packer file provisioner in a template, Packer validates that the source file exists before running the build. This seems like a good idea, except it makes it impossible to use the following technique:

{
  "provisioners": [{
    "type": "shell-local",
    "command": "git clone <URL> /tmp/foo"
  },{
    "type": "file",
    "source": "/tmp/foo",
    "destination": "/tmp"
  }]
}

(I've omitted the builders section from the template above).

The shell-local provisioner creates the /tmp/foo folder locally (in this case, by using git clone), which I then want to upload to the remote server.

Unfortunately, Packer gives the following error:

packer build foo.json

1 error(s) occurred:

* Bad source '/tmp/foo': stat /tmp/foo: no such file or directory

[marema31]

It most of the case it will be the expected behavior, a new option to explicitly ignore this error could be a good idea. Currently we have used a workaround to use an empty file explicitly removed by the shell-local provisionner. I'll try to propose a PR. Any preference on the option's name ?

[mwhooker]

I believe this duplicates #3210

This has definitely been requested before, but we find that using the shell-local provisioner and then uploading the results is often the wrong way to do things. We recommend doing as much work in the container as possible, ensuring that the build doesn't depend on your local environment. See more context here: #2877

I think if we were going to change anything it would be to alter the file provisioner to use go-getter (as in #3118), which would allow you to download git repositories using the file provisioner

[marema31]

This will correspond to my needs. Thanks for the link.

[mwhooker]

I'm going to close this since I don't think we're planning on changing the file upload prepare checks. One thing you can do in the meantime is use directory upload. So git clone to a directory and upload the parent. That way it'll work regardless of the contents of the directory.

@marema31 If you plan to work on the go-getter patch, I think there's a lot you can work off of in #3118. One thing it does which I'm not sure of is remove the stat check. I think creating an entirely new provisioner would be the way to go. Feel free to ping me if you want to chat about it.

[brikis98]

This has definitely been requested before, but we find that using the shell-local provisioner and then uploading the results is often the wrong way to do things. We recommend doing as much work in the container as possible, ensuring that the build doesn't depend on your local environment.

The reason I want to do the git clone on the local machine is because it's a private Git repo. I don't want to have to send any auth details to the EC2 instance just to checkout the code. So a flag that disables the file provisioners stat check would be great.

I think if we were going to change anything it would be to alter the file provisioner to use go-getter (as in #3118), which would allow you to download git repositories using the file provisioner

Assuming that go-getter downloads the code locally and then uploads it to the EC2 instance, I think this would work too!

[marema31]

I think #3891 is just about the same need.

@mwhooker: for you that will imply to create a new provisionner and leave the file provisionner as it is actually ?

[mwhooker]

yeah it looks like #3891 does what this issue suggests

This gets into a philosophical debate. There's some question if the shell-local provisioner is a good thing to have at all. We're thinking that packer should not manipulate the local filesystem at all, and that if you want to do some pre-run work, you should do it in a separate script that runs before packer. The shell-local provisioner is sort of an odd-ball; it doesn't really fit in with our guiding principals.

If you don't generate any local files as part of the packer build, then you don't need to upload files that don't exist before packer runs.

I personally want to keep the way the file provisioner works now for a couple of reasons, which I'm not going to get into right now, but they stem from what I said above.

the new provisioner would be for fetching files on the remote side.

[marema31]

The other usage we have for shell-local is to use external program to call enterprise APIs with informations about the VM to permit the following providers to work. It's never our first choice, but it's a nice to have (like the command module of Ansible or the exec from Puppet)

[marema31]

@mwhooker I can have a look to this provisionner, but I have some difficulty to the workflow. Since the fetching will be done from the remote side, that will necessitate to send go-getter binary via the communicator and execute it with the provided parameters lists. Or I misunderstood the usage of go-getter?

[mwhooker]

The other usage we have for shell-local is to use external program to call enterprise APIs with informations about the VM to permit the following providers to work.

Are you using template variables as part of the shell-local step? That's a good use case that we should keep in mind

I can have a look to this provisionner, but I have some difficulty to the workflow. Since the fetching will be done from the remote side, that will necessitate to send go-getter binary via the communicator and execute it with the provided parameters lists. Or I misunderstood the usage of go-getter?

The go-getter provider would just download files locally. you would still need to upload them with the file provisioner. I think it needs more thought and I wouldn't necessarily work on it right now.

If you really need this to work, and none of the work-arounds are sufficient (like uploading a directory), then I encourage you to build packer with the patch in #3891 applied

[marema31]

Are you using template variables as part of the shell-local step? That's a good use case that we should keep in mind
Yes

The go-getter provider...
No I don't have urgent need on this, I saw that as an opportunity to accelerate the file upload for windows templates we are doing (since file transfer via WinRM is quite slow) but the current solution is OK for us.

[develar]

So strange.

{
    "type": "shell-local",
    "command": "git archive master -o .app.tar.gz"
},

What's bad? Why packer forced me to use yet another preparation script before...