Packer: Permission denied running plugin since packer 1.5.0

[gvcgael]

Overview of the Issue

Since Packer 1.5.0, we could not run vsphere-clone and vsphere-iso plugins anymroe.

Reproduction Steps

Install vsphere plugins https://github.com/jetbrains-infra/packer-builder-vsphere

Run a build.json using these plugins

Packer version

Packer 1.5.0

Operating system and Environment details

Linux 64 bits

Log Fragments and crash.log files

$ PACKER_LOG=1 packer.io -only=vsphere-clone build.json
2019/12/20 11:24:36 [INFO] Packer version: 1.5.0 [go1.13.5 linux amd64]
2019/12/20 11:24:36 [DEBUG] Discovered plugin: vsphere-clone = /usr/local/bin/packer-builder-vsphere-clone.linux
2019/12/20 11:24:36 [DEBUG] Discovered plugin: vsphere-iso = /usr/local/bin/packer-builder-vsphere-iso
2019/12/20 11:24:36 using external builders [vsphere-clone vsphere-iso]
2019/12/20 11:24:36 Attempting to open config file: /home/user/.packerconfig
2019/12/20 11:24:36 [WARN] Config file doesn't exist: /home/user/.packerconfig
2019/12/20 11:24:36 Setting cache directory: /home/user/centos7_http/packer_cache
2019/12/20 11:24:36 Plugin could not be found at  (exec: "": executable file not found in $PATH). Checking same directory as executable.
2019/12/20 11:24:36 Current exe path: /usr/local/bin/packer.io
2019/12/20 11:24:36 Creating plugin client for path: /usr/local/bin
2019/12/20 11:24:36 Starting plugin: /usr/local/bin []string{"/usr/local/bin"}
Failed to initialize build 'vsphere-clone': error initializing builder 'vsphere-clone': fork/exec /usr/local/bin: permission denied
2019/12/20 11:24:36 [INFO] (telemetry) Finalizing.
2019/12/20 11:24:38 [WARN] (telemetry) Error finalizing report. This is safe to ignore. Post https://checkpoint-api.hashicorp.com/v1/telemetry/packer: context deadline exceeded
2019/12/20 11:24:38 waiting for all plugin processes to complete...
make: *** [Makefile:22: vsphere_validate] Error 1
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/games:/home/user/.local/bin:/home/user/.local/bin

[azr]

Hey @gvcgael thanks for opening, This wont solve your issue but I think the rights on the binary are incorrect. Have you tried chmod +x after installing it ?

If you fix this this still won't work because the code of the vsphere builder needs to be updated to work with packer 1.5: there was a bunch of breaking changes in 1.5. The vsphere builder is currently being merged in Packer in #8480 and we miss a bunch if CLA signature before we can do so. Only then you will be able to use it with Packer 1.5.

In the meantime I recommend using an old version of Packer that worked for you.

[gvcgael]

Hey @gvcgael thanks for opening, This wont solve your issue but I think the rights on the binary are incorrect. Have you tried chmod +x after installing it ?

Yes, all the binaries have the chmod +x.

If you fix this this still won't work because the code of the vsphere builder needs to be updated to work with packer 1.5: there was a bunch of breaking changes in 1.5. The vsphere builder is currently being merged in Packer in #8480 and we miss a bunch if CLA signature before we can do so. Only then you will be able to use it with Packer 1.5.

Ok no problem, we will remain on packer 1.4.5. Great news to here that the vsphere builder will be merged to upstream packer :)

Thanks for your work.

[Panthaaaa]

Same thing happens with https://github.com/wata727/packer-post-processor-amazon-ami-management

[SwampDragons]

@Panthaaaa this will happen with _all_ third party plugins that have not updated their interfaces. Downgrade to 1.4.5 until the plugin maintainers update their plugins.

[SwampDragons]

PLUGIN MAINTAINERS: For an explanation of how to implement the ConfigSpec() function, see our docs: https://www.packer.io/guides/hcl/component-object-spec

For an example of how to do it, plus how to let your provisioners access the new generatedData argument, see https://github.com/SwampDragons/packer-provisioner-comment/commit/970d9d10c34a3e3c63351a788e4fd36a0650d844

[wata727]

@SwampDragons Thanks for your guidance. I'm a maintainer of a plugin, but it seems that permission denied occurs even if I add the ConfigSpec as described in the guide. What have I missed? https://github.com/wata727/packer-post-processor-amazon-ami-management/pull/59

[TJM]

OK, I am not a developer, but I am trying to grind this one out...

• The API version changed here: https://github.com/hashicorp/packer/commit/18bb4ffb44f98107f4a643e8c3023a71bc76f154

So, we are going to test right there...

Testing procedure:

• First I validated that I could build packer using the directions https://github.com/hashicorp/packer/blob/master/.github/CONTRIBUTING.md#setting-up-packer-for-dev (see note)
  
  
  
  • NOTE: ADD ~/go/bin to PATH. The packer build (make generate) depends on this, otherwise you will get stuck on mapstructure-to-hcl2 is not in $PATH errors.
  
  
• From inside the packer source directory...
• Checkout the commit with the API Change (this is nothing but the API number bump):
  
  
  
  • git checkout 18bb4ff
  
  
• Build: go build -o bin/packer-1.5.0-18bb4ff
• Test: ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.0-18bb4ff validate packer.json

Failed to initialize build 'virtualbox-iso': error initializing post-processor 'artifactory': Incompatible API version with plugin. Plugin version: 4, Ours: 5

NOTE: This result is expected. Test successful :)

• Checkout the previous commit: git checkout 18bb4ff~ (which is 21a8493)

Previous HEAD position was 18bb4ffb4 bump API version since this change makes things backwards incompatible.
HEAD is now at 21a849354 fix hook data loading

• Build: go build -o bin/packer-1.5.0-21a8493
• Test: ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.0-21a8493 validate packer.json

Template validated successfully.

Perfect.

Fix Plugin for new API: https://github.com/pyToshka/packer-post-processor-artifactory/pull/6

I am sure there are "dummy" plugins which would be more appropriate to test with, but alas, this is the one I am working on.

$ ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.0-18bb4ff validate packer.json
Template validated successfully.

OK, lets grab the actual changes now :)

• git checkout 0f6d1be
• go build -o bin/packer-1.5.0-0f6d1be
• Test:

$ ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.0-0f6d1be validate packer.json
Template validated successfully.

Excellent.

• Just for grins, I reverted my changes to the plugin and tested again...

$ ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.0-0f6d1be validate packer.json
Failed to initialize build 'virtualbox-iso': error initializing post-processor 'artifactory': Incompatible API version with plugin. Plugin version: 4, Ours: 5

Perfect. so up to here we are good. Things are working as expected.

So, the "problem" we are experiencing is after the API version bump, but before the 1.5.0 release... as in somewhere in here: https://github.com/hashicorp/packer/compare/21a849354b4e19308c7745a96eec8d865bc2c94e...a8a3758cc806961b4f633d4cb6020de7e93a33d0

I think :)

[TJM]

I followed through testing each commit, and this is the one that appears to make it stop working:

commit 0785c2f6fca9c22bf25528e0176042799dd79df9
Author: Adrien Delorme <[email protected]>
Date:   Tue Dec 17 11:25:56 2019 +0100

The one before it (6bc5a95c3a92231b663258a5545d232db4d5eed0):

$ ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.0-6bc5a95c3 validate packer.json
Template validated successfully.

This one (0785c2f6fca9c22bf25528e0176042799dd79df9):

$ ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.0-0785c2f6f validate packer.json
Failed to initialize build 'virtualbox-iso': error initializing post-processor 'artifactory': fork/exec /Users/tmcneely/go/src/github.com/hashicorp/packer/bin: permission denied

... sadly, that is a mega commit with over 300 changed files :)

[TJM]

This is the "packer.json" file I was using above: https://github.com/pyToshka/packer-post-processor-artifactory/blob/master/example/packer.json

Here is the full log output... (its not too long)

$ PACKER_LOG=1 ~/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev validate packer.json
2019/12/27 09:02:36 [INFO] Packer version: 1.5.2-dev [go1.13.5 darwin amd64]
2019/12/27 09:02:36 [DEBUG] Discovered plugin: artifactory = /Users/tmcneely/.packer.d/plugins/packer-post-processor-artifactory
2019/12/27 09:02:36 using external post-processors [artifactory]
2019/12/27 09:02:36 Attempting to open config file: /Users/tmcneely/.packerconfig
2019/12/27 09:02:36 Setting cache directory: /Users/tmcneely/go/src/github.com/pyToshka/packer-post-processor-artifactory/example/packer_cache
2019/12/27 09:02:36 [ERR] Checkpoint error: EOF
2019/12/27 09:02:36 Plugin could not be found at /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-iso (exec: "/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-iso": stat /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-builder-virtualbox-iso: no such file or directory). Checking same directory as executable.
2019/12/27 09:02:36 Current exe path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Creating plugin client for path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Starting plugin: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev []string{"/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev", "plugin", "packer-builder-virtualbox-iso"}
2019/12/27 09:02:36 Waiting for RPC address for: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [INFO] Packer version: 1.5.2-dev [go1.13.5 darwin amd64]
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Attempting to open config file: /Users/tmcneely/.packerconfig
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Setting cache directory: /Users/tmcneely/go/src/github.com/pyToshka/packer-post-processor-artifactory/example/packer_cache
2019/12/27 09:02:36 packer-1.5.2-dev plugin: args: []string{"packer-builder-virtualbox-iso"}
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [ERR] Checkpoint error: EOF
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Plugin address: unix /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin125487656
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Waiting for connection...
2019/12/27 09:02:36 Received unix RPC address for /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: addr is /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin125487656
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Serving a plugin connection...
2019/12/27 09:02:36 Plugin could not be found at /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell (exec: "/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell": stat /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell: no such file or directory). Checking same directory as executable.
2019/12/27 09:02:36 Current exe path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Creating plugin client for path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Starting plugin: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev []string{"/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev", "plugin", "packer-provisioner-shell"}
2019/12/27 09:02:36 Waiting for RPC address for: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [INFO] Packer version: 1.5.2-dev [go1.13.5 darwin amd64]
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Attempting to open config file: /Users/tmcneely/.packerconfig
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Setting cache directory: /Users/tmcneely/go/src/github.com/pyToshka/packer-post-processor-artifactory/example/packer_cache
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [ERR] Checkpoint error: EOF
2019/12/27 09:02:36 packer-1.5.2-dev plugin: args: []string{"packer-provisioner-shell"}
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Plugin address: unix /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin507706428
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Waiting for connection...
2019/12/27 09:02:36 Received unix RPC address for /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: addr is /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin507706428
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Serving a plugin connection...
2019/12/27 09:02:36 Plugin could not be found at /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell (exec: "/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell": stat /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-provisioner-shell: no such file or directory). Checking same directory as executable.
2019/12/27 09:02:36 Current exe path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Creating plugin client for path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Starting plugin: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev []string{"/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev", "plugin", "packer-provisioner-shell"}
2019/12/27 09:02:36 Waiting for RPC address for: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [INFO] Packer version: 1.5.2-dev [go1.13.5 darwin amd64]
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Attempting to open config file: /Users/tmcneely/.packerconfig
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Setting cache directory: /Users/tmcneely/go/src/github.com/pyToshka/packer-post-processor-artifactory/example/packer_cache
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [ERR] Checkpoint error: EOF
2019/12/27 09:02:36 packer-1.5.2-dev plugin: args: []string{"packer-provisioner-shell"}
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Plugin address: unix /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin943682608
2019/12/27 09:02:36 Received unix RPC address for /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: addr is /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin943682608
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Waiting for connection...
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Serving a plugin connection...
2019/12/27 09:02:36 Plugin could not be found at /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant (exec: "/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant": stat /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev-PACKERSPACE-plugin-PACKERSPACE-packer-post-processor-vagrant: no such file or directory). Checking same directory as executable.
2019/12/27 09:02:36 Current exe path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Creating plugin client for path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Starting plugin: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev []string{"/Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev", "plugin", "packer-post-processor-vagrant"}
2019/12/27 09:02:36 Waiting for RPC address for: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [INFO] Packer version: 1.5.2-dev [go1.13.5 darwin amd64]
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Attempting to open config file: /Users/tmcneely/.packerconfig
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Setting cache directory: /Users/tmcneely/go/src/github.com/pyToshka/packer-post-processor-artifactory/example/packer_cache
2019/12/27 09:02:36 packer-1.5.2-dev plugin: [ERR] Checkpoint error: EOF
2019/12/27 09:02:36 packer-1.5.2-dev plugin: args: []string{"packer-post-processor-vagrant"}
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Plugin address: unix /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin737118340
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Waiting for connection...
2019/12/27 09:02:36 Received unix RPC address for /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: addr is /var/folders/pj/dyscjyhn3m9_v224j2z5jywct5qv5k/T/packer-plugin737118340
2019/12/27 09:02:36 packer-1.5.2-dev plugin: Serving a plugin connection...
2019/12/27 09:02:36 Plugin could not be found at  (exec: "": executable file not found in $PATH). Checking same directory as executable.
2019/12/27 09:02:36 Current exe path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev
2019/12/27 09:02:36 Creating plugin client for path: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin
2019/12/27 09:02:36 Starting plugin: /Users/tmcneely/go/src/github.com/hashicorp/packer/bin []string{"/Users/tmcneely/go/src/github.com/hashicorp/packer/bin"}
2019/12/27 09:02:36 [INFO] (telemetry) Finalizing.
Failed to initialize build 'virtualbox-iso': error initializing post-processor 'artifactory': fork/exec /Users/tmcneely/go/src/github.com/hashicorp/packer/bin: permission denied
2019/12/27 09:02:37 waiting for all plugin processes to complete...
2019/12/27 09:02:37 /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: plugin process exited
2019/12/27 09:02:37 /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: plugin process exited
2019/12/27 09:02:37 /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: plugin process exited
2019/12/27 09:02:37 /Users/tmcneely/go/src/github.com/hashicorp/packer/bin/packer-1.5.2-dev: plugin process exited

[TJM]

As per above, I have a simple fix for this issue with PR #8543 ... and if you are sick of seeing the "Plugin could not be found at /usr/local/bin/packer-PACKERSPACE-plugin-PACKERSPACE-...." messages, checkout #8542 :)

[rui-su-bnet]

Bump, running into the same issue as well. (Responding to TJM below that is what we did as well)

[TJM]

For what its worth, downgrading to 1.4.5 worked for us until I built the fix mentioned above, but we really wanted HCL :)

[TJM]

@SwampDragons Can we change this back to a bug, please? I think I have proven through my testing that even when the external module has been updated to support the ConfigSpec interface, packer still tries to execute its directory instead of the external plugin. Thanks!

[SwampDragons]

Thanks for your patience on this one -- I've been out on vacation. I'll try to reproduce and merge your fix today.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.