Atlantis: Bug: [Docker] Custom Terraform plugins not found on plan

[smiller171]

I have a custom Terraform plugin at ${workspace}/terraform.d/plugins/linux_amd64/terraform-provider-errorcheck_v1.1.5. The plugin is found during init, but during plan I get the following error

exit status 1: running "version=$(git rev-parse --short HEAD); [[ -z \"${version}\" ]] && { echo '$version is not set' ; exit 1; }; PATH=/home/atlantis/.atlantis/bin:$PATH terraform${ATLANTIS_TERRAFORM_VERSION} plan -input=false -no-color -out ${PLANFILE} -var nginx_container_version=${version} -var php_container_version=${version} -var-file=\"$(source ../../.atlantis/workspaceselect.sh).tfvars\"\n" in "/home/atlantis/.atlantis/repos/rateco/rateco.ca/227/default/terraform/app": 

Error: Failed to instantiate provider "errorcheck" to obtain schema: fork/exec /home/atlantis/.atlantis/repos/rateco/rateco.ca/227/default/terraform/app/terraform.d/plugins/linux_amd64/terraform-provider-errorcheck_v1.1.5: no such file or directory



Switched to workspace "test".

atlantis.yaml:

version: 3
automerge: true
projects:
  - name: app
    dir: terraform/app
    terraform_version: v0.12.2
    autoplan:
      when_modified:
        - "**"
        - "../../code/**"
        - "../modules/**"
      enabled: true
    workflow: app

workflows:
  app:
    plan:
      steps:
        - run: rm -rf .terraform
        - init:
            extra_args: [-backend-config=backend.tfvars]
        - run: >
            PATH=/home/atlantis/.atlantis/bin:$PATH            
            terraform${ATLANTIS_TERRAFORM_VERSION} workspace select -no-color "$(source ../../.atlantis/workspaceselect.sh)"
        - init:
            extra_args: [-backend-config=backend.tfvars]
        - run: >
            version=$(git rev-parse --short HEAD);
            [[ -z "${version}" ]] && { echo '$version is not set' ; exit 1; };
            PATH=/home/atlantis/.atlantis/bin:$PATH
            terraform${ATLANTIS_TERRAFORM_VERSION} plan -input=false -no-color
            -out ${PLANFILE}
            -var nginx_container_version=${version}
            -var php_container_version=${version}
            -var-file="$(source ../../.atlantis/workspaceselect.sh).tfvars"

[lkysow]

Hmm, it should work.

• Can you confirm via run steps and ls that the second init isn't deleting that file?
• Also why are you running init twice?
• Also I'm assuming you've run this locally and it works fine?
• Can you show your ../../.atlantis/workspaceselect.sh script?

[smiller171]

I'm running init twice because otherwise workspace selection breaks. I need to init both before and after workspace selection (or at least I did on 0.11) which is super annoying and I don't fully understand why, but this works.

This does work locally

Here's the workspace selection script:

#!/bin/bash
case "${BASE_BRANCH_NAME}" in
  develop)
    export WORKSPACE_COMPUTED=test
    ;;
  master)
    export WORKSPACE_COMPUTED=prod
    ;;
  *)
  exit 1
  ;;
esac

echo "${WORKSPACE_COMPUTED}"

I'll confirm that the file is still there shortly

[smiller171]

workflows:
  app:
    plan:
      steps:
        - run: rm -rf .terraform
        - run: tfenv install
        - init:
            extra_args: [-backend-config=backend.tfvars]
        - run: terraform workspace select -no-color "$(source ../../.atlantis/workspaceselect.sh)"
        - init:
            extra_args: [-backend-config=backend.tfvars]
        - run: >
            set -ex;
            pwd;
            ls -la terraform.d/plugins/linux_amd64/;
        - run: >
            version=$(git rev-parse --short HEAD);
            [[ -z "${version}" ]] && { echo '$version is not set' ; exit 1; };
            terraform plan -input=false -no-color
            -out ${PLANFILE}
            -var nginx_container_version=${version}
            -var php_container_version=${version}
            -var-file="$(source ../../.atlantis/workspaceselect.sh).tfvars"

Error: Failed to instantiate provider "errorcheck" to obtain schema: fork/exec /home/atlantis/.atlantis/repos/rateco/rateco.ca/229/default/terraform/app/terraform.d/plugins/linux_amd64/terraform-provider-errorcheck_v1.1.5: no such file or directory



Switched to workspace "test".

+ pwd
/home/atlantis/.atlantis/repos/rateco/rateco.ca/229/default/terraform/app
+ ls -la terraform.d/plugins/linux_amd64/
total 31596
drwxr-xr-x 2 atlantis atlantis     4096 Jun 25 18:22 .
drwxr-xr-x 4 atlantis atlantis     4096 Jun 25 18:22 ..
-rwxr-xr-x 1 atlantis atlantis 32344506 Jun 25 18:22 terraform-provider-errorcheck_v1.1.5

[lkysow]

Hmm, I'm stumped. Can you provide me a sample terraform file I can use to trigger the issue? Also once you have that trimmed down file, can you double check that the most simple workflow fails:

        - init
        - run: >
            set -ex;
            pwd;
            ls -la terraform.d/plugins/linux_amd64/;
        - plan

[smiller171]

I can set up a basic repo that includes the plugin. It's an incredibly simple plugin that just allows for custom validation to throw errors on plan

Edit: This error isn't from the plugin itself though

[smiller171]

I haven't yet tested this in Atlantis, but here's the repo https://github.com/smiller171/atlantis-plugin-bug-poc

[smiller171]

Unfortunately my minimal test case failed to fail. I'm gonna try building on this a bit to see if I can reproduce

[smiller171]

I can't get it to fail running locally on my Mac. Will need to test running in Fargate like I am in production

[smiller171]

The PR I'm testing with: https://github.com/smiller171/atlantis-plugin-bug-poc/pull/1

[lkysow]

Thanks for your hard work, this will make it much easier for me to help.

[smiller171]

It fails as soon as I run it in Docker. Not sure if this is a Linux issue or a Docker issue. I'm working on bringing my test case back down to as minimal as possible

[smiller171]

All right. When running in Docker (locally) it fails even without a custom workflow

https://github.com/smiller171/atlantis-plugin-bug-poc/pull/1

[smiller171]

I don't know if this is an issue with Docker or an issue with the Linux build vs the Mac build

[smiller171]

BTW this plugin is fantastic for catching errors that would normally not be caught until apply and is open source: https://github.com/rhythmictech/terraform-provider-errorcheck

[lkysow]

Hey Scott, so I can reproduce the issue just by running terraform in a linux Docker image. So I don't think the bug is with Atlantis at all.

docker run -v $(pwd):/tf -it --rm --entrypoint=sh hashicorp/terraform:latest
cd /tf
rm -rf .terraform
terraform init

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.errorcheck: version = "~> 1.1"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
/tf # terraform plan

Error: Failed to instantiate provider "errorcheck" to obtain schema: fork/exec /tf/terraform.d/plugins/linux_amd64/terraform-provider-errorcheck_v1.1.5: no such file or directory

ls /tf/terraform.d/plugins/linux_amd64/terraform-provider-errorcheck_v1.1.5
/tf/terraform.d/plugins/linux_amd64/terraform-provider-errorcheck_v1.1.5

[smiller171]

Well that's interesting. Looks like I've got more debugging to do on my end. Thanks.

[lkysow]

👍 I'm going to close this for now. If it looks like this is a bug with Atlantis I'll re-open.

[remilapeyre]

Hi @lkysow, I think I'm having the same issue, here's my Dockerfile:

FROM golang:1.12

COPY terraform-provider-pypiserver/ terraform-provider-pypiserver/
RUN cd terraform-provider-pypiserver && go install

FROM runatlantis/atlantis:v0.8.1

LABEL maintainer="Rémi Lapeyre <[email protected]>"

COPY --from=0 /go/bin/terraform-provider-pypiserver /home/atlantis/.terraform.d/plugins/terraform-provider-pypiserver

RUN chown -R atlantis:atlantis /home/atlantis/.terraform.d/
RUN chmod +x /home/atlantis/.terraform.d/plugins/terraform-provider-*

In the final Docker image, the provider does not work:

bash-4.4# /home/atlantis/.terraform.d/plugins/terraform-provider-pypiserver 
bash: /home/atlantis/.terraform.d/plugins/terraform-provider-pypiserver: No such file or directory

but I tested it in the golang:1.12 image and it works fine there. I think it may be related to https://superuser.com/questions/1176200/no-such-file-when-it-exists but I'm not sure.

[remilapeyre]

For what it's worth I finally solved my issue by changing

RUN cd terraform-provider-pypiserver && go install

to

RUN cd terraform-provider-pypiserver && CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"'

which seems to have solved my issue. On my Mac I cannot run strace in the Atlantis Docker image so I could not find what was the missing library.

[smiller171]

Thanks @remilapeyre I forgot to compile a statically linked binary!

[remilapeyre]

I tought go was supposed to statically linked libraries, isn't that the case?

I think we may need to look at what are the missing libraries in the Atlantis image to add them, my provider was just calling some https endpoints and many others will probably have the same issue.

Surprisingly, when cross-compiling from MacOS to linux_amd64 everything was working nicely

[lkysow]

It's a known issue with CGO and Alpine Linux I think. We use CGO_ENABLED=0 in our build

https://github.com/runatlantis/atlantis/blob/1e030634a81b308c4922cbcee5ba928a8fb962b6/Makefile#L26-L27

[remilapeyre]

Would it make sense to use debian-slim to avoid this then?

[lkysow]

If more people run into this issue then maybe. As is it, I wouldn't want to change the base image that's worked fine for a couple years until now.

As a workaround you can build your own image on top of debian-slim and use Docker multi-stage builds to pull the atlantis binary in from runatlantis/atlantis.

[smiller171]

I think the Ubuntu images are even smaller than Debian Slim now. It's at least close.

@remilapeyre Go _can_ compile static binaries, and you are, but I'm not. I'm also not specifying CGO_ENABLED=0

[Heiko-san]

just ran into the same issue, took quite a while to realize it's the alpine image, it's working with debian buster