Terraform: darwin/arm64 build

[dansimau]

I didn't see an existing issue, so I thought I'd open an issue to track building an arm64 (Apple Silicon) binary for macOS.

After migrating to a new Mac, I have seen at least one issue using Terraform almost straight away:

module.certs_us-east-1["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m27s elapsed]
module.certs["wildcard"].aws_acm_certificate_validation.validation: Still creating... [34m57s elapsed]
module.certs_us-east-1["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m37s elapsed]
module.certs["wildcard"].aws_acm_certificate_validation.validation: Still creating... [35m7s elapsed]

assertion failed [abi_info.kind == AbiKind::TranslatedCode]: emulated forward to an arm pc that isn't in translated code. arm_pc=0x1053794d8 abi_kind=6 emulation_interval=[0x105464e60,0x105464e74) instruction_interval=[0x105464e4c, 0x105464e74) x86_rip=0x106148c
(ThreadContextRegisterState.cpp:677 move_to_instruction_boundary)


^C

(CTRL-C did not kill it; when I killed the provider processes they became zombies; I had to SIGKILL the main terraform process.)

Current Terraform Version

Terraform v0.14.2

References

• https://github.com/golang/go/issues/38485 (will at least require Go build support for darwin/arm64)

[apparentlymart]

Hi @dansimau! Thanks for raising this.

There are some things that must happen before we could produce official Terraform releases on arm64 macOS, or before such releases would be practically useful:

• The Go toolchain we use must have production-quality support for that platform. Based on available information, it seems that Go 1.15 (our current version) has only early, experimental support, and that full support is expected in Go 1.16 which is not due until February.
• We will need to acquire hardware with the new CPU in order to verify that all of the platform features we depend on are available in Go's new port to the platform.

• At least the most commonly-used Terraform providers must have at least one release that supports this platform, so that Terraform will be able to install and run those plugins. (A Terraform executable without any provider plugins is not useful.)

  The provider development teams are also constrained by the availability of a suitable Go toolchain and hardware, so they will probably also only be able to do make experimental efforts until Go 1.16 is released, in which case there will be some lag after the Go 1.16 release before sufficient providers are available.

• Our policy for Go language upgrades is that we only accept patch-level upgrades in minor releases, and so the earliest we would adopt Go 1.16 is in the forthcoming v0.15 release. Given that the dependencies here include releases of several other pieces of software, it seems that we should not assume that everything lands in the correct order for macOS arm64 support to necessarily be included in v0.15, but it seems like that's the _earliest_ release where we could support it.

Given that Apple's strategy for gradual ecosystem migration is emulation, I feel optimistic that problems in the current version of the emulation layer will be gradually addressed as the Apple team becomes aware of the problems, so hopefully future updates to the arm64 version of macOS will address the problems that are currently causing the assertion failure you saw. We'll monitor Go releases to see if there are any minor release updates to Go 1.15 that might be issued to improve compatibility with Apple's emulator when running x86_64 binaries, in which case we may be able to include them in v0.14.x minor releases.

[dansimau]

Makes sense.

After using it a bit more, I have concluded that Terraform is basically unusable under emulation on macOS arm64 (Big Sur 11.01).

Here are a sample of some more errors I have been getting:

$ terraform import module.auth.aws_security_group.presignup-lambda sg-XXX
module.auth.aws_security_group.presignup-lambda: Importing from ID "sg-XXX"...
module.auth.aws_security_group.presignup-lambda: Import prepared!
  Prepared aws_security_group for import
module.auth.aws_security_group.presignup-lambda: Refreshing state... [id=sg-XXX]

^C
^C^C^C
^C
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C

(Eventually I killed the provider processes).

$ terraform import module.auth.aws_iam_role.presignup-lambda cognito-presignup-lambda

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to retrieve schema from provider "registry.terraform.io/hashicorp/aws":
rpc error: code = Unavailable desc = connection error: desc = "transport:
authentication handshake failed: context deadline exceeded"

(Is the second time I have had to reinit).

$ terraform apply

Error: timeout while waiting for plugin to start


assertion failed [abi_info.kind == AbiKind::TranslatedCode]: emulated forward to an arm pc that isn't in translated code. arm_pc=0x104d204d8 abi_kind=6 emulation_interval=[0x104d77908,0x104d7791c) instruction_interval=[0x104d778f4, 0x104d7791c) x86_rip=0x4044049
(ThreadContextRegisterState.cpp:677 move_to_instruction_boundary)

At the moment it's roughly a coin toss as to whether a command will work or not.

[danieldreier]

@dansimau as it happens, I just got a an M1 Macbook, so I'm able to reproduce this. I did a state import of a security group, and it worked perfectly for me. However, I have my shell set to run under emulation by default, and when I use the default shell, I (on one run, haven't been able to reproduce since then) see the same behavior as you where it hangs waiting for the plugin to start.

Can you try running a shell under rosetta with a command like arch -x86_64 bash and then re-run these commands to see if the issue you're seeing still happens running in an emulated shell?

[dansimau]

@danieldreier I was already running an x86_64 bash (v5), because I was using /usr/local/bin/bash from Homebrew that was copied over after running Migration Assistant.

[dansimau]

Just FWIW, another symptom:

I randomly opened Activity Monitor and found this provider process using 100% CPU. Not even sure where it came from because I stopped running Terraform locally (unless the vscode language server runs it?)

Anyway, I will update to macOS 11.1 and see if there is any change (there is at least one report of Rosetta bug fixes in 11.1).

Side note, I had a lot of issues with other (non-Terraform) Go binaries compiled for darwin/x86_64 (e.g. my VSCode environment ground to a halt). I compiled Go HEAD for arm64, trashed my $GOHOME and reinstalled/recompiled everything. Now that environment is mostly working and stable (which is to be expected) with the notable exception of anything using cgo.

[tuannvm]

@dansimau

Anyway, I will update to macOS 11.1 and see if there is any change (there is at least one report of Rosetta bug fixes in 11.1).

I had same problem when kubectl & terraform x86 went crazy and ate up a lot of CPU. But upgrading to macOS 11.1 fixed that.

[dansimau]

Upgraded to 11.1 and it didn't change anything. The first time I ran Terraform it hung with the same symptoms (provider at ~100% CPU).

Here's a sample of the process:
https://gist.github.com/dansimau/b424531d9ad69acd1dcc08f2713341e0

[steveetm]

For v0.12 users, terraform plan -out was crashing me as well but since 11.1 it is fixed.

[billinghamj]

For anyone who just wants to get on with being able to do their job...

# on your Apple Silicon (/opt/homebrew) brew installation, _not_ your Intel (/var/local) brew installation
brew install --build-from-source terraform

Make sure your brew bin is in your PATH (e.g. echo 'eval $(/opt/homebrew/bin/brew shellenv)' >> $HOME/.zprofile).

Make sure to restart your terminal if you've just changed your PATH or installed terraform to /opt/homebrew.

git clone https://github.com/hashicorp/terraform-provider-aws.git
cd terraform-provider-aws
git checkout v3.22.0
cd tools
go get -d github.com/pavius/impi/cmd/impi
cd ..
make tools
make build

Then copy from $GOPATH/bin/terraform-provider-aws to:

~/.terraform.d/plugins/registry.terraform.io/hashicorp/aws/3.22.0/darwin_arm64/terraform-provider-aws_v3.22.0_x5

Then back in your workspace:

rm .terraform.lock.hcl
terraform init

It should pick up the arm64 build from your global plugins (https://www.terraform.io/docs/commands/cli-config.html#implied-local-mirror-directories)

The same basic process should work for all providers

The resultant .terraform.lock.hcl file should not be committed as-is, because it will break TF for other developers. Please see @apparentlymart's comment below for a better way to handle that

[apparentlymart]

The strategy of building the providers locally for unsupported architectures is certainly a possible workaround. Thanks for sharing the instructions, @billinghamj.

One thing I'd add here is that following your instructions exactly will cause the .terraform.lock.hcl file to _only_ contain the checksum for your local darwin_arm64 build, which means that the configuration will therefore only be usable on that platform: if you have coworkers using other platforms then the installation will fail because the packages for their platform won't be recorded in the lock file.

If you are in an environment where you need to mix both supported and unsupported platforms then probably the best approach would be to review the git diff created by that final terraform init, see which checksum it recorded in the lock file, and then git checkout .terraform.lock.hcl before adding that new checksum _in addition to_ the "official" checksums Terraform learned from the upstream registry. If you commit that result to the repository then that should leave you in a state where both the official packages and your unofficial build are both acceptable.

---

Relatedly, I'm not sure if the local build process described here serves as a _reproducible build_ such that two different people running those steps on their own laptops would end up producing identical binaries. If not, you may run into similar trouble if you have multiple people in your environment that are using this new platform and creating their own builds. To mitigate that, you could produce one "official-within-my-company" build of the provider and share it with all users, thus causing everyone to agree on what the checksum ought to be.

[billinghamj]

Yeah, in my case I'm just probably not going to commit the modified lock file, and we likely will just share the binaries between each other too.