Nomad: Illegal instruction (core dumped) on armv6l

[ttyS0]

When trying to run nomad (v0.5.6) on an armv6l Raspberry Pi (Gen 1 and PiZero), the executable crashes. It does, however run on armv7l Raspberry Pi (Gen 2 & Gen 3). I'm running Arch Linux on the Pis in question, and using the Linux Arm nomad binary downloaded directly from nomadproject.io.

The following is the last bit of output from strace:

set_tls(0x76fe71e0, 0x76fe78d8, 0x76fe9000, 0x76fe71e0, 0x76fe9000) = 0
mprotect(0x76f79000, 8192, PROT_READ)   = 0
mprotect(0x76fa6000, 4096, PROT_READ)   = 0
mprotect(0x10f0000, 4096, PROT_READ)    = 0
mprotect(0x76fe8000, 4096, PROT_READ)   = 0
munmap(0x76faa000, 59413)               = 0
set_tid_address(0x76fe6d88)             = 290
set_robust_list(0x76fe6d90, 12)         = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x76f83300, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x76e61460}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x76f8341c, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x76e61460}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
ugetrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x5db82} ---
+++ killed by SIGILL (core dumped) +++
Illegal instruction (core dumped)

[schmichael]

Our 32-bit ARM build is currently for armv6h or armhf or however you want to say "hard float" :)

We don't have any armv6l boxes handy and the various ARM flavors have proven exceedingly difficult targets to hit -- especially with CGO enabled. We'd like to produce builds without CGO which would make supporting various ARM flavors far far easier. I'm not sure we'll have any resources to invest in fixing this until we removed the CGO dependency.

[ttyS0]

I inadvertently muddled things by running from the uname -m output. The systems in question are hard float, but the difference is in the FPU. The Pi Gen 1 and PiZero systems have VFPv2, while the Pi Gen2 and Gen3 have VFPv3-D16. The nomad binary shows VFPv3-D16, which explains why it doesn't run on the older Pis.

I definitely realize that supporting the smaller Raspberry Pis might not be worth the effort, and just wanted to clarify. :)

[sean@rpi ~]$ readelf -A /usr/bin/ls
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6


[sean@rpi ~]$ readelf -A ./nomad
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "7-A"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3-D16
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6

[muellerkyle]

I ran into this same issue attempting to run nomad on a Raspberry Pi zero and I was able to get nomad to compile to a state where it runs on the ARMv6l. The caveat is that I haven't fully tested nomad on the zero yet. But, I did want to document how I was able to get it to run.

I have an ARMv7 board (I think it's a Raspberry Pi 2) which has more memory than the Zero too.

I cloned this repo and checking out the latest tag on the ARMv7 box. I then modified the GNUmakefile file and added the GOARM=6 flag for the Go compiler to this section:

pkg/linux_arm/nomad: $(SOURCE_FILES) ## Build Nomad for linux/arm
        @echo "==> Building $@ with tags $(GO_TAGS)..."
        @CGO_ENABLED=1 GOOS=linux GOARCH=arm CC=arm-linux-gnueabihf-gcc-5 \
                go build \
                -ldflags $(GO_LDFLAGS) \
                -tags "$(GO_TAGS)" \
                -o "$@"

I then finished with a make bootstrap and a make pkg/linux_arm/nomad and used scp to get the binary from the pkg/linux_arm directory to my Pi Zero. And I was able to run it without the Illegal instruction that I get from the pre-built binary.

I'll note if I run into any further issues here on this ticket, if anyone wants. But, I did get it to build and thought I would note how I did it. For anyone else following this path, you need GO 1.9 and at least gcc 5 on your build Pi.

Thanks!

Update: I was able to get redis running on the Pi Zero using the example job. It spun up without issue.

[schmichael]

@muellerkyle Thanks for the fix! Would you, @ttyS0, and anyone else interested in the 32bit ARM build mind testing the binary I attached to #3823?

[muellerkyle]

I'll pull a copy of your build and try to get it tested tonight.
Thank you!

Should there be a seperate GNUMake profile for the ARMv6 vs v7?

[preetapan]

@muellerkyle https://github.com/hashicorp/nomad/pull/3823#discussion_r165363507

[ttyS0]

It still crashes for me with Illegal instruction on a fresh install of Raspbian on a Pi Zero W.

The binary is showing up as a arch v7 for me.

root@raspberrypi:~# wget -q https://github.com/hashicorp/nomad/files/1683761/nomad.gz
root@raspberrypi:~# gunzip nomad.gz
root@raspberrypi:~# chmod a+x nomad
root@raspberrypi:~# ls -l nomad
-rwxr-xr-x 1 root root 35782112 Feb  1 00:17 nomad
root@raspberrypi:~# sha256sum nomad
8e9d527c0d501e41ded3b134f833ba2bb0f62220f8579ad3a410de76329116b7  nomad
root@raspberrypi:~# ./nomad
Illegal instruction
root@raspberrypi:~# uname -a
Linux raspberrypi 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
root@raspberrypi:~# readelf -A ./nomad
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "7-A"
  Tag_CPU_arch: v7
  Tag_CPU_arch_profile: Application
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-2
  Tag_FP_arch: VFPv3-D16
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
root@raspberrypi:~# readelf -A /bin/ls
Attribute Section: aeabi
File Attributes
  Tag_CPU_name: "6"
  Tag_CPU_arch: v6
  Tag_ARM_ISA_use: Yes
  Tag_THUMB_ISA_use: Thumb-1
  Tag_FP_arch: VFPv2
  Tag_ABI_PCS_wchar_t: 4
  Tag_ABI_FP_rounding: Needed
  Tag_ABI_FP_denormal: Needed
  Tag_ABI_FP_exceptions: Needed
  Tag_ABI_FP_number_model: IEEE 754
  Tag_ABI_align_needed: 8-byte
  Tag_ABI_align_preserved: 8-byte, except leaf SP
  Tag_ABI_enum_size: int
  Tag_ABI_VFP_args: VFP registers
  Tag_CPU_unaligned_access: v6
root@raspberrypi:~# 

:(

[muellerkyle]

I had the same result (Illegal Instruction). I was unable to get that binary to work.
I noticed that the linked ticket did have another user, @nicholasjackson , confirming a working executable built on Pi 3

Ah, nevermind. I see that he did not get it to run.

I can post my binary for testing. I believe that I compiled in on a Raspberry Pi 2.

I've uploaded my binary here: https://github.com/muellerkyle/nomadpizero/releases/download/v0.7.1/nomad.tar.gz

[schmichael]

Yeah, so far I don't think anyone has figured out how to cross-compile for ARMv6 using our root Vagrant box which cuts our releases.

Might have to leave ARMv6 up to community support for the time being... If nothing else I'd be happy to add a note to our downloads page about minimum ARM requirements if someone can concisely describe them.

[nicholasjackson]

@schmichael for the time being I am happy to build on my Pi 3, the releases for 0.7.0 and 0.7.1 are currently there, I will build a 0.8.0 dev release today.

https://github.com/nicholasjackson/nomad-arm6l/releases

[shantanugadgil]

Hi, not sure if this related or not, but when trying to get binaries built on RHEL 6 (x86_64) to run on RHEL 5 (x86_64), I would add the linker hash type to "both".

GNU ld option:
-Wl,--hash-style=both

[schmichael]

@shantanugadgil How does linker hash style relate to arm builds?

[shantanugadgil]

@schmichael

It was just a matter of "psychic debugging" on my part. :)
The symptom of 'crashes at startup with illegal instruction' made me remember things.
Typically I would see a 'floating point exception' during my experiments.
I read the words 'float', 'illegal instruction' and thought of the linker flag.

[mafonso]

I got 0.8.4 to work on the pi zero using the vagrant box to build it by chaning CGO_ENABLED to 0

diff --git a/GNUmakefile b/GNUmakefile
index e6dcbfd..62a12a4 100644
--- a/GNUmakefiletry
+++ b/GNUmakefile
@@ -85,7 +85,7 @@ pkg/linux_amd64/nomad: $(SOURCE_FILES) ## Build Nomad for linux/amd64

 pkg/linux_arm/nomad: $(SOURCE_FILES) ## Build Nomad for linux/arm
        @echo "==> Building $@ with tags $(GO_TAGS)..."
-       @CGO_ENABLED=1 GOOS=linux GOARCH=arm CC=arm-linux-gnueabihf-gcc-5 \
+       @CGO_ENABLED=0 GOOS=linux GOARCH=arm CC=arm-linux-gnueabihf-gcc-5 \
                go build \
                -ldflags $(GO_LDFLAGS) \
                -tags "$(GO_TAGS)" \

after copying the binary to the pi zero

pi@p02:/tmp $ uname -a
Linux p02 4.14.52+ #1123 Wed Jun 27 17:05:32 BST 2018 armv6l GNU/Linux
pi@p02:/tmp $ ./nomad version
Nomad v0.8.4 (75c5fb48c53cfa95926a941a55b3f06553be2d0f+CHANGES)

but readelf -A ./nomad returns nothing for some reason (static?)
The same binary runs on a Pi 2 and Pi 3 with an armv7l as well.

Update: I made some edits to this comment because I tried to reproduce all steps from fresh code and realised that I threw myself (and others) in the wrong direction.
The only change that allowed me to get a working binary from the vagrant builder was changing CGO to 0 for the arm build.

[myoung34]

This process also worked for me
Fwiw Consul works fine on armv6l and uses CGO_ENABLED=0 here and here

[bltavares]

I recently found the core dump error when trying nomad on a small Pi Zero cluster.

Using https://github.com/nicholasjackson/nomad-arm6l instructions, I was able to use a Docker image to cross compile, and run on a PiZero W.

I've made a PR there: https://github.com/nicholasjackson/nomad-arm6l/pull/2, but I would love to include this into Nomad build process as well. I'll take a look at this at some point during the week

[bltavares]

I've put a compiled zip file on my fork of the builder project.

https://github.com/bltavares/nomad-arm6l/releases/tag/v0.8.7-armv6l

I was able to get it running on Pi Zero, but not on Pi 3. I'll give it a try to add a new target for armv6l on the build script if this binary works for others. I didn't test further yet, other than connecting it to the cluster.

Update: Testing this version, it does not allocate docker containers due to some cgo cross-compilation error. But using the new PR does allocate containers properly.

[bltavares]

I was able to produce a new armel binary and add to the build infrastructure to the project. It was quite easy after learning which are the new dependencies required. Really easy to contribute, congrats team.

@myoung34 @schmichael I've opened a new PR. https://github.com/hashicorp/nomad/pull/5525
Would you be able to try them using a test workload? I've not reached the point of running workloads on my local cluster yet.

The artifacts should work after a vagrant up and make prerelease && make release from the VM.

[bltavares]

I've pre-compiled a new armel release using v0.9.0, if anyone is interested. https://github.com/bltavares/nomad/releases/tag/v0.9.0-armel

[myoung34]

@bltavares Ill test 0.9.0 in the next few days

[cgbaker]

There is also @nicholasjackson's project, https://github.com/nicholasjackson/nomad-arm6l. He has released a Nomad 0.9.0 binary as well.

We are not going to add armel to our official releases. We are going to leave this up to community and third-party avenues, like Nic's project.

[otto-dev]

Any armv6l binary for 0.10.0? I made a request here https://github.com/nicholasjackson/nomad-arm6l/issues/5

I have a bunch of Rasberry Pi Zeros, Docker is running, Consul is running, this one is the last one missing

[otto-dev]

I managed to compile 0.10.0 for Raspberry Pi Zero based on https://github.com/bltavares/nomad by replacing arm-linux-gnueabihf-gcc-5 with arm-linux-gnueabi-gcc-5 (note the missing hf) in the GNUmakefile. That's at least what I think the main difference was.

It was a messy process of trial and error carried out in a docker container (I don't know go), and I doubt I can reproduce the steps.