Caddy: Arm64 version 2.1.0 and 2.1.1 run the same error

[IDSSC]

Arm64 version 2.1.0 and 2.1.1 run the same error

SIGILL: illegal instruction
PC=0x427ca0 m=0 sigcode=1

goroutine 1 [running, locked to thread]:
github.com/klauspost/cpuid.getProcFeatures(0xfaaf60, 0x4000104990, 0x11dab7f, 0xc, 0x4000069e18, 0x421cd4, 0x216e6c0, 0x421cd0, 0x4000069e28, 0x51a5c, ...)
github.com/klauspost/[email protected]/cpuid_arm64.s:15 fp=0x4000069dd0 sp=0x4000069dd0 pc=0x427ca0
github.com/klauspost/cpuid.addInfo(0x216e6c0)
github.com/klauspost/[email protected]/detect_arm64.go:42 +0x20 fp=0x4000069e00 sp=0x4000069dd0 pc=0x422fd0
github.com/klauspost/cpuid.Detect()
github.com/klauspost/[email protected]/cpuid.go:289 +0x70 fp=0x4000069e20 sp=0x4000069e00 pc=0x421d60
github.com/klauspost/cpuid.init.0()
github.com/klauspost/[email protected]/cpuid.go:272 +0x24 fp=0x4000069e30 sp=0x4000069e20 pc=0x421cd4
runtime.doInit(0x20d4720)
runtime/proc.go:5420 +0x9c fp=0x4000069e70 sp=0x4000069e30 pc=0x51a5c
runtime.doInit(0x20e9620)
runtime/proc.go:5415 +0x58 fp=0x4000069eb0 sp=0x4000069e70 pc=0x51a18
runtime.doInit(0x20e6820)
runtime/proc.go:5415 +0x58 fp=0x4000069ef0 sp=0x4000069eb0 pc=0x51a18
runtime.doInit(0x20e6940)
runtime/proc.go:5415 +0x58 fp=0x4000069f30 sp=0x4000069ef0 pc=0x51a18
runtime.doInit(0x20d3a60)
runtime/proc.go:5415 +0x58 fp=0x4000069f70 sp=0x4000069f30 pc=0x51a18
runtime.main()
runtime/proc.go:190 +0x1b8 fp=0x4000069fd0 sp=0x4000069f70 pc=0x44828
runtime.goexit()
runtime/asm_arm64.s:1148 +0x4 fp=0x4000069fd0 sp=0x4000069fd0 pc=0x74604

r0 0x216e6c0
r1 0x4000068380
r2 0x4000069e00
r3 0x7
r4 0xfaaf60
r5 0x0
r6 0x11dab7f
r7 0xcf
r8 0xf
r9 0x4
r10 0x0
r11 0xd
r12 0x1
r13 0x802aaaa00aaaa
r14 0xf8
r15 0x0
r16 0x34
r17 0x68
r18 0x0
r19 0xd0
r20 0x4000069ca8
r21 0x40001203a0
r22 0x4000002000
r23 0x0
r24 0x0
r25 0x0
r26 0x20d4748
r27 0x216e748
r28 0x4000000180
r29 0x4000069dc8
lr 0x422fd0
sp 0x4000069dd0
pc 0x427ca0
fault 0x0

[mholt]

Oh, fun. That looks like a bug in github.com/klauspost/cpuid then.

/cc @klauspost - have you seen this before? Are we "holding it wrong"? 😁

[klauspost]

It must be a problem with this: https://github.com/klauspost/cpuid/pull/44

@fwessels could you take a look?

[fwessels]

Looks like the following instruction is failing:

    mrs x0, id_aa64pfr0_el1  /* Processor Feature Register 0 */

What arm64 chip is this being run on?

(maybe paste the output from lscpu ?)

[IDSSC]

lscpu

Architecture: aarch64
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Vendor ID: ARM
Model: 4
Model name: Cortex-A53
Stepping: r0p4
CPU max MHz: 1512.0000
CPU min MHz: 100.0000
BogoMIPS: 48.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32

[fwessels]

Thanks, what sort of server is this? Something like Pine64 perhaps?

[IDSSC]

Hardware model: Amlogic s905d2

[fwessels]

@IDSSC can you clone https://github.com/fwessels/investigate-arm64 and run go test -v?

Eg. on an AWS Graviton2 instance I am getting:

$ go test -v 
=== RUN   TestMidr
 implementer:  0x41
     variant:   0x3
architecture:   0xf
    part num: 0xd0c
    revision:   0x1
--- PASS: TestMidr (0.00s)
=== RUN   TestProcFeatures
Processor Feature Register 0: 0x0000000000110011
--- PASS: TestProcFeatures (0.00s)
=== RUN   TestInstAttributess
Instruction Set Attribute Register 0: 0x0000100010211120
Instruction Set Attribute Register 1: 0x0000000000100001
--- PASS: TestInstAttributess (0.00s)
PASS

[IDSSC]

Docker container golang:1.14.4-alpine3.11
$ go test -v
fork/exec /tmp/go-build217936674/b001/investigate-arm64.test: permission denied
FAIL _/media/EXTHD/ext/src/investigate-arm64 0.001s

[fwessels]

Looks like the program was not executed, can you try running it with sudo ?

[IDSSC]

Neither root permissions nor normal permissions can be run

[klauspost]

Possibly Related: https://github.com/golang/go/issues/37345

[klauspost]

yes, id_aa64isar0_el1 is privileged. The FreeBSD kernel traps the exception and returns correct values to userland (at least on "recent" version of FreeBSD, I think it appears in FreeBSD 12.0, not sure it works in 11.x)

@IDSSC What OS are you running?

The Go runtime will likely have the same problems.

[IDSSC]

The host system is similar to Ubuntu 18.04

Caddy2.0.0 can be started before, and errors start to be reported after 2.1.0

1. Running in a self-built docker container based on alpine3.11, using official or self-compiled caddy arm64 file to run error
2. Running in a self-built Docker container based on Ubuntu 18.04.4, using the official or self-compiled caddy arm64 file to run an error
3. Running on the host system, using the caddy arm64 file compiled by the official or self-run error

All three methods report the same error

root@ubuntu:/# uname -a
Linux ubuntu 4.9.76 #1 SMP PREEMPT Thu Apr 11 20:39:46 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
root@ubuntu:/# caddy -version
SIGILL: illegal instruction
PC=0x427ca0 m=0 sigcode=1

goroutine 1 [running, locked to thread]:
github.com/klauspost/cpuid.getProcFeatures(0xfaaf60, 0x400007e930, 0x11dab7f, 0xc, 0x4000063e18, 0x421cd4, 0x216e6c0, 0x421cd0, 0x4000063e28, 0x51a5c, ...)
github.com/klauspost/[email protected]/cpuid_arm64.s:15 fp=0x4000063dd0 sp=0x4000063dd0 pc=0x427ca0
github.com/klauspost/cpuid.addInfo(0x216e6c0)
github.com/klauspost/[email protected]/detect_arm64.go:42 +0x20 fp=0x4000063e00 sp=0x4000063dd0 pc=0x422fd0
github.com/klauspost/cpuid.Detect()
github.com/klauspost/[email protected]/cpuid.go:289 +0x70 fp=0x4000063e20 sp=0x4000063e00 pc=0x421d60
github.com/klauspost/cpuid.init.0()
github.com/klauspost/[email protected]/cpuid.go:272 +0x24 fp=0x4000063e30 sp=0x4000063e20 pc=0x421cd4
runtime.doInit(0x20d4720)
runtime/proc.go:5420 +0x9c fp=0x4000063e70 sp=0x4000063e30 pc=0x51a5c
runtime.doInit(0x20e9620)
runtime/proc.go:5415 +0x58 fp=0x4000063eb0 sp=0x4000063e70 pc=0x51a18
runtime.doInit(0x20e6820)
runtime/proc.go:5415 +0x58 fp=0x4000063ef0 sp=0x4000063eb0 pc=0x51a18
runtime.doInit(0x20e6940)
runtime/proc.go:5415 +0x58 fp=0x4000063f30 sp=0x4000063ef0 pc=0x51a18
runtime.doInit(0x20d3a60)
runtime/proc.go:5415 +0x58 fp=0x4000063f70 sp=0x4000063f30 pc=0x51a18
runtime.main()
runtime/proc.go:190 +0x1b8 fp=0x4000063fd0 sp=0x4000063f70 pc=0x44828
runtime.goexit()
runtime/asm_arm64.s:1148 +0x4 fp=0x4000063fd0 sp=0x4000063fd0 pc=0x74604

r0 0x216e6c0
r1 0x4000062380
r2 0x4000063e00
r3 0x5
r4 0xfaaf60
r5 0x0
r6 0x11dab7f
r7 0xeb
r8 0xf
r9 0x4
r10 0x0
r11 0xd
r12 0x1
r13 0x802aaaa00aaaa
r14 0xf8
r15 0x0
r16 0x34
r17 0x68
r18 0x0
r19 0xd0
r20 0x4000063ca8
r21 0x400019a3a0
r22 0x4000002000
r23 0x0
r24 0x0
r25 0x0
r26 0x20d4748
r27 0x216e748
r28 0x4000000180
r29 0x4000063dc8
lr 0x422fd0
sp 0x4000063dd0
pc 0x427ca0
fault 0x0

[klauspost]

What do you mean "similar to Ubuntu 18.04"?

The reason I am asking is that the OS seems to lack support for intercepting these instructions, so I would like to know more about it to see if this is a general problem. The main issue is that there is no way of testing for this AFAICT.

The ticket above indicates that this would also affect the Go runtime. from v1.14 and going forward.

[IDSSC]

I tested the version of 2.1.0 and 2.1.1, this error can be reproduced in arm64 Android, Ubuntu, alpine
But the 2.0.0 program is running normally, it can be determined that this error was introduced in 2.1.0

[mholt]

Caddy 2.0.0 used cpuid v1.2.3; Caddy 2.1.0 (and 2.1.1) uses cpuid v1.3.0.

[klauspost]

@mholt You can revert to v1.2.5 for now. It should be fine for the use you have. I will work with Frank on getting the issue resolved outside of here.