Weave: No ARM image for the Weave Net 2.3.0 release

[remonlam]

Hi all,

I'm trying to build a Kubernetes 1.10 environment on 4 Raspberry Pi's, running the latest release of Raspbian Stretch.

However it seems that the latest Docker images on the Docker Hub (weaveworks/weave-kube:2.3.0) does not have a ARM(HF) image.

What you expected to happen?

An ARM based image for the 2.3.0 release of Weave Net.

What happened?

During the deployment of the weave-net containers you see: exec format error's.

How to reproduce it?

Trying to run the container on just Docker, I'll get the same exec error.
docker run -it weaveworks/weave-kube:2.3.0 standard_init_linux.go:190: exec user process caused "exec format error"

Anything else we need to know?

weave-net 2.2.1 seems to have a ARM image

Versions:

$ docker version
18.03.0-ce

$ uname -a
Linux n0-nodename 4.14.30-v7+ #1102 SMP Mon Mar 26 16:45:49 BST 2018 armv7l GNU/Linux

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/arm"}

Logs:

Network:

[bboreham]

I ran the publish step again - try it now.

[remonlam]

Thanks @bboreham on Docker I don't see any errors, will test K8S tonight

[remonlam]

Thanks! it's working now, will close this issue.

[arussellsaw]

i'm having this issue too. k8s 1.10 on raspberry pi

strangely my master is running successfully, but the second node i've added is seeing exec format error

[bboreham]

Different processor? Please run uname -a

[remonlam]

Did you remove the old weave images from the worker nodes?

[arussellsaw]

i removed the old weave images yeah. it's at home now so i'll check that later, but one thought, the first node is a Raspberry Pi 3 B, the second one is the newer Pi 3 B+

[james-powis]

I will create another issue if needed, but have a Pi 3 B+ which is having the same issues as expressed in this issue with 2.2.1 and 2.3.0

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.34-v7+ #1110 SMP Mon Apr 16 15:18:51 BST 2018 armv7l GNU/Linux

[brb]

@james-powis Could you run docker run --rm weshigbee/manifest-tool inspect weaveworks/weave:2.3.0 and paste the output here?

I've just spotted that we don't publish arch variant for any image while e.g. Alpine does it (run docker run --rm weshigbee/manifest-tool inspect alpine:latest and grep for Variant).

[james-powis]

Here is on the model 3 b+:

pi@raspberrypi:~ $ docker run --rm weshigbee/manifest-tool inspect weaveworks/weave:2.3.0
standard_init_linux.go:190: exec user process caused "exec format error"

docker run --rm weshigbee/manifest-tool inspect alpine:latest
standard_init_linux.go:190: exec user process caused "exec format error"

I get much more output on my old 3b.

[james-powis]

even using manifest-tool v0.7.0 same story even on the weshigbee/manifest-tool on the 3b and 3b+

[brb]

Sorry, my bad - the weshigbee/manifest-tool image exists only for amd64.

[bgruszka]

Same here on my RPi3 :(

HypriotOS/armv7: pirate@alcohol1 in ~
$ docker run --rm weshigbee/manifest-tool inspect weaveworks/weave:2.3.0
Unable to find image 'weshigbee/manifest-tool:latest' locally
latest: Pulling from weshigbee/manifest-tool
10a267c67f42: Pull complete
fb5937da9414: Pull complete
684941781a3a: Pull complete
Digest: sha256:eed06b548a0d4a34c6ec685bbf2bc6b9fcc952d735b9ecb7aea1c1f3024cb7c6
Status: Downloaded newer image for weshigbee/manifest-tool:latest
standard_init_linux.go:190: exec user process caused "exec format error"
HypriotOS/armv7: pirate@alcohol1 in ~

Any news on that? :)

[brb]

Sorry, I haven't had time to look into this issue properly.

What Docker image of Alpine Linux can run on your R-PI?

[luxas]

So I don't think the weshigbee/manifest-tool Docker image itself is multiarch, so to check whether the Weave image is multiarch doesn't work if you're trying to run an amd64 binary to determine that.
The Weave image is built for the RPi 2 and newer (armhf, GOARM=7). Can you give more information about the environment where _the Weave Net images themselves fail (not where the manifest-tool fails_)?
Thanks!

[mckern]

I'm having the same problem. I installed manifest-tool locally, and produced this:

$ manifest-tool inspect weaveworks/weave:2.3.0
Name:   weaveworks/weave:2.3.0 (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:02914df933ffd52c6daf8c4ced33e48dad36e4d4fd9e684d5004cd72236ced60
 * Contains 4 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:a34edd09494d8386cb803631aae01e9b68b9bd2e8a66edd203c98e7b5ac00aef
1  Mfst Length: 1368
1     Platform:
1           -      OS: linux
1           - OS Vers:
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant:
1           - Feature:
1     # Layers: 5
         layer 1: digest = sha256:88286f41530e93dffd4b964e1db22ce4939fffa4a4c665dab8591fbab03d4926
         layer 2: digest = sha256:ad4e50ed2c0864c172278d5d8c7fb42f16fd61970e4c2dd608ca08edd29d6720
         layer 3: digest = sha256:b3f4c952e7c2927e480f84f01c39a99a0e882c8437778b589e423b2f95710c32
         layer 4: digest = sha256:5e27cb7f1c2b15fbae335d957f5a0faca0249c022be3ec5bc431d86a5426f855
         layer 5: digest = sha256:f9dfb03c1d7b30788594c05e4b71e6d370829c47a9da7e0f36d6fab9d765c167

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:df6243c3964aaf957fbcc02d278bcd0a9b5e8aad2713877cba840b5961847a3b
2  Mfst Length: 1787
2     Platform:
2           -      OS: linux
2           - OS Vers:
2           - OS Feat: []
2           -    Arch: arm
2           - Variant:
2           - Feature:
2     # Layers: 7
         layer 1: digest = sha256:0864efeeb5cb8dca4eb53e5d6fd38486daee80fa326fe36d1ad254f8fa6bb310
         layer 2: digest = sha256:080d529007c9f23d34293d57bcb09ff52df62b428cd0de400438e01e7a42f085
         layer 3: digest = sha256:d3285738f2749bcf1cd9e55391b73dbdfd3f4912421812052ca57d892623a496
         layer 4: digest = sha256:1f1f6d0b030522298e0e10a21c2756a0ea7fffcf30edaefdf8e003288d73e639
         layer 5: digest = sha256:88f44245e5bd8540954142bbcdb64f9d83a852a280d2bf396c54bd0a1e1caa5f
         layer 6: digest = sha256:69f06a51f0444ccbd4f49c600ca54b500324f082ddbd184a00991a57aa7a16e2
         layer 7: digest = sha256:ed6e778a8141c06b59526d7ff19ca620dca982b0186014c13b447dc879d2f241

3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:bdabaa1f459e05ade98b3758edd26a5556d820cc035a00feedcf8d1276ebd186
3  Mfst Length: 1787
3     Platform:
3           -      OS: linux
3           - OS Vers:
3           - OS Feat: []
3           -    Arch: arm64
3           - Variant:
3           - Feature:
3     # Layers: 7
         layer 1: digest = sha256:bb473f0ebc12fde1bd45c1bd3c46f2d3aab367b1b7739464771455b9972f7894
         layer 2: digest = sha256:95a435f629156f89100b4013689a5cf8fdcd7e3e52ebd3575bf20db2a72655d0
         layer 3: digest = sha256:ebdaa8d2c96770c4b62ace23ccf7509803818b5b9b936b93f568b910da23331e
         layer 4: digest = sha256:b2d8bd9674275b5e84f565ce9c0f7512769dcd53e714005afdf9ad63f5bed413
         layer 5: digest = sha256:831c04980cd8889e8faee73e3355cfc57dfd5f8fe98a1508c5c91237566d983f
         layer 6: digest = sha256:086ebe9fb9db1c14bdd74aa79338053248e2b8a3b7ae28da081a40fb50da176d
         layer 7: digest = sha256:beecc23c3e4537a5c57cef8c793d4f60487caec233d815b6f416125ae950a113

4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:ee2c26648ddaf66895c4086f57aa8c382d0e45e3a4681a862f0d303bd265c374
4  Mfst Length: 1787
4     Platform:
4           -      OS: linux
4           - OS Vers:
4           - OS Feat: []
4           -    Arch: ppc64le
4           - Variant:
4           - Feature:
4     # Layers: 7
         layer 1: digest = sha256:1e52418956f7d2a8ea35e8e6e3318fd08e005b27457d77868c225e7433bbfa02
         layer 2: digest = sha256:ed6f332c99ba2e15b9c8422a60f82415ab796807672570755b180a18a09d60dd
         layer 3: digest = sha256:d7ae08d952b259ce619e95fc50d78ef97fefb7c98d81cb63618d817d3e1d5b40
         layer 4: digest = sha256:f6c2c07059b3069612f101caf8f488ff23e7295b578597dfc37aca44412e3383
         layer 5: digest = sha256:169f6f8507280de32dbfba2976d267114b568e1a0bf0bcfc5b8e5db5d8412e1e
         layer 6: digest = sha256:e9a23937e4db46bbdd3d5d28d65fce7904a80d955bec3a1be26360b532569f71
         layer 7: digest = sha256:362470c6273f52ef01772ff6e14c2eee87d39ea18424942245f04207ebbf767a

[brb]

@mckern @bgruszka @james-powis @arussellsaw

Could you run the following on your Pi 3 B+ and paste outputs here:

1. uname -a
2. docker inspect weaveworks/weave-kube:2.3.0
3. docker run --entrypoint=/bin/sh -ti weaveworks/weave-kube:2.3.0

[brb]

Someone was able to run (regardless the kernel panic) Weave Net 2.3.0 on RPI 3 B+: https://github.com/weaveworks/weave/issues/3314

[bernhara]

Yes.

But to solve the problem with the "wrong exec format", I rebuilt my own image from scratch, directly on the PI3 with a Rasbian distrib.
That image was working perfectly, up to latest Raspbian update (or Docker for Raspbian...), where I got the mentioned Kernel Panic (=> PI freeze).

Since I upgrade only parts of my PIs, I have still some units running with that image and one other which crashes after less than one minute run of WeaveNet.

[brb]

@bernhara Thanks, _"up to latest Raspbian update"_ made me to realize the cause for the kernel panic: https://github.com/weaveworks/weave/issues/3314#issuecomment-396257270.

[alexellis]

@bboreham It's unclear from this thread whether the Docker image is now compatible with ARMHF.

I can confirm that the image weaveworks/weave-kube:2.3.0 will now run on Raspberry Pi 3B+ without an exec error.

Alex

[alexellis]

I suggest closing this issue now that the reported issue is resolved.

[fkpwolf]

Works on my Odroid XU4 with Ubuntu 18.04. Good job.

My master node is amd-64, and I have to change kube-porxy pod on arm with image url "gcr.io/google_containers/kube-proxy-arm:v1.10.4".

[bboreham]

Thanks for the note @fkpwolf . I'll close this since the original point seems addressed, but please do point out anything I've missed and/or open new issues.