Weave: Segmentation fault on ARM with version 2.5.2

[olavt]

I installed weave on a Raspberry PI 3 master node by issuing these commands:

``
$ sudo kubeadm init

$ kubectl apply -f \
"https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
``

This resulted in a problem with one of the weave pods:

olavt@k8s-master-1:~ $ kubectl get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE
coredns-5644d7b6d9-8stg2 0/1 Pending 0 18m
coredns-5644d7b6d9-kcv2h 0/1 Pending 0 18m
etcd-k8s-master-1 1/1 Running 0 18m
kube-apiserver-k8s-master-1 1/1 Running 0 18m
kube-controller-manager-k8s-master-1 1/1 Running 1 18m
kube-proxy-n9whx 1/1 Running 0 18m
kube-scheduler-k8s-master-1 1/1 Running 1 18m
weave-net-95vdr 1/2 CrashLoopBackOff 8 18m

olavt@k8s-master-1:~ $ kubectl logs weave-net-95vdr -c weave --namespace=kube-system
Segmentation fault (core dumped)

[frederic-blanc]

I have the exact same issue on Raspberry PI 4 with this version.
Two week ago, it works but I do not remember the version used at the time

[HaveFun83]

same here rpi 4 with weavenet 2.5.2

[HindrikStegenga]

Same here on pi 3b master + 3x 2b nodes.

[toraxe]

Same here pi 3b worker.. Any workaround?

[schornig]

same here on rpi3 & 4

[dudefellah]

I just wanted to add a note in case it's helpful to anybody; the problem (for me, anyway) is coming from ipset being called in the launch.sh shell script. I tried running this on hypriot nodes which are running Debian Buster and kernel verison 4.19.75 and it would be my guess that the ipset available in alpine:3.6 (the base image being used by the weave-* images) doesn't play nicely with the kernel modules available for this kernel release.

If this is true, a fix might be to build the weaver containers with more recent base docker images. I might try this at some point. If I do and it works, I can report back. Otherwise, hopefully people at weaveworks can fix this up at some point.

[dudefellah]

Played with this a little bit and I think I'm wrong about ipset. It seems to crash at different times. My best guess now might just be that it's running out of memory. I only have 1gb pi 3s to test on. A pi 4 cluster with more memory might be worth a try...

[toraxe]

Vierd thing.. my setup started to work on rassberry pi buster 3b+ automatically after about 1000 restarts..

[g2david]

The script will point me to this file
https://cloud.weave.works/k8s/v1.10/net.yaml

I have modified to 2.5.0. I am still getting the same error.

Kube is 1.16?
pi@n0:~ $ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:15:39Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/arm"}

[eloylp]

Same here, RPI4 node. Weave v2.5.2 . Kubernetes v1.16.2

[Letme]

I came googling from another issues and determined that we are missing arm images again using:

docker inspect weaveworks/weave-kube:2.5.2

which prints only amd64. Can someone confirm that?

Ran on RPI4 Weave 2.5.2 Kube 1.16.2

I looked on dockerhub and image exists, so there must be something wrong of why I got amd64 version on my docker on master node...

[bboreham]

I don't think you'd get a segmentation fault from an incompatible binary, it would fail to execute.
If you have different symptoms, please click on the 'new issue' button to avoid confusing the thread of conversation.

[Letme]

Its exactly the same issue if you ask me. I did run some commands from #3276 to maybe get more information.

# uname -a
Linux raspberrypi4-master 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

$ docker inspect weaveworks/weave-kube:2.5.2
[
    {
        "Id": "sha256:cd043b725d8265a17e33514a1b9b1d85deb6480e5c1fa643ea5868a87b2e48b5",
        "RepoTags": [
            "weaveworks/weave-kube:2.5.2"
        ],
        "RepoDigests": [
            "weaveworks/weave-kube@sha256:368b5e86902dcb0cc2b2f59e56a1381a5dd9b401ed309d84044ede6366dcb5c6"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2019-05-16T11:58:25.616119241Z",
        "Container": "e9e58d73a06539ae7a4103bb7ac133336642b0dc00ba8478998ffb4f23a104f2",
        "ContainerConfig": {
            "Hostname": "e9e58d73a065",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ",
                "LABEL org.opencontainers.image.revision=4b5d827d941aaf24ae0fd2313e4e3f4496b3a936"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:8ac1d1c079efe34e0c545aec919df8ec3c6433eaa78061aeb01b27d6ea5a83e7",
            "Volumes": null,
            "WorkingDir": "/home/weave",
            "Entrypoint": [
                "/home/weave/launch.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "Weaveworks <[email protected]>",
                "org.label-schema.description": "Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery",
                "org.label-schema.name": "Weave Net",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.url": "https://weave.works",
                "org.label-schema.vcs-ref": "4b5d827d941aaf24ae0fd2313e4e3f4496b3a936",
                "org.label-schema.vcs-url": "https://github.com/weaveworks/weave",
                "org.label-schema.vendor": "Weaveworks",
                "org.opencontainers.image.description": "Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery",
                "org.opencontainers.image.revision": "4b5d827d941aaf24ae0fd2313e4e3f4496b3a936",
                "org.opencontainers.image.source": "https://github.com/weaveworks/weave",
                "org.opencontainers.image.title": "weave-kube",
                "org.opencontainers.image.url": "https://weave.works",
                "org.opencontainers.image.vendor": "Weaveworks",
                "works.weave.role": "system"
            }
        },
        "DockerVersion": "18.09.2",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "ArgsEscaped": true,
            "Image": "sha256:8ac1d1c079efe34e0c545aec919df8ec3c6433eaa78061aeb01b27d6ea5a83e7",
            "Volumes": null,
            "WorkingDir": "/home/weave",
            "Entrypoint": [
                "/home/weave/launch.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "maintainer": "Weaveworks <[email protected]>",
                "org.label-schema.description": "Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery",
                "org.label-schema.name": "Weave Net",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.url": "https://weave.works",
                "org.label-schema.vcs-ref": "4b5d827d941aaf24ae0fd2313e4e3f4496b3a936",
                "org.label-schema.vcs-url": "https://github.com/weaveworks/weave",
                "org.label-schema.vendor": "Weaveworks",
                "org.opencontainers.image.description": "Weave Net creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery",
                "org.opencontainers.image.revision": "4b5d827d941aaf24ae0fd2313e4e3f4496b3a936",
                "org.opencontainers.image.source": "https://github.com/weaveworks/weave",
                "org.opencontainers.image.title": "weave-kube",
                "org.opencontainers.image.url": "https://weave.works",
                "org.opencontainers.image.vendor": "Weaveworks",
                "works.weave.role": "system"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 134156204,
        "VirtualSize": 134156204,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/03d9aca920cc11c19de694be193274e85ceb8ddaa9a982d1d8ea2942bf32dc83/diff:/var/lib/docker/overlay2/a73039218e22e09fc368f3574a5d98256d037dbfdde22cfb0a768521637bd685/diff:/var/lib/docker/overlay2/f757f94f6818b06d762c92d658656993e73d1c961d3203d7b4ff6a513bf4a01c/diff:/var/lib/docker/overlay2/1025b4f03a9ed79fe7c3147e18c16583d36241ff82bdc0f18c955217c3e0059e/diff:/var/lib/docker/overlay2/e71f48fb20a993e467ad4e2b862f2ffa90eecb4657acdf313ccae9dbb80c6535/diff:/var/lib/docker/overlay2/a6601368bc03b701f5e3a8ac937ea0f563917fb9bb128a33028a7dc92d2e6627/diff:/var/lib/docker/overlay2/a53791bd72158d6c6ffec70a17c28ee37520a9647cba7bbcda985083d3c17840/diff",
                "MergedDir": "/var/lib/docker/overlay2/6dc8993d3510f23660ed9e0305ccde7444806fef952ce338aaffa1b121f33708/merged",
                "UpperDir": "/var/lib/docker/overlay2/6dc8993d3510f23660ed9e0305ccde7444806fef952ce338aaffa1b121f33708/diff",
                "WorkDir": "/var/lib/docker/overlay2/6dc8993d3510f23660ed9e0305ccde7444806fef952ce338aaffa1b121f33708/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:5861ca2df442f6563fad2d964aa1d22125f3a6ff195875e8e9d8311431566779",
                "sha256:05ef96dee356c92fe3d875389a148dc0c06fe3bfabb64e8a75999d097170dd19",
                "sha256:1f2872f66897a38c98044022fdd1f723cf65b1081db8ca6821d28c8658698459",
                "sha256:1a650b210fdf883ebe894d22374b4a57d81a7ea2e3cc595a5dc08008df8e0392",
                "sha256:139a946552ea7e33ec13857f6f68c970781f12b06a8d317b099346013a1e7312",
                "sha256:31ae8ec7fb022a296fc82fce234e9021d96d9ce8f4e43ab23889a401648785bc",
                "sha256:be99cb2be199b121c9f1b732e29f4bb7194b0fa4c349ecdb5c374ab37c5ccbe9",
                "sha256:2c6018abe927e134fde55e531a8cbd34dd00a22d9b76e75287315b240b09a1ea"
            ]
        },
        "Metadata": {
            "LastTagTime": "2019-10-28T08:18:34.357736385Z"
        }
    }
]
$ docker run --entrypoint=/bin/sh -ti weaveworks/weave-kube:2.5.2
/home/weave # ^C

$ kubectl get pods --namespace=kube-system
NAME                                          READY   STATUS              RESTARTS   AGE
coredns-5644d7b6d9-cxqgh                      0/1     ContainerCreating   0          16h
coredns-5644d7b6d9-shtxz                      0/1     ContainerCreating   0          16h
etcd-raspberrypi4-master                      1/1     Running             2          16h
kube-apiserver-raspberrypi4-master            1/1     Running             2          16h
kube-controller-manager-raspberrypi4-master   1/1     Running             2          16h
kube-proxy-rrv6l                              1/1     Running             2          16h
kube-proxy-ztn6d                              1/1     Running             0          15h
kube-scheduler-raspberrypi4-master            1/1     Running             2          16h
tiller-deploy-648c4865d-ck4xs                 0/1     ContainerCreating   0          14h
weave-net-7kb2w                               2/2     Running             0          14h
weave-net-w9ctw                               1/2     CrashLoopBackOff    174        14h
$ kubectl --namespace=kube-system logs weave-net-w9ctw weave
Segmentation fault (core dumped)

[bboreham]

When I pull what looks like the same image, it looks like it contains ARM binaries:

$ docker pull weaveworks/weave-kube-arm:2.5.2    
[...]
Digest: sha256:368b5e86902dcb0cc2b2f59e56a1381a5dd9b401ed309d84044ede6366dcb5c6
$ docker inspect weaveworks/weave-kube-arm:2.5.2
[
    {
        "Id": "sha256:cd043b725d8265a17e33514a1b9b1d85deb6480e5c1fa643ea5868a87b2e48b5",
        "RepoTags": [
            "weaveworks/weave-kube-arm:2.5.2"
        ],
        "RepoDigests": [
            "weaveworks/weave-kube-arm@sha256:368b5e86902dcb0cc2b2f59e56a1381a5dd9b401ed309d84044ede6366dcb5c6"
        ],
...
$ docker create --name=xx weaveworks/weave-kube-arm:2.5.2
1430af7e74acdec85e9ea935b9919e6209713ef5d4b7762514f81a288a3b7b04
$ docker cp xx:/bin/busybox the-busybox
$ file the-busybox 
the-busybox: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, stripped
$ docker cp xx:/usr/sbin/ipset the-ipset
$ file the-ipset
the-ipset: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, stripped

[Letme]

you pulled weave-kube-arm:2.5.2 while default kubectl apply command pulls weave-kube:2.5.2 . I do agree the RepoDigests is the same sha (368b5e86902d) which is arm, but

$ docker pull weaveworks/weave-kube:2.5.2
2.5.2: Pulling from weaveworks/weave-kube
Digest: sha256:8fea236b8e64192c454e459b40381bd48795bd54d791fa684d818afdc12bd100
Status: Image is up to date for weaveworks/weave-kube:2.5.2
docker.io/weaveworks/weave-kube:2.5.2

I assume above needs to pull weave-kube-arm?

[bboreham]

What's supposed to happen is that docker fetches the multi-platform manifest, inspects that, then pulls the correct image.
There is no single image named weave-kube-arm:2.5.2.
See https://hub.docker.com/r/weaveworks/weave-kube/tags

[Letme]

OK, so something in multi-platform manifest is not correctly interpreted by docker. I tried with both rapsbian-stretch and buster (tried both with 18.09 and with latest 19.03) packages, but there was no difference. What else could be wrong?

I can also confirm it tries to run the defective node on the RPI4:

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=raspberrypi4-master
kube-system   weave-net-w9ctw                               1/2     CrashLoopBackOff   205        16h   IP.add.ress   raspberrypi4-master   <none>           <none>

[Letme]

I enabled experimental feature for docker for docker pull --platform flag (to try with platform arm), but there is no difference.

[Letme]

Hm, I made a reboot, but now it scheduled that pod (w9ctw) to different node and it is failing with:

 kubectl logs -n kube-system weave-net-9vcg4 weave
DEBU: 2019/10/28 15:09:26.368563 [kube-peers] Checking peer "ce:71:f5:2a:e7:ff" against list &{[{ce:71:f5:2a:e7:ff raspberrypi3-node} {62:7f:70:b8:7b:0b raspberrypi4-master}]}
INFO: 2019/10/28 15:09:27.262699 Command line options: map[db-prefix:/weavedb/weave-net http-addr:127.0.0.1:6784 nickname:raspberrypi3-node name:ce:71:f5:2a:e7:ff no-dns:true port:6783 conn-limit:100 host-root:/host ipalloc-range:10.32.0.0/12 docker-api: expect-npc:true no-fastdp:true datapath:datapath ipalloc-init:consensus=2 metrics-addr:0.0.0.0:6782]
INFO: 2019/10/28 15:09:27.262896 weave  2.5.2
FATA: 2019/10/28 15:09:27.403223 Existing bridge type "bridged_fastdp" is different than requested "bridge". Please do 'weave reset' and try again

while the first one is now running normally.

$ kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=raspberrypi4-master
kube-system   weave-net-w9ctw                               2/2     Running   214        17h   IP.add.ress   raspberrypi4-master   <none>           <none>

So maybe removing an image and pulling it with --platform flag made a difference? My docker is now running without experimental flag and I have also fixed runc (rebuilt from source according to https://github.com/kubernetes/kubernetes/issues/76531)

[Letme]

I did the same trick (pulled docker image with --platform arm flag manually) on the node (Which was failing) then after reboot, it all came back up running normally.

[tha2015]

I did the same trick (pulled docker image with --platform arm flag manually) on the node (Which was failing) then after reboot, it all came back up running normally.

Can you provide specific command which you used? I'm having the same problem on RPi3 (BTW This issue didn't happen one month ago)

[Letme]

I also remember I had a node 2 months ago and did not see this issue. Dockerhub changed something? Because I tried older version (I think 2.3.0) but it also did not work (so maybe I tried too old version).

@tha2015 I turned on docker experimental by adding "experimental": true in /etc/docker/daemon.json then restarted it with sudo systemctl restart docker, then removed the docker image docker rmi imghashofweave-kube and then pulled again with docker pull --platform arm weave-kube:2.5.2, then i rebooted RPI3 worker.

[tha2015]

BTW I used a different Raspbian version at that time. I will try to flash my SD card tonight with older Raspbian version to see if it helps

[pdfruth]

Like others, I started having this problem too.
I have a perfectly working four node cluster (1 master, 3 workers), all based on Raspbian Buster running on RPi4B w/ 4GB.
It appeared to start happening on one of the nodes, after 'apt upgrade && apt update' one day recently.
So, I deleted the broken node from the cluster, and then re-imaged the SD card with the base-image I had initially used to create each node from (circa Aug 15 2019). After re-joining the node back into the kube cluster, the weave docker images were re-pulled (as a result of the daemonset placing a weave pod on the node once again). And all is working once again.
So, I think it has more to do with some change introduced in recent Raspbian Buster udpates.
For me, systems running these kernel versions DO WORK (uname -r);
4.19.57-v7l+
4.19.66-v7l+
A system running this kernel version (or later) DOES NOT work;
4.19.75-v7l+

For me, it looks like the SEGV is happening very soon after the prog/weave-kube/launch.sh script starts running. I stuck a 'set -x' near the beginning of that script to see how far it gets. The SEGV happens on the call to /home/weave/weaver at line 103 of the script. This line;
PEERNAME=$(/home/weave/weaver $EXTRA_ARGS --print-peer-name --host-root=$HOST_ROOT --db-prefix="$DB_PREFIX")

That appears to be a call to a bit of Go code in prog/weaver/main.go
Looking at that code, I'd guess the SEGV is happening during the call to the "peerName" function on line 243, cuz that Go code never returns the Mac address of the 'weave' virtual bridge interface like it's supposed to (probably because something prevented the creation of the bridge in the first place).
This bit (lines 243 - 247);
name := peerName(routerName, bridgeConfig.WeaveBridgeName, dbPrefix, hostRoot)
if justPeerName {
fmt.Printf("%s\n", name)
os.Exit(0)
}

Really reaching now... Looking at the peerName function starting on line 708 of prog/weaver/main.go, I'd further conjecture the problem is happening on the call to net.InterfaceByName(bridgeName) on line 710.

[dudefellah]

@pdfruth thanks for that.

One thing I have noticed is that /proc/sys/net/bridge is not available to containers under Raspbian buster. I'm guessing all previous versions of Rasbian are affected. If the br_netfilter isn't accessible from inside a container, then any number of crazy issues might crop up. Including something like in this issue.

It would be good to test running a newer kernel version (this appears to be fixed in newer releases, I can read /proc/sys/net/bridge on my Ubuntu laptop just fine) with br_netfilter made available on the container side and see if that fixes things. I will personally probably go back to using flannel for now, since it seems like it'll be a pain getting a newer kernel release up and running under raspbian, but if someone is inclined to do so, I think that might be the way to go.

[tha2015]

I found some suggestions to run rpi-update to upgrade kernel to 4.19.79. I will try it tonight.
See https://github.com/iiab/iiab/issues/1993
and https://github.com/raspberrypi/linux/issues/3271

[tha2015]

I ran "sudo rpi-update" and restarted, after that the "weave-net" pod status changed to Running. I guess it fixed the problem.

[pdfruth]

@tha2015 would you please report back with the captured output from these commands on that node where you just ran 'rpi-update'

sudo dpkg --list | grep raspberrypi-kernel
uname -a

[tha2015]

This is the result

$sudo dpkg --list | grep raspberrypi-kernel; uname -a

ii  raspberrypi-kernel                    1.20190925+1-1                      armhf        Raspberry Pi bootloader
Linux homepi 4.19.80-v7+ #1275 SMP Mon Oct 28 18:27:03 GMT 2019 armv7l GNU/Linux

[tha2015]

Not sure if it is related, but although weave-net pod is running after rpi-update, I still has issues with routing. Currently I'm trying to install metallb and its controller pod has some errors when trying to connect to kubernetes API.
I don't know if it is a Kubernetes issue or weave-net issue because I have limited knowledge about how it works.

$ kubectl logs controller-6bcfdfd677-xgh56  --namespace=metallb-system
...
E1030 05:47:01.592449       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.ConfigMap: Get https://10.96.0.1:443/api/v1/namespaces/metallb-system/configmaps?fieldSelector=metadata.name%3Dconfig&limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I also tried to run a busybox pod to check the connection and got similar errors

$ kubectl run busybox -it --image=busybox --restart=Never -- sh
/ # wget -O- https://10.96.0.1:443/api/v1/namespaces/metallb-system/configmaps?fieldSelector=metadata.name%3Dconfig&limit=500&resourceVersion=0
/ # Connecting to 10.96.0.1:443 (10.96.0.1:443)
wget: can't connect to remote host (10.96.0.1): Connection timed out

[dudefellah]

I ended up giving this a try since rpi-update is easy enough to run. After updating the kernel, I've managed to successfully run the weave-kube containers and have also installed Helm tiller, which needs access to the Kubernetes API in order to work. It's my guess that this particular issue is solved with a newer kernel release.

[tha2015]

Not sure if it is related, but although weave-net pod is running after rpi-update, I still has issues with routing. Currently I'm trying to install metallb and its controller pod has some errors when trying to connect to kubernetes API.

I think I found the reason. My Raspberry Pi uses iptables 1.8.2 but
"Weave Net does not work on hosts running iptables 1.8 or above" (see here) also this weave issue

$ iptables --version
iptables v1.8.2 (nf_tables)

So I ran below commands from Kubernetes doc

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

And now it works

[ralfellner]

Hi,

I encountered the same problem with a fresh installation of k8s v1.16.2 on HypriotOS v1.11.4 running on several raspberry pis. This release of HypriotOS is based on Debian Buster and came with Kernel 4.19.75-v7+. I tried to use the most recent version of weave (2.6.0) that already seems to use Alpine 3.8 as base image. I had already switched iptables etc. to legacy mode (k8s setup guide lists issues with kube-proxy if nft is used). To sum it up, the following things did not work for me:

• Upgrade to most recent version of weave (2.6.0) including Alpine 3.8 as base image
• Switch iptables etc. to legacy mode

As @pdfruth, @tha2015 suggested, I ran a rpi-update to switch to the latest kernel version
4.19.81-v7+. After the reboot, the weave containers started without segfaulting.

[chrisreddington]

I'm also struggling with this one, I'm trying to use kubeadm to setup a new Kubernetes cluster on a set of Raspberry Pis (Model 3B+).

I have upgraded to the latest kernel using sudo rpi-update -
Linux k8s-master 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

I'm also using iptables in legacy mode -
iptables --version
iptables v1.8.2 (legacy)

Also using the docker pull weaveworks/weave-npc:2.6.0 --platform arm and same for weave-kube

Seem to keep getting the segfault -
kubectl logs weave-net-w8wgg -n kube-system -c weave
Segmentation fault (core dumped)

Not sure if there are any additional suggestions, but looks like i'm blocked by the issues in the original post, though different version.

[Letme]

@christianreddington the latest kernel version is not 75 (as that one has a bug as you can see up) so make sure you get 81.

[chrisreddington]

@christianreddington the latest kernel version is not 75 (as that one has a bug as you can see up) so make sure you get 81.

I'll keep digging... I had thought that when reading above, but have attempted to re-run sudo rpi-update, but getting the response that "Your firmware is already up to date"

[EDIT]
Looks like there were some cached files. I followed some of the discussion over at https://www.raspberrypi.org/forums/viewtopic.php?t=105917 to trigger the latest firmware update (4.19.83-v7+).

Can confirm that after restart, all works fine for me as well. Thank you!

[struxoje]

I ran "sudo rpi-update" and restarted, after that the "weave-net" pod status changed to Running. I guess it fixed the problem.

This fixed the problem for me as well. Thank you!

[hakan458]

rpi-update also fixed my problem, would be good to know why though.

[bboreham]

Most people seem to be saying this is fixed elsewhere, closing.

[pdfruth]

rpi-update also fixed my problem, would be good to know why though.

@hakan458 and other interested parties. For the sake of posterity....

A change was introduced in the upstream Linux kernel, which then found it's way into the RPi kernel (by way of commit c0ccb4d). That change then made it's way into the stable apt channel sometime around the time the 4.19.75-xxx RPi kernel was released. That change is ultimately what causes the SEGV.

The good news is, that change has subsequently been reverted (in commit 68a2665). The bad news is, that commit has not yet made it's way into the stable apt channel. To-date, it is only available in recent kernel builds that are available via an 'rpi-update'. Until that revert is delivered in a kernel via the stable apt channel, the only way to get it is to do an 'rpi-update'. This is generally considered a risky thing to do because there may be other breaking changes introduced in the bleeding edge kernel delivered via 'rpi-update'.

I posed the question to the kernel maintainers (here) to see when we might expect commit 68a2665 to find it's way into the stable apt channel. Unfortunately, it appears there are other dependencies holding things up at the moment. Hopefully, we'll see the log-jam free up soon.

[damntourists]

Not sure if rpi-update is solving the issue anymore. I tried setting up kubernetes on my Raspberry Pi 3 (buster) and I'm still getting 2019-12-14T05:40:38.294847959Z Segmentation fault (core dumped) even after running rpi-update and rebooting.
I installed with

$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

One of three pods came up. The other two segfault while the functional one is outputting:

2019-12-14T05:40:33.253863859Z DEBU: 2019/12/14 05:40:33.253087 [kube-peers] Checking peer "1a:5c:3e:f6:03:c5" against list &{[{c6:89:83:10:73:4e k8s-master-1} {12:f5:f0:e7:05:cd k8s-node-1} {1a:5c:3e:f6:03:c5 k8s-node-2}]}
2019-12-14T05:40:33.435152254Z INFO: 2019/12/14 05:40:33.434600 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true host-root:/host http-addr:127.0.0.1:6784 ipalloc-init:consensus=2 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:1a:5c:3e:f6:03:c5 nickname:k8s-node-2 no-dns:true port:6783]
2019-12-14T05:40:33.435310483Z INFO: 2019/12/14 05:40:33.434844 weave  2.6.0
2019-12-14T05:40:33.822855139Z INFO: 2019/12/14 05:40:33.822503 Re-exposing 169.254.122.157/16 on bridge "weave"
2019-12-14T05:40:33.918986081Z INFO: 2019/12/14 05:40:33.918659 Bridge type is bridged_fastdp
2019-12-14T05:40:33.919108008Z INFO: 2019/12/14 05:40:33.918737 Communication between peers is unencrypted.
2019-12-14T05:40:34.038437233Z INFO: 2019/12/14 05:40:34.037482 Our name is 1a:5c:3e:f6:03:c5(k8s-node-2)
2019-12-14T05:40:34.038558587Z INFO: 2019/12/14 05:40:34.037786 Launch detected - using supplied peer list: [192.168.2.210 192.168.2.211]
2019-12-14T05:40:34.173848386Z INFO: 2019/12/14 05:40:34.173426 Checking for pre-existing addresses on weave bridge
2019-12-14T05:40:34.175455157Z INFO: 2019/12/14 05:40:34.175138 weave bridge has address 169.254.122.157/16
2019-12-14T05:40:34.190909324Z INFO: 2019/12/14 05:40:34.190593 [allocator 1a:5c:3e:f6:03:c5] No valid persisted data
2019-12-14T05:40:34.206215887Z INFO: 2019/12/14 05:40:34.205875 [allocator 1a:5c:3e:f6:03:c5] Initialising via deferred consensus
2019-12-14T05:40:34.206530262Z INFO: 2019/12/14 05:40:34.206139 Sniffing traffic on datapath (via ODP)
2019-12-14T05:40:34.207984846Z INFO: 2019/12/14 05:40:34.207541 ->[192.168.2.210:6783] attempting connection
2019-12-14T05:40:34.209138179Z INFO: 2019/12/14 05:40:34.208688 ->[192.168.2.211:6783] attempting connection
2019-12-14T05:40:34.209873961Z INFO: 2019/12/14 05:40:34.209466 ->[192.168.2.210:6783] error during connection attempt: dial tcp :0->192.168.2.210:6783: connect: connection refused
2019-12-14T05:40:34.210916461Z INFO: 2019/12/14 05:40:34.210615 ->[192.168.2.211:6783] error during connection attempt: dial tcp :0->192.168.2.211:6783: connect: connection refused
2019-12-14T05:40:34.21656594Z INFO: 2019/12/14 05:40:34.216241 Listening for HTTP control messages on 127.0.0.1:6784
2019-12-14T05:40:34.21819318Z INFO: 2019/12/14 05:40:34.217352 Listening for metrics requests on 0.0.0.0:6782
2019-12-14T05:40:34.595763719Z INFO: 2019/12/14 05:40:34.595302 [kube-peers] Added myself to peer list &{[{c6:89:83:10:73:4e k8s-master-1} {12:f5:f0:e7:05:cd k8s-node-1} {1a:5c:3e:f6:03:c5 k8s-node-2}]}
2019-12-14T05:40:34.633619503Z DEBU: 2019/12/14 05:40:34.632539 [kube-peers] Nodes that have disappeared: map[]
2019-12-14T05:40:36.497956466Z INFO: 2019/12/14 05:40:36.497418 ->[192.168.2.211:6783] attempting connection
2019-12-14T05:40:36.499399851Z INFO: 2019/12/14 05:40:36.499033 ->[192.168.2.211:6783] error during connection attempt: dial tcp :0->192.168.2.211:6783: connect: connection refused
2019-12-14T05:40:37.023003Z INFO: 2019/12/14 05:40:37.022209 ->[192.168.2.210:6783] attempting connection
2019-12-14T05:40:37.024905708Z INFO: 2019/12/14 05:40:37.024385 ->[192.168.2.210:6783] error during connection attempt: dial tcp :0->192.168.2.210:6783: connect: connection refused
2019-12-14T05:40:39.056665896Z INFO: 2019/12/14 05:40:39.055593 ->[192.168.2.210:6783] attempting connection
2019-12-14T05:40:39.058051209Z INFO: 2019/12/14 05:40:39.057278 ->[192.168.2.210:6783] error during connection attempt: dial tcp :0->192.168.2.210:6783: connect: connection refused
2019-12-14T05:40:40.000681921Z INFO: 2019/12/14 05:40:39.999927 ->[192.168.2.211:6783] attempting connection
2
[ ...and so on... ]

EDIT:
Since this is for a newer version, I've opened a new issue at https://github.com/weaveworks/weave/issues/3748

[subhashdasyam]

for me both

sudo rpi-update

followed by

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

worked

[pdfruth]

For those who have been waiting....
A new kernel, with the requisite fixes to address the SEGV, has been released to the mainstream apt repositories.

Upon doing an 'apt update && apt upgrade' today, I see this kernel version installed;
sudo apt list raspberrypi-kernel
raspberrypi-kernel/testing,now 1.20200114-1 armhf [installed]

uname -a
Linux pikube1.localdomain 4.19.93-v7l+ #1290 SMP Fri Jan 10 16:45:11 GMT 2020 armv7l GNU/Linux

Testing a kube cluster on a four node cluster of RPi4's with weave overlay network... so far all is good.