2.8.1 exec format errors on Raspberry Pi 4 homelab cluster

[cannfoddr]

What you expected to happen?

Upgrading my homelab Pi 4 cluster to Kubenetes V1.25-5. Followed my usual kubadm process and updated weave to latest 2.8.1 version

What happened?

After upgrade completed on controller and work nodes I noticed that weave-net pods were all in crashloopbackoff state.

How to reproduce it?

Install 2.8.1 on Ubuntu Pi 4 system

Anything else we need to know?

Standalone Kubeadm cluster install 5xPi 4 4GB
Linux pikubewrk002 4.19.105-v8-28 #28 SMP PREEMPT Mon Feb 24 17:59:40 PST 2020 aarch64 aarch64 aarch64 GNU/Linux

I've deleted 2.8.1 and installed 2.7.0 and things are now working

Versions:

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:51:05Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.5", GitCommit:"804d6167111f6858541cef440ccc53887fbbc96a", GitTreeState:"clean", BuildDate:"2022-12-08T10:08:09Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/arm64"}
WARNING: version difference between client (1.23) and server (1.25) exceeds the supported minor version skew of +/-1

describe on a pod gave:

Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Pulling  28m (x28 over 149m)     kubelet  Pulling image "weaveworks/weave-kube:latest"
  Warning  BackOff  3m41s (x720 over 148m)  kubelet  Back-off restarting failed container

kubectl log shows:

Dec 11 13:49:40 pikubewrk002 kubelet[569]: E1211 13:49:40.381375     569 remote_runtime.go:233] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"0214c9a3ce15f53e45f93d2b33b69f65705833dfaad1b30dfcd8451612b1d51b\": netplugin failed with no error message: fork/exec /opt/cni/bin/weave-net: exec format error"

[jpetazzo]

Hi!

I'm investigating a similar issue on ARM servers.
It looks like the weaveworks/weave-kube:latest arm64 image is broken; try to use weaveworks/weave-kube:2.8.1 instead:

ubuntu@node1:~$ uname -a
Linux node1 5.15.0-1021-oracle #27-Ubuntu SMP Fri Oct 14 20:04:20 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@node1:~$ sudo docker run --entrypoint sh -ti  weaveworks/weave-kube:latest -c "apk -q add file && file weaver"
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
weaver: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=1a29ea0e5d7e33dcee87512f32068e17dae6f78c, stripped
ubuntu@node1:~$ sudo docker run --entrypoint sh -ti  weaveworks/weave-kube:2.8.1 -c "apk -q add file && file weaver"
weaver: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=892358108af8a6f0d0c5427a34943ebfe471b84c, stripped

I think the YAML manifests should be updated to pin the image to 2.8.1; I'll try to report that on issue #3960 since that's where that YAML manifest was set up.