Kube-prometheus: Default image for prometheus operator not able to run on Raspbian / armv7

[geerlingguy]

What did you do?

I tried deploying everything in the manifests directory (first the setup dir, then the parent dir), and everything deploys, but when I check my cluster, the prometheus-operator pod is stuck in in a CrashLoopBackOff.

When I deploy the Prometheus operator from bundle.yaml directly, this issue does not occur, and the prometheus-operator pod launches successfully, allowing me to create Prometheus resources.

If I check the logs for the operator container in the operator pod it returns:

kubectl logs prometheus-operator-5cfbdc9b67-hdcwj -n monitoring -c prometheus-operator
standard_init_linux.go:211: exec user process caused "exec format error"

This indicates the image in use (https://github.com/coreos/kube-prometheus/blob/master/manifests/setup/prometheus-operator-deployment.yaml#L29, or quay.io/coreos/prometheus-operator:v0.39.0) doesn't have an armv7-compatible version, or maybe some other setting is in conflict that causes the operator to not launch.

Did you expect to see some different?

Environment

• Prometheus Operator version:

  v0.5.0

• Kubernetes version information:

  Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5+k3s1", GitCommit:"58ebdb2a2ec5318ca40649eb7bd31679cb679f71", GitTreeState:"clean", BuildDate:"2020-05-06T23:42:31Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/arm"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.5+k3s1", GitCommit:"58ebdb2a2ec5318ca40649eb7bd31679cb679f71", GitTreeState:"clean", BuildDate:"2020-05-06T23:42:31Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/arm"}

• Kubernetes cluster kind:

  K3s on Raspberry Pi with HypriotOS

• Manifests:

manifests/setup/prometheus-operator-deployment.yam

• Prometheus Operator Logs:

kubectl logs prometheus-operator-5cfbdc9b67-hdcwj -n monitoring -c prometheus-operator
standard_init_linux.go:211: exec user process caused "exec format error"

• Prometheus Logs:

N/A

Anything else we need to know?:

N/A

[geerlingguy]

It seems the kube-rbac-proxy that ships with the other pods is also incompatible:

kubectl logs node-exporter-bgq2t -n monitoring -c kube-rbac-proxy
standard_init_linux.go:211: exec user process caused "exec format error"

kubectl get pods -n monitoring
NAME                                   READY   STATUS             RESTARTS   AGE
grafana-86b55cb79f-slvtb               1/1     Running            0          9m57s
kube-state-metrics-dbb85dfd5-lxnk5     0/3     CrashLoopBackOff   18         9m33s
node-exporter-sgtjs                    1/2     CrashLoopBackOff   6          9m13s
node-exporter-bgq2t                    1/2     CrashLoopBackOff   6          9m13s
node-exporter-jj6tt                    1/2     CrashLoopBackOff   6          9m13s
node-exporter-f8zfh                    1/2     CrashLoopBackOff   6          9m13s
node-exporter-b7z8c                    1/2     CrashLoopBackOff   6          9m13s
node-exporter-4gz2c                    1/2     CrashLoopBackOff   6          9m13s
node-exporter-b25b5                    1/2     CrashLoopBackOff   6          9m13s
prometheus-adapter-5cd5798d96-6xdjv    0/1     CrashLoopBackOff   6          8m11s
prometheus-operator-5cfbdc9b67-hdcwj   0/2     CrashLoopBackOff   14         12m

[geerlingguy]

For now I'm going to try to get the derivative https://github.com/carlosedp/cluster-monitoring working.

[brancz]

cc @paulfantom I think you recently looked into this?

[geerlingguy]

With the release of Raspberry Pi OS 64-bit (arm64 arch), this might be different. I may be able to test again to see if the containers have arm64 images available.

[paulfantom]

That is strange because I am using and tested those images on Raspbian (raspberry pi 3B+ armv7) and on ubuntu 20.04 (raspberry pi 4B arm64). I'll take a look during weekend to check those issues.

Could you force to use quay.io/coreos/prometheus-operator:v0.39.0-arm or quay.io/coreos/prometheus-operator:v0.39.0-arm64 to check if this is only a problem with manifest generation?

[geerlingguy]

@paulfantom the Prometheus images run fine, it’s the kube-rbac-proxy images that run alongside that cause the exec format error.

[paulfantom]

Ahhh... Sorry, read it too early in the morning. There are no releases of kube-rbac-proxy with multi-arch support right now. Only one tag works with it as can be seen in https://quay.io/repository/brancz/kube-rbac-proxy?tab=tags

Could you check if quay.io/brancz/kube-rbac-proxy:HEAD-2020-05-07-2fd48d73 works for you?

---

@brancz could you release v0.5.1 of kube-rbac-proxy?

[geerlingguy]

I probably won't get the time to test this in the next couple days (a bit overloaded right now), but promise to test it as soon as I get a few cycles on the cluster (right now it's down for a rebuild ;-).

[dmfrey]

Any update on this issue?

[geerlingguy]

I've been out sick since the day after I posted the earlier comment, all my testing was on hold and now I'm a bit backlogged... but I still plan on getting it tested soon.

[paulfantom]

Note: there is a new release of kube-rbac-proxy (v0.6.0) with support for multi-arch.

[tommyjmquinn]

OK so i tried this out with
quay.io/brancz/kube-rbac-proxy:HEAD-2020-06-12-281be3a9
as I couldn't see v0.6.0 in quay

It gets a little further but then we fail because I don't believe kube-state-metrics has an ARM64 build?
was using:
quay.io/coreos/kube-state-metrics:v1.9.5
in kube-state-metrics-deployment.yaml

I also see errors in the other two containers in the same pod but I believe those are because the kube-state-metrics container failed container failed

[tommyjmquinn]

OK I moved on from that by pulling an arm64build from
/carlosedp/kube-state-metrics/
as he has pushed an arm64 build to docker hub

[dmfrey]

Should I try an update?

On Wed, Jun 17, 2020, 8:04 PM tommyjmquinn [e-mail] wrote:

OK I moved on from that by pulling an arm64build from
/carlosedp/kube-state-metrics/
as he has pushed an arm64 build to docker hub

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/coreos/kube-prometheus/issues/545#issuecomment-645689183,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHHFTCRW4447JLE5EUAFG3RXFK2JANCNFSM4NIQO7TA
.

[tommyjmquinn]

Next erro I found is that Prometheus adapter is specified as an arm64 build in
prometheus-adapter-deployment.yaml
it specifies
quay.io/coreos/k8s-prometheus-adapter-amd64:v0.7.0

im searching for an arm64 build now

[tommyjmquinn]

found one ...
https://hub.docker.com/r/carlosedp/k8s-prometheus-adapter/tags

[tommyjmquinn]

OK got this working Ill write up a blog post at some point but everything anyone needs to get this working in a pinch is above :)

[paulfantom]

OK so i tried this out with
quay.io/brancz/kube-rbac-proxy:HEAD-2020-06-12-281be3a9
as I couldn't see v0.6.0 in quay

You used newer than v0.6.0, which is listed just below HEAD-2020-06-12-281be3a9 in https://quay.io/repository/brancz/kube-rbac-proxy?tab=tags

It gets a little further but then we fail because I don't believe kube-state-metrics has an ARM64 build?

There is an open PR about it in https://github.com/kubernetes/kube-state-metrics/pull/1130

Next erro I found is that Prometheus adapter is specified as an arm64 build in
prometheus-adapter-deployment.yaml

Upstream images are multi-arch: https://hub.docker.com/r/directxman12/k8s-prometheus-adapter. We should probably switch to that, can you create a PR?

Summing up, the only major problem right now to run full kube-prometheus on ARM/ARM64 is providing multi-arch builds for KSM.

[paulfantom]

Upstream kube-state-metrics work on ARM support is in https://github.com/kubernetes/kube-state-metrics/pull/1190

[hwdef]

Now, I ran into the same problem when using pi os (32bit)and the latest kube-prometheus.