Lens: Prometheus doesn't install on arm64 cluster

[rockfordlhotka]

Describe the bug
I do not have Prometheus installed on my arm64 microk8s cluster. I clicked the install button in Lens settings and it attempted to deploy, but the pod was unable to start - crashloopbackoff.

To Reproduce
Steps to reproduce the behavior:

1. Connect to an arm64 cluster
2. Click around to ensure Lens is connected/working other than Prometheus
3. Go to settings for the cluster
4. Click on install prometheus
5. Click back to dashboard and see errors as pod fails to initialize

Expected behavior
I expected prometheus to install and come up, lighting up the dashboard in Lens.

Environment (please complete the following information):

• Lens Version: 3.4.0
• OS: Windows 10 (client); Ubuntu 18.04 (k8s nodes); Microk8s 1.18.2
• Installation method (e.g. snap or AppImage in Linux): Win10 installer

[BlackMix]

Hi @rockfordlhotka
New version runs Prometheus , but u need now change deploy in kube-state-metrics
and change image for carlosedp/kube-state-metrics:v1.9.6
that image have support for arch amd64, arm, arm64
coreos/kube-state-metrics have only amd64 and not support for others arch.

I run in raspberry pi 4 arm64 kernel

[jnummelin]

This should be "fixed" in 3.5.0 already via https://github.com/lensapp/lens/pull/400. With "fixed" I mean that kube state metrics should now be left in "Pending" state as it cannot find proper node as itäs using only amd64 selector

[BlackMix]

@jnummelin with new version I got warning message, that not good idea, better add one image with amd64, arm, arm64 builds and so everyone can use Lens, Carlos image is fine have recent build from metrics and 100k+ downloads

[rockfordlhotka]

@BlackMix if I understand correctly, you are saying that I can't have Lens install Prometheus.

I need to find the deployment file (which is where?), edit it, and install Prometheus by hand?

[BlackMix]

@rockfordlhotka nop, u install normal in Lens and stop kube-state-metrics deploy with scale 0, so u edit deploy with new image: carlosedp/kube-state-metrics:v1.9.6
after edit image back scale for 1

edit: Prometheus in last version work fine, only kube-state-metrics not work in arm64, so need use other image.

[rockfordlhotka]

Oh, ok. I need to do some learning 😄

[BlackMix]

I think now this fix it #484
I tested and run without errors.
node arm64

[rockfordlhotka]

I finally got enough time to try this again - latest version of microk8s on arm64 and latest version of Lens. It does make a good effort, but kube-state-metrics doesn't start on any of the nodes because they don't match the node selector. I'm afraid I am not sure what that means?

[rockfordlhotka]

Is the problem because there's no selector for arm64?

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - amd64

[cristiangauma]

Yes, this is one of the problems. But when I tried to add the value in the matchExpressions statements, then it failed to start the pod with a CrashLoopBackOff problem.

The good news are that until now kube-state-metrics didn't have support (image) for arm64, but it seems that they started to work on this 11 days ago. You can follow the issue and PR in their official Github project:

Issue: https://github.com/kubernetes/kube-state-metrics/issues/1037
PR: https://github.com/kubernetes/kube-state-metrics/pull/1190

[rockfordlhotka]

My current workaround is to repurpose a 2007 era laptop that was gathering dust as a k8s node, so I have one amd64 node that can run this particular container.

[luis-garza]

Is there any news about the issue?

[fmdlc]

I make it work in my local Raspberry Pi cluster under the arm64/v7 architecture.

For that, I had to edit these nodeAffinity annotations for the kube-state-metrics deployment in the lens- metrics namespace. Otherwise, the pods won't be scheduled to run since it is unable to find any matching expression.

     affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - arm64
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
                  - key: beta.kubernetes.io/arch
                    operator: In
                    values:
                      - arm64

Also, I had to update the Deployment image, to point to the specific image for ARM. (To know which images tags use, check here). I updated the deployment image in order to use k8s.gcr.io/kube-state-metrics/kube-state-metrics-arm64:2.0.0-alpha.1.

Using the provided Quay repository doesn't work since the image was build for the amd64 architecture.

I would recommend using Moby BuildKit to build multi-architecture images (at least for the most popular architectures), or updating the Registry to use the above listed images.

[pixie79]

I have tested the above using the latest alpha release - k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0-alpha.3

This all appears to work correctly, looking at the image build I would suggest replacing that as the default as it supports - (arch: amd64, arm, arm64, ppc64le and s390x)

I also note they have stated they will stop producing the quay image soon. By switching the image you probably dont need the node selectors.

The only issue i do see is the pod counter says i have 0 out of 440. A, i dont know where the 440 is configured and b i have lots of pods running :) - but CPU & Memory look good which is the main thing.

Tested on a Raspberry PI Arm - Microk8s cluster

Thanks

[organicnz]

@pixie79 could you share your manifest, mate? I'd love to test it on my Nvidia Jetson Nano node. My previous attempts failed quite miserably :)

[Vaskozl]

I can confirm that it works with k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0-beta,

The quay.io only supports amd64, we should switch to the gcr.io image by default.