Lens: Metrics is not available when using AWS EKS + Weavenet

[getvivekv]

Describe the bug
The metrics feature is not available when on EKS cluster with weavenet as the CNI used. This works fine on bare metal installation. After removing the AWS CNI from the EKS cluster, the installation of the Metrics server requires the networking to change to HostNetwork = true and communication IP = Internal. I have noticed that Lens uses its own Metrics server and Prometheus so there may be some additional tweaks required.

To Reproduce
Steps to reproduce the behavior:

1. Setup a cluster on AWS EKS
2. Remove AWS CNI
3. Install Weavenet
4. Connect using Lens and install the inbuilt metrics server

Expected behavior
Lens UI shows the metrics

Screenshots
Lens UI doesn't show metrics and complains " Metrics not available at the moment"

Environment (please complete the following information):

• Lens Version: 3.5.1
• OS: [e.g. OSX] Windows

[toabi]

Funny, I just replaced the AWS VPC CNI with Calico and had the same effect… Metrics don't work and the whole thing is generally more sluggish than before. It worked before so it really has to do something with CNIs.

Lens 3.5.2 on macOS

[simwak]

Exactly the same with EKS and Cilium. No metrics and a lot slower than before.

Lens 3.5.2 on Windows

[toabi]

Yes, after a few days using Lens with a EKS+CNI cluster it's pretty close to unusable actually :(

I wonder if it's so slow because it tries to always fetch some metrics which it can't?

[simwak]

Is it in overlay mode or the EKS standard CNI mode (every Pod get's a VPC routable adress)?
Secondary private IPs on your worker nodes mean you use the EKS CNI mode.

[toabi]

No it's a full replacement with overlay network. Pods get some 192.168.* addresses.

[Siddharthk]

same issue using Lens 3.5.3 with EKS using AWS CNI + Calico

[toabi]

The issue still exists in 3.6

[mordax777]

Problem still persist in 3.6.8

[gelblars]

Workaround: just edit the StatefulSet of prometheus and add hostNetwork: true to the spec section

...
spec:
  replicas: 1
  selector:
    matchLabels:
      name: prometheus
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: prometheus
    spec:
      volumes:
        - name: config
          configMap:
            name: prometheus-config
            defaultMode: 420
        - name: rules
          configMap:
            name: prometheus-rules
            defaultMode: 420
      initContainers:
        - name: chown
          image: 'docker.io/alpine:3.9'
          command:
            - chown
            - '-R'
            - '65534:65534'
            - /var/lib/prometheus
          resources: {}
          volumeMounts:
            - name: data
              mountPath: /var/lib/prometheus
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      containers:
        - name: prometheus
          image: 'docker.io/prom/prometheus:v2.17.2'
          args:
            - '--web.listen-address=0.0.0.0:9090'
            - '--config.file=/etc/prometheus/prometheus.yaml'
            - '--storage.tsdb.path=/var/lib/prometheus'
            - '--storage.tsdb.retention.time=2d'
            - '--storage.tsdb.retention.size=5GB'
            - '--storage.tsdb.min-block-duration=2h'
            - '--storage.tsdb.max-block-duration=2h'
          ports:
            - name: web
              hostPort: 9090
              containerPort: 9090
              protocol: TCP
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: rules
              mountPath: /etc/prometheus/rules
            - name: data
              mountPath: /var/lib/prometheus
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 9090
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 10
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /-/ready
              port: 9090
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 10
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: prometheus
      serviceAccount: prometheus
      hostNetwork: true
      securityContext: {}
...