Autoscaler: VPA 'no pods found'

[hprotzek]

We are running a vpa for Prometheus Operator instances on GKE 1.18.6-gke.3504. We thought after the issue #2159 got fixed, this should be working. But we now we get No recommendation provided from the recommendation engine: No pods found.

Other vpa configurations, with non operator resources are working as expected. Is this possible again related to the CRDs of the Prometheus Operator?

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
spec:
  targetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: prometheus-example
  updatePolicy:
    updateMode: Auto
status:
  conditions:
  - lastTransitionTime: "2020-09-22T06:14:30Z"
    message: 'No recommendation provided from the recommendation engine: No pods found.'
    reason: 'No recommendation provided from the recommendation engine: No pods found.'
    status: "False"
    type: RecommendationProvided
  recommendation: {}

apiVersion: apps/v1
kind: StatefulSet
  name: prometheus-example
  namespace: prometheus-example
  ownerReferences:
  - apiVersion: monitoring.coreos.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: Prometheus
    name: prometheus-example
    uid: 857262d5-e10a-4eff-8d9f-d84696821b5f
spec:
...

[bskiba]

@jbartosik can you check if the fix for #2159 is included in GKE 1.18.6-gke.3504?

[bskiba]

I've checked and the mentioned fix will be available starting from version 1.18.6-gke.5600 which means it's not yet available. Would it be possible for you to check back with the newer version when it comes out?

[hprotzek]

Thanks @bskiba for checking that. I'll wait until 1.18.6-gke.5600 and will report back.

[hprotzek]

@bskiba After upgrading GKE to 1.18.9-gke.801 the vpa config for the Prometheus Operator resources are working as expected. Thank you for helping :-)

[bskiba]

Thanks for confirming! Glad it helped.