Autoscaler: VPA: NoPodsMatched

[kam1kaze]

HI,

For some reason VPA shows NoPodsMatched status for running POD. Also it correctly calculates Recommendation for such POD.

Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-gke.7", GitCommit:"f264644fff244bce8ca71b3344f8cc239af7bc37", GitTreeState:"clean", BuildDate:"2019-09-20T23:16:19Z", GoVersion:"go1.11.13b4", Compiler:"gc", Platform:"linux/amd64"}
VPA version is 0.6.3 with --min-replicas=1 option 

For example:

❯ kubectl get vpa lv-mysqlsync -o yaml
apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"lv-mysqlsync","app.kubernetes.io/managed-by":"Tiller","app.kubernetes.io/name":"mysqlsync","helm.sh/chart":"mysqlsync-0.1.0"},"name":"lv-mysqlsync","namespace":"default"},"spec":{"resourcePolicy":{"containerPolicies":[{"containerName":"mysqlsync","maxAllowed":{"cpu":"5000m","memory":"2Gi"},"minAllowed":{"cpu":"500m","memory":"1Gi"}}]},"targetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"lv-mysqlsync"},"updatePolicy":{"updateMode":"auto"}}}
  creationTimestamp: "2019-10-17T20:19:32Z"
  generation: 81
  labels:
    app.kubernetes.io/instance: lv-mysqlsync
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: mysqlsync
    helm.sh/chart: mysqlsync-0.1.0
  name: lv-mysqlsync
  namespace: default
  resourceVersion: "46236527"
  selfLink: /apis/autoscaling.k8s.io/v1beta2/namespaces/default/verticalpodautoscalers/lv-mysqlsync
  uid: 6e5a6de3-f11b-11e9-b4e6-42010aca0008
spec:
  resourcePolicy:
    containerPolicies:
    - containerName: mysqlsync
      maxAllowed:
        cpu: 5000m
        memory: 2Gi
      minAllowed:
        cpu: 500m
        memory: 1Gi
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: lv-mysqlsync
  updatePolicy:
    updateMode: Auto
status:
  conditions:
  - lastTransitionTime: "2019-10-17T22:32:57Z"
    message: No pods match this VPA object
    reason: NoPodsMatched
    status: "True"
    type: NoPodsMatched
  - lastTransitionTime: "2019-10-17T22:32:57Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: mysqlsync
      lowerBound:
        cpu: 1213m
        memory: "1799606177"
      target:
        cpu: 1388m
        memory: "1836551791"
      uncappedTarget:
        cpu: 1388m
        memory: "1836551791"
      upperBound:
        cpu: "5"
        memory: 2Gi

I see such error in vpa-admission-controller logs

2019-10-18 02:58:57.070 EEST
vpa-admission-controller
] [] nil [] []} {&CrossVersionObjectReference{Kind:Deployment,Name:lv-mysqlsync,APIVersion:apps/v1,} 0xc00000c8e0 0xc006112f40} {0xc006113060 [{NoPodsMatched True 2019-10-17 22:32:57 +0000 UTC NoPodsMatched No pods match this VPA object} {RecommendationProvided True 2019-10-17 22:32:57 +0000 UTC }]}}

and at the same time vpa-recommender uses currect selector and calculates correct data

2019-10-18 02:58:56.673 EEST
vpa-recommender
Using selector app.kubernetes.io/instance=lv-mysqlsync,app.kubernetes.io/name=mysqlsync for VPA default/lv-mysqlsync

Deployment:

❯ kubectl get deployments.apps lv-mysqlsync -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "10"
  creationTimestamp: "2019-08-29T15:11:54Z"
  generation: 10
  labels:
    app.kubernetes.io/instance: lv-mysqlsync
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: mysqlsync
    helm.sh/chart: mysqlsync-0.1.0
  name: lv-mysqlsync
  namespace: default
  resourceVersion: "45893673"
  selfLink: /apis/apps/v1/namespaces/default/deployments/lv-mysqlsync
  uid: 55c6d06d-ca6f-11e9-bd80-42010aca0008
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: lv-mysqlsync
      app.kubernetes.io/name: mysqlsync
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: lv-mysqlsync
        app.kubernetes.io/name: mysqlsync
    spec:
      containers:
...

[bskiba]

There is some combination of events that causes the condition to be set uncorrectly, and I have not yet identified what it is. (Most of the times it seems to work correctly).
Do you have more than one VPA object defined? Have you observed it multiple times or just one instance?

[calixwu]

Same issue. Actually I found this happens very oftenly.

[bskiba]

Thanks! Then it should be easy to reproduce, I'll try to get to this next week

[NeilW]

I've probably hit the same issue. The recommender seems to pick up the selector properly, but the admission controller says no pods matched and the updater doesn't then apply the recommendation.

The fault arises with the standard hamster yaml file if you keep removing and re-creating the VPA. If you then remove the deployment and redeploy, the VPA picks the pods up again.

[PatricioDanos]

Same as @NeilW I've gotten to the same issue. I had a deployment, and then I created my VPA, and even though it created the recommendation, it didn't find any Pods. However, I deleted and recreated the deployment, and it worked just fine using the exact same specs.

In my case, I was using the Microsoft AKS example https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-application for my testings.

EDIT: Spoke too soon. It's showing NoPodsMatched.

Processing vpa: &{{VerticalPodAutoscaler autoscaling.k8s.io/v1beta2} {my-vpa default d2591616-d52a-47e6-97ea-2d2eacfd1dca 13105 53 2019-11-05 14:02:35 ││ +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"my-vpa ││ ","namespace":"default"},"spec":{"targetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"azure-vote-front"},"updatePolicy":{"updateMode":"Auto"}}} ││ ] [] nil [] []} {&CrossVersionObjectReference{Kind:Deployment,Name:azure-vote-front,APIVersion:apps/v1,} 0xc000432838 <nil>} {0xc0006b2940 [{NoPodsMatched True 2019-11-05 14:03:24 +0000 UTC NoPodsM ││ atched No pods match this VPA object} {RecommendationProvided True 2019-11-05 14:44:24 +0000 UTC }]}}

[bskiba]

@PatricioDanos I believe that the logic itself is correct (VPA sees the pods and calculates the right recommendation for you) but it sets the NoPodsMatched condition incorrectly. (I don't know the root cause yet)

@NeilW Can you describe in more detail what you observe? Ideally can you paste your VPA object status when the issue is there? Is the recommendation present? Are pods getting new resource requests assigned?
This issue is to my current understanding about VPA working correctly, but incorrectly marking the VPA object with NoPodsMatched=True

[NeilW]

I get it by deploying the standard hamster example VPA on a clean cluster

$ kubectl describe vpa/hamster-vpa
Name:         hamster-vpa
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"hamster-vpa","namespace":"d...
API Version:  autoscaling.k8s.io/v1beta2
Kind:         VerticalPodAutoscaler
Metadata:
  Creation Timestamp:  2019-11-07T15:04:55Z
  Generation:          2
  Resource Version:    2548021
  Self Link:           /apis/autoscaling.k8s.io/v1beta2/namespaces/default/verticalpodautoscalers/hamster-vpa
  UID:                 08e5a6b5-a581-407e-b996-8dc86887778f
Spec:
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         hamster
  Update Policy:
    Update Mode:  Auto
Status:
  Conditions:
    Last Transition Time:  2019-11-07T15:05:25Z
    Message:               No pods match this VPA object
    Reason:                NoPodsMatched
    Status:                True
    Type:                  NoPodsMatched
    Last Transition Time:  2019-11-07T15:05:25Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  hamster
      Lower Bound:
        Cpu:     585m
        Memory:  262144k
      Target:
        Cpu:     587m
        Memory:  262144k
      Uncapped Target:
        Cpu:     587m
        Memory:  262144k
      Upper Bound:
        Cpu:     1432m
        Memory:  262144k
Events:          <none>

It looks like pods are still getting recommendations, and that they get assigned over time.

[bskiba]

Yeah, so I believe this is a reporting problem not a problem with a logic per-se. I'll try to root cause it and if I can't in a reasonable time, I'll consider removing the condition since it mostly confuses people, while the idea was exactly the opposite :(

[anikafuchs]

Hello, is there any update on how to avoid the condition to become NoPodsMatched status?
I am facing the same issues:

Status:
Conditions:
Last Transition Time: 2020-03-19T10:12:48Z
Message: No pods match this VPA object
Reason: NoPodsMatched
Status: True
Type: NoPodsMatched
Last Transition Time: 2020-03-19T10:12:48Z
Status: True
Type: RecommendationProvided

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kinghrothgar]

Any ETA on this? This makes it hard to have monitoring for mis-configured VPAs

[bskiba]

/remove-lifecycle stale
I'm hoping that #3156 helps with the issue, but it's not released yet. I am planning a release this month.

[bskiba]

/lifecycle frozen