HI,
For some reason VPA shows NoPodsMatched status for running POD. Also it correctly calculates Recommendation for such POD.
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-gke.7", GitCommit:"f264644fff244bce8ca71b3344f8cc239af7bc37", GitTreeState:"clean", BuildDate:"2019-09-20T23:16:19Z", GoVersion:"go1.11.13b4", Compiler:"gc", Platform:"linux/amd64"}
VPA version is 0.6.3 with --min-replicas=1 option
For example:
❯ kubectl get vpa lv-mysqlsync -o yaml
apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"lv-mysqlsync","app.kubernetes.io/managed-by":"Tiller","app.kubernetes.io/name":"mysqlsync","helm.sh/chart":"mysqlsync-0.1.0"},"name":"lv-mysqlsync","namespace":"default"},"spec":{"resourcePolicy":{"containerPolicies":[{"containerName":"mysqlsync","maxAllowed":{"cpu":"5000m","memory":"2Gi"},"minAllowed":{"cpu":"500m","memory":"1Gi"}}]},"targetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"lv-mysqlsync"},"updatePolicy":{"updateMode":"auto"}}}
creationTimestamp: "2019-10-17T20:19:32Z"
generation: 81
labels:
app.kubernetes.io/instance: lv-mysqlsync
app.kubernetes.io/managed-by: Tiller
app.kubernetes.io/name: mysqlsync
helm.sh/chart: mysqlsync-0.1.0
name: lv-mysqlsync
namespace: default
resourceVersion: "46236527"
selfLink: /apis/autoscaling.k8s.io/v1beta2/namespaces/default/verticalpodautoscalers/lv-mysqlsync
uid: 6e5a6de3-f11b-11e9-b4e6-42010aca0008
spec:
resourcePolicy:
containerPolicies:
- containerName: mysqlsync
maxAllowed:
cpu: 5000m
memory: 2Gi
minAllowed:
cpu: 500m
memory: 1Gi
targetRef:
apiVersion: apps/v1
kind: Deployment
name: lv-mysqlsync
updatePolicy:
updateMode: Auto
status:
conditions:
- lastTransitionTime: "2019-10-17T22:32:57Z"
message: No pods match this VPA object
reason: NoPodsMatched
status: "True"
type: NoPodsMatched
- lastTransitionTime: "2019-10-17T22:32:57Z"
status: "True"
type: RecommendationProvided
recommendation:
containerRecommendations:
- containerName: mysqlsync
lowerBound:
cpu: 1213m
memory: "1799606177"
target:
cpu: 1388m
memory: "1836551791"
uncappedTarget:
cpu: 1388m
memory: "1836551791"
upperBound:
cpu: "5"
memory: 2Gi
I see such error in vpa-admission-controller logs
2019-10-18 02:58:57.070 EEST
vpa-admission-controller
] [] nil [] []} {&CrossVersionObjectReference{Kind:Deployment,Name:lv-mysqlsync,APIVersion:apps/v1,} 0xc00000c8e0 0xc006112f40} {0xc006113060 [{NoPodsMatched True 2019-10-17 22:32:57 +0000 UTC NoPodsMatched No pods match this VPA object} {RecommendationProvided True 2019-10-17 22:32:57 +0000 UTC }]}}
and at the same time vpa-recommender uses currect selector and calculates correct data
2019-10-18 02:58:56.673 EEST
vpa-recommender
Using selector app.kubernetes.io/instance=lv-mysqlsync,app.kubernetes.io/name=mysqlsync for VPA default/lv-mysqlsync
Deployment:
❯ kubectl get deployments.apps lv-mysqlsync -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "10"
creationTimestamp: "2019-08-29T15:11:54Z"
generation: 10
labels:
app.kubernetes.io/instance: lv-mysqlsync
app.kubernetes.io/managed-by: Tiller
app.kubernetes.io/name: mysqlsync
helm.sh/chart: mysqlsync-0.1.0
name: lv-mysqlsync
namespace: default
resourceVersion: "45893673"
selfLink: /apis/apps/v1/namespaces/default/deployments/lv-mysqlsync
uid: 55c6d06d-ca6f-11e9-bd80-42010aca0008
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: lv-mysqlsync
app.kubernetes.io/name: mysqlsync
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: lv-mysqlsync
app.kubernetes.io/name: mysqlsync
spec:
containers:
...
kam1kaze
All 13 comments
There is some combination of events that causes the condition to be set uncorrectly, and I have not yet identified what it is. (Most of the times it seems to work correctly).
Do you have more than one VPA object defined? Have you observed it multiple times or just one instance?
bskiba
on 22 Oct 2019
Same issue. Actually I found this happens very oftenly.
calixwu
on 23 Oct 2019
Thanks! Then it should be easy to reproduce, I'll try to get to this next week
bskiba
on 23 Oct 2019
I've probably hit the same issue. The recommender seems to pick up the selector properly, but the admission controller says no pods matched and the updater doesn't then apply the recommendation.
The fault arises with the standard hamster yaml file if you keep removing and re-creating the VPA. If you then remove the deployment and redeploy, the VPA picks the pods up again.
NeilW
on 5 Nov 2019
Same as @NeilW I've gotten to the same issue. I had a deployment, and then I created my VPA, and even though it created the recommendation, it didn't find any Pods. However, I deleted and recreated the deployment, and it worked just fine using the exact same specs.
In my case, I was using the Microsoft AKS example https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-application for my testings.
EDIT: Spoke too soon. It's showing NoPodsMatched.
Processing vpa: &{{VerticalPodAutoscaler autoscaling.k8s.io/v1beta2} {my-vpa default d2591616-d52a-47e6-97ea-2d2eacfd1dca 13105 53 2019-11-05 14:02:35 ││ +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"my-vpa ││ ","namespace":"default"},"spec":{"targetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"azure-vote-front"},"updatePolicy":{"updateMode":"Auto"}}} ││ ] [] nil [] []} {&CrossVersionObjectReference{Kind:Deployment,Name:azure-vote-front,APIVersion:apps/v1,} 0xc000432838 <nil>} {0xc0006b2940 [{NoPodsMatched True 2019-11-05 14:03:24 +0000 UTC NoPodsM ││ atched No pods match this VPA object} {RecommendationProvided True 2019-11-05 14:44:24 +0000 UTC }]}}
PatricioDanos
on 5 Nov 2019
@PatricioDanos I believe that the logic itself is correct (VPA sees the pods and calculates the right recommendation for you) but it sets the NoPodsMatched condition incorrectly. (I don't know the root cause yet)
@NeilW Can you describe in more detail what you observe? Ideally can you paste your VPA object status when the issue is there? Is the recommendation present? Are pods getting new resource requests assigned?
This issue is to my current understanding about VPA working correctly, but incorrectly marking the VPA object with NoPodsMatched=True
bskiba
on 7 Nov 2019
I get it by deploying the standard hamster example VPA on a clean cluster
$ kubectl describe vpa/hamster-vpa
Name: hamster-vpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"hamster-vpa","namespace":"d...
API Version: autoscaling.k8s.io/v1beta2
Kind: VerticalPodAutoscaler
Metadata:
Creation Timestamp: 2019-11-07T15:04:55Z
Generation: 2
Resource Version: 2548021
Self Link: /apis/autoscaling.k8s.io/v1beta2/namespaces/default/verticalpodautoscalers/hamster-vpa
UID: 08e5a6b5-a581-407e-b996-8dc86887778f
Spec:
Target Ref:
API Version: apps/v1
Kind: Deployment
Name: hamster
Update Policy:
Update Mode: Auto
Status:
Conditions:
Last Transition Time: 2019-11-07T15:05:25Z
Message: No pods match this VPA object
Reason: NoPodsMatched
Status: True
Type: NoPodsMatched
Last Transition Time: 2019-11-07T15:05:25Z
Status: True
Type: RecommendationProvided
Recommendation:
Container Recommendations:
Container Name: hamster
Lower Bound:
Cpu: 585m
Memory: 262144k
Target:
Cpu: 587m
Memory: 262144k
Uncapped Target:
Cpu: 587m
Memory: 262144k
Upper Bound:
Cpu: 1432m
Memory: 262144k
Events: <none>
It looks like pods are still getting recommendations, and that they get assigned over time.
NeilW
on 7 Nov 2019
Yeah, so I believe this is a reporting problem not a problem with a logic per-se. I'll try to root cause it and if I can't in a reasonable time, I'll consider removing the condition since it mostly confuses people, while the idea was exactly the opposite :(
bskiba
on 8 Nov 2019
Hello, is there any update on how to avoid the condition to become NoPodsMatched status?
I am facing the same issues:
Status:
Conditions:
Last Transition Time: 2020-03-19T10:12:48Z
Message: No pods match this VPA object
Reason: NoPodsMatched
Status: True
Type: NoPodsMatched
Last Transition Time: 2020-03-19T10:12:48Z
Status: True
Type: RecommendationProvided
anikafuchs
on 19 Mar 2020
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 17 Jun 2020
Any ETA on this? This makes it hard to have monitoring for mis-configured VPAs
kinghrothgar
on 30 Jun 2020
/remove-lifecycle stale
I'm hoping that #3156 helps with the issue, but it's not released yet. I am planning a release this month.
bskiba
on 15 Jul 2020
/lifecycle frozen
bskiba
on 15 Jul 2020
Related issues
hprotzek
·
5Comments
jadelafuente
·
4Comments
clamoriniere
·
5Comments
johanneswuerbach
·
5Comments
tjliupeng
·
6Comments
Most helpful comment
Yeah, so I believe this is a reporting problem not a problem with a logic per-se. I'll try to root cause it and if I can't in a reasonable time, I'll consider removing the condition since it mostly confuses people, while the idea was exactly the opposite :(