Pulumi-kubernetes: Headless Service for StatefulSet hangs on create/update "Finding Pods to direct traffic to"
Headless service required for StatefulSets with appropriate label selectors and clusterIP set to "None"tries to wait for pods. Workaround is to abort the create, do a refresh, and then subsequent updates are ok.
protometa
All 26 comments
I'm still having this issue with the latest Pulumi. The StatefulSet headless service still does have a pod selector. I don't think this case was addressed in the merge.
protometa
on 19 Dec 2018
I'm also having this issue. I think for headless StatefulSet service Pulumi shouldn't wait for any pods even if the service spec includes selector.
pbzdyl
on 8 Jan 2019
THis seems to have been closed erroneously... I must have searched for issues with "headless" in them and selected this instead of a headless service issue. We'll fix soon.
hausdorff
on 8 Jan 2019
If I understand the problem correctly, I believe this issue is fixed in master already by https://github.com/pulumi/pulumi-kubernetes/pull/307
Prior to that change, the await logic didn't handle StatefulSet, so Pulumi would show it as ready as soon as it was created.
I took another look at the Service awaiter code, and it seems to be valid according to the current semantics; as you noted, the Service will not show ready until all selected Pods are ready, even for a headless Service.
We'll cut a new release soon, and if that doesn't fix your problem, I'll need some more detail on expected vs actual results you're seeing.
lblackstone
on 8 Jan 2019
@lblackstone sounds like we should close the issue and let @protometa and @pbzdyl re-open if it persists?
hausdorff
on 9 Jan 2019
@hausdorff I would like to test but has the fix been released? I updated all npm packages and the issue is still there.
pbzdyl
on 14 Jan 2019
@pbzdyl we're releasing today, I had hoped to have you try a dev release, though. Either way, though.
hausdorff
on 14 Jan 2019
@hausdorff I can try a dev release but I would need some guidance how to do it :)
pbzdyl
on 14 Jan 2019
@pbzdyl you can just put "@pulumi/kubernetes": "dev" as the version in package.json
hausdorff
on 14 Jan 2019
@hausdorff I tested and it seems it still doesn't work. I have changed the version to dev as you suggested, removed npm_modules and package-lock.json:
```$ npm i
[email protected] install
/node_modules/grpc
node-pre-gyp install --fallback-to-build --library=static_library
node-pre-gyp WARN Using needle for node-pre-gyp https download
[grpc] Success: "
@pulumi/[email protected] install
/node_modules/@pulumi/gcp
node scripts/install-pulumi-plugin.js resource gcp v0.16.4
[resource plugin gcp-0.16.4] installing
@pulumi/[email protected] install
/node_modules/@pulumi/kubernetes
node scripts/install-pulumi-plugin.js resource kubernetes v0.18.1-dev.1546907245+g69d6a81
[resource plugin kubernetes-0.18.1-dev.1546907245+g69d6a81] installing
Downloading plugin: 16.12 MiB / 16.12 MiB [=======================] 100.00% 11s
[email protected] postinstall
/node_modules/protobufjs
node scripts/postinstall
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN services@ No description
npm WARN services@ No repository field.
npm WARN services@ No license field.
added 221 packages from 623 contributors and audited 897 packages in 23.819s
found 0 vulnerabilities
$ pulumi up
Previewing update (
Type Name Plan
pulumi:pulumi:Stack <redacted>
- ├─ pulumi:pulumi:StackReference
read
└─
└─
- └─ kubernetes:core:Service
create
Resources:
+ 1 to create
14 unchanged
Do you want to perform this update? yes
Updating (
Type Name Status Info
pulumi:pulumi:Stack <redacted>
- ├─ pulumi:pulumi:StackReference
read
└─
└─
- └─ kubernetes:core:Service
creating failed 1 error
Diagnostics:
kubernetes:core:Service (
error: Plan apply failed: 2 errors occurred:
* Timeout occurred for '<redacted>'
* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
Headless service:
```$ kubectl describe svc <redacted>
Name: <redacted>
Namespace: default
Labels: app=example
deployment=example
Annotations: <none>
Selector: app=example,deployment=example
Type: ClusterIP
IP: None
Session Affinity: None
Events: <none>
Pods:
kubectl get pods --selector app=example,deployment=example
NAME READY STATUS RESTARTS AGE
<redacted>-0 1/1 Running 0 22h
I can provide more information if needed.
pbzdyl
on 15 Jan 2019
@pbzdyl Sorry for the delay. I just released v0.19.0
Can you give it one more shot with that release, and then I'll follow up if you're still having problems?
lblackstone
on 15 Jan 2019
@lblackstone I have just tested v0.19.0 and I got the same issue and output as in my previous comment.
pbzdyl
on 16 Jan 2019
Let me also add .status info for the impacted StatefulSet and its pod:
StatefulSet.status:
status:
collisionCount: 0
currentReplicas: 1
currentRevision: <redacted>-f66f69fc
observedGeneration: 4
readyReplicas: 1
replicas: 1
updateRevision: <redacted>-f66f69fc
updatedReplicas: 1
Pod.status:
```
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2019-01-14T10:49:39Z
status: "True"
type: Initialized - lastProbeTime: null
lastTransitionTime: 2019-01-14T10:49:58Z
status: "True"
type: Ready - lastProbeTime: null
lastTransitionTime: null
status: "True"
type: ContainersReady - lastProbeTime: null
lastTransitionTime: 2019-01-14T10:49:39Z
status: "True"
type: PodScheduled
containerStatuses: - containerID: docker://4a4d134aa8b92e4bbdaa9c7325c5df01690d523416afae646529457907a39d24
image:
imageID:
lastState: {}
name:
ready: true
restartCount: 0
state:
running:
startedAt: 2019-01-14T10:49:58Z
hostIP:
phase: Running
podIP:
qosClass: Burstable
startTime: 2019-01-14T10:49:39Z```
pbzdyl
on 16 Jan 2019
@pbzdyl What labels do you have set for the StatefulSet/Pod? Your Service is selecting for app=example,deployment=example, so I would expect that to match.
All of the following should be identical:
- Service
.spec.selector.<label> - StatefulSet
.spec.selector.matchLabels.<label> - StatefulSet
.spec.template.metadata.labels.<label>
Reference: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#components
lblackstone
on 16 Jan 2019
@lblackstone here it is:
StatefulSet .spec.selector.matchLabels = { app: example, deployment: example }
StatefulSet .spec.template.metadata.labels = { app: example, deployment: example }
Service .spec.selector = { app: example, deployment: example }
Also:
$ kubectl get pods --selector app=example,deployment=example
NAME READY STATUS RESTARTS AGE
<redacted>-0 1/1 Running 0 2d
which shows that all the metadata and selectors are configured correctly.
pbzdyl
on 16 Jan 2019
@pbzdyl Alright, just wanted to confirm. I'll take another look at the provider code today.
lblackstone
on 16 Jan 2019
I will be happy to provide any other information you might need to troubleshoot this issue.
pbzdyl
on 16 Jan 2019
@pbzdyl I haven't been able to reproduce this behavior. I'm able to create a StatefulSet + headless Service on GKE like this:
Pulumi snippet
const ssLabels = {app: `ss-test`, deployment: `example`};
const ssName = "ss-service";
const ssService = new k8s.core.v1.Service("statefulservice", {
metadata: {name: ssName},
spec: {
ports: [{port: 80, name: "web"}],
clusterIP: "None",
selector: ssLabels,
}
}, {provider: k8sProvider});
const ss = new k8s.apps.v1.StatefulSet("statefulset", {
metadata: {labels: ssLabels, name: "foo"},
spec: {
selector: {matchLabels: ssLabels},
serviceName: ssName,
replicas: 3,
template: {
metadata: {labels: ssLabels},
spec: {
terminationGracePeriodSeconds: 10,
containers: [
{
name: "nginx",
image: "nginx:stable",
ports: [
{
containerPort: 80,
name: "web"
}
],
volumeMounts: [
{
name: "www",
mountPath: "/usr/share/nginx/html"
}
]
}
],
},
},
volumeClaimTemplates: [
{
metadata: {name: "www"},
spec: {
accessModes: ["ReadWriteOnce"],
resources: {requests: {storage: "1Gi"}}
}
}
]
},
}, {provider: k8sProvider});
StatefulSet info
{
"apiVersion": "apps/v1",
"kind": "StatefulSet",
"metadata": {
"creationTimestamp": "2019-01-17T20:23:37Z",
"generation": 1,
"labels": {
"app": "ss-test",
"deployment": "example"
},
"name": "foo",
"namespace": "default",
"resourceVersion": "20781",
"selfLink": "/apis/apps/v1/namespaces/default/statefulsets/foo",
"uid": "c595cfde-1a95-11e9-9fc4-42010a8a0120"
},
"spec": {
"podManagementPolicy": "OrderedReady",
"replicas": 3,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "ss-test",
"deployment": "example"
}
},
"serviceName": "ss-service",
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "ss-test",
"deployment": "example"
}
},
"spec": {
"containers": [
{
"image": "nginx:stable",
"imagePullPolicy": "IfNotPresent",
"name": "nginx",
"ports": [
{
"containerPort": 80,
"name": "web",
"protocol": "TCP"
}
],
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/usr/share/nginx/html",
"name": "www"
}
]
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"terminationGracePeriodSeconds": 10
}
},
"updateStrategy": {
"rollingUpdate": {
"partition": 0
},
"type": "RollingUpdate"
},
"volumeClaimTemplates": [
{
"metadata": {
"creationTimestamp": null,
"name": "www"
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "1Gi"
}
}
},
"status": {
"phase": "Pending"
}
}
]
},
"status": {
"collisionCount": 0,
"currentReplicas": 3,
"currentRevision": "foo-778b5999d8",
"observedGeneration": 1,
"readyReplicas": 3,
"replicas": 3,
"updateRevision": "foo-778b5999d8",
"updatedReplicas": 3
}
}
Service info
Name: ss-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=ss-test,deployment=example
Type: ClusterIP
IP: None
Port: web 80/TCP
TargetPort: 80/TCP
Endpoints: 10.40.1.10:80,10.40.1.9:80,10.40.2.8:80
Session Affinity: None
Events: <none>
The main difference I noticed from the output you provided is that your service doesn't include any Endpoints; not sure if you omitted that, or if they are missing. The await logic is based on the endpoints being present, so it may be a configuration problem in your program.
Happy to help troubleshoot if you can provide the relevant code snippet, either here or DM on our community slack channel.
lblackstone
on 17 Jan 2019
Thank you @lblackstone. This is it: I have a headless service without any ports but it is a valid use case for statefulsets: having such a service doesn't publish any ports but manages a DNS subdomain for DNS names of the statefulset pods. In such scenario there will be no service endpoint created for the service and there will be
Thus I think for headless services without any ports the awaiter logic must be different.
pbzdyl
on 18 Jan 2019
@pbzdyl Can you also verify that your StatefulSet's .spec.serviceNamematches the Service .metadata.name?
I still don't think we're on the same page. A headless Service will not have an external IP, but my understanding is that it should still have Endpoint(s) that point to the internal Pod IPs from your StatefulSet.
It would be really helpful to see the pulumi snippets that you're using to create the Service and StatefulSet (with values redacted is fine).
lblackstone
on 18 Jan 2019
Hi @lblackstone. Let me include all the info I have:
statefulApp.ts:
import * as pulumi from '@pulumi/pulumi'
import * as k8s from '@pulumi/kubernetes'
export interface StatefulAppArgs {
readonly namespace: pulumi.Input<string>
readonly name: pulumi.Input<string>
readonly dataVolumeSize: pulumi.Input<string>
readonly logsVolumeSize: pulumi.Input<string>
}
export class StatefulApp extends pulumi.ComponentResource {
readonly selector: pulumi.Input<{ [key: string]: pulumi.Input<string> }>
constructor(resourceName: string, args: StatefulAppArgs, opts: any) {
super('StatefulApp', resourceName, {}, opts)
this.selector = {
app: args.name
}
const headlessServiceName = `${args.name}-statefulset`
const metadata = {
namespace: args.namespace,
labels: this.selector
}
new k8s.apps.v1.StatefulSet(resourceName, {
metadata: {
...metadata,
name: args.name
},
spec: {
serviceName: headlessServiceName,
selector: { matchLabels: metadata.labels },
template: {
metadata: {
...metadata,
name: args.name
},
spec: {
securityContext: { fsGroup: 100 },
containers: [{
name: args.name,
image: 'busybox',
command: ['sleep', '3600000'],
volumeMounts: [
{ name: 'data', mountPath: '/app/data' },
{ name: 'logs', mountPath: '/ap/logs' }
]
}]
}
},
volumeClaimTemplates: [
{
metadata: { name: 'data' },
spec: {
accessModes: ['ReadWriteOnce'],
storageClassName: 'standard',
resources: { requests: { storage: args.dataVolumeSize } }
}
},
{
metadata: { name: 'logs' },
spec: {
accessModes: ['ReadWriteOnce'],
storageClassName: 'standard',
resources: { requests: { storage: args.logsVolumeSize } }
}
}
]
}
}, {
parent: this,
provider: opts.providers.kubernetes
})
new k8s.core.v1.Service(`${resourceName}-headless`, {
apiVersion: 'v1',
kind: 'Service',
metadata: {
...metadata,
name: headlessServiceName
},
spec: {
clusterIP: 'None',
selector: metadata.labels
}
}, {
parent: this,
provider: opts.providers.kubernetes
})
}
}
Used in index.ts like that:
import { StatefulApp } from './statefulApp'
import { config } from './config'
new StatefulApp('example', {
namespace: 'default',
name: 'example',
dataVolumeSize: '1Gi',
logsVolumeSize: '1Gi',
}, {
providers: {
kubernetes: config.k8sProvider
}
})
$ pulumi up
Previewing update (<redacted>):
Type Name Plan
pulumi:pulumi:Stack <redacted>
+ ├─ StatefulApp example create
+ │ ├─ kubernetes:core:Service example-headless create
+ │ └─ kubernetes:apps:StatefulSet example create
>- └─ pulumi:pulumi:StackReference <redacted> read
Resources:
+ 3 to create
15 unchanged
Do you want to perform this update? yes
Updating (<redacted>):
Type Name Status Info
pulumi:pulumi:Stack <redacted>
+ ├─ StatefulApp example created
+ │ ├─ kubernetes:core:Service example-headless **creating failed** 1 error
+ │ └─ kubernetes:apps:StatefulSet example created
>- └─ pulumi:pulumi:StackReference <redacted> read
Diagnostics:
kubernetes:core:Service (example-headless):
error: Plan apply failed: 2 errors occurred:
* Timeout occurred for 'example-statefulset'
* Service does not target any Pods. Selected Pods may not be ready, or field '.spec.selector' may not match labels on any Pods
Resources:
+ 2 created
15 unchanged
Duration: 11m3s
Permalink: https://app.pulumi.com/<redacted>/<redacted>/updates/44
error: update failed
$ kubectl get statefulset -o yaml example
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: 2019-01-19T09:41:03Z
generation: 1
labels:
app: example
name: example
namespace: default
resourceVersion: "408683"
selfLink: /apis/apps/v1/namespaces/default/statefulsets/example
uid: 5640bbc7-1bce-11e9-99cd-4201ac100387
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: example
serviceName: example-statefulset
template:
metadata:
creationTimestamp: null
labels:
app: example
name: example
namespace: default
spec:
containers:
- command:
- sleep
- "3600000"
image: busybox
imagePullPolicy: Always
name: example
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /app/data
name: data
- mountPath: /ap/logs
name: logs
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 100
terminationGracePeriodSeconds: 30
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:
- metadata:
creationTimestamp: null
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
volumeMode: Filesystem
status:
phase: Pending
- metadata:
creationTimestamp: null
name: logs
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
volumeMode: Filesystem
status:
phase: Pending
status:
collisionCount: 0
currentReplicas: 1
currentRevision: example-68477db846
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: example-68477db846
updatedReplicas: 1
kubectl get pod -o yaml example-0
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
example'
creationTimestamp: 2019-01-19T09:41:03Z
generateName: example-
labels:
app: example
controller-revision-hash: example-68477db846
statefulset.kubernetes.io/pod-name: example-0
name: example-0
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: example
uid: 5640bbc7-1bce-11e9-99cd-4201ac100387
resourceVersion: "408682"
selfLink: /api/v1/namespaces/default/pods/example-0
uid: 5644c264-1bce-11e9-99cd-4201ac100387
spec:
containers:
- command:
- sleep
- "3600000"
image: busybox
imagePullPolicy: Always
name: example
resources:
requests:
cpu: 100m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /app/data
name: data
- mountPath: /ap/logs
name: logs
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-5d6lx
readOnly: true
dnsPolicy: ClusterFirst
hostname: example-0
nodeName: gke-default-pool-1-5f32c1bf-8hfl
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 100
serviceAccount: default
serviceAccountName: default
subdomain: example-statefulset
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: data
persistentVolumeClaim:
claimName: data-example-0
- name: logs
persistentVolumeClaim:
claimName: logs-example-0
- name: default-token-5d6lx
secret:
defaultMode: 420
secretName: default-token-5d6lx
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2019-01-19T09:42:04Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2019-01-19T09:44:36Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: null
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: 2019-01-19T09:42:04Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://1237cec318b10ca01c35693cd30bd2409f1832d0d54b430629002db774c9c216
image: docker.io/library/busybox:latest
imageID: docker.io/library/busybox@sha256:bbb143159af9eabdf45511fd5aab4fd2475d4c0e7fd4a5e154b98e838488e510
lastState: {}
name: example
ready: true
restartCount: 0
state:
running:
startedAt: 2019-01-19T09:44:35Z
hostIP: 10.88.0.22
phase: Running
podIP: 10.89.2.3
qosClass: Burstable
startTime: 2019-01-19T09:42:04Z
kubectl get svc -o yaml example-statefulset
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2019-01-19T09:40:56Z
labels:
app: example
name: example-statefulset
namespace: default
resourceVersion: "407942"
selfLink: /api/v1/namespaces/default/services/example-statefulset
uid: 51ed105a-1bce-11e9-99cd-4201ac100387
spec:
clusterIP: None
selector:
app: example
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Dependencies:
"@pulumi/gcp": "^0.16.5"
"@pulumi/kubernetes": "^0.19.0"
"@pulumi/pulumi": "^0.16.11"
I think this might be caused by a bug in Kubernetes API <1.12 and it was fixed in 1.12 by this PR.
I guess it will take a while until Kubernetes 1.12 is available in GKE so it would be very helpful to have the awaiter logic changed for Kubernetes API with version <1.12 so it doesn't block pulumi update for so long when there is a headless service without ports.
pbzdyl
on 19 Jan 2019
@pbzdyl Interesting! Thanks for the updated info. I'll take another look today.
lblackstone
on 22 Jan 2019
@pbzdyl I just wanted to say, thank you so much for being so persistent! If you'd like a shirt or a beanie, let me know and I'll send it your way. Drop me a line @ [email protected]
hausdorff
on 23 Jan 2019
@pbzdyl You should be able to test out this fix in the latest dev release.(@pulumi/[email protected])
lblackstone
on 25 Jan 2019
It worked - thank you @lblackstone.
pbzdyl
on 26 Jan 2019
Related issues
HagaiBarel
·
5Comments
oliverholliday
·
9Comments
dotansimha
·
4Comments
hausdorff
·
9Comments
zarkone
·
10Comments
Most helpful comment
It worked - thank you @lblackstone.