minikube 1.11.0: PVC stuck in pending, storage provisioner parse error
Issue Description
Persistent Volume Claims created by various operators (in this example, the Strimzi Kafka operator) remain stuck in the Pending state. Upon examination of the output of the storage-provisioner pod in kube-system, there appears to be a JSON parse error. The JSON appears to be valid, and the same operator-generated volume claim works as expected on other Kubernetes implementations (GKE, Docker for Mac)
Steps to reproduce the issue:
- Start Minikube: minikube start --driver=parallels (Mac OS 10.15, Parallels 15)
- Install Strimzi Kafka operator:
helm repo add strimzi https://strimzi.io/charts/
helm install strimzi strimzi/strimzi-kafka-operator --namespace kafka --set watchAnyNamespace=true - Deploy a Kafka instance:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
namespace: development
name: kafka
spec:
kafka:
version: 2.4.0
replicas: 1
listeners:
plain: {}
tls: {}
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
log.message.format.version: "2.3"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 1Gi
deleteClaim: true
zookeeper:
replicas: 1
storage:
type: persistent-claim
size: 1Gi
deleteClaim: true
Full output of failed command:
The following error is repeatedly emitted by the storage-provisioner pod:
First line: [storage-provisioner storage-provisioner] E0530 04:50:23.157132 1 reflector.go:205] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:411: Failed to list *v1.PersistentVolumeClaim: v1.PersistentVolumeClaimList: Items: []v1.PersistentVolumeClaim: v1.PersistentVolumeClaim: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 5301 ...:{},"k:{\"... at {"kind"
Full error emitted by storage-provisioner
[storage-provisioner storage-provisioner] E0530 04:50:23.157132 1 reflector.go:205] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:411: Failed to list *v1.PersistentVolumeClaim: v1.PersistentVolumeClaimList: Items: []v1.PersistentVolumeClaim: v1.PersistentVolumeClaim: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 5301 ...:{},"k:{\"... at {"kind":"PersistentVolumeClaimList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/persistentvolumeclaims","resourceVersion":"2745"},"items":[{"metadata":{"name":"data-mysql-0","namespace":"development","selfLink":"/api/v1/namespaces/development/persistentvolumeclaims/data-mysql-0","uid":"b9bdebe8-117f-49ad-bb07-0eb1492192fe","resourceVersion":"1246","creationTimestamp":"2020-05-30T04:38:36Z","labels":{"kubedb.com/kind":"MySQL","kubedb.com/name":"mysql"},"annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:36Z\",\"renewTime\":\"2020-05-30T04:39:06Z\",\"leaderTransitions\":0}","pv.kubernetes.io/bind-completed":"yes","pv.kubernetes.io/bound-by-controller":"yes","volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:38:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:labels":{".":{},"f:kubedb.com/kind":{},"f:kubedb.com/name":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{},"f:volumeName":{}},"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}}},{"manager":"storage-provisioner","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:39:06Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:control-plane.alpha.kubernetes.io/leader":{}}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"2Gi"}},"volumeName":"pvc-b9bdebe8-117f-49ad-bb07-0eb1492192fe","storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Bound","accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"}}},{"metadata":{"name":"data-redis-0","namespace":"development","selfLink":"/api/v1/namespaces/development/persistentvolumeclaims/data-redis-0","uid":"4e064ada-dfb7-48de-9093-111dc45e0651","resourceVersion":"1299","creationTimestamp":"2020-05-30T04:38:55Z","labels":{"kubedb.com/kind":"Redis","kubedb.com/name":"redis"},"annotations":{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:55Z\",\"renewTime\":\"2020-05-30T04:39:25Z\",\"leaderTransitions\":0}","pv.kubernetes.io/bind-completed":"yes","pv.kubernetes.io/bound-by-controller":"yes","volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:38:55Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:pv.kubernetes.io/bind-completed":{},"f:pv.kubernetes.io/bound-by-controller":{},"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:labels":{".":{},"f:kubedb.com/kind":{},"f:kubedb.com/name":{}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{},"f:volumeName":{}},"f:status":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:phase":{}}}},{"manager":"storage-provisioner","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:39:25Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:control-plane.alpha.kubernetes.io/leader":{}}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"volumeName":"pvc-4e064ada-dfb7-48de-9093-111dc45e0651","storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Bound","accessModes":["ReadWriteOnce"],"capacity":{"storage":"1Gi"}}},{"metadata":{"name":"data-kafka-zookeeper-0","namespace":"development","selfLink":"/api/v1/namespaces/development/persistentvolumeclaims/data-kafka-zookeeper-0","uid":"fabbaafe-f1d9-40e3-a342-0e2e95e5c83f","resourceVersion":"2745","creationTimestamp":"2020-05-30T04:43:44Z","labels":{"app.kubernetes.io/instance":"kafka","app.kubernetes.io/managed-by":"strimzi-cluster-operator","app.kubernetes.io/name":"zookeeper","app.kubernetes.io/part-of":"strimzi-kafka","strimzi.io/cluster":"kafka","strimzi.io/kind":"Kafka","strimzi.io/name":"kafka-zookeeper"},"annotations":{"strimzi.io/delete-claim":"true","volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"ownerReferences":[{"apiVersion":"kafka.strimzi.io/v1beta1","kind":"Kafka","name":"kafka","uid":"599d91ef-9823-47ba-a67c-bd1c447c6d6f","controller":false,"blockOwnerDeletion":false}],"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"okhttp","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:43:44Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:strimzi.io/delete-claim":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:strimzi.io/cluster":{},"f:strimzi.io/kind":{},"f:strimzi.io/name":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"599d91ef-9823-47ba-a67c-bd1c447c6d6f\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}},"f:status":{"f:phase":{}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-05-30T04:48:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:finalizers":{".":{},"v:\"kubernetes.io/pvc-protection\"":{}}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Pending"}}]}
Pretty-printed JSON:
{
"kind": "PersistentVolumeClaimList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/persistentvolumeclaims",
"resourceVersion": "2745"
},
"items": [
{
"metadata": {
"name": "data-mysql-0",
"namespace": "development",
"selfLink": "/api/v1/namespaces/development/persistentvolumeclaims/data-mysql-0",
"uid": "b9bdebe8-117f-49ad-bb07-0eb1492192fe",
"resourceVersion": "1246",
"creationTimestamp": "2020-05-30T04:38:36Z",
"labels": {
"kubedb.com/kind": "MySQL",
"kubedb.com/name": "mysql"
},
"annotations": {
"control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:36Z\",\"renewTime\":\"2020-05-30T04:39:06Z\",\"leaderTransitions\":0}",
"pv.kubernetes.io/bind-completed": "yes",
"pv.kubernetes.io/bound-by-controller": "yes",
"volume.beta.kubernetes.io/storage-provisioner": "k8s.io/minikube-hostpath"
},
"finalizers": [
"kubernetes.io/pvc-protection"
],
"managedFields": [
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "v1",
"time": "2020-05-30T04:38:36Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:pv.kubernetes.io/bind-completed": {},
"f:pv.kubernetes.io/bound-by-controller": {},
"f:volume.beta.kubernetes.io/storage-provisioner": {}
},
"f:labels": {
".": {},
"f:kubedb.com/kind": {},
"f:kubedb.com/name": {}
}
},
"f:spec": {
"f:accessModes": {},
"f:resources": {
"f:requests": {
".": {},
"f:storage": {}
}
},
"f:volumeMode": {},
"f:volumeName": {}
},
"f:status": {
"f:accessModes": {},
"f:capacity": {
".": {},
"f:storage": {}
},
"f:phase": {}
}
}
},
{
"manager": "storage-provisioner",
"operation": "Update",
"apiVersion": "v1",
"time": "2020-05-30T04:39:06Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:control-plane.alpha.kubernetes.io/leader": {}
}
}
}
}
]
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "2Gi"
}
},
"volumeName": "pvc-b9bdebe8-117f-49ad-bb07-0eb1492192fe",
"storageClassName": "standard",
"volumeMode": "Filesystem"
},
"status": {
"phase": "Bound",
"accessModes": [
"ReadWriteOnce"
],
"capacity": {
"storage": "2Gi"
}
}
},
{
"metadata": {
"name": "data-redis-0",
"namespace": "development",
"selfLink": "/api/v1/namespaces/development/persistentvolumeclaims/data-redis-0",
"uid": "4e064ada-dfb7-48de-9093-111dc45e0651",
"resourceVersion": "1299",
"creationTimestamp": "2020-05-30T04:38:55Z",
"labels": {
"kubedb.com/kind": "Redis",
"kubedb.com/name": "redis"
},
"annotations": {
"control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"1d078d69-a22f-11ea-a37d-001c42492410\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-05-30T04:38:55Z\",\"renewTime\":\"2020-05-30T04:39:25Z\",\"leaderTransitions\":0}",
"pv.kubernetes.io/bind-completed": "yes",
"pv.kubernetes.io/bound-by-controller": "yes",
"volume.beta.kubernetes.io/storage-provisioner": "k8s.io/minikube-hostpath"
},
"finalizers": [
"kubernetes.io/pvc-protection"
],
"managedFields": [
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "v1",
"time": "2020-05-30T04:38:55Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:pv.kubernetes.io/bind-completed": {},
"f:pv.kubernetes.io/bound-by-controller": {},
"f:volume.beta.kubernetes.io/storage-provisioner": {}
},
"f:labels": {
".": {},
"f:kubedb.com/kind": {},
"f:kubedb.com/name": {}
}
},
"f:spec": {
"f:accessModes": {},
"f:resources": {
"f:requests": {
".": {},
"f:storage": {}
}
},
"f:volumeMode": {},
"f:volumeName": {}
},
"f:status": {
"f:accessModes": {},
"f:capacity": {
".": {},
"f:storage": {}
},
"f:phase": {}
}
}
},
{
"manager": "storage-provisioner",
"operation": "Update",
"apiVersion": "v1",
"time": "2020-05-30T04:39:25Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:control-plane.alpha.kubernetes.io/leader": {}
}
}
}
}
]
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "1Gi"
}
},
"volumeName": "pvc-4e064ada-dfb7-48de-9093-111dc45e0651",
"storageClassName": "standard",
"volumeMode": "Filesystem"
},
"status": {
"phase": "Bound",
"accessModes": [
"ReadWriteOnce"
],
"capacity": {
"storage": "1Gi"
}
}
},
{
"metadata": {
"name": "data-kafka-zookeeper-0",
"namespace": "development",
"selfLink": "/api/v1/namespaces/development/persistentvolumeclaims/data-kafka-zookeeper-0",
"uid": "fabbaafe-f1d9-40e3-a342-0e2e95e5c83f",
"resourceVersion": "2745",
"creationTimestamp": "2020-05-30T04:43:44Z",
"labels": {
"app.kubernetes.io/instance": "kafka",
"app.kubernetes.io/managed-by": "strimzi-cluster-operator",
"app.kubernetes.io/name": "zookeeper",
"app.kubernetes.io/part-of": "strimzi-kafka",
"strimzi.io/cluster": "kafka",
"strimzi.io/kind": "Kafka",
"strimzi.io/name": "kafka-zookeeper"
},
"annotations": {
"strimzi.io/delete-claim": "true",
"volume.beta.kubernetes.io/storage-provisioner": "k8s.io/minikube-hostpath"
},
"ownerReferences": [
{
"apiVersion": "kafka.strimzi.io/v1beta1",
"kind": "Kafka",
"name": "kafka",
"uid": "599d91ef-9823-47ba-a67c-bd1c447c6d6f",
"controller": false,
"blockOwnerDeletion": false
}
],
"finalizers": [
"kubernetes.io/pvc-protection"
],
"managedFields": [
{
"manager": "okhttp",
"operation": "Update",
"apiVersion": "v1",
"time": "2020-05-30T04:43:44Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:strimzi.io/delete-claim": {}
},
"f:labels": {
".": {},
"f:app.kubernetes.io/instance": {},
"f:app.kubernetes.io/managed-by": {},
"f:app.kubernetes.io/name": {},
"f:app.kubernetes.io/part-of": {},
"f:strimzi.io/cluster": {},
"f:strimzi.io/kind": {},
"f:strimzi.io/name": {}
},
"f:ownerReferences": {
".": {},
"k:{\"uid\":\"599d91ef-9823-47ba-a67c-bd1c447c6d6f\"}": {
".": {},
"f:apiVersion": {},
"f:blockOwnerDeletion": {},
"f:controller": {},
"f:kind": {},
"f:name": {},
"f:uid": {}
}
}
},
"f:spec": {
"f:accessModes": {},
"f:resources": {
"f:requests": {
".": {},
"f:storage": {}
}
},
"f:volumeMode": {}
},
"f:status": {
"f:phase": {}
}
}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "v1",
"time": "2020-05-30T04:48:45Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:volume.beta.kubernetes.io/storage-provisioner": {}
},
"f:finalizers": {
".": {},
"v:\"kubernetes.io/pvc-protection\"": {}
}
}
}
}
]
},
"spec": {
"accessModes": [
"ReadWriteOnce"
],
"resources": {
"requests": {
"storage": "1Gi"
}
},
"storageClassName": "standard",
"volumeMode": "Filesystem"
},
"status": {
"phase": "Pending"
}
}
]
}
Analysis
It looks like the JSON parser in the storage provisioner doesn't like the escape sequence in the managedFields.f:ownerReferences section.
wcjohnson
👍1
All 8 comments
I did some searching and discovered the following comment on another issue:
https://github.com/kubernetes/minikube/issues/7828#issuecomment-627182963
minikube config set kubernetes-version v1.16.0 as suggested by the commenter did resolve this issue for me. This may therefore be a duplicate, though I am not certain the cause is the same in both cases.
wcjohnson
on 30 May 2020
👍1
Unfortunately I think this is the same as in #7218, the updated storage-provisioner didn't make it in v1.11.0 (which is still running the previous v1.8.1, from two years ago).
afbjorklund
on 30 May 2020
Got the same issue and switching to 1.16.0 seems to work.
hadim
on 9 Jun 2020
👍1
Some recent work has gone into the storage-provisioner, so it's feasible that we may be able to fix this in a near-term release.
tstromberg
on 22 Jul 2020
For those looking for a temporary fix for this and some related issues (particularly the persistent storage being lost due to being in tmpfs), I have forked the newest storage provisioner (which does work with Kubernetes 1.18) and made a small change allowing the hostpath location to be changed.
Source code is at https://github.com/wcjohnson/minikube/tree/storage-provisioner-pvdir-arg
Kubernetes resources for replacing the storage provisioner: https://raw.githubusercontent.com/wcjohnson/minikube/storage-provisioner-pvdir-arg/hack/storage_provisioner_fixes/storage-provisioner.yaml
To deploy:
# STEP 1: get updated storage provisioner yaml
wget https://raw.githubusercontent.com/wcjohnson/minikube/storage-provisioner-pvdir-arg/hack/storage_provisioner_fixes/storage-provisioner.yaml
# STEP 2: start a minikube
minikube config set driver docker
minikube start
# STEP 3: disable bugged provisioner
minikube addons disable storage-provisioner
# STEP 4: create persistent storage path inside the docker container
# (can be skipped if you aren't bitten by the persistent data loss bug)
minikube ssh
sudo mkdir /persistent_volumes
sudo chmod 777 /persistent_volumes
exit
# STEP 5: apply the fixed storage provisioner
# (remove the -pvdir option from the command line if you didn't do step 4)
kubectl apply -f ./storage-provisioner.yaml
For me this fixed all my storage-related issues. Looking forward to when it lands in the mainline.
wcjohnson
on 23 Jul 2020
👍1
@wcjohnson we updated storage provsioner in minikube v1.12.2 do you mind trying with that see if it was fixed?
if not I would be happy to accept a PR that fixes this
medyagh
on 12 Aug 2020
👍1
@medyagh I can confirm the new storage provisioner resolves this problem.
wcjohnson
on 24 Aug 2020
great to hear that ! thank you for confirming
medyagh
on 9 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
javajon
路
3Comments
vainikkaj
路
3Comments
ahmetb
路
3Comments
tstromberg
路
3Comments
cherish3620
路
3Comments