Brigade: Errors in brigade-api pod

[salimd]

Output of brig version:
v1.3.0

Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-26T06:16:15Z", GoVersion:"go1.14", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
bare-metal

I've installed brigade on a newly installed k8s and errors are flooding in brigade-api pods. I'm not sure but it might be related to the managedFields feature introduced in Kubernetes 1.18.

E0424 10:31:59.612146 1 reflector.go:205] github.com/brigadecore/brigade/pkg/storage/kube/apicache/liststore.go:36: Failed to list *v1.Pod: v1.PodList: Items: []v1.Pod: v1.Pod: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 1337 ...:{},"k:{\"... at {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/brigade/pods","resourceVersion":"390513"},"items":[{"metadata":{"name":"brigade-kashti-5d8546994-qc48q","generateName":"brigade-kashti-5d8546994-","namespace":"brigade","selfLink":"/api/v1/namespaces/brigade/pods/brigade-kashti-5d8546994-qc48q","uid":"9de5265f-c69d-4e91-9eb4-d56a1fecf12a","resourceVersion":"23578","creationTimestamp":"2020-04-22T22:00:26Z","labels":{"app":"kashti","pod-template-hash":"5d8546994"},"annotations":{"checksum/config-js":"bc87ca954545f87dec6694db42c28c030b794101a1e6928ed19c0fe084bedf95","checksum/config-nginx":"c7b4a6c805acdb0d9ffcd3740c02f43ee5e528ed7346c06a87ceca40f412d4d6","cni.projectcalico.org/podIP":"10.138.67.1/32","cni.projectcalico.org/podIPs":"10.138.67.1/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"brigade-kashti-5d8546994","uid":"d560ced1-1472-4805-831f-32c1c76b472d","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-04-22T22:00:26Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:checksum/config-js":{},"f:checksum/config-nginx":{}},"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"d560ced1-1472-4805-831f-32c1c76b472d\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"kashti\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":80,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/etc/nginx/conf.d\"}":{".":{},"f:mountPath":{},"f:name":{}},"k:{\"mountPath\":\"/usr/share/nginx/html/assets/js/settings\"}":{".":{},"f:mountPath":{},"f:name":{}}}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"config-js\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{}},"f:name":{}},"k:{\"name\":\"config-nginx\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:name":{}},"f:name":{}}}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2020-04-22T22:00:27Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2020-04-22T22:00:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.138.67.1\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"config-js","configMap":{"name":"brigade-kashti-js","defaultMode":420}},{"name":"config-nginx","configMap":{"name":"brigade-kashti-nginx","defaultMode":420}},{"name":"default-token-74d47","secret":{"secretName":"default-token-74d47","defaultMode":420}}],"containers":[{"name":"kashti","image":"brigadecore/kashti:v0.4.0","ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"config-js","mountPath":"/usr/share/nginx/html/assets/js/settings"},{"name":"config-nginx","mountPath":"/etc/nginx/conf.d"},{"name":"default-token-74d47","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":80,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":80,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"worker01.example.com","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-22T22:00:26Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-22T22:00:36Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-22T22:00:36Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-22T22:00:26Z"}],"hostIP":"10.111.64.23","podIP":"10.138.67.1","podIPs":[{"ip":"10.138.67.1"}],"startTime":"2020-04-22T22:00:26Z","containerStatuses":[{"name":"kashti","state":{"running":{"startedAt":"2020-04-22T22:00:32Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"brigadecore/kashti:v0.4.0","imageID":"docker-pullable://brigadecore/kashti@sha256:752d20123ffab185f19975c080aa9655637800c3fcdcaa947b943c55ccee91b5","containerID":"docker://4e2b711885a191dc3c0bd340791386b9b89f41a861b088ae80a487db575c13ac","started":true}],"qosClass":"BestEffort"}},{"metadata":{"name":"brigade-vacuum-1587718800-r7q75","generateName":"brigade-vacuum-1587718800-","namespace":"brigade","selfLink":"/api/v1/namespaces/brigade/pods/brigade-vacuum-1587718800-r7q75","uid":"1d0b43eb-ace4-42db-8284-2c61badf763d","resourceVersion":"376065","creationTimestamp":"2020-04-24T09:00:00Z","labels":{"app":"brigade-vacuum","controller-uid":"6b35a8c8-5142-49ad-a8a9-1f66b224badd","job-name":"brigade-vacuum-1587718800"},"annotations":{"cni.projectcalico.org/podIP":"10.138.67.33/32","cni.projectcalico.org/podIPs":"10.138.67.33/32"},"ownerReferences":[{"apiVersion":"batch/v1","kind":"Job","name":"brigade-vacuum-1587718800","uid":"6b35a8c8-5142-49ad-a8a9-1f66b224badd","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-04-24T09:00:00Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:controller-uid":{},"f:job-name":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"6b35a8c8-5142-49ad-a8a9-1f66b224badd\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"brigade-vacuum\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BRIGADE_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"VACUUM_AGE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"VACUUM_MAX_BUILDS\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"VACUUM_SKIP_RUNNING_BUILDS\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2020-04-24T09:00:02Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.138.67.32\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2020-04-24T09:00:03Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}}]},"spec":{"volumes":[{"name":"brigade-vacuum-token-bnds8","secret":{"secretName":"brigade-vacuum-token-bnds8","defaultMode":420}}],"containers":[{"name":"brigade-vacuum","image":"brigadecore/brigade-vacuum:v1.2.1","env":[{"name":"VACUUM_AGE","value":"720h"},{"name":"VACUUM_MAX_BUILDS","value":"3"},{"name":"VACUUM_SKIP_RUNNING_BUILDS","value":"false"},{"name":"BRIGADE_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"brigade-vacuum-token-bnds8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"OnFailure","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"brigade-vacuum","serviceAccount":"brigade-vacuum","nodeName":"worker01.example.com","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T09:00:00Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-04-24T09:00:00Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-04-24T09:00:00Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T09:00:00Z"}],"hostIP":"10.111.64.23","podIP":"10.138.67.32","podIPs":[{"ip":"10.138.67.32"}],"startTime":"2020-04-24T09:00:00Z","containerStatuses":[{"name":"brigade-vacuum","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2020-04-24T09:00:01Z","finishedAt":"2020-04-24T09:00:01Z","containerID":"docker://592edb387b4dfde1249615c86d55daba5f0609c5ba38ba7a2a8f85e6e178e228"}},"lastState":{},"ready":false,"restartCount":0,"image":"brigadecore/brigade-vacuum:v1.2.1","imageID":"docker-pullable://brigadecore/brigade-vacuum@sha256:0b9245247f93feb584dd0c63a387f6c9922d97bde51b597fab3b5a00906fc649","containerID":"docker://592edb387b4dfde1249615c86d55daba5f0609c5ba38ba7a2a8f85e6e178e228","started":false}],"qosClass":"BestEffort"}},{"metadata":{"name":"brigade-vacuum-1587715200-r8qjc","generateName":"brigade-vacuum-1587715200-","namespace":"brigade","selfLink":"/api/v1/namespaces/brigade/pods/brigade-vacuum-1587715200-r8qjc","uid":"85d1ed4d-6248-4fea-bc11-9338df532bba","resourceVersion":"366005","creationTimestamp":"2020-04-24T08:00:01Z","labels":{"app":"brigade-vacuum","controller-uid":"92b1962a-2bd0-4c26-b1f3-f81047e1c2d5","job-name":"brigade-vacuum-1587715200"},"annotations":{"cni.projectcalico.org/podIP":"10.135.53.33/32","cni.projectcalico.org/podIPs":"10.135.53.33/32"},"ownerReferences":[{"apiVersion":"batch/v1","kind":"Job","name":"brigade-vacuum-1587715200","uid":"92b1962a-2bd0-4c26-b1f3-f81047e1c2d5","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-04-24T08:00:01Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:controller-uid":{},"f:job-name":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"92b1962a-2bd0-4c26-b1f3-f81047e1c2d5\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"brigade-vacuum\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BRIGADE_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"VACUUM_AGE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"VACUUM_MAX_BUILDS\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"VACUUM_SKIP_RUNNING_BUILDS\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2020-04-24T08:00:03Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.135.53.32\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2020-04-24T08:00:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}}]},"spec":{"volumes":[{"name":"brigade-vacuum-token-bnds8","secret":{"secretName":"brigade-vacuum-token-bnds8","defaultMode":420}}],"containers":[{"name":"brigade-vacuum","image":"brigadecore/brigade-vacuum:v1.2.1","env":[{"name":"VACUUM_AGE","value":"720h"},{"name":"VACUUM_MAX_BUILDS","value":"3"},{"name":"VACUUM_SKIP_RUNNING_BUILDS","value":"false"},{"name":"BRIGADE_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"brigade-vacuum-token-bnds8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"OnFailure","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"brigade-vacuum","serviceAccount":"brigade-vacuum","nodeName":"worker02.example.com","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T08:00:01Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-04-24T08:00:01Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-04-24T08:00:01Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T08:00:01Z"}],"hostIP":"10.111.64.24","podIP":"10.135.53.32","podIPs":[{"ip":"10.135.53.32"}],"startTime":"2020-04-24T08:00:01Z","containerStatuses":[{"name":"brigade-vacuum","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2020-04-24T08:00:02Z","finishedAt":"2020-04-24T08:00:02Z","containerID":"docker://aff8ab258720f1be504ef5e1502dfdfe008131159aebf4d6bd3458605c3da09a"}},"lastState":{},"ready":false,"restartCount":0,"image":"brigadecore/brigade-vacuum:v1.2.1","imageID":"docker-pullable://brigadecore/brigade-vacuum@sha256:0b9245247f93feb584dd0c63a387f6c9922d97bde51b597fab3b5a00906fc649","containerID":"docker://aff8ab258720f1be504ef5e1502dfdfe008131159aebf4d6bd3458605c3da09a","started":false}],"qosClass":"BestEffort"}},{"metadata":{"name":"brigade-api-7b69f7dc9c-9zvj5","generateName":"brigade-api-7b69f7dc9c-","namespace":"brigade","selfLink":"/api/v1/namespaces/brigade/pods/brigade-api-7b69f7dc9c-9zvj5","uid":"e108c47f-6751-4773-9e7d-b5e677875841","resourceVersion":"390440","creationTimestamp":"2020-04-24T10:24:40Z","labels":{"app":"brigade-api","pod-template-hash":"7b69f7dc9c"},"annotations":{"cni.projectcalico.org/podIP":"10.135.53.36/32","cni.projectcalico.org/podIPs":"10.135.53.36/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"brigade-api-7b69f7dc9c","uid":"0f1cf148-f5d4-46d9-9f9d-04157f7fb6ba","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:24:40Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"0f1cf148-f5d4-46d9-9f9d-04157f7fb6ba\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"brigade\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BRIGADE_API_PORT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"BRIGADE_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":7745,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:24:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:24:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.135.53.36\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"brigade-api-token-2zlv6","secret":{"secretName":"brigade-api-token-2zlv6","defaultMode":420}}],"containers":[{"name":"brigade","image":"brigadecore/brigade-api:v1.3.0","ports":[{"containerPort":7745,"protocol":"TCP"}],"env":[{"name":"BRIGADE_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"BRIGADE_API_PORT","value":"7745"}],"resources":{},"volumeMounts":[{"name":"brigade-api-token-2zlv6","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":7745,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":7745,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"brigade-api","serviceAccount":"brigade-api","nodeName":"worker02.example.com","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:40Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:45Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:45Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:40Z"}],"hostIP":"10.111.64.24","podIP":"10.135.53.36","podIPs":[{"ip":"10.135.53.36"}],"startTime":"2020-04-24T10:24:40Z","containerStatuses":[{"name":"brigade","state":{"running":{"startedAt":"2020-04-24T10:24:43Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"brigadecore/brigade-api:v1.3.0","imageID":"docker-pullable://brigadecore/brigade-api@sha256:35dcd3f98e662e0d526d349488bb79b9b888730faccf20ca05c422daac5594a8","containerID":"docker://51228066ef29c590c267d97265d9b411ca87cb04a8c417b7e0d7692fd5a8d1e6","started":true}],"qosClass":"BestEffort"}},{"metadata":{"name":"brigade-vacuum-1587722400-jlzlv","generateName":"brigade-vacuum-1587722400-","namespace":"brigade","selfLink":"/api/v1/namespaces/brigade/pods/brigade-vacuum-1587722400-jlzlv","uid":"5167c552-2ea0-4133-9abf-23e3052fd6b5","resourceVersion":"386245","creationTimestamp":"2020-04-24T10:00:09Z","labels":{"app":"brigade-vacuum","controller-uid":"1e22fdbb-9793-4657-b71d-8c2f7c3b076b","job-name":"brigade-vacuum-1587722400"},"annotations":{"cni.projectcalico.org/podIP":"10.135.53.35/32","cni.projectcalico.org/podIPs":"10.135.53.35/32"},"ownerReferences":[{"apiVersion":"batch/v1","kind":"Job","name":"brigade-vacuum-1587722400","uid":"1e22fdbb-9793-4657-b71d-8c2f7c3b076b","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:00:09Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:controller-uid":{},"f:job-name":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"1e22fdbb-9793-4657-b71d-8c2f7c3b076b\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"brigade-vacuum\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BRIGADE_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"VACUUM_AGE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"VACUUM_MAX_BUILDS\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"VACUUM_SKIP_RUNNING_BUILDS\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:00:11Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.135.53.34\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:00:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}}]},"spec":{"volumes":[{"name":"brigade-vacuum-token-bnds8","secret":{"secretName":"brigade-vacuum-token-bnds8","defaultMode":420}}],"containers":[{"name":"brigade-vacuum","image":"brigadecore/brigade-vacuum:v1.2.1","env":[{"name":"VACUUM_AGE","value":"720h"},{"name":"VACUUM_MAX_BUILDS","value":"3"},{"name":"VACUUM_SKIP_RUNNING_BUILDS","value":"false"},{"name":"BRIGADE_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"brigade-vacuum-token-bnds8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"OnFailure","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"brigade-vacuum","serviceAccount":"brigade-vacuum","nodeName":"worker02.example.com","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Succeeded","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:00:09Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:00:09Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:00:09Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:00:09Z"}],"hostIP":"10.111.64.24","podIP":"10.135.53.34","podIPs":[{"ip":"10.135.53.34"}],"startTime":"2020-04-24T10:00:09Z","containerStatuses":[{"name":"brigade-vacuum","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2020-04-24T10:00:10Z","finishedAt":"2020-04-24T10:00:10Z","containerID":"docker://c1c79503a3990bb16341c372534c99f79bcc87ea9813b0cc1f1010dd4228746a"}},"lastState":{},"ready":false,"restartCount":0,"image":"brigadecore/brigade-vacuum:v1.2.1","imageID":"docker-pullable://brigadecore/brigade-vacuum@sha256:0b9245247f93feb584dd0c63a387f6c9922d97bde51b597fab3b5a00906fc649","containerID":"docker://c1c79503a3990bb16341c372534c99f79bcc87ea9813b0cc1f1010dd4228746a","started":false}],"qosClass":"BestEffort"}},{"metadata":{"name":"brigade-ctrl-7795565757-cwp9b","generateName":"brigade-ctrl-7795565757-","namespace":"brigade","selfLink":"/api/v1/namespaces/brigade/pods/brigade-ctrl-7795565757-cwp9b","uid":"f203e33a-6030-48fb-882f-ce67f7ff3b9f","resourceVersion":"390468","creationTimestamp":"2020-04-24T10:24:40Z","labels":{"app":"brigade-ctrl","pod-template-hash":"7795565757"},"annotations":{"cni.projectcalico.org/podIP":"10.135.53.37/32","cni.projectcalico.org/podIPs":"10.135.53.37/32"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"brigade-ctrl-7795565757","uid":"6600c8c4-18d0-4562-9349-74ef64cb1ba8","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:24:40Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"6600c8c4-18d0-4562-9349-74ef64cb1ba8\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"brigade\"}":{".":{},"f:command":{},"f:env":{".":{},"k:{\"name\":\"BRIGADE_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"BRIGADE_WORKER_IMAGE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"BRIGADE_WORKER_PULL_POLICY\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"BRIGADE_WORKER_SERVICE_ACCOUNT\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:24:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2020-04-24T10:24:47Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.135.53.37\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"brigade-ctrl-token-ssbjt","secret":{"secretName":"brigade-ctrl-token-ssbjt","defaultMode":420}}],"containers":[{"name":"brigade","image":"brigadecore/brigade-controller:v1.3.0","command":["/usr/bin/brigade-controller"],"env":[{"name":"BRIGADE_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"BRIGADE_WORKER_IMAGE","value":"brigadecore/brigade-worker:v1.3.0"},{"name":"BRIGADE_WORKER_PULL_POLICY","value":"IfNotPresent"},{"name":"BRIGADE_WORKER_SERVICE_ACCOUNT","value":"brigade-worker"}],"resources":{},"volumeMounts":[{"name":"brigade-ctrl-token-ssbjt","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"brigade-ctrl","serviceAccount":"brigade-ctrl","nodeName":"worker02.example.com","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:40Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:47Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:47Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-04-24T10:24:40Z"}],"hostIP":"10.111.64.24","podIP":"10.135.53.37","podIPs":[{"ip":"10.135.53.37"}],"startTime":"2020-04-24T10:24:40Z","containerStatuses":[{"name":"brigade","state":{"running":{"startedAt":"2020-04-24T10:24:46Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"brigadecore/brigade-controller:v1.3.0","imageID":"docker-pullable://brigadecore/brigade-controller@sha256:d47cc40adcc6c50447e4f6cd4ff30af6f9082f574f7f6c132bba6f4b02bf633f","containerID":"docker://d109317b9fc0e642fa74085e3e2a6174254e81428a6054243eb17bf6eaf3c8c8","started":true}],"qosClass":"BestEffort"}}]}

[krancour]

@salimd I don't know much about managedFields, but glancing at the docs, it looks like it's possibly intended to help avoid race conditions when multiple collaborators may be applying changes to a single resource?

The brigade API is used exclusively by Kashti and is _read only_ (it will likely have an expanded role in Brigade 2.0, but that's neither here nor there), so I'm not inferring that managedFields is playing any role in this issue.

The errors you see look like they could be a classic case of the service account being used by the API having insufficient permissions. I can infer that the cluster has RBAC enabled. Did you install Brigade with RBAC enabled as well? Did the user account you used to install Brigade have sufficient permissions to delegate to the different service accounts used by Brigade? (I think the install would probably have failed outright if that weren't the case, but I'm asking anyway.)

[salimd]

Hi @krancour,

I don't think RBAC is the issue as I've deployed it with RBAC resources provided with the helm chart. It is also working fine with same manifests on two other clusters. The main difference between them afaik is the Kubernetes version.

I also tried to run brigade API using cluster-admin privileges, the errors were still there.

I appreciate any further advice.

Thanks.

$ kubectl describe rolebinding/brigade-api
Name:         brigade-api
Labels:       app.kubernetes.io/name=brigade
Annotations:  Role:
  Kind:       Role
  Name:       brigade-api
Subjects:
  Kind            Name         Namespace
  ----            ----         ---------
  ServiceAccount  brigade-api

$ kubectl describe Role/brigade-api
Name:         brigade-api
Labels:       app.kubernetes.io/name=brigade
Annotations:  PolicyRule:
  Resources   Non-Resource URLs  Resource Names  Verbs
  ---------   -----------------  --------------  -----
  pods/log    []                 []              [get list watch]
  pods        []                 []              [get list watch]
  secrets     []                 []              [get list watch]

[krancour]

@salimd, @vdice and I have both been able to reproduce this and we discovered an apparent incompatibility between an older version of k8s client-go library and k8s 1.18 clusters. We already had upgraded that library in the master branch. Switching to brig CLI and Docker images built from master resolved the issue for both of us.

So... we've decided to accelerate the Brigade 1.4 release based on these findings.

Thank you for reporting this!

[salimd]

@krancour I've just tried the brigade-api:c1ec6fc image and it just works.

Thank you!