Jx: GC cronjob actions failing?

[tdcox]

Using:

NAME VERSION
jx 1.2.133
jenkins x platform 0.0.1175
kubernetes cluster v1.9.7-gke.1
kubectl v1.9.6
helm client v2.9.1+g20adb27
helm server v2.9.1+g20adb27
git git version 2.15.1 (Apple Git-101)

I am seeing lots of failed pods on our cluster with names starting with jenkins-x-gc-activities and jenkins-x-gc-previews. Looks like there is an issue with permissions as the logs are reporting:

error: pipelineactivities.jenkins.io is forbidden: User "system:serviceaccount:jx:default" cannot list pipelineactivities.jenkins.io in the namespace "jx": Unknown user "system:serviceaccount:jx:default"

[rawlingsj]

yeah I'll be working on this today. not too sure what's up with it yet but hopefully it wont take much

[almerico]

Are any workaround?
Have something similar on 1.3.20.
kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-5d88987b8d-965hr 1/1 Running 0 1h
jenkins-x-chartmuseum-77dcdf7f87-2q2tr 1/1 Running 0 1h
jenkins-x-docker-registry-76dd6bddd4-6qxtr 1/1 Running 0 1h
jenkins-x-gc-activities-1529658000-4wtnf 0/1 Error 0 16m
jenkins-x-gc-activities-1529658000-6rpgd 0/1 Error 0 15m
jenkins-x-gc-activities-1529658000-97cq8 0/1 Error 0 15m
jenkins-x-gc-activities-1529658000-cfwbj 0/1 Error 0 14m
jenkins-x-gc-activities-1529658000-csbrx 0/1 Error 0 15m
jenkins-x-gc-activities-1529658000-jdltm 0/1 Error 0 13m
jenkins-x-gc-activities-1529658000-pdqt8 0/1 Error 0 15m
jenkins-x-gc-previews-1529658000-4tgj4 0/1 Error 0 15m
jenkins-x-gc-previews-1529658000-blz52 0/1 Error 0 14m
jenkins-x-gc-previews-1529658000-bmp5k 0/1 Error 0 15m
jenkins-x-gc-previews-1529658000-bqm57 0/1 Error 0 15m
jenkins-x-gc-previews-1529658000-jszzj 0/1 Error 0 16m
jenkins-x-gc-previews-1529658000-nw4qb 0/1 Error 0 13m
jenkins-x-gc-previews-1529658000-wmpr6 0/1 Error 0 15m
jenkins-x-heapster-57b558bf9b-vxxjz 2/2 Running 0 1h
jenkins-x-mongodb-5fb7868bd-9sc45 1/1 Running 0 1h
jenkins-x-monocular-api-584fcdb959-jvh2w 1/1 Running 1 1h
jenkins-x-monocular-prerender-74f49ffdc7-x6nlw 1/1 Running 0 1h
jenkins-x-monocular-ui-74d48c8cdb-hmw7m 1/1 Running 0 1h
jenkins-x-nexus-786fcdbd98-bl2d2 1/1 Running 0 1h
pipelinecontroller-5c5f5df7ff-vbq5p 1/1 Running 0 1h
skaffold-6c9595bc8c-vhxch 1/1 Running 0 13m

kubectl logs jenkins-x-gc-activities-1529658000-4wtnf
error: pipelineactivities.jenkins.io is forbidden: User "system:serviceaccount:jx:default" cannot list pipelineactivities.jenkins.io in the namespace "jx": Unknown user "system:serviceaccount:jx:default"
$ jx version
NAME VERSION
jx 1.3.20
jenkins x platform 0.0.1436
kubernetes cluster v1.8.10-gke.0
kubectl v1.10.3
helm client v2.9.1+g20adb27
helm server v2.9.1+g20adb27
git git version 2.15.2 (Apple Git-101.1)

[rawlingsj]

Sorry - I've been juggling a couple of things at the moment. Will try and sort this today. It doesn't cause any harm these failed pods but the gc job doesn't need fixing up.

[rawlingsj]

for now you can delete the cronjobs so you don't get the pods scheduled

kubectl delete cronjob jenkins-x-gc-activities jenkins-x-gc-previews

[liuxiaopiao]

Met the same issue and will watch it.

[almerico]

After upgrade skaffold stops working

o.s.cloud.kubernetes.StandardPodUtils : Failed to get pod with name:[skaffold-6cc58b4c5b-mwwp7]. You should look into this if things aren't working as you expect. Are you missing serviceaccount permissions?

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/jx/pods/skaffold-6cc58b4c5b-mwwp7. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "skaffold-6cc58b4c5b-mwwp7" is forbidden: User "system:serviceaccount:jx:default" cannot get pods in the namespace "jx": Unknown user "system:serviceaccount:jx:default".

$ kubectl get pod,pvc -n jx
NAME READY STATUS RESTARTS AGE
pod/jenkins-7d584ff648-fqrt7 1/1 Running 0 4d
pod/jenkins-x-chartmuseum-77dcdf7f87-snp6k 1/1 Running 0 4d
pod/jenkins-x-docker-registry-76dd6bddd4-pt9hq 1/1 Running 0 4d
pod/jenkins-x-gc-activities-1530113400-5klxg 0/1 Error 0 35m
pod/jenkins-x-gc-activities-1530113400-7t58w 0/1 Error 0 37m
pod/jenkins-x-gc-activities-1530113400-9kth6 0/1 Error 0 37m
pod/jenkins-x-gc-activities-1530113400-fvlvs 0/1 Error 0 37m
pod/jenkins-x-gc-activities-1530113400-rvbsp 0/1 Error 0 34m
pod/jenkins-x-gc-activities-1530113400-tg9gc 0/1 Error 0 36m
pod/jenkins-x-gc-activities-1530113400-w99hz 0/1 Error 0 36m
pod/jenkins-x-gc-previews-1530111600-4q4f2 0/1 Error 0 1h
pod/jenkins-x-gc-previews-1530111600-6j524 0/1 Error 0 1h
pod/jenkins-x-gc-previews-1530111600-bmn7w 0/1 Error 0 1h
pod/jenkins-x-gc-previews-1530111600-f86fw 0/1 Error 0 1h
pod/jenkins-x-gc-previews-1530111600-fss78 0/1 Error 0 1h
pod/jenkins-x-gc-previews-1530111600-mwm2s 0/1 Error 0 1h
pod/jenkins-x-gc-previews-1530111600-qsjkx 0/1 Error 0 1h
pod/jenkins-x-heapster-57b558bf9b-pmzcn 2/2 Running 0 4d
pod/jenkins-x-mongodb-5fb7868bd-ltzmh 1/1 Running 0 4d
pod/jenkins-x-monocular-api-d7d6cc9b7-rbmlq 1/1 Running 2 4d
pod/jenkins-x-monocular-prerender-74f49ffdc7-79zx8 1/1 Running 0 4d
pod/jenkins-x-monocular-ui-589c5c757c-tn29f 1/1 Running 0 4d
pod/jenkins-x-nexus-786fcdbd98-r7wgm 1/1 Running 0 4d
pod/maven-d9jr3 2/2 Running 0 22m
pod/pipelinecontroller-5c5f5df7ff-5cpcm 1/1 Running 0 4d
pod/skaffold-6cc58b4c5b-mwwp7 0/1 CrashLoopBackOff 8 18m

jx 1.3.36

[adomenech73]

Same for:

$ jx version
NAME               VERSION
jx                 1.3.42
jenkins x platform 0.0.1607
kubernetes cluster v1.10.4-gke.2
kubectl            v1.11.0
helm client        v2.9.1+g20adb27
helm server        v2.9.1+g20adb27
git                git version 2.17.1

~Just an update: In my case I got this problem when using Linux to create jx environment but not with MacOS~
It hapened after a while

[livelyRyan]

Met the same issue and will watch it.

[rawlingsj]

This was fixed by https://github.com/jenkins-x/jx/pull/1190

[patrickleet]

still seeing this in EKS

[rawlingsj]

Can you add the logs from one of the failed pods please?

[mroverton1]

Seems to be timing. When I install on heavily loaded system I get these.
While installing on Docker for macOS.

[patrickleet]

➜  ~ kubectl logs -n jx jenkins-x-gc-activities-1533081600-zfbdj
error: 500 Server Error

[patrickleet]

@rawlingsj ^

[iamonkara]

pretty much the same issue

~|⇒ k get pods --all-namespaces -o wide | ag activ
jx            jenkins-x-gc-activities-1536613200-222zs                0/1       Error       0          1h        100.96.3.32     ip-172-20-32-137.ec2.internal
jx            jenkins-x-gc-activities-1536613200-2248l                0/1       Error       0          49m       100.96.2.150    ip-172-20-113-40.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22642                0/1       Error       0          1h        100.96.3.241    ip-172-20-32-137.ec2.internal
jx            jenkins-x-gc-activities-1536613200-226pt                0/1       Error       0          4h        100.96.2.179    ip-172-20-113-40.ec2.internal
jx            jenkins-x-gc-activities-1536613200-229k4                0/1       Error       0          3h        100.96.3.50     ip-172-20-32-137.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22b5h                0/1       Error       0          4h        100.96.3.148    ip-172-20-32-137.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22gft                0/1       Error       0          3h        100.96.2.154    ip-172-20-113-40.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22h7q                0/1       Error       0          3h        100.96.2.167    ip-172-20-113-40.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22kjh                0/1       Error       0          35m       100.96.2.13     ip-172-20-113-40.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22p4n                0/1       Error       0          5h        100.96.2.203    ip-172-20-113-40.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22rbx                0/1       Error       0          3h        100.96.3.203    ip-172-20-32-137.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22rlf                0/1       Error       0          2h        100.96.2.123    ip-172-20-113-40.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22rmm                0/1       Error       0          2h        100.96.3.6      ip-172-20-32-137.ec2.internal
jx            jenkins-x-gc-activities-1536613200-22sng                0/1       Error       0          5h        100.96.3.181    ip-172-20-32-137.ec2.internal
.
.
.

I ended up deleting the cronjobs

kubectl delete cronjob jenkins-x-gc-activities jenkins-x-gc-previews

as suggested above, not sure if that's the right approach

[polothy]

Running into the same issue, eventually Jenkins stops responding to requests (EG: jx console page fails to load). Getting this for logs on the gc-activities pods:

❯ kubectl logs --tail=100 jenkins-x-gc-activities-1537207200-zwb8m -n jxmt
error: deployments.apps "prow-build" is forbidden: User "system:serviceaccount:jxmt:jenkins-x-gc-activities" cannot get deployments.apps in the namespace "jxmt"

When I installed JX, I did not install with the prow option. The install was basically jx create cluster eks and hooking it up to Bitbucket Cloud. I built the cluster this morning and Jenkins is unresponsive. Here is JX version:

❯ jx version
NAME               VERSION
jx                 1.3.275
jenkins x platform 0.0.2447
kubernetes cluster v1.10.3
kubectl            v1.11.3
helm client        v2.10.0+g9ad53aa
helm server        v2.10.0+g9ad53aa
git                git version 2.19.0

[polothy]

Created https://github.com/jenkins-x/jx/issues/1705 since this issue is closed.