Test-infra: prevent tests from running kubectl against the build/test cluster
most tests should run against an external cluster, not the cluster in which the prowjob's pod is created.
we can decorate pods with AutomountServiceAccountToken: false, but we may want to allow some kind of opt-out.
We could just paste this onto every job, but that doesn't sound super great, and probably most prow users will want this behavior by default. This will help catch misconfigured tests that were supposed to run against some other cluster but instead wound up talking to prow.
BenTheElder
All 23 comments
/cc @cjwagner @kargakis @stevekuznetsov @sebastienvas
This came up because I noticed some strange objects deployed in our build cluster, I suspect some tests accidentally deployed helm things to the build cluster instead of an e2e cluster.
BenTheElder
on 27 Apr 2018
/area prow
/kind feature
BenTheElder
on 27 Apr 2018
@cjwagner pointed out that we can configure the service account instead, I think this might be worth having in the pod decorator though since probably most prow users should be opting in their pods to having access to the test-cluster?
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
WDYT?
BenTheElder
on 27 Apr 2018
We can configure it in both the service account and the pod. The pod's configuration takes precedence.
Are we using the default service account to access boskos projects? If so we'll need to change that. We want to limit access to the build cluster without limiting access to boskos test projects.
cjwagner
on 27 Apr 2018
Boskos projects use entirely different credentials. We should probably at
least add a note to the prow guide(s) covering this as well once we fix
things on our end.
On Fri, Apr 27, 2018, 2:29 PM Cole Wagner notifications@github.com wrote:
We can configure it in both the service account and the pod. The pod's
configuration takes precedence.Are we using the default service account to access boskos projects? If so
we'll need to change that. We want to limit access to the build cluster
without limiting access to boskos test projects.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/test-infra/issues/7901#issuecomment-385099245,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA4Bq-WnMf4byGnxoL8QzI_V7eOo6wofks5ts43GgaJpZM4TrBt0
.
BenTheElder
on 27 Apr 2018
I believe RBAC is also an option here.
cjwagner
on 28 Apr 2018
I believe RBAC is also an option here.
not without recreating the cluster though, but probably good enough for everyone else? having any sort of credential can provide suprising behavior though.
We've also previously seen that the kubectl context is in default namespace test-pods when running in a prow pod which can cause problems even when using a boskos cluster (objects that don't specify namespace are implicitly created here instead of default, but the namespace won't exist in the other cluster).
BenTheElder
on 28 Apr 2018
rbac is your friend
kargakis
on 28 Apr 2018
RBAC seems like the appropriate solution here. Not sure I want to not mount a service account into test pods from plank as that may limit how tests can works -- for instance we are working on tests right now that we _do_ want to touch the build cluster, and it would be great to use in-cluster config for those.
stevekuznetsov
on 28 Apr 2018
Even with RBAC this will not fix the bizzare kubectl context. Tests pods
really don't need to automount the cluster creds.
On Sat, Apr 28, 2018, 2:45 PM Steve Kuznetsov notifications@github.com
wrote:
RBAC seems like the appropriate solution here. Not sure I want to not
mount a service account into test pods from plank as that may limit how
tests can works -- for instance we are working on tests right now that we
do want to touch the build cluster, and it would be great to use
in-cluster config for those.—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/test-infra/issues/7901#issuecomment-385207608,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA4Bq8Rm14xPUcAgie1Z6bcDnq7QFVpsks5ttOMNgaJpZM4TrBt0
.
BenTheElder
on 30 Apr 2018
for instance we are working on tests right now that we do want to touch the build cluster, and it would be great to use in-cluster config for those.
Override it for just those tests? (to enable it)
For most tests it's suprising to have the default namespace be test-pods and in some situations it seems they're creating and never cleaning up resources in the test cluster which is a problem.
We could default decorate pods to not automount credentials but not decorate if they explicitly specify credential mounting.
BenTheElder
on 30 Apr 2018
You can delete the default service account for the test-pod NS and create
one that does not cred Access. I think we had the same issue and that
resolved it.
On Mon, Apr 30, 2018, 6:47 PM Benjamin Elder notifications@github.com
wrote:
for instance we are working on tests right now that we do want to touch
the build cluster, and it would be great to use in-cluster config for those.Override it for just those tests? (to enable it)
For most tests it's suprising to have the default namespace be test-pods
and in some situations it seems they're creating and never cleaning up
resources in the test cluster which is a problem.We could default decorate pods to not automount credentials but not
decorate if they explicitly specify credential mounting.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/test-infra/issues/7901#issuecomment-385457728,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQ80Z4IKW1qGX1laRoEfIRMiFNu9KWCmks5tt0AhgaJpZM4TrBt0
.
sebastienvas
on 1 May 2018
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 30 Jul 2018
/remove-lifecycle stale
@cjwagner @krzyzacy we still either need to reconfigure our build cluster(s) to fix this, or enhance prow to decorate AutomountServiceAccountToken: false ... we could maybe just default this field if it is unset in the prowjob spec so that only jobs that explicitly need it have it, and they clearly opt in.
If we go the modify-cluster-config route instead, we should probably document this (in the getting started guide ?)
BenTheElder
on 30 Jul 2018
Alternatively we can go the really fun route and figure out how to transplant boskos to a new cluster and finally get RBAC, and just make sure the guide covers this :D
That actually seems like the correct route going forward, but we'll need a good solution for migrating boskos in particular @krzyzacy
Perhaps we can have a brief planned outage at some point to migrate the build cluster + boskos?
BenTheElder
on 30 Jul 2018
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
fejta-bot
on 28 Oct 2018
/remove-lifecycle stale
BenTheElder
on 30 Oct 2018
Auto deploy prow just hit this. cc @fejta
BenTheElder
on 9 Nov 2018
what do you mean by prow is super unhappy?
fejta
on 9 Nov 2018
senlu@senlu:~/work/src/k8s.io/test-infra/prow$ kubectl get po
NAME READY STATUS RESTARTS AGE
branchprotector-1541721240-bsrmt 1/1 Running 0 47m
deck-7d7d5c5b4-kmm8x 1/1 Running 0 22h
deck-7d7d5c5b4-mztk7 1/1 Running 0 22h
deck-7d7d5c5b4-n7ttc 1/1 Running 0 22h
ghproxy-6d94f789c8-z6k2b 1/1 Running 0 45d
hook-545fb946f7-25h5c 1/1 Running 0 22h
hook-545fb946f7-2c759 1/1 Running 0 22h
hook-545fb946f7-5xgt6 1/1 Running 0 22h
hook-545fb946f7-bsvs9 1/1 Running 0 22h
horologium-7865768dd6-rg264 1/1 Running 0 22h
needs-rebase-75cb567498-b8z22 1/1 Running 0 22h
plank-678f777545-wmf4t 1/1 Running 0 22h
pushgateway-3554809893-g4g5g 1/1 Running 0 7d
pushgateway-proxy-4026406893-wzcxs 1/1 Running 1 7d
sinker-d58cfb9ff-mjgzt 1/1 Running 0 22h
tide-db9b4d646-89fz8 1/1 Running 0 22h
tot-f5c494f6c-lzjhq 1/1 Running 0 22h
senlu@senlu:~/work/src/k8s.io/test-infra/prow$ kubectl get po -n=test-pods
NAME READY STATUS RESTARTS AGE
cm-acme-http-solver-swpjj 1/1 Running 0 33m
cm-acme-http-solver-x75h4 1/1 Running 0 33m
deck-58ffbff564-bwmld 0/1 ContainerCreating 0 33m
deck-58ffbff564-rt26w 0/1 ContainerCreating 0 33m
deck-58ffbff564-v9bd8 0/1 ContainerCreating 0 33m
ghproxy-5fdb9b4d75-hdjc2 1/1 Running 0 33m
hook-8457db5df6-4989h 0/1 ContainerCreating 0 33m
hook-8457db5df6-4lsfw 0/1 ContainerCreating 0 33m
hook-8457db5df6-jg89g 0/1 ContainerCreating 0 33m
hook-8457db5df6-wc5dz 0/1 ContainerCreating 0 33m
horologium-7b666c6cfb-2t7vv 0/1 ContainerCreating 0 33m
needs-rebase-7c5f8d845d-zqd4c 0/1 ContainerCreating 0 33m
plank-564677db95-4hf57 0/1 ContainerCreating 0 33m
pushgateway-7569b4584b-fmjxf 1/1 Running 0 33m
pushgateway-proxy-75c9684bcd-klr7h 1/1 Running 0 33m
sinker-f8774f79f-nclms 0/1 ContainerCreating 0 33m
tide-68947fbdbb-v56cq 0/1 ContainerCreating 0 33m
tot-7cfd685f77-pd85x 1/1 Running 0 33m
/this-is-fine
krzyzacy
on 9 Nov 2018
@krzyzacy: 
In response to this:
senlu@senlu:~/work/src/k8s.io/test-infra/prow$ kubectl get po NAME READY STATUS RESTARTS AGE branchprotector-1541721240-bsrmt 1/1 Running 0 47m deck-7d7d5c5b4-kmm8x 1/1 Running 0 22h deck-7d7d5c5b4-mztk7 1/1 Running 0 22h deck-7d7d5c5b4-n7ttc 1/1 Running 0 22h ghproxy-6d94f789c8-z6k2b 1/1 Running 0 45d hook-545fb946f7-25h5c 1/1 Running 0 22h hook-545fb946f7-2c759 1/1 Running 0 22h hook-545fb946f7-5xgt6 1/1 Running 0 22h hook-545fb946f7-bsvs9 1/1 Running 0 22h horologium-7865768dd6-rg264 1/1 Running 0 22h needs-rebase-75cb567498-b8z22 1/1 Running 0 22h plank-678f777545-wmf4t 1/1 Running 0 22h pushgateway-3554809893-g4g5g 1/1 Running 0 7d pushgateway-proxy-4026406893-wzcxs 1/1 Running 1 7d sinker-d58cfb9ff-mjgzt 1/1 Running 0 22h tide-db9b4d646-89fz8 1/1 Running 0 22h tot-f5c494f6c-lzjhq 1/1 Running 0 22h senlu@senlu:~/work/src/k8s.io/test-infra/prow$ kubectl get po -n=test-pods NAME READY STATUS RESTARTS AGE cm-acme-http-solver-swpjj 1/1 Running 0 33m cm-acme-http-solver-x75h4 1/1 Running 0 33m deck-58ffbff564-bwmld 0/1 ContainerCreating 0 33m deck-58ffbff564-rt26w 0/1 ContainerCreating 0 33m deck-58ffbff564-v9bd8 0/1 ContainerCreating 0 33m ghproxy-5fdb9b4d75-hdjc2 1/1 Running 0 33m hook-8457db5df6-4989h 0/1 ContainerCreating 0 33m hook-8457db5df6-4lsfw 0/1 ContainerCreating 0 33m hook-8457db5df6-jg89g 0/1 ContainerCreating 0 33m hook-8457db5df6-wc5dz 0/1 ContainerCreating 0 33m horologium-7b666c6cfb-2t7vv 0/1 ContainerCreating 0 33m needs-rebase-7c5f8d845d-zqd4c 0/1 ContainerCreating 0 33m plank-564677db95-4hf57 0/1 ContainerCreating 0 33m pushgateway-7569b4584b-fmjxf 1/1 Running 0 33m pushgateway-proxy-75c9684bcd-klr7h 1/1 Running 0 33m sinker-f8774f79f-nclms 0/1 ContainerCreating 0 33m tide-68947fbdbb-v56cq 0/1 ContainerCreating 0 33m tot-7cfd685f77-pd85x 1/1 Running 0 33m/this-is-fine
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
k8s-ci-robot
on 9 Nov 2018
heh, that's not good
fejta
on 9 Nov 2018
I think the issue here is that it runs in test-pods namespace and doesn't specify where it wants to schedule jobs
fejta
on 9 Nov 2018
Related issues
BenTheElder
·
4Comments
MrHohn
·
4Comments
spiffxp
·
3Comments
BenTheElder
·
4Comments
stevekuznetsov
·
4Comments