Pipeline: Build docker image with kaniko fails, when serviceaccount configured with imagePullSecrets

[sravankumar777]

Expected Behavior

Kaniko build steps should successful run when Serviceaccount configured with dockerregistry secrets under "imagePullSecrets", by not only checking for secrets under "Secrets".

Actual Behavior

Kaniko build step fails when serviceaccount configured with dockerregistry secrets under "imagePullSecrets".

Steps to Reproduce the Problem

1. Configure serviceaccount with dockerregistry secrets under "imagePullSecrets".

apiVersion: v1
imagePullSecrets:
- name: <name>-docker-config
kind: ServiceAccount
metadata:
  name: <name>-sa
  namespace: <namespace>

1. Run taskrun by using this serviceaccount.
2. Observe the logs for the taskrun which show error for authentication required.

Additional Info

• Kubernetes version:

  Output of kubectl version:

  Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
  

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

$ kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'
v0.11.0

I tried to replice the example scenario with change in serviceaccount having imagePullSecrets only.

This looks like an issue not considering both "secrets" & "imagePullSecrets" while checking for dockerregistry secrets.

[bobcatfish]

Hey @sravankumar777 ! Two things that come to mind:

• The way any particular Task is handling secrets is specific to the Task - are you using the Kaniko Task from the catalog (https://github.com/tektoncd/catalog/tree/v1beta1/kaniko)? Any issues you are encountering might be related to the Task or to Kaniko itself
• my limited understanding of imagepullsecrets is that it exists so you can use it to PULL an image, but not to PUSH an image (and the tekton controller uses it to pull as well https://github.com/tektoncd/pipeline/blob/e2755583d52ae46907790d40ba4886d55611cd23/docs/container-contract.md#entrypoint) so im not certain it would make sense for the kaniko task to use imagePullSecrets

Also we have some docs here about how auth works: https://github.com/tektoncd/pipeline/blob/master/docs/auth.md

[sravankumar777]

Hi Christie,

Thanks for your reply!!!

For 1st case, i have not tried using with kaniko task from catalog. Would be checking about it this week.
For 2nd case, i have provided the permission on the secrets for image push and pull operations.
For more information, i have created a harbor robot account with both image pull & push permissions. It did worked for me.
Same permissions, i tried to access with the tekton-kaniko task.
My suggestion would be to check for docker registry permissions on secrets & imagePullSecrets.

[nikhil-thomas]

@sravankumar777 hi, did you get it working.
If yes, shall we close this issue.
if no, could you share a link to your task definition.

you could also reach us on https://tektoncd.slack.com/

[sravankumar777]

Hi @nikhil-thomas,

We got it working. My suggestion would be to handle from Kaniko image to check both imagePulSecrets & secrets for any service account.
Hope this suggestion would be considered and provide the fix.

Thanks.

[rakhbari]

First let me say, I am using the kaniko Task from the catalog. Well, an exact copy of the kaniko task in the catalog. Long story as to why a copy, but just know the content is the same.

And also that I've already gotten past this issue by adding the Quay secret to the secrets in my serviceaccount. Originally it was only under imagePullSecrets and I was getting a 401 error, but as soon as I added it under secrets, that error went away.

But If you take a close look at the error message I was getting:
[image-build-and-push] error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "hub.tess.io/rakhbari/ngcddemo2:0.0.2": creating push check transport for hub.tess.io failed: GET https://hub.tess.io/v2/auth?scope=repository%3Arakhbari%2Fngcddemo2%3Apush%2Cpull&service=ecr.vip.ebayc3.com: unsupported status code 401
The error msg clearly states that the problem is with checking push permissions, not the actual push itself. It's trying to do a GET /v2/auth call to check for repo permissions and I have a hunch the serviceaccount needs the Quay secret under secrets for that call to succeed.

In any case, as far as I'm concerned the issue is resolved. As long as this gets clearly called out in the docs, I think we can resolve/close this issue.

[nikhil-thomas]

@sravankumar777
im glad that you got it resolved.

The kaniko image doen't know about the secrets we attach to the service account. As our creds-init logic in Tekton reads the secrets specfied in the service account and sets up the creds in the Task pods.

So if I understand correctly, your suggestion is that the creds-init should check the pull-secrets as well.

However, at present as @rakhbari said we could specify it as a secret. Will that good enough for now. Or is the support for a separate imagePullSecrets a critical requirement.

(On a side note, you can add the dockerconfig on your Task as a volume mount.)

@rakhbari Thanks :+1:

cc @sbwsg

[sbwsg]

Reading the reference for imagePullSecrets in podspec it seems pretty clear, at least to me, that the field is only intended to support the container runtime fetch images for the pod's containers:

ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use.

There isn't any mention of supporting an in-container credential usecase. I'd definitely prefer not to go out of our way to augment or modify the behaviour of a field that already has a well-established meaning in k8s.

I'm going to close this for now since I think using secrets is perfectly acceptable. Unless we can establish a strong reason with no good alternatives to add this additional behaviour for imagePullSecrets (and get that design broadly accepted by API WG) I don't think we should pursue this.