Pipeline: Cannot clone private Git repository as an input PipelineResource in tekton task

[g000444555]

Expected Behavior

Clone the private Git repository using the supplied secret.

Actual Behavior

Fails with a misleading error.

Steps to Reproduce the Problem

As documented in https://github.com/tektoncd/pipeline/blob/master/docs/resources.md I have configured a private GitHub repository as a PipelineResource for a task, and have created the relevant secret as well:

---
apiVersion: v1
kind: Secret
metadata:
  name: github-secrets
type: Opaque
data:
  token: github_personal_access_token_secret # in base64 encoded form
---
apiVersion: tekton.dev/v1alpha1
kind: PipelineResource
metadata:
  name: my-repo-git
spec:
  type: git
  params:
    - name: revision
      value: master
    - name: url
      value: https://github.my-company.com/my-team/my-repo.git
  secrets:
    - fieldName: authToken
      secretName: github-secrets
      secretKey: token

Now when I am using the above PipelineResource as an input to a task:

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: my-task
spec:
  inputs:
    resources:
      - name: my-repo-git
        type: git
  steps:
    - name: print-info
      image: image-registry.openshift-image-registry.svc:5000/default/my-task-runner-image:latest
      imagePullPolicy: Always
      command: ["/bin/sh"]
      args:
        - "-c"
        - >
          echo "List the contents of /workspace, expecting to find the Git repository in there" &&
          ls -R /workspace

I get the following error:

[test : git-source-my-repo-git-qhsvq] {"level":"warn","ts":1580370880.0396245,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
[test : git-source-my-repo-git-qhsvq] {"level":"error","ts":1580370880.653762,"logger":"fallback-logger","caller":"git/git.go:40","msg":"Error running git [fetch --depth=1 --recurse-submodules=yes origin master]: exit status 128\nfatal: could not read Username for 'https://github.my-company.com': No such device or address\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:40\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:91\nmain.main\n\t/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:39\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}
[test : git-source-my-repo-git-qhsvq] {"level":"error","ts":1580370881.232096,"logger":"fallback-logger","caller":"git/git.go:40","msg":"Error running git [pull --recurse-submodules=yes origin]: exit status 1\nfatal: could not read Username for 'https://github.my-company.com': No such device or address\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:40\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:94\nmain.main\n\t/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:39\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}
[test : git-source-my-repo-git-qhsvq] {"level":"warn","ts":1580370881.2322857,"logger":"fallback-logger","caller":"git/git.go:95","msg":"Failed to pull origin : exit status 1"}
[test : git-source-my-repo-git-qhsvq] {"level":"error","ts":1580370881.2347631,"logger":"fallback-logger","caller":"git/git.go:40","msg":"Error running git [checkout master]: exit status 1\nerror: pathspec 'master' did not match any file(s) known to git.\n","stacktrace":"github.com/tektoncd/pipeline/pkg/git.run\n\t/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:40\ngithub.com/tektoncd/pipeline/pkg/git.Fetch\n\t/go/src/github.com/tektoncd/pipeline/pkg/git/git.go:97\nmain.main\n\t/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:39\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}
[test : git-source-my-repo-git-qhsvq] {"level":"fatal","ts":1580370881.2348266,"logger":"fallback-logger","caller":"git-init/main.go:40","msg":"Error fetching git repository: exit status 1","stacktrace":"main.main\n\t/go/src/github.com/tektoncd/pipeline/cmd/git-init/main.go:40\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}

The error could not read Username for 'https://github.my-company.com': No such device or address indicates that it looks for a username. The sourceSecret in build configs has different format:

apiVersion: v1
kind: Secret
metadata:
  name: github-secrets
type: kubernetes.io/basic-auth
data:
  username: my-github-username
  password: github_personal_access_token_secret # in base64 encoded form

However, even when trying the above secret type I still get the same error.

How do I configure the secret for a private repository?

Additional Info

Related (though different) issues:

1. https://github.com/tektoncd/pipeline/issues/1603
2. https://github.com/tektoncd/pipeline/issues/1375

[vdemeester]

Hi @g000444555, thanks for the issue.

The way to use secret in tekton is a bit different than usual, see auth.md.

• Your secret need to be annotated with tekton.dev/git-0: https://github.my-company.com
• You need to add that secret to a serviceaccount
• You need to run you Task or Pipeline with that serviceAccount (using serviceAccountName)

/kind question

[g000444555]

That's a very good answer. Thank you very much.

[zertan]

Hi, can I use any other git repo provider (such as a local one) with the tekton pipeline git resource?

[sbwsg]

@zertan Yes it should work with any git repo. But you might find the git-clone task from the catalog more useful: https://github.com/tektoncd/catalog/tree/master/git

The git pipelineresource can be difficult to inspect/debug when there are problems fetching from a repo.

[desourav]

It doesn't work despite creating a secret with the recommended annotation and data fields

apiVersion: v1
kind: Secret
metadata:
  name: ssh-key
  annotations:
    tekton.dev/git-0: github.company.com # Described below
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: <base64 encoded>
  # This is non-standard, but its use is encouraged to make this more secure.
  known_hosts: <base64 encoded>

"msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin master]: exit status 128\nfatal: could not read Username for 'https://github.company.com': No such device or address\n"

[yudatou123]

@zertan Yes it should work with any git repo. But you might find the git-clone task from the catalog more useful: https://github.com/tektoncd/catalog/tree/master/git

The git pipelineresource can be difficult to inspect/debug when there are problems fetching from a repo.

now have some plans to optimize this problem ？

[CoderPoet]

It doesn't work despite creating a secret with the recommended annotation and data fields

apiVersion: v1
kind: Secret
metadata:
  name: ssh-key
  annotations:
    tekton.dev/git-0: github.company.com # Described below
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: <base64 encoded>
  # This is non-standard, but its use is encouraged to make this more secure.
  known_hosts: <base64 encoded>

"msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin master]: exit status 128\nfatal: could not read Username for 'https://github.company.com': No such device or address\n"

My performance is the same as yours, is there any solution for this issue

The same configuration, tekton pipeline 0.12 no problem, 0.13, 0.14 version will have problems

[sbwsg]

Please provide a copy of the Secret YAML, ServiceAccount YAML, and Task YAML that you're using. Remove any sensitive data in those YAMLs.

"msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin master]: exit status 128\nfatal: could not read Username for 'https://github.company.com': No such device or address\n"

This error ^ implies that Tekton is not considering your secret as a valid SSH secret. Unfortunately the error message isn't enough to know why. It's also interesting that you mention this problem only started with 0.13. It could be a bug or it could be that some of our validation or processing of secrets changed. I don't remember that being the case but it would be great to see the YAMLs involved to try and work it out.

[sbwsg]

One other thing that would be really useful to help debug is the TaskRun YAML after the TaskRun has ended. You can get this with kubectl get -o yaml taskrun <taskrun-name>.

[Shahard2]

any solution to this one?

[sbwsg]

Will need YAMLs described in my previous comments in order to help debug. Remove any sensitive values before posting them.

[Shahard2]

@sbwsg can i put the service account inside 'Task' or 'PipelineResource' or 'Pipeline'? Because I don't have a 'TaskRun' for that use-case. and I want to use ssh key from a secret

[sbwsg]

It's not possible to include a service account in a Task, PipelineResource or Pipeline. What's your specific use case? It might help if you could offer a bit more info on what you're trying to do.

[Shahard2]

Wrote you in slack (k8s community)

[hrvolapeter]

@sbwsg @CoderPoet Does github.com that is used with ssh-auth qualify for "name" https://github? It sound like it could be caused by wrong protocol.
I would expect that with http protocol it looks only for basic-auth key. If that's the case how can you dynamically rewrite protocol?

"msg":"Error running git [fetch --recurse-submodules=yes --depth=1 origin master]: exit status 128\nfatal: could not read Username for 'https://github.company.com': No such device or address\n"

Edit: Wrong protocol was indeed the problem at least in my case. Error with the https protocol and ssh-auth suggest wrong protocol this can be fixed in the github case by using value: $(body.repository.ssh_url) instead of value:$(body.repository.html_url)