Cloud-builders: How do I access a private Github repository within a cloud builder?

[avinassh]

I have two private repositories on Github, A and B. Both are Golang projects. Project A depends on repository B. I have linked both repositories to Google Source Code and consented with permissions.

Now in Container builder, when I try to build for project A, it is not able to access repository B. In the build logs, I see:

Step #1: [WARN]  Unable to checkout github.com/avi/api

Step #1: [ERROR] Update failed for github.com/avi/api: Unable to get repository

The above happens when I try to run glide install:

steps:
- name: 'gcr.io/cloud-builders/glide'
  args: ['install', '.']

Later I thought may I could clone the repo first and make glide to use local repo, so I tried:

steps:
- name: 'gcr.io/cloud-builders/git'
  args: ['clone', '[email protected]:avinassh/api.git']

But it failed saying:

Step #0: Already have image (with digest): gcr.io/cloud-builders/git
Starting Step #0
Step #0: Cloning into 'ssh_clone'...
Step #0: Host key verification failed.
Step #0: fatal: Could not read from remote repository.
Step #0: 
Step #0: Please make sure you have the correct access rights
Step #0: and the repository exists.
Finished Step #0

and when I tried HTTPS instead of SSH, I got following error:

Step #0: Cloning into 'api'...
Step #0: fatal: could not read Username for 'https://github.com': No such device or address

[Philmod]

Hey,

We don’t have an integrated solution as of today, but we have seen one solution coming from users. The trick is to use a Deploy or User Key that you could store in GCS for example, and pull it as a previous build step.

[DocBradfordSoftware]

Is there an example out there of configuring .netrc, or any other way to configure the build environment to hold the git authorization for a service account, so that You wouldn't have to include a secret in the package.json or other package scheme (go)?

[bendory]

We don't have a canonical example at this time. We have seen users do this by storing the secret in GCS and then pulling it into their build using the gsutil build step.

[DocBradfordSoftware]

@bendory, thanks. That works fine if you are scripting bash commands, but it won't work, if you want to use a package manager to pull from git private repos without putting the private token in your package.json file. Since there doesn't seem to be a way to get a username/password for a service account, I think I will just create a fake user email, that I can authenticate at github or google source, then include that in a .netrc that I include in GCS.

[bendory]

This newly-published doc on Using Encrypted Files in your build should be helpful.

[purohit]

@bendory thanks for the tutorial.

Any idea how to actually get the git cloud builder steps to remember the settings from previous steps? I do a git config --global credentials 'store --file [unencrypted credentials file]', but in the next git clone steps, it still tries to use gcloud.sh. I can confirm that's being used by echoing. Otherwise, I'm not sure how git private repos are being used. Here's what I have so far:

steps:
# Unencrypt the git credentials
- name: 'gcr.io/cloud-builders/gcloud'
  args:
  - kms
  - decrypt
  - --ciphertext-file=git-credentials.enc
  - --plaintext-file=git-credentials
  - --location=global
  - --keyring=github-read
  - --key=github-read-key
- name: 'gcr.io/cloud-builders/git'
  args: ['config', '--global', 'credential.helper', "'store --file /workspace/git-credentials'"]
# Echo git credentials: (Incorrectly outputs `gcloud.sh` instead the file above)
- name: 'gcr.io/cloud-builders/git'
  args: ['config', '--global', '--list']
# Pull source repo
# Doesn't work, same problem as above (it's not using the store, as if the command remembered nothing)
- name: 'gcr.io/cloud-builders/git'
  args: ['clone', 'https://github.com/myprivate/repo.git']

[bendory]

@ImJasonH is working on a better solution for this, but in the meantime, I think the following workaround will help you move forward:

steps:
# Unencrypt the git credentials
# Note that I changed your --plaintext-file arg to .git-credentials
- name: 'gcr.io/cloud-builders/gcloud'
  args:
  - kms
  - decrypt
  - --ciphertext-file=git-credentials.enc
  - --plaintext-file=.git-credentials
  - --location=global
  - --keyring=github-read
  - --key=github-read-key
# Note that I changed your --file arg to .git-credentials
- name: 'gcr.io/cloud-builders/git'
  args: ['config', '--global', 'credential.helper', "'store --file /workspace/.git-credentials'"]
  env: ['HOME=/workspace']
# Echo git credentials
- name: 'gcr.io/cloud-builders/git'
  args: ['config', '--global', '--list']
  env: ['HOME=/workspace']
# Pull source repo
# Doesn't work, same problem as above (it's not using the store, as if the command remembered nothing)
- name: 'gcr.io/cloud-builders/git'
  args: ['clone', 'https://github.com/myprivate/repo.git']
  env: ['HOME=/workspace']

Full explanation, in case the wizardry isn't clear...
Because the /workspace directory tree are the only files that persist between build steps, you need your git credentials to live there. git expects to store config in ~ and read credentials from ~/.git-credentials , but $HOME isn't persisted. So what I've done here is persisted the credentials in /workspace/.git-credentials and then set HOME=/workspace so that git's configuration files are persisted across build steps.

Does that make sense? Does it help move you forward?

Disclaimer: I didn't run the above build steps, they may be not quite right, please give a try...

[purohit]

@bendory Thank you for the response and explanation, it did help me move forward :D

[lestrrat]

I followed this, and I can't seem to get this to work. what's the correct way to do it?

Step #5: git: 'credential-store --file /workspace/.git-credential' is not a git command. See 'git --help'.

[purohit]

@lestrrat it doesn't look like your command is using the right syntax (it's git config credential.helper [...]: https://git-scm.com/docs/git-credential-store)

[lestrrat]

@purohit that's not what I typed, that's the git error after executing.

With the quoting as is in https://github.com/GoogleCloudPlatform/cloud-builders/issues/44#issuecomment-311517375, I get

Step #6: 01:22:37.646078 git.c:369               trace: built-in: git 'clone' 'https://github.com/my_project/my_repo.git'
Step #6: Cloning into 'bunto'...
Step #6: 01:22:37.670798 run-command.c:369       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/my_project/my_repo.git'
Step #6: 01:22:38.058218 run-command.c:369       trace: run_command: 'git credential-gcloud.sh get'
Step #6: 01:22:38.059135 run-command.c:228       trace: exec: '/bin/sh' '-c' 'git credential-gcloud.sh get' 'git credential-gcloud.sh get'
Step #6: 01:22:38.081128 git.c:594               trace: exec: 'git-credential-gcloud.sh' 'get'
Step #6: 01:22:38.081269 run-command.c:369       trace: run_command: 'git-credential-gcloud.sh' 'get'
Step #6: 01:22:38.420456 run-command.c:369       trace: run_command: 'git credential-'\''store --file /workspace/.git-credentials'\'' get'
Step #6: 01:22:38.421205 run-command.c:228       trace: exec: '/bin/sh' '-c' 'git credential-'\''store --file /workspace/.git-credentials'\'' get' 'git credential-'\''store --file /workspace/.git-credentials'\'' get'
Step #6: 01:22:38.422737 git.c:594               trace: exec: 'git-credential-store --file /workspace/.git-credentials' 'get'
Step #6: 01:22:38.422808 run-command.c:369       trace: run_command: 'git-credential-store --file /workspace/.git-credentials' 'get'
Step #6: git: 'credential-store --file /workspace/.git-credentials' is not a git command. See 'git --help'.
Step #6: fatal: could not read Username for 'https://github.com': No such device or address
Finished Step #6

With quotes changed to

  args:
    - 'config'
    - '--global'
    - 'credential.helper'
    - 'store --file /workspace/.git-credentials'

Step #6: 01:24:14.408691 git.c:369               trace: built-in: git 'clone' 'https://github.com/my_project/my_repo.git'
Step #6: Cloning into 'bunto'...
Step #6: 01:24:14.434288 run-command.c:369       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/my_project/my_repo.git'
Step #6: 01:24:14.800590 run-command.c:369       trace: run_command: 'git credential-gcloud.sh get'
Step #6: 01:24:14.801279 run-command.c:228       trace: exec: '/bin/sh' '-c' 'git credential-gcloud.sh get' 'git credential-gcloud.sh get'
Step #6: 01:24:14.818346 git.c:594               trace: exec: 'git-credential-gcloud.sh' 'get'
Step #6: 01:24:14.818435 run-command.c:369       trace: run_command: 'git-credential-gcloud.sh' 'get'
Step #6: 01:24:15.104658 run-command.c:369       trace: run_command: 'git credential-store --file /workspace/.git-credentials get'
Step #6: 01:24:15.105289 run-command.c:228       trace: exec: '/bin/sh' '-c' 'git credential-store --file /workspace/.git-credentials get' 'git credential-store --file /workspace/.git-credentials get'
Step #6: 01:24:15.106617 git.c:594               trace: exec: 'git-credential-store' '--file' '/workspace/.git-credentials' 'get'
Step #6: 01:24:15.106677 run-command.c:369       trace: run_command: 'git-credential-store' '--file' '/workspace/.git-credentials' 'get'
Step #6: fatal: could not read Username for 'https://github.com': No such device or address

So this seems to have been a problem with how the Cloud Builder's interpretation of command arguments when certain quoting rules are applied.

Subsequent errors turned out to be a problem with how I wrote the git-credentials file. I will post a wrap up when everything works.

[biggestT]

@bendory I finally seems to have got this working with SSH as well. Thought that was nice because of Githubs deploy keys feature. The setup I use is like this:

cloudbuild.yaml

steps:
- name: 'gcr.io/cloud-builders/gcloud'
  args:
  - kms
  - decrypt
  - --ciphertext-file=deployment/id_rsa.enc
  - --plaintext-file=deployment/id_rsa
  - --location=[location]
  - --keyring=[keyring]
  - --key=[key]

# Prepare for github cloning
- name: 'gcr.io/cloud-builders/gcloud'
  dir: deployment
  entrypoint: ./git-prepare.sh
  volumes:
  - name: 'ssh-setup'
    path: /root/.ssh

- name: 'gcr.io/cloud-builders/git'
  env:
  - GIT_SSH=deployment/git-ssh.sh 
  args:
  - clone
  - [email protected]:[org]/[github-reponame]
  volumes:
  - name: 'ssh-setup'
    path: /root/.ssh

deployment/git-prepare.sh

#!/bin/bash

chmod 600 id_rsa
ssh-keyscan github.com > /root/.ssh/known_hosts

deployment/git-ssh.sh

#!/bin/bash

ssh -i /workspace/deployment/id_rsa $1 $2

[bendory]

@biggestT Thanks for the write-up!

[cedrelek]

Here i s an other way :
Create with gcloud a source repository.
https://cloud.google.com/source-repositories/docs/
In the console it is :
https://console.cloud.google.com/code/develop/repo

If your have for example a git repository : [email protected]:org42/billing-service.git
Name the source repository org42-billing-service in gcloud, connect this source to your github private repository.
Then in your cloudbuilder.yaml :

 name: 'gcr.io/cloud-builders/git'
 args:
  - clone
  - https://source.developers.google.com/p/$PROJECT_ID/r/org42-billing-service

[alexanderGalushka]

@Philmod @bendory has the integration solution been worked on?
@lestrrat I also have a trouble with credential.helper store, have you got it to work?

[Philmod]

Hey @alexanderGalushka , we now have a tutorial about this topic: https://cloud.google.com/container-builder/docs/access-private-github-repos
Does that help?

[alexanderGalushka]

@Philmod I think it should, I'm using govendor golang package manager and by default it is https git clone, I'll try to git config --global url."[email protected]:".insteadOf "https://github.com/" and follow provided tutorial, thank you.

[lestrrat]

@Philmod for the record, I wrote up what I did here https://medium.com/@lestrrat/taming-google-container-builder-22a6dded155c

[pepedocs]

@Philmod I have tried the tutorial you have specified at

https://cloud.google.com/container-builder/docs/access-private-github-repos

It worked for me but I had to do a little bit of tweaking of the cloudbuild.yaml, I guess this tutorial was designed for non-triggered builds only. In my case, I needed to use this builder/git for a branch triggered build because the default cloned/mounted repo was a detached repo (it didn't link to the parent and does not retrieve the branches), hence I have to use this builder so I can make use of some git commands, for example checking if a file/dir has changed. Just an FYI.

[imjasonh]

@pepedocs By default when the build is on a Git repo, we do a "shallow clone" of the repo, which contains only the commit being built. This results in a faster build when the repo is very large, since the shallow clone doesn't have to fetch the full history of the repo, or other branches.

If you want to "unshallow" the clone, you can do so by adding this step:

- steps:
  name: gcr.io/cloud-builders/git
  args: ['fetch', '--unshallow']

[pepedocs]

@ImJasonH Great. So that just mean I do not have to use the git cloud builder then for my use case. I just have to use the cloned repo (with --unshallow) and then just run git commands from there. Thank you.

---UPDATE---

@ImJasonH I guess you were referring to the this git cloud builder when I said that the cloned repo was detached (see my previous post). Just to clarify that the "detached" repo I was referring to was the one cloned and mounted (to the container) by the github triggered build (not the one cloned by the git cloud builder). However, it helps that you mentioned that the git cloud builder clones a shallow repo. I'm still going to use this git cloud builder to properly clone a repo and run git commands with it. Thank you.

[avinassh]

So, I ended up writing a little tool which gets temporary access token from Github and use that token in builds.

Code - Fidelius

post explaining more details - link

[hegdeph]

@pepedocs
What was your solution for triggered builds ?

[pepedocs]

@hegdeph For the cloned repo by GCB I ran an unshallow git fetch command. For other repos I used Google kms to clone the repo.

[automaticalldramatic]

Did anyone here find a solution ? I have my builds failing because of a go.mod dependency on a private repository. I can clone the repo in my cloud builds but the app deploy step fails with

fatal: could not read Username for 'https://github.com': terminal prompts disabled

I have tried setting the git config --global flag as well, but to no avail :(

- name: 'gcr.io/cloud-builders/git'
    entrypoint: 'git'
    args:
      - config
      - --global
      - --add
      - [email protected]:.insteadOf
      - https://github.com/

Also, @bendory why was this closed without a resolution ?

[Shimi]

I opened a feature request at https://issuetracker.google.com/issues/150198815 to solve this problem

[crjacinro]

Any update on this problem?

I have a private Go repo sitting on Gitlab which is a dependency of another go app. When I do, app deploy, build is failing because it cannot access my private repo.

I was able to setup a GCloud Secret Manager to authorize tokens in Gcloud builds and was able to trigger a build successfuly but the app deploy still fails. It seems that the app deploy is not using the tokens I setup in cloud build.

I followed the steps in this document:

https://cloud.google.com/cloud-build/docs/access-private-github-repos

[bendory]

For support with the hosted Google Cloud Build service, please escalate to your Google Cloud Support team or use the public issue tracker at
https://issuetracker.google.com/issues/new?component=190802&template=1162743.