Cloud-builders: Make /workspace a git directory when running Builds using the GitHub App

[scopelf]

I'm using cloud-build as a CI in GitHub and want to add Codecov to display the code coverage of my tests. However, this doesn't work since Codecov can't access the git information because the /workspace is not a git directory. I propose making /workspace a git folder when using the GitHub App as build trigger.

The current option would be to clone the repo and then running Codecov, which seems to be unnecessary since the code is already in the /workspace.

[alex-tzinov]

Would love to have this as well. We are running clould-build alongside Cypress and Cypress Dashboard which relies on having a .git folder to be able to pull off Author and Message metadata to publish in test results dashboard on Cypress. Spent a good amount of time troubleshooting why our Cypress - GCB integration wasn't working before realizing that workspace isn't a git directory

[alex-tzinov]

This seems like a potential solution https://github.com/GoogleCloudPlatform/cloud-builders/issues/236

[alex-tzinov]

So it seems like the solution in 239 above only works when using the cli
(gcloud builds submit --config=clouldbuild.yaml) but does not work via a commit trigger through the GitHub App. The GitHub app does not seem to respect the gcloudignore file

[scopelf]

I did find a way to circumvent the problem (for codecov). I'm passing all pre-defined git substitutions to codecov via dash commands.

bash <(curl -s https://codecov.io/bash) -y .codecov.yml -B ${BRANCH_NAME} -C ${COMMIT_SHA} -b ${BUILD_ID} -T ${TAG_NAME}

However, it would be still easier if the git information would be available.

[peterlogg]

+1 for the ability to do this also

[ptinsley]

+1 I’m porting our current build harness over to cloud build and this has become a blocker for me. I retrieve the commit message, author and git origin url (so I can build links to the commit and branch) from the .git dir for use in slack notifications.

[dinvlad]

I found this guide helpful on how to make it work with Codecov specifically: https://medium.com/google-cloud/code-coverage-on-google-cloud-build-7fcf1b67f182

TLDR: yarn codecov --disable=detect --commit=${COMMIT_SHA} --branch=${BRANCH_NAME}

That being said, I absolutely +1 on making it a .git folder, at least as an option.

[eddiemonge]

It would also make it possible to run commit lint checking if .git was available

[zbutt-muvaki]

i am running test coverage that requires submodule to be pulled in to build the app ... need the .git folder ... cant init the submodule

- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  id: initialize-submodules
  args: 
  - '-c'
  - |
        git config -f .gitmodules submodule.core.url https://source.developers.google.com/p/$PROJECT_ID/r/github_submodule-repo
        git submodule init
        git submodule update

anyone reading.. this is how i did my work around

edit

- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  id: initialize-submodules
  args: 
  - '-c'
  - |
        git init
        git remote add origin https://source.developers.google.com/p/$PROJECT_ID/r/github_main-repo
        git fetch --depth=1 origin $COMMIT_SHA
        git reset --hard FETCH_HEAD
        git config -f .gitmodules submodule.core.url https://source.developers.google.com/p/$PROJECT_ID/r/github_submodule-repo
        git submodule init
        git submodule update

[dinvlad]

We've just ran into the same issue for SourceClear: SourceClear only supports git repos, trying to run it in gcr.io/cloud-builders/npm on a private repo. Any solutions/workarounds would be highly appreciated..

EDIT: I was able to work around that specifically for SourceClear using the following environment variables:

- SRCCLR_SCM_URI=https://github.com/${_REPO_OWNER}/${REPO_NAME}
- SRCCLR_SCM_REF=${BRANCH_NAME}
- SRCCLR_SCM_REF_TYPE=branch
- SRCCLR_SCM_REV=${REVISION_ID}

Still, such workarounds may not always be available for a particular tool.

[boredland]

This was working for the "mirror" repos (which you can see here in "Including the repository history in a build" https://cloud.google.com/cloud-build/docs/running-builds/create-manage-triggers) - but apparently the new gh-triggers are only half-implemented. If you have a repo outside the google universe you'll currently rely on dirty stuff like this:

steps:
  - name: git-image
    id: PREPARE
    entrypoint: bash
    args:
      - '-c'
      - |
        ssh-keyscan -t rsa github.com > /tmp/known_hosts
        shopt -s dotglob && rm -rf *
        git init
        git remote add origin [email protected]:MobileheadHolding/sputnik.git
        GIT_SSH_COMMAND='ssh -i /tmp/release.id_rsa -o UserKnownHostsFile=/tmp/known_hosts' \
        git fetch --depth=5 origin $BRANCH_NAME
        git checkout $BRANCH_NAME
        git reset --hard $COMMIT_SHA

[willieowens]

+1

[Datamance]

+1, things like sentry-cli that rely on commit history to manage releases can't function without this. It seems really silly to have to set up authentication for GitHub when the integration between GCB<->GitHub is already there, and then delete the source that was pulled down in the tarball just to re-clone everything.

[dinvlad]

I'd add that if we could have an option to use even a shallow clone instead of just the tarball, this would go a long way, and would kill two birds with one stone (performance and preserving the repo structures).

[Datamance]

@dinvlad Here's what I did that is _slightly_ less hackish than the above answers:

- id: get-git-history
  name: gcr.io/cloud-builders/git
  entrypoint: bash
  args:
    - '-c'
    - |
      git clone https://github.com/lookit/lookit-api.git --no-checkout
      mv lookit-api/.git .
      git fetch --unshallow
      rm -rf lookit-api

--no-checkout only downloads the .git folder, but then you gotta move it from the cloned folder to the cloud builder workspace. Then you clean up and you're good!

[dinvlad]

Nice! Only problem is, most of our repos are private ;) We can pass GitHub token via trigger config obviously, but that's a lot of extra work for something this basic..

[matthiasa4]

+1 for adding .git folder.

@dinvlad - I actually solved it using Google Cloud Secret Manager which makes this rather easy - put your GitHub OAuth token and retrieve in the Cloud Build image:

Adding secret:

echo $GITHUB_TOKEN | gcloud secrets create github-token --data-file=- --replication-policy=automatic

Retrieving secret in Build pipeline:

GITHUB_TOKEN=$(gcloud secrets versions access latest --secret=$GITHUB_SECRET_NAME)
COMMAND=$(echo 'git clone https://[email protected]/ORGANISATION/REPOSITORY' | sed -e "s/GITHUB_TOKEN/$GITHUB_TOKEN/" )
eval $COMMAND

Hope this helps :)

[dinvlad]

In addition to everything mentioned above, the GitHub App trigger appears to change file permissions to 664 from 644 (as recorded in Git).

I discovered that in a painful way, while using MD5 hash of files in a ZIP archive for deploying Cloud Functions through Terraform. MD5 hash differs when running the same command with the same files locally vs Cloud Build, and the hexdump/zipinfo shows that attributes of all files inside were changed as above.

[noelbk]

Here's a solution I also posted to https://github.com/GoogleCloudPlatform/cloud-builders/issues/236#issuecomment-621708050

We need the .git directory for several reasons, including git describe, git submodule, and git lfs. Here's a cloudbuild.yaml file I made to hack around this. This checks out the .git directory and checks out a private submodule with git-lfs files.

steps:
- name: 'MY_IMAGE_WITH_GIT_LFS'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    set -eux

    # cloudbuild doesn't check out the .git directory, so I have to
    # check it out again to get the .git directory, submodules, and
    # LFS files
    eval $(ssh-agent -s)
    mkdir -p -m 0700 /root/.ssh
    cat ci/host-github.com >> /root/.ssh/known_hosts

    # MYREPO1.key was made from ssh-keygen then added to the github
    # repo's Deploy Keys https://github.com/USER/REPO/settings/keys
    chmod 0600 ci/MYREPO1.key
    ssh-add ci/MYREPO1.key
    git init
    git lfs install
    git remote add origin [email protected]:/GITHUB_USER/$REPO_NAME.git
    git fetch --depth=1 origin $COMMIT_SHA
    git reset --hard FETCH_HEAD
    # have to switch to MYSUBREPO2.key to check out the MYSUBREPO2 submodule
    ssh-add -D
    chmod 0600 ci/MYSUBREPO2.key
    ssh-add ci/MYSUBREPO2.key
    # if there were other private repos here, I'd have to switch keys
    # and update them individually
    git submodule update --init --recursive
    # finally the .git repo is rebuild and checked out

    make test

[dinvlad]

This needs native integration! Baking your SSH keys into an image is not the best practice from security standpoint. Might be better to fetch them from Secret Manager, but once again, this is a workaround we should not be using. Cloud Build App should just fetch .git directory by default, as any other CI/CD systems do (and Cloud Build itself in non-App mode).

[seenkey]

This needs to be addressed in Cloud Build itself, not cloud-builders.

You can follow along on https://issuetracker.google.com/issues/136435027