Cloud-builders: Fetch repositories with submodules

[onionjake]

It is unclear how repositories with submodules can be built.

This is what the build currently does:

Initialized empty Git repository in /workspace/.git/
... <fetches from the single specified repository> ...

It is not clear if it is possible to use the cloud-builders/git container somehow to fetch all of the repositories. Is the workspace cleared between each buildstep? What if they are executed in parallel?

[skelterjohn]

Are you sure no subrepo is fetched? We populate the workspace like so:

git config --local credential.helper gcloud.sh && \
(git remote add origin $REMOTE || true) && \
git fetch --depth=1 --recurse-submodules=yes --tags origin $REV && \
git reset --hard $REV

The workspace is initialized once before any build steps run, and it is not cleared between steps. If steps are executed in parallel, normal concurrent disk semantics apply (so be careful about writing the same file from two build steps).

[onionjake]

Yes. This is the the output before the BUILD step. There is no mention of attempts to fetch submodules.

Initialized empty Git repository in /workspace/.git/
From https://source.developers.google.com/p/<my-project>/r/<my-repo>
 * branch            <my-sha1-long> -> FETCH_HEAD
HEAD is now at <my-sha1-short> use cloudbuild
BUILD

[mhr3]

I'm also unable to fetch submodules, which is probably related to using Google Cloud Repository that mirrors a GitHub repo. After running git submodule update using the cloud-builders/git image, I just get:

Step #0: Host key verification failed.
Step #0: fatal: Could not read from remote repository.

~Is there any workaround? Can I somehow get to the deploy key that was used for the mirroring and convince git to use that to update the submodules?~

Nevermind, I forgot that deploy keys are specific to a single repo.

[cedrelek]

You can use the source repositories of gcloud instead of github. Read my comment in the #44
I perform a step in my cloudbuild.yaml which rename the repositories :

- id: renameGitHubUrl
  name: debian
  args:
  - sed
  - -i
  - 's#[email protected]:___GitHubId___/\(.*\)\.git#https://source.developers.google.com/p/___GCLOUD_PROJECT_ID___/r/github-___GitHubId___-\1#g'
  - .gitmodules

As soon as your private github repository are well named in gcloud source repositories, it will be correct. After this step you can submodule init and update.

[juancampa]

Wondering if this is scheduled to be fixed anytime soon. @ImJasonH

[GM12Tick]

Is this still an issue?

[imjasonh]

@gmistick Yes, this is still not solved. The issue is that the repo you build from is a mirror that the builder service account has access to, and that account might not have access to private submodules defined in the repo.

/cc @wlynch

[bussyjd]

Can we have it please? @juancampa @wlynch

[cbuchacher]

We currently use the approach described here: https://cloud.google.com/cloud-build/docs/access-private-github-repos
Instead of the git clone step we do git submodule update.

[pst]

For anyone coming across this: An easier way than creating deploy keys, encrypting them with kms and adding multiple additional steps to your pipeline is to simply have a Google mirror the submodule as well. Then run two commands to temporarily change the submodule config on disk to point to the Google mirror before updating the submodule. Like so:

# Update submodules
- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    git config -f .git/config submodule.one.url https://source.developers.google.com/p/[project_id]/r/[repo_one]
    git config -f .gitmodules submodule.one.url https://source.developers.google.com/p/[project_id]/r/[repo_one]
    git submodule init
    git submodule update

[youens]

For what it's worth, pst's comment saved the day. THANKS!

That said, I must admit I had to poke around to make sense of it, so perhaps these more detailed steps will save the next person time...

To mirror the submodule, go to Cloud Source Repositories. That gave me a repo like:
https://source.cloud.google.com/[my-project]/github_[my-sub-repo-name]

In order for Cloud Build to gain access, I needed to rewrite like:
https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]

With that new url, I followed the example above... but I did not need to update .git/config, only .gitmodules.

So:

- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    git config -f .gitmodules submodule.[my-sub-repo-name].url https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]
    git submodule init
    git submodule update

I added this to my cloudbuild.yaml as the first step. Cloud Build then had no issues cloning the submodule during the build.

Your mileage may vary with a different setup. I do hope Google adds proper support to do this automatically in the near future. What a pain.

[Fleker]

Unfortunately the advice above didn't seem to work in my case, I just cloned the repo in my cloud build configuration.

[mairh]

For what it's worth, pst's comment saved the day. THANKS!

That said, I must admit I had to poke around to make sense of it, so perhaps these more detailed steps will save the next person time...

To mirror the submodule, go to Cloud Source Repositories. That gave me a repo like:
https://source.cloud.google.com/[my-project]/github_[my-sub-repo-name]

In order for Cloud Build to gain access, I needed to rewrite like:
https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]

With that new url, I followed the example above... _but_ I did not need to update .git/config, only .gitmodules.

So:

- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    git config -f .gitmodules submodule.[my-sub-repo-name].url https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]
    git submodule init
    git submodule update

I added this to my cloudbuild.yaml as the first step. Cloud Build then had no issues cloning the submodule during the build.

Your mileage may vary with a different setup. I do hope Google adds proper support to do this automatically in the near future. What a pain.

@pst @youens What did you update in your .gitmodules file after the change in yaml. I also made the changes you wrote in your comment but getting the following error.

Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/git
Step #1: fatal: not a git repository (or any parent up to mount point /)
Step #1: Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Step #1: fatal: not a git repository (or any parent up to mount point /)
Step #1: Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/git" failed: exit status 128

[rodolfo-picoreti]

@mairh if you triggering the build from Github it will not work since the .git folder will not be present. All repositories need to be fetched from Cloud Source Repositories.

[popaaaandrei]

Guys,

I never ever thought this would actually work, but it did :))

- id: git-submodule
  name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  env: ['GIT_DISCOVERY_ACROSS_FILESYSTEM=1']
  args:
  - '-c'
  - |
    git init
    git config -f .gitmodules submodule.[my-sub-repo-name].url https://source.developers.google.com/p/[my-project]/r/github_[my-sub-repo-name]
    git submodule init
    git submodule update

💃

[arinto]

Mine is like this, but with these following conditions

1. Original private repo is in Github
2. Integration with Cloudbuild Github App
3. Parent Github repo must be mirrored to Google Cloud Source Repositories.
4. Submodule repos also need to be mirrored to Google Cloud Source Repositories

Here's the Cloudbuild step:

  - id: 'initialize_repo'
    name: 'gcr.io/cloud-builders/git'
    entrypoint: 'bash'
    env: ['GIT_DISCOVERY_ACROSS_FILESYSTEM=1']
    args:
      - '-c'
      - |
        git init
        git clean  -d  -f . #clean current working directory
        git remote add origin https://source.developers.google.com/p/[gcp-project]/r/[parent-repo-name-in-google-cloud-source]
        git fetch origin $BRANCH_NAME
        git checkout $COMMIT_SHA #checkout at current commit
        git config -f .gitmodules submodule.[submodule-repo-name].url https://source.developers.google.com/p/[gcp-project]/r/[submodule-repo-name-in-google-cloud-source]
        git submodule init
        git submodule update

Notes: well, it's very very very hacky. Basically we re-initialize the repo and discard the source code obtained via Github App. Hopefully Cloudbuild will have better support for git submodule in the future :)

[bendory]

Note: please report issues with and feature requests for the hosted Google Cloud Build service to your Google Cloud Support team, or use the public issue tracker at
https://issuetracker.google.com/issues/new?component=190802&template=1162743.

I'm closing this issue as "off-topic."