Cloud-builders: Google Cloud Build for Github does not fetch submodules

[williamli]

I am using the Google Cloud Build for Github and it seems that it checkouts a copy (non git) of source code excluding submodules into the work directory.

Is there a way to have the submodules checked out as well?

[williamli]

My current way around this is to setup GitHub private repo permissions and manually pull in other repos as one of the first few steps of my builds.

[CallumAttryde]

Also seeing this, not sure why the --recurse-submodules flag is not included in the default git clone hook. For my use case of simply building the contained Dockerfile, the following cloudbuild config works (though note I only have public modules so YMMV):

steps:
- name: gcr.io/cloud-builders/git
  args: ['submodule', 'update', '--init', '--recursive']
- name: gcr.io/cloud-builders/docker
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/code', '.']
images: ['gcr.io/$PROJECT_ID/code']

[captain-kark]

Here's what I did, and overall I'm pretty happy with it.

First, go to the google cloud tutorial on setting up access to private github repos, since this will come up quite a bit and represents the outline of what I used to get a custom docker image put together for the single purpose of cloning submodules. The important steps include giving access to the cloudbuilder service account to the KMS, creating an ssh key to use for cloning, and setting that key for use by google cloud builder as a github deploy key.

Next, make a Dockerfile that looks something like this:

FROM gcr.io/cloud-builders/gcloud

COPY github_gcb_deploy_key_rsa.enc /
COPY known_hosts /root/.ssh/known_hosts
COPY ssh_config /root/.ssh/config
COPY submodule_cloner.sh /

WORKDIR /workspace

ENTRYPOINT ["/bin/bash"]
CMD ["/submodule_cloner.sh"]

You get the encrypted private key from this step in the tutorial. Just take the encrypted private key and toss it into the docker image directly. Note that I've changed the name away from the tutorial's key name to something more meaningful than github-key.

Here is the source code for the submodule_cloner.sh script. Be sure to change the values marked CHANGE_ME to something you use, or just use the ones from the tutorial.

#!/bin/bash

gcloud kms decrypt \
       --ciphertext-file=/github_gcb_deploy_key_rsa.enc \
       --plaintext-file=/root/.ssh/id_rsa \
       --location=CHANGE_ME \
       --keyring=CHANGE_ME \
       --key=CHANGE_ME

chmod 600 /root/.ssh/id_rsa

git submodule update --init

The ssh config is pretty straightforward, but by including it in the image you get away from the mess of a cloudbuild.yaml file they have in their example in the tutorial.

Host github.com
     IdentityFile /root/.ssh/id_rsa

Same goes for the known_hosts file, they cover how you should generate that in the tutorial as well. Again, just drop it into the image directly to avoid dealing with volumes in your cloudbuild.yaml file.

Finally, your clone submodule step in your project's resulting cloudbuild.yaml file is really clean and easy to use.

steps:
- name: gcr.io/${PROJECT_ID}/tools/github/submodule_cloner
  id: 'configuration-submodule'
# more steps here ...

You can only use this deploy key in a single repo, as github does not allow you to reuse deploy keys. If you need more than one key, you should make the key name/ring/region environment variables, and check in more than one .enc private key file to support multiple keys.

[tonymet]

I'm using the github trigger, and submodules are not being cloned. Here is my workaround, derived from the private repo tutorial

I put id_rsa.enc and known_hosts in storage, but feel free to put them in your build

e.g.

gsutil ls  gs://${PROJECT_ID}_cloudbuild/deps/root/.ssh/
gs://XXXXX_cloudbuild/deps/root/.ssh/
gs://XXXXX_cloudbuild/deps/root/.ssh/id_rsa.enc
gs://XXXXX_cloudbuild/deps/root/.ssh/known_hosts

steps:
- name: 'gcr.io/cloud-builders/gsutil'
  args:
  - 'cp'
  - '-r'
  - 'gs://${PROJECT_ID}_cloudbuild/deps/root/.ssh'
  - '/root/'
  volumes:
  - name: 'ssh'
    path: /root/.ssh
- name: 'gcr.io/cloud-builders/gcloud'
  args:
  - kms
  - decrypt
  - --ciphertext-file=/root/.ssh/id_rsa.enc
  - --plaintext-file=/root/.ssh/id_rsa
  - --location=global
  - --keyring=${PROJECT_ID}
  - --key=github-key
  volumes:
  - name: 'ssh'
    path: /root/.ssh
# Set up git with key and domain.
- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    chmod 600 /root/.ssh/id_rsa
    cat <<EOF >/root/.ssh/config
    Hostname github.com
    IdentityFile /root/.ssh/id_rsa
    EOF
  volumes:
  - name: 'ssh'
    path: /root/.ssh
- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
   - '-c'
   - |
      git submodule init
      git submodule update
  volumes:
  - name: 'ssh'
    path: /root/.ssh

[bendory]

This GitHub issue tracker is intended for bugs with the officially supported builder images specifically.
Please report issues and feature requests regarding the GCB service to our public issue tracker at
https://issuetracker.google.com/issues/new?component=190802&template=1162743

[jakebiesinger-onduo]

FYI this is now tracked as https://issuetracker.google.com/issues/123060361

[popaaaandrei]

Try this (I use BitBucket but change the url appropriately):

- id: git-submodule
  name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  env: ['GIT_DISCOVERY_ACROSS_FILESYSTEM=1']
  args:
  - '-c'
  - |
    git init
    git config -f .gitmodules submodule.packages.url https://source.developers.google.com/p/[project]/r/bitbucket_[submodule]
    git submodule init
    git submodule update

It works for me.
Thanks

[killthekitten]

@jakebiesinger-onduo the link is not accessible from the outside, is there anything public?

[jakebiesinger-onduo]

@killthekitten Sorry, Google rewrites my links on their tracker. Try https://issuetracker.google.com/issues/123060361

You need a Google account, I think, but not one that can access google corp resources. I've update my link above.

[killthekitten]

@jakebiesinger-onduo sweeet, this one works 👍 thanks!

[cblage]

Thanks @popaaaandrei - you just saved me a ton of time!

[edmondo1984]

Do @popaaaandrei do you need other steps before that step to clone the main repository? The submodules are not cloned for me, i.e. git submodule init / git submodule update does not update the .git/config

[here-nerd]

Thanks @popaaaandrei, this works for me.
It's good to note that submodule.[packages].url .. the [packages] is the name of your submodule as written in .gitmodules.

@edmondo1984, you might need to mirror your submodule repo in Google Cloud Repository before you can fetch it. This way you don't have to read the submodule repo directly from BitBucket or GitHub at the cloudbuild-time.

[Mhm0ud]

I did the following and it's working for me, and it's more efficient if you have multiple submodules:
I followed these instructions to access my private repo from google cloud:

1. Create an SSH key
2. Store the private SSH key in Secret Manager
3. Add the public SSH key to your private repository's deploy keys (if you need to access more than one repo, create a user or use an existing user who has access to these repos and put the deploy key in this account, not in the repo itself > from GitHub account > settings > SSH keys)
4. Grant permissions to Cloud Build service account to access Secret Manager
5. Add the public SSH key to known hosts (I stored the public key as a variable in cloud build and you can use GitHub secret to store it)

*Use this command to get the public key and don't copy it from .pub file
ssh-keyscan -t rsa github.com > known_hosts.github

Then these steps in the cloud build file:

- name: 'gcr.io/cloud-builders/git'
  secretEnv: ['SSH_KEY']
  entrypoint: 'bash'
  args:
  - -c
  - |
    echo "$$SSH_KEY" >> /root/.ssh/id_rsa
    chmod 400 /root/.ssh/id_rsa
    echo ${_SSH_PUBLIC_KEY} >> /root/.ssh/known_hosts
  volumes:
  - name: 'ssh'
    path: /root/.ssh

- name: 'gcr.io/cloud-builders/git'
  entrypoint: 'bash'
  args:
   - '-c'
   - |
      git submodule init
      git submodule update
  volumes:
  - name: 'ssh'
    path: /root/.ssh

availableSecrets:
  secretManager:
  - versionName: projects/[GCP_Project]/secrets/[SECRET_NAME]/versions/latest
    env: 'SSH_KEY'