Jib: Maven jib:dockerBuild in Google Cloud Build fails

[dgvigil]

Description of the issue:
Attempting to build and deploy a container from within Google Cloud Build.
jib:build wants me to authenticate within the same project where GCB and GCR run. It feels wrong to attempt to add this so I am attempting to use jib:dockerBuild.

  - name: 'gcr.io/cloud-builders/mvn'
    id: MVN_COMPILE
    args: ['compile','jib:dockerBuild', '-Dmaven.test.skip=true', '-Ddocker.image.prefix=gcr.io/$PROJECT_ID', '-s', '.mvn/settings.xml']

images:
  - 'gcr.io/$PROJECT_ID/my-company-project'

But I am getting an error that Google Cloud Build does not have a docker daemon running. I can run both the jib:build and jib:dockerBuild locally, but need the CI to work as a trigger in Google Cloud Builder.

Steps to reproduce:
Simply using Google cloud build via triggers or gcloud builds submit .

Environment:

jib-maven-plugin Configuration:

            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>jib-maven-plugin</artifactId>
                <version>1.0.0</version>
            </plugin>

Log output:

Step #3 - "MVN_COMPILE": [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:1.0.0:dockerBuild (default-cli) on project my-company-project: Build to Docker daemon failed, perhaps you should make sure Docker is installed and you have correct privileges to run it -> [Help 1]

[chanseokoh]

Hi @dgvigil,

GCB recently dropped Docker from gcr.io/cloud-builders/mvn. In fact, now the image has essentially become the official mvn Docker Hub image. Therefore, jib:dockerBuild no longer works.

Rather, jib:build should just work and push the built image to the GCR. Unforunately, as you mentioned, you have an authentication issue with jib:build on GCB (internal bug: 124388903).

UPDATE: the following workaround for the authentication issue is no longer necessary with recent Jib versions. Note you can't still do jib:dockerBuild on gcr.io/cloud-builders/mvn, because the Maven image doesn't have Docker installed.

obsolete workaround

However, there is a bug that prevents jib:build from working. On GCB, /builder/home/.docker/config.json, which holds the GCR credentials, is not being copied/linked into /root/.docker/config.json.

That said, one ugly workaround you can use is to link that file yourself. Something like

steps:
- name: gcr.io/cloud-builders/mvn
  dir: /root
  entrypoint: bash
  args:
  - -c
  - # Links the Docker config to /root/.docker/config.json so that Jib picks it up.
    # Note that this is only a temporary workaround.
    # See https://github.com/GoogleContainerTools/jib/pull/1500.
    |
    mkdir .docker &&
    ln -s /builder/home/.docker/config.json .docker/config.json
  volumes:
  - name: user.home
    path: /root

- name: gcr.io/cloud-builders/mvn
  args: ['-B', 'compile', 'jib:build', '-Dmaven.test.skip=true', ...]
  volumes:
  - name: user.home
    path: /root

(I know this is not pretty. Should be unnecessary once the bug is fixed.)

Or,

# This is untested, but should work basically.
- name: gcr.io/cloud-builders/mvn
  entrypoint: bash
  args:
  - -c
  - |
    ln -s /builder/home/.docker /root/.docker &&
    mvn -B compile jib:build -Dimage=gcr.io/$PROJECT_ID/chanseok-test

Or, see https://github.com/GoogleContainerTools/jib/issues/1500#issuecomment-477364975.

[chanseokoh]

UPDATE: the following workarounds to sidestep the Docker credential issue are no longer needed in recent Jib versions.

obsolete workarounds

Another option is to have docker-credential-gcr on $PATH by whatever means:

- name: gcr.io/cloud-builders/mvn
  entrypoint: bash
  args:
  - -c
  - |
    export PATH=$$PATH:. &&
    curl -fsSL https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz | tar zxv &&
    chmod +x docker-credential-gcr &&
    mvn -B compile jib:build -Dimage=gcr.io/$PROJECT_ID/chanseok-test

or

- name: gcr.io/cloud-builders/mvn
  dir: /usr/local/bin
  entrypoint: bash
  args:
  - -c
  - |
    curl -fsSL https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v1.5.0/docker-credential-gcr_linux_amd64-1.5.0.tar.gz | tar zxv &&
    chmod +x docker-credential-gcr
  volumes:
  - name: usr.local.bin
    path: /usr/local/bin
- name: gcr.io/cloud-builders/mvn
  args: ['-B', 'compile', 'jib:build', '-Dimage=gcr.io/$PROJECT_ID/chanseok-test']
  volumes:
  - name: usr.local.bin
    path: /usr/local/bin

[janwillies]

Thanks for the workarounds @chanseokoh. Appreciate a quick heads-up here once the internal bug is fixed.

[dgageot]

UPDATE: the following workarounds to sidestep the Docker credential issue are no longer needed in recent Jib versions.

obsolete workarounds

I found a simpler solution:

- name: gcr.io/cloud-builders/mvn
  args: ['-Duser.home=/builder/home', 'compile', 'jib:build', ...]

This makes sure that the user.home seen by java/maven is aligned with $HOME.

This might also make the build run faster because maven will stop downloading files under /root, that's inside the docker container, and rather download them under the /builder/home volume.
That also has the advantage of being able to share that folder between multiple steps (I think).

[chanseokoh]

@dgageot in fact we were discouraging setting user.home to /builder/home, because the home directory of the user root points to /root on GCB (for example, Maven caches artifacts under /root/.m2) and -Duser.home= changes that JVM-wise globally. So for example, -Duser.home=/builder/home makes Maven cache artifacts under /builder/home/.m2 instead of /root/.m2, which is a breaking change. There has been a discussion around what the home directory of the user root should actually be on GCB, and because currently it is /root we have been recommending not overriding it.

[dgageot]

So, on GCB, $HOME is set to /builder/home. It's only java's user.home that is set to /root for a reason that I'm still investigating.

Also, yes, if you set the user.home to /builder/home it will indeed also download maven artifacts under /builder/home/.m2 which is great because then you are not writing inside the container but in a shared volume that is faster and shared by build steps and could be cached.

[chanseokoh]

Oh, yeah, my mistake about $HOME, sorry. To be fair, I did know $HOME points to /builder/home, which is very confusing actually (I did note this in the internal bug before), because the OS-defined home directory for the user root is still /root. (The Maven runs as root and stores artifacts under /root/.m2.)

$ getent passwd $( id -u )
root:x:0:0:root:/root:/bin/bash

So, it is not that only Java is thinking the home of the root user is /root. It is rather that $HOME alone happens to point to a different directory /builder/home.

[dgageot]

I think java is basically reading the user's home directory from /etc/password -> root
GCB is setting the HOME variable to /builder/home when docker running the build step.

[dgageot]

https://stackoverflow.com/questions/1503284/java-system-getpropertyuser-home-returns/1503500#1503500

[dgageot]

Anyways, I don't really see the risk of changing user.home

[chanseokoh]

This issue will be one of those that the new feature #1902 will fix. Once #1902 gets in, no more extra configuration will be needed for GCB. Will update here once it goes live.

[chanseokoh]

1.5.0 released that can pick up the Google Application Default Credentials on most GCP environments including GCB. In many cases, Jib will work out of the box on GCB by default with no extra configuration.

If anyone sees an issue, please follow up on #1903.

@dgvigil @ractive @janwillies @grimrose @Matrix89 @priyawadhwa @dgageot @Alos