Cloud-builders: Mounting files in /workspace to docker run steps

[FA-tvonmoll]

This is somewhat related to #345.

Following the steps in this comment works totally fine for me, and I'm able to view the decrypted contents of that file. The trouble happens when I attempt to mount that file as a bind volume in later steps.

I might simply be missing what the absolute path of files in /workspace actually are.

Example cloudbuild.yaml (this results in cat: read error: Is a directory)

steps:
    - name: 'alpine'
      args: ['ls', '-la']
    - name: 'gcr.io/cloud-builders/gcloud'
      args: ['kms', 'decrypt', '--location=global', '--keyring=<ring>', '--key=<key>', '--ciphertext-file=.foo.enc', '--plaintext-file=.hidden.txt']
    - name: 'alpine'
      args: ['ls', '-la']
    - name: 'alpine'
      args: ['cat', '.hidden.txt']
    - name: 'gcr.io/cloud-builders/docker'
      args: ['run', '-v', '/workspace/.hidden.txt:/data/.hidden.txt', 'alpine', 'cat', '/data/.hidden.txt']

Trying with relative ./.hidden.txt is also getting me nowhere.

The goal is to get to the point where I can compose up everything for integration testing as described here. The compose cloud builder behaves identically to the vanilla docker one in this scenario.

Thanks!

[bendory]

The supported way to do this is to use the volume option to mount a volume into the decrypt step, write the plaintext file into that volume, then use the same volume option in the step that uses the file.

See documentation here: https://cloud.google.com/cloud-build/docs/build-config (search for "volumes")

[FA-tvonmoll]

Thanks, @bendory !

I tried with a non-default volume, but it had the same result. cloudbuild.yaml example is:

steps:
    - name: 'gcr.io/cloud-builders/gcloud'
      args: ['kms', 'decrypt', '--location=global', '--keyring=<ring>', '--key=<key>', '--ciphertext-file=.foo.enc', '--plaintext-file=/secrets/.hidden.txt']
      volumes:
      - name: 'secret-files'
        path: '/secrets'
    - name: 'alpine'
      args: ['cat', '/secrets/.hidden.txt']  # this works
      volumes:
      - name: 'secret-files'
        path: '/secrets'
    - name: 'gcr.io/cloud-builders/docker'
      args: ['run', '-v', '/secrets/.hidden.txt:/data/.hidden.txt', 'alpine', 'cat', '/data/.hidden.txt'] # this fails
      volumes:
      - name: 'secret-files'
        path: '/secrets'

Am I missing something?

[bendory]

What is it that you are trying to do in that last build step where you docker run just to cat? What is the actual build step you are trying to run there that needs access to /data/.hidden.txt?

[FA-tvonmoll]

That .hidden.txt would be a config file. My goal is to be able to run the container from the newly-built image and supply that config file to run integration tests in the following build step (ultimately, this would be a $ compose up rather than running a single container as described here https://cloud.google.com/cloud-build/docs/configuring-builds/build-test-deploy-artifacts#running_unit_tests_and_integration_tests).

[bendory]

So you presumably need the file to land at /data/.hidden.txt, right? Is there a reason that you can't define the volume as /data rather than /secrets so that it simply lands there?

[FA-tvonmoll]

Yep, that's the idea!

Are you saying that I don't need to include the -v flags in the docker run step? I'm still having trouble getting that volume to mount.

Here are some minimally-reproducible examples using (mostly) the same yaml from the example in the build-config docs (and simplified from before because the issue doesn't appear to have anything to do with the decrypt step). Regardless of the purpose, /persistent_volume/file should end up in the container, correct?

steps:
- name: 'ubuntu'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        echo "Hello, world!" > /persistent_volume/file
- name: 'gcr.io/cloud-builders/docker'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  args: ['run', 'alpine', 'ls', '-la']

Output of that is below (/persistent_volume isn't there).

Using default tag: latest
latest: Pulling from cloud-builders/metadata
Digest: sha256:7b1cc5e18ed10a2138a7c6c1db586fd1fe9ecadc24ff45fb731003bd4407afe6
Status: Image is up to date for gcr.io/cloud-builders/metadata:latest
2018/09/21 18:29:17 Started spoofed metadata server
2018/09/21 18:29:17 Build id = localbuild_94d2d071-a6c4-40a8-862d-6bd2e9345383
2018/09/21 18:29:17 status changed to "BUILD"
BUILD
Starting Step #0
Step #0: Already have image (with digest): ubuntu
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: total 60
Step #1: drwxr-xr-x    1 root     root          4096 Sep 21 22:29 .
Step #1: drwxr-xr-x    1 root     root          4096 Sep 21 22:29 ..
Step #1: -rwxr-xr-x    1 root     root             0 Sep 21 22:29 .dockerenv
Step #1: drwxr-xr-x    2 root     root          4096 Sep 11 20:23 bin
Step #1: drwxr-xr-x    5 root     root           340 Sep 21 22:29 dev
Step #1: drwxr-xr-x    1 root     root          4096 Sep 21 22:29 etc
Step #1: drwxr-xr-x    2 root     root          4096 Sep 11 20:23 home
Step #1: drwxr-xr-x    5 root     root          4096 Sep 11 20:23 lib
Step #1: drwxr-xr-x    5 root     root          4096 Sep 11 20:23 media
Step #1: drwxr-xr-x    2 root     root          4096 Sep 11 20:23 mnt
Step #1: dr-xr-xr-x  208 root     root             0 Sep 21 22:29 proc
Step #1: drwx------    2 root     root          4096 Sep 11 20:23 root
Step #1: drwxr-xr-x    2 root     root          4096 Sep 11 20:23 run
Step #1: drwxr-xr-x    2 root     root          4096 Sep 11 20:23 sbin
Step #1: drwxr-xr-x    2 root     root          4096 Sep 11 20:23 srv
Step #1: dr-xr-xr-x   13 root     root             0 Sep 21 22:29 sys
Step #1: drwxrwxrwt    2 root     root          4096 Sep 11 20:23 tmp
Step #1: drwxr-xr-x    7 root     root          4096 Sep 11 20:23 usr
Step #1: drwxr-xr-x   11 root     root          4096 Sep 11 20:23 var
Finished Step #1

Including -v and mounting it (similar to before) includes /persistent_volume, but it's empty:

steps:
- name: 'ubuntu'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        echo "Hello, world!" > /persistent_volume/file
- name: 'gcr.io/cloud-builders/docker'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  args: ['run', '-v', '/persistent_volume:/persistent_volume', 'alpine', 'ls', '-la', 'persistent_volume']

Output of that is below (cleaned up the first half, which is the same as above):

Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: total 8
Step #1: drwxr-xr-x    2 root     root          4096 Sep 21 22:42 .
Step #1: drwxr-xr-x    1 root     root          4096 Sep 21 22:42 ..
Finished Step #1

[FA-tvonmoll]

Ah, ok; I think I've got it.

For folks coming here after-the-fact: you _do_ still need to mount the volume normally (-v) in addition to using the volumes cloudbuild property. The trick is using the _name_ rather than the path of that volume. You can still drop that volume wherever you want inside the container.

steps:
- name: 'ubuntu'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        echo "Hello, world!" > /persistent_volume/file
- name: 'gcr.io/cloud-builders/docker'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  args: ['run', '-v', 'vol1:/data', 'alpine', 'cat', 'data/file']

Running locally results in the below (command and full stdout included).

18:56 $ cloud-build-local --config=cloudbuild.yaml --dryrun=false .
2018/09/21 18:56:32 Warning: The server docker version installed (18.06.1-ce) is different from the one used in GCB (17.12.0-ce)
2018/09/21 18:56:32 Warning: The client docker version installed (18.06.1-ce) is different from the one used in GCB (17.12.0-ce)
Using default tag: latest
latest: Pulling from cloud-builders/metadata
Digest: sha256:7b1cc5e18ed10a2138a7c6c1db586fd1fe9ecadc24ff45fb731003bd4407afe6
Status: Image is up to date for gcr.io/cloud-builders/metadata:latest
2018/09/21 18:56:39 Started spoofed metadata server
2018/09/21 18:56:39 Build id = localbuild_ac999140-ca4b-4978-8667-764b175a95ad
2018/09/21 18:56:39 status changed to "BUILD"
BUILD
Starting Step #0
Step #0: Already have image (with digest): ubuntu
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: Hello, world!
Finished Step #1
2018/09/21 18:56:43 Failed to delete volume "vol1": exit status 1
2018/09/21 18:56:43 status changed to "DONE"
DONE

[polarisdigitech]

Ah, ok; I think I've got it.

For folks coming here after-the-fact: you _do_ still need to mount the volume normally (-v) in addition to using the volumes cloudbuild property. The trick is using the _name_ rather than the path of that volume. You can still drop that volume wherever you want inside the container.

steps:
- name: 'ubuntu'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        echo "Hello, world!" > /persistent_volume/file
- name: 'gcr.io/cloud-builders/docker'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  args: ['run', '-v', 'vol1:/data', 'alpine', 'cat', 'data/file']

Running locally results in the below (command and full stdout included).

18:56 $ cloud-build-local --config=cloudbuild.yaml --dryrun=false .
2018/09/21 18:56:32 Warning: The server docker version installed (18.06.1-ce) is different from the one used in GCB (17.12.0-ce)
2018/09/21 18:56:32 Warning: The client docker version installed (18.06.1-ce) is different from the one used in GCB (17.12.0-ce)
Using default tag: latest
latest: Pulling from cloud-builders/metadata
Digest: sha256:7b1cc5e18ed10a2138a7c6c1db586fd1fe9ecadc24ff45fb731003bd4407afe6
Status: Image is up to date for gcr.io/cloud-builders/metadata:latest
2018/09/21 18:56:39 Started spoofed metadata server
2018/09/21 18:56:39 Build id = localbuild_ac999140-ca4b-4978-8667-764b175a95ad
2018/09/21 18:56:39 status changed to "BUILD"
BUILD
Starting Step #0
Step #0: Already have image (with digest): ubuntu
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: Hello, world!
Finished Step #1
2018/09/21 18:56:43 Failed to delete volume "vol1": exit status 1
2018/09/21 18:56:43 status changed to "DONE"
DONE

This was very helpful. This should be added in the documentations.

[zwstella]

Hello, I found the post super helpful. One question: Why can't we just mount default /workspace? As from the documentation https://cloud.google.com/cloud-build/docs/build-config /workspace will be persisted by default across steps.

[gustavovalverde]

But, what if I'd like to build an image which uses the mounted volume by cloud build? I can mount while using docker run, but not when using docker build

cc: @sanastos

[andreipradan]

Hello, I found the post super helpful. One question: Why can't we just mount default /workspace? As from the documentation https://cloud.google.com/cloud-build/docs/build-config /workspace will be persisted by default across steps.

@zwstella I also mounted with the above solution, Thank you @FA-tvonmoll !
However digging in a little bit more I found out how to use the existing workspace volume to not have to define custom volumes to each step.

tl;dr

steps:
- name: 'ubuntu'
  args: [ "touch", "foo" ]
- name: 'ubuntu'
  args: [ "ls", "-l" ]
- name: 'gcr.io/cloud-builders/docker'
  args: [ "run", "--volume", "/workspace:/workspace", "--rm", "ubuntu", "ls", "-l", "/workspace" ]

Source: https://github.com/GoogleCloudPlatform/cloud-builders/issues/323#issuecomment-411130485
Thanks @bendory :)

It's a little bit cleaner :)
Hope this helps!

[ggunia]

But, what if I'd like to build an image which uses the mounted volume by cloud build? I can mount while using docker run, but not when using docker build

cc: @sanastos

Did you solved that issue?

[Li-ReDBox]

The documentations need to be updated with better examples and explanations. Each build step is a simplified docker run -v /workspace:/workspace args .... a.k.a. /workspace is automatically mounted. But if your step is a args: [ "run", ... ], you have to mount a volume for persistent volume even it is /workspace.

[samhiatt]

After running a docker build, for some reason I am unable to access the built files in the default workspace from the gsutil image. I was able to get this to work by adding a docker run command before running the gsutil image.

For example, given the fllowing Dockerfile:

FROM ubuntu:latest
WORKDIR /workspace
RUN echo "Hey bar" > bar.txt
RUN mkdir /persist && cp * /persist

and the following cloudbuild.yaml:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: [
        'build',
        '--file', 'Dockerfile',
        '-t', 'gcr.io/$PROJECT_ID/cloudbuild_test:latest',
        '.'
    ]
    volumes:
      - name: 'vol1'
        path: '/persist'

  - name: 'gcr.io/cloud-builders/gsutil'
    entrypoint: "bash"
    args: [ "-c", "ls -l /workspace /persist" ]
    volumes:
      - name: 'vol1'
        path: '/persist'

the gsutil ls shows /persistent to be empty.

However, if I add a docker run step explicitly mounting the /persistent volume in the docker run command and also include the volumes attribute in the step, then following steps that mount /persistent do have the expected files in that directory. This is what works:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    args: [
        'build',
        '--file', 'Dockerfile',
        '-t', 'gcr.io/$PROJECT_ID/cloudbuild_test:latest',
        '.'
    ]

  - name: 'gcr.io/cloud-builders/docker'
    args: [ "run", "--rm", "-v", "vol1:/persist", "gcr.io/$PROJECT_ID/cloudbuild_test:latest" ]
    volumes:
      - name: 'vol1'
        path: '/persist'

  - name: 'gcr.io/cloud-builders/gsutil'
    entrypoint: "bash"
    args: [ "-c", "ls -l /workspace /persist" ]
    volumes:
      - name: 'vol1'
        path: '/persist'

Notice I don't need the volumes attribute on the docker build step, and I don't even need to run any command in the docker run step to get it to work.

Not sure if this is expected behavior. It definitely seems counter-intuitive to require both "-v", "vol1:/persist" and the volumes step attribute. It was also surprising that the first example doesn't persist the contents of /persist if the volumes attribute is included on the build step.

This took me forever to figure out, so hopefully posting here helps someone else in the same situation.

[bivald]

Leaving this reminder here since this comes up in Google searches:

If you successfully mount the volume as per @FA-tvonmoll suggestion (which I did) and get permission denied when trying to write anything to the volume, remember that you need to appropriate permissions for your docker image user. So if you change the user in your Dockerfile, you might need to run --user root :)