Cloud-builders: Save and restore Docker image cache layers?

[acasademont]

We're using a multi-stage docker file and the intermediate steps are not pushed to the repository and therefore, the following build will need to rebuild all those intermediate steps. --cache-from might be helpful but those intermediate steps would need to be available in the repository and that forces us to create individual docker build --target builds for each step and then push every step to the repository, a bit of a PITA.

Is there any way I could save and restore the whole local image cache of the builder instance to and from a GCS bucket? Can I assume /var/lib/docker or that would be a big mess?

[Philmod]

Hey Albert,

We don't have a fully integrated caching story yet.

There are a few solutions possible to you:

1. Push the intermediate images to GCR, then pull them at the beginning of the build and use --cache-from
2. Push the cache to/from GCS

Here is a documentation about speeding up your builds.

I hope this helped,
Philmod

[egsy]

@acasademont my coworker and I have been working on this and found a solution that works until GCP fully supports caching... it's in a private repo though. @mattrobenolt care to comment?

[mattrobenolt]

We ended up using something like this:

steps:
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args: [
    '-c',
    'docker pull us.gcr.io/$PROJECT_ID/$REPO_NAME:builder || true',
  ]
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args: [
    '-c',
    'docker pull us.gcr.io/$PROJECT_ID/$REPO_NAME:latest || true',
  ]
- name: 'gcr.io/cloud-builders/docker'
  args: [
    'build',
    '--target', 'builder',
    '-t', 'us.gcr.io/$PROJECT_ID/$REPO_NAME:builder',
    '--cache-from', 'us.gcr.io/$PROJECT_ID/$REPO_NAME:builder',
    '.'
  ]
- name: 'gcr.io/cloud-builders/docker'
  args: [
    'build',
    '-t', 'us.gcr.io/$PROJECT_ID/$REPO_NAME:latest',
    '-t', 'us.gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA',
    '--cache-from', 'us.gcr.io/$PROJECT_ID/$REPO_NAME:builder',
    '--cache-from', 'us.gcr.io/$PROJECT_ID/$REPO_NAME:latest',
    '.'
]
images: [
  'us.gcr.io/$PROJECT_ID/$REPO_NAME:builder',
  'us.gcr.io/$PROJECT_ID/$REPO_NAME:latest',
  'us.gcr.io/$PROJECT_ID/$REPO_NAME:$COMMIT_SHA',
]

The idea is that we explicitly pull and build each stage, then push each stage as a tagged image.

So the Dockerfile for this ends up being something like:

FROM foo AS builder
...

FROM bar
...

Notice the --target argument in the first build step.

[acasademont]

Thans @Philmod @egsy and @mattrobenolt! Something similar is what I ended up doing. But still feels a bit fishy. If there was an easy way to save & restore the contents of /var/lib/docker (with all the cached images) these steps wouldn't be necessary. Also, right now the time spent on the different docker pulls is more than 1 minute, not the fastest thing ever...