Cloud-builders: Ignore failed step?

[acasademont]

Hi all,

The first step of my build is a docker pull of an image to be used in the --cache-from of the docker build step. If the docker pull fails because the image is not present the whole build will fail. That first step is really optional, the docker build will just run as normal without the cached image.

Is there any way to make the builder ignore a failed step which is not critical? In bash I would just write docker pull || true but I can't seem to make it work here. Thanks!

[Philmod]

Hey Albert,

We don't have an integrated solution to skip a failed step, but you can always use bash:

- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    docker pull gcr.io/$PROJECT_ID/my-image || exit 0

[acasademont]

That's a nice workaround indeed, thanks @Philmod

I'm wondering if this could be something you would consider as a feature request maybe? That is, the possibility of defining an allow_failure: true option on a build step.

- name: 'gcr.io/cloud-builders/docker'
  args: ['pull', 'gcr.io/$PROJECT_ID/my-image']
  allow_failure: true

I believe this should be pretty easy to implement

[bendory]

Not only would we consider it, it is currently on our roadmap for prioritization.

[acasademont]

awesome thanks!

[MrBlaise]

@bendory Is there any news on this feature?

[bendory]

no updates; still on the list, but given the ease of work-around, it hasn't yet bubbled up as a priority

[emarx]

@bendory this workaround doesn't allow one to see how the build went if you're using cloud builder for CI. It would be great to see that the build failed without stopping all the other steps. Is the plan when skipping failed steps to show the build as failed? Do you have any recommendation for running tests on many packages and seeing which packages failed without short circuiting?

[bendory]

Indeed, this is a "workaround" but not a solution. But I think @Philmod's suggestion above meets your case, does it not? The failed step will be ignored and the build will proceed -- whether the step in question is a test or some other functionality. What am I not understanding?

[emarx]

It would be nice to see the step as failed, even if the build "succeeded" in the sense that it kept running. Using the workaround will display the build step as successful, when it is in fact failed, but we decided to proceed with the other build steps in the file.

[heygambo]

Unfortunately the hack doesn't work for me.

I'm trying to cache node_modules as described here: https://cloud.google.com/cloud-build/docs/speeding-up-builds#caching_directories_with_google_cloud_storage

The first time this build script is running there is nothing on the google storage. I'm wondering if there is a way how to ignore that? I've tried to add || exit 0 as you've described but google cloud builder still exits with an error.

This is how I've build it together:

- name: gcr.io/cloud-builders/gsutil
  args: ['cp', 'gs://${PROJECT_ID}_cloudbuild/web_app_node_modules_cache.tar.gz', 'node_modules.tar.gz', '||', 'exit', '0']

This is how the build step fails.

Am I not using exit 0 correctly?

[bendory]

Your build step is not running in a shell, and gsutil doesn't understand what to do with || exit 0. This should work for you:

- name: 'gcr.io/cloud-builders/gsutil
   entrypoint: '/bin/bash'
   args: [ '-c', 'gsutil cp gs:/.../....tar.gz ... || true ]

[heygambo]

I've tried something similar also. Nothing worked.

I'm getting around this now by creating an empty node_modules cache folder, taring it and uploading it if it doesn't exist.

gsutil -n only copies the file over if it doesn't exist:

No-clobber. When specified, existing files or objects at the destination will not be overwritten. Any items that are skipped by this option will be reported as being skipped. This option will perform an additional GET request to check if an item exists before attempting to upload the data. This will save retransmitting data, but the additional HTTP requests may make small object transfers slower and more expensive.

- name: alpine:latest
  args:
  - mkdir
  - -p
  - tmp
- name: alpine:latest
  args:
  - tar
  - cfz
  - tmp_node_modules.tar.gz
  - tmp
- name: gcr.io/cloud-builders/gsutil
  args:
  - cp
  - -n
  - tmp_node_modules.tar.gz
  - gs://${PROJECT_ID}_cloudbuild/web_app_node_modules_cache.tar.gz
- name: gcr.io/cloud-builders/gsutil
  args:
  - cp
  - gs://${PROJECT_ID}_cloudbuild/web_app_node_modules_cache.tar.gz
  - node_modules.tar.gz
- name: alpine:latest
  args:
  - tar
  - xfz
  - node_modules.tar.gz
- name: gcr.io/cloud-builders/yarn
  args:
  - install
  - --non-interactive
- name: alpine:latest
  args:
  - tar
  - cfz
  - node_modules.tar.gz
  - node_modules
- name: gcr.io/cloud-builders/gsutil
  args:
  - cp
  - node_modules.tar.gz
  - gs://${PROJECT_ID}_cloudbuild/web_app_node_modules_cache.tar.gz
- name: alpine:latest
  args:
  - rm
  - -f
  - node_modules.tar.gz

[jeremyjh]

@bendory I know the "entrypoint" key is documented on this page, but I think it would be helpful if this workaround were explicitly mentioned near the top of https://cloud.google.com/cloud-build/docs/create-custom-build-steps, before it talks about building custom images. This should be the first thing most people turn to when they want to execute a bit of bash, and I discovered it here in this issue (though, on retrospect, it is obvious).

[acasademont]

Maybe this issue should remain open until the feature is implemented?

[jannaspam]

Hey. I would like to add an use case for this feature. We're using caching feature from Docker. To do that, we need to pull an image first. But it doesn't exist for new images. So simply speaking, it's a "chicken & egg" problem. Example below:

steps:
  - name: gcr.io/cloud-builders/docker
    id: pull_image_A_cache
    args:
      - pull
      - gcr.io/${PROJECT_ID}/image_A:latest

  - name: gcr.io/cloud-builders/docker
    id: build_image_A
    args:
      - build
      - --file=Dockerfile
      - --tag=gcr.io/${PROJECT_ID}/image_A:latest
      - --cache-from
      - gcr.io/${PROJECT_ID}/image_A:latest
      - .

If we could allow the first step to fail, that would be cool :)

[NullPrice]

I've been working around this with the above suggestions. We have multiple different test steps (unit, intergration etc) that we run in parallel and if we don't use the workaround of returning a exit code of 0 the pipeline simply exits without waiting for the rest of the parallel steps to return an exit code themselves.

An unfortunate side affect of the workaround is that I have to have a step that 'checks' the test result, and it becomes verbose and hard for people to work out what has actually failed.

Really wish there were some configuration options around this as others have mentioned.

[gijswobben]

Any progress / updates on this matter? The workaround isn't really an option for my use case..

[0x962]

Would love to have this feature built in. Unfortunately, don't see a lot of activity happening here...

[tfsparks]

I will add my perspective on the utility of allowing a failed step.

Here is a practical real-world example:

• Build infrastructure
• Run infrastructure tests
• Tear down infrastructure

If I fail the test step I won't remove the infrastructure I created in the first step and will continue being billed. I need to have a step that tears the deployed resources down regardless of the result of the prior steps. Without allowing the tests to fail as a warning, and drawing attention to that step in the process, I end up in a lose-lose scenario. I am either hiding an important piece of information to get the utility I need, or I am being billed for things that are broken. I am sure this extends to applications as well and not just the infrastructure example.

I propose a config setting allow_failure at the step level to treat a failure as a warning and continue the pipeline which should also change the visual language of the resulting step in the logs. Ideally we could also have a way to force a step to always run.

Gitlab-CI has an example of this capability in multiple forms:
Allow a step to fail and treat it as a warning:
https://docs.gitlab.com/ee/ci/yaml/README.html#allow_failure
Always run a step regardless of prior steps (treats a failure as a failure instead of a warning, but allows cleanup type operations):
https://docs.gitlab.com/ee/ci/yaml/README.html#when

[bbhoss]

Is this tracked in Google's issuetracker somewhere?

[severi]

This would indeed be a nice feature. Also what I'd love to see implemented is a post build action (like https://jenkins.io/doc/book/pipeline/syntax/#post ) which would allow to run some steps after the build has finished/failed (e.g. uploading the test reports to object storage so the failure could be better analyzed).

[tfsparks]

@bbhoss I just saw your question and created one here:
https://issuetracker.google.com/issues/128353446

Feel free to star it as it doesn't appear Github issues are getting much love. Re-reading the contribution info it looks like issues on Github are supposed to be specifically about the builder containers, not the Cloud Build tool itself.

[niedbalski]

This should be handled as a feature request.

[selslack]

@bendory is it still on your roadmap? Any updates on the story?

[ReallyLiri]

I found a simple (but a bit violent) workaround - build cancel itself if some condition matches.

  - name: 'gcr.io/cloud-builders/gcloud'
    id: 'Cancel current build if on master'
    entrypoint: 'sh'
    args:
      - '-c'
      - |
        test $BRANCH_NAME = "master" && gcloud builds cancel $BUILD_ID > /dev/null || true

Note the service account that runs the builds ([email protected]) should have appropriate permissions to cancel build.

[trialblaze-bt]

This issue/request is 2+ years in the pipeline. Is there any plan to have this common feature properly integrated with GCB? Thanks

[bendory]

This Issue tracker is for bugs and issues regarding the cloud-builders build steps.

To report an issue with the hosted Google Cloud Build service or to request a feature in the hosted service, please report it to your Google Cloud Support team or use the public issue tracker at
https://issuetracker.google.com/issues/new?component=190802&template=1162743.