Toolkit: Please support something like "allow-failure" for a given job

[mvo5]

We use github actions in our "snapd" project and we love them.

One small feature we would love to see is a way to mark a test job as "allow-failure" (or a term along these lines) [0]. This would simply mean that the overall /pulls overview page would show the PR as with the little green tick-mark (and maybe in the tooltip 5/6 OK, 1 ignored). It would still show as a failure in the details view (maybe with a different icon?).

Our use-case is that we have some CI environments that fail frequently because of external factors like repository mirrors that are our of sync etc. We still want to run the CI on these systems but not get distracted too much by these out-of-our-control issues.

Hope this makes sense.

Thanks!
Michael

[0] E.g.

jobs:
  spread:
    runs-on: ubuntu-latest
    allow-failure: true
    steps:
    - do-some-flaky-stuff

[thboop]

Hey @mvo5 Thanks for the feature request!

We do support marking a step to allow failure via continue-on-error.

We also support marking specific checks as required
That will allow you to merge pr's without a particular job succeeding.

I think the latter should solve your issue at the job level, is there a reason that doesn't well for you?

[mvo5]

Hey @thboop. Thanks for your quick reply!

Yeah, it's really just about the little green tick at the pull-request overview page. AFAICT when one job (even if it's not required) fails the overview PR list will show this PR as failed. Having a way to mark certain jobs as not rquired would still show the pulls as green (or yellow?) instead of red.

But I do understand this is a bit of a specific request, so feel free to close it if you think it's a bit too odd. We had it with our old CI system and I liked it.

[fingolfin]

I don't think it is that specific. People have requested this before: https://github.community/t5/GitHub-Actions/continue-on-error-allow-failure-UI-indication/td-p/37033

I just came here via Google as I was surprised I couldn't find anything like this in the documentation. It's standard with e.g. Travis CI

[jhenstridge]

I guess one way to handle this would be for jobs with an allow-failure flag set to generate a neutral conclusion on failure instead of failure. As I understand it, this would let the overall check suite conclusion be success if such a job fails.

[jadephilipoom]

I'm also an Actions user who would love to see this feature. My team's project has a long build and a few jobs that are flaky and frequently take a while to fix. We do just ignore failures on those for merging PRs, but being able to make the green tick agree with that convention would be really nice.

[jakesyl]

This would be great

[ghostsquad]

in the interim, what if there was a step that simply posted the "allowed failures" in a comment (that same comment updated each time the workflow ran). Similar CodeCov.

according to the docs, this is a detectable condition:

https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#steps-context

steps..outcome | string | The result of a completed step before continue-on-error is applied. Possible values are success, failure, cancelled, or skipped. When a continue-on-error step fails, the outcome is failure, but the final conclusion is success.
-- | -- | --
steps..conclusion | string | The result of a completed step after continue-on-error is applied. Possible values are success, failure, cancelled, or skipped. When a continue-on-error step fails, the outcome is failure, but the final conclusion is success.

[henryiii]

I thought that jobs.<job_id>.continue-on-error sounded exactly like this feature, but it doesn't seem to work like that. For me, at least, setting a job to "continue-on-error" still causes a red x for the workflow; it doesn't seem to have any effect at all.

[dopplershift]

By default, when any job within a workflow fails, all the other jobs are cancelled (unless 'fail-fast' is False).continue-on-error specifies that a certain job failing should not trigger cancelling other jobs. (I would definitely love some way to keep a job from signalling a failing check.)

[ramsey]

Not only does continue-on-error not work as expected (as mentioned by everyone on this thread), but the example in the documentation appears to result in a syntax error that I cannot figure out how to get around. As a result, I've been trying all sorts of permutations to figure out if this syntax error is causing continue-on-error not to work as expected.

Maybe it is?

Here's the documentation example verbatim:

runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental }}
strategy:
  fail-fast: false
  matrix:
    node: [11, 12]
    os: [macos-latest, ubuntu-18.04]
    experimental: [false]
    include:
      - node: 13
        os: ubuntu-18.04
        experimental: true

My version looks similar:

unit-tests:
  name: "Unit Tests"

  runs-on: ${{ matrix.operating-system }}
  continue-on-error: ${{ matrix.experimental }}

  strategy:
    fail-fast: false
    matrix:
      dependencies:
        - "lowest"
        - "highest"
      php-version:
        - "7.4"
        - "8.0"
      operating-system:
        - "ubuntu-latest"
      experimental: [false]
      include:
        - php: "8.0"
          composer-options: "--ignore-platform-reqs"
          experimental: true

When this runs, it fails at the unit-tests job, due to the following error. (See the failure here.)

Error when evaluating 'runs-on' for job 'unit-tests'. (Line: 121, Col: 14): Unexpected value ''

Somehow, it's getting an empty string for runs-on, or at least, that's what this error seems to indicate to me.

When I remove the line that defines experimental: [false] (and only this line), the build runs successfully (see here).

So, maybe the successful build is not actually taking into account the continue-on-error property? Does this syntax error help point to some kind of culprit that could resolve this issue for everyone on this thread?

[nschonni]

@ramsey your issue doesn't seem related to the thread. Your job is failing because your include doesn't have a value for operating-system

[ramsey]

Why does my include work without operating-system when I remove experimental: [false] from the matrix?

[ramsey]

@nschonni You were right. Turns out, I had my matrix messed up. Sorry for the churn here, y'all. 😀

[szepeviktor]

This is one solution

      - name: Install dependencies
        id: composer-run
        continue-on-error: true
        run: composer update --${{ matrix.dependency-version }} --prefer-dist --no-interaction --no-suggest

      - name: Execute tests
        if: steps.composer-run.outcome == 'success' && steps.composer-run.conclusion == 'success'
        run: vendor/bin/phpunit

continue-on-error causes difference in outcome and conclusion!

[ljharb]

@thboop any update? "continue on error" isn't "allow failure", unfortunately - as has been explained upthread, "allow failure" means, mark the overarching workflow as successful even if this job or step fails.

[stevepiercy]

Another use case is to be able to test future versions of Python to ensure compatibility for our project. This gives us a heads up of what to expect and deal with issues as they come up, and keeps a history.

[MSP-Greg]

I recently revisited this when I noticed the jobs.<job_id>.continue-on-error option in the docs. Couldn't recall if I'd tried it before. I believe at some point in time, it may have not been available? Anyway, it is helpful, but still lacking. Forgive me, but below I've described why I think a proper 'allow-failure' is important.

There is one 'top level' pass/fail indicator that is important, and that is the indicator associated with the push/pull-request event that triggered the CI run. For a maintainer/owner reviewing PR's, this indicates whether they need to look at the CI logs. For a contributor, who may not understand all the Action workflow syntax, it shows whether their PR passed CI. Currently, the option does not affect this indicator.

It does affect the workflow's indicator in the list of workflow runs shown on the repo's 'Actions' page/tab. It also can affect badges, since they can be tied to a specific workflow file.

So, the above option does not affect the most important indicator. It seems to be based on jobs, and ignores the containership of jobs within the workflow file, which is what the option affects.

Finally, it's been mentioned in an above comment that 'allow-failure' functionality is desired for testing against 'master/main' branches of major dependencies, which may include the language used. This can be an important feedback loop, and the current functionality of jobs.<job_id>.continue-on-error works well for that, as clicking the 'top level' pass/fail indicator in the web UI will show what jobs have failed, even if they are set to 'allow-failure' by using jobs.<job_id>.continue-on-error.

[geerlingguy]

Just chiming in with another +1 to this; I have a number of workflows where I test on stable + nightly, or stable + latest, and I want to test latest/nightly, but don't want the job to fail _and still be marked as a pass in the UI_ if that test fails.

I still want to be able to get into the test results to see what fails, but it would be nice if the UI would show a grey dot, or an exclamation point, or anything besides a green check for the failed Job.

As someone who is migrating a number of open source projects over from Travis CI, this is probably the number one painful omission.

[MSP-Greg]

@geerlingguy

Maybe we've got a terminology issue, but in Travis, a job that's marked as 'allow-failure' is shown as failed. Conversely, the 'top level' indicator is shown as passed.

That's part of the reason for my post, as I wanted to describe exactly what I thought was needed. Also, Actions has another level of containership in part of the UI, the workflow file. In Travis and AppVeyor, that didn't exist.

And, in Actions, that containership is only shown in the repo 'Actions` page, it doesn't exist when looking at commits or pull requests that show a 'top level' indicator...

EDIT: Rephrasing, I would like the correct status of all jobs to be reported anywhere a job list exists in the UI. Any pass/fail indicators summarizing that list, in whatever way, should be aware of the 'allow-failure' settings of jobs, and ignore their result. Hence, those indicators will show pass/fail based only on jobs not marked as 'allow-failure'...

[MSP-Greg]

Well, we all know how helpful examples are...

I created a commit in a fork that has two workflow files, one of which I added jobs.<job_id>.continue-on-error to a single job.

Lets look at the 'top level' indicators, see the commit 'Actions - allow failure - non_mri.yml':

https://github.com/MSP-Greg/puma/tree/allow-failure
https://github.com/MSP-Greg/puma/commits/allow-failure

Both 'top level' indicators show failed, which is what's wrong with the current implementation.

When either are clicked, they show one job failing, which is what I'd prefer. If one has prior knowledge of what jobs run 'allow-failure', it's very easy to check.

Now, let's look at the 'Actions' page run list.

https://github.com/MSP-Greg/puma/actions?query=branch%3Aallow-failure

The two workflow indicators ('MRI' & 'non_MRI') for the 'Actions - allow failure - non_mri.yml' commit both show passed. This is inconsistent with the indicator mentioned above for the commit.

[haltman-at]

Just one more person trying to migrate from Travis, who thought I could use continue-on-error to make certain jobs allowed failures, only to find that continue-on-error didn't work like I thought... please add this! As others have mentioned, if an allowed failure fails, it means that that job should get a red X, but the overall workflow should not.

[OsamaSayegh]

I'd like to +1 this. At Discourse, we're moving all our repos to Github Actions, and one of our gems (rails_multisite) is tested against multiple Rails versions and Rails master. Tests against Rails master are informative and are allowed to fail. So it'd be really nice if they didn't cause the overall workflow to display a scary ❌ if they failed.

[szepeviktor]

The problem is that we - CI users - have to justify why allow-failure is a vital option.

[matteobachetti]

+1: at Stingraysoftware and DISCOS we would really love this feature.

[haltman-at]

The problem is that we - CI users - have to justify why allow-failure is a vital option.

@szepeviktor Do you think the above arguments don't sufficiently cover that? (@thboop?)

[szepeviktor]

Do you think the above arguments don't sufficiently cover that?

The problem is that GitHub needs our arguments at all!

All mature CI-s - e.g. Travis CI - have this feature.

[henryiii]

All mature CI-s

Of which GHA is _by far_ the youngest. It's also build on a simple, composable design instead of hard-coding things like workflows for a specific tool that may change (looking at you, Travis CI!).

I think we are getting one thing at a time. allow-failure, as well as single/failing job restarting, skip-ci, and probably other things _may_ come in the future. I don't don't think eloquent arguments are needed, just time to prioritize, design, test, and deploy. We have already been getting some great features, like manual workflow triggering.

[obfusk]

I've been using continue-on-error and it mostly provides the functionality that I've used travis' allow_failures for.

But there are some UI inconsistencies: the /:user/:repo/actions page gives a nice :heavy_check_mark: and the badge correctly says "passing", but the /:user/:repo/commits/:branch page shows :x:.

[Borda]

I've been using continue-on-error and it mostly provides the functionality that I've used travis' allow_failures for.

Can you please share example?
So far I know continue-on-error is set per step, but allow_failures is set per confirmation - e.g. all windows builds from matrix build are allowed to fail...

[obfusk]

Can you please share example?

jobs:
  build:
    continue-on-error: ${{ matrix.warnings == 'error' }}
    strategy:
      matrix:
        # ...
        warnings: [default, error]
    - name: Test
      run: |
        if [ "$WARNINGS" = error ]; then
          # do something that is allowed to fail, like compiling with -Wall -Werror
        else
          # ...
        fi
      env:
        WARNINGS: ${{ matrix.warnings }}
# ...

[eregon]

I tried continue-on-error recently,

• if used at the step level, then everything appears green and the only way to notice the failure would be to manually open the step. There is also an annotation Check failure on line 1 in .github but that doesn't link to the proper place.
  That's not enough notification, and there seem to be no reasonable way to see if that job actually passed or failed.
  Example: https://github.com/eregon/sequel/runs/1476292451?check_suite_focus=true#step:7:27
  I think it would be far more useful if continue-on-error at the step level would behave more like at the job level, and mark the CI job as failed or failed-but-expected.
  The failed step itself should have a red or yellow mark, so it can be noticed without having to guess which step failed.
• if used at the job level, then the job is marked as failed, which at least can be noticed. And the whole workflow is still marked as successful when looking in the Actions tab. But on the PR it's rather confusing, and it looks like the CI broke when it did not.
  Example: https://github.com/jeremyevans/sequel/pull/1737/checks?check_run_id=1479678900
  
  The big red Some checks were not successful sounds like something broke, but no, an error/failure there is expected so it shouldn't be marked as "CI failed". And on the wheel there is a bit of red.
  The whole area is encircled in yellow (I'm not sure why), which seems OK.
  So there I would propose to mark the failed job in yellow, and possibly use another symbol. And then to use the same color in the wheel for jobs which are continue-on-error: true. And also change the text so it would be something like All expected checks have passed in green (the wording for all checks are green is All checks have passed).
  The whole area should then be encircled in green, because it's safe to merge without looking at logs.

[rwwagner90]

We definitely need a way to ignore failures, rather than using continue-on-error and making everything look green. We should be able to visually see the errors, but not have it fail the build.

[ktomk]

not allowing failed builds fails to integrate with next versions which are normally allowed to fail but constantly checked. was easy on travis-ci, is not possible with github actions. the docs about migrating from travis-ci do not cover this as well. just my 2 cents.

[mjftw]

Adding another 👍🏼 to this.

Did this get resolved?

This would solve the problem we're currently having on our Python project where we're slowly migrating our code to use static type checking using pyright. We want static typing violations to be flagged in the PR, but we don't want a big scary X on the PRs just yet.

A softer form of failure, like an expected failure as mentioned above would be much nicer.

[fbidu]

Hey all, just passing by to say that this is the second or third time that I found this issue while searching for an 'allow-failure-like option'. I would really love to have this around!

[sayakbiswas]

Would love to have this added. Currently I have a test matrix and a dependent job that should execute when a specific configuration from the matrix is successful even if the others fail. continue-on-error makes it so that the failed configurations show up with a green check mark

[1313e]

Something like this would be extremely useful for testing packages that rely on other packages to be updated first.
For example, if one were to check if their Python package is Python 3.10 compatible (and all of its dependencies as well), one could test their package for Python 3.10 and mark it as allowed-to-fail until it finally stops failing.
This way, every time the test suite is ran, it basically checks if all dependencies are Python 3.10 compatible, but it won't fail the job if that has not happened yet.

[lamont-granquist]

Travis CI has this as their allow_failures feature:

https://docs.travis-ci.com/user/customizing-the-build/#jobs-that-are-allowed-to-fail

This can be useful to do testing on unreleased versions of upstream products (beta-testing against their unreleased code) so you can get an idea of when stuff starts failing upstream.

We've used this for many years to test code against ruby-head. None of these tests are required for "production" development and deployment but it can be highly useful to have the canary running all the time. The results should be allowed to go red, but be completely ignored for the overall result of the tests.

The documentation on GH actions continue-on-error method even looks like it trying to do exactly this, but the result is very much not what we want. It almost looks like product and/or development tried to copy the feature but misunderstood what its use case was or just misunderstood the semantics.

[glensc]

Here's my simple version of the experimental: true behaving like Travis allow-failure: true behavior. If experimental is true, the failing job status is ignored, marked as "passed".

• https://github.com/zf1s/zf1/pull/59/files

...
    strategy:
      matrix:
        experimental:
          - false
        php:
          - "5.3"
          - "5.4"
...
        include:
          - php: "8.0"
            experimental: true
...
    steps:
...

      - name: "Run PHPUnit tests (Experimental: ${{ matrix.experimental }})"
        env:
          FAILURE_ACTION: "${{ matrix.experimental == true }}"
        run: vendor/bin/phpunit --verbose || $FAILURE_ACTION

[lilydjwg]

@glensc No. The job status should be "failed" if it has errors, but the status for the whole workflow should ignore the status of the said job. I do want to know if my project works with latest development dependencies or not, especially when the new upstream version is about to release. Manually reading the logs works but takes more effort than Travis CI's.

[glensc]

@lilydjwg well. apparently what you (we) want is not possible at this point. my goal was to run CI for the experimental platform, but the failure of that job may not result in the whole pipeline failure. as that would only confuse contributors to the project.

[ljharb]

@glensc right, the point is that the experimental platform when failing should be marked as failed, but the overarching pipeline should be marked as successful. The last part is what Github Actions' "continue on error" fails to do (travis-ci's allow_failure does both)

[ktomk]

The run .travis.yml action has an example in turning errors into warnings in case the build failed but allow-failure is on (this is ported from Travis-CI). the way it work on Github actions just has a different model and this needs an entirely new setting and class of what constitutes are failure in a build. Travis-CI for example allows to configure the allowed failures centrally, on Github build job expansion works differently.

[eregon]

FWIW I filed a GitHub Support issue about this, if you also want this to progress it might be a good idea to tell them as well.

Thank you for contacting GitHub Support and providing detailed feedback.
I've documented and passed on your specific request to the relevant team as such, your feedback is in the right hands.
While I don't have a timeline to share for possible changes, we really appreciate you writing in with this detailed feedback helping us make GitHub work better for you!
Please let us know if you need any further help and we will be happy to assist.

Concrete suggestions how to fix this: https://github.com/actions/toolkit/issues/399#issuecomment-738700569

[geerlingguy]

For about the 20th time in the past year, as I continue migrating my Travis CI jobs over to GitHub Actions, I've been bitten by this weird behavior again. I really wish there was something like Travis' 'allow-failure', where it could still show up as failed in the UI, but would not fail the entire build.

[driesvints]

I recently implemented nightly builds for PHP 8.1 in Laravel and used continue-on-error like @thboop suggested. It works pretty well when you combine it with a matrix build. I think the only thing that's missing is indeed something like a different color indication for builds that continued on error. Something like a yellow status and a different icon.

[ktomk]

I think the only thing that's missing is indeed something like a different color indication for builds that continued on error. Something like a yellow status and a different icon.

Maybe just this as a feature on it's own may already solve most of this issue. Good thinking.

[eregon]

The main problem for continue-on-error at job level is a PR will be shown as having "failed builds" even though the builds that failed are all continue-on-error jobs: https://github.com/actions/toolkit/issues/399#issuecomment-738700569

For continue-on-error on a step there is 0 indication the step or the job failed, which makes it a sure way to never see any issue...

[ktomk]

the builds you allow to fail you just turn into green (0) while painting it yellow or red. done.

[tomato42]

@ktomk the problem is that you both want to have it reported (so you know when it stops failing, and you can promote that new version to "fully supported" status) and have no impact on the PR CI passing.

[zikaeroh]

Given continue-on-error is step-scoped, and there can be many steps that can fail in a job (e.g, if I'm targeting a new compiler, it could crash at compile time, test time, in benchmarks, maybe even building the compiler itself), I think it's a poor replacement for allow-failure (which is job-scoped on Travis).

As a hack if you have a simple check, maybe it's reasonable, but I feel this issue goes further than just adding a visual indicator for "neutral" builds or allowing some failed builds to not completely fail the whole workflow. I don't really want to have to come up with some complicated logic to track the status of prior failures to try and make sure I get that neutral status passed through even though the problem was a few steps ago.

[ktomk]

@ktomk the problem is that you both want to have it reported (so you know when it stops failing, and you can promote that new version to "fully supported" status) _and_ have no impact on the PR CI passing.

No, the problem is that those who like a different service better don't want to pay it and jump over to GHA and then complain that it's not the same.

Take ktomk/run-travis-yml as an example: It does highlight the allow-failure jobs as errors/warnings (and with the reason) and let the CI pass green as those failures ain't any. It's not that it wouldn't be possible to achieve the same (CI passing even while build job failures), you still have to check for errors thought. Add some "report to the PR" or send out an Email, this all looks possible to me.

However if the decoration would be available as an option, then it would be more integrated. That's all I wanted express with my earlier comment. With the best intentions.

[ktomk]

As a hack if you have a simple check, maybe it's reasonable, but I feel this issue goes further than just adding a visual indicator for "neutral" builds or allowing some failed builds to not completely fail the whole workflow.

Yes it goes further, however if you implement to allow a failure on a build job - which you can do on GHA already today - , what you would still miss is the visual indicator.

continue-on-error is indeed a different concept. It can help when you come from the Travis-CI allow-failure mindset, but it does not match well, often not at all.

[ktomk]

For continue-on-error on a step there is 0 indication the step or the job failed, which makes it a sure way to never see any issue...

Well, that's the command: continue-on-error. You set it if you want to get the error out of the way. It does not unbind you from reading the logs... . It is not something similar to _allow-failure_ as known from Travis-CI where you don't need to read the logs and get the message in the summary. But the concept of actions on GHA and the build matrix on TCI is entirely different.

[eregon]

It is in large parts an issue of presenting continue-on-error job and steps.
No indication at all for a continue-on-error step means one cannot even find out if a step failed without expanding each step in the log of every build and analyze the output, which of course nobody does or has time for. A green job should mean "every step succeeded".
So the step-level continue-on-error seems no better than doing || true which is of course terrible for logging and finding issues.

The job-level continue-on-error has kind of the opposite issue where the PR looks like it breaks CI (shown as red with failing builds) even though the only job failing is marked as continue-on-error in the workflow and so it is an expected failure.

I think my comment https://github.com/actions/toolkit/issues/399#issuecomment-738700569 captures clearly what needs to happen here to make this feature actually usable beyond just || true.

Now the question is how do we get the GitHub Actions dev team to look at this?
cc @chrispat @thboop

(probably this should be an actions/runner issue but that is details)

[ktomk]

@eregon: please see steps context and Setting an Error Message, these are visible in the summary, no need to check each log.

[MSP-Greg]

@ktomk

I'm a maintainer of a repo using a common technique for showing a failed job in the summary, but is still not as friendly as seeing it in the job list shown in a PR or the repo's Actions tab.

So, given multiple issues in multiple repos, and also probably a few in the Support Community forum, this is a feature many people have asked for. Or, demand for it shows that people aren't interested in the available workarounds...

[ktomk]

@MSP-Greg: Yes, exactly that, wrote about it in my previous comments. I find it weird to name these workarounds, this is just how it works. But no need to argue with me, that last question remains how this can get some traction on the Github side. Can somebody please call the technical support?

[ioquatix]

This PR should be passing, because the failing jobs are all marked as continue-on-errror: true. It's super confusing. Can we please get a fix for this?

cc @bryanmacfarlane @chrispat @ericsciple

[henryiii]

It's continue-on-error, not pass-on-error or similar. I admit the latter would be very useful, and is what this issue is asking for, but claiming it's a bug that continue-on-error doesn't do exactly what it says it does (continue until the job is complete instead of instantly stopping it when an error is encountered) isn't helpful. It does not change the fail to a pass, it just continues instead of instantly stopping.

[ljharb]

@henryiii given that every CI system but actions seems to have “allow failure” but does not have “continue on error”, it’s at least a bug in the minds of those who shipped the latter in response to requests for the former.