Cirrus-ci-docs: GitHub checks reported from multiple builds

Created on 21 Oct 2019  路  20Comments  路  Source: cirruslabs/cirrus-ci-docs

Expected Behavior

When multiple builds are executed in sequence (for example, commit build -> PR build), only results from the latest build should be reported to GitHub.

Real Behavior

Dependabot PRs frequently result in mixed-up commit and PR build results in GitHub. Even though the PR build passes, GitHub still reports "Scheduled" for some tasks. The misreported tasks need to be re-executed to get proper results in GitHub.

Related Info

Screenshot 2019-10-21 10 39 43

bug
Source
picture connorbrinton

All 20 comments

Related: #473

RDIL picture RDIL on 21 Oct 2019

@RDIL agreed, that issue is somewhat related. In my case however, I don't mind the fact that Cirrus CI runs multiple pipelines on a branch. Actually, it's kindof nice if I only want to run certain tasks in a PR context, and not a commit context.

My specific issue is that if a new pipeline supersedes and autocancels an older pipeline, I only want to see GitHub status check results from the new pipeline 馃檪

connorbrinton picture connorbrinton on 21 Oct 2019

Yeah I know, just pointing out that these issues are related in that they are both very similar, and as such they can be grouped.

RDIL picture RDIL on 21 Oct 2019
👍1

Hey @connorbrinton, which task you had to re-run to get the checks go green? Don't see any re-runs for https://cirrus-ci.com/build/5765266985713664 馃

fkorotkov picture fkorotkov on 21 Oct 2019

I didn't actually re-run any checks in that particular instance 馃槄 I merged a PR that had green checks and Dependabot automatically rebased the PR with bad checks.

This happens fairly often though, so if there's no other PR to merge, I do have to actually re-run the checks. I'm afraid I don't have an example readily available though 馃槵

connorbrinton picture connorbrinton on 21 Oct 2019

Since it's happening fairy often for you. Could you please the next time post a link of a task that finished but a corresponding check is still yellow? I'll check the logs to see why it happened.

fkorotkov picture fkorotkov on 21 Oct 2019

Yup, I'll update here when it happens next 馃檪

connorbrinton picture connorbrinton on 21 Oct 2019
👍1

Closing for now. Please re-open once you have a task I can investigate.

fkorotkov picture fkorotkov on 22 Oct 2019

Dependabot just did its weekly run, so I have a bunch of tasks for you to investigate now 馃檪

First PR:

Second PR:

connorbrinton picture connorbrinton on 28 Oct 2019

I've deployed an improvement to the locking mechanism for such race conditions. Thank you for the links! Let's see how it'll behave.

fkorotkov picture fkorotkov on 28 Oct 2019
👍1

@fkorotkov locking has worked much better over the past month 馃檪 However I still run into this problem from time to time. Out of 9 Dependabot PRs created today, 1 of them is showing results from the branch build in GitHub:

Both "Build and Push" and "Run Tests" (the only two tasks) are misreported on the PR page.

Have any changes been deployed recently affecting the locking mechanism, or is there anything in the logs that might explain why the wrong task status are reported?

Thanks! 馃槃

connorbrinton picture connorbrinton on 16 Dec 2019

Hey @connorbrinton, I would love to take a look and help out, but since the repo looks private, can you describe what you are seeing more in depth, or provide a screenshot (even if partially censored)?

(this is optional, you don't need to disclose this to me if you don't want to or can't, since I am not an employee at Cirrus Labs)

RDIL picture RDIL on 16 Dec 2019

Yup, can do @RDIL 馃檪 I actually realized that a second Dependabot PR (besides the one I mentioned above) also has misreported status checks. It's a mix of misreported and properly reported status checks, so it's more interesting:

Screenshot 2019-12-16 12 17 56

Clicking on "Details" for each status check and opening the Cirrus CI page, the builds associated with each status check are:

From just these URLs, we can tell that something's wrong, since the status checks should only show the results from a single build.

Opening build 5447007296028672, every single task is cancelled with the message "Auto-canceled because of a newer PR build!".

In build 6666649494618112, all tasks completed successfully and the manually triggered task is ready to be executed. However, because the wrong build results were reported to GitHub, the PR can't be merged 鈽癸笍

connorbrinton picture connorbrinton on 16 Dec 2019

Ah, so this is an issue with returning status checks. I can understand why this could be flaky, since Cirrus may be getting confused with all the different builds and where to send status checks back to.

RDIL picture RDIL on 16 Dec 2019

It seems there was a race condition on GH side. Could you please confirm that on Checks tab you see two separate entries for Cirrus on the left hand side?

fkorotkov picture fkorotkov on 16 Dec 2019

Here's what I have for the first PR:

Screenshot 2019-12-16 13 00 53

And the second PR:

Screenshot 2019-12-16 13 01 48

connorbrinton picture connorbrinton on 16 Dec 2019

Interesting. Seems they've fixed it in the UI. From the logs I see two separate GH Check Runs:

  1. For the canceled task: https://github.com/cirruslabs/cirrus-ci-docs/runs/350613260
  2. For the PR task: https://github.com/cirruslabs/cirrus-ci-docs/runs/350613231

I cannot see the pages since the repo is private but you should be able to see it.

Anyways I know how it can be improved from the Cirrus side and I'm working on a fix.

fkorotkov picture fkorotkov on 16 Dec 2019

Yeah, it seems GH hides everything it doesn't think is the latest run. Here's what those links look like: 馃檪

Canceled task:

Screenshot 2019-12-16 13 16 24

PR task:

Screenshot 2019-12-16 13 17 12

Thanks for all of your hard work on Cirrus! 馃槃

connorbrinton picture connorbrinton on 16 Dec 2019

Got the change ready and will deploy first thing tomorrow morning. 馃檶

fkorotkov picture fkorotkov on 16 Dec 2019

Sounds great, thanks @fkorotkov! 馃槃

connorbrinton picture connorbrinton on 16 Dec 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

aslushnikov picture aslushnikov  路  5Comments
EpicEric picture EpicEric  路  3Comments
dotdoom picture dotdoom  路  4Comments
RDIL picture RDIL  路  4Comments
januswel picture januswel  路  4Comments