Freecodecamp: Restart CI for failed jobs

Created on 20 Oct 2018 · 19Comments · Source: freeCodeCamp/freeCodeCamp

I am helping out with the QA process and applied a filter using status:failure and see over 1500 open PRs with red Xs indicating something failed. The following types of errors are showing up in this list.

Unable to find './azure-pipelines.yml'

Travis CI errors involving folder names with upper case letters.
Example:

Warning: TODO - Define Testing.
Error:
Upper case characters found in 3D, all folder names must be lower case
Found in :
/home/travis/build/freeCodeCamp/freeCodeCamp/guide/english/3D

I know the one Travis CI error involving the folder issue can be resolved by manually restarting the build, because @Bouncey fixed the offending folders, but the sheer volume of these would be a pain to do manually. Is there an automated solution to rebuild these?

DevOps

Source

RandellDawson

👍2

Most helpful comment

@QuincyLarson I believe this issue has been resolved by this PR. I will let @raisedadead make the final determination.

RandellDawson on 28 Dec 2018

🎉2

All 19 comments

Here is another error I am seeing:

RandellDawson on 21 Oct 2018

@Bouncey @raisedadead Do you have any ideas for how we could do an automatic rebuild of these failing PRs? I think it would be worth the time to figure out such a way since this has come up a lot in the past and manually triggering these builds isn't a great use of our contributors' scarce time.

QuincyLarson on 21 Oct 2018

@Bouncey , @raisedadead I found travis-ci/travis.rb#344 on how to restart the build process for PRs which failed. This would make it so that no one has to manually restart the 1500+ PRs which have failed in the last 2 days due to the uppercase folder issues @Bouncey fixed earlier today. Not sure if this particular solution will work, but it gives me hope there is a way to automate these. Another option uses Python.

RandellDawson on 21 Oct 2018

I manually restarted about 50 a few hours ago and about 20-30 this morning. I finally decided to give up in hopes of an automated solution.

RandellDawson on 21 Oct 2018

Sorry, I contributed to one of them by not having the file named index.md, which I think I understood after reading the log for Travis CI a second time. I was responding to the Hactoberfest mailing. I was perhaps being to hasty.

leebut on 21 Oct 2018

I have left a cron job running on a VM. This might take a long long time.

raisedadead on 21 Oct 2018

👍1

The jobs are restarting. I hope in a few hours everything will be queued.

raisedadead on 21 Oct 2018

The restart of most buils is complete. 575 of them did not go thru. I'll see what I can do.

raisedadead on 21 Oct 2018

@raisedadead What criteria was used to choose which of the 1500+ to rebuild? I ask, because I am still seeing several which original failed for having uppercase names (i.e. the 3D and Tableau folders) which was fixed by @Bouncey. It does not appear the builds for these PRs were restarted.

Here are just a few examples of the many I have found so far:

https://github.com/freeCodeCamp/freeCodeCamp/pull/23617
https://github.com/freeCodeCamp/freeCodeCamp/pull/23618
https://github.com/freeCodeCamp/freeCodeCamp/pull/23622
https://github.com/freeCodeCamp/freeCodeCamp/pull/23764
https://github.com/freeCodeCamp/freeCodeCamp/pull/23767
https://github.com/freeCodeCamp/freeCodeCamp/pull/23769
https://github.com/freeCodeCamp/freeCodeCamp/pull/23771
https://github.com/freeCodeCamp/freeCodeCamp/pull/23772

Maybe another script could restart the build for these types?

UPDATE: It took me about 7 hours, but I every single one of the 575 open PRs with a failure status and manually restarted about 250 builds, identified roughly 280 PRs for @raisedadead to write a script to automatically rebuild do to the problems I reported in freeCodeCamp/freeCodeCamp#24688. The rest were various folder and file name fixes and several invalid. I also labeled about 30 merge conflicts, which I hope to learn how to resolve tomorrow.

The following 2 PRs had a similar error:

This job was cancelled because the "Auto Cancellation" feature is currently enabled,
and a more recent build (#XXXXX) for pull request #XXXXX came in while this job was waiting to be processed.

https://github.com/freeCodeCamp/freeCodeCamp/pull/26002
https://github.com/freeCodeCamp/freeCodeCamp/pull/25740

The following PRs had the error below:

Error: One or more locations are missing, all are required.
api: undefined
home: undefined
forum: undefined
forumProxy: undefined

RandellDawson on 22 Oct 2018

🎉1

Hi @RandellDawson

Firstly thanks a ton for spending so much time and efforts on the manual audit. 👏!

I'll try to clarify as much as I can for the benefit of all the @freeCodeCamp/moderators

...seeing several which original failed for having uppercase names...

These indeed need restarts or fixes as you already figured out. We have a test now, that will fail these for lack of correct naming of the files. This is a VALID failure. Lets discuss on the automation in our meeting later today.

...cancelled because the "Auto Cancellation" feature is currently enabled,...

I have enabled auto cancellation for PRs and branches to avoid the strain on the CI.

Two scenarios will cancel the builds on PR:

If a PR is closed without merge.
Fresh commits were added to the PR,

The previous build if pending to run will get cancelled. If additional commits are added the last job in the queue is run, which will correspond to the latest commit on the PR. This reduces the no of jobs on the CI.

A rare side-effect of this is: If a PR is accidentally closed and re-opened, then a restart is required. A moderator can do so when they notice it.

...Error: One or more locations are missing, all are required...

You would notice this is most likely to happen on PRs that have a merge conflict. We have merged a fix for the error. These PRs may have been made before the fix: https://github.com/freeCodeCamp/freeCodeCamp/pull/24587 and hence need a rebase. Which basically means getting the conflicts to be resolved.

The best way to address this, is to ask the OP of the PR to close and re-open a new PR, since the files are conflicted anyways.

raisedadead on 23 Oct 2018

👍2

@raisedadead the following error has started popping up in some of the recent Travis failures:

Failed Tests: There were 6 failures
Use @each to Map Over Items in a List
✖ Your .blue-bg class should have a background-color of blue.
✖ Your .black-bg class should have a background-color of black.
✖ Your .red-bg class should have a background-color of red.
✖ Your .blue-bg class should have a background-color of blue.
✖ Your .black-bg class should have a background-color of black.
✖ Your .red-bg class should have a background-color of red.
total: 4015
passing: 4009
failing: 6
duration: 2m 29.6s

Nirajn2311 on 23 Oct 2018

👍2

@Nirajn2311 @raisedadead recently solution for SASS challenge was added, but tests can't check a solution for SASS, only pure HTML/CSS/JS. I work on it and going to make PR soon.

ValeraS on 23 Oct 2018

👍2

@raisedadead the following error has started popping up in some of the recent Travis failures:

Failed Tests: There were 6 failures
Use @each to Map Over Items in a List
✖ Your .blue-bg class should have a background-color of blue.
✖ Your .black-bg class should have a background-color of black.
✖ Your .red-bg class should have a background-color of red.
✖ Your .blue-bg class should have a background-color of blue.
✖ Your .black-bg class should have a background-color of black.
✖ Your .red-bg class should have a background-color of red.
total: 4015
passing: 4009
failing: 6
duration: 2m 29.6s

Same error for CI for me PR #27608

jagzmz on 24 Oct 2018

made a PR #27716

ValeraS on 24 Oct 2018

👍2

I guess, a good option would be to use Travis CI build stages. I'll explore this a bit

raisedadead on 24 Oct 2018

👍1

Travis CI build stages are better as it will be easier to identify the problems instead of going through one long log.

Nirajn2311 on 24 Oct 2018

Guys, if the script has been run fully, #21951 hasn't rebuilt

thecodingaviator on 16 Nov 2018

@RandellDawson I noticed many of the PR's I've been removing have already been restarted to fix broken builds, which is awesome! Thanks for writing this script.

Do we still need this issue to stay open or can we close it?

QuincyLarson on 28 Dec 2018

@QuincyLarson I believe this issue has been resolved by this PR. I will let @raisedadead make the final determination.

RandellDawson on 28 Dec 2018

🎉2

Was this page helpful?

0 / 5 - 0 ratings