Node: 403 Forbidden for the most recent coverage reports

[RaisinTen]

Why am I getting a 403 Forbidden when I'm trying to see the most recent coverage reports?

*
* JS Coverage - https://coverage.nodejs.org/coverage-adae822035b64053/index.html
* C++ Coverage - https://coverage.nodejs.org/coverage-adae822035b64053/cxxcoverage.html
*
* JS Coverage - https://coverage.nodejs.org/coverage-f06f3c0ea0405ef2/index.html
* C++ Coverage - https://coverage.nodejs.org/coverage-f06f3c0ea0405ef2/cxxcoverage.html

[gireeshpunathil]

I too get it - let me put it in the core repo.

/cc @nodejs/website @nodejs/build may be?

[bnb]

AFAIK there were some c8 changes recently? Maybe related?

[Trott]

If I had to guess, it is/was a full disk. But that's totally a guess.

Unrelated to c8, I'm pretty sure. Some successful new-c8 runs happened.

Not seeing any builds at https://ci.nodejs.org/view/All/job/node-test-commit-coverage/ which is puzzling (unless that's not where we run this stuff anymore?).

Kicking off a run there to see if it magically fixes itself because why not? https://ci.nodejs.org/view/All/job/node-test-commit-coverage/62/

[richardlau]

Not seeing any builds at https://ci.nodejs.org/view/All/job/node-test-commit-coverage/ which is puzzling (unless that's not where we run this stuff anymore?).

I think https://ci.nodejs.org/job/node-test-commit-linux-coverage-daily/ is the job (https://github.com/nodejs/build/pull/2468 recently updated the link in nodejs/build). Looks to be broken for the last few days -- haven't looked into it but maybe the job was broken by recent changes/cleanups to get the coverage working with GitHub Actions?

[Trott]

Looks like the coverage runs OK but the transfer of the report to the server fails?

20:17:21 + rsync -r out coveragedata:coverage-out
20:47:21 Build timed out (after 30 minutes). Marking the build as failed.
20:47:21 Build was aborted
20:47:21 Sending e-mails to: [email protected]
20:47:21 rsync: connection unexpectedly closed (78669875 bytes received so far) [sender]
20:47:21 rsync error: unexplained error (code 143) at io.c(226) [sender=3.1.0]

[mhdawson]

@richardlau maybe we can pair on getting that fixed up. We currently need to do periodic cleanup or the rsync times out. If we can do that together then maybe you can help improve as I've not been able to prioritise that. If you can find a hour in my calendar tomorrow we could do it together.

[richardlau]

@richardlau maybe we can pair on getting that fixed up. We currently need to do periodic cleanup or the rsync times out. If we can do that together then maybe you can help improve as I've not been able to prioritise that. If you can find a hour in my calendar tomorrow we could do it together.

@mhdawson unfortunately it looks like both of our calendars tomorrow are fairly full, although I may get an hour back if the Release WG meeting gets cancelled.

I also don't appear to be able to ssh into infra-softlayer-ubuntu1404-x64-2 (I'm being prompted for a password). The most recent jobs are failing with a read only filesystem, which I'm guessing could be related.
https://ci.nodejs.org/job/node-test-commit-linux-coverage-daily/692/nodes=benchmark/console

stderr: error: could not lock config file .git/config: Read-only file system

[mhdawson]

@richardlau I seemed to be able to login with the test key that I have ok. I rebooted the machine. Can you check if that resolves the problem and if not find a time in our calendar that will work even if its not tomorrow.

[richardlau]

@mhdawson I now can't connect to infra-softlayer-ubuntu1404-x64-2 at all (time outs) and it doesn't appear to have come back in Jenkins either. I've sent out an invite for us to pair.

[rvagg]

I've done a soft and a hard reboot of that machine and it's not coming back online, ruh roh

[mhdawson]

One last thing to try is a power off/on. Doing that now

[mhdawson]

Power on/off did not work.

That machine was only being used for benchmarks

Benchmark WG is no longer active and I have an open issue to de-charter so getting the machine working for the benchmarks does not make sense.

There is also an effort to transition to be newer/better coverage data capture approach, it's just not quite ready yet. In the interim
I'm thinking we use a different machine to run the coverage job (needs to be the same machine every time) to bridge the gap.

[richardlau]

As previously mentioned infra-softlayer-ubuntu1404-x64-2 hasn't come back. @mhdawson and I have set up test-nearform_intel-ubuntu1604-x64-1 to run the coverage job on the CI.
Test build: https://ci.nodejs.org/job/node-test-commit-linux-coverage-daily/695/

We've also disabled the benchmark-footprint-* jobs that were running on infra-softlayer-ubuntu1404-x64-2 for now.

[richardlau]

It's quite possible the job itself will need to be changed to take into account the changes in https://github.com/nodejs/node/pull/35767. I'll take a look on Monday.

[richardlau]

Last few runs of https://ci.nodejs.org/job/node-test-commit-linux-coverage-daily/ have been green and the links for the last few days on https://coverage.nodejs.org/ appear to be working. I'll close this out but feel free to reopen if there's anything amiss.

I've also tidied up the job a bit to stop installing and running c8 a second time (it's already done as part of make coverage) and to remove the tmp directory before rsyncing the reports over to the data server: https://ci.nodejs.org/job/node-test-commit-linux-coverage-daily/jobConfigHistory/showDiffFiles?timestamp1=2020-11-20_12-50-51&timestamp2=2020-11-23_15-56-14

[RaisinTen]

Thanks for doing this! :slightly_smiling_face: