Virtual-environments: Unable to connect to azure.archive.ubuntu.com flaky failure on ubuntu-latest

[traversaro]

Describe the bug
On some jobs (apparently in a non-deterministic way) commands such as apt-get install fail with the following error:
~
Get:1 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic/main amd64 gdb amd64 8.2-0ubuntu1~18.04 [3024 kB]
Get:2 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic/main amd64 gdbserver amd64 8.2-0ubuntu1~18.04 [292 kB]
Get:3 http://ppa.launchpad.net/ondrej/php/ubuntu bionic/main amd64 libxml2-dev amd64 2.9.10+dfsg-2+ubuntu18.04.1+deb.sury.org+1 [821 kB]
Get:4 http://ppa.launchpad.net/ondrej/php/ubuntu bionic/main amd64 libxml2 amd64 2.9.10+dfsg-2+ubuntu18.04.1+deb.sury.org+1 [726 kB]
Get:5 http://ppa.launchpad.net/ondrej/php/ubuntu bionic/main amd64 libgraphicsmagick-q16-3 amd64 1.3.30+hg15796-1+ubuntu18.04.1+deb.sury.org+2 [1181 kB]
Get:6 http://ppa.launchpad.net/ondrej/php/ubuntu bionic/main amd64 libgraphicsmagick++-q16-12 amd64 1.3.30+hg15796-1+ubuntu18.04.1+deb.sury.org+2 [144 kB]
Err:7 http://azure.archive.ubuntu.com/ubuntu bionic/main amd64 libdouble-conversion1 amd64 2.0.1-4ubuntu1
Could not connect to azure.archive.ubuntu.com:80 (52.177.174.250), connection timed out
Ign:8 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libqt5core5a amd64 5.9.5+dfsg-0ubuntu2.5
Ign:9 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libwayland-server0 amd64 1.16.0-1ubuntu1.1~18.04.3
Err:10 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libgbm1 amd64 19.2.8-0ubuntu0~18.04.3
Unable to connect to azure.archive.ubuntu.com:http:
Err:11 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxcb-xfixes0 amd64 1.13-2~ubuntu18.04
Unable to connect to azure.archive.ubuntu.com:http:
Err:12 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libegl-mesa0 amd64 19.2.8-0ubuntu0~18.04.3
Unable to connect to azure.archive.ubuntu.com:http:
Err:13 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libegl1 amd64 1.0.0-2ubuntu2.3
~

For an example of such failure, check https://github.com/robotology/idyntree/pull/668/checks?check_run_id=560084383 . I would not be too surprised about CI jobs failing for networks problems, but this specific issue seems to be extremely frequent. I did not collect precise data on how much frequently this happens, but my impression that it happens in the order of magnitude of 1 out of 10 builds.

This issue is already being discussed on GitHub Community Forum at https://github.community/t5/GitHub-Actions/Install-dependencies-in-Ubuntu-flakes-several-times-a-wekk/td-p/51785 , but the GitHub support suggested me to open an issue here as well.

Area for Triage:

Question, Bug, or Feature?:
Bug

Virtual environments affected

• [ ] macOS 10.15
• [x] Ubuntu 16.04 LTS
• [x] Ubuntu 18.04 LTS
• [ ] Windows Server 2016 R2
• [ ] Windows Server 2019

Expected behavior
I would expect commands such as apt-get install to work correctly.

Actual behavior
It is not clear what is triggering this behavior, but sometimes apt-get install fails.

[MSP-Greg]

I added -o Acquire::Retries=3, and today it worked, then failed for a while, then starting working again.

It may not be a GitHub issue, but they (and especially MSFT) have the resources to mirror azure.archive.ubuntu.com so Actions CI is reliable...

[boegel]

We've been seeing issues like this fairly frequently since Sunday March 29th...

[boegel]

Using -o Acquire::Retries=3 is not helping for us, see for example https://github.com/easybuilders/easybuild-easyblocks/pull/2012/checks?check_run_id=563883264

[MCOfficer]

I mentioned this on the github community post, but you can probably use apt-spy2 to find a working mirror before using apt.

[MSP-Greg]

@boegel

Sorry, I didn't mean to imply that using -o Acquire::Retries=3 was a fix. What I meant was that if one is using it and apt-get still times out, there is really a problem.

[boegel]

I mentioned this on the github community post, but you can probably use apt-spy2 to find a working mirror before using apt.

That adds a couple of minutes to each run, but it seems to work well to circumvent the problems, thanks a lot!

For other, see for example https://github.com/easybuilders/easybuild-easyconfigs/pull/10341

[znarf]

On our side, we get 503 errors on almost all runs past few hours:

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/universe/f/fonts-arphic-bkai00mp/fonts-arphic-bkai00mp_2.10-17_all.deb  503  Service Unavailable [IP: 52.177.174.250 80]

[deivid-rodriguez]

I added this to my workflow before apt-get install as a workaround:

sudo sed -i 's/azure\.//' /etc/apt/sources.list
sudo apt-get update

[paoloczi]

Several workarounds proposed, but no comments about fixing this for real. This is a serious issue making GitHub Actions unusable for anybody working with Ubuntu, and should be looked at with high priority.

[alepauly]

@traversaro (and everyone else affected) - We've engaged the mirror hosts and they have added capacity to improve its reliability. I'll keep this open until we can confirm the issue is resolved. Please let us know if this continues to occur. Thank you for the reports.

[MSP-Greg]

@alepauly

Thank you.

Last saw a problem at 23:15 GMT (3 hours ago), at which point I changed servers. Did the update/change happen after that?

EDIT: 02:25 GMT - reverted server change, and no timeout errors.

[alepauly]

@MSP-Greg, I don't have he exact time where the fixes were put in place yet but I believe it was in the last two hours. Please let me know if you see more issues, thanks!

[traversaro]

@traversaro (and everyone else affected) - We've engaged the mirror hosts and they have added capacity to improve its reliability. I'll keep this open until we can confirm the issue is resolved. Please let us know if this continues to occur. Thank you for the reports.

Thanks a lot @alepauly !

[hloeung]

@alepauly has something changed with GitHub's Actions on the 24th March?

[alepauly]

@alepauly has something changed with GitHub's Actions on the 24th March?

@hloeung we make improvements constantly so, yes. If you're having a specific problem with the tools installed please file an issue in this repo so we can help.

[pestophagous]

Noting another workaround, since there are many interested parties on this thread:

https://github.com/ros2/rmw_cyclonedds/pull/134/files

I'm not the author of that, but I found it a few days ago when I saw this azure-related outage was happening to me. I took the same approach as the ros2/rmw_cyclonedds team, and I've been quite happy with it.

Based on search results I found on April 1 (before this present issue #675 had been created), this is a recurring problem: https://github.community/t5/GitHub-Actions/sudo-apt-install-fails-with-Unable-to-connect-to-azure-archive/td-p/32154

[hloeung]

@alepauly how can I get in touch with you? Would like to discuss something without having to file an issue.

[boegel]

We changed to using apt-spy2 a couple of days ago which helped for a while, but today tests starting failing again because of what looks like a faulty mirror picked by apt-spy2:

E: Failed to fetch http://mirrors.codec-cluster.org/ubuntu/pool/universe/l/lua-bit32/lua-bit32_5.3.0-3_amd64.deb  403  Forbidden [IP: 65.49.71.107 80]

So, we've disabled the use of apt-spy2 again, based on @alepauly's comment that the issues with azure.archive.ubuntu.com should be largely resolved now... (see https://github.com/easybuilders/easybuild-easyconfigs/pull/10374)

[traversaro]

It is not an extensive test, but personally I did not have any more azure.archive.ubuntu.com-related failures in the last few days, thanks @alepauly .

[deivid-rodriguez]

Neither did us, everything seems fine now :+1:.

[miketimofeev]

@traversaro @deivid-rodriguez thanks for the updates! We're going to close the issue, but feel free to open a new one if there are any similar issues in the future.

[akhilerm]

Hitting the same issue in ubuntu bionic. when running
apt-get install

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libudev-dev_237-3ubuntu10.41_amd64.deb  404  Not Found [IP: 52.177.174.250 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[MCOfficer]

Not the same issue - the mirror is reachable, but doesn't have the file you request. Did you run apt-get update?

[akhilerm]

@MCOfficer Was the package recently removed from azure.ubuntu.com ? Because it was working till yesterday.

Yep. its working after doing apt-get update

[MCOfficer]

@MCOfficer Was the package recently removed from azure.ubuntu.com ? Because it was working till yesterday.

When a package is updated, the old version may be removed from the mirror. If you get 404's, 99% of the time apt-get update fixes it by fetching the mirror's new index.

[jessevdp]

I can confirm that running apt-get update before any calls to apt-get install on GitHub Actions seems to resolve this issue. We started experiencing this problem just today.