Virtual-environments: azure.archive.ubuntu.com (default apt mirror for at least ubuntu-latest runners) appears to have had its content removed

[sfoster1]

Description
The mirror of first resort on at least the ubuntu-latest/ubuntu-2018 runners is http://azure.archive.ubuntu.com/ubuntu/, with packages served from http://azure.archive.ubuntu.com/ubuntu/pool/...

At 15:33 UTC 11/20/2020 all content in http://azure.archive.ubuntu.com/ubuntu/pool appears to have been removed (screenshot attached), which causes apt installs on those ubuntu runners to fail, since the repository looks ok but the package content isn't present:

Obviously this isn't the fault of the code or maintainers of this repo, but it might be a good idea to switch default apt mirrors until/unless this is fixed, and you may have communications with the maintainers of that mirror.

Area for Triage:
Packages

Question, Bug, or Feature?:
Question

Virtual environments affected

• [ ] Ubuntu 16.04
• [x] Ubuntu 18.04
• [ ] Ubuntu 20.04
• [ ] macOS 10.15
• [ ] macOS 11.0
• [ ] Windows Server 2016 R2
• [ ] Windows Server 2019

Expected behavior
apt-get install -y libsystemd-dev succeeds

Actual behavior

Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  libsystemd-dev
0 upgraded, 1 newly installed, 0 to remove and 12 not upgraded.
Need to get 184 kB of archives.
After this operation, 743 kB of additional disk space will be used.
Err:1 http://azure.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsystemd-dev amd64 237-3ubuntu10.42
  404  Not Found [IP: 52.147.219.192 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd-dev_237-3ubuntu10.42_amd64.deb  404  Not Found [IP: 52.147.219.192 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

(from https://github.com/Opentrons/opentrons/pull/6793/checks?check_run_id=1432068369)

Repro steps

This build: https://github.com/Opentrons/opentrons/pull/6793/checks?check_run_id=1432068369 shows the issue; I think any apt-get install on an ubuntu-latest or ubuntu-2018 runner will have the issue.

[sfoster1]

Given that you can do this: http://azure.archive.ubuntu.com/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ubuntu/ I think someone on their end might have made a mistake involving symlinks

[maxim-lobanov]

@sfoster1 , thank you for report.
Unfortunately, we can't do something from image side on short term.

We can build new image with mirrors but full deployment of this image will take a week. So I guess the initial issue will be fixed earlier.

[sfoster1]

@maxim-lobanov OK, I totally understand. Do you folks have a point of contact with someone at canonical to notify them about this?

And would you like me to leave this open so there's something people can find if they google the issue they're seeing, or close it since there's nothing you can do?

[sfoster1]

And for reference of anybody finding this via google - you can remove this mirror from your runner's source list with this code:

sudo sed -i 's/azure\.//' /etc/apt/sources.list 
sudo apt-get update 

(there's almost certainly a better way to do this, but this way does work).

[maxim-lobanov]

@sfoster1 , Looks like everything is good with azure.archive.ubuntu.com

Please try to update apt-get update before install.

sudo apt-get update
sudo apt-get install -y libsystemd-dev

works perfectly for me.
Please pay attention that apt cache can be a bit outdated on image.

Looks like apt was updated recently and link to deb package was changed:

before: http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd-dev_237-3ubuntu10.42_amd64.deb
after:  http://azure.archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd-dev_237-3ubuntu10.43_amd64.deb

that it is the reason why package is not found without apt update.

One more interesting thing is that:
1) http://azure.archive.ubuntu.com/ubuntu/ doesn't contain pool folder
2) http://azure.archive.ubuntu.com/ubuntu/pool/ is available and contains all files

[sfoster1]

Oh, that's really strange. Thanks for looking into this! I'll happily close this now.

[hloeung]

There are two types of archives - full contains local copy of /pool and partial does not, but instead proxies back to the specified full archive (usually the closest). This is why you don't see /pool in the generated index when visiting /ubuntu/ but it's accessible and showing all contents via /ubuntu/pool/.

As for the symlink, that's intentional mostly for community operated mirrors.