Virtual-environments: Ubuntu jobs fail to fetch apt packages with 402 Payment Required

[JasonGross]

Description

E: Failed to fetch https://packagecloud.io/github/git-lfs/ubuntu/dists/bionic/InRelease  402  Payment Required [IP: 54.183.38.243 443]
E: The repository 'https://packagecloud.io/github/git-lfs/ubuntu bionic InRelease' is no longer signed.

Area for Triage:
Packages

Question, Bug, or Feature?:
Bug

Virtual environments affected

• [ ] Ubuntu 16.04
• [x] Ubuntu 18.04
• [ ] Ubuntu 20.04
• [ ] macOS 10.15
• [ ] macOS 11.0
• [ ] Windows Server 2016 R2
• [ ] Windows Server 2019

Expected behavior
Packages can be installed

Actual behavior
Job fails on package install step with 402

Repro steps
https://github.com/mit-plv/fiat-crypto/runs/1355905419?check_suite_focus=true

[buvinghausen]

@JasonGross it impacts all versions of Ubuntu

https://packagecloud.io/github/git-lfs/ubuntu/dists/focal/InRelease
https://packagecloud.io/github/git-lfs/ubuntu/dists/bionic/InRelease
https://packagecloud.io/github/git-lfs/ubuntu/dists/xenial/InRelease

The issue is our friends at packagecloud.io apparently can't access their key due to non payment.

We are trying to remove the packagecloud line from the sources.list so when you run apt it doesn't try to ping them. I will post a workaround here if it works.

[buvinghausen]

Added a step to just remove git-lfs before we do any apt stuff seems to work.

      - name: remove git-lfs
        shell: bash
        run: |
          cat /etc/apt/sources.list
          ls /etc/apt/sources.list.d
          sudo rm /etc/apt/sources.list.d/github_git-lfs.list
          sudo rm /etc/apt/sources.list.d/github_git-lfs.list.save

Credit goes to @kyle-assurely

[buvinghausen]

FYi the keys are back online...

[andy-mishechkin]

I think, we ca close this ticket
If this issue still not decided - feel free to reopen

[buvinghausen]

@andy-mishechkin yes the issue can be closed this did make it crystal clear we need to look into a self hosted runner so that we don't have 500+ dependencies that don't relate to our software stack any of which can take us down if something goes wrong.