Iiab: yarn update failures due to IIAB / internetarchive on Ubuntu 18.04, 16.04, etc

[holta]

@mitra42 these errors have been ongoing for a day or two now...I was hoping they would just go away, pending yarn maintainers cleaning up their keys/signatures...but so far no.

Are you aware and/or can you recommend any corrective action?

Pastes from 2 different OS's below, in case that helps...

root@box:~# cat /etc/issue.net
Ubuntu 18.04.4 LTS

root@box:~# cat /etc/apt/sources.list.d/yarn.list
deb http://dl.yarnpkg.com/debian/ stable main

root@box:~# apt update
Get:1 http://dl.yarnpkg.com/debian stable InRelease [17.1 kB]
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://linux.teamviewer.com/deb stable InRelease [9,388 B]
Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:6 https://deb.nodesource.com/node_10.x bionic InRelease
Hit:7 http://ppa.launchpad.net/ansible/ansible/ubuntu bionic InRelease
Hit:8 http://security.ubuntu.com/ubuntu bionic-security InRelease
Err:1 http://dl.yarnpkg.com/debian stable InRelease
  The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <[email protected]>
Fetched 26.5 kB in 1s (20.3 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://dl.yarnpkg.com/debian stable InRelease: The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <[email protected]>
W: Failed to fetch http://dl.yarnpkg.com/debian/dists/stable/InRelease  The following signatures were invalid: EXPKEYSIG 23E7166788B63E1E Yarn Packaging <[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.

root@box:~# cat /etc/issue.net
Ubuntu 16.04.6 LTS

root@box:~# cat /etc/apt/sources.list.d/yarn.list
deb http://dl.yarnpkg.com/debian/ stable main

root@box:~# apt update
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Get:4 http://dl.yarnpkg.com/debian stable InRelease [17.1 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
Hit:6 http://ppa.launchpad.net/ansible/ansible/ubuntu bionic InRelease
Hit:7 https://deb.nodesource.com/node_10.x xenial InRelease
Hit:8 http://ppa.launchpad.net/stebbins/handbrake-releases/ubuntu xenial InRelease
Err:4 http://dl.yarnpkg.com/debian stable InRelease
  The following signatures were invalid: KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1580619281  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983
Fetched 124 kB in 0s (158 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://dl.yarnpkg.com/debian stable InRelease: The following signatures were invalid: KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1580619281  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983
W: Failed to fetch http://dl.yarnpkg.com/debian/dists/stable/InRelease  The following signatures were invalid: KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1580619281  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983
W: Some index files failed to download. They have been ignored, or old ones used instead.

[jvonau]

Try editing /etc/apt/sources.list.d/yarn.list from
deb http://dl.yarnpkg.com/debian/ stable main
to
deb [ allow-insecure=yes ] http://dl.yarnpkg.com/debian/ stable main

[indivisualvj]

i removed yarn.list, purged yarn and reinstalled like instructions say.
works fine.
https://classic.yarnpkg.com/en/docs/install/#debian-stable

[mitra42]

Yarn is going through a new release process, just these last week or two, evolving to yarn 2. There may be some glitches while that happens. If what @jvonau says worked then lets leave it at that for at least a few weeks until yarn settles down. I'll happily jump on any issues, but am panicing to be ready for the conference that both @holta and I will be at for the rest of the week.

[mitra42]

@indivisualvj - can you clarify, when you say "like instructions say" do you mean @jvonau 's instructions that worked, or some other ones ?

[indivisualvj]

@indivisualvj - can you clarify, when you say "like instructions say" do you mean @jvonau 's instructions that worked, or some other ones ?
sorry. thought you'd see the link i added.

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

On Ubuntu 16.04 or below and Debian Stable, you will also need to configure the NodeSource repository to get a new enough version of Node.js.

Then you can simply:

sudo apt update && sudo apt install

[mitra42]

@holta
Those lines above look like they should be part of roles/yarn/tasks/main.yml which currently has

- name: "Yarn | Ensure Debian package is in sources list"
  lineinfile:
    dest: /etc/apt/sources.list.d/yarn.list
    regexp: 'deb http://dl.yarnpkg.com/debian/ stable main'
    line: 'deb http://dl.yarnpkg.com/debian/ stable main'
    state: present

I didn't build that ansible, and am not confident turning bash into ansible to make that work.

[mitra42]

I just took a second look and it appears that if I'm reading it correctly, the only difference between current yarn/tasks/main.yml and these instructions is that lines 16 and 17 use http rather than https which I wouldnt have thought would cause this to fail.

@holta - were you reinstalling ontop of something that could have this bad key already ?

[holta]

@holta - were you reinstalling ontop of something that could have this bad key already ?

@mitra42 I was reinstalling on top of an existing IIAB & its Internet Archive (http://box:4244).

And as @indivisualvj and @jvonau hints, 'apt purge yarn' and 'rm /etc/apt/sources.list.d/yarn.list' did work did allow apt update to complete without errors.
Which in turn allowed the 'Offline Internet Archive' to install again (using ./runrole --reinstall internetarchive after PR #2238 is merged!)

But this is not the kind of tinkering that our developing world field implementers know how to deal with (so hopefully the Yarn install/update process becomes much less messy going forward!)

[mitra42]

Yes - I totally agree that its not what we'd expect in-country implementers to figure out, it looks at first glance that the key used to sign yarn expired, so if you update over the top of an old installation its "deb" that is failing.

But yarn has been going through some changes, it looks like there was a move from v1 to v2 that took a lot of people by surprise - I don't think that is what is hitting you as I haven't seen this problem yet on any of the Dockerfiles (which also install yarn), but I would appreciate if someone who succeeded in doing this could confirm what version of yarn is installed (yarn --version at a shell prompt should hopefully still report 1.x.x).

Yarn is a really standard package, you really can't install anything on javascript without either yarn or npm, and along with many people I switched to yarn because it was MORE predictable than npm, so this has taken me by surprise as well.

[holta]

FWIW machines (Ubuntu 18.04 & 16.04) that experienced the above-mentioned glitch were running yarn 1.21.1, which appears to be the very latest (released 2019-12-11) according to https://github.com/yarnpkg/yarn/releases

Anyway let's hope that the people who package up yarn (for 'apt') can keep their distribution/signing keys from expiring next time~

[jvonau]

name: "Yarn | GPG"
apt_key:
url: https://dl.yarnpkg.com/debian/pubkey.gpg
state: present

IIAB only imports yarn's deb signing key once, the underlying key changed so prior installs would have the expired key installed while new installs would happily just grab the new key and use it without issue. There is no bug in IIAB with this transient issue, in the wild installations would need to update to the new signing key without IIAB being in the mix as apt is failing as per above. Think curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - from above would be enough to ensure apt works with the yarn repo without resorting to any of the above workarounds. A quick test would be to install the key on a failing machine and run apt update to ascertain if the error went away and the error does go away.

[holta]

Think curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - from above would be enough to ensure apt works with the yarn repo without resorting to any of the above workarounds. A quick test would be to install the key on a failing machine and run apt update to ascertain if the error went away and the error does go away.

Confirmed on the Ubuntu 16.04 machine: running curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - indeed makes the apt update error go away...

[jvonau]

Now make a note of the 'upstream snafu' in the role's readme, unless there was some warning to the up coming change announced somewhere...

[jvonau]

https://github.com/yarnpkg/releases/commit/649a87723204e3a55e7d878ff7911c5b41292117

[holta]

@mitra42 I'll close this now if you think that's best?

Thanks @indivisualvj & @jvonau for explaining the root cause!

[mitra42]

Jerry - did you push that curl into.a change somewhere?

If I understand your diagnosis correctly the part

- name: "Yarn | GPG"
  apt_key:
    url: https://dl.yarnpkg.com/debian/pubkey.gpg
    state: present

failed, because apt_key decided (on machines that are updating) that an expired key was "present". It passed on new installations (or in my Dockerfile installs) because there was no key. This actually feels like a ansible bug (with their apt_key function) to me.

Strange thing to me though, is that I'd have expected, on an install that is updating, that the step of installing yarn would be skipped because the only reason it would have a bad key would be if yarn was installed already ?

[jvonau]

Nothing failed except upstream, your expectation of yarn being skipped on an updating install is correct as we only perform the yarn setup once. There is no bug in ansible's apt_key routine, it does what we ask, install the available key if not present.

[holta]

Transient thing. @jvonau agrees let's close it.