Crystal: GPG error trying to upgrade Crystal from Debian repository

[matiasgarciaisaia]

Reportedly, running apt-get update gives GPG errors since 0.23.0 was published:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://dist.crystal-lang.org crystal InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4EBAC6667697DD2

W: Failed to fetch https://dist.crystal-lang.org/apt/dists/crystal/InRelease

W: Some index files failed to download. They have been ignored, or old ones used instead.

This is due to the repo now being signed with a subkey (A4EBAC6667697DD2) of the previously used key (09617FD37CC06B54).

At the moment, running apt-key adv --keyserver keys.gnupg.net --recv-keys A4EBAC6667697DD2 as root solves the issue, and it _may_ be possible that running apt-key adv --keyserver keys.gnupg.net --recv-keys 09617FD37CC06B54 works too (it re-downloads the master key, now with the subkey added).

The second command is included in the setup script, so fresh new installations doesn't have this issue.

I'm not sure if there's a way to sign the repo using the subkey _on behalf of_ the master key, so users don't need to do any update? If not, we should at least inform everyone how to fix this.

If you have examples of _things_ that broke due to this change (ie, CI builds, whatever), please post them here.

[RX14]

So you added a new GPG signing subkey (which modifies the public key) and all the new packages are signed with that? I think the solution is just to get everyone to re-download the GPG pubkey.

[mverzilli]

@RX14, yes, that is the situation. The problem is how to get everyone to update the pubkey effectively (that is, in a way that's likely to occur without needing them to come here and report that they can't get the package to install). Do you have any ideas on that?

[RX14]

@mverzilli Most distributions seem to have packages containing the maintainers' pubkeys. In this situation they would just push a new package (obviously signed by an older key) and have it update the system keyring. This isn't really applicable to our situation, so I would guess there really isn't an automated solution to this. Perhaps contact some debian maintainers?

[ivy]

Like @RX14 mentioned, the solution a lot of maintainers use is a *-keyring package (e.g. debian-keyring, ubuntu-keyring, gnome-keyring, deb.torproject.org-keyring, etc.). That's not too helpful here though since it would have had to be installed ahead of time. 😞

I'm by no means an expert in Apt package maintenance but out of curiosity, I did some searching around for what others have done in this situation:

• Cloudflare put a note about their key change on their package index: https://pkg.cloudflare.com/#gpg-key-update
• Nginx published a blog post: https://www.nginx.com/blog/updating-gpg-key-nginx-products/

Maybe there needs to be a note somewhere prominent?

I'd also add that Dockerfile.release appears to be in need of an update.

[mverzilli]

@RX14 @ivy Thanks for your suggestions! So yes, I guess we could take the same approach as Nginx and write a release blog post explaining how to solve it.

Thanks for pointing out that Dockerfile.release needs an update too.

[matiasgarciaisaia]

We still have the master key, though - so it MAY be possible to roll back
the repo to the old key, then implement the keyring, then upgrade the repo
to a new key.

But we'll see the what's best.

Maybe we just roll back for now until we make the decision without being in
a rush.

[ivy]

@matiasgarciaisaia What was the original reason for the key change?

A seamless transition is definitely possible. I might've jumped the gun on opening a PR. 😅

As far as I know, you can't publish an Apt repository signed by multiple keys. There'd have to be some time for users to upgrade in stages. First, to a new dependency on a crystal-keyring package that includes both keys. Then after some time, make the switch to signing with the new key and remove the old from the keyring.

[straight-shoota]

Has the key only changed for APT repository or others as well?

[mverzilli]

@ivy the reason was that we wanted to empower more people to generate Crystal releases but we didn't want everyone to share the same key. We have obviously underestimated the consequences.

[ivy]

@mverzilli Interesting. When you mentioned empowering more people, were you only referring to team members?

I think a build server would be the best solution for automating the signing and release of packages. Maintainers would instead sign Git tags with their personal key and the build system would sign and release the final package on their behalf after authorizing them. This would make it so that a few trusted people could cut a release without having access to the master key or having to set anything up locally. It also seems to be a standard practice as Apt keys can cause a lot of harm when compromised.

I wouldn't mind taking a look at this further. Is omnibus-crystal where this work is happening?

[matiasgarciaisaia]

Exactly as Martin said.

For now, it's only core team - and being able to cut a release would hardly
ever mean you're NOT part of the core team.

Having a CI is the long-term solution, sure - but we've just got a single
command build process, so we're still missing a couple of steps. Not that
we don't want to do it, but we aren't there yet.

Effectively, crystal-lang/omnibus-crystal is where the party is going on.
Look for the current branch, though, as we still haven't merged it - I
don't see why, now that I think about it.

Thanks for getting involved 👌

[matiasgarciaisaia]

BTW, the key only changed for APT repos. For CentOS - the repo is still at
0.22.0, without changes since its original release.

[ivy]

@matiasgarciaisaia No problem. I'm more than happy to help out. 😄

I looked through the Omnibus repository and I think I may have found multiple security vulnerabilities. Who should I talk to about this?

[matiasgarciaisaia]

[email protected]

You can find our GPG key on https://crystal-lang.org/community#security

Thanks a lot!

[matiasgarciaisaia]

Hi everyone!

We've just updated the APT repository - now signed with the old key once again.

If you can please confirm wether it works or not, that'd be great to hear.

We'll keep working on the keyring package and that stuff, but we wanted to fix the repo first.

[straight-shoota]

On Travis-CI installs with apt are working again 👍 But #4647 still prohibits successful builds.

[matiasgarciaisaia]

@straight-shoota thanks for confirming!

We're working on #4647. Could you share a link to one of the Travis builds to help us understand the issue?

[straight-shoota]

Sure, https://travis-ci.org/straight-shoota/crinja/builds/245963652
Base image for Crystal builds on Travis-CI is an outdated Ubuntu precise 12.04

[konovod]

Base image for Crystal builds on Travis-CI is an outdated Ubuntu precise 12.04

it supports Ubuntu trusty 14.04, if you add dist: trusty line to travis.yml. But on 14.04 the error is same.

[matiasgarciaisaia]

@straight-shoota @konovod let's please follow that discussion in #4647 ;-)

I'll keep this one open for now to work on the keyring thingy, unless we decide at some point in the future to make a specific issue for that 👍

[ivy]

I should have some more time to hack on this over the weekend.

Building Omnibus Packages for Debian, makes it look pretty trivial to add Debian-specific customizations. While I'm at it, do we also want to add hard dependencies on the other required packages (e.g. gcc and friends)?

[Fryguy]

I'm also passed this error now and blocked by #4647 on Travis. Example failing Travis build is https://travis-ci.org/Fryguy/cargo_shorts/builds/250606812

[jhass]

Uhm, I guess this is fixed by now?

[bcardiff]

Yes, we are actually not using the subkey for signing.