K3s: Cannot pull public image from Github Container Registry (GHCR) anonymously

[windsource]

Environmental Info:
K3s Version:
k3s version v1.19.3+k3s1 (974ad30b)

Node(s) CPU architecture, OS, and Version:

Linux manjaro 5.4.67-1-MANJARO #1 SMP PREEMPT Wed Sep 23 14:20:18 UTC 2020 x86_64 GNU/Linux

Cluster Configuration:

1 master, 0 workers

Describe the bug:

I am unable to pull a public image from Github Container Registry (GHCR) without authentication. I get the error: "failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden". When I provide imagePullSecrets it works.

With the docker client pulling the same image from GHCR without authentication works fine.

Steps To Reproduce:

• Installed K3s:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.19.3+k3s1 sh -s - --write-kubeconfig-mode 644

To reproduce the problem you can pull the image from the command line with:

sudo k3s ctr image pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0: resolving      |--------------------------------------| 
elapsed: 0.2 s                                               total:   0.0 B (0.0 B/s)                                         
ctr: failed to resolve reference "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

When using docker client it works:

docker pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0    
1.0.0: Pulling from windsource/nextcloud-influxdb-tracks-importer
df20fa9351a1: Already exists 
58c8db755c6d: Pull complete 
6dfe9eecf726: Pull complete 
3c96a94aa10c: Pull complete 
Digest: sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0
Status: Downloaded newer image for ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0

Expected behavior:
Public images from GHCR should be pulled anonymously without credentials (see also announcement from Github that states "With the container registry you can ...Access public container images anonymously."

Actual behavior:

Pulling images using k3s requires credentials for GHCR to pull public images.

Additional context / logs:

[brandond]

During the beta period, GHCR had several issues with their support for containerd due to not following the spec properly. See: https://github.com/containerd/containerd/issues/3291#issuecomment-494511092. I suspect that this is another instance of that.

[windsource]

Thanks @brandond. I also suspect that the problem is somewhere in the collaboration of containerd and GHCR. But the issue you refer to marks all problems as solved.

I further investigated the problem and it does actually depend on the image itself.

Pulling the public image ghcr.io/csantanapr/nginx:latest anonymously with containerd works but pulling the public image ghcr.io/windsource/nextcloud-influxdb-tracks-importer:1.0.0 anoymously does not work.

What could be the difference?

[brandond]

I just pushed a random dev image up to ghcr.io, marked it public, and was able to pull it using ctr:

brandond@sealand:~$ sudo ctr image pull ghcr.io/brandond/k3s:v1.19.3-k3s-766cd212-dirty-amd64
[sudo] password for brandond:
ghcr.io/brandond/k3s:v1.19.3-k3s-766cd212-dirty-amd64:                            resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:76d097a2bc11b6dc0c2c490ebdfb96e2ef0356e33f952dc8680b24cfa25ce500: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ee5eb13a956855cb691fda55d6fdf4c0933dc9eb2622cc38675e9bf72eeaee6f:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3171f9e0644db6285a740a701ea5f1a771852fccf0dcf7447d8354852c726c06:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:cfbf6dce60c89bac1943d07d22cb77137e7f0545cfd798a83d80803e594d595c:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:f3c9dda8bdbcc25774709eec635ed668b65eafffa0b4db6bab2a8a755d6a2381:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 8.8 s                                                                    total:  48.3 M (5.5 MiB/s)
unpacking linux/amd64 sha256:76d097a2bc11b6dc0c2c490ebdfb96e2ef0356e33f952dc8680b24cfa25ce500...
done

Can you share a screenshot of your package privacy settings? Note that you have to manage the privacy through the new package UI. There's a big red 'make public' button that goes away once you've clicked it once.

[windsource]

Here you see that the package ghcr.io/windsource/nextcloud-influxdb-tracks-importer is public: https://github.com/windsource?tab=packages.

The package settings look like:

[brandond]

I literally just pulled/tagged/pushed your image to my account and now I can pull it fine. That would suggest that this is an issue with something on the GHCR side, but exactly what I have no idea. What happens if you delete the package and then push it again and make it public?

On my dev host with Docker:

brandond@dev01:~$ docker pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest
latest: Pulling from windsource/nextcloud-influxdb-tracks-importer
df20fa9351a1: Already exists
58c8db755c6d: Pull complete
6dfe9eecf726: Pull complete
3c96a94aa10c: Pull complete
Digest: sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0
Status: Downloaded newer image for ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest

brandond@dev01:~$ docker tag ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest ghcr.io/brandond/nextcloud-influxdb-tracks-importer:latest

brandond@dev01:~$ docker push ghcr.io/brandond/nextcloud-influxdb-tracks-importer:latest
The push refers to repository [ghcr.io/brandond/nextcloud-influxdb-tracks-importer]
18a7ed190020: Mounted from windsource/nextcloud-influxdb-tracks-importer
f2dfa8b4b885: Mounted from windsource/nextcloud-influxdb-tracks-importer
fb7d69d86d94: Mounted from windsource/nextcloud-influxdb-tracks-importer
50644c29ef5a: Mounted from windsource/nextcloud-influxdb-tracks-importer
1.0.0: digest: sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0 size: 1155

On my k3s host:

brandond@seago:~$ sudo ctr image pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest: resolving      |--------------------------------------|
elapsed: 0.1 s                                                total:   0.0 B (0.0 B/s)
ctr: failed to resolve reference "ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

brandond@seago:~$ sudo ctr image pull ghcr.io/brandond/nextcloud-influxdb-tracks-importer:latest
ghcr.io/brandond/nextcloud-influxdb-tracks-importer:latest:                       resolved       |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3c96a94aa10c342b7ac8dc8f854d14dbabe57cabc5999dbf2796d5eeb942edc4:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:e8780064e542093a966d0a52204f730019c6b087029b53b1a111268815d296cc:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:58c8db755c6dc5035b49eaa57b4696e6074db57d623fd2568d3d02751b092781:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6dfe9eecf7268812efdbc6f9b16239dd7d4c1b6a74aaa74fc4a55c2bdc43d21e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c:    exists         |++++++++++++++++++++++++++++++++++++++|
elapsed: 2.8 s                                                                    total:  4.0 Mi (1.4 MiB/s)
unpacking linux/amd64 sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0...
done

[zlogic]

I'm having the exact same issue with GHCR repos in my account (https://github.com/users/zlogic/packages/container/package/nanorss-go and https://github.com/users/zlogic/packages/container/package/vogon-go). Both packages are built and pushed through Github Actions.

It seems like the issue is caused by changes on Github's side, because

• I've used the exact same k3s cluster (same k3s version) and the same ghcr.io registry, and it worked without any issues in September - k3s was able to pull images anonymously on September 29
• Nothing changed on my side since then - same k3s version, same registry
• The same 403 error appeared after I've pushed an updated tag to those repos and tried to pull from them again - about a week ago

[windsource]

My package is also built and pushed through Github Actions. But I am not sure if this is related to the problem. I also get the error message when I pull images from my Github account that are not created by Github actions, e.g.

$ docker pull nginx
$ docker tag nginx ghcr.io/windsource/nginx
$ docker push ghcr.io/windsource/nginx
# Goto github.com and make package public
$ sudo ctr image pull ghcr.io/windsource/nginx:latest
ghcr.io/windsource/nginx:latest: resolving      |--------------------------------------| 
elapsed: 0.2 s                   total:   0.0 B (0.0 B/s)                                         
ctr: failed to resolve reference "ghcr.io/windsource/nginx:latest": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

[clarkbw]

Not sure what's happening here. We're investigating further...

[markphelps]

👋🏻 Hi from the GH Container Registry team! We just pushed a fix to our service that should fix the issue that @windsource was seeing.

It has to do with how containerd requests multiple scopes when doing the initial auth request. It's related to this existing issue in containerd.

More specifically, GHCR wasnt handling the possibility of multiple scopes in the URL params correctly.

Verification

$ sudo ctr image pull ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest
ghcr.io/windsource/nextcloud-influxdb-tracks-importer:latest:                     resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:3c96a94aa10c342b7ac8dc8f854d14dbabe57cabc5999dbf2796d5eeb942edc4:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:e8780064e542093a966d0a52204f730019c6b087029b53b1a111268815d296cc:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:58c8db755c6dc5035b49eaa57b4696e6074db57d623fd2568d3d02751b092781:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6dfe9eecf7268812efdbc6f9b16239dd7d4c1b6a74aaa74fc4a55c2bdc43d21e:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:a9a6214e87593bac0f0345f04025799c14120855bfded59c35a3a7bf727b77b0...
done

[windsource]

Hi @markphelps and @clarkbw, thanks a lot! Now it works.