Mybinder.org-deploy: Removing a image from the OVH registry

[betatim]

For some repos (example) pulling images from the OVH registry fails with "filesystem layer verification failed for digest":

  Normal   Created    1m                kubelet, node-a031e240-a864-4a1a-87ff-55bdd6ec7cb6  Created container
  Normal   Pulling    57s (x2 over 1m)  kubelet, node-a031e240-a864-4a1a-87ff-55bdd6ec7cb6  pulling image "binder-registry.mybinder.ovh/binder-ovh/r2d-f18835fd-noamross-2dgams-2din-2dr-2dcourse-1fc161:ac4c2572f553e1365e2d3eb16fe8db548fe689f1"
  Warning  Failed     14s (x2 over 1m)  kubelet, node-a031e240-a864-4a1a-87ff-55bdd6ec7cb6  Failed to pull image "binder-registry.mybinder.ovh/binder-ovh/r2d-f18835fd-noamross-2dgams-2din-2dr-2dcourse-1fc161:ac4c2572f553e1365e2d3eb16fe8db548fe689f1": rpc error: code = Unknown desc = filesystem layer verification failed for digest sha256:bccc34851423dc8ff2c6044209eccfd863c8cb6284ef94bdc5cb218ba69f9306
  Warning  Failed     14s (x2 over 1m)  kubelet, node-a031e240-a864-4a1a-87ff-55bdd6ec7cb6  Error: ErrImagePull
  Normal   BackOff    0s (x2 over 1m)   kubelet, node-a031e240-a864-4a1a-87ff-55bdd6ec7cb6  Back-off pulling image "binder-registry.mybinder.ovh/binder-ovh/r2d-f18835fd-noamross-2dgams-2din-2dr-2dcourse-1fc161:ac4c2572f553e1365e2d3eb16fe8db548fe689f1"
  Warning  Failed     0s (x2 over 1m)   kubelet, node-a031e240-a864-4a1a-87ff-55bdd6ec7cb6  Error: ImagePullBackOff

Is there a way for us to delete this image from the registry and see what happens when it is rebuilt?

I don't think this is a one-off problem (one user getting unlucky) with transferring the image but instead something is wrong with the registry/image as all people launching this repo get this problem.

A next step is to find out why the image got corrupted.

cc @mael-le-gal

[betatim]

This is another repo for which this happens https://ovh.mybinder.org/v2/gh/DS-100/textbook/master

[sgibson91]

What I'm about to type is from my Azure experience, so may or may not be relevant.

Specific permissions have to be allocated in order to delete images from a private repo, in the same way that permissions are needed to push to a private repo (or pull from). In the Azure infrastructure, Users who have permissions are Owners or Contributors on the subscription (read project, I'm guessing for Google Cloud translation) and a specific "delete" permission is given to resources to automagically do this. The "delete" permission is entirely separate to the push or pull permission. See these docs for details.

My guess would be (providing OVH works in a similar way to Azure) either someone at OVH will have to delete that image manually (also guessing that you don't have access) or assign a delete permission to a resource in order to clean out the registry.

I may also be entirely off the mark :)

[betatim]

I looked into this a bit. We can/have to use the docker registry API to delete the image. When you do that the returned status code is 405 The operation is unsupported.. I think this is because deleting images has to be enabled.

@mael-le-gal could you take a look at that? I think the registry is controlled from somewhere we don't have access to.

[betatim]

As temporary cheat I edited the configmap that holds the registry config to add this. After that the following lines of Python did the job of deleting the image(s):

import requests
from tornado.httputil import url_concat

image ='binder-ovh/r2d-f18835fd-ds-2d100-2dtextbook-089fa5'
tag = '725460fe030050c0effc029c334337f8a1477335'
url = 'https://binder-registry.mybinder.ovh'
token_url = 'https://auth-binder-registry.mybinder.ovh/auth?service=binder-registry'

user ='binder'
pw = 'look me up in the secrets'

# get a token that can read and write
r = requests.get(url_concat(token_url, {"scope": "repository:{}:*".format(image)}), auth=(user, pw))
token = r.json()['token']

r = requests.head("{}/v2/{}/manifests/{}".format(url, image, tag), headers={"Authorization": "Bearer {}".format(token), 'Accept': 'application/vnd.docker.distribution.manifest.v2+json'})

digest = r.headers['Docker-Content-Digest']

# this should return a 202
r = requests.delete("{}/v2/{}/manifests/{}".format(url, image, digest), headers={"Authorization": "Bearer {}".format(token)})

[mael-le-gal]

I added that :

storage:
  delete:
    enabled: true

in our configuration deployment for the binder registry

Is that the only thing you change in the configmap ? Does that solve your issue ?

[betatim]

That is the only thing I changed and indeed let me delete the images. Thanks for adding it. Should we add the helm chart(?) that deploys the registry to this repo or do you think it isn't worth the effort once we debugged it?

There is one remaining problem with https://ovh.mybinder.org/v2/gh/noamross/gams-in-r-course/master It builds and during the push to the registry there is one layer that times out in SWIFT (I think). This then leads to a status 500 being returned to the docker client. End result for the user: the build failed. I have no idea how to fix that or even where to start looking. There are a few GitHub issues I found from a few years back about exactly this issue but the conclusion in them seems to be "don't use swift" or "somehow fixed itself". Neither of them very useful :)

Maybe you can find someone at OVH who knows a bit more about running docker registries backed by swift. I know nothing about this :-/

[mael-le-gal]

Should we add the helm chart(?) that deploys the registry to this repo or do you think it isn't worth the effort once we debugged it?

Indeed that could also be added on the deployment. However, that was our quickest way to provide you with a custom docker registry and if you say that you have problem with swift, maybe a better solution would be to provide you a more standard docker registry.

I'm going to look at that and I let you know.

[mael-le-gal]

Ok, the problem should be solved for the image of this repository : noamross/gams-in-r-course/master, are you aware of similar problems with other images ?

Here is what I did to solve the issue :

• I checked the logs of the docker-registry pod and I found the following error :

swift: Timeout expired while waiting for segments of /docker/registry/v2/blobs/sha256/bc/bccc34851423dc8ff2c6044209eccfd863c8cb6284ef94bdc5cb218ba69f9306/data to show up

• I tried to download the same object /docker/registry/v2/blobs/sha256/bc/bccc34851423dc8ff2c6044209eccfd863c8cb6284ef94bdc5cb218ba69f9306/data directly from swift => It seems to be empty whereas it should contains data.
• I deleted that object from swift
• I tried again to spawn the noamross/gams-in-r-course/master notebook on binder => it works

So what happened ?
Hard to say how we get into that situation but it seems that our docker registry pushed an incomplete or corrupt data file once, and it was then unable to push it again without deleting the initial one.

So what next ?
The good point is that we now have a workarround solution in case the problem shows up again.
The bad point is that it seems to be a randomly happening error.

As I say earlier, the solution to deploy a docker registry over swift was for me the quickest way to provide you a dedicated registry. But there is another team in OVH who is working on a registry as a service and it should be released soon. So I think that it will be a good idea to migrate on their service when we will have the chance.

[betatim]

This is the only image I know of that had this problem.

Thanks for fixing this and the nice write up!

Let's keep using the registry-with-swift for now and then swap to the registry as a service.

I'll close this issue because I think it is done if not then please reopen it.