Mybinder.org-deploy: [OVH] Image layer needs removing

[betatim]

There is another case like #1016 here: https://discourse.jupyter.org/t/i-get-read-timed-out-errors-on-when-pushing-image-ovh/1394

Could you remove the objects from SWIFT again for us @mael-le-gal? Is there a way we can organise for more people to be able to delete things from SWIFT?

[mael-le-gal]

This time it seems to be a different error because it happens during the push and not on the pull.

When the error was on the pull we could see this error in the docker registry log :

swift: Timeout expired while waiting for segments of /docker/registry/v2/blobs/sha256/09/0931d59cf7ecec93048b810715e19591c09e88ec23efb183c1ee916a9aebcd27/data to show up

For deleting files on swift you can install the python swift client :

pip install python-swiftclient

Then you can find the file in swift with the following command :

swift --os-auth-url https://auth.cloud.ovh.net/v2.0 --os-username $USERNAME --os-password $PASSWORD --os-tenant-id $TENANT_ID  --os-region-name GRA5 list binder-registry | grep /docker/registry/v2/blobs/sha256/09/0931d59cf7ecec93048b810715e19591c09e88ec23efb183c1ee916a9aebcd27/data

And then delete the file with the following command :

swift --os-auth-url https://auth.cloud.ovh.net/v2.0 --os-username $USERNAME --os-password $PASSWORD --os-tenant-id $TENANT_ID  --os-region-name GRA5 delete binder-registry file/docker/registry/v2/blobs/sha256/09/0931d59cf7ecec93048b810715e19591c09e88ec23efb183c1ee916a9aebcd27/data

When you have k8s access, you can have the $USERNAME with :

 kubectl get secret binder-registry-docker-registry-secret -n ovh -o json | jq '.data.swiftUsername' -r | base64 --decode

you can have the $PASSWORD with :

 kubectl get secret binder-registry-docker-registry-secret -n ovh -o json | jq '.data.swiftPassword' -r | base64 --decode

you can have the $TENANT_ID with :

kubectl get pod binder-registry-docker-registry-57754485b5-889j7 -n ovh -o json | jq '.spec.containers[0].env[] | select(.name=="REGISTRY_STORAGE_SWIFT_TENANTID")' | jq '.value' -r

[mael-le-gal]

I'm on it to try figuring out what's happening

[mael-le-gal]

@betatim, after some research I think that this problem has no links with the docker registry this time. I don't see any error in the registry log about this images.

I find similar error on stackoverflow topics and it seems to be more related to docker client.

Pushing image
Pushing image
Error during build: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

My question is : in the binder stack what are de pods responsible for the docker push command ? Is it dind ?

[mael-le-gal]

I tried it again now and the problem seems to have disappear. So it seems to be randomly happening.

I guess that this error appear only when the build occur on one node of our cluster (I was unable to figure out which one for now).

As dind use the underlying docker daemon of the host, I think the problem is coming from the docker client of that specific hosts.

This is only a guess, it's hard to validate it now that the error disappeared

[mael-le-gal]

For the issue related to the docker registry, I choose to increase the allocated ressources for that deployment. Maybe that's a reason why some layers where pushed corrupted.

Your docker registry pod now takes 2 vcores and 10Go RAM at least. We will see if we still have this recurrent problem after that.

In case that doesn't solve anything, we are still waiting for the official realease of the registry as a service in OVH. I talked with the related team today and it should be available very soon now.

[betatim]

I've increased the traffic again, let's see what happens. Closing this issue for now as there isn't anything to do :)