Distribution: failed to garbage collect: failed to mark: s3aws: Path not found: /docker/registry/v2/repositories

[paochiang]

registry image: registry:2

/etc/docker/registry/config.yml

version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  s3:
    accesskey: *******
    secretkey: *******
    region: us-east-2
    regionendpoint: https://161.189.205.192:31920
    bucket: registry
    encrypt: false
    secure: false
    skipverify: true
    v4auth: false
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
notifications:
  events:
    includereferences: true
  endpoints:
    - name: alistener
      disabled: false
      url: http://gemini-api:80/gemini_api/notifications/registry/event
      timeout: 2s
      threshold: 10
      backoff: 5s
      ignoredmediatypes:
        - application/octet-stream
      ignore:
        mediatypes:
          - application/octet-stream
        actions:
          - pull

command (in container):

registry garbage-collect /etc/docker/registry/config.yml

error log :

failed to garbage collect: failed to mark: s3aws: Path not found: /docker/registry/v2/repositories

Any help on how to fix it?
Thanks a lot

[thomasf]

I just bumped into this as well when moving to a s3 object storage.

The problem

It seems like s3aws.Walk() assumes that /docker/registry/v2/repositories is a file system directory. It's not a directory, it it's a path prefix and since there is no files there it seems like it can't read the situation properly.

A work around

After I put an empty dummy object named empty into the /docker/registry/v2/repositories of the s3 server (Linode object storage in my case) garbage collection finished successfully.

[paochiang]

I just bumped into this as well when moving to a s3 object storage.

The problem

It seems like s3aws.Walk() assumes that /docker/registry/v2/repositories is a file system directory. It's not a directory, it it's a path prefix and since there is no files there it seems like it can't read the situation properly.

A work around

After I put an empty object named empty into the /docker/registry/v2/repositories of the s3 server (Linode object storage in my case) garbage collection finished successfully.

It works.
Thank you @thomasf for contributing the solution.

[thomasf]

@paochiang It's still probably a bug even if there is a work around though, maybe best to keep this issue open

[Antauri]

In Harbor which uses the registry internally, when doing a GC you will see the message: "Failed to start GC: 500". Poking around, you'll probably get to the logs of "registryctl" which will show the "failed to mark: s3aws" error on its logs. You'll then find this ticket for which the workaround exists to put an empty file, which works on Digital Ocean Spaces. After putting a random file in /docker/registry/v2/repositories it was able to GC.

Crazy ...
Keep the issue open please, pending a fix. Upstreams (Harbor which use the registry) are affected.

[huww98]

I did some debugging. the issue originates from here:
https://github.com/docker/distribution/blob/9690d843fa153973c16f6c86fe9bce3f2421f9ce/registry/storage/driver/s3-aws/s3.go#L968-L977

This seems to be a third-party S3 implementation issue. In my case, I use Ceph Object Storage. the returned objects.KeyCount equals to len(objects.Contents). But according to S3 documentation, it should equal to len(objects.Contents) + len(objects.CommonPrefixes).

When enumerating manifeats, it sees no Contents, only CommonPrefixes, and objectCount remains to be 0, thus Path not found error.

Given there are a few incompatible implementations, maybe we should just use len(objects.Contents) + len(objects.CommonPrefixes) instead of KeyCount? I can confirm this works, and this should also result in cleaner code.