Cli: Docker manifest won't authenticate if TLS is not terminated by registry

[agronholm]

Description

I have a setup where nginx is sitting as the front-end web server and is reverse proxying for Docker registry. Apparently this kind of setup confuses the docker CLI to not send the authentication headers when creating a manifest list on the registry, because the registry server is oblivious about TLS.

Steps to reproduce the issue:

1. Set up nginx (with TLS) and a docker registry behind it
2. Push an image (registry.address/image_name) to the registry
3. Run docker manifest create registry.address/manifest_name registry.address/image_name

Describe the results you received:

no such manifest: registry.address/image_name
Pulling registry.address/image_name works fine however!

Describe the results you expected:

The manifest list being created on the registry.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:   18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:10:01 2018
 OS/Arch:   linux/amd64
 Experimental:  true
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:    Wed Mar 21 23:08:31 2018
  OS/Arch:  linux/amd64
  Experimental: true

Output of docker info:

Containers: 10
 Running: 10
 Paused: 0
 Stopped: 0
Images: 472
Server Version: 18.03.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 414
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-119-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.795GiB
Name: devib
ID: YWI7:LAUS:45DD:ZOTC:IVL6:N4DK:CBFR:QJ5I:GWDC:TFCE:JMGH:BOMQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

This is on an Azure VM.
Nginx configuration for the registry:

server {
    listen 443 ssl http2;
    server_name redacted.example.org;
    access_log /var/log/nginx/registry-access.log;
    error_log /var/log/nginx/registry-error.log notice;

    # Allow large enough uploads
    client_max_body_size 0;

    location / {
        proxy_pass http://localhost:5000;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        auth_basic "Registry realm";
        auth_basic_user_file /etc/docker-registry-htpasswd;
    }
}

[clnperez]

Feel free to assign this one to me. I've been talking to @agronholm about it and can take a look at it now.

[n4ss]

@agronholm Can you please share the configuration of the registry?

[n4ss]

@clnperez let me know if I can help ^_^

[agronholm]

@n4ss I have not touched the configuration that comes with the registry image. My Ansible task was defined as follows:

    - name: create the registry container
      docker_container:
        image: registry:2
        name: registry
        pull: true
        state: started
        restart_policy: always
        user: 1000:1000
        published_ports:
          - 127.0.0.1:5000:5000
        volumes:
          - /var/local/docker-registry:/var/lib/registry
          - /etc/ssl/certs/redacted.crt:/certs/domain.crt:ro
          - /etc/ssl/private/redacted.key:/certs/domain.key:ro
          - /etc/docker-registry-htpasswd:/auth/htpasswd:ro
        log_driver: syslog
        log_options: tag=registry
        env:
          DOCKER_OPTS: --insecure-registry {{ registry.url }}

Several of those options were probably left over from various attempts to get the registry working, but I don't think they're a problem.

[clnperez]

I set up an nginx proxy and basic htpasswd auth using the howoto @ https://github.com/docker/docker.github.io/blob/master/registry/recipes/nginx.md (after i tried doing it locally and running into the bcrypt issue, which was, as this guy described, a time sink: https://github.com/docker/distribution/issues/655#issuecomment-143500360 :laughing: ). Using the setup from the registry repo, the manifest create succeeded.

Can you try tweaking your nginx config based on the doc link I pasted? There's an interesting discussion linked in that doc as well, but it also complicates things because that scenario has nginx and the registry behind Amazon's ELB. I'm not sure if the Azure setup might also be updating headers in a similar fashion.

[agronholm]

Yes, with the addition of one header it seems to work, thank you! Now the big question: why do all other commands besides manifest work without this?

[clnperez]

Thanks for trying that out. That's my question now as well.

@n4ss Do you happen to know how the docker push flow works, for example? Does it just always include the auth header no matter what?

[clnperez]

Oh, @agronholm, which header did you not have in the config? I'd like to recreate what you were doing exactly.

[agronholm]

It was this line that made all the difference (along with the corresponding map directive):
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

[clnperez]

Ping @n4ss. Any idea or another person to ask?

I'm a little swamped, otherwise I would dig around a bit more myself but since we've got a workaround I'm less inclined to lose sleep over this at the moment. :D

[denefoster]

Having the same issue with a registry behind traefik, image pull/push works fine manifest create results in:

no such manifest: xxx/docker-light-baseimage:amd64

Debug output shows the response being auth required:
DEBU[0000] hostDir: /etc/docker/certs.d/xxx DEBU[0000] endpoints for xxx/docker-light-baseimage:amd64: [{false https://xxx v2 false false true 0xc0004d8a80}] DEBU[0002] continuing on error (errcode.Error) unauthorized: authentication required no such manifest: xxx/docker-light-baseimage:amd64

Happy to provide any configs to help