Docker-gitlab: docker registry 404

Created on 28 Aug 2016 · 12Comments · Source: sameersbn/docker-gitlab

I try to get the registry working but every time i get 404 errors.

docker login registry.gitlab.example.com
Username: abc 
Password: 
Login Succeeded
docker push registry.gitlab.example.com/gimler/docker-php

result

The push refers to a repository [registry.gitlab.example.com/gimler/docker-php]
c56b7dabbc7a: Preparing 
Error: Status 404 trying to push repository gimler/docker-php: "<!DOCTYPE html>\n<html>\n<head>\n  <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n  <title>The page you're looking for could not be found (404)</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: normal;\n      color: #456;\n    }\n\n    h2 {\n      font-size: 24px;\n      color: #666;\n      line-height: 1.5em;\n    }\n\n    h3 {\n      color: #456;\n      font-size: 20px;\n      font-weight: normal;\n      line-height: 28px;\n    }\n\n    hr {\n      max-width: 800px;\n      margin: 18px auto;\n      border: 0;\n      border-top: 1px solid #EEE;\n      border-bottom: 1px solid white;\n    }\n\n    img {\n      max-width: 40vw;\n    }\n\n    .container {\n      margin: auto 20px;\n    }\n  </style>\n</head>\n\n<body>\n  <h1>\n    <img src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=\" alt=\"GitLab Logo\" /><br />\n    404\n  </h1>\n  <div class=\"container\">\n    <h3>The page you're looking for could not be found.</h3>\n    <hr />\n    <p>Make sure the address is correct and that the page hasn't moved.</p>\n    <p>Please contact your GitLab administrator if you think this is a mistake.</p>\n  </div>\n</body>\n</html>\n"

docker-compose.yml

postgresql:
  restart: always
  image: sameersbn/postgresql:9.5-1
  environment:
    - DB_USER=root
    - DB_PASS=***
    - DB_NAME=gitlabhq
    - DB_EXTENSION=pg_trgm
  volumes:
    - "$PWD/data/postgresql:/var/lib/postgresql"

gitlab:
  restart: always
  image: sameersbn/gitlab:8.11.2-1
  links:
    - redis:redisio
    - postgresql:postgresql
    - registry:registry
  expose:
    - "80"
  ports:
    - "10022:22"
  environment:
    - VIRTUAL_HOST=gitlab.example.com,registry.gitlab.example.com
    - LETSENCRYPT_HOST=gitlab.example.com,registry.gitlab.example.com
    - [email protected]

    - DEBUG=true
    - TZ=Europe/Berlin
    - GITLAB_TIMEZONE=Berlin

    - GITLAB_SECRETS_DB_KEY_BASE=***
    - GITLAB_SECRETS_SECRET_KEY_BASE=***
    - GITLAB_SECRETS_OTP_KEY_BASE=***

    - GITLAB_HOST=gitlab.example.com
    - GITLAB_PORT=443
    - GITLAB_HTTPS=true
    - GITLAB_SSH_PORT=10022
    - GITLAB_RELATIVE_URL_ROOT=

    - GITLAB_REGISTRY_ENABLED=true
    - GITLAB_REGISTRY_HOST=registry.gitlab.example.com
    - GITLAB_REGISTRY_PORT=443
    - GITLAB_REGISTRY_API_URL=http://registry:5000
    - GITLAB_REGISTRY_KEY_PATH=/certs/registry-auth.key

    - GITLAB_BACKUP_SCHEDULE=daily
    - GITLAB_BACKUP_TIME=01:00
  volumes:
    - "$PWD/data/gitlab:/home/git/data"
    - "$PWD/certs:/certs"

redis:
  restart: always
  image: sameersbn/redis:latest
  volumes:
    - "$PWD/data/redis:/var/lib/redis"

gitlab-runner:
  restart: always
  image: gitlab/gitlab-runner:alpine-v1.5.2
  links:
    - gitlab:gitlab
  volumes:
    - "$PWD/config/gitlab-runner:/etc/gitlab-runner"
    - "/var/run/docker.sock:/var/run/docker.sock"

registry:
    restart: always
    image: registry:2.5.0
    volumes:
      - "$PWD/data/registry:/registry"
      - "$PWD/certs:/certs"
    environment:
      - REGISTRY_LOG_LEVEL=debug
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
      - REGISTRY_AUTH_TOKEN_REALM=https://gitlab.example.com/jwt/auth
      - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
      - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
      - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry-auth.crt
      - REGISTRY_STORAGE_DELETE_ENABLED=true

Source

gimler

👍5

Most helpful comment

I use https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion as letsencrypt proxy

I have found solution to this issue. I am also using "proxy-companion" for letsencrypt certificates and jwilder/nginx-proxy as docker reverse proxy server.

Thing that did the trick is registry nginx port. I have changed port to 80 since all incoming traffic is going to VIRTUAL_PORT=80. As we are hitting our reverse proxy on port 443 traffic is forwarded to VIRTUAL_HOST port and that is 80 of gitlab container. Then we need our registry nginx to listen on port 80 and it should work fine...

So my current config looks like this..

registry_external_url 'https://docker.my-domain.com'

gitlab_rails['registry_enabled'] = true

registry['enable'] = true

registry_nginx['enable'] = true
registry_nginx['listen_https'] = false
registry_nginx['listen_port'] = 80
registry_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

Next problem that you will face is limitation of jwilder/nginx-proxy is the client_max_body_size limit.
You have to increase the body size limit in order to upload docker images to your registry over reverse proxy.
You can do that by creating custom jwilder/nginx-proxy image with client_max_body_size 0; in nginx.conf.
Or you can mount additional proxy config file using -v /path/to/my_proxy.conf:/etc/nginx/conf.d/my_proxy.conf:ro. You can find more information on this here.

After that you should be able to use gitlab docker registry without problems.

nikolicjakov on 27 Sep 2018

👍4 🚀1

All 12 comments

yep, same here. note that i run this behind traefik (external load-balancer) and thus i did not set SSL_REGISTRY_{KEY,CERT}_PATH - see https://github.com/sameersbn/docker-gitlab/pull/714#issuecomment-226882702

    # https://github.com/sameersbn/docker-gitlab/pull/714#issuecomment-226882702
    # "My current implementation will now work like the normal ssl proxy for gitlab if you didn't provide a cert and key for SSL_REGISTRY_CERT & SSL_REGISTRY_KEY"
    # - SSL_REGISTRY_KEY_PATH=/certs/registry.key
    # - SSL_REGISTRY_CERT_PATH=/certs/registry.crt

guybrush on 29 Aug 2016

I use https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion as letsencrypt proxy

gimler on 29 Aug 2016

Hey,

if you use an external registry api change your url to GITLAB_REGISTRY_API_URL=https://registry.gitlab.example.com.

solidnerd on 29 Aug 2016

ok. i get it to work but only on port 5500 set - GITLAB_REGISTRY_PORT=5500 and use SSL_REGISTRY_KEY_PATH and SSL_REGISTRY_CERT_PATH.

Now it works in gitlab-ci.yml but not from external.

docker push registry.gitlab.example.com:5500/gimler/docker-php

return

error parsing HTTP 403 response body: no error details found in HTTP response body: "{\"message\":\"forbidden\",\"status\":\"error\",\"http_status\":403}"

gimler on 29 Aug 2016

I use my self an external registry and I don't have these problems. Run the container registry with a minimal configuration env's. Have a look a this example:

GITLAB_REGISTRY_ENABLED=true
GITLAB_REGISTRY_HOST=registry.gitlab.example.com
GITLAB_REGISTRY_API_URL=https://registry.gitlab.example.com
GITLAB_REGISTRY_KEY_PATH=/certs/registry_auth.key
GITLAB_REGISTRY_ISSUER=gitlab-issuer

REGISTRY_LOG_LEVEL=info
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
REGISTRY_AUTH_TOKEN_REALM=https://gitlab.example.com/jwt/auth
REGISTRY_AUTH_TOKEN_SERVICE=container_registry
REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry_auth.crt
REGISTRY_STORAGE_DELETE_ENABLED=true

solidnerd on 29 Aug 2016

❤1

Perhaps your proxy configuration is different.

## Set a variable to help us decide if we need to add the
## 'Docker-Distribution-Api-Version' header.
## The registry always sets this header.
## In the case of nginx performing auth, the header will be unset
## since nginx is auth-ing before proxying.
map $upstream_http_docker_distribution_api_version $docker_distribution_api_version {
  'registry/2.0' '';
  default registry/2.0;
}

server {
  listen *:80;
  server_name  registry.gitlab.example.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice
  return 301 https://$http_host:$request_uri;
  access_log  /var/log/nginx/registry.gitlab.example.com/access.log;
  error_log   /var/log/nginx/registry.gitlab.example.com/error.log;
}

server {
  listen *:443 ssl http2;
  server_name registry.gitlab.example.com;

  #Logging
  access_log  /var/log/nginx/registry.gitlab.example.com/access.log;
  error_log   /var/log/nginx/registry.gitlab.example.com/error.log;

  #SSL
  ssl_certificate /etc/letsencrypt/live/registry.gitlab.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/registry.gitlab.example.com/privkey.pem;

  # Recommendations from https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
  ssl_protocols TLSv1.1 TLSv1.2;
  ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  # disable any limits to avoid HTTP 413 for large image uploads
  client_max_body_size 0;

  # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
  chunked_transfer_encoding on;

  location / {
    # Do not allow connections from docker 1.5 and earlier
    # docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
    if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*\$" ) {
      return 404;
    }

    ## If $docker_distribution_api_version is empty, the header will not be added.
    ## See the map directive above where this variable is defined.
    add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;

    proxy_pass                          http://gitlab_registry_1:5000;
    proxy_set_header  Host              $http_host;   # required for docker client's sake
    proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
    proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_read_timeout                  900;
  }
}

solidnerd on 29 Aug 2016

I use it as a external registry now all works ;) thanks @SolidNerd

gimler on 30 Aug 2016

yep, using the registry externally works - thanks!

guybrush on 30 Aug 2016

@gimler hi! I'm confusing, I have tried to find the file to place external registry. In documentation they mention gitlab.rb but I don´t find. my gitlab is sameersbn 8.16.5 and the registry is 2.4.1.
The error that throws me is :"Error: Status 404 trying to push repository group/name: "

ghost on 23 Feb 2017

@FranVeraC Did you find a solution? I have the exact same error

julienTocci on 14 Mar 2018

basically my file was that i have tried to use http://registry:5000 as GITLAB_REGISTRY_API_URL change it to the external url https://registry.gitlab.example.com without the port number fix this form me.

see comment https://github.com/sameersbn/docker-gitlab/issues/876#issuecomment-243050517

gimler on 14 Mar 2018

I use https://github.com/JrCs/docker-letsencrypt-nginx-proxy-companion as letsencrypt proxy

I have found solution to this issue. I am also using "proxy-companion" for letsencrypt certificates and jwilder/nginx-proxy as docker reverse proxy server.

So my current config looks like this..

registry_external_url 'https://docker.my-domain.com'

gitlab_rails['registry_enabled'] = true

registry['enable'] = true

registry_nginx['enable'] = true
registry_nginx['listen_https'] = false
registry_nginx['listen_port'] = 80
registry_nginx['proxy_set_headers'] = {
  "Host" => "$http_host",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

After that you should be able to use gitlab docker registry without problems.

nikolicjakov on 27 Sep 2018

👍4 🚀1

Was this page helpful?

0 / 5 - 0 ratings

Related issues

docker registry log warns about redis

rakesh-k · 3Comments

fatal: git fetch-pack: expected shallow

cyklodev · 3Comments

Upgrade to 13.7 which also upgrades to postgres 12 requires DB migration instructions

tomastzn · 4Comments

Unlinking stale socket /var/run/supervisor.sock cp: error writing '/home/git/gitlab-shell/config.yml': No space left on device cp: failed to extend '/home/git/gitlab-shell/config.yml': No space left on device

ImmanuelFredrick · 3Comments

8.13 has been released

lenovouser · 3Comments