Docker-gitlab: "getsockopt: connection refused" when execute "docker login" on my own registry

[colas-pomies]

Hi,

First of all : thank for all your great work

I use your gitlab installation and now I try to add the registry. After depoying all containers I create a docker gitlab-runner and I configure the .gitlab-ci.yml to execute this command before script :
docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.mysite.fr:5500

When doing this, I got the error :
Error response from daemon: Get https://registry.gitlab.mysite.fr:5500/v1/users/: dial tcp XX.XX.XX.XX:5500: getsockopt: connection refused

Do you have any idea to solve this problems. I use this docker compose :

version: '2'

services:
  redis:
    restart: always

    image: sameersbn/redis:latest

    command:
    - --loglevel warning

    volumes:
    - /srv/docker/gitlab/redis:/var/lib/redis:Z

  postgresql:
    restart: always

    image: sameersbn/postgresql:9.5-1

    volumes:
    - /srv/docker/gitlab/postgresql:/var/lib/postgresql:Z

    environment:
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production
    - DB_EXTENSION=pg_trgm

  gitlab:
    restart: always

    image: sameersbn/gitlab:8.11.5

    depends_on:
    - redis
    - postgresql

    ports:
    - "80:80"
    - "5500:5500"
    - "10022:22"

    volumes:
    - /srv/docker/gitlab/gitlab/data:/home/git/data:Z
    - /srv/docker/gitlab/gitlab/logs:/var/log/gitlab
    - /srv/docker/gitlab/gitlab_registry/certs:/certs

    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlabhq_production

    - REDIS_HOST=redis
    - REDIS_PORT=6379

    - TZ=Europe/Paris
    - GITLAB_TIMEZONE=Paris

    - GITLAB_HTTPS=false
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=gitlab.mysite.fr
    - GITLAB_PORT=80
    - GITLAB_SSH_PORT=10022
    - GITLAB_RELATIVE_URL_ROOT=
    - GITLAB_SECRETS_DB_KEY_BASE=d6D7bcMVVmpkdQrzFF96kX3ffqXDVKhklf4bQHxssZstqcBBFqfnzG2N96jj7VqS
    - GITLAB_SECRETS_SECRET_KEY_BASE=MRRfSR5xdJVQQqbgN8r4vdZZw7fdw56Kh2R5Vgj9XJRf6PLHbw9mx42LBvCBFShr
    - GITLAB_SECRETS_OTP_KEY_BASE=rtzpnc8zfjLjMSflwDFnmWJPPBxgXx2fBx3sznLbKFW9ZJQV59dbvZRPSM6MpMd5

    - GITLAB_ROOT_PASSWORD=
    - GITLAB_ROOT_EMAIL=

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
    - GITLAB_NOTIFY_PUSHER=false

    - [email protected]
    - [email protected]
    - [email protected]

    - GITLAB_BACKUP_SCHEDULE=daily
    - GITLAB_BACKUP_TIME=01:00

    - SMTP_ENABLED=false
    - SMTP_DOMAIN=www.example.com
    - SMTP_HOST=smtp.gmail.com
    - SMTP_PORT=587
    - [email protected]
    - SMTP_PASS=password
    - SMTP_STARTTLS=true
    - SMTP_AUTHENTICATION=login

    - IMAP_ENABLED=false
    - IMAP_HOST=imap.gmail.com
    - IMAP_PORT=993
    - [email protected]
    - IMAP_PASS=password
    - IMAP_SSL=true
    - IMAP_STARTTLS=false

    - OAUTH_ENABLED=false
    - OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=
    - OAUTH_ALLOW_SSO=
    - OAUTH_BLOCK_AUTO_CREATED_USERS=true
    - OAUTH_AUTO_LINK_LDAP_USER=false
    - OAUTH_AUTO_LINK_SAML_USER=false
    - OAUTH_EXTERNAL_PROVIDERS=

    - OAUTH_CAS3_LABEL=cas3
    - OAUTH_CAS3_SERVER=
    - OAUTH_CAS3_DISABLE_SSL_VERIFICATION=false
    - OAUTH_CAS3_LOGIN_URL=/cas/login
    - OAUTH_CAS3_VALIDATE_URL=/cas/p3/serviceValidate
    - OAUTH_CAS3_LOGOUT_URL=/cas/logout

    - OAUTH_GOOGLE_API_KEY=
    - OAUTH_GOOGLE_APP_SECRET=
    - OAUTH_GOOGLE_RESTRICT_DOMAIN=

    - OAUTH_FACEBOOK_API_KEY=
    - OAUTH_FACEBOOK_APP_SECRET=

    - OAUTH_TWITTER_API_KEY=
    - OAUTH_TWITTER_APP_SECRET=

    - OAUTH_GITHUB_API_KEY=
    - OAUTH_GITHUB_APP_SECRET=
    - OAUTH_GITHUB_URL=
    - OAUTH_GITHUB_VERIFY_SSL=

    - OAUTH_GITLAB_API_KEY=
    - OAUTH_GITLAB_APP_SECRET=

    - OAUTH_BITBUCKET_API_KEY=
    - OAUTH_BITBUCKET_APP_SECRET=

    - OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=
    - OAUTH_SAML_IDP_CERT_FINGERPRINT=
    - OAUTH_SAML_IDP_SSO_TARGET_URL=
    - OAUTH_SAML_ISSUER=
    - OAUTH_SAML_LABEL="Our SAML Provider"
    - OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:transient
    - OAUTH_SAML_GROUPS_ATTRIBUTE=
    - OAUTH_SAML_EXTERNAL_GROUPS=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=
    - OAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=

    - OAUTH_CROWD_SERVER_URL=
    - OAUTH_CROWD_APP_NAME=
    - OAUTH_CROWD_APP_PASSWORD=

    - OAUTH_AUTH0_CLIENT_ID=
    - OAUTH_AUTH0_CLIENT_SECRET=
    - OAUTH_AUTH0_DOMAIN=

    - OAUTH_AZURE_API_KEY=
    - OAUTH_AZURE_API_SECRET=
    - OAUTH_AZURE_TENANT_ID=

    - GITLAB_REGISTRY_ENABLED=true
    - GITLAB_REGISTRY_HOST=registry.gitlab.mysite.fr
    - GITLAB_REGISTRY_PORT=5500
    - GITLAB_REGISTRY_API_URL=http://registry:5000
    - GITLAB_REGISTRY_KEY_PATH=/certs/registry-auth.key
    - GITLAB_REGISTRY_ISSUER=gitlab-issuer

  registry:
    restart: always

    image: registry:2.4.1

    volumes:
    - /srv/docker/gitlab/gitlab/data/shared/registry:/registry
    - /srv/docker/gitlab/gitlab_registry/certs:/certs

    environment:
    - REGISTRY_LOG_LEVEL=info
    - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/registry
    - REGISTRY_AUTH_TOKEN_REALM=http://gitlab.mysite.fr/jwt/auth
    - REGISTRY_AUTH_TOKEN_SERVICE=container_registry
    - REGISTRY_AUTH_TOKEN_ISSUER=gitlab-issuer
    - REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/certs/registry-auth.crt
    - REGISTRY_STORAGE_DELETE_ENABLED=true

I also generate sel signed certificate for registry like this :

mkdir -p /srv/docker/gitlab/gitlab_registry/certs
cd /srv/docker/gitlab/gitlab_registry/certs
openssl req -nodes -newkey rsa:4096 -keyout registry-auth.key -out registry-auth.csr -subj "/CN=gitlab-issuer"
openssl x509 -in registry-auth.csr -out registry-auth.crt -req -signkey registry-auth.key -days 3650

I hope you have some idea to solve the problem.

Thanks

[panoramic-machine]

I have the exact same problem. My setting are nearly identical to yours.

[maikelvl]

Maybe you made a typo: registry.gitlab.colaspomies.fr:5500 or registry.gitlab.mysite.fr:5500?

[colas-pomies]

... Sorry mistake when I write the issue. I wrote mysite instead of colaspomies. I just forget to replace it when I wrote the issue. I change it.

[panoramic-machine]

In my case I am positive that I am typing it correctly.

[colas-pomies]

Me too I am typing correctly in my configuration !

[maikelvl]

Try set the GITLAB_REGISTRY_PORT to 5000 (I think you could remove the 5500:5500 from gitlab)

[maikelvl]

Also you should connect to port 5000 (the registry itself) docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.mysite.fr:5000

The registry will then use GitLab to authenticate.

[colas-pomies]

Okay so I try different case and for each I reinstall all :

• stop & rm all docker
• rm /srv/docker/gitlab && rm /srv/docker/gitlab-runner
• recreate ssl certificate
• execute docker-compose
• reinstall gitlab-runner

1 - set GITLAB_REGISTRY_PORT to 5000
2 - connect direct to port 5000 with docker login
3 - set GITLAB_REGISTRY_PORT to 5000 && connect direct to port 5000 with docker login

And I've got the same error :

Error response from daemon: Get https://registry.gitlab.mysite.fr:5500/v1/users/: dial tcp XX.XX.XX.XX:5500: getsockopt: connection refused

[maikelvl]

Mmm... That's strange because 5500 shouldn't be defined anywhere.

[colas-pomies]

I applied (or I try to apply) the method defined here for the registry

[colas-pomies]

If you want more information this is ALL commands I execute :

• Before all, in my domain name page configuration I set :

gitlab.mysite.fr -> my server IP
registry.gitlab.mysite.fr -> my server IP

• Stop all my docker and delete them

docker stop gitlab-runner mysite_gitlab_1 mysite_registry_1 mysite_postgresql_1 mysite_redis_1;
docker rm gitlab-runner mysite_gitlab_1 mysite_registry_1 mysite_postgresql_1 mysite_redis_1;

• Clean the persistant directory for gitlab and create certificate (with root user)

rm -r /srv/docker/gitlab;
mkdir -p /srv/docker/gitlab/gitlab_registry/certs;
cd /srv/docker/gitlab/gitlab_registry/certs;
openssl req -nodes -newkey rsa:4096 -keyout registry-auth.key -out registry-auth.csr -subj "/CN=gitlab-issuer";
openssl x509 -in registry-auth.csr -out registry-auth.crt -req -signkey registry-auth.key -days 3650;

• Execute docker-compose with this this docker-compose.yml (same as define upper in the issue) : docker-compose.yml
  docker-compose up -d
• Waiting until gitlab is avaible at gitlab.mysite.fr (~5min)
• Connect to gitlab.mysite.fr and do :
  
  
  
  • Set root mdp
  
  
  • Create new user (myuser) and connect with it
  
  
  • Create new project : my-other-site.fr
  
  
• Clean the persistant directory for gitlab-runner (with user root)
  rm -r /srv/docker/gitlab-runner
• Create gitlab-runner with docker

docker run -d --name gitlab-runner --restart always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /srv/docker/gitlab-runner/config:/etc/gitlab-runner \
  gitlab/gitlab-runner:latest

• Register the gitlab-runner to my gitlab (with the url and token get from the runner part of gitlab)

docker exec -it gitlab-runner gitlab-ci-multi-runner register -n \
  --url http://gitlab.mysite.fr/ci \
  --registration-token TOKEN_FROM_GITLAB \
  --executor docker \
  --description "My Docker Runner" \
  --docker-image "docker:latest" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

• From my computer I add some file in the repo
  
  
  
  • A codedirectory wich contains an index.html file
  
  
  • A Dockerfile : Dockerfile
  
  
  • A .gitlab-ci.yml : .gitlab-ci.yml
  
  
• When I push this the gitlab-ci is activate and execute the before_script :
  docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.gitlab.mysite.fr:5500
• TADAAAAAAAA !!! In the logs of the pipeline we can see that we have the error :
  Error response from daemon: Get https://registry.gitlab.mysite.fr:5500/v1/users/: dial tcp XX.XX.XX.XX:5500: getsockopt: connection refused

I execute all these steps and nothing more !

[maikelvl]

I see.

So after diggin' through, I've found the cause and a solution.

Short story: you should not want to use self-signed certificates in production. It's a pain because every docker client has to have your .crt.

When using a self-signed:

• As stated on the registry docs: "Be sure to use the name myregistrydomain.com as a CN." In this case 'gitlab-issuer' is not a valid CN, use registry.gitlab.mysite.fr.
• Copy your generated registry-auth.crt file to /etc/docker/certs.d/registry.gitlab.mysite.fr:5000/ca.crt on the machine where the Docker daemon is running.
• The docker daemon has to be initialised with the --insecure-registry-flag. I did apply this with my docker-machine thanks to this post (see comment from RAY at MAY 14, 2016)
• Add port "5000:5000" to the registry service, else the runner is unable to connect to it. (I assume the 5500 on the gitlab service is not being used, therefore could be removed)
• Add these env vars to the registry service: (see the docs)
  REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry-auth.crt
REGISTRY_HTTP_TLS_KEY=/certs/registry-auth.key
• Make sure you have the docker.sock mounted with your gitlab-multi-runner
• Make sure to mount the docker.sock inside your runners:
  docker exec -it gitlab-runner vi /etc/gitlab-runner/config.toml
  and add the docker.sock:
  volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]

Since I've done a lot of trail and error, it could be that some steps are unnecessary. I hope I did not forget anything. 😅

I recommend to use valid ssl certificates for the registry as well as GitLab itself. Use LetsEncrypt for instance.

[maikelvl]

Just seeing your latest comment 😅

[colas-pomies]

Woh ! Okay I will try all of this tomorow. I know for valid ssl and i program to add letsencrypt (need to learn how to use it before). I will comment with the result tomorow and hope close the issue

[colas-pomies]

Okay so It's work but not totaly. I do all the modification you tell me to do and I connect to port 5000 and not to port 5500 in the docker login command.

The issue now is that I can't connect to my registry page in gitlab (http://gitlab.mysite.fr/myuser/myproject/container_registry).

This problem appear only when I add these line in my registry docker :

    - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry-auth.crt
    - REGISTRY_HTTP_TLS_KEY=/certs/registry-auth.key

And when these line are not here I can connect to my registry page but I cant connect to the registry with docker login. I will try to use LetsEncrypt and tell you if it's work but if you have some idea to solve this problem with self signed certificate it will be awsome !!

[maikelvl]

mmm... maybe you could find the cause by inspecting logs:
docker exec -it ..._gitlab_1 tail -n 100 /var/log/gitlab/gitlab/production.log ?

[colas-pomies]

Started GET "/myuser/my-other-site.fr/container_registry" for XX.XX.XX.XX at 2016-09-15 23:44:59 +0200
Processing by Projects::ContainerRegistryController#index as HTML
Parameters: {"namespace_id"=>"myuser", "project_id"=>"my-other-site.fr"}
Completed 500 Internal Server Error in 381ms (ActiveRecord: 13.5ms)

Faraday::ConnectionFailed (wrong status line: "x15x03x01x00x02x02"):
lib/container_registry/client.rb:19:in repository_tags' lib/container_registry/repository.rb:22:inmanifest'
lib/container_registry/repository.rb:31:in tags' app/controllers/projects/container_registry_controller.rb:8:inindex'
lib/gitlab/request_profiler/middleware.rb:15:in call' lib/gitlab/middleware/go.rb:16:incall'

[maikelvl]

Could you paste your current docker-compose.yml?

[colas-pomies]

In my last docker-compose I had - GITLAB_REGISTRY_API_URL=http://registry:5000 for gitlab container. I change it with - GITLAB_REGISTRY_API_URL=https://registry:5000 and now I got the error :

Started GET "/myuser/my-other-site.fr/container_registry" for XX.XX.XX.XX at 2016-09-16 00:02:54 +0200
Processing by Projects::ContainerRegistryController#index as HTML
  Parameters: {"namespace_id"=>"myuser", "project_id"=>"my-other-site.fr"}
Completed 500 Internal Server Error in 397ms (ActiveRecord: 18.5ms)

Faraday::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed):
  lib/container_registry/client.rb:19:in `repository_tags'
  lib/container_registry/repository.rb:22:in `manifest'
  lib/container_registry/repository.rb:31:in `tags'
  app/controllers/projects/container_registry_controller.rb:8:in `index'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'

Do you want my all docker-compose ?

[maikelvl]

Ok. Maybe with a stripped down env list.

[colas-pomies]

[colas-pomies]

[maikelvl]

[colas-pomies]

[solidnerd]

[colas-pomies]

[colas-pomies]

[rutsky]

[reynold-lariza]

[maikelvl]

[reynold-lariza]

[reynold-lariza]

[reynold-lariza]

[reynold-lariza]

[reynold-lariza]

[reynold-lariza]

[maikelvl]

[rbucker]

[rbucker]