Origin: Can not pull image from gitlab private registry

[slmzig]

I Can not pull image from gitlab private registry

How to use the Container Registry

First log in to GitLab’s Container Registry using your GitLab username and password. If you have 2FA enabled you need to use a personal access token:

docker login registry.gitlab.com

Version

oc version
oc v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7
features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.6.1+008f2d5
kubernetes v1.6.1+5115d708d7

Steps To Reproduce

1. oc new-project test2
2. oc project test2
3.

oc secrets new-dockercfg secret     --docker-server=https://registry.gitlab.com     --docker-username=user     --docker-password="pass"     [email protected]
secret/secret

4.

oc secrets link builder secret --for=pull
oc secrets link default secret --for=pull
oc secrets link deployer secret --for=pull

5.

oc new-app --docker-image='registry.gitlab.com/user/imagename:latest' --loglevel=5
I1227 12:37:21.263940   77342 newapp.go:486] Docker client did not respond to a ping: Get http://unix.sock/_ping: dial unix /var/run/docker.sock: connect: permission denied
I1227 12:37:21.264210   77342 dockerimagelookup.go:79] checking remote registry for "registry.gitlab.com/user/imagename:latest"
I1227 12:37:22.356499   77342 dockerimagelookup.go:214] image import failed: image.ImageImportStatus{Tag:"latest", Status:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"Internal error occurred: Get https://registry.gitlab.com/v2/user/imagename/manifests/latest: denied: access forbidden", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc42072aff0), Code:500}, Image:(*image.Image)(nil)}
W1227 12:37:22.356612   77342 dockerimagelookup.go:220] Docker registry lookup failed: Get https://registry.gitlab.com/v2/user/imagename/manifests/latest: denied: access forbidden
F1227 12:37:22.356776   77342 helpers.go:119] error: no match for "registry.gitlab.com/user/imagename:latest"

The 'oc new-app' command will match arguments to the following types:

  1. Images tagged into image streams in the current project or the 'openshift' project
     - if you don't specify a tag, we'll add ':latest'
  2. Images in the Docker Hub, on remote registries, or on the local Docker engine
  3. Templates in the current project or the 'openshift' project
  4. Git repository URLs or local paths that point to Git repositories

--allow-missing-images can be used to point to an image that does not exist yet.

Current Result

image did not pull from registry
denied: access forbidden

Expected Result

images must pull from registry

Additional Information

oc get all -o json -n test2
{
    "apiVersion": "v1",
    "items": [],
    "kind": "List",
    "metadata": {},
    "resourceVersion": "",
    "selfLink": ""
}

oc describe secret/secret
Name:       secret
Namespace:  test2
Labels:     <none>
Annotations:    <none>

Type:   kubernetes.io/dockercfg

Data
====
.dockercfg: {"https://registry.gitlab.com":{"username":"user","password":"pass","email":"[email protected]","auth":"someauth"}}

oc describe serviceaccount/default
Name:       default
Namespace:  test2
Labels:     <none>
Annotations:    <none>

Image pull secrets: default-dockercfg-8h0hq
                    secret

Mountable secrets:  default-dockercfg-8h0hq
                    default-token-8dc0x

Tokens:             default-token-6s4bl
                    default-token-8dc0x

oc describe serviceaccount/builder
Name:       builder
Namespace:  test2
Labels:     <none>
Annotations:    <none>

Image pull secrets: builder-dockercfg-w3lfm
                    secret

Mountable secrets:  builder-token-t7kzh
                    builder-dockercfg-w3lfm

Tokens:             builder-token-c1nt6
                    builder-token-t7kzh

oc describe serviceaccount/deployer
Name:       deployer
Namespace:  test2
Labels:     <none>
Annotations:    <none>

Image pull secrets: deployer-dockercfg-zqnxx
                    secret

Mountable secrets:  deployer-token-sbl9c
                    deployer-dockercfg-zqnxx

Tokens:             deployer-token-sbl9c
                    deployer-token-z61sv



md5-255dd18c827de6b5e7d2c5074991f4b7



oc adm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/home/centos/.kube/config'
Info:  Using context for cluster-admin access: 'default/oshift.com/system:admin'
[Note] Performing systemd discovery

[Note] Running diagnostic: ConfigContexts[api-gateway/oshift.com/admin]
       Description: Validate client config context is complete and has connectivity

Info:  For client config context 'api-gateway/oshift.com/admin':
       The server URL is 'https://oshift.com'
       The user authentication is 'admin/oshift.com'
       The current project is 'api-gateway'
       Successfully requested project list; has access to project(s):
         [datasources datasources-china-copy hello-openshift hola superpython test test2 testproject]

[Note] Running diagnostic: ConfigContexts[default/oshift.com/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  For client config context 'default/oshift.com/system:admin':
       The server URL is 'https://oshift.com'
       The user authentication is 'system:admin/oshift.com'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [datasources datasources-china-copy default hello-openshift hola kube-public kube-system logging management-infra openshift ...]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

ERROR: [DCli2012 from diagnostic DiagnosticPod@openshift/origin/pkg/diagnostics/client/run_diagnostics_pod.go:156]
       See the errors below in the output from the diagnostic pod:
       [Note] Running diagnostic: PodCheckAuth
              Description: Check that service account credentials authenticate as expected

       Info:  Service account token successfully authenticated to master
       ERROR: [DP1014 from diagnostic PodCheckAuth@openshift/origin/pkg/diagnostics/pod/auth.go:174]
              Request to integrated registry timed out; this typically indicates network or SDN problems.

       [Note] Running diagnostic: PodCheckDns
              Description: Check that DNS within a pod works as expected

       [Note] Summary of diagnostics execution (version v3.6.1+008f2d5):
       [Note] Errors seen: 1

[Note] Running diagnostic: NetworkCheck
       Description: Create a pod on all schedulable nodes and run network diagnostics from the application standpoint

ERROR: [DNet2001 from diagnostic NetworkCheck@openshift/origin/pkg/diagnostics/network/run_pod.go:83]
       Checking network plugin failed. Error: User "admin" cannot get clusternetworks at the cluster scope

[Note] Skipping diagnostic: AggregatedLogging
       Description: Check aggregated logging integration for proper configuration
       Because: Master configuration is unreadable

[Note] Running diagnostic: ClusterRegistry
       Description: Check that there is a working Docker registry

ERROR: [DClu1006 from diagnostic ClusterRegistry@openshift/origin/pkg/diagnostics/cluster/registry.go:206]
       The "docker-registry" service exists but has no associated pods, so it
       is not available. Builds and deployments that use the registry will fail.

[Note] Running diagnostic: ClusterRoleBindings
       Description: Check that the default ClusterRoleBindings are present and contain the expected subjects

Info:  clusterrolebinding/cluster-readers has more subjects than expected.

       Use the `oadm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount management-infra management-admin    }.
Info:  clusterrolebinding/cluster-readers has extra subject {ServiceAccount default router    }.

Info:  clusterrolebinding/self-provisioners has more subjects than expected.

       Use the `oadm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/self-provisioners has extra subject {ServiceAccount management-infra management-admin    }.

[Note] Running diagnostic: ClusterRoles
       Description: Check that the default ClusterRoles are present and contain the expected permissions

[Note] Running diagnostic: ClusterRouterName
       Description: Check there is a working router

ERROR: [DClu2007 from diagnostic ClusterRouter@openshift/origin/pkg/diagnostics/cluster/router.go:157]
       The "router" DeploymentConfig exists but has no running pods, so it
       is not available. Apps will not be externally accessible via the router.

[Note] Skipping diagnostic: MasterNode
       Description: Check if master is also running node (for Open vSwitch)
       Because: (DClu3008) Master config provided but unable to parse: open /etc/origin/master/master-config.yaml: permission denied

[Note] Skipping diagnostic: MetricsApiProxy
       Description: Check the integrated heapster metrics can be reached via the API proxy
       Because: The heapster service does not exist in the openshift-infra project at this time,
       so it is not available for the Horizontal Pod Autoscaler to use as a source of metrics.

[Note] Running diagnostic: NodeDefinitions
       Description: Check node records on master

WARN:  [DClu0003 from diagnostic NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:113]
       Node is-oshift-master-01.novalocal is ready but is marked Unschedulable.
       This is usually set manually for administrative reasons.
       An administrator can mark the node schedulable with:
           oadm manage-node is-oshift-master-01.novalocal --schedulable=true

       While in this state, pods should not be scheduled to deploy on the node.
       Existing pods will continue to run until completed or evacuated (see
       other options for 'oadm manage-node').

[Note] Running diagnostic: RouteCertificateValidation
       Description: Check all route certificates for certificates that might be rejected by extended validation.

[Note] Running diagnostic: ServiceExternalIPs
       Description: Check for existing services with ExternalIPs that are disallowed by master config

ERROR: [DH0002 from diagnostic ServiceExternalIPs@openshift/origin/pkg/diagnostics/host/util.go:38]
       Could not read master config file '/etc/origin/master/master-config.yaml':
       (*os.PathError) open /etc/origin/master/master-config.yaml: permission denied

Info:  Unreadable master config; skipping this diagnostic.

[Note] Running diagnostic: AnalyzeLogs
       Description: Check for recent problems in systemd service logs

Info:  Checking journalctl logs for 'origin-master' service
Info:  Checking journalctl logs for 'origin-node' service
Info:  Checking journalctl logs for 'docker' service

[Note] Running diagnostic: MasterConfigCheck
       Description: Check the master config file

ERROR: [DH0002 from diagnostic MasterConfigCheck@openshift/origin/pkg/diagnostics/host/util.go:38]
       Could not read master config file '/etc/origin/master/master-config.yaml':
       (*os.PathError) open /etc/origin/master/master-config.yaml: permission denied

[Note] Running diagnostic: NodeConfigCheck
       Description: Check the node config file

ERROR: [DH1002 from diagnostic NodeConfigCheck@openshift/origin/pkg/diagnostics/host/check_node_config.go:38]
       Could not read node config file '/etc/origin/node/node-config.yaml':
       (*os.PathError) open /etc/origin/node/node-config.yaml: permission denied

[Note] Running diagnostic: UnitStatus
       Description: Check status for related systemd units

[Note] Summary of diagnostics execution (version v3.6.1+008f2d5):
[Note] Warnings seen: 1
[Note] Errors seen: 7

[slmzig]

Any ideas?

[piar1989]

I have the same problem

[sylvainar]

Same here, I can pull the image when the project visibility is on public, but not on private, I've made the whole workflow of adding secrets and stuff, and I can pull with my access token from my computer.

I'm trying to pull from Gitlab.com integrated registry.

[slmzig]

It pulls image from private repo only when I add to pod

serviceAccount: default

and run yaml from console, but from web it do not pull image from private repo

kind: Pod
apiVersion: v1
metadata:
  name: crm
  creationTimestamp:
  labels:
    name: ds
spec:
  containers:
  - env:
    - name: JAVA_OPTS
      value: "-Dpidfile.path=/dev/null"
    name: crm
    image: registry.gitlab.com/user/project:latest
    ports:
    - containerPort: 8090
      protocol: TCP
    resources: {}
    volumeMounts: {}
    terminationMessagePath: "/dev/termination-log"
    imagePullPolicy: IfNotPresent
    capabilities: {}
    securityContext:
      capabilities: {}
      privileged: false
  volumes: {}
  restartPolicy: Always
  dnsPolicy: ClusterFirst
  serviceAccount: default
status: {}

[sylvainar]

LOL, got it.

So gitlab make the authentification in two times, first gitlab.com then registry.gitlab.com. Actually the error we got was the first one that was being dropped.
Few people got this error because usually they put their gitlab in domain.tld and the registry in domain.tld:5000 so the authentification works.

Just duplicate what you've done for registry.gitlab.com, but for gitlab.com.

[slmzig]

@sylvainar Thanks

[nikolicjakov]

@sylvainar

Hi, i am having the same problem. I have latest GitLab running behind custom domain and i cant get it to work.

I have tested creating 2 secrets one for _gitlab.mydomain.com_ and one for _registry.mydomain.com_. But nothing seems to help.

I have also tested using curl to get the domain of authentication and it is gitlab.mydomain.com.

curl -I https://registry.mydomain.com/v2/ |& grep -i www-authenticate
Www-Authenticate: Bearer realm="https://gitlab.mydomain.com/jwt/auth",service="container_registry"

Am i doing something wrong here because i am keep getting "access forbidden"..??

[nikolicjakov]

I would just like to post an update for all of you that are having trouble getting this to work.
This is what i have did and what has worked out for me.

1. Login to your gitlab docker registry and gitlab domain using docker..

docker login registry.mydomain.com

and

docker login gitlab.mydomain.com

This will create docker _config.json_ in your user home directory under _~/.docker/config.json_
Now you can use this file to generate openshift secret using this command.

oc create secret generic mysecretname --from-file=.dockerconfigjson=~/.docker/config.json --type=kubernetes.io/dockerconfigjson

After that you can link this secret to serviceaccounts and pull your images without any issue.

oc secrets link default mysecretname --for=pull
oc secrets link builder mysecretname --for=pull
oc secrets link deployer mysecretname --for=pull

Now you can use openshift web interface to deploy images from your private GitLab image registry.

[uocxp]

my gitlab is deployed in a subdirectory _tld/gitlab_ and the registry as subdomain _registry.tld:5000_. I can do docker login registry.tld:5000 without any problem but it fails when docker login tld/gitlab i don't know if it lays on the subdirectory.
i can pull the image with docker pull registry.tld:5000/image without any problem but i not able to pull it using oc import-image registry.tld:5000/image --confirm I get : denied: access forbidden

why it does work with docker pull but not with oc?

[debianmaster]

Found the fix.
Based on this https://stackoverflow.com/questions/47993222/can-not-pull-image-from-gitlab-private-registryopenshift
```
So gitlab make the authentification in two times, first gitlab.com then registry.gitlab.com. Actually the error we got was the first one that was being dropped.

Just duplicate what you've done for registry.gitlab.com, but for gitlab.com```

i had create another secret that is referring to gitlab.com similar to egistry.gitlab.com then it fixed the issue

[nikolicjakov]

Found the fix.
Based on this https://stackoverflow.com/questions/47993222/can-not-pull-image-from-gitlab-private-registryopenshift

So gitlab make the authentification in two times, first gitlab.com then registry.gitlab.com. Actually the error we got was the first one that was being dropped.

Just duplicate what you've done for registry.gitlab.com, but for gitlab.com```

i had create another secret that is referring to  `gitlab.com`   similar to  `egistry.gitlab.com` then it fixed the issue

This is the same fix that I have suggested...

[sylvainar]

That's also what I posted several comments ago.