Origin: accessing ImageStream from private repository with auth delegated to a different server requires creating two secrets instead of one

Created on 27 Jun 2016  ·  90Comments  ·  Source: openshift/origin

I want to declare an ImageStream that tracks an external docker image located on an authenticated docker registry.

My use-case is:

  • build docker images for our internal applications as part of our CI pipeline
  • have the CI pipeline push the docker image to external registry (eg. gitlab) on success
  • track the docker images from many OpenShift environments (using ImageStreams)
  • redeploy all relevant applications in OpenShift when their docker image has been updated (using image change triggers)

So far, I have found no way to declare an ImageStream which uses a secret to pull the image.

I have found 2 workarounds:

  • either use S2I source builds (then what we deploy wont be the docker image build in CI)
  • either use an inline dockerfile (eg. FROM registry.internal.mycompany.com/myapp) but then how do I trigger a new build since ImageStream still cannot follow the source image).

I have also tried without success to run docker login on the Openshift hosts, in the hope that the image puller would run as same user and implicitly use my dockercfg but no success.

Version
$ oc version
oc v1.2.0
kubernetes v1.2.0-36-g4a3f9c5
Steps To Reproduce

Run the following commands (after adjusting docker image source to point to an authenticated docker registry).

$ cat <<EOF | tee is.yaml
apiVersion: v1
kind: ImageStream
metadata:
  name: myapp
spec:
  tags:
  - from:
      kind: DockerImage
      name: registry.internal.company.com/projects/myapp:latest
    name: latest
EOF
$ oc create -f is.yaml
imagestream "myapp" created
$ oc get is
NAME               DOCKER REPO                               TAGS      UPDATED
myapp              10.10.166.188:5000/gio/myapp              latest    
$ oc import-image myapp
The import completed successfully.

Name:           myapp
Created:        14 seconds ago
Labels:         <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2016-06-27T14:15:06Z
Docker Pull Spec:   10.10.166.188:5000/project/myapp

Tag Spec                                Created     PullSpec    Image
latest  registry.internal.company.com/projects/myapp:latest 13 seconds ago          import failed: Internal error occurred: Get https://registry.intern....
Current Result

Image fails to import because the image puller does not use any secret.

Expected Result

ImageStream should provide a way to specify a secret to use when pulling the image (with a mechanism similar to pullSecret in BuildConfig dockerStrategy).

componenimage kinbug prioritP1
Source
picture jperville

Most helpful comment

I have done more testing @soltysh and indeed the name of secret can be anything, as long as there is two secrets:

  • one with a --docker-server matching the authentication endpoint (gitlab.internal.example.com), used by the ImageStream
  • and one with a --docker-server matching the public registry endpoint (registry.internal.example.com), used when pulling pods images.

Yes, my authentication is done versus a different server (using the new integrated container registry from gitlab), hence the need for two secrets because of the two endpoints. I just wish there was a way to DRY that but for the moment duplicating the secrets is acceptable.

Also, I have tried adding the secrets only to the default service account (and not to the deployer), and it still works. Thanks for pointing that to me.

picture jperville on 30 Jun 2016
👍5

All 90 comments

@jperville have you checked our docs at https://docs.openshift.org/latest/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries I think that topic fits your use case. If not lemme know where you're having problems, I'll help.

soltysh picture soltysh on 29 Jun 2016

Yes @soltysh , I know how to reference docker images on other registries in my pod definitions, however I would really like my pods to be automatically redeployed if their docker images are updated; to me this is the point of the ImageStream feature.

If there is another method, how to use use it with oc new-app --docker-image=some-external-registry/image?

jperville picture jperville on 29 Jun 2016
👍2

@jperville my bad, I wanted to point you to https://docs.openshift.org/latest/dev_guide/managing_images.html#private-registries apologies

soltysh picture soltysh on 29 Jun 2016

Thanks @soltysh . I have created the secret, however I see no way with ImageStream object or with the oc import-image command to tell explicitly "use this secret when pulling the image".

Assuming I am in the 'project' project, here are my secrets:

$ oc secrets new-dockercfg registry-login --docker-server=registry.example.com \
  --docker-username=someuser --docker-password=somepassword [email protected]
$ oc secrets add serviceaccount/default secrets/registry-login --for=pull
$ oc secrets add serviceaccount/builder secrets/registry-login
$ oc get secrets/registry-login
NAME             TYPE                      DATA      AGE
registry-login   kubernetes.io/dockercfg   1         26s

I want to create the following image Stream:

$ cat <<EOF | tee is.yaml
apiVersion: v1
kind: ImageStream
metadata:
  name: myapp
spec:
  tags:
  - from:
      kind: DockerImage
      name: registry.internal.example.com/myapp:latest
    name: latest
EOF
$ oc create -f is.yaml
$ oc get is
NAME      DOCKER REPO                       TAGS      UPDATED
myapp     10.10.87.191:5000/default/myapp   latest

If I oc import-image the image, here is the error I get:

$ oc import-image myapp
oc describe is myapp
Name:           myapp
Created:        About a minute ago
Labels:         <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2016-06-29T12:56:33Z
Docker Pull Spec:   10.10.87.191:5000/default/myapp

Tag Spec                                Created         PullSpec    Image
latest  registry.si.perfect-memory.com/core/exchange-manager:latest About a minute ago          import failed: Internal error occurred: Get https://registry.si.per...

$ oc get is/myapp -o yaml
apiVersion: v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.dockerRepositoryCheck: 2016-06-29T12:56:33Z
  creationTimestamp: 2016-06-29T12:56:32Z
  generation: 2
  name: myapp
  namespace: default
  resourceVersion: "639"
  selfLink: /oapi/v1/namespaces/default/imagestreams/myapp
  uid: e75ff849-3df8-11e6-8434-080027c23ce5
spec:
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: registry.internal.example.com/myapp:latest
    generation: 2
    importPolicy: {}
    name: latest
status:
  dockerImageRepository: 10.10.87.191:5000/default/myapp
  tags:
  - conditions:
    - generation: 2
      lastTransitionTime: 2016-06-29T12:56:33Z
      message: 'Internal error occurred: Get https://registry.internal.example.com/myapp/manifests/latest:
        token auth attempt for registry: https://gitlab.example.com/jwt/auth?scope=repository%3Amyapp%3Apull&service=container_registry
        request failed with status: 403 Forbidden'
      reason: InternalError
      status: "False"
      type: ImportSuccess
    items: null
    tag: latest

Note that in the error above, the registry public endpoint is https://registry.internal.example.com which delegates authentication by redirecting to https://gitlab.example.com (a different domain name but still part of the same environment); that authentication fails with 403 .

From reading https://docs.openshift.org/latest/dev_guide/managing_images.html#private-registries I was expecting the secret to be automatically used to authenticate to the external registry, but it seems it is not used.

If the secret is to be associated with a specific serviceaccount so that all image stream pulls use that secret automatically, can it be documented somewhere?

jperville picture jperville on 29 Jun 2016

Your use-case is different from what I've been testing/working on so far. Generally all this works like this: during importing image the importer will match against all secrets, in the same project the ImageStream is defined in, based on the hostname (see here). Having said that you start with changing the secret name so that it matches the registry uri, or rather auth endpoint uri in your case. If either won't help attach me log from import operation at level 8, I'll have a look, it might be that your use-case requires more work.

soltysh picture soltysh on 29 Jun 2016

I have tried again naming the secret after the registry name (eg. `oc get secrets/registry.internal.example.com) without success.

You will find below a trace of oc import-image myapp at level 8 below (with the real host and image names):

I0629 13:43:47.224945   29105 loader.go:242] Config loaded from file /root/.kube/config
I0629 13:43:47.227135   29105 round_trippers.go:264] GET https://192.168.56.220:8443/api
I0629 13:43:47.227148   29105 round_trippers.go:271] Request Headers:
I0629 13:43:47.227152   29105 round_trippers.go:274]     Accept: application/json, */*
I0629 13:43:47.227156   29105 round_trippers.go:274]     User-Agent: oc/v1.2.0 (linux/amd64) kubernetes/b12cbb6
I0629 13:43:47.227160   29105 round_trippers.go:274]     Authorization: Bearer 5x00cpdmr9mbsO1OuoZbyQJzc9BCN5Mi2SNzW86zQss
I0629 13:43:47.279596   29105 round_trippers.go:289] Response Status: 200 OK in 52 milliseconds
I0629 13:43:47.279620   29105 round_trippers.go:292] Response Headers:
I0629 13:43:47.279626   29105 round_trippers.go:295]     Cache-Control: no-store
I0629 13:43:47.279630   29105 round_trippers.go:295]     Content-Type: application/json
I0629 13:43:47.279634   29105 round_trippers.go:295]     Date: Wed, 29 Jun 2016 13:43:47 GMT
I0629 13:43:47.279638   29105 round_trippers.go:295]     Content-Length: 136
I0629 13:43:47.279681   29105 request.go:870] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.56.220:443"}]}
I0629 13:43:47.279909   29105 round_trippers.go:264] GET https://192.168.56.220:8443/apis
I0629 13:43:47.279918   29105 round_trippers.go:271] Request Headers:
I0629 13:43:47.279921   29105 round_trippers.go:274]     Accept: application/json, */*
I0629 13:43:47.279925   29105 round_trippers.go:274]     User-Agent: oc/v1.2.0 (linux/amd64) kubernetes/b12cbb6
I0629 13:43:47.279929   29105 round_trippers.go:274]     Authorization: Bearer 5x00cpdmr9mbsO1OuoZbyQJzc9BCN5Mi2SNzW86zQss
I0629 13:43:47.282089   29105 round_trippers.go:289] Response Status: 200 OK in 2 milliseconds
I0629 13:43:47.282111   29105 round_trippers.go:292] Response Headers:
I0629 13:43:47.282116   29105 round_trippers.go:295]     Content-Length: 778
I0629 13:43:47.282120   29105 round_trippers.go:295]     Cache-Control: no-store
I0629 13:43:47.282124   29105 round_trippers.go:295]     Content-Type: application/json
I0629 13:43:47.282128   29105 round_trippers.go:295]     Date: Wed, 29 Jun 2016 13:43:47 GMT
I0629 13:43:47.282162   29105 request.go:870] Response Body: {"kind":"APIGroupList","groups":[{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.56.220:443"}]},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.56.220:443"}]},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"192.168.56.220:443"}]}]}
I0629 13:43:47.282985   29105 round_trippers.go:264] GET https://192.168.56.220:8443/oapi
I0629 13:43:47.282995   29105 round_trippers.go:271] Request Headers:
I0629 13:43:47.282999   29105 round_trippers.go:274]     User-Agent: oc/v1.2.0 (linux/amd64) openshift/2e62fab
I0629 13:43:47.283003   29105 round_trippers.go:274]     Authorization: Bearer 5x00cpdmr9mbsO1OuoZbyQJzc9BCN5Mi2SNzW86zQss
I0629 13:43:47.283007   29105 round_trippers.go:274]     Accept: application/json, */*
I0629 13:43:47.284778   29105 round_trippers.go:289] Response Status: 200 OK in 1 milliseconds
I0629 13:43:47.284801   29105 round_trippers.go:292] Response Headers:
I0629 13:43:47.284808   29105 round_trippers.go:295]     Cache-Control: no-store
I0629 13:43:47.284814   29105 round_trippers.go:295]     Content-Type: application/json
I0629 13:43:47.284820   29105 round_trippers.go:295]     Date: Wed, 29 Jun 2016 13:43:47 GMT
I0629 13:43:47.284826   29105 round_trippers.go:295]     Content-Length: 93
I0629 13:43:47.284856   29105 request.go:870] Response Body: {"kind":"APIVersions","apiVersion":"v1","versions":["v1"],"serverAddressByClientCIDRs":null}
I0629 13:43:47.285248   29105 round_trippers.go:264] GET https://192.168.56.220:8443/oapi/v1/namespaces/default/imagestreams/myapp
I0629 13:43:47.285263   29105 round_trippers.go:271] Request Headers:
I0629 13:43:47.285268   29105 round_trippers.go:274]     Accept: application/json, */*
I0629 13:43:47.285274   29105 round_trippers.go:274]     User-Agent: oc/v1.2.0 (linux/amd64) openshift/2e62fab
I0629 13:43:47.285279   29105 round_trippers.go:274]     Authorization: Bearer 5x00cpdmr9mbsO1OuoZbyQJzc9BCN5Mi2SNzW86zQss
I0629 13:43:47.288234   29105 round_trippers.go:289] Response Status: 200 OK in 2 milliseconds
I0629 13:43:47.288254   29105 round_trippers.go:292] Response Headers:
I0629 13:43:47.288259   29105 round_trippers.go:295]     Cache-Control: no-store
I0629 13:43:47.288263   29105 round_trippers.go:295]     Content-Type: application/json
I0629 13:43:47.288267   29105 round_trippers.go:295]     Date: Wed, 29 Jun 2016 13:43:47 GMT
I0629 13:43:47.288272   29105 round_trippers.go:295]     Content-Length: 1122
I0629 13:43:47.288302   29105 request.go:870] Response Body: {"kind":"ImageStream","apiVersion":"v1","metadata":{"name":"myapp","namespace":"default","selfLink":"/oapi/v1/namespaces/default/imagestreams/myapp","uid":"e75ff849-3df8-11e6-8434-080027c23ce5","resourceVersion":"639","generation":2,"creationTimestamp":"2016-06-29T12:56:32Z","annotations":{"openshift.io/image.dockerRepositoryCheck":"2016-06-29T12:56:33Z"}},"spec":{"tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"registry.si.perfect-memory.com/core/exchange-manager:latest"},"generation":2,"importPolicy":{}}]},"status":{"dockerImageRepository":"10.10.87.191:5000/default/myapp","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2016-06-29T12:56:33Z","reason":"InternalError","message":"Internal error occurred: Get https://registry.si.perfect-memory.com/v2/core/exchange-manager/manifests/latest: token auth attempt for registry: https://gitlab.si.perfect-memory.com/jwt/auth?scope=repository%3Acore%2Fexchange-manager%3Apull\u0026service=container_registry request failed with status: 403 Forbidden","generation":2}]}]}}
I0629 13:43:47.289274   29105 request.go:555] Request Body: {"kind":"ImageStreamImport","apiVersion":"v1","metadata":{"name":"myapp","namespace":"default","resourceVersion":"639","creationTimestamp":null},"spec":{"import":true,"images":[{"from":{"kind":"DockerImage","name":"registry.si.perfect-memory.com/core/exchange-manager:latest"},"to":{"name":"latest"},"importPolicy":{}}]},"status":{}}
I0629 13:43:47.289321   29105 round_trippers.go:264] POST https://192.168.56.220:8443/oapi/v1/namespaces/default/imagestreamimports
I0629 13:43:47.289328   29105 round_trippers.go:271] Request Headers:
I0629 13:43:47.289331   29105 round_trippers.go:274]     Accept: application/json, */*
I0629 13:43:47.289335   29105 round_trippers.go:274]     User-Agent: oc/v1.2.0 (linux/amd64) openshift/2e62fab
I0629 13:43:47.289338   29105 round_trippers.go:274]     Authorization: Bearer 5x00cpdmr9mbsO1OuoZbyQJzc9BCN5Mi2SNzW86zQss
I0629 13:43:47.289342   29105 round_trippers.go:274]     Content-Type: application/json
I0629 13:43:47.698538   29105 round_trippers.go:289] Response Status: 201 Created in 409 milliseconds
I0629 13:43:47.698569   29105 round_trippers.go:292] Response Headers:
I0629 13:43:47.698577   29105 round_trippers.go:295]     Cache-Control: no-store
I0629 13:43:47.698584   29105 round_trippers.go:295]     Content-Type: application/json
I0629 13:43:47.698590   29105 round_trippers.go:295]     Date: Wed, 29 Jun 2016 13:43:47 GMT
I0629 13:43:47.698643   29105 request.go:870] Response Body: {"kind":"ImageStreamImport","apiVersion":"v1","metadata":{"name":"myapp","namespace":"default","selfLink":"/oapi/v1/namespaces/default/imagestreamimports/myapp","uid":"8113c15e-3dff-11e6-8434-080027c23ce5","resourceVersion":"639","creationTimestamp":"2016-06-29T13:43:47Z"},"spec":{"import":true,"images":[{"from":{"kind":"DockerImage","name":"registry.si.perfect-memory.com/core/exchange-manager:latest"},"to":{"name":"latest"},"importPolicy":{}}]},"status":{"import":{"metadata":{"name":"myapp","namespace":"default","uid":"e75ff849-3df8-11e6-8434-080027c23ce5","resourceVersion":"639","generation":2,"creationTimestamp":"2016-06-29T12:56:32Z","annotations":{"openshift.io/image.dockerRepositoryCheck":"2016-06-29T12:56:33Z"}},"spec":{"tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"registry.si.perfect-memory.com/core/exchange-manager:latest"},"generation":2,"importPolicy":{}}]},"status":{"dockerImageRepository":"10.10.87.191:5000/default/myapp","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2016-06-29T12:56:33Z","reason":"InternalError","message":"Internal error occurred: Get https://registry.si.perfect-memory.com/v2/core/exchange-manager/manifests/latest: token auth attempt for registry: https://gitlab.si.perfect-memory.com/jwt/auth?scope=repository%3Acore%2Fexchange-manager%3Apull\u0026service=container_registry request failed with status: 403 Forbidden","generation":2}]}]}},"images":[{"status":{"metadata":{},"status":"Failure","message":"Internal error occurred: Get https://registry.si.perfect-memory.com/v2/core/exchange-manager/manifests/latest: token auth attempt for registry: https://gitlab.si.perfect-memory.com/jwt/auth?scope=repository%3Acore%2Fexchange-manager%3Apull\u0026service=container_registry request failed with status: 403 Forbidden","reason":"InternalError","details":{"causes":[{"message":"Get https://registry.si.perfect-memory.com/v2/core/exchange-manager/manifests/latest: token auth attempt for registry: https://gitlab.si.perfect-memory.com/jwt/auth?scope=repository%3Acore%2Fexchange-manager%3Apull\u0026service=container_registry request failed with status: 403 Forbidden"}]},"code":500}}]}}
The import completed successfully.

I0629 13:43:47.699732   29105 round_trippers.go:264] GET https://192.168.56.220:8443/oapi/v1/namespaces/default/imagestreams/myapp
I0629 13:43:47.699744   29105 round_trippers.go:271] Request Headers:
I0629 13:43:47.699750   29105 round_trippers.go:274]     Authorization: Bearer 5x00cpdmr9mbsO1OuoZbyQJzc9BCN5Mi2SNzW86zQss
I0629 13:43:47.699756   29105 round_trippers.go:274]     Accept: application/json, */*
I0629 13:43:47.699761   29105 round_trippers.go:274]     User-Agent: oc/v1.2.0 (linux/amd64) openshift/2e62fab
I0629 13:43:47.702488   29105 round_trippers.go:289] Response Status: 200 OK in 2 milliseconds
I0629 13:43:47.702509   29105 round_trippers.go:292] Response Headers:
I0629 13:43:47.702515   29105 round_trippers.go:295]     Cache-Control: no-store
I0629 13:43:47.702518   29105 round_trippers.go:295]     Content-Type: application/json
I0629 13:43:47.702522   29105 round_trippers.go:295]     Date: Wed, 29 Jun 2016 13:43:47 GMT
I0629 13:43:47.702526   29105 round_trippers.go:295]     Content-Length: 1122
I0629 13:43:47.702556   29105 request.go:870] Response Body: {"kind":"ImageStream","apiVersion":"v1","metadata":{"name":"myapp","namespace":"default","selfLink":"/oapi/v1/namespaces/default/imagestreams/myapp","uid":"e75ff849-3df8-11e6-8434-080027c23ce5","resourceVersion":"639","generation":2,"creationTimestamp":"2016-06-29T12:56:32Z","annotations":{"openshift.io/image.dockerRepositoryCheck":"2016-06-29T12:56:33Z"}},"spec":{"tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"registry.si.perfect-memory.com/core/exchange-manager:latest"},"generation":2,"importPolicy":{}}]},"status":{"dockerImageRepository":"10.10.87.191:5000/default/myapp","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2016-06-29T12:56:33Z","reason":"InternalError","message":"Internal error occurred: Get https://registry.si.perfect-memory.com/v2/core/exchange-manager/manifests/latest: token auth attempt for registry: https://gitlab.si.perfect-memory.com/jwt/auth?scope=repository%3Acore%2Fexchange-manager%3Apull\u0026service=container_registry request failed with status: 403 Forbidden","generation":2}]}]}}
Name:           myapp
Created:        47 minutes ago
Labels:         <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2016-06-29T12:56:33Z
Docker Pull Spec:   10.10.87.191:5000/default/myapp

Tag Spec                                Created     PullSpec    Image
latest  registry.si.perfect-memory.com/core/exchange-manager:latest 47 minutes ago          import failed: Internal error occurred: Get https://registry.si.per...
jperville picture jperville on 29 Jun 2016

Sorry for not being clear, I need logs from openshift master, since the oc import-image just invokes the import on the server and the actual importing is being done on the master.

soltysh picture soltysh on 29 Jun 2016

Have no special logs appear on the master after I run the oc import-image command.

Here is how processes look on the master VM:

[root@master ~]# ps aux | grep origin
root      20908  0.3  2.4 578588 46580 ?        Ssl  12:37   0:25 /usr/bin/openshift start node --config=/etc/origin/node/node-config.yaml --loglevel=2
root      44545  3.4  9.3 834148 176644 ?       Ssl  14:48   0:13 /usr/bin/openshift start master --config=/etc/origin/master/master-config.yaml --loglevel=8

Notice the loglevel=8 on the master.

I get some verbose messages in journalctl -f -u origin-master when the master restarts, but no mew logs are visible when running the oc import-image command.

jperville picture jperville on 29 Jun 2016

You should see the logs like this one in the log. Without them I won't be able to dig what's going on.

soltysh picture soltysh on 29 Jun 2016

I'm sorry but I really don't see anything in the origin-master log after the booting messages (the last ones being about listing service account tokens). In neither the origin-master nor on the origin-node, both running at loglevel=8, anything shows when I try my oc import-image.

jperville picture jperville on 29 Jun 2016

@jperville Assuming those processes were started by systemd units the journalctl command used should be fine.
@soltysh loglevel for systemd units is set via the OPTIONS variable in /etc/sysconfig/origin-master and restarting the service.

sdodson picture sdodson on 29 Jun 2016

@sdodson yes I have set the loglevel in /etc/sysconfig, as shown by the output of systemctl status origin-master.service below:

Redirecting to /bin/systemctl status  origin-master.service
● origin-master.service - Origin Master Service
   Loaded: loaded (/usr/lib/systemd/system/origin-master.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-06-29 15:50:12 UTC; 4h 8min ago
     Docs: https://github.com/openshift/origin
 Main PID: 55796 (openshift)
   CGroup: /system.slice/origin-master.service
           └─55796 /usr/bin/openshift start master --config=/etc/origin/master/master-config.yaml --loglevel=8

Still nothing interesting in the origin-master log when trying to oc import-image the image.

jperville picture jperville on 29 Jun 2016

@jperville thanks for confirming, @soltysh had asked me on irc how to properly configure loglevel for systemd units, sorry if that comment was a bit out of context.

However I have an idea, is the certificate for registry.si.perfect-memory.com trusted by the master host? If not you'll need to add the insecure annotation to the image stream, I think the error message for that condition is more clear than a 500 error, but I may be mistaken. See https://docs.openshift.org/latest/dev_guide/managing_images.html#insecure-registries

sdodson picture sdodson on 29 Jun 2016

I will try tomorrow the insecure annotation; for information the private registry is using a regular domain-validated certificate which poses no problem with docker pull.

jperville picture jperville on 29 Jun 2016

After trying all combinations of insecureRegistry and naming the secret after the registry URL and the registry authentication endpoint, I am proud to announce success with the following combination:

  • insecureRegistry annotation: NOT set
  • name of secret: hostname of registry authentication endpoint, which may NOT be the same as the hosthame of the public registry endpoint (that is: gitlab.internal.example.com instead of repository.internal.example.com).

Thank you very much for the help @soltysh and @sdodson.

Could you add a note in the documentation about the way secrets have to be named when working with authenticated image streams? This was not easy to figure out, I didn't even know about having to name the secret in a specific manner in the first place.

Sample working image stream definition:

apiVersion: v1
kind: ImageStream
metadata:
  name: myapp
spec:
  tags:
  - from:
      kind: DockerImage
      name: registry.internal.example.com/myapp:latest
    name: latest

Command-line to make a suitably named secret and add it to the default service accounts:

for name in gitlab.internal.example.com ; do
  oc secrets new-dockercfg ${name } --docker-server=${name } \
    --docker-username=someuser --docker-password=somepassword [email protected]
  oc secrets add serviceaccount/default secrets/${name } --for=pull
  oc secrets add serviceaccount/builder secrets/${name }
done

PS: found nothing relevant in the master log about image streams, even at loglevel8.

jperville picture jperville on 30 Jun 2016

After successfully importing my authenticated ImageStream, I finally can oc new-app --image-stream=myapp.

Surprisingly, after oc create-ing the application, the pod will refuse to start because ErrImagePull (can't find the credentials) so I added back the secret named after the public registry hostname (here: registry.internal.example.com).

In the end, I had to duplicate my secret and add it to the default and deployer service accounts twice to get my application working.

For now, this is OK, I have a workaround, but it would be nice to be able to have a single place to declare which secret to use to pull which docker image, which would work both for ImageStream and for instanciating pods.

jperville picture jperville on 30 Jun 2016

Could you add a note in the documentation about the way secrets have to be named when working with authenticated image streams? This was not easy to figure out, I didn't even know about having to name the secret in a specific manner in the first place.

Actually after re-testing and re-checking the code the key is created from the value passed to --docker-server, so the secret name does not matter. Apologies for the confusion, but I wonder why it didn't work for you, though :/

Secondly, I don't understand why you need to update both deployer and default serviceaccount. Since your image is used only for deployment, default should be sufficient. Again, I just verified that.

The only thing that comes to my mind and would be worth checking would be the fact that your auth is done against different server, but that shouldn't be a problem, imho.

The logs I was looking into for look like that:

I0630 13:32:16.941779    5048 credentials.go:169] Being asked for https://auth.docker.io/token, trying index.docker.io/v1 for legacy behavior
I0630 13:32:16.941800    5048 credentials.go:174] Being asked for //index.docker.io/v1, trying docker.io for legacy behavior
I0630 13:32:16.941809    5048 credentials.go:177] Unable to find a secret to match //docker.io (docker.io)

And these should be available at level 5, already. I'm not sure why you're not seeing those :(

Finally, I'd really appreciate if you could give it a try once again with above details.

soltysh picture soltysh on 30 Jun 2016

I have done more testing @soltysh and indeed the name of secret can be anything, as long as there is two secrets:

  • one with a --docker-server matching the authentication endpoint (gitlab.internal.example.com), used by the ImageStream
  • and one with a --docker-server matching the public registry endpoint (registry.internal.example.com), used when pulling pods images.

Yes, my authentication is done versus a different server (using the new integrated container registry from gitlab), hence the need for two secrets because of the two endpoints. I just wish there was a way to DRY that but for the moment duplicating the secrets is acceptable.

Also, I have tried adding the secrets only to the default service account (and not to the deployer), and it still works. Thanks for pointing that to me.

jperville picture jperville on 30 Jun 2016
👍5

@jperville thanks for the detailed info, I've changed the tile of the bug and recategorized this a bit, since it seems like a bug that we should address. I'll leave this open and will try to tackle that. Thanks a lot for your report and patience!

soltysh picture soltysh on 1 Jul 2016

what are the endpoints @jperville ? trying to use the saas version of registry without success

hernandev picture hernandev on 29 Jul 2016

I am also encountering this issue, or something remarkably similar.

Output from the logs on my registry:

 level=warning msg="error authorizing context: authorization token required"

It appears that no secrets are being used to authenticate when I run oc import-image. However, my authentication mechanism is running on the same machine. It just uses this simple mechanism. It's running on the same machine, but a different port - would I still need to provide 2 secrets?

I've tried lots of what's been posted in here (though I'm not sure if I got everything in the correct order) and in the official documentation, with no joy yet.

Can post a much more elaborate report tomorrow with logs and sample outputs.

jacderida picture jacderida on 24 Aug 2016

I'm reproducing the issue using a 2 node Origin cluster and a simple Docker registry.

The registry uses this image running on port 5000, along with this simple authentication mechanism, running on port 5001 on the same machine.

[root@master ~]# oc version
oc v1.2.1
kubernetes v1.2.0-36-g4a3f9c5

I'll push a simple image up to that registry:

jacderida@jacderida-ubuntu-14-04 » ~/dev/dockerfiles/ansible/2.1.1.0 [master] $ docker \\
    login \\
    support-registry.westeurope.cloudapp.azure.com:5000
Username: testuser
Password: 
Login Succeeded
jacderida@jacderida-ubuntu-14-04 » ~/dev/dockerfiles/ansible/2.1.1.0 [master] $ docker \\
    push \\
    support-registry.westeurope.cloudapp.azure.com:5000/ansible:2.1.1.0
The push refers to a repository [support-registry.westeurope.cloudapp.azure.com:5000/ansible]
a5e118e704e3: Pushed 
f8e9b7723ab0: Pushed 
cd643ef4f979: Pushed 
87d57d63fda7: Pushed 
bd50d37a703c: Pushed 
2f71b45e4e25: Pushed 
2.1.1.0: digest: sha256:ca86385a1fc5e327d8b9a74a3b72b69b34377ca774f32e33acb18ba49ed0953b size: 6434

Some sample output from the registry:

Aug 25 21:59:55 registry docker[63002]: time="2016-08-25T21:59:55Z" level=info msg="response completed" go.version=go1.5.2 http.request.host="support-registry.westeurope.cloudapp.azure.com:5000" http.request.id=d9cd1455-f521-4cdf-92cf-34fcc3321419 http.request.method=HEAD http.request.remoteaddr="88.111.205.71:25721" http.request.uri="/v2/ansible/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.13.0-53-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))" http.response.contenttype="application/octet-stream" http.response.duration=4.911727ms http.response.status=200 http.response.written=0 instance.id=c34f7e44-c3d4-4cdb-95a5-d550c115bb2c version=v2.2.1
Aug 25 21:59:55 registry docker[63002]: 88.111.205.71 - - [25/Aug/2016:21:59:55 +0000] "HEAD /v2/ansible/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 0 "" "docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.13.0-53-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))"
Aug 25 21:59:55 registry docker[63002]: time="2016-08-25T21:59:55Z" level=info msg="response completed" go.version=go1.5.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v1+prettyjws" http.request.host="support-registry.westeurope.cloudapp.azure.com:5000" http.request.id=317a3aa8-4126-49d7-94e3-e0537e16dbe8 http.request.method=PUT http.request.remoteaddr="88.111.205.71:25722" http.request.uri="/v2/ansible/manifests/2.1.1.0" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.13.0-53-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))" http.response.duration=53.575396ms http.response.status=201 http.response.written=0 instance.id=c34f7e44-c3d4-4cdb-95a5-d550c115bb2c version=v2.2.1

The push seems to be fine.

I'll create a new project:

[root@master ~]# oc new-project private-registry
Now using project "private-registry" on server "https://master.f2qcrn2xk5kete4vnnzyykkywf.ax.internal.cloudapp.net:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    $ oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git

to build a new hello-world application in Ruby.

Now I'll create the Docker secret and link it to the service account:

[root@master ~]# oc secrets new-dockercfg support-registry --docker-server=support-registry.westeurope.cloudapp.azure.com:5000 --docker-username=testuser --docker-password=testpassword [email protected]
secret/support-registry
[root@master ~]# oc describe secrets/support-registry
Name:       support-registry
Namespace:  private-registry
Labels:     <none>
Annotations:    <none>

Type:   kubernetes.io/dockercfg

Data
====
.dockercfg: {"support-registry.westeurope.cloudapp.azure.com:5000":{"username":"testuser","password":"testpassword","email":"[email protected]","auth":"dGVzdHVzZXI6dGVzdHBhc3N3b3Jk"}}


[root@master ~]# oc secrets add serviceaccount/default secret/support-registry --for=pull
[root@master ~]# oc describe serviceaccount/default
Name:       default
Namespace:  private-registry
Labels:     <none>

Tokens:             default-token-j444i
                    default-token-xawzg

Image pull secrets: default-dockercfg-xfzh4
                    support-registry

Mountable secrets:  default-token-xawzg
                    default-dockercfg-xfzh4

The secret is applied to the service account, and appears to be correct.

Now, in theory, according to the documentation, I should be able to use oc import-image to produce an ImageStream with tags, since it should pick up the secret that's been added.

I'll try this now. This registry is just a test setup with a self-signed certificate, so we need to specify that it's insecure.

[root@master ~]# oc import-image ansible-test-stream --from=support-registry.westeurope.cloudapp.azure.com:5000/ansible --confirm --insecure --all
The import completed successfully.

Name:           ansible-test-stream
Created:        Less than a second ago
Labels:         <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2016-08-26T17:49:40Z
Docker Pull Spec:   172.30.148.0:5000/private-registry/ansible-test-stream
Tags:           <none>

It fails to actually read any of the tags for the image.

Here's the output from the registry:

Aug 26 17:49:40 registry docker[63002]: time="2016-08-26T17:49:40Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.5.2 http.request.host="support-registry.westeurope.cloudapp.azure.com:5000" http.request.id=e64eca8f-d5c9-492a-bc5c-bf8ccf32ff80 http.request.method=GET http.request.remoteaddr="52.174.57.91:52267" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" instance.id=c34f7e44-c3d4-4cdb-95a5-d550c115bb2c version=v2.2.1

It appears the secret isn't being used.

Can you advise if there are any workarounds for this, or if I'm misunderstanding something and doing it wrong?

jacderida picture jacderida on 26 Aug 2016

Does anyone have any ideas on a workaround for this?

I've been able to work around it on Origin by pushing directly to the internal registry, which automatically creates an ImageStream. However, I'm actually working on a commercial project that uses the Enterprise Edition v3.2. I thought I would be able to apply the workaround there, but on that version, it just says "denied" when I try to push to the internal registry:

[root@master ~]# docker push 172.30.148.0:5000/dcp/logstash:5.0
The push refers to a repository [172.30.148.0:5000/dcp/logstash]
e1ad378129fb: Pushed 
93a100127e3d: Pushed 
e50ed304c0dd: Pushed 
71503a74018d: Pushed 
ce16e390f094: Pushed 
0b383639d4d5: Pushed 
563c5b7b2163: Pushed 
ee099b0f1b68: Pushed 
79a0c35fe416: Pushed 
92d0e56b50e8: Pushed 
71bfa8a0a84e: Pushed 
7d756ba01c56: Pushed 
b5412699bc1b: Pushed 
66d8e5ee400c: Pushed 
2f71b45e4e25: Pushed 
denied

I have setup the permissions in the exact same way:

[root@master ~]# oc describe policyBindings :default -n dcp
Name:                   :default
Created:                2 days ago
Labels:                 <none>
Annotations:                <none>
Last Modified:              2016-08-29 22:26:12 +0000 UTC
Policy:                 <none>
RoleBinding[admin]:          
                    Role:           admin
                    Users:          oneilcr, cluster_admin
                    Groups:         <none>
                    ServiceAccounts:    <none>
                    Subjects:       <none>
RoleBinding[system:deployers]:       
                    Role:           system:deployer
                    Users:          <none>
                    Groups:         <none>
                    ServiceAccounts:    deployer
                    Subjects:       <none>
RoleBinding[system:image-builders]:  
                    Role:           system:image-builder
                    Users:          oneilcr, cluster_admin
                    Groups:         <none>
                    ServiceAccounts:    builder
                    Subjects:       <none>
RoleBinding[system:image-pullers]:   
                    Role:           system:image-puller
                    Users:          <none>
                    Groups:         system:serviceaccounts:dcp
                    ServiceAccounts:    <none>
                    Subjects:       <none>
RoleBinding[system:registry]:        
                    Role:           system:registry
                    Users:          oneilcr, cluster_admin
                    Groups:         <none>
                    ServiceAccounts:    <none>
                    Subjects:       <none>

Even if I'm logged in as a cluster admin it won't let me push on this project. I also cannot push in the openshift project either, even when logged in as a user who has cluster admin permissions.

Unfortunately, for this particular project I'm working on, we need to use Enterprise, and I don't have the option of using a public registry on a temporary basis. We just can't put our code up to a public registry. So it looks like I'm looking for another workaround to the original question :).

jacderida picture jacderida on 30 Aug 2016

This is affecting me, too.
Also using gitlab, and I've tried creating a secret for both registry.company.com & git.company.com, but it doesn't seem to use it all!

My secrets:

git.company.com kubernetes.io/dockercfg 1 9m
registry.company.com kubernetes.io/dockercfg 1 10m

Error when trying to import-image:

message: 'Internal error occurred: Get https://registry.company.com:4000/v2/pieter.steyn/server->config/manifests/latest:
token auth attempt for registry: https://git.company.com:3000/jwt/auth?>scope=repository%3Apieter.steyn%2Fserver-config%3Apull&service=container_registry
request failed with status: 403 Forbidden'

I don't see my username even being passed here.

The only difference here is that I'm using custom ports instead of standard https. So I'll try hack around that next.

It would still be awesome to be able to specify which key to use when doing import-image.
Any advice would be welcome!

psteyn picture psteyn on 31 Aug 2016

Removing the custom ports did not solve the problem for me.

psteyn picture psteyn on 31 Aug 2016

@psteyn If it's an option for you, you can get around it temporarily by just pushing direct to the internal registry. It's actually a pretty good alternative, as pushing there automatically creates an ImageStream.

jacderida picture jacderida on 31 Aug 2016

@jacderida nice to know that's an option, can you give us, that don't know that much about openshit a quick guide on pusing it there? Im not sure if i need to create create some configs first

hernandev picture hernandev on 31 Aug 2016

@hernandev Sure, give me about an hour or so and I'll show you the steps I took to push to the internal registry.

jacderida picture jacderida on 31 Aug 2016

@jacderida sure, take your time, many thanks

hernandev picture hernandev on 31 Aug 2016

@hernandev @psteyn

These steps will push to the internal registry. Hope it's useful for you.

Create a normal user:

[root@master ~]# htpasswd /etc/htpasswd demo
New password: 
Re-type new password: 
Adding password for user demo

Login and create a new project as that user:

[root@master ~]# oc login -u demo -p demo
[root@master ~]# oc new-project test
Now using project "test" on server "https://master.f2qcrn2xk5kete4vnnzyykkywf.ax.internal.cloudapp.net:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    $ oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-hello-world.git

to build a new hello-world application in Ruby.

Configure the user with system:registry and system:image-builder roles. You will need to login as the system:admin user to do this:

[root@master ~]# oc login -u system:admin
You have access to the following projects and can switch between them with 'oc project <projectname>':

  * default
  * logging
  * management-infra
  * openshift
  * openshift-infra
  * private-registry
  * test (current)
  * test-project

Using project "test".

[root@master ~]# oadm policy add-role-to-user system:image-builder demo -n test
[root@master ~]# oadm policy add-role-to-user system:registry demo -n test
[root@master ~]# oadm policy add-role-to-user admin demo -n test
[root@master ~]# oc login -u demo -p demo
Login successful.

Using project "test".
[root@master ~]# oc describe policyBindings :default -n test
Name:                   :default
Created:                9 minutes ago
Labels:                 <none>
Annotations:                <none>
Last Modified:              2016-08-31 18:39:37 +0000 UTC
Policy:                 <none>
RoleBinding[admin]:          
                    Role:           admin
                    Users:          system:admin, demo
                    Groups:         <none>
                    ServiceAccounts:    <none>
                    Subjects:       <none>
RoleBinding[system:deployers]:       
                    Role:           system:deployer
                    Users:          <none>
                    Groups:         <none>
                    ServiceAccounts:    deployer
                    Subjects:       <none>
RoleBinding[system:image-builders]:  
                    Role:           system:image-builder
                    Users:          demo
                    Groups:         <none>
                    ServiceAccounts:    builder
                    Subjects:       <none>
RoleBinding[system:image-pullers]:   
                    Role:           system:image-puller
                    Users:          <none>
                    Groups:         system:serviceaccounts:test
                    ServiceAccounts:    <none>
                    Subjects:       <none>
RoleBinding[system:registry]:        
                    Role:           system:registry
                    Users:          demo
                    Groups:         <none>
                    ServiceAccounts:    <none>
                    Subjects:       <none>

So the demo user is an admin, and has the system:registry and system:image-builder roles.

Now login to the internal registry:

[root@master ~]# docker login -u $(oc whoami) -p $(oc whoami -t) -e [email protected] 172.30.148.0:5000
WARNING: login credentials saved in /root/.docker/config.json
Login Succeeded

Finally, tag and push a test image:

[root@master ~]# docker pull busybox
Using default tag: latest
Trying to pull repository docker.io/library/busybox ... 
latest: Pulling from docker.io/library/busybox
8ddc19f16526: Pull complete 
Digest: sha256:a59906e33509d14c036c8678d687bd4eec81ed7c4b8ce907b888c607f6a1e0e6
Status: Downloaded newer image for docker.io/busybox:latest
[root@master ~]# docker tag busybox 172.30.148.0:5000/test/busybox
[root@master ~]# docker push 172.30.148.0:5000/test/busybox
The push refers to a repository [172.30.148.0:5000/test/busybox]
8ac8bfaff55a: Pushed 
latest: digest: sha256:af4eb8b6a19961203576995c51c7b9bf414def6c8451b9b6e15f47affd043508 size: 2088

You can see the ImageStream it's created:

[root@master ~]# oc get imagestreams
NAME      DOCKER REPO                      TAGS      UPDATED
busybox   172.30.148.0:5000/test/busybox   latest    About a minute ago

You should be able to replace busybox with your own container.

jacderida picture jacderida on 31 Aug 2016

@jacderida on my case, Im interest on push the images from a CI system, and the application itself would be not depending on openshift pushing code, it's already port of the image

would a normal login against the registry works, using the oc credentials?

hernandev picture hernandev on 31 Aug 2016

@hernandev It should work. You would need to expose the registry so that you can push to it from Jenkins or whatever you're using.

https://docs.openshift.com/enterprise/3.2/install_config/install/docker_registry.html#exposing-the-registry

Unfortunately I haven't done that yet, so I can't provide steps. One of my colleagues is working on it.

jacderida picture jacderida on 31 Aug 2016

Btw, I just want to say, "auth delegated to a different server" seems to be a red herring. This occurs regardless of whether you're using separate auth or not.

jacderida picture jacderida on 31 Aug 2016

@jacderida many thanks

hernandev picture hernandev on 31 Aug 2016

@jacderida Thank you for the suggestion.

I initially tried using the internal registry, but it has issues with imagestreams when using Docker 1.10+.
Unfortunately I created my cluster using the openshift-ansible code, and THAT relies on Docker 1.10, and I actually want to keep using ansible in order to create playbooks for our services.

Please see:
https://lists.openshift.redhat.com/openshift-archives/users/2016-June/msg00056.html

(Just realized that I can probably get away with using the internal Docker 1.10, IF I build & push my images using a docker 1.9 server, but still...please read on anyway)

We then looked at Gitlab's registry / CI solution, and it seems ideal, but obviously I can't get it to work.

As a last resort, I've built my own docker registry using Docker 1.9, and this is working.
Unfortunately this adds another layer of complexity into our planned system design, and it would be wonderful if we can solve the gitlab issue, specifically.

PS, auth delegated to a different server may not be a red herring, as the output is clearly different than a normal htpassword secure registry. Perhaps rather meant to say an external oauth instead of htpassword.

psteyn picture psteyn on 1 Sep 2016

Ah ok, apologies. Perhaps it is 2 different issues, because the secrets don't seem to be used even for a simple registry.

jacderida picture jacderida on 1 Sep 2016

@psteyn I've just been hit with that very issue!

Sep 01 17:49:57 node atomic-openshift-node[6898]: E0901 17:49:57.869817    6898 pod_workers.go:138] Error syncing pod 7520d78a-706c-11e6-a97a-000d3a21597c, skipping: failed to "StartContainer" for "datagram" with ErrImagePull: "manifest unknown: manifest unknown"

If I'm understanding this correctly, I could rebuild the container with Docker 1.10 and push it to the registry.

The source container must have came from a Docker 1.9 setup.

jacderida picture jacderida on 1 Sep 2016

@psteyn @jacderida apologies, I was on PTO for the past 2-weeks. I'll see how my commitments look like and will try to address that soonish :wink:

soltysh picture soltysh on 2 Sep 2016

For our use case on shipping images between two clusters
We followed this one
https://blog.openshift.com/remotely-push-pull-container-images-openshift/

and it worked great for 1.2 origin
we have upgraded the cluster lately to 1.3, and now imports are failing
The oc omports are failing reason being unauthorised

message: you may not have access to the Docker image "registry"
      reason: Unauthorized
      status: "False"
      type: ImportSuccess

The logs on docker repository

level=debug msg="anonymous token request" go.version=go1.6.3 http.request.host=registry.test.origin.paas http.request.id=d55c5ece-da69-4ccf-86b6-d1eae19f6305 http.request.method=GET http.request.remot

And finally bails out using with this error on registry

msg="OpenShift client error: User \"system:anonymous\" cannot create localsubjectaccessreviews in project \"core-dev\"" go.version=go1.6.3

and this

level=error msg="error authorizing context: access denied"

I could directly use the service account and the token to sucessfully login into the registry

docker login -u jenkinsa -p token registry.test.origin.paas
Email:
login credentials saved in /root/.docker/config.json
Login Succeeded

Also the docker secret was created using the json file generated above ,once the login on above was sucessfull as

oc secrets new dockersecret .dockerconfigjson=/root/.docker/config.json

oc describe secrets dockersecret
Name:       dockersecret
Namespace:  core
Labels:     <none>
Annotations:    <none>

Type:   kubernetes.io/dockerconfigjson

Data
====
.dockerconfigjson:  1235 bytes

My curosity is if the oc import command is picking up the dockersecret.?, from docker registry logs it seems like it is not. Also is there a way to make oc import imagestreams to flag this dockersecret to use.

At the moment the contents if dockerconfigjson seems like some hash, and i am not sure how credentials of picked up following the docker server name that used to be in the docker json file

apiVersion: v1 data: .dockerconfigjson: ewoJImF1dGhzIjogewoJCSJyZWdpc3RyeS5vcmlnaW4ucGFhcy5lbmlybyI6IHsKCQkJImF1dGgiOiAiYW1WdWEybHVjMkU2WlhsS2FHSkhZMmxQYVVwVFZYcEpNVTVwU1hOSmJsSTFZME5KTmtscmNGaFdRMG81TG1WNVNuQmpNMDFwVDJsS2NtUlhTbXhqYlRWc1pFZFdla3d6VG14amJscHdXVEpXYUZreVRuWmtWelV3U1dsM2FXRXpWbWxhV0VwMVdsaFNiR041TlhCaWVUbDZXbGhLTW1GWFRteFpWMDVxWWpOV2RXUkRPWFZaVnpGc1l6TkNhRmt5VldsUGFVcHJXbGRhYUdSWGVEQkphWGRwWVROV2FWcFlTblZhV0ZKc1kzazFjR0o1T1hwYVdFb3lZVmRPYkZsWFRtcGlNMVoxWkVNNWVscFhUbmxhV0ZGMVltMUdkRnBUU1RaSmJYQnNZbTEwY0dKdVRtaE1XRkoyWVRKV2RVeFVRWGxhZWxGNVNXbDNhV0V6Vm1sYVdFcDFXbGhTYkdONU5YQmllVGw2V2xoS01tRlhUbXhaVjA1cVlqTldkV1JET1hwYVdFb3lZVmRPYkV4WFJtcFpNamt4WW01UmRXSnRSblJhVTBrMlNXMXdiR0p0ZEhCaWJrNW9TV2wzYVdFelZtbGFXRXAxV2xoU2JHTjVOWEJpZVRsNldsaEtNbUZYVG14WlYwNXFZak5XZFdSRE9YcGFXRW95WVZkT2JFeFhSbXBaTWpreFltNVJkV1JYYkd0SmFtOXBUakpGZUUxdFJURlplbEYwVFZSV2JFNTVNSGhOVjFVeVRGZEpkMWxxUlhSTlJFRXhUVVJWTWxsdFVtdE5SRlY2U1dsM2FXTXpWbWxKYW05cFl6TnNlbVJIVm5SUGJrNXNZMjVhY0ZreVZtaFpNazUyWkZjMU1FOXRVbXhhYlVZeFlraFJObUZ0Vm5WaE1teDFZekpGYVdaUkxsQk5PWEZJTTNWSFYwd3piVXAwYkZONVVsQjJlSE5MWDNsZlZXVldPVlZoVERsb1JGWjBibkpCVldwdlUyUnBNbEJGYTAxT1NtOTNTRXQ1VWpod1VVeHdPVW8zV1V4VGRtVnhNVzFtY2tWbVFsUmliWGt3Y1ZGc1UzbEtNRVZrY0hsb1RIb3pWMUp0ZVVGR1VHaFlhbEpvWWtGeFMyeEVlblphWkdoMlRGbE1aRUp0VDJaWWVYRXhXV05NTWt4YVowNHpWbTlTZG05R09WZEhVV04zTWpkbWVEUndjMDB0V0RSblIwMW1hbDl3U3pWUFpHWjJhekZvV2tWcGRucG5ja2hHZVVKRFpYWlFRMDkxUlZsdlRUWldhbEJFTjFSQlZYZzNOV2hZVDFSM2NWQmljMFYwVkc5aU9FWm5iWGhhWW5sQllYa3RaelF4VDFKbVJYVkxOemswVlRWaFpEZFlWemRmZUdwTVZqbEpPRXBsTW1GMVJ6Rkxjazl1WlVjMFluZDRZemgyY3pNMVoyTlJiRU5WTlhRNFMzUXplV2REU2pKTlVUZFpabGRGUzBwRVgydHNPVzF4WHpSZmVXOUNWRTVZVFcxSGR3PT0iLAoJCQkiZW1haWwiOiAiIgoJCX0KCX0KfQo= kind: Secret metadata: creationTimestamp: 2016-10-25T18:02:24Z name: dockersecret namespace: core resourceVersion: "10198535" selfLink: /api/v1/namespaces/core/secrets/dockersecret uid: 2e9c8a66-9add-11e6-ab2a-2c44fd84c3dc type: kubernetes.io/dockerconfigjson```

Please let me know where can i find more logs/info on this failure.. as this setup was working very well in 1.2 (origin), Any workaround would be helpful too.

Also let me know if i can provide more logs in debugging the problem

sudhirpandey picture sudhirpandey on 25 Oct 2016

From the same project i can easily deploy the app using 

oc new-app --docker-image=registry.test.origin.paas/namespace/image:tag,

and the deployment is successfull. so it leads me to believe that oc import image is not picking up the dockersecrtes to sync the image streams from remote registry

sudhirpandey picture sudhirpandey on 27 Oct 2016

I can finally confirm that log level 5 on masters reveals that .. it cannot find the secrete for importing the image

importing remote Docker repository registry=https://registry.test.origin.paasrepository=core-dev/corefront-api insecure=true Unable to find a secret to match https://registry.test.origin.paas:443/openshift/token (registry.test.origin.paas:443/openshift/token) unauthorized: authentication required Retrying request to a v2 Docker registry after encountering error (0 attempts remaining): unauthorized: authentication required unable to get manifest by tag "prod-green" for repository &importer.importRepository{....errcode.Errors{errcode.Error{Code:1002, Message:"authentication required", Detail:[]interface {}{map[string]interface {} ...Message:"authentication required", Detail: --Reason:"Unauthorized", Message:"you may not have access to the Docker image ```

sudhirpandey picture sudhirpandey on 27 Oct 2016

This method is responsible for matching the secret with the remote repository. As you see from the lookup call you need to match the remote endpoint with what you have in the secret, that should do the trick.

soltysh picture soltysh on 27 Oct 2016

yes i have now created docker secret to have
--docker-server=https://registry.test.origin.paas:443/openshift/token and now imagestream import works.

before i could just use --docker-server=registry.test.origin.paas and it used to have worked. so for now being able to import the imagestream and to acutally pull the image from this secured registry , i am having two secrets.

one secret with --docker-server=https://registry.test.origin.paas:443/openshift/token (import imagestream)..
and another with --docker-server=registry.test.origin.paas, which is then assigned to a default service acoount , which actually pulls the image while deploying.

sudhirpandey picture sudhirpandey on 27 Oct 2016
😕1 👍1

Hello,

I still having issue to pull images on openshift from gitlab docker registry when its private.

I have tried creating two secrets as I described above and i dont know if i am missing something.

thank you.

weslleycamilo picture weslleycamilo on 14 Feb 2017

Hi,
I'm facing the same issue using Portus docker registry. Although I've configured secrets as specified in documentation I'm facing Import failed (Unauthorized): you may not have access to the Docker image "internal-registry.example.com/mynamespace/sampleimage" while trying to import image. Using two secrets didn't do the trick for me. Also trying to create new app with docker image specified doesn't work. On loglevel =6 I've got following output and no success:

I0220 08:14:42.993361   21752 dockerimagelookup.go:73] checking remote registry for "internal-registry.example.com/mynamespace/sampleimage"
I0220 08:14:44.163734   21752 round_trippers.go:318] POST https://openshift.example.com:8443/oapi/v1/namespaces/sampleimage/imagestreamimports 201 Created in 1169 milliseconds
I0220 08:14:44.164160   21752 dockerimagelookup.go:208] image import failed: api.ImageImportStatus{Tag:"latest", Status:unversioned.Status{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ListMeta:unversioned.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"you may not have access to the Docker image \"internal-registry.example.com/mynamespace/sampleimage\"", Reason:"Unauthorized", Details:(*unversioned.StatusDetails)(nil), Code:401}, Image:(*api.Image)(nil)}
F0220 08:14:44.164245   21752 helpers.go:110] error: no match for "internal-registry.example.com/mynamespace/sampleimage"
majk-p picture majk-p on 20 Feb 2017

error: no match for "internal-registry.example.com/mynamespace/sampleimage"

This clearly show the logic matching secrets couldn't match the secret with the registry URL. Make sure to double check the URL this code is hitting is the same in the secret.

soltysh picture soltysh on 23 Feb 2017

It's exactly the same as logging into docker registry with the secret data and then doing sudo docker pull internal-registry.example.com/mynamespace/sampleimage works well. Additionally creating pod with raw yaml kubectl create -f pod.yml where internal-registry.example.com/mynamespace/sampleimage is used works as well. The problem only appears when trying to either import image or run oc new-app internal-registry.example.com/mynamespace/sampleimage and only if the sampleimage is not public (requires authentication for download).

majk-p picture majk-p on 23 Feb 2017

I'm having exactly the same issue with Gitlab's Registry. I'm able to reference images from ImageStream, but unable to run a deploy, it fails to pull from the Registry (even though I can see image metadata and everything).

I've also checked oc images and my images are there (?), don't understand why my deploy fails. This is weird.

A good idea might be to check this against Gitlab+Registry combo as Gitlab's popularity will surely expose this functionality to a large number of users. A documentation HOWTO might also be a good idea.

dkarlovi picture dkarlovi on 24 Feb 2017

I'm having exactly the same issue with Gitlab's Registry. I'm able to reference images from ImageStream, but unable to run a deploy, it fails to pull from the Registry (even though I can see image metadata and everything).

@dkarlovi this is because when running your image by default we're using default service account, which does not have access to that remote registry. This mechanism is completely separate from the one used for import. You need to explicitly allow it to do so with:

oc secrets link default <pull_secret_name> --for=pull

You can read more about it here: https://docs.openshift.org/latest/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries

soltysh picture soltysh on 24 Feb 2017

@majk-p to be able to help you more I might need to know exact steps you performed. Iow. did you create the secret using oc secrets new-dockercfg specifying every parameter manually like here or did you use oc secrets new SECRET .dockerconfigjson=path/to/.docker/config.json. What is that registry you're trying to access, is it possible it has double auth like from the initial author of this issue?

soltysh picture soltysh on 24 Feb 2017

@soltysh Actually I've even wrote a script not to forget any single step in the process. It was originally intended to be run as last step of GitLab CI/CD. Here's related part:

echo "Openshift login"
if [[ -n "$OC_CA_PEM" ]]; then
    echo "Certificate found, adding"
    echo "$OC_CA_PEM" > "$TMP_CERT_PATH"
    oc login "${OC_URL}" --token="$OC_TOKEN" --certificate-authority="$TMP_CERT_PATH"
else
    echo "No cert, just token"
    oc login "${OC_URL}" --token="$OC_TOKEN"
fi

if ! oc project "$OC_PROJECT" &> /dev/null; then
  echo "Creating project: $OC_PROJECT..."
  oc new-project "$OC_PROJECT"
  oc project "$OC_PROJECT"
else
  echo "Project ${OC_PROJECT} already exists and will be used"
fi

echo "Deleting old application..."
oc delete all -l "app=${CI_ENVIRONMENT_SLUG}" || true
echo ""

echo "Delete previous secret..."
oc delete secrets/internal-registry
echo "Create secret..."
oc secrets new-dockercfg internal-registry \
    --docker-server="$CI_REGISTRY" --docker-username="$CI_REGISTRY_LOGIN" \
    --docker-password="$CI_REGISTRY_TOKEN" --docker-email="[email protected]"
oc secrets link serviceaccount/default secrets/internal-registry --for=pull
echo ""

echo "Importing image..."
oc delete is "${CI_REGISTRY_IMAGE}"
oc import-image "$CI_REGISTRY_IMAGE" --from="$IMAGE" --confirm
echo ""

echo "Creating application..."
oc new-app "--name=${CI_ENVIRONMENT_SLUG}" --docker-image="$IMAGE"
echo ""

As to your question, how can i test if registry requires double auth? As I've mentioned, I'm forced to use Portus docker registry server.

majk-p picture majk-p on 24 Feb 2017
👍1

@soltysh thanks for your answer. The sa/default already has both secrets attached (gitlab.example.com and registry.example.com, as suggested in this issue).

I can see them both in "Image pull secrets" in oc describe sa default.

dkarlovi picture dkarlovi on 24 Feb 2017

@soltysh Additional finding. After creating an application template deployment works perfectly. The problem seems to affect only the image-import logic as creating deployment from template does not create image stream. Calling oc new-app myregistry.com/namespace/image seems to be implicitly trying to create an image stream and fails as well during image import.

majk-p picture majk-p on 24 Feb 2017

@majk-p thanks for all the input and sorry for the long delay. I guess I haven't played with portus before and it looks tightly related to that repo. I'll try to look into it more.

soltysh picture soltysh on 1 Mar 2017

While looking at https://bugzilla.redhat.com/show_bug.cgi?id=1462606 I've started digging more into how we handle secrets in import vs how it's handled in pulls in k8s. It looks like when are matching the url for the registry we're actually matching against the url that the http client is trying access, and not the url of the server. This leads to the requirement of having two secrets one for the actual registry, and the other one for the authn url. I'll see what is doable wrt to this.

soltysh picture soltysh on 22 Jun 2017
👍1

It seems like exactly the same issue. It would be great to have it fixed, or at least to have a well described workaround.

majk-p picture majk-p on 22 Jun 2017

@majk-p simple workaround is to create 2 secrets: one with the registry address and the other with authn endpoint address. That was proven to be working as expected. See the linked PR, where I've started a dirty hack on a possible approach to solve the problem, though.

soltysh picture soltysh on 23 Jun 2017

See this comment in the linked PR for reasons why. I'm re-assiging this to @miminar.

soltysh picture soltysh on 14 Sep 2017

this has been turned into a trello card: https://trello.com/c/o8tqoSAp/1345-support-importing-from-registries-having-delegated-authorization

bparees picture bparees on 3 Oct 2017

Hi,

I'm having same kind of issue while trying to deal with ImageStream and a private authenticated registry stored in Artifactory. I keep getting a 401 Unauthorized response when trying oc import-image ..... I don't really know why. Also I wondered if all these things with ImageStream makes it possible to use them to PUSH to the registry with secret. I explain myself, I'm trying to use fabric8 maven plugin with a BuildConfig and it looks like it will only deal with ImageStream and that's why I wonder if these can be used to also push to private authenticated registries.

Thank you in advance for your help

Here is how I proceed:

oc describe secret artifactory
Name:       artifactory
Namespace:  projectname
Labels:     <none>
Annotations:    <none>

Type:   kubernetes.io/dockercfg

Data
====
.dockercfg: {"private.docker-registry.example.com":{"password":"superstrongpassword","auth":"cHJqX2l0c19vdsauc2hpZnQ6c3BtZFp5VGNPMTB5M1g4S0hHQmY2cHRQZkNLWDkxaHA=","email":"[email protected]","username":"userwithrights"}}

oc get sa default -o yaml
apiVersion: v1
imagePullSecrets:
- name: artifactory
- name: default-dockercfg-7vpcl
kind: ServiceAccount
metadata:
  creationTimestamp: 2017-08-08T08:47:15Z
  name: default
  namespace: projectname
  resourceVersion: "1835862"
  selfLink: /api/v1/namespaces/projectname/serviceaccounts/default
  uid: 2d7343a6-7c16-11e7-b2f8-005056920908
secrets:
- name: default-token-g2l7f
- name: default-dockercfg-7vpcl


oc import-image imagestream --from=private.docker-registry.example.com --confirm

The import completed successfully.

Name:           imagestream
Namespace:      projectname
Created:        Less than a second ago
Labels:         <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2017-10-12T12:51:40Z
Docker Pull Spec:   <none>
Unique Images:      0
Tags:           1

latest
  tagged from private.docker-registry.example.com

  ! error: Import failed (Unauthorized): you may not have access to the Docker image "private.docker-registry.example.com"
      Less than a second ago
zonArt picture zonArt on 12 Oct 2017

@zonArt You should be able to overcome this by adding the auth server of the private registry to you dockerconfig secret:

{
        "private.docker-registry.example.com": { ... },
        "auth.docker-registry.example.com":    { /* the same as above */ }
}

If you don't know the auth server name, you can:

$ curl  -I https://private.docker-registry.example.com/v2/ |& grep -i www-authenticate
Www-Authenticate: Bearer realm="https://auth.docker-registry.example.com/token",service="registry.docker.io"

Please let me know if it works for you.

miminar picture miminar on 12 Oct 2017
👍2

@miminar i don't think the dockercfg secret was created correctly?

When i create one properly:

$ oc secrets new-dockercfg mysecret --docker-server=docker.io --docker-username=bparees --docker-password=REDACTED --docker-email=REDACTED

my describe shows binary data, not the actual dockercfg json with a cleartext password:

$ oc describe secret mysecret
Name:       mysecret
Namespace:  test
Labels:     <none>
Annotations:    <none>

Type:   kubernetes.io/dockercfg

Data
====
.dockercfg: 121 bytes

@zonArt how did you create your dockercfg secret?

bparees picture bparees on 12 Oct 2017

(and i don't think we actually know that @zonArt's registry is using delegated auth?)

bparees picture bparees on 12 Oct 2017

@bparees I created the secret with a java json node object sent via a RestTemplate to the openshift rest api. Anyway, I just did the exact same thing as you and still got the same behavior:

~/w/E/g/workspace (master|✚3…) $ oc secrets new-dockercfg mysecret --docker-server=docker.io --docker-username=bparees --docker-password=REDACTED --docker-email=REDACTED
secret/mysecret
~/w/E/g/workspace (master|✚3…) $ oc describe secret mysecret
Name:       mysecret
Namespace:  projectname
Labels:     <none>
Annotations:    <none>

Type:   kubernetes.io/dockercfg

Data
====
.dockercfg: {"docker.io":{"username":"bparees","password":"REDACTED","email":"REDACTED","auth":"YnBhcmVlczpSRURBQ1RFRA=="}}

And yes you're right I don't have any delegated auth service in place.
Maybe some more information could help:

oc version
oc v1.3.0
kubernetes v1.3.0+52492b4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://openshift.example.com:8443
openshift v1.5.1
kubernetes v1.5.2+43a9be4

Should I maybe align my oc binary version with openshift's version ?

zonArt picture zonArt on 12 Oct 2017

I wonder if this is another flavor of https://bugzilla.redhat.com/show_bug.cgi?id=1476330

in which the secret k8s creates is not the right format for artifactory.

@zonArt Can you try the workaround suggested in https://bugzilla.redhat.com/show_bug.cgi?id=1476330#c1 ?

bparees picture bparees on 12 Oct 2017

Still the same (but not exactly). Is it a problem if different kind of secrets are linked to a serviceaccount ?
What happened now is the following:

oc import-image anewtest --from=private.docker-registry.example.com --confirm
The import completed successfully.

Name:           anewtest
Namespace:      projectname
Created:        Less than a second ago
Labels:         <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2017-10-12T15:07:09Z
Docker Pull Spec:   <none>
Unique Images:      0
Tags:           1

latest
  tagged from private.docker-registry.example.com

  ! error: Import failed (InternalError): Internal error occurred: Get https://registry-1.docker.io/v2/library/private.docker-registry.example.com/manifests/latest: unauthorized: incorrect username or password
      Less than a second ago

Notice that the --from=private.docker-registry.example.com might be wrong as it is not pointing directly to an image but rather to the root of the registry, but I made the same mistake before and the error was different so maybe I'm coming closer. After trying to put the image location, the issue remains exactly the same:

oc delete is anewtest
imagestream "anewtest" deleted
~/w/E/g/workspace (master|✚3…) $ oc import-image anewtest --from=private.docker-registry.example.com/myimage --confirm
The import completed successfully.

Name:           anewtest
Namespace:      projectname
Created:        Less than a second ago
Labels:         <none>
Annotations:        openshift.io/image.dockerRepositoryCheck=2017-10-12T15:08:01Z
Docker Pull Spec:   <none>
Unique Images:      0
Tags:           1

latest
  tagged from private.docker-registry.example.com/myimage

  ! error: Import failed (Unauthorized): you may not have access to the Docker image "private.docker-registry.example.com/myimage"
      Less than a second ago
zonArt picture zonArt on 12 Oct 2017

@miminar maybe you're right somehow, I just tried your suggestion and here is what I got:

curl -sI https://private.docker-registry.example.com/v2/ | grep -i www-authenticate
WWW-Authenticate: Bearer realm="https://docker-registry.example.com:443/artifactory/api/docker/private/v2/token",service="docker-registry.example.com:443"

Is that expected ? Should I add anything special in my secret to handle that ?

zonArt picture zonArt on 12 Oct 2017

One more thing I wanted to share is that with my existing "artifactory" secret, I have no issue pulling images within a DeploymentConfig referencing the full path to the registry (without going through an ImageStream)

zonArt picture zonArt on 12 Oct 2017

you definitely need to specify the --from as "registry.hostname.com/namespace/repositoryname" (the same way you'd specify the image if you were doing a "docker pull", but leave off the tag.

is private.docker-registry.example.com/myimage that value? i don't see a namespace path segment which makes me suspicious. Can you share the yaml for the deploymentconfig that is working for you? Is that in the same namespace as the imagestream?

bparees picture bparees on 12 Oct 2017

yes it is the value, here is a DC which is working fine:

apiVersion: v1
kind: DeploymentConfig
metadata:
  name: docker-demo-app
  namespace: projectname
spec:
  replicas: 1
  selector:
    name: docker-demo-app
  strategy:
    resources: {}
    rollingParams:
      intervalSeconds: 1
      maxSurge: 25%
      maxUnavailable: 25%
      timeoutSeconds: 600
      updatePeriodSeconds: 1
    type: Rolling
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: docker-demo-app
    spec:
      containers:
      - image: private.docker-registry.example.com/myimage
        imagePullPolicy: Always
        name: docker-demo-app
        terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
  test: false
  triggers:
  - type: ConfigChange
zonArt picture zonArt on 12 Oct 2017

@miminar well i'm out of ideas... any known issues in v1.5 (the server level @zonArt is running) w/ respect to imagestream import? Any logging we can enable to ensure the secret is being used during import?

@zonArt regarding why your deploymentconfig works, is there any chance the image already existed on the node? (in which case it didn't actually pull it using the secret)

bparees picture bparees on 12 Oct 2017

@bparees not at all it has always been working this way and as I said, when the registry is accessed/required from let say "directly" (with spec.containers.image for instance), I never had any problem pulling it. It's just recently that I tried to used ImageStream (I first thought it was only possible with the openshift internal registry though never used them before) and experienced the issue.

I really think that the secret is absolutely not taken into account as when I check our Artifactory server's logs, the GET request is made with Anonymous user.

If you can point me to other kind of logs I can look at, I'll provide you details

zonArt picture zonArt on 13 Oct 2017

@zonArt From your prior comment:

tagged from private.docker-registry.example.com

! error: Import failed (InternalError): Internal error occurred: Get https://registry-1.docker.io/v2/library/private.docker-registry.example.com/manifests/latest: unauthorized: incorrect username or password
Less than a second ago

Looks inauthentic :-)

WWW-Authenticate: Bearer realm="https://docker-registry.example.com:443/artifactory/api/docker/private/v2/token",service="docker-registry.example.com:443"

I believe the port part prevents the keyring from matching the credentials. Can you please add the entry with the port to your dockerconfig secret? Or modify your proxy/lb in front of your registry to send port-less location?

miminar picture miminar on 13 Oct 2017

@miminar I added port and protocol and nothing changed

zonArt picture zonArt on 16 Oct 2017

One more question:
Regarding ImageStream object proxying a private authenticated registry, once I'll manage to get the secret working (if I get it one day), will it work to push to this registry ?
This is the finality of my problem, actually, as I pull using the registry's url and not the ImageStream (during build process)

zonArt picture zonArt on 16 Oct 2017

I'm not exactly sure what you're asking, but the secret that you use to import the image should also be usable to push to the source registry (ie if you reference the secret in a buildconfig that has your source registry as the output image target).

bparees picture bparees on 16 Oct 2017

Hi,

Regarding the the possibility to push to an ImageStream (thanks to a BuildConfig) I got no problem in doing so by setting the pushSecret parametre. The issue was on the fabric8-maven-plugin which was overriding my BC (a PR was opened to propose a fix even if I'm not hundred per cent sure it's the right approach).

Regarding the pull secret, I'm still not able to get the tags from the ImageStream but I probably miss something. If you could suggest some logs and/or config files I could provide to help diagnose the issue further, I would appreciate.

zonArt picture zonArt on 18 Oct 2017

@miminar does image import do any useful logging that would show us if the secret is being used when we try to do the import?

@zonArt the buildconfig that successfully did the push is in the same project as the imagestream that can't import?

bparees picture bparees on 18 Oct 2017

@bparees yes it is, having this BC:

apiVersion: v1
kind: BuildConfig
metadata:
  name: buildconfig
  namespace: projectname
spec:
  output:
    pushSecret:
      name: artifactory
    to:
      kind: ImageStream
      name: imagestream
  postCommit: {}
  resources: {}
  runPolicy: Serial
  source:
    binary: {}
    type: Binary
  strategy:
    dockerStrategy: {}
    type: Docker

makes the build success without any issue (thanks to the pushSecret section)

zonArt picture zonArt on 18 Oct 2017

@miminar does image import do any useful logging that would show us if the secret is being used when we try to do the import?

@zonArt you can try to run the import cmd with --loglevel=5 which may reveal some more info.

miminar picture miminar on 19 Oct 2017

Here is the info: (I see nothing really relevant though)

The import completed successfully.

Name:                   imagename
Namespace:              projectname
Created:                47 hours ago
Labels:                 app=imagename
                        group=com.sca.fo
                        provider=fabric8
                        version=0.9.7.0-SNAPSHOT
Annotations:            openshift.io/image.dockerRepositoryCheck=2017-10-18T10:13:40Z
Docker Pull Spec:       private.docker-registry.example.com/imagename
Unique Images:          0
Tags:                   1

latest
  tagged from private.docker-registry.example.com/imagename

  ~ importing latest image ...
  ! error: Import failed (Unauthorized): you may not have access to the Docker image "private.docker-registry.example.com/imagename"
      43 hours ago

Even with loglevel 8:

I1019 11:13:20.056412   25102 loader.go:330] Config loaded from file /home/nicko/.kube/config
I1019 11:13:20.057152   25102 round_trippers.go:296] GET https://openshift.example.com:443/api
I1019 11:13:20.057176   25102 round_trippers.go:303] Request Headers:
I1019 11:13:20.057182   25102 round_trippers.go:306]     Accept: application/json, */*
I1019 11:13:20.057215   25102 round_trippers.go:306]     User-Agent: oc/v1.3.0+52492b4 (linux/amd64) kubernetes/52492b4
I1019 11:13:20.057234   25102 round_trippers.go:306]     Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXX
I1019 11:13:20.148693   25102 round_trippers.go:321] Response Status: 200 OK in 91 milliseconds
I1019 11:13:20.148748   25102 round_trippers.go:324] Response Headers:
I1019 11:13:20.149010   25102 round_trippers.go:327]     Cache-Control: no-store
I1019 11:13:20.149094   25102 round_trippers.go:327]     Content-Type: application/json
I1019 11:13:20.149260   25102 round_trippers.go:327]     Content-Length: 146
I1019 11:13:20.149301   25102 round_trippers.go:327]     Date: Thu, 19 Oct 2017 09:13:20 GMT
I1019 11:13:20.149496   25102 request.go:901] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]}
I1019 11:13:20.150642   25102 round_trippers.go:296] GET https://openshift.example.com:443/apis
I1019 11:13:20.150768   25102 round_trippers.go:303] Request Headers:
I1019 11:13:20.150785   25102 round_trippers.go:306]     Accept: application/json, */*
I1019 11:13:20.150791   25102 round_trippers.go:306]     User-Agent: oc/v1.3.0+52492b4 (linux/amd64) kubernetes/52492b4
I1019 11:13:20.150795   25102 round_trippers.go:306]     Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXX
I1019 11:13:20.166422   25102 round_trippers.go:321] Response Status: 200 OK in 15 milliseconds
I1019 11:13:20.166473   25102 round_trippers.go:324] Response Headers:
I1019 11:13:20.166553   25102 round_trippers.go:327]     Cache-Control: no-store
I1019 11:13:20.166591   25102 round_trippers.go:327]     Content-Type: application/json
I1019 11:13:20.166599   25102 round_trippers.go:327]     Content-Length: 2284
I1019 11:13:20.166616   25102 round_trippers.go:327]     Date: Thu, 19 Oct 2017 09:13:20 GMT
I1019 11:13:20.166675   25102 request.go:901] Response Body: {"kind":"APIGroupList","groups":[{"name":"apps","versions":[{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v2alpha1","version":"v2alpha1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]},{"name":"certificates.k8s.io","versions":[{"groupVersion":"certificates.k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"certificates.k8s.io/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]},{"name":"policy","versions":[{"groupVersion":"policy/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"policy/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]},{"name":"storage.k8s.io","versions":[{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"openshift.example.com:8443"}]}]}
I1019 11:13:20.171753   25102 round_trippers.go:296] GET https://openshift.example.com:443/oapi
I1019 11:13:20.172030   25102 round_trippers.go:303] Request Headers:
I1019 11:13:20.172154   25102 round_trippers.go:306]     Accept: application/json, */*
I1019 11:13:20.172237   25102 round_trippers.go:306]     User-Agent: oc/v1.3.0 (linux/amd64) openshift/3ab7af3
I1019 11:13:20.172374   25102 round_trippers.go:306]     Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXX
I1019 11:13:20.181403   25102 round_trippers.go:321] Response Status: 200 OK in 8 milliseconds
I1019 11:13:20.181458   25102 round_trippers.go:324] Response Headers:
I1019 11:13:20.181470   25102 round_trippers.go:327]     Content-Type: application/json
I1019 11:13:20.181481   25102 round_trippers.go:327]     Content-Length: 93
I1019 11:13:20.181488   25102 round_trippers.go:327]     Date: Thu, 19 Oct 2017 09:13:20 GMT
I1019 11:13:20.181497   25102 round_trippers.go:327]     Cache-Control: no-store
I1019 11:13:20.181535   25102 request.go:901] Response Body: {"kind":"APIVersions","apiVersion":"v1","versions":["v1"],"serverAddressByClientCIDRs":null}
I1019 11:13:20.181942   25102 round_trippers.go:296] GET https://openshift.example.com:443/oapi/v1/namespaces/projectname/imagestreams/imagename
I1019 11:13:20.182013   25102 round_trippers.go:303] Request Headers:
I1019 11:13:20.182031   25102 round_trippers.go:306]     Accept: application/json, */*
I1019 11:13:20.182041   25102 round_trippers.go:306]     User-Agent: oc/v1.3.0 (linux/amd64) openshift/3ab7af3
I1019 11:13:20.182051   25102 round_trippers.go:306]     Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXX
I1019 11:13:20.218828   25102 round_trippers.go:321] Response Status: 200 OK in 36 milliseconds
I1019 11:13:20.218860   25102 round_trippers.go:324] Response Headers:
I1019 11:13:20.218867   25102 round_trippers.go:327]     Content-Length: 1255
I1019 11:13:20.218873   25102 round_trippers.go:327]     Date: Thu, 19 Oct 2017 09:13:20 GMT
I1019 11:13:20.218879   25102 round_trippers.go:327]     Cache-Control: no-store
I1019 11:13:20.218884   25102 round_trippers.go:327]     Content-Type: application/json
I1019 11:13:20.219792   25102 request.go:901] Response Body: {"kind":"ImageStream","apiVersion":"v1","metadata":{"name":"imagename","namespace":"projectname","selfLink":"/oapi/v1/namespaces/projectname/imagestreams/imagename","uid":"48da075a-b31f-11e7-b4b4-005056920908","resourceVersion":"5544817","generation":61,"creationTimestamp":"2017-10-17T09:41:00Z","labels":{"app":"imagename","group":"com.sca.fo","provider":"fabric8","version":"0.9.7.0-SNAPSHOT"},"annotations":{"openshift.io/image.dockerRepositoryCheck":"2017-10-18T10:13:40Z"}},"spec":{"dockerImageRepository":"private.docker-registry.example.com/imagename","tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"private.docker-registry.example.com/imagename"},"generation":61,"importPolicy":{},"referencePolicy":{"type":"Source"}}]},"status":{"dockerImageRepository":"private.docker-registry.example.com/imagename","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2017-10-17T14:02:04Z","reason":"Unauthorized","message":"you may not have access to the Docker image \"private.docker-registry.example.com/imagename\"","generation":55}]}]}}
I1019 11:13:20.220956   25102 request.go:562] Request Body: {"kind":"ImageStreamImport","apiVersion":"v1","metadata":{"name":"imagename","namespace":"projectname","resourceVersion":"5544817","creationTimestamp":null},"spec":{"import":true,"images":[{"from":{"kind":"DockerImage","name":"private.docker-registry.example.com/imagename"},"to":{"name":"latest"},"importPolicy":{}}]},"status":{}}
I1019 11:13:20.221118   25102 round_trippers.go:296] POST https://openshift.example.com:443/oapi/v1/namespaces/projectname/imagestreamimports
I1019 11:13:20.221157   25102 round_trippers.go:303] Request Headers:
I1019 11:13:20.221244   25102 round_trippers.go:306]     Accept: application/json, */*
I1019 11:13:20.221326   25102 round_trippers.go:306]     Content-Type: application/json
I1019 11:13:20.221397   25102 round_trippers.go:306]     User-Agent: oc/v1.3.0 (linux/amd64) openshift/3ab7af3
I1019 11:13:20.221404   25102 round_trippers.go:306]     Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXX
I1019 11:13:21.386356   25102 round_trippers.go:321] Response Status: 201 Created in 1164 milliseconds
I1019 11:13:21.386525   25102 round_trippers.go:324] Response Headers:
I1019 11:13:21.386586   25102 round_trippers.go:327]     Content-Type: application/json
I1019 11:13:21.386641   25102 round_trippers.go:327]     Content-Length: 1898
I1019 11:13:21.386693   25102 round_trippers.go:327]     Date: Thu, 19 Oct 2017 09:13:21 GMT
I1019 11:13:21.386729   25102 round_trippers.go:327]     Cache-Control: no-store
I1019 11:13:21.387031   25102 request.go:901] Response Body: {"kind":"ImageStreamImport","apiVersion":"v1","metadata":{"name":"imagename","namespace":"projectname","selfLink":"/oapi/v1/namespaces/projectname/imagestreamimports/imagename","uid":"c010cd09-b4ad-11e7-b4b4-005056920908","resourceVersion":"5544817","creationTimestamp":"2017-10-19T09:13:20Z"},"spec":{"import":true,"images":[{"from":{"kind":"DockerImage","name":"private.docker-registry.example.com/imagename"},"to":{"name":"latest"},"importPolicy":{}}]},"status":{"import":{"metadata":{"name":"imagename","namespace":"projectname","uid":"48da075a-b31f-11e7-b4b4-005056920908","resourceVersion":"5544817","generation":61,"creationTimestamp":"2017-10-17T09:41:00Z","labels":{"app":"imagename","group":"com.sca.fo","provider":"fabric8","version":"0.9.7.0-SNAPSHOT"},"annotations":{"openshift.io/image.dockerRepositoryCheck":"2017-10-18T10:13:40Z"}},"spec":{"dockerImageRepository":"private.docker-registry.example.com/imagename","tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"private.docker-registry.example.com/imagename"},"generation":61,"importPolicy":{},"referencePolicy":{"type":"Source"}}]},"status":{"dockerImageRepository":"private.docker-registry.example.com/imagename","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2017-10-17T14:02:04Z","reason":"Unauthorized","message":"you may not have access to the Docker image \"private.docker-registry.example.com/imagename\"","generation":55}]}]}},"images":[{"status":{"metadata":{},"status":"Failure","message":"you may not have access to the Docker image \"private.docker-registry.example.com/imagename\"","reason":"Unauthorized","code":401},"tag":"latest"}]}}
I1019 11:13:21.391126   25102 round_trippers.go:296] GET https://openshift.example.com:443/oapi/v1/namespaces/projectname/imagestreams/imagename
I1019 11:13:21.391417   25102 round_trippers.go:303] Request Headers:
I1019 11:13:21.391485   25102 round_trippers.go:306]     User-Agent: oc/v1.3.0 (linux/amd64) openshift/3ab7af3
I1019 11:13:21.391537   25102 round_trippers.go:306]     Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXX
I1019 11:13:21.391594   25102 round_trippers.go:306]     Accept: application/json, */*
I1019 11:13:21.420987   25102 round_trippers.go:321] Response Status: 200 OK in 29 milliseconds
I1019 11:13:21.421122   25102 round_trippers.go:324] Response Headers:
I1019 11:13:21.421159   25102 round_trippers.go:327]     Cache-Control: no-store
I1019 11:13:21.421189   25102 round_trippers.go:327]     Content-Type: application/json
I1019 11:13:21.421215   25102 round_trippers.go:327]     Content-Length: 1255
I1019 11:13:21.421245   25102 round_trippers.go:327]     Date: Thu, 19 Oct 2017 09:13:21 GMT
I1019 11:13:21.421470   25102 request.go:901] Response Body: {"kind":"ImageStream","apiVersion":"v1","metadata":{"name":"imagename","namespace":"projectname","selfLink":"/oapi/v1/namespaces/projectname/imagestreams/imagename","uid":"48da075a-b31f-11e7-b4b4-005056920908","resourceVersion":"5544817","generation":61,"creationTimestamp":"2017-10-17T09:41:00Z","labels":{"app":"imagename","group":"com.sca.fo","provider":"fabric8","version":"0.9.7.0-SNAPSHOT"},"annotations":{"openshift.io/image.dockerRepositoryCheck":"2017-10-18T10:13:40Z"}},"spec":{"dockerImageRepository":"private.docker-registry.example.com/imagename","tags":[{"name":"latest","annotations":null,"from":{"kind":"DockerImage","name":"private.docker-registry.example.com/imagename"},"generation":61,"importPolicy":{},"referencePolicy":{"type":"Source"}}]},"status":{"dockerImageRepository":"private.docker-registry.example.com/imagename","tags":[{"tag":"latest","items":null,"conditions":[{"type":"ImportSuccess","status":"False","lastTransitionTime":"2017-10-17T14:02:04Z","reason":"Unauthorized","message":"you may not have access to the Docker image \"private.docker-registry.example.com/imagename\"","generation":55}]}]}}
The import completed successfully.

Name:           imagename
Namespace:      projectname
Created:        47 hours ago
Labels:         app=imagename
            group=com.sca.fo
            provider=fabric8
            version=0.9.7.0-SNAPSHOT
Annotations:        openshift.io/image.dockerRepositoryCheck=2017-10-18T10:13:40Z
Docker Pull Spec:   private.docker-registry.example.com/imagename
Unique Images:      0
Tags:           1

latest
  tagged from private.docker-registry.example.com/imagename

  ~ importing latest image ...
  ! error: Import failed (Unauthorized): you may not have access to the Docker image "private.docker-registry.example.com/imagename"
      43 hours ago
zonArt picture zonArt on 19 Oct 2017

@zonArt I'm sorry, the log from the master service needs to be captured from the time of import with --loglevel=5.

miminar picture miminar on 19 Oct 2017

@miminar Mmmmh ok, where are these logs located ? I'm having quite a hard time to figure it out, looks like everything gets spread in daemon.log, is that possible ? And there is nothing in daemon.log related to import-image

zonArt picture zonArt on 19 Oct 2017

@zonArt do you run the cluster containerized? If yes, you need to docker logs <name-your-master-container>. If you run the openshift-master as a service (using e.g. systemd), you will need:

$ journalctl -u atomic-openshift-master.service -f | tee image-import.log
$ oc import-image --from=...  ...`
miminar picture miminar on 19 Oct 2017

@miminar thanks, here is the linked log output:

oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.111972   49915 panics.go:76] PUT /api/v1/nodes/openshift.example.com/status: (8.546383ms) 200 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37192]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.126211   49915 panics.go:76] GET /api/v1/nodes?resourceVersion=0: (1.260212ms) 200 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.372724   49915 panics.go:76] GET /api: (2.46666ms) 200 [[oc/v1.3.0+52492b4 (linux/amd64) kubernetes/52492b4] xx.xx.xxx.xx:52931]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.378061   49915 panics.go:76] GET /apis: (2.888212ms) 200 [[oc/v1.3.0+52492b4 (linux/amd64) kubernetes/52492b4] xx.xx.xxx.xx:52931]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.392347   49915 panics.go:76] GET /oapi: (2.966155ms) 200 [[oc/v1.3.0 (linux/amd64) openshift/3ab7af3] xx.xx.xxx.xx:52931]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.400887   49915 panics.go:76] GET /api/v1/namespaces/default/services/docker-registry: (1.231769ms) 404 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.402395   49915 panics.go:76] GET /api/v1/namespaces/default/services/docker-registry: (936.554µs) 404 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.402904   49915 panics.go:76] GET /oapi/v1/namespaces/projectname/imagestreams/imagename: (8.08406ms) 200 [[oc/v1.3.0 (linux/amd64) openshift/3ab7af3] xx.xx.xxx.xx:52931]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.458501   49915 panics.go:76] GET /api/v1/namespaces/projectname/resourcequotas: (1.685423ms) 200 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172] 
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.482718   49915 panics.go:76] GET /api/v1/namespaces/projectname/secrets: (7.610672ms) 200 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:09 openshift origin-master[49915]: I1019 15:28:09.483875   49915 panics.go:76] GET /oapi/v1/namespaces/projectname/imagestreams/imagename/secrets: (9.783863ms) 200 [[openshift/v1.5.1 (linux/amd64) openshift/7b451fc] xxx.xx.xxx.xxx:37172]
oct 19 15:28:10 openshift origin-master[49915]: I1019 15:28:10.519661   49915 panics.go:76] GET /api/v1/namespaces/default/services/docker-registry: (1.552176ms) 404 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:10 openshift origin-master[49915]: I1019 15:28:10.522288   49915 panics.go:76] GET /api/v1/namespaces/default/services/docker-registry: (1.832307ms) 404 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:10 openshift origin-master[49915]: I1019 15:28:10.523486   49915 trace.go:61] Trace "Create /oapi/v1/namespaces/projectname/imagestreamimports" (started 2017-10-19 15:28:09.417974961 +0200 CEST):
oct 19 15:28:10 openshift origin-master[49915]: [37.877215ms] [37.877215ms] About to convert to expected version
oct 19 15:28:10 openshift origin-master[49915]: [38.019256ms] [142.041µs] Conversion done 
oct 19 15:28:10 openshift origin-master[49915]: [41.246518ms] [3.227262ms] About to store object in database
oct 19 15:28:10 openshift origin-master[49915]: [1.105178634s] [1.063932116s] Object stored in database
oct 19 15:28:10 openshift origin-master[49915]: [1.105189942s] [11.308µs] Self-link added
oct 19 15:28:10 openshift origin-master[49915]: "Create /oapi/v1/namespaces/projectname/imagestreamimports" [1.105465153s] [275.211µs] END
oct 19 15:28:10 openshift origin-master[49915]: I1019 15:28:10.523543   49915 panics.go:76] POST /oapi/v1/namespaces/projectname/imagestreamimports: (1.108359098s) 201 [[oc/v1.3.0 (linux/amd64) openshift/3ab7af3] xx.xx.xxx.xx:52931]
oct 19 15:28:10 openshift origin-master[49915]: I1019 15:28:10.538851   49915 panics.go:76] GET /api/v1/namespaces/default/services/docker-registry: (1.943446ms) 404 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:10 openshift origin-master[49915]: I1019 15:28:10.541654   49915 panics.go:76] GET /api/v1/namespaces/default/services/docker-registry: (1.886461ms) 404 [[openshift/v1.5.2+43a9be4 (linux/amd64) kubernetes/43a9be4] xxx.xx.xxx.xxx:37172]
oct 19 15:28:10 openshift origin-master[49915]: I1019 15:28:10.542445   49915 panics.go:76] GET /oapi/v1/namespaces/projectname/imagestreams/imagename: (10.077725ms) 200 [[oc/v1.3.0 (linux/amd64) openshift/3ab7af3] xx.xx.xxx.xx:52931]

Still not sure to find any interesting info, except on the fact it tries to reach the inexistent internal registry (which doesn't really help by the way)

zonArt picture zonArt on 19 Oct 2017

It case it may interest anybody: I found the solution in RedHat KB:
https://access.redhat.com/solutions/3019071
I never thought you needed to also specify the port in case it runs on 443. I therefore added a comment on the above mentioned article.

Thanks again for your support

zonArt picture zonArt on 8 Nov 2017

@miminar

@zonArt

You should be able to overcome this by adding the auth server of the private registry to you
dockerconfig secret:

{
"private.docker-registry.example.com": { ... },
"auth.docker-registry.example.com": { /* the same as above */ }
}

If you don't know the auth server name, you can:

$ curl -I https://private.docker-registry.example.com/v2/ |& grep -i www-authenticate
Www-Authenticate: Bearer realm="https://auth.docker-registry.example.com/token",service="registry.docker.io"

Please let me know if it works for you.

This is actually working, thanks

zonArt picture zonArt on 31 Jan 2018

@jperville jperville thanks for posting the solution! This works for me also (y)

andutt picture andutt on 28 May 2019

Thanks @zonArt , this is the only solution that works for me!

nqbao picture nqbao on 1 Aug 2019

For the convenience of people finding this through a search engine, here is a condensed answer. Thank you to everyone above for the excellent guidance.

Suppose you want an ImageStream that references a secured external registry such as GitLab, and the auth for said registry is done through a different hostname e.g. gitlab.com rather than registry.gitlab.com. The following has been tested in OpenShift Online 4.2.

First we find out the exact URI of the auth server with:

$ curl -sI https://registry.gitlab.com/v2/ | grep -i Www-Authenticate

Www-Authenticate: Bearer realm="https://gitlab.com/jwt/auth",service="container_registry"

We will therefore use https://gitlab.com/jwt/auth below.

Create secret

Have a dockerconfig.json file with contents like:

{"auths":
    {
        "registry.gitlab.com":{"username":"AzureDiamond","password":"hunter2","email":"[email protected]","auth":"QXp1cmVEaWFtb25kOmh1bnRlcjI="},
        "https://gitlab.com/jwt/auth":{"username":"AzureDiamond","password":"hunter2","email":"[email protected]","auth":"QXp1cmVEaWFtb25kOmh1bnRlcjI="}
    }
}

Then do:

kubectl create secret generic regsecret --from-file=.dockerconfigjson=dockerconfig.json --type=kubernetes.io/dockerconfigjson

Important: It seems necessary to name it regsecret. I haven't had success if the secret has any other name. If anyone can think why, please say.

Set default service account to use this secret:

kubectl get serviceaccounts default -o json |
     jq 'del(.metadata.resourceVersion)'|
     jq 'setpath(["imagePullSecrets"];[{"name":"regsecret"}])' |
     kubectl replace serviceaccount default -f -

Verify:

$ kubectl get serviceaccount default -o yaml

apiVersion: v1
imagePullSecrets:
- name: regsecret
- name: default-dockercfg-ls2wj
kind: ServiceAccount
... snip ...

Create the image stream

kind: ImageStream
apiVersion: image.openshift.io/v1
metadata:
  name: foobar
spec:
  tags:
    - name: latest
      from:
        kind: DockerImage
        name: registry.gitlab.com/foobar:latest

You should now have a working ImageStream.

3dbrows picture 3dbrows on 23 Jan 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

smarterclayton picture smarterclayton  ·  72Comments
alejandronb picture alejandronb  ·  51Comments
csrwng picture csrwng  ·  46Comments
stevekuznetsov picture stevekuznetsov  ·  67Comments
rhcarvalho picture rhcarvalho  ·  51Comments