Ingress-nginx: harbor fialed to push images wien nginx ingress controller

[garyyang85]

I am using harbor as my docker registry. Now docker login works with below log:

127.0.0.1 - [127.0.0.1] - - [18/Aug/2017:23:55:57 +0000] "GET /v2/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 262 0.000 [upstream-default-backend] 10.254.74.11:8080 21 0.000 404 127.0.0.1 - [127.0.0.1] - - [18/Aug/2017:23:55:57 +0000] "GET /v2/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 262 0.001 [upstream-default-backend] 10.254.74.11:8080 21 0.001 404 9.x.x.x - [9.x.x.x] - - [18/Aug/2017:23:55:57 +0000] "GET /v2/ HTTP/1.1" 401 87 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 262 0.006 [default-harbor-80] 10.254.4.5:80 87 0.006 401 9.x.x.x - [9.x.x.x] - admin [18/Aug/2017:23:55:57 +0000] "GET /service/token?account=admin&client_id=docker&offline_token=true&service=token-service HTTP/1.1" 200 906 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 391 0.040 [default-harbor-80] 10.254.4.5:80 1100 0.040 200 9.x.x.x - [9.x.x.x] - - [18/Aug/2017:23:55:57 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 1308 0.005 [default-harbor-80] 10.254.4.5:80 2 0.005 200

but docker push failed with error response:
c0de73ac9968: Preparing
Error: Status 404 trying to push repository library/mytest: "default backend - 404"

logs:
127.0.0.1 - [127.0.0.1] - - [18/Aug/2017:23:58:02 +0000] "GET /v2/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 262 0.001 [upstream-default-backend] 10.254.74.11:8080 21 0.001 404 127.0.0.1 - [127.0.0.1] - - [18/Aug/2017:23:58:02 +0000] "POST /v2/library/mytest/blobs/uploads/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 327 0.001 [upstream-default-backend] 10.254.74.11:8080 21 0.001 404 127.0.0.1 - [127.0.0.1] - - [18/Aug/2017:23:58:03 +0000] "GET /v1/_ping HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 267 0.001 [upstream-default-backend] 10.254.74.11:8080 21 0.001 404 127.0.0.1 - [127.0.0.1] - admin [18/Aug/2017:23:58:03 +0000] "PUT /v1/repositories/library/mytest/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/3.10.0-514.2.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 501 0.001 [upstream-default-backend] 10.254.74.11:8080 21 0.001 404

I am using image nginx-ingress-controller:0.9.0-beta.11, when replace the ingress controller with nginxdemos/nginx-ingress, it works. I am trying to understand the every detail parameters in these two images, but seems they are not the same. When I tried to change the parameters in our nginx image, it reports unknown parameters...

[aledbf]

@garyyang85 the ingress controllers are not compatible because are two different projects.

[garyyang85]

@aledbf any thoughts about this issue? I can access the URL http://harbor.default.example.com, and also docker login harbor.default.example.com and docker pull/push "service-ip"/library/busybox, but when docker push/pull harbor.default.example.com/library/busybox, it reports error. It always goes to the default back end, but not the harbor.default.example.com upstream server.

[aledbf]

@garyyang85 please post the ingress you defined. Did you increase the size of uploads?

[garyyang85]

@aledbf , the ingress is defined as below. Also tried to delete one host, does not work:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/configuration-snippet: proxy_set_header Host $host;
    ingress.kubernetes.io/proxy-body-size: "0"
  creationTimestamp: 2017-07-06T08:28:14Z
  name: harbor
  namespace: default
  resourceVersion: "6728038"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/harbor
  uid: 0e2f30d1-6225-11e7-95a8-fa163e23f57d
spec:
  rules:
  - host: harbor.default.example.com
    http:
      paths:
      - backend:
          serviceName: harbor
          servicePort: 80
  - host: harbor.default.example.com
    http:
      paths:
      - backend:
          serviceName: harbor
          servicePort: 443
status:
  loadBalancer:
    ingress:
    - ip: 192.168.100.73

[garyyang85]

from the error logs, it always think the request is from 127.0.0.1? I tired to debug all the ip related parameters, $the_real_ip and $remote_addr are always 127.0.0.1 during docker push operation.

[aledbf]

@garyyang85 the harbor service is using https? (servicePort: 443)
If that's the case you need to add the annotation ingress.kubernetes.io/secure-backends: "true"

[aledbf]

Closing. Please reopen if the issue continues after the use of the annotation ^^

[jessehu]

The root cause is docker will first connect to port 443 (SSL) but got 404 error.
Need to add the following snippet in ingress yaml file which makes ingress forward requests on port 443 to Harbor service. Without it, ingress only forward requests on port 80 to Harbor service.

spec:
  tls:
  - hosts:
    - harbor.default.example.com

[JinsYin]

@jessehu @aledbf I met the same issue when Ingress forward requests on port 80 to Harbor service.

• Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: harbor-ingress
  namespace: harbor
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: dockerhub.cloud.local
    http:
      paths:
      - path: /
        backend:
          serviceName: ui
          servicePort: 80
      - path: /v2
        backend:
          serviceName: registry
          servicePort: 5000
      - path: /service
        backend:
          serviceName: ui
          servicePort: 80

• docker push

$ docker push dockerhub.cloud.local/library/pause:01

# nginx-ingress-controller log
192.168.8.220 - [192.168.8.220] - - [16/Mar/2018:02:26:26 +0000] "GET /v2/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.2.0-27-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 240 0.001 [upstream-default-backend] 172.1.74.159:8080 21 0.001 404
192.168.8.220 - [192.168.8.220] - - [16/Mar/2018:02:26:26 +0000] "POST /v2/library/pause/blobs/uploads/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.2.0-27-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 304 0.001 [upstream-default-backend] 172.1.74.159:8080 21 0.001 404
192.168.8.220 - [192.168.8.220] - - [16/Mar/2018:02:26:26 +0000] "POST /v2/library/pause/blobs/uploads/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.2.0-27-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 304 0.001 [upstream-default-backend] 172.1.74.159:8080 21 0.001 404
192.168.8.220 - [192.168.8.220] - - [16/Mar/2018:02:26:26 +0000] "GET /v1/_ping HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.2.0-27-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 245 0.000 [upstream-default-backend] 172.1.74.159:8080 21 0.000 404
192.168.8.220 - [192.168.8.220] - admin [16/Mar/2018:02:26:26 +0000] "PUT /v1/repositories/library/pause/ HTTP/1.1" 404 21 "-" "docker/1.13.1 go/go1.7.5 git-commit/092cba3 kernel/4.2.0-27-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.13.1 \x5C(linux\x5C))" 623 0.001 [upstream-default-backend] 172.1.74.159:8080 21 0.000 404

• curl

$ curl -X GET "dockerhub.cloud.local/v2/"
$ curl -X POST "dockerhub.cloud.local/v2/library/pause/blobs/uploads/"

# nginx-ingress-controller log
192.168.8.220 - [192.168.8.220] - - [16/Mar/2018:02:29:33 +0000] "GET /v2/ HTTP/1.1" 200 2 "-" "curl/7.35.0" 88 0.004 [harbor-registry-repo] 172.1.74.177:5000 2 0.004 200
192.168.8.220 - [192.168.8.220] - - [16/Mar/2018:02:30:44 +0000] "POST /v2/library/pause/blobs/uploads/ HTTP/1.1" 202 0 "-" "curl/7.35.0" 117 0.018 [harbor-registry-repo] 172.1.74.177:5000 0 0.018 202

[jessehu]

@JinsYin You need to add this in your ingress.yml

spec:
  tls:
  - hosts:
    - dockerhub.cloud.local

BTW you can try the official Harbor Helm Chart 1.4.0 beta https://github.com/vmware/harbor/tree/master/contrib/helm/harbor.
This is the ingress.yaml https://github.com/vmware/harbor/blob/master/contrib/helm/harbor/templates/ingress/ingress.yaml

[JinsYin]

@jessehu Thank you so much, It is working.