Harbor: Running Harbor with HTTP behind a HTTPS Reverse Proxy (nginx)

[Flipez]

I've tried to get harbor running behind (another) nginx reverse proxy.

So the first setup looked like this:

nginx (host,ssl) -> harbor-nginx(non-ssl) -> harbor

If I setup the first nginx with ssl redirect it basically works but the docker client send the first try in plain which I do not want.

I tried different settings but none of them works because something inside harbor seems to redirect the requests to port 80. So if I stop exposing port 80 on the host the login fails with a connection refused even if I submit domain:443 or https://domain.

Are there any guidelines for this scenario?

The simple reason is that I do not want to maintain neither the SSL settings nor the cert within the container. This is globally handled by ansible.

Please let me know if any config of the current or previous setup would be useful.

# docker version
Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:51:12 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.1-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 22:50:04 2017
 OS/Arch:      linux/amd64
 Experimental: false

[totojack]

+1

[mdgreenwald]

+1

[jsumners]

This is just the issue I was looking for prior to installing for the first time. In my organization we use HAProxy for SSL termination. I currently have docker-distribution running while I research VMWare's solution. With docker-distribution this sort of configuration can be tricky (https://github.com/docker/distribution/issues/2225), so I expected to find such an issue for Harbor.

This is a configuration that needs to be supported.

[spark8899]

Jan 4 17:26:35 172.19.0.1 registry[22654]: time="2018-01-04T09:26:35.950809267Z" level=error msg="response completed with error" auth.user.name=admin err.code="blob unknown" err.detail=sha256:9ec141d7a6d669ded82f1d0977abf28eb71ae48f767babaa4651dcb9474dcb4f err.message="blob unknown to registry" go.version=go1.7.3 http.request.host=reg.sparkknow.com http.request.id=4761d537-1500-4009-9ddb-96ca993422cf http.request.method=HEAD http.request.remoteaddr=172.31.128.101 http.request.uri="/v2/library/alpine/blobs/sha256:9ec141d7a6d669ded82f1d0977abf28eb71ae48f767babaa4651dcb9474dcb4f" http.request.useragent="docker/17.12.0-ce go/go1.9.2 git-commit/c97c6d6 kernel/3.10.0-693.11.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.12.0-ce \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.729592ms http.response.status=404 http.response.written=157 instance.id=6a311010-3363-437c-b3e4-8c88eb62dfdf service=registry vars.digest="sha256:9ec141d7a6d669ded82f1d0977abf28eb71ae48f767babaa4651dcb9474dcb4f" vars.name="library/alpine" version=v2.6.2

I have error too.

[rolep]

Probably it can be used as https behind https proxy.

Somehow I got it running with following:
traefik (https, letsencrypt) <-> harbor (nginx, https, self-signed certs) <-> harbor

Actually, thought it will be complete ssl-passthrough to harbor-nginx, but the result is more convenient, as users see letsencrypt cert.
Traefik auto-generates letsencrypt cert (for a bundle of names), InsecureSkipVerify = true to disable cert check for backends. Also have traefik.frontend.passTLSCert=true (because of it thought it will pass-through) and traefik.protocol=https.

Certs are self-signed by local CA. In harbor.cfg selected https, path to cert.key and bundle-cert.pem.
Also using them for token service (off-topic: how to actually use/generate tokens for docker login?).

[wangycc]

+1
traefik (https, letsencrypt) <-> harbor (nginx, https, self-signed certs) <-> harbor

traefik(443) <-> harbor(nginx,443,self-signed certs),insecureSkipVerify = true

harbor self-signed:
https://github.com/vmware/harbor/blob/master/docs/configure_https.md

[ensean]

An extra section for using external nginx in the harbor.cfg file would be great.

After a few days's trying, I got the following combination work finally...

nginx1 (https, letsencrypt) <-> harbor proxy (nginx, http) <-> harbor ui/registry etc.

The most import is to follow this troubleshooting remove the following line from common/templates/nginx/nginx.http.conf

proxy_set_header X-Forwarded-Proto $scheme;

[lxkaka]

@ensean could you show your nginx1 configuration? thanks in advance

[ensean]

@lxkaka here goes my nginx1 conf

upstream harbor {
  server harbor_proxy_ip:8080;
}

server {
    listen 443 ssl;
    server_name harbor.mycomp.com;

    ssl_certificate      /etc/nginx/conf.d/mycomp.com.crt;
    ssl_certificate_key  /etc/nginx/conf.d/mycomp.com.key;

    client_max_body_size 0;
    chunked_transfer_encoding on;

    location / {
        proxy_pass http://harbor/;
        proxy_set_header  Host              $http_host;
        proxy_set_header  X-Real-IP         $remote_addr;
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;

        proxy_buffering off;
        proxy_request_buffering off;
    }
}

[k7]

After a struggle, I have fixed it.

I have the service like this:

nginx (host,ssl) -> harbor-nginx(non-ssl) -> harbor

First, I emulate login via curl, it was good.

curl -u admin:passwd123 "https://my.registry.company.com/service/token?account=admin&client_id=docker&offline_token=true&service=harbor-registry"

{
  "token": "xxxxxx",
  "expires_in": 1800,
  "issued_at": "2018-09-27T03:42:27Z"
}

It means there was other things error.

Via tcpdump, I found docker login will make a http request https://my.registry.company.com/v2/ first.

curl -X GET -I "https://my.registry.company.com/v2/"

HTTP/1.1 401 Unauthorized
Server: nginx/1.12.2
Date: Thu, 27 Sep 2018 07:20:58 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 87
Connection: keep-alive
Docker-Distribution-Api-Version: registry/2.0
Set-Cookie: beegosessionID=75df8e3e78eacf5848a83877908ca96; Path=/; HttpOnly
Www-Authenticate: Bearer realm="http://my.registry.company.com/v2//service/token",service="harbor-registry"

Now, I found the problem. realm="http://my.registry.company.com/v2//service/token". it was http, not https.

So I edit common/config/registry/config.yml，and fixed it.

auth:
  token:
    issuer: harbor-token-issuer
    realm: https://my.registry.company.com/service/token  # http -> https
    rootcertbundle: /some/root.crt
    service: harbor-registry

Great, It works.

docker login private-registry.tupu.ai -u admin
Login Succeeded

[rruu]

and you can changes prepare script with and reinstall
- public_url = protocol + "://" + hostname
+ public_url = "https" + "://" + hostname

[shinebayar-g]

Hey @k7 Thank you very much for the explanation. Without changing that particular config, I was getting

Error response from daemon: Get https://goharbor.mydomain.com/v2/: unauthorized: authentication required

error when trying to login. After changing that it works! However if I run install.sh script it overrides that changed settings. So I just restarted all docker process instead of re-install.

Oh just noticed @rruu mentioned that we can change template's public_url option to change it.

[jakirpatel]

I faced a similar issue while running Harbor with HTTP behind an HTTPS Reverse Proxy.
I did the following changes:
In harbor.cfg:

1. Make sure the hostname points to the VIP. (In my case its reverse proxy https load balancer nginx)

2. customize_crt = off( As external source that means https nginx will do this)

1. In config of nginx common
   /common/templates/nginx/nginx.http.conf
   Comment this line: proxy_set_header X-Forwarded-Proto $$scheme;
   Make sure to comment it / , /v2/ and /service/ endpoints.
2. Prepare your harbor with ./prepare.
   After preparation in common/config/registry/config.yml make realm https.

auth:
  token:
    issuer: harbor-token-issuer
    realm: http://vip-address/service/token
    rootcertbundle: /etc/registry/root.crt
    service: harbor-registry

Change to:

auth:
  token:
    issuer: harbor-token-issuer
    realm: https://vip-address/service/token
    rootcertbundle: /etc/registry/root.crt
    service: harbor-registry

1. Start the harbor.

This solved my problem.

[CrystalMethod]

I can confirm that Harbor works behind an HTTPS reverse proxy (Synology NAS/Nginx) with SSL termination. In addition there is Traefik as another proxy in between - however, this only has the task of routing to different Docker based services. All communication uses port 80 of Harbor Proxy.

[nicholasamorim]

I can confirm it runs with the modifications that are shown in here _but_ docker login always returns me 400 Bad Request. Everything else works fine and emulating login via curl works fine too.

The funny thing is that if I enter the incorrect login/password, the behaviour works as expected, telling me I'm unauthorized.

[mxplusb]

I had the same problem here as most folks, I wanted to run Harbor behind a reverse proxy (I'm using Nginx), but I also wanted Clair, Chart Museum, and Notary, which means HTTPS all the way. Here is what I did, thanks to @jakirpatel for the initial work.

1. If Harbor with Notary, Clair, and Chart Museum is running, run docker-compose -f ./docker-compose.yml -f ./docker-compose.notary.yml -f ./docker-compose.clair.yml -f ./docker-compose.chartmuseum.yml down -v
2. Generate your self-signed HTTPS certificates from the Harbor instructions.
3. In harbor.cfg, set your hostname to the external VIP, so vip.domain.com.
4. In harbor.cfg, set customize_cert = on
5. In the nginx HTTPS config, /common/templates/nginx/nginx.https.conf, comment out all instances of proxy_set_header X-Forwarded-Proto $$scheme;
6. Run ./prepare --with-notary --with-clair --with-chartmuseum
7. Start everything back up with docker-compose -f ./docker-compose.yml -f ./docker-compose.notary.yml -f ./docker-compose.clair.yml -f ./docker-compose.chartmuseum.yml up -d
8. On your reverse proxy server (I'm using Nginx as an example since that's what I use), create the harbor.conf file.
9. Populate it with this:

# this block is to redirect 80 to 443, i.e. HTTP to HTTPS
server {
        listen 80;
        server_name vip.domain.com;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443;
        server_name     vip.domain.com;
        add_header      Strict-Transport-Security "max-age=31536000" always;
        # your certs, I'm using let's encrypt domain wildcards.
        ssl_certificate /etc/letsencrypt/live/domain.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/domain.com/privkey.pem;
        ssl_stapling on;
        ssl_stapling_verify on;

        location / {
                client_max_body_size 2000m;
                proxy_buffering off;
                proxy_ssl_verify off;
                proxy_set_header Host $http_host;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_pass https://harbor.internal.domain.com;
    }
}

Edit: I'm still working through the Notary configs. I've never used Notary before so I only partially understand it. If I get it working, I'll update here.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[DiegoSouzaPW]

Edit: I'm still working through the Notary configs. I've never used Notary before so I only partially understand it. If I get it working, I'll update here.

No News for Notary?, i try without success too

[soosap]

@k7 pointed me into the right direction. I was experiencing 401 Unauthorized for docker login myco.registry.com -u admin. I solved this by setting the value of auth.token.issuer to issuer: harbor-token-issuer inside the registry config. Somehow I believed I could choose a random value there. 🤯

[slzzz]

+1
traefik (https, letsencrypt) <-> harbor (nginx, https, self-signed certs) <-> harbor

traefik(443) <-> harbor(nginx,443,self-signed certs),insecureSkipVerify = true

harbor self-signed:
https://github.com/vmware/harbor/blob/master/docs/configure_https.md
404