Harbor: 502 Bad Gateway

Created on 11 May 2017 · 7Comments · Source: goharbor/harbor

I have beat my head against the wall for days.

I am trying to set up Harbor with notary and SSL access on Centos 7

I have followed the install instruction, TO THE LETTER.

Everything starts.

I get a 502 Bad Gateway every time. I have ripped and reinstalled, per the instructions here at many times and no change.

Either the instructions are wrong or incomplete.

Is there a missing instruction or step that the author assumes is done??? HAs anyone followed the instructions, as written, and been able to get it to work? Without filling in the missing instructions on their own?

I need help as I can not figure out why it is not working.

Source

glzavert

Most helpful comment

I fixed my issue and it was tied to the secret key location. The docker-compose.yml file has the host file path hard coded to the system drive. Not sure why you did not use relative paths and kept everything under ../harbor. After I created the directory and updated my harbor.cfg file it all worked.

Now for the whining. This is very bad. If we are to deploy in a real environment, you likely WILL not want to use the system drive, or for that matter, any other physical drive on the server. You would want harbor to be installed on attached storage so that you could better manage resources and provide HA/DR capabilities. To that end, all the storage locations in the docker-compose file should be relative to it and under the harbor directory. Also, the harbor.cfg file should all for relative pathing as well.

That's my view, so take it for what it is worth.

glzavert on 16 May 2017

👍7

All 7 comments

Please provide the logs under directory /var/log/harbor/.

ywk253100 on 11 May 2017

@ywk253100 same problem, the logs below

==> /var/log/harbor/2017-05-16/adminserver.log <==
May 16 11:18:43 localhost adminserver[1079]: 2017-05-16T03:18:43Z [DEBUG] [driver_json.go:46]: path of configuration file: /etc/adminserver/config/config.json
May 16 11:18:43 localhost adminserver[1079]: 2017-05-16T03:18:43Z [INFO] the path of key used by key provider: /etc/adminserver/key
May 16 11:18:43 localhost adminserver[1079]: 2017-05-16T03:18:43Z [INFO] configurations read from storage driver are null, will load them from environment variables
May 16 11:18:43 localhost adminserver[1079]: 2017-05-16T03:18:43Z [FATAL] [main.go:46]: failed to initialize the system: read /etc/adminserver/key: is a directory
May 16 11:19:08 localhost adminserver[1079]: 2017-05-16T03:19:08Z [INFO] initializing system configurations...
May 16 11:19:08 localhost adminserver[1079]: 2017-05-16T03:19:08Z [INFO] the path of json configuration storage: /etc/adminserver/config/config.json
May 16 11:19:08 localhost adminserver[1079]: 2017-05-16T03:19:08Z [DEBUG] [driver_json.go:46]: path of configuration file: /etc/adminserver/config/config.json
May 16 11:19:08 localhost adminserver[1079]: 2017-05-16T03:19:08Z [INFO] the path of key used by key provider: /etc/adminserver/key
May 16 11:19:08 localhost adminserver[1079]: 2017-05-16T03:19:08Z [INFO] configurations read from storage driver are null, will load them from environment variables
May 16 11:19:08 localhost adminserver[1079]: 2017-05-16T03:19:08Z [FATAL] [main.go:46]: failed to initialize the system: read /etc/adminserver/key: is a directory

==> /var/log/harbor/2017-05-16/jobservice.log <==
May 16 11:19:10 localhost jobservice[1079]: 2017-05-16T03:19:10Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:12 localhost jobservice[1079]: 2017-05-16T03:19:12Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:14 localhost jobservice[1079]: 2017-05-16T03:19:14Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:16 localhost jobservice[1079]: 2017-05-16T03:19:16Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:18 localhost jobservice[1079]: 2017-05-16T03:19:18Z [FATAL] [main.go:31]: failed to initialize configurations: failed to ping adminserver: failed to connect to tcp:adminserver:80 after 60 seconds
May 16 11:19:18 localhost jobservice[1079]: 2017-05-16T03:19:18Z [INFO] Config path: /etc/jobservice/app.conf
May 16 11:19:18 localhost jobservice[1079]: 2017-05-16T03:19:18Z [INFO] initializing configurations...
May 16 11:19:18 localhost jobservice[1079]: 2017-05-16T03:19:18Z [INFO] key path: /etc/jobservice/key
May 16 11:19:18 localhost jobservice[1079]: 2017-05-16T03:19:18Z [INFO] initializing client for adminserver http://adminserver ...
May 16 11:19:18 localhost jobservice[1079]: 2017-05-16T03:19:18Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host

==> /var/log/harbor/2017-05-16/mysql.log <==
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note] InnoDB: Waiting for purge to start
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note] InnoDB: 5.6.35 started; log sequence number 1749532
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note] Server hostname (bind-address): '*'; port: 3306
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note] IPv6 is available.
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note]   - '::' resolves to '::';
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note] Server socket created on IP: '::'.
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Warning] 'proxies_priv' entry '@ root@240b4f568407' ignored in --skip-name-resolve mode.
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note] Event Scheduler: Loaded 0 events
May 16 11:18:17 localhost mysql[1079]: 2017-05-16 03:18:17 1 [Note] mysqld: ready for connections.
May 16 11:18:17 localhost mysql[1079]: Version: '5.6.35'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server (GPL)

==> /var/log/harbor/2017-05-16/proxy.log <==
May 16 11:13:44 localhost proxy[1079]: 2017/05/16 03:13:44 [error] 5#5: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.74.20, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.3:80/favicon.ico", host: "192.168.74.29", referrer: "https://192.168.74.29/"
May 16 11:13:44 localhost proxy[1079]: 192.168.74.20 - "GET /favicon.ico HTTP/1.1" 502 575 "https://192.168.74.29/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.000 0.000 .
May 16 11:13:44 localhost proxy[1079]: 192.168.74.20 - "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.000 0.000 .
May 16 11:13:44 localhost proxy[1079]: 2017/05/16 03:13:44 [error] 5#5: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.74.20, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.3:80/", host: "192.168.74.29"
May 16 11:13:44 localhost proxy[1079]: 2017/05/16 03:13:44 [error] 5#5: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.74.20, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.3:80/favicon.ico", host: "192.168.74.29", referrer: "https://192.168.74.29/"
May 16 11:13:44 localhost proxy[1079]: 192.168.74.20 - "GET /favicon.ico HTTP/1.1" 502 575 "https://192.168.74.29/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.000 0.000 .
May 16 11:13:44 localhost proxy[1079]: 192.168.74.20 - "GET / HTTP/1.1" 502 575 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.000 0.000 .
May 16 11:13:44 localhost proxy[1079]: 2017/05/16 03:13:44 [error] 5#5: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.74.20, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.0.3:80/", host: "192.168.74.29"
May 16 11:13:44 localhost proxy[1079]: 2017/05/16 03:13:44 [error] 5#5: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.74.20, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.3:80/favicon.ico", host: "192.168.74.29", referrer: "https://192.168.74.29/"
May 16 11:13:44 localhost proxy[1079]: 192.168.74.20 - "GET /favicon.ico HTTP/1.1" 502 575 "https://192.168.74.29/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 0.001 0.001 .

==> /var/log/harbor/2017-05-16/registry.log <==
May 16 11:16:25 localhost registry[1079]: time="2017-05-16T03:16:25.812860005Z" level=info msg="debug server listening localhost:5001"
May 16 11:16:25 localhost registry[1079]: time="2017-05-16T03:16:25.905120922Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.7.3 instance.id=bedb72df-8046-4b66-bac5-24b7bdcf8a08 service=registry version="v2.4.1+unknown"
May 16 11:16:25 localhost registry[1079]: time="2017-05-16T03:16:25.905857435Z" level=debug msg="configured \"token\" access controller" go.version=go1.7.3 instance.id=bedb72df-8046-4b66-bac5-24b7bdcf8a08 service=registry version="v2.4.1+unknown"
May 16 11:16:25 localhost registry[1079]: time="2017-05-16T03:16:25.905898104Z" level=info msg="listening on [::]:5000" go.version=go1.7.3 instance.id=bedb72df-8046-4b66-bac5-24b7bdcf8a08 service=registry version="v2.4.1+unknown"
May 16 11:18:15 localhost registry[1079]: time="2017-05-16T03:18:15.997500233Z" level=info msg="configuring endpoint harbor (http://ui/service/notifications), timeout=3s, headers=map[]" go.version=go1.7.3 instance.id=66a6c5b7-00ba-42a7-a056-e6d136b75f3e service=registry version="v2.4.1+unknown"
May 16 11:18:15 localhost registry[1079]: time="2017-05-16T03:18:15.997555286Z" level=info msg="redis not configured" go.version=go1.7.3 instance.id=66a6c5b7-00ba-42a7-a056-e6d136b75f3e service=registry version="v2.4.1+unknown"
May 16 11:18:16 localhost registry[1079]: time="2017-05-16T03:18:16.007095366Z" level=info msg="debug server listening localhost:5001"
May 16 11:18:16 localhost registry[1079]: time="2017-05-16T03:18:16.036069111Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.7.3 instance.id=66a6c5b7-00ba-42a7-a056-e6d136b75f3e service=registry version="v2.4.1+unknown"
May 16 11:18:16 localhost registry[1079]: time="2017-05-16T03:18:16.03624501Z" level=debug msg="configured \"token\" access controller" go.version=go1.7.3 instance.id=66a6c5b7-00ba-42a7-a056-e6d136b75f3e service=registry version="v2.4.1+unknown"
May 16 11:18:16 localhost registry[1079]: time="2017-05-16T03:18:16.036279816Z" level=info msg="listening on [::]:5000" go.version=go1.7.3 instance.id=66a6c5b7-00ba-42a7-a056-e6d136b75f3e service=registry version="v2.4.1+unknown"

==> /var/log/harbor/2017-05-16/ui.log <==
May 16 11:19:10 localhost ui[1079]: 2017-05-16T03:19:10Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:12 localhost ui[1079]: 2017-05-16T03:19:12Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:14 localhost ui[1079]: 2017-05-16T03:19:14Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:16 localhost ui[1079]: 2017-05-16T03:19:16Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host
May 16 11:19:18 localhost ui[1079]: 2017-05-16T03:19:18Z [FATAL] [main.go:78]: failed to initialize configurations: failed to ping adminserver: failed to connect to tcp:adminserver:80 after 60 seconds
May 16 11:19:18 localhost ui[1079]: 2017-05-16T03:19:18Z [INFO] Config path: /etc/ui/app.conf
May 16 11:19:18 localhost ui[1079]: 2017-05-16T03:19:18Z [INFO] initializing configurations...
May 16 11:19:18 localhost ui[1079]: 2017-05-16T03:19:18Z [INFO] key path: /etc/ui/key
May 16 11:19:18 localhost ui[1079]: 2017-05-16T03:19:18Z [INFO] initializing client for adminserver http://adminserver ...
May 16 11:19:18 localhost ui[1079]: 2017-05-16T03:19:18Z [ERROR] [utils.go:94]: failed to connect to tcp://adminserver:80, retry after 2 seconds :dial tcp: lookup adminserver: no such host

oiuww09fn on 16 May 2017

@QuentinJi Could you try the steps mentioned in #2208 ?

ywk253100 on 16 May 2017

secretkey_path does not work well

oiuww09fn on 16 May 2017

That's my view, so take it for what it is worth.

glzavert on 16 May 2017

👍7

please see the log of vi /var/log/harbor/adminserver.log
You will find the error
Dec 6 19:06:26 172.18.0.1 adminserver[5242]: 2018-12-06T11:06:26Z [FATAL] [main.go:46]: failed to initialize the system: read /etc/adminserver/key: is a directory