Renovate: Docker returns 503 Service Unavailable when behind proxy

Created on 16 Apr 2019 · 29Comments · Source: renovatebot/renovate

What Renovate type are you using?
Renovate CLI via the official docker image.

Describe the bug
Our CI platform is now sitting behind a proxy. Proxy variables are provided both HTTP and HTTPS, upper and lower case. Renovate works successfully for all dependency types we use except Docker, which consistently returns a 503 Service Unavailable error.

Did you see anything helpful in debug logs?

 WARN: docker registry failure: internal error (repository=group/repo)
       "registry": "https://index.docker.io",
       "dockerRepository": "library/python",
       "err": {
         "name": "HTTPError",
         "host": "index.docker.io",
         "hostname": "index.docker.io",
         "method": "GET",
         "path": "/v2/library/python/tags/list?n=10000",
         "protocol": "https:",
         "url": "https://index.docker.io/v2/library/python/tags/list?n=10000",
         "gotOptions": {
           "path": "/v2/library/python/tags/list?n=10000",
           "protocol": "https:",
           "slashes": true,
           "auth": null,
           "host": "index.docker.io",
           "port": null,
           "hostname": "index.docker.io",
           "hash": null,
           "search": "?n=10000",
           "pathname": "/v2/library/python/tags/list",
           "href": "https://index.docker.io/v2/library/python/tags/list?n=10000",
           "headers": {
             "user-agent": "https://github.com/renovatebot/renovate",
             "accept": "application/json",
             "accept-encoding": "gzip, deflate"
           },
           "hooks": {
             "beforeError": [],
             "init": [],
             "beforeRequest": [],
             "beforeRedirect": [],
             "beforeRetry": [],
             "afterResponse": []
           },
           "retry": {
             "methods": {},
             "statusCodes": {},
             "errorCodes": {},
             "maxRetryAfter": 10000
           },
           "decompress": true,
           "throwHttpErrors": true,
           "followRedirect": true,
           "stream": false,
           "form": false,
           "json": true,
           "cache": false,
           "useElectronNet": false,
           "gotTimeout": {"request": 10000},
           "method": "GET"
         },
         "statusCode": 503,
         "statusMessage": "Service Unavailable",
         "headers": {
           "cache-control": "no-cache",
           "connection": "close",
           "content-type": "text/html"
         },
         "body": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n\n",
         "message": "Response code 503 (Service Unavailable)",
         "stack": "HTTPError: Response code 503 (Service Unavailable)\n    at EventEmitter.emitter.on (/usr/src/app/node_modules/got/source/as-promise.js:74:19)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"
       }

To Reproduce
Sit Renovate Cl behind a proxy and attempt to renovate a docker file, or something containing docker images (e.g. gitlab-ci.yml).

Expected behavior
Renovation to occur on docker images as previously.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Using the Renovate Docker image I can manually replicate the registry handshake so the base configuration appears ok:

- curl -i 'https://index.docker.io/v2/library/python/tags/list'
- export AUTH_RESPONSE=$(curl 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/python:pull')
- echo $AUTH_RESPONSE
- echo "$AUTH_RESPONSE" | grep -Po -e '\"token\"\:\"\K[^\"]*'
- export BEARER_TOKEN=$(echo "$AUTH_RESPONSE" | grep -Po -e '\"token\"\:\"\K[^\"]*')
- echo $BEARER_TOKEN
- 'curl https://index.docker.io/v2/library/python/tags/list -i --header "Authorization: Bearer $BEARER_TOKEN"'

help wanted priority-3-normal

Source

adam-moss

All 29 comments

@adam-moss I will need your help to troubleshoot this further. For example:

Which server do you think is returning that 503? i.e. is it reaching all the way to Docker but Docker returns 503?
Can you work out in which version of Renovate this broke?

Potentially if you work out (2) then (1) doesn't matter as much

rarkins on 16 Apr 2019

@adam-moss how is that curl example working with a proxy in place? Does curl autodetect HTTPS_PROXY settings? Or are you running it from a proxyless environment?

rarkins on 16 Apr 2019

Yep, works in place. You can set the --proxy flag on curl to manually set (or override) the env vars.

Going to be hard to find (2), 'cos we pull the latest image before each run of Renovate (so daily)

adam-moss on 16 Apr 2019

We think it might be https://github.com/renovatebot/renovate/blob/4b9a18ff663b85183af85f349a27511324c4aa7d/lib/datasource/docker/index.js#L47, but haven't conclusively verified yet

adam-moss on 16 Apr 2019

In that case though can you work out which day it first stopped? Also I think we should print the version of Renovate with each run

rarkins on 16 Apr 2019

27th Feb is the first time it was ran under the proxy and started through those warnings.

Agree re the version 👍

adam-moss on 16 Apr 2019

I might have misunderstood. Do you mean Docker has never worked with the proxy?

rarkins on 16 Apr 2019

So according to the headers it’s trying without authentication. 503 is a strange error if so, but maybe that’s how they handle it. So you agree with this analysis so far?

rarkins on 16 Apr 2019

Previously our proxy was completely transparent to the end user and CI, so we never had an issue. Now it needs to be explicitly set, and we have this issue only in relation to docker when using renovate.

adam-moss on 16 Apr 2019

Yes, your analysis seems ok 👍

adam-moss on 16 Apr 2019

We have the same problem - the last time it worked was on March, 6th. The error appeared the first time on Mar 7, 2019 8am CET (we are running renovate hourly 8am - 6pm).

derkoe on 9 May 2019

proxy-agent seems to be the problem.

I wrote a small test program comaring that with the old implementation (using tunnel) - tunnel works, proxy-agent does not.

const got = require("got");
const ProxyAgent = require("proxy-agent");
const tunnel = require("tunnel");

async function testProxyAgent() {
  const apiCheckResponse = await got("https://index.docker.io/v2/", {
    throwHttpErrors: false,
    agent: new ProxyAgent()
  });
  console.info(`ProxyAgent:
  - ${apiCheckResponse.statusCode}
  - ${apiCheckResponse.headers['www-authenticate']}
  - ${apiCheckResponse.body}`)
}

async function testTunnel() {
  const apiCheckResponse = await got("https://index.docker.io/v2/", {
    throwHttpErrors: false,
    agent: tunnel.httpsOverHttp({
      proxy: {
        host: 'localhost',
        port: 3128
      }
    })
  });
  console.info(`tunnel:
  - ${apiCheckResponse.statusCode}
  - ${apiCheckResponse.headers['www-authenticate']}
  - ${apiCheckResponse.body}`)
}

testProxyAgent();
testTunnel();

Output:

tunnel:
  - 401
  - Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
  - {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

ProxyAgent:
  - 503
  - undefined
  - <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

derkoe on 9 May 2019

👍1

@derkoe thank you for your assistance in narrowing down the problem. Strange that Docker appears to be the only endpoint throwing errors.

I don't mind which package we use but we must be capable of proxying all node.js requests and honouring NO_PROXY too

rarkins on 9 May 2019

@gajus I just saw you recently wrote https://github.com/gajus/global-agent

Would that work here?

We need to support HTTP_PROXY etc but I think we can hopefully work around that quite easily.

rarkins on 9 May 2019

👍1

@derkoe it's slightly outside your time range but I wonder if this commit did it: https://github.com/renovatebot/renovate/commit/34d270fd793d78e88190de07ed0e9e8dabd44dc0

rarkins on 9 May 2019

@rarkins does not seem so - I just tried renovate/renovate:14.35 (which is before the error first occured). Maybe Docker Hub has changed some headers.

BTW there is an error in proxy.js - it should be options instead of opts (but that's not fixing it):
https://github.com/renovatebot/renovate/blob/5c817b1315d09a1ee84c34568b2791799e68e8d0/lib/proxy.js#L19

derkoe on 9 May 2019

I can reproduce this problem locally now at least. Trying to narrow it down. When I run a debug http server and relay it with serveo, I see these headers for the query to /v2/:

User-Agent: https://github.com/renovatebot/renovate
Accept-Encoding: gzip, deflate

but I fail to reproduce:

curl -i -H 'accept-encoding:gzip, deflate' https://index.docker.io/v2/HTTP/1.1 200 Connection established

HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io"
Date: Thu, 09 May 2019 13:57:33 GMT
Content-Length: 87
Strict-Transport-Security: max-age=31536000

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

rarkins on 9 May 2019

I need to give up on this one for now. Any further help would be appreciated.

rarkins on 9 May 2019

I guess you'll need a "real" proxy like squid because there is a CONNECT phase before the real GET.

You could use https://hub.docker.com/r/minimum2scp/squid

docker run -d -p 3128:3128 minimum2scp/squid
export http_proxy=http://127.0.0.1:3128

derkoe on 9 May 2019

Actually I ran through squid locally first each time including for curl

rarkins on 9 May 2019

Just replaced proxy.js with

const envKeys = Object.keys(process.env).map(key => key.toLocaleUpperCase());
if (envKeys.includes('HTTP_PROXY') || envKeys.includes('HTTPS_PROXY')) {
  require('global-agent/bootstrap');
  global.GLOBAL_AGENT.HTTP_PROXY =
    process.env.HTTP_PROXY || process.env.HTTPS_PROXY;
  global.GLOBAL_AGENT.NO_PROXY = process.env.NO_PROXY;
}

and everything seems to work now.

The only problem I see is that https://github.com/gajus/global-agent does not distinguish between http_proxy and https_proxy.

derkoe on 13 May 2019

@derkoe thanks for trying that. Any reason why you chose to implement that way rather than by setting GLOBAL_AGENT_HTTP_PROXY prior to initialization?

@gajus do you plan to support/distinguish between HTTP and HTTPS proxies in future, in case they're different?

rarkins on 13 May 2019

@gajus do you plan to support/distinguish between HTTP and HTTPS proxies in future, in case they're different?

I don't see a reason not to support it.

Best would be to raise an issue and discuss it there.

gajus on 13 May 2019

@rarkins no - you could do this as well.

Regarding http and https - I guess most people won't need this but some might have a special setup for https.

derkoe on 13 May 2019

@gajus I created https://github.com/gajus/global-agent/issues/5 for discussion, thanks. Also created another issue about use of traditional env variables, but that one is not essential for here - just nice to have.

rarkins on 13 May 2019

Waiting on the merge of https://github.com/gajus/global-agent/pull/8

I'll fork and merge soon if we need to, until the main repo is ready to accept the PR.

rarkins on 17 Jun 2019

@derkoe @fgreinacher I think we may be ready to go? Something like this:

if (process.env.HTTP_PROXY || process.env.HTTPS_PROXY) {
  require('global-agent/bootstrap');
  global.GLOBAL_AGENT.HTTP_PROXY = process.env.HTTP_PROXY;
  global.GLOBAL_AGENT.HTTPS_PROXY =
    process.env.HTTPS_PROXY || process.env.HTTP_PROXY;
  global.GLOBAL_AGENT.NO_PROXY = process.env.NO_PROXY;
}

rarkins on 19 Jun 2019

Yep, the HTTPS_PROXY fallback logic is not necessary though - that’s already handled by global-agent. Might have a look at this tomorrow or on the weekend.