Describe the bug
I set up a cluster in AKS and executed the commands required to browse to it and they work from my machine. But when I execute them in an Azure CLI Docker container, it hangs and eventually exits with an error.
To Reproduce
Steps to reproduce the behavior.
1) Create a K8S cluster
2) Launch an Azure CLI container: docker run -it -p 8001:8001 microsoft/azure-cli:latest
3) az login
4) az aks install-cli
5) az aks get-credentials --resource-group my-resource-group --name my-k8s-name
6) az aks browse --resource-group my-resource-group --name my-k8s-name --disable-browser
7) Navigate to http://{container-ip}:8001
Expected behavior
See the Kubernetes dashboard
Environment summary
Additional context
This is what appears eventually after running the aks browse command. It also appears when you cancel the running aks browse command:
HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/knack/cli.py", line 197, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 372, in execute
six.reraise(sys.exc_info())
File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 346, in execute
result = cmd(params)
File "/usr/local/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 182, in __call__
return self.handler(args, kwargs)
File "/usr/local/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 436, in default_command_handler
result = op(command_args)
File "/usr/local/lib/python3.6/site-packages/azure/cli/command_modules/acs/custom.py", line 1381, in aks_browse
requests.post('http://localhost:8888/closeport/8001')
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, *kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, *kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 524, in request
resp = self.send(prep, *send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 637, in send
r = adapter.send(request, *kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('
Something similar happens to me too error: error upgrading connection: error dialing backend: dial tcp: lookup aks-default-34133888-1 on 172.30.0.10:53: server misbehaving
Generally having to use az aks browse is a big pain as it doesn't work as reliably as kubectl proxy and it's also inconvenient when you need to work with clusters in different subscriptions. Is there a way to use kubectl proxy directly?
UPD: after deleting kubernetes-dashboard via kubectl I can connect again.
I'm having this issue now:
HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10fd414a8>: Failed to establish a new connection: [Errno 61] Connection refused'))
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x10fd414a8>: Failed to establish a new connection: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001(Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10fd414a8>: Failed to establish a new connection: [Errno 61] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/knack/cli.py", line 197, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 372, inexecute
six.reraise(*sys.exc_info())
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 346, inexecute
result = cmd(params)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 182, in__call__
return self.handler(*args, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/__init__.py", line 436, in default_command_handler
result = op(**command_args)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/command_modules/acs/custom.py", line 1381, in aks_browse
requests.post('http://localhost:8888/closeport/8001')
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10fd414a8>: Failed to establish a new connection: [Errno 61] Connection refused'))
Unfortunately the workaround suggested by @ilyalukyanov does not work anymore
I think why it worked for me was for another reason. Maybe the pod was rescheduled on another node. However, when I tried cordoning the suspected node to force rescheduling on another one, the pod started but proxy was timing out and failing (but failing differently).
Restarting the node didn't help too. The only thing that helped was recreating the node from scratch (by de-scaling the cluster and scaling it back - lucky it was dev environment).
I have the same problem with azure-cli 2.0.49
macOS mojave
az aks browse --resource-group AKS-demo --name aks-demo
Merged "aks-demo" as current context in /var/folders/rh/5x2rl5p50hb4cd1zf0s2m46m0000gn/T/tmpx08su8oz
Proxy running on http://127.0.0.1:8001/
Press CTRL+C to close the tunnel...
Forwarding from 127.0.0.1:8001 -> 9090
Forwarding from [::1]:8001 -> 9090
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
Handling connection for 8001
E1031 09:53:44.482047 42072 portforward.go:303] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:8001->127.0.0.1:57370: write tcp4 127.0.0.1:8001->127.0.0.1:57370: write: broken pipe
^CHTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x103da5cf8>: Failed to establish a new connection: [Errno 61] Connection refused'))
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 956, in send
self.connect()
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x103da5cf8>: Failed to establish a new connection: [Errno 61] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/adapters.py", line 440, in send
timeout=timeout
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x103da5cf8>: Failed to establish a new connection: [Errno 61] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/knack/cli.py", line 197, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 372, in execute
six.reraise(*sys.exc_info())
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 346, in execute
result = cmd(params)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 182, in __call__
return self.handler(*args, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/core/__init__.py", line 436, in default_command_handler
result = op(**command_args)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/azure/cli/command_modules/acs/custom.py", line 1381, in aks_browse
requests.post('http://localhost:8888/closeport/8001')
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.49/libexec/lib/python3.7/site-packages/requests/adapters.py", line 508, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8888): Max retries exceeded with url: /closeport/8001 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x103da5cf8>: Failed to establish a new connection: [Errno 61] Connection refused'))
+1
I have the same issue azure-cli (2.0.49), macOS High Sierra
Happens on Windows 10 (version 1803) too, with azure-cli (2.0.49).
Same problem. It worked yesterday and now it's not.
OSX Mojave, Azure Cli .51
Updating the cluster helped, I'm on 11.5 now.
I investigated this with Azure kubernetes services.
By tracing the logs from kubernetes-dashboard in the kube-system namespace i found that the dashboard was listening on port 9090. Using the following "browse" command did the trick.
az aks browse --resource-group RESOURCE --name NAME --listen-port 9090
Not totally related but might help some folks out.
Does this relate to? https://github.com/Azure/azure-cli/issues/7914
I managed to work around this by adding socat installation and exposing a port in my Dockerfile:
FROM microsoft/azure-cli:latest
...
RUN apk add --no-cache socat
EXPOSE 8888
...
When I run docker I specify:
docker run -it -p 8888:8888 MYIMAGE
Then I forward 8888 traffic to 9090 (apologies if this isn't quite right terminology, I'm not a networks expert) and then run "az aks browse".
socat TCP-LISTEN:8888,fork TCP:127.0.0.1:9090 &
az aks browse --resource-group RESOURCE --name NAME --listen-port 9090 --disable-browser
I can then access the Dashboard on my host via http://localhost:8888/
Can this issue be reopened?
I've tested the fix and it only seems to fix the problem partially.
When I run the fix, it no longer spits out the error, but simply exits the proxy.
You can test it as follows (tested this on a mac running 10.14.5):
Run az aks browse --resource-group my-resource-group --name my-cluster-name
The proxy will start and will open http://127.0.0.1:8001
Wait a couple of minutes without doing anything and you'll see that the az command silently exits…
I experience the same thing as Pieter -- I'm on linux pop!_os 19.04. az-cli 2.0.58 aks 1.13.9. I'm attempting aks upgrade to 1.14.5 to see if that helps.
Still got this error now. My az --version:
azure-cli 2.0.60 *
acr 2.2.2 *
acs 2.3.19 *
advisor 2.0.0 *
ams 0.4.2 *
appservice 0.2.15 *
backup 1.2.1 *
batch 4.0.0 *
batchai 0.4.7 *
billing 0.2.0 *
botservice 0.1.8 *
cdn 0.2.0 *
cloud 2.1.0 *
cognitiveservices 0.2.4 *
command-modules-nspkg 2.0.2 *
configure 2.0.20 *
consumption 0.4.2 *
container 0.3.15 *
core 2.0.60 *
cosmosdb 0.2.8 *
dla 0.2.4 *
dls 0.1.8 *
dms 0.1.2 *
eventgrid 0.2.1 *
eventhubs 0.3.4 *
extension 0.2.3 *
feedback 2.1.4 *
find 0.3.0 *
hdinsight 0.3.2 *
interactive 0.4.1 *
iot 0.3.6 *
iotcentral 0.1.6 *
keyvault 2.2.12 *
kusto 0.2.0 *
lab 0.1.5 *
maps 0.3.3 *
monitor 0.2.10 *
network 2.3.4 *
nspkg 3.0.3 *
policyinsights 0.1.1 *
profile 2.1.3 *
rdbms 0.3.8 *
redis 0.4.1 *
relay 0.1.3 *
reservations 0.4.1 *
resource 2.1.11 *
role 2.4.2 *
search 0.1.1 *
security 0.1.0 *
servicebus 0.3.3 *
servicefabric 0.1.14 *
signalr 1.0.0 *
sql 2.1.9 *
sqlvm 0.1.0 *
storage 2.3.1 *
telemetry 1.0.1 *
vm 2.2.16 *
Python location '/opt/az/bin/python3'
Extensions directory '/home/shenghang/.azure/cliextensions'
Python (Linux) 3.6.5 (default, Mar 7 2019, 19:49:28)
[GCC 5.4.0 20160609]
Legal docs and information: aka.ms/AzureCliLegal
I don't know why I cannot see aks version here.
The resolution to this issue was specifically about not spitting out the ugly error shown above when the connection drops. We separately need to handle the connection drops. I've filed issue #11580 for that and PR #10668 is in progress to solve it.
Most helpful comment
Can this issue be reopened?
I've tested the fix and it only seems to fix the problem partially.
When I run the fix, it no longer spits out the error, but simply exits the proxy.
You can test it as follows (tested this on a mac running 10.14.5):
Run
az aks browse --resource-group my-resource-group --name my-cluster-nameThe proxy will start and will open
http://127.0.0.1:8001Wait a couple of minutes without doing anything and you'll see that the
azcommand silently exits…