Supervisor: Home Assistant can't connect to the outside internet

[Sephyr0s]

Hi there,

My Home assistant instance (running on Ubuntu on an old laptop) is having trouble accessing the internet. The docker image and Ubuntu installation itself however have no trouble whatsoever pinging for example google.com and the instance is accessible through the web. I can also still install add-ons. The issue is occurring since a few hours, and I wasn't editing when it started occurring.

2018-08-22 12:00:56 ERROR (SyncWorker_14) [locationsharinglib.Service] Could not open pickle file, either file does not exist or no read access.
2018-08-22 12:01:01 ERROR (MainThread) [homeassistant.components.device_tracker] Error setting up platform google_maps
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 56, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 180, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /ServiceLogin (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Try again',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/homeassistant/components/device_tracker/__init__.py", line 183, in async_setup_platform
disc_info)
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(self.args, *self.kwargs)
File "/usr/local/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py", line 44, in setup_scanner
scanner = GoogleMapsScanner(hass, config, see)
File "/usr/local/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py", line 63, in __init__
hass.config.path(CREDENTIALS_FILE))
File "/usr/local/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py", line 370, in __init__
cookies_file=cookies_file)
File "/usr/local/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py", line 198, in __init__
self._authenticate()
File "/usr/local/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py", line 234, in _authenticate
initial_form = self._initialize()
File "/usr/local/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py", line 240, in _initialize
response = self._session.get(url)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 525, in get
return self.request('GET', url, *kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
resp = self.send(prep, *send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 513, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='accounts.google.com', port=443): Max retries exceeded with url: /ServiceLogin (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Try again',))
2018-08-22 12:01:06 ERROR (MainThread) [homeassistant.components.cloud] Error fetching Cognito keyset: Cannot connect to host cognito-idp.us-east-1.amazonaws.com:443 ssl:None [Try again]
...
2018-08-22 12:02:05 WARNING (MainThread) [homeassistant.components.sensor.buienradar] Unable to retrieve xml data from Buienradar.(Msg: Cannot connect to host api.buienradar.nl:80 ssl:None [Try again], status: None,)

[frenck]

Are you running Ubuntu 18.04?

If that is that case, run the following as root:

echo '{"dns": ["8.8.8.8", "8.8.4.4"]}' > /etc/docker/daemon.json

Restart your system and done.

[Sephyr0s]

It's there (https://snag.gy/dkD5hU.jpg )

But still the same errors.

The weird thing is that downloading and installing add-ons works just fine...

And no, I'm on 16.04 I'll do the upgrade to 18.04 and see if it makes a difference.

[frenck]

@Sephyr0s I would strongly recommend sticking to 16.04 for now...
18.04 with Docker and the way Hassio uses it, can be a pain.

[Sephyr0s]

Alright so I didn't upgrade... any clue on what it can be in that case?

[frenck]

I have no clue.
The only thing I can think off is an outdated AppArmor profile.
This can be updated by running the Hass.io (Generic Linux) installer again.
(this can be run on an existing system without issues).

[frenck]

@Sephyr0s Have you tried updating/re-installing? Did it solve your issue?

[Sephyr0s]

Yes multiple times, seeing other topics about the issue I'm not alone though. I kinda gave up already and rendered HA unusible for now.

[emouawad]

Am i can confirm - I have the same issue on Ubuntu 16.04.5 - that's probably the DNS issue?

[pvizeli]

yes. Docker 18.06 don't support systemd resolver. You need set specific DNS server for docker in docker config or use NetworkManager like HassOS. The issue should be fix in next docker version

[TheCellMC]

same issue here. What can I do to get my docker's DNS fixed ?

[frenck]

@TheCellMC https://github.com/home-assistant/hassio/issues/663#issuecomment-415046978

[emouawad]

but doing that didn't solve it in my case - i have to run
docker exec hassio_supervisor socat UDP-LISTEN:53,fork UDP:127.0.0.11:53

[deftdawg]

@emouawad's recommendation worked for me as well...

docker exec $(docker ps | grep supervisor | cut -d' ' -f1) socat UDP-LISTEN:53,fork UDP:127.0.0.11:53
allowed name resolution to start working for my hassio instance running in docker. Before that I also tried manually overwriting /etc/resolv.conf and that worked for as long as the instance was alive, emouawad's workaround survives hass instance restarts.

Edit: Looks like changing it to my router's IP I can get lan and internet resolution working, sweet! 😄
docker exec $(docker ps | grep supervisor | cut -d' ' -f1) socat UDP-LISTEN:53,fork UDP:192.168.1.1:53

[emouawad]

glad it worked for you. Still i don't know of a permanent fix, have to execute this command on each restart or use some kind of autorun.

[frenck]

Try upgrading Docker to 18.09, which has an fix for Systemd Resolved systems

[emouawad]

Docker version 18.09.0, build 4d60db4
Home Assistant 0.82.0
Still not fixed

[deftdawg]

Hi All... expanding on @emouawad's workaround for this, I've created a shell script that will restart home-assistant and the supervisor every so often based on the number of socat connections.

To use it create a root crontab (user must be able to sudo restart services without password) with the following entry:

# Check every 10 minutes, restart HA at anytime if the number of socat connections is over 25000
  */10 * *   *   *    if [ $(ps aux | grep -v grep | grep socat | wc -l) -le 20 ] || [ $(ps aux | grep -v grep | grep socat | wc -l) -gt 25000 ]; then sudo /root/bounce-ha.sh; fi

25000 for the number of socat connections is just an arbitrary number, adjust as needed.

Next create the following script in /root/bounce-ha.sh (or change location in cronjob above to where you put it) and set this script +x:

#!/usr/bin/env bash

SUPERVISOR=$(docker ps | grep supervisor | cut -d' ' -f1)

if [ ! -z  "${SUPERVISOR}" ]; then
  echo "Shutting down Supervisor"
  sudo systemctl stop hassio-supervisor.service

  while [ ! -z "$(docker ps | grep ${SUPERVISOR})" ]; do
    echo "Waiting for Supervisor to finish shutdown..."
    sleep 1
  done
  SUPERVISOR=""
fi

sudo systemctl start hassio-supervisor.service
while [ -z  "${SUPERVISOR}" ]; do
  echo "Waiting for Supervisor to startup..."
  SUPERVISOR=$(docker ps | grep supervisor | cut -d' ' -f1)
  sleep 1
done

echo "Injecting supervisor with socat process to properly resolve DNS..."
sudo docker exec -d ${SUPERVISOR} socat UDP-LISTEN:53,fork UDP:192.168.1.1:53

HASSIO=$(sudo docker ps | grep homeassistant/qemux86-64-homeassistant | cut -d" " -f1)
if [ ! -z "${HASSIO}" ]; then
  echo "hass.io container already running, shut it down to force a restart..."
  sudo docker exec ${HASSIO} poweroff
fi

Replace 192.168.1.1 with the IP of your DNS (in my case my router is providing DNS).
If you're on a Pi instead of x86_64, you'll need to adjust the HASSIO= ... qemu-binary ... to your arch for it to work.

[JumpMaster]

I've had the same issue.
Ubuntu 18.04.1
Docker 18.06.1

I ran through the following article but the fix for me was to change /etc/docker/daemon.json from
{"dns": ["8.8.8.8", "8.8.4.4"]}
to
{"dns": ["192.168.1.1"]}

I then rebooted the Ubuntu host.

https://l-lin.github.io/post/2018-09-03-docker_ubuntu_18_dns/

192.168.1.1 is the IP of my Google WiFi router which also handles DNS and is the same DNS server used by the Ubuntu host.

[emouawad]

@JumpMaster And fix persists across host reboots?
The modified /etc/docker/daemon.json is the one from the host too, not a docker container right? (i.e: homeassistant or hassio_supervisor)?
thanks

[JumpMaster]

Yes that's correct. It survives reboots and the change was made to the host. No change to the docker containers. It seems to be Ubuntu/Docker related and not the fault of hassio.

I'm baffled why this change resolves the issue. 8.8.8.8 and 192.168.1.1 are both valid DNS servers. Without this change I would receive multiple HTTPSConnectionPool errors within the home-assistant.log relating to DarkSky and CloudFlare on their outbound connection attempts. This is typically triggered or worsened by restarting Home Assistant from the configuration screen.

Since implementing this change 12 hours ago I've performed several changes requiring HA to be restarted and have seen no further errors in the logs.

[emouawad]

@JumpMaster Unfortunately this does not work in my case.
So that we are on the same page: are you restarting your hardware or just homeassistant?

Ubuntu 16.04.5 LTS
Docker version 18.09.0, build 4d60db4

[JumpMaster]

To implement the fix I restarted the hardware (sudo reboot from the host OS)

[emouawad]

Not working - it must be something else.
Am thinking because i am on Ubuntu 16.04.5 LTS VS yours Ubuntu 18.04.1

[Jossie90]

I am also on 18.04.1, and the fix with daemon.json did not resolve the issue for me. I have temporarily solved it by running socat (jumpkick/emouawa's solutions).

[mattiaskagstrom]

I'm on Ubuntu 14.04.5 LTS with Docker version 18.06.1-ce, build e68fc7a and i have the same problem.

[br34k0r]

Same here with Ubuntu 18.04 - This is really annoying.

Is this problem only happening on ubuntu or also on other distros like centos?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.