Supervisor: hassio-supervisor fails to start and attach to an not running hassio_supervisor

[vajonam]

Describe the issue you are experiencing

hassio-supervisor fails to start and attach to an already created but not runing hassio_supervisor

On an ordinary reboot, I find that hassio_supervisor never restart correctly, and I cannot start with systemctl start hassio-supervisor.service

It looks like the /usr/sbin/hassio-supervisor is hanging on the docker start --attach hassio_supervisor.

I am running docker version

Docker version 20.10.4

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:   buster

What is the used version of the Supervisor?

supervisor-2021.02.11

What type of installation are you running?

Home Assistant Supervised

Which operating system are you running on?

Debian

What is the version of your installed operating system?

Debian GNU/Linux 10 (buster)

What version of Home Assistant Core is installed?

core-2021.2.3

Steps to reproduce the issue

Try and reboot with an updated debian system and notice the supervisor doesn't come back up.

docker rm -f hassio_supervisor, allows it come back up correctly.

Anything in the Supervisor logs that might be useful for us?

# Put your logs below this line

Nothing to log. 

[vajonam]

I have narrowed it down to this line.

https://github.com/home-assistant/supervised-installer/blob/db7e0e0a6d4c73c23496e38d5ad27ed31de8c714/files/hassio-supervisor#L37

[vajonam]

Think its similar to this. https://github.com/home-assistant/supervisor/issues/1529 but no resolution there. I was able to get everything to work as expected commenting out the first or of the || statement

# Run supervisor
mkdir -p "${HASSIO_DATA}"
# ([ "${HASSIO_IMAGE_ID}" = "${HASSIO_CONTAINER_ID}" ] && docker start --attach hassio_supervisor) || 
runSupervisor

[jhampson-dbre]

did this break with Docker 20.10.4? I haven't had any problem on 20.10.3.

[Ansarowsky]

I have the same issue, I can manually start supervisor by docker start hassio-supervisor, but it won't start by systemd. I have to wait for it to time out (1min 30s) when I'm stopping it too.

[vajonam]

did this break with Docker 20.10.4? I haven't had any problem on 20.10.3.

That is good guess. I did an apt upgrade a few days ago. Only noticed the non-startup of stuff after a reboot a few days later. So it’s good guess.

[vajonam]

I have the same issue, I can manually start supervisor by docker start hassio-supervisor, but it won't start by systemd. I have to wait for it to time out (1min 30s) when I'm stopping it too.

Which version of docker-ce are you running?

[Ansarowsky]

I have the same issue, I can manually start supervisor by docker start hassio-supervisor, but it won't start by systemd. I have to wait for it to time out (1min 30s) when I'm stopping it too.

Which version of docker-ce are you running?

20.10.4, I installed it an hour ago.

[scianos]

I can confirm the same behavior:

root@hac-primary:~# docker --version
Docker version 20.10.4, build d3cb89e
root@hac-primary:~# dpkg --list | grep docker
ii  docker-ce                            5:20.10.4~3-0~ubuntu-focal        amd64        Docker: the open-source application container engine
ii  docker-ce-cli                        5:20.10.4~3-0~ubuntu-focal        amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras            5:20.10.4~3-0~ubuntu-focal        amd64        Rootless support for Docker.
root@hac-primary:~# uname -a
Linux hac-primary 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@hac-primary:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:   focal

Fixed by setting the last line in the supervisor script to simply execute the runSupervisor function and kick off the docker container:

# Run supervisor
mkdir -p "${HASSIO_DATA}"
runSupervisor

[scianos]

One note re: my comment above - I now also see failures on udev and privileged container detection as well on this release; I'm digging in a bit to see if I can pinpoint where pyudev is failing on my system.... but that is likely a different bug report.

[jhampson-dbre]

So is docker start --attach hassio_supervisor being ran now, when on previous Docker version it wasn't? or was it running on both versions, but on 20.10.4 it is hanging or erroring out?

[scianos]

It looks like it's hard hanging on a futex/mutex re: the docker attach command as of 20.10.4 (NOTE - the SIGINT is me testing behavior of the process)

root@hac-primary:~# ps uax | grep 'docker start --attach'
root       17634  0.0  2.7 683688 56008 pts/0    Sl+  04:47   0:00 docker start --attach hassio_supervisor
root       18748  0.0  0.0   6436   672 pts/2    R+   04:48   0:00 grep --color=auto docker start --attach
root@hac-primary:~# strace -p17634 -f
strace: Process 17634 attached with 8 threads
[pid 17641] futex(0xc0003d8bc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17640] epoll_pwait(4,  <unfinished ...>
[pid 17639] futex(0x56442c778bf8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17638] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 17636] futex(0x56442c778d20, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17635] restart_syscall(<... resuming interrupted read ...> <unfinished ...>
[pid 17637] futex(0xc000076bc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17634] futex(0x56442c75bdc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17638] <... restart_syscall resumed>) = -1 ETIMEDOUT (Connection timed out)
[pid 17638] futex(0x56442c75b110, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 17635] <... restart_syscall resumed>) = 0
[pid 17638] futex(0xc000076bc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 17635] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 17638] <... futex resumed>)        = 1
[pid 17637] <... futex resumed>)        = 0
[pid 17638] futex(0x56442c75f9e0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=4, tv_nsec=999551361} <unfinished ...>
[pid 17637] futex(0xc000076bc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17635] <... nanosleep resumed>NULL) = 0
[pid 17635] futex(0x56442c75b110, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 17634] <... futex resumed>)        = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
[pid 17634] --- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---
[pid 17634] futex(0x56442c778d20, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 17634] rt_sigreturn({mask=[]} <unfinished ...>
[pid 17636] <... futex resumed>)        = 0
[pid 17634] <... rt_sigreturn resumed>) = 202
[pid 17636] futex(0x56442c75b110, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 17634] futex(0x56442c75bdc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 17636] <... futex resumed>)        = 1
[pid 17635] <... futex resumed>)        = 0
[pid 17636] futex(0xc000076bc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 17635] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>

[jhampson-dbre]

sounds similar to this issue in Docker engine repo
https://github.com/moby/moby/issues/42093#issuecomment-787100537

[remyderuysscher]

Workaround: By downgrading docker-ce and cli HA supervised starts again.

[firestrife23]

another temporary solution by holding back the upgrade by issuing this command:

sudo apt-mark hold docker-ce
sudo apt-mark hold docker-ce-cli

to cancel the hold when it's safe to upgrade.

sudo apt-mark unhold docker-ce
sudo apt-mark unhold docker-ce-cli

Editted for typos

[andriej]

How about issuing alert as it was done with previous docker issue @pvizeli ?

[vajonam]

even with the downgrade when there is a established container but not running (case of reboot). I am seeing some issues where the container doesn't start up cleanly. Think there are some other issues here too.

[cont-finish.d] done.
[s6-finish] waiting for services.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
[19:09:29] INFO: Using udev information from host
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[19:09:39] INFO: Using udev information from host
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...[cont-finish.d] executing container finish scripts...
exited 0.

all this goes away if we "rm" the container and start a fresh without doing a docker start or docker start --attach

[jhampson-dbre]

I don't know if this would play a part here, but docker install instructions list 3 packages docker-ce docker-ce-cli containerd.io. does containerd.io also need downgrade? I only see the other 2 packages mentioned in workaround above.

[remyderuysscher]

I don't know if this would play a part here, but docker install instructions list 3 packages docker-ce docker-ce-cli containerd.io. does containerd.io also need downgrade? I only see the other 2 packages mentioned in workaround above.

I have downgraded two HA supervised installs with docker-ce and cli. This works for Ubuntu (Odroid) and Raspbian (PI 4).

[belov-ve]

Same problem after Raspbian update (PI 3). Container don't start with key --attach (docker start --attach hassio_supervisor). Docker version 20.10.4
Downgrading docker-ce to 19.03.15 did not help.
Everything worked after reinstalling the Home Assistant Supervised:
https://github.com/home-assistant/supervised-installer

[Kryoh-hash]

Same here, i did updated my Raspbian for the latest docker-ce a couple a days ago, as soon as i did the supervisor went belly up when i rebooted my Pi. A few head scratches, and some hours looking in to it, I had to uninstall the docker.ce and docker-ce-cli and containerd.io downgrade to 20.10.3 and locked it, so far its working as used to be.

i use the : docker-ce=5:20.10.3~3-0~raspbian-buster docker-ce-cli=5:20.10.3~3-0~raspbian-buster containerd.io
to downgrade.

[sheminasalam]

I also downgrade one more docker component and now every thing is fine.

sudo apt install docker-ce=5:20.10.3~3-0~raspbian-buster
sudo apt install docker-ce-cli=5:20.10.3~3-0~raspbian-buster
sudo apt install docker-ce-rootless-extras=5:20.10.3~3-0~raspbian-buster

[belov-ve]

After recovery Home Assistant Supervised, everything works on docker 5:20.10.4~3-0~raspbian-buster

[azsak79]

I tried everything but it doesn't start :(

[ads1230]

For Ubuntu Server on Pi this worked for me

Find your specific docker version using apt-cache madison docker-ce

Then downgrade using

sudo apt-get install docker-ce=5:20.10.3~3-0~ubuntu-focal containerd.io
sudo apt-get install docker-ce-cli=5:20.10.3~3-0~ubuntu-focal containerd.io
sudo apt-get install docker-ce-rootless-extras=5:20.10.3~3-0~ubuntu-focal containerd.io
sudo reboot

everything running smoothly again.

[azsak79]

Im using rasperian os with docker.

[wes1993]

Did someone have found a solution? There is an official thread like the issue in December?

[vajonam]

This helps worked for a few folks

https://github.com/home-assistant/supervisor/issues/2642#issuecomment-787122346

look in /usr/sbin/hassio-supervisor

[wes1993]

Thanks a lot for your reply :)
Stefano

[mamrai1]

Upgrading to docker-ce to 20.10.5 seems to fix the issue...

[remyderuysscher]

Upgrading to docker-ce to 20.10.5 seems to fix the issue...

Confirmed.

[ads1230]

Upgrading to docker-ce to 20.10.5 seems to fix the issue...

All working.

[vajonam]

Yup works! Closing issue. docker bug. Based on the PR. https://github.com/docker/cli/pull/2987 It might come up again as moby investigates why this happens.