Supervisor: Supervised installation not working on Docker 20.x

[andriej]

Describe the issue

Upgrading to the latest Debian 10 stable on 'healthy supported supervised' installation makes HA's components never come back again: supervisor, observer and all dockers including HA

Steps to reproduce

1. Have working Debian 10 installation
2. apt-get update && apt-get upgrade (docker-ce and containerd especially to version 20.x)
3. reboot (even it's not needed as dockers doesn't come back up)

After that, the only working 'docker ps' docker will be observer with no possibility to come back online.
restarting of the supervisor and doing ha su repair helps to get HA (and only HA, without addons) back online
But it's not persistent and lost after reboot.

Enviroment details

• Operating System:: Debian 10.7
• Supervisor version:: latest (2020.12.06)
• Home Assistant version: latest (0.118.5)

Supervisor logs

Supervisor logs

Supervisor doensn't come back up

System Information

System Information

arch: amd64
channel: stable
docker: 20.10.0
features:
- reboot
- shutdown
- services
- network
- hostname
hassos: null
homeassistant: 0.118.5
hostname: brix
logging: info
machine: qemux86-64
operating_system: Debian GNU/Linux 10 (buster)
supervisor: 2020.12.6
supported: true
supported_arch:
- amd64
- i386
timezone: Europe/Warsaw

What's quick-fix for those that have their whole house down, is to downgrade docker:
apt install docker-ce=5:19.03.14~3-0~debian-buster
and restore full-snapshot...

EDIT:
Full rollback as in alert until fixed:

apt install docker-ce=5:19.03.14~3-0~debian-buster
apt install docker-ce-cli=5:19.03.14~3-0~debian-buster
apt install containerd.io=1.3.9-1

[balloob]

Please do not respond with "same problem here", instead click on the 👍 reaction on the first post.

We are aware of the issue but will not have time to fix it until after the Home Assistant Conference.

[andriej]

@balloob maybe it's worth to post a blog regarding this or some social-media (FB/Twitter)?
I'm afraid most of users that has 'clean Supervised' will just apt upgrade like I did and face few hours of downtime (it took 4 for me before I decided to skip debugging and restored full snapshot with 21 addons).

[pvizeli]

It comes frome:

1) https://github.com/moby/moby/pull/41297
Look like they get reverted, because that is wrong inside systemd boot context of multi-users

2) https://github.com/moby/moby/pull/40491
Fixed by https://github.com/docker/docker-py/commit/1757c974fa3a05b0e9b783af85242b18df09d05d and we need to wait for an update

EDIT:
I asked for a new release: https://github.com/docker/docker-py/issues/2720
Reported issue on docker: https://github.com/moby/moby/issues/41772

[ludeeus]

To add some more context to this issue witch is also described in the alert

The issue has 2 parts.

Filtering has been removed from the API that lists container images, with that change the Supervisor is not able to handle the response correctly. It looks like that will be resolved in https://github.com/docker/docker-py/commit/1757c974fa3a05b0e9b783af85242b18df09d05d which is the library the Supervisor uses to communicate with Docker.

The second part, is changes to the systemd service for Docker, that change results in the service for hassio-supervisor being blocked at startup.

The quick-fix given in the issue description will for the most part work, but it's not enough, see the alert for a full rollback of Docker.

[pdecat]

Until support is implemented, here's the command to prevent packages from being upgraded:

apt-mark hold containerd.io=1.3.9-1 docker-ce=5:19.03.14~3-0~debian-buster docker-ce-cli=5:19.03.14~3-0~debian-buster

And the command to revert that:

apt-mark unhold containerd.io docker-ce docker-ce-cli

[aladin2000]

Describe the issue

Upgrading to the latest Debian 10 stable on 'healthy supported supervised' installation makes HA's components never come back again: supervisor, observer and all dockers including HA

Steps to reproduce

1. Have working Debian 10 installation
2. apt-get update && apt-get upgrade (docker-ce and containerd especially to version 20.x)
3. reboot (even it's not needed as dockers doesn't come back up)

After that, the only working 'docker ps' docker will be observer with no possibility to come back online.
restarting of the supervisor and doing ha su repair helps to get HA (and only HA, without addons) back online
But it's not persistent and lost after reboot.

Enviroment details

• Operating System:: Debian 10.7
• Supervisor version:: latest (2020.12.06)
• Home Assistant version: latest (0.118.5)

Supervisor logs

Supervisor logs

System Information

System Information
What's quick-fix for those that have their whole house down, is to downgrade docker:
apt install docker-ce=5:19.03.14~3-0~debian-buster
and restore full-snapshot...

EDIT:
Full rollback as in alert until fixed:

apt install docker-ce=5:19.03.14~3-0~debian-buster
apt install docker-ce-cli=5:19.03.14~3-0~debian-buster
apt install containerd.io=1.3.9-1

Same for me but i kept docker as is and fixed the
Addon coredns image not available

with :
````
ha supervisor repear
ha supervisor restart

````

[aladin2000]

Oups i just reboot the Host....
What is my work around ?

10 déc. 2020 18:27:22 mamrai1 [e-mail]:

No if you reboot the host supervisor will not start and it will remove all containers...

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub[https://github.com/home-assistant/supervisor/issues/2359#issuecomment-742669981], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AD7K4IZFQFEOT2QELXD4CHTSUEAHTANCNFSM4UUGNEEA].
[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/CAYAAABXXxDfAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAmSURBVGiB7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAJwY+QwABivJx1AAAAABJRU5ErkJggg==###24x24:true###][Image de pistage][https://github.com/notifications/beacon/AD7K4IY2R6XHMFO3WWBINRLSUEAHTA5CNFSM4UUGNEEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFRCD5HI.gif]

[aladin2000]

Is there a way to react before délétion  and perhaps boot in single user and fix on before normal reboot ?

10 déc. 2020 18:46:37 mamrai1 [e-mail]:

If you have portainer install start supervisor and wait.. It may need to restart it again from portainer to re-download home assistant. After that manually pull each image of the add-ons you use and then start them from the supervisor

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub[https://github.com/home-assistant/supervisor/issues/2359#issuecomment-742682315], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AD7K4I6Y5KE67P347F6SP2LSUECPZANCNFSM4UUGNEEA].
[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/CAYAAABXXxDfAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAmSURBVGiB7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAJwY+QwABivJx1AAAAABJRU5ErkJggg==###24x24:true###][Image de pistage][https://github.com/notifications/beacon/AD7K4I54TYLIWSRSWNFILN3SUECPZA5CNFSM4UUGNEEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFRCG5SY.gif]

[aladin2000]

So if I start downgrade docker , start all containers  except supervisor... Then i can manage the problem

10 déc. 2020 18:58:43 mamrai1 [e-mail]:

Only if they are running before starting the supervisor...

On Thu, 10 Dec 2020, 19:56 Aladin, [e-mail] wrote:

Is there a way to react before délétion and perhaps boot in single user
and fix on before normal reboot ?

10 déc. 2020 18:46:37 mamrai1 [e-mail]:

If you have portainer install start supervisor and wait.. It may need to
restart it again from portainer to re-download home assistant. After that
manually pull each image of the add-ons you use and then start them from
the supervisor

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub[
https://github.com/home-assistant/supervisor/issues/2359#issuecomment-742682315],
or unsubscribe[
https://github.com/notifications/unsubscribe-auth/AD7K4I6Y5KE67P347F6SP2LSUECPZANCNFSM4UUGNEEA
].

[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/CAYAAABXXxDfAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAmSURBVGiB7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAJwY+QwABivJx1AAAAABJRU5ErkJggg==###24x24:true###][Image
de pistage][
https://github.com/notifications/beacon/AD7K4I54TYLIWSRSWNFILN3SUECPZA5CNFSM4UUGNEEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFRCG5SY.gif
]

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/supervisor/issues/2359#issuecomment-742688596,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANVCNGABOSOP2Z57CKM3TLLSUEDTNANCNFSM4UUGNEEA
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub[https://github.com/home-assistant/supervisor/issues/2359#issuecomment-742690310], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AD7K4I6LCTJVSQUNIEI7QT3SUED5FANCNFSM4UUGNEEA].
[data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAD8AAAA/CAYAAABXXxDfAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAAmSURBVGiB7cEBDQAAAMKg909tDwcUAAAAAAAAAAAAAAAAAAAAJwY+QwABivJx1AAAAABJRU5ErkJggg==###24x24:true###][Image de pistage][https://github.com/notifications/beacon/AD7K4I7K4GRH65GIEZN5WD3SUED5FA5CNFSM4UUGNEEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFRCI4BQ.gif]

[pvizeli]

Please don't abuse github as chat system. We have discord for this.

This Issue is to track solutions and progress. All other is written on the alert: https://alerts.home-assistant.io/#docker_2010.markdown

[pvizeli]

For history. Docker will release soon 20.10.1, which solves the issue with systemd, and we can work around the other bug with the docker-library. 20.10.0 will never get working

[crisflashin]

This are the commands to rollback in ubuntu 18.04 (bionic)

apt install containerd.io=1.3.9-1
apt install docker-ce=5:19.03.14~3-0~ubuntu-bionic
apt install docker-ce-cli=5:19.03.14~3-0~ubuntu-bionic

and this, for prevent packages from being upgraded:

apt-mark hold containerd.io=1.3.9-1 docker-ce=5:19.03.14~3-0~ubuntu-bionic docker-ce-cli=5:19.03.14~3-0~ubuntu-bionic

For other distributions, you can check with

cat /etc/os-release

and you can replace the commands that contain ubuntu-bionic with your version, in this format:

%ID%-%VERSION_CODENAME%

(mods: if you want to merge this comment in the first want, you re welcome!)

[frenck]

@crisflashin The Home Assistant project does not support Ubuntu. Thus the initial post or alerts will not be updated with that information.

[hacker1024]

In the meantime, perhaps, could some sort of check for the broken Docker version be implemented? I had to ask on Discord to find the alert and this issue - it's not very easy to research.

[frenck]

If Docker doesn't work, there is not much to bootstrap that can check it. We have an issue here and put out an alert. If you are affected by this, you are running a Supervised installation, which is considered an advanced case and we assume one is capable of handling system administration.

[hacker1024]

If Docker doesn't work, there is not much to bootstrap that can check it.

Wouldn't a check against docker --version at startup suffice?

If you are affected by this, you are running a Supervised installation, which is considered an advanced case and we assume one is capable of handling system administration.

Fair enough, I suppose.

[ludeeus]

Wouldn't a check against docker --version at startup suffice?

Since nothing is starting, that would not work.

[jon-hedgerows]

If Docker doesn't work, there is not much to bootstrap that can check it. We have an issue here and put out an alert. If you are affected by this, you are running a Supervised installation, which is considered an advanced case and we assume one is capable of handling system administration.

I think you need to rethink what an "alert" means. There's nothing on the homeassistant.io website, nothing in the blog, nothing in the forums.

@crisflashin The Home Assistant project does not support Ubuntu. Thus the initial post or alerts will not be updated with that information.

ok, fine, I get that you don't support ubuntu (and completely get that you can't support every operating system that some user arbitrarily decides they want to use), but it wouldn't take a lot to update the initial post and the alert (what alert?) to point users at relevant user experiences on unsupported operating systems. That's just helping the community help itself, and that's always a good thing.

[bcutter]

If Docker doesn't work, there is not much to bootstrap that can check it. We have an issue here and put out an alert. If you are affected by this, you are running a Supervised installation, which is considered an advanced case and we assume one is capable of handling system administration.

I think you need to rethink what an "alert" means. There's nothing on the homeassistant.io website, nothing in the blog, nothing in the forums.

Today I decided to switch from HASS OS to Supervised Mode. Luckily I was made aware of the alert on https://alerts.home-assistant.io (https://alerts.home-assistant.io/#docker_2010.markdown). Guess how? By a user comment in a Home Assistant facebook group...

My two cents even I was lucky for today and could just downgrade the freshly installed docker as mentioned in the alert:
1) The alert site doesn´t provide any subscription method as far as I´m aware of. No eMails, no RSS feed, nothing. Do users need to have a look at this page manually?
2) There´s no mention on https://status.home-assistant.io.

So I have to agree with @jon-hedgerows by saying the communication part has a bit room for improvement. Maybe in general, even users running the Supervised mode need take care of a lot more things as documentation is saying, that´s the deal.

[JMVS]

Following on what others say, it might be a good thing to integrate those alerts into HA, like a notification.

[bcutter]

Following on what others say, it might be a good thing to integrate those alerts into HA, like a notification.

Well that might be too late. Sending notifications to a system which might already be down (cause e. g. in this case the base like docker is gone) is not very reliable. I´d vote for simply copy/pasting the existing eMail notification feature provided on https://status.home-assistant.io. But I guess for that discussion this issue isn´t the right place.

Let´s see how to fix this so updating to latest Docker will be possible again.

[thaJeztah]

Docker 20.10.1 was released with an updated systemd unit file, and should probably fix this issue

[aladin2000]

Great ....
But I will probably wait for other feedbacks before go to that

[pvizeli]

Thanks, @thaJeztah for the news

[pezlin]

Does this mean that it should be ok to run apt update/upgrade now?

[ludeeus]

You first need Supervisor 2020.12.7, when you have that you can update Docker

[andriej]

I can confirm it works - forced Supervisor update, apt upgrade and reboot worked (as usually) - flawlessly.

[thaJeztah]

I can confirm it works - forced Supervisor update, apt upgrade and reboot worked (as usually) - flawlessly.

Thanks for confirming, @andriej, good to hear!

[Whyrl35]

I also confirm that it works with the last docker-ce package

[pezlin]

I can confirm it works - forced Supervisor update, apt upgrade and reboot worked (as usually) - flawlessly.

Hw to force the update? Do I need to be in beta?

[ludeeus]

Instructions for that has been added to the bottom of this: https://alerts.home-assistant.io/#docker_2010.markdown

[ne0ark]

The latest version released 8 hours ago doesn't seem to work:

2020-12-16T16:50:54.971525000Z [cont-finish.d] executing container finish scripts...


2020-12-16T16:50:54.974430000Z [cont-finish.d] done.


2020-12-16T16:50:54.974644000Z [s6-finish] waiting for services.


2020-12-16T16:50:54.981835000Z [s6-init] making user provided files available at /var/run/s6/etc...exited 0.


2020-12-16T16:50:55.008861000Z [s6-init] ensuring user provided files have correct perms...exited 0.


2020-12-16T16:50:55.014174000Z [fix-attrs.d] applying ownership & permissions fixes...


2020-12-16T16:50:55.015743000Z [fix-attrs.d] done.


2020-12-16T16:50:55.016405000Z [cont-init.d] executing container initialization scripts...


2020-12-16T16:50:55.017646000Z [cont-init.d] udev.sh: executing... 


2020-12-16T16:50:55.037199000Z [16:50:55] INFO: Update udev information


2020-12-16T16:50:55.191551000Z [s6-finish] sending all processes the TERM signal.


2020-12-16T16:50:58.214902000Z [s6-finish] sending all processes the KILL signal and exiting.

[pvizeli]

If you run a supported system, please open a new issue