Supervisor: Unhealthy State / Unsupported OS

Created on 3 Dec 2020 · 18Comments · Source: home-assistant/supervisor

Describe the issue

Addon installs/changes/deletes, and supervisor upgrades blocked due to "unhealthy state".
Investigating the errors/links provided by HASS showed a list of docker issues. These were all checked and are passing on my system (docker is latest version, no extra containers, etc). Additional messages state "unsupported OS". I'm running HASS Supervisor on a bare metal Arch Linux install, and have been for just under 18 months.

Steps to reproduce

Attempt to install or remove any addons. Attempt to update or downgrade HASS Supervisor/core.

Enviroment details

Version 2020.12.2
Newest Version 2020.12.2
Channel stab
Hostname hass.redactedyo.io
IP Address 172.20.0.40/23
Operating System Arch Linux
Linux hass.redactedyo.io 5.9.11-arch2-1 #1 SMP PREEMPT Sat, 28 Nov 2020 02:07:22 +0000 x86_64 GNU/Linux
Docker version 19.03.14-ce le

Supervisor logs

20-12-02 23:12:23 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.SETUP
20-12-02 23:12:24 WARNING (MainThread) [supervisor.resolution.evaluations.base] Detected unsupported OS: Arch Linux (more-info: https://www.home-assistant.io/more-info/unsupported/os)
20-12-02 23:12:24 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
20-12-02 23:12:24 INFO (MainThread) [__main__] Running Supervisor
20-12-02 23:12:24 WARNING (MainThread) [supervisor.core] System running in a unsupported environment!
20-12-02 23:12:24 CRITICAL (MainThread) [supervisor.core] System running in a unhealthy state and need manual intervention!

bug

Source

spurgelaurels

All 18 comments

Likely related to #2334 #2330 and probably a bunch of other incoming ones. Seems to be affecting everyone who doesn't run HASSOS.

I did find a temporary workaround by adding {"ignore_conditions": ["healthy"]} to jobs.json in my HASS supervisor directory. I posted this in a forum with a CRUSHINGLY MASSIVE WARNING TO ANYONE WHO TRIES THIS THAT IT SHOULD ONLY BE USED TO PUT THE SYSTEM IN A STATE WHERE YOU CAN ACTUALLY FIX IT and that applies here too.

spurgelaurels on 3 Dec 2020

👍1

I run Ubuntu on one of my systems and does not have issues

But this is however expected, and not a bug.
In the UI (under the unsupported message in the "Supervisor" card in my screenshot) there will be a similar text for unhealthy with a dialog that provides links to how you can fix the system.

ludeeus on 3 Dec 2020

😕1 👎1

But this is however expected, and not a bug.

I wholeheartedly disagree. This post wasn't created because I encountered an error; it was created because I encountered a critical error that when fixed using the information in the documentation it persisted. Furthermore, when HASS was told to ignore an unhealthy state, it continued operating just fine without any errors or warnings whatsoever.

In the UI (under the unsupported message in the "Supervisor" card in my screenshot) there will be a similar text for unhealthy with a dialog that provides links to how you can fix the system.

This is correct, however, as I stated in my initial description, I followed the link and instructions only to find that my system was actually properly configured and installed.

The Docker environment is not working properly

This links to https://www.home-assistant.io/more-info/unhealthy/docker which details what you're describing. I have detailed my docker version so it's not that. I am not running LXC so it's not that. I am not running any non-HASS docker containers, and even removed a bunch of older containers just to be sure,so it's not that. And finally, it's running as a privileged user.

spurgelaurels on 3 Dec 2020

That leaves 1 potential issue, the configuration of the docker runner, it needs to be using journald as the logger, and overlay2 as the storage driver.

ludeeus on 3 Dec 2020

Server Version: 19.03.14-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: journald

I really hope it leaves more than 1 potential issue.

spurgelaurels on 3 Dec 2020

Actually, that was removed as a reason.

For the unhealthy reason docker:

System runs as LXC
Supervisor is not privileged
Docker version (used_version > 19.03.0)
Having images on the host that has been blacklisted:
- watchtower
- ouroboros

Nothing else can set that status

ludeeus on 3 Dec 2020

Hence, why I've opened this as a bug.

To cover the other items:

~ » ps aux | grep hass                                                                         daphne@hass
root         836  0.0  0.0  10136  3608 ?        Ss   16:26   0:00 /bin/sh /usr/sbin/hassio-supervisor
root         859  0.0  0.0  10136  1928 ?        S    16:26   0:00 /bin/sh /usr/sbin/hassio-supervisor

The only containers I'm running that are not homeassistant or hassioaddons are zigbee2mqtt/zigbee2mqtt-edge-amd64:edge and tsightler/ring-mqtt:4.2.1 and this issue started when I was running zigbee2mqtt stable (found this error when I was trying to remove it to install edge so that I can help koenkk fix a completely unrelated issue)

I want to add as well, that this isn't just an error showing up in my logs. When this happened, I was unable to upgrade, install, or change anything with addons or the core system. This unfortunately left my entire home in a broken state, and came on very suddenly.

spurgelaurels on 3 Dec 2020

What do these say:

docker images -a
docker exec -it hassio_cli ha host info
docker exec -it hassio_cli ha supervisor info
docker exec -it hassio_cli ha info
docker exec -it hassio_cli ha resolution info

ludeeus on 3 Dec 2020

ha resolution:

issues:
context: supervisor
reference: null
type: update-rollback
uuid:
suggestions []
unhealthy:

supervisor
unsupported: []

slayorktc on 3 Dec 2020

docker images -a

homeasssitant/amd64-hassio-supervisor 2020.12.2
homeasssitant/amd64-hassio-supervisor latest

slayorktc on 3 Dec 2020

@slayorktc you have something completely different, you are already in discord so you should continue there

ludeeus on 3 Dec 2020

docker images -a

REPOSITORY                               TAG                 IMAGE ID            CREATED             SIZE
zigbee2mqtt/zigbee2mqtt-edge-amd64       edge                73d1ae852776        28 hours ago        588MB
homeassistant/amd64-hassio-supervisor    2020.12.2           a4a4fd8804e2        33 hours ago        289MB
homeassistant/amd64-hassio-supervisor    latest              a4a4fd8804e2        33 hours ago        289MB
homeassistant/amd64-hassio-cli           2020.11.1           8042e06b5b1f        3 days ago          83.3MB
homeassistant/qemux86-64-homeassistant   0.118.4             bb5bb150a2f7        7 days ago          1.13GB
homeassistant/amd64-addon-ssh            8.10.0              a6b2b897177c        7 days ago          158MB
homeassistant/amd64-hassio-dns           2020.11.0           41ebcae8f030        4 weeks ago         114MB
hassioaddons/vscode-amd64                2.8.2               fc1e5f2eb472        7 weeks ago         692MB
homeassistant/amd64-hassio-observer      2020.10.1           82619539144c        7 weeks ago         80MB
hassioaddons/node-red-amd64              7.2.5               e15eba9cc109        7 weeks ago         245MB
homeassistant/amd64-addon-letsencrypt    4.11.0              2a368d49a8b9        2 months ago        286MB
tsightler/ring-mqtt                      4.2.1               e507ba039f34        2 months ago        146MB
homeassistant/amd64-hassio-multicast     3                   3cefe14067ff        3 months ago        70.3MB
homeassistant/amd64-hassio-audio         17                  bce169b2d85f        3 months ago        94.3MB
homeassistant/amd64-addon-mosquitto      5.1                 345767b9a9ce        14 months ago       33.2MB

docker exec -it hassio_cli ha host info

chassis: desktop
cpe: ""
deployment: ""
disk_free: 87.4
disk_total: 107.2
disk_used: 14.4
features:
- reboot
- shutdown
- services
- network
- hostname
hostname: hass.redactedyo.io
kernel: 5.9.11-arch2-1
operating_system: Arch Linux

docker exec -it hassio_cli ha supervisor info


addons:
- description: Fully featured VSCode experience, to edit your HA config in the browser, including auto-completion!
  icon: true
  logo: true
  name: Visual Studio Code
  repository: a0d7b954
  slug: a0d7b954_vscode
  state: started
  update_available: true
  version: 2.8.2
  version_latest: 2.9.1
- description: Flow-based programming for the Internet of Things
  icon: true
  logo: true
  name: Node-RED
  repository: a0d7b954
  slug: a0d7b954_nodered
  state: started
  update_available: true
  version: 7.2.5
  version_latest: 7.2.11
- description: Manage certificate from Let's Encrypt
  icon: true
  logo: true
  name: Let's Encrypt
  repository: core
  slug: core_letsencrypt
  state: stopped
  update_available: false
  version: 4.11.0
  version_latest: 4.11.0
- description: An Open Source MQTT broker
  icon: true
  logo: true
  name: Mosquitto broker
  repository: core
  slug: core_mosquitto
  state: started
  update_available: false
  version: "5.1"
  version_latest: "5.1"
- description: Integrate Ring Alarm, Smart Lighting and other devices via MQTT
  icon: false
  logo: false
  name: Ring Devices
  repository: a98c085d
  slug: a98c085d_ring_mqtt
  state: started
  update_available: false
  version: 4.2.1
  version_latest: 4.2.1
- description: Allow logging in remotely to Home Assistant using SSH
  icon: true
  logo: true
  name: Terminal & SSH
  repository: core
  slug: core_ssh
  state: started
  update_available: false
  version: 8.10.0
  version_latest: 8.10.0
- description: Development build of the Zigbee2mqtt add-on
  icon: true
  logo: true
  name: Zigbee2mqtt Edge
  repository: 45df7312
  slug: 45df7312_zigbee2mqtt_edge
  state: started
  update_available: false
  version: edge
  version_latest: edge
addons_repositories:
- https://github.com/hassio-addons/repository
- https://github.com/rs1932/ring-alarm-hassio-addon
- https://github.com/tsightler/ring-mqtt-hassio-addon
- https://github.com/zigbee2mqtt/hassio-zigbee2mqtt
arch: amd64
channel: stable
debug: false
debug_block: false
diagnostics: null
healthy: false
ip_address: 172.30.32.2
logging: info
supported: false
timezone: America/Toronto
update_available: false
version: 2020.12.2
version_latest: 2020.12.2
wait_boot: 5

docker exec -it hassio_cli ha info


arch: amd64
channel: stable
docker: 19.03.14-ce
features:
- reboot
- shutdown
- services
- network
- hostname
hassos: null
homeassistant: 0.118.4
hostname: hass.redactedyo.io
logging: info
machine: qemux86-64
operating_system: Arch Linux
supervisor: 2020.12.2
supported: false
supported_arch:
- amd64
- i386
timezone: America/Toronto

docker exec -it hassio_cli ha info

issues: []
suggestions: []
unhealthy:
- docker
unsupported:
- docker_version
- job_conditions
- os

spurgelaurels on 4 Dec 2020

The docker version is the issue, that is not a version that can be checked,
https://docs.docker.com/engine/reference/commandline/version/ The version should report as X.X.X without any pre/postfix

Try reinstalling Docker or report that to the maintainer of the docker daemon you have installed.

ludeeus on 4 Dec 2020

Sounds to me like this might be a bug in Home assistant, as there are a lot of distributions that add a suffix to their version numbers. In this case, it's a distinction between community edition (CE) and enterprise edition (EE).

Docker version (used_version > 19.03.0)

Client:
 Version:           19.03.14-ce
 API version:       1.40
 Go version:        go1.15.5
 Git commit:        5eb3275d40
 Built:             Tue Dec  1 23:20:14 2020
 OS/Arch:           linux/amd64
 Experimental:      false

spurgelaurels on 4 Dec 2020

I'll also add that if this suffix is the issue, the version of docker itself is operating fine, and a string difference should certainly not introduce a broken state on a production system.

spurgelaurels on 5 Dec 2020

👍1

FYI: https://github.com/home-assistant/supervisor/issues/2334#issuecomment-739291720