Supervisor: Better documentation on supervisor updating and DNS handling is needed

[ntompson]

So I've recently suffered from the CoreDNS issue nuking my local DNS server configuration (I logged an issue against hassos here). I acknowledge the great and rapid response by the hassio team. But through this experience, I learnt some things that surprised me. I think they are issues and should be addressed somehow.

Firstly: I had no idea that the supervisor is auto-upgraded. This is all kinds of wrong:

• I decide when to upgrade hassio / hassos and home assistant. I wait for others to try new major releases and wait for evidence of stability. I wait until I am home and can fix stuff if it breaks
• The supervisor is the only component of hassio that is auto-upgraded. This means that if I choose not to update home assistant or hassio / hassos, supervisor can get ahead, creating conflicts
• The auto-upgrading behaviour of supervisor is not (as far as I can tell) documented anywhere. That's not cool. At the very least, I should know what to expect.

Secondly: I now have no idea how I am supposed to configure a local DNS server.

• The HassOS guide implies that NetworkManager config is the way to go, but - in my case at least - that is now getting nuked by CoreDNS.
• I can't find any documentation to tell me whether NetworkManager config getting toasted is by design or is a bug (none of the current issues / PRs suggest to me that it is a bug).
• In fact, I can't find any documentation about CoreDNS or the CLI hassio dns command set.

I realise that my issue may come across as a rant, but this particularly experience is out of character with what I have come to expect from my 18 months of life with home assistant. Documentation is usually sufficient (if a little terse) and release notes are very clear and solid. I know what to expect and if something breaks it is because I did something stupid. But in this case, something broke, I hadn't touched anything, and I had no idea where to look. Like many people, I lost most of a day trying to figure it out.

I would be happy to contribute documentation on these two points, but I am completely in the dark as to what is expected. In an otherwise awesome project, this needs fixing.

[frenck]

I decide when to upgrade hassio / hassos and home assistant. I wait for others to try new major releases and wait for evidence of stability. I wait until I am home and can fix stuff if it breaks

This part is a little more complex and not as simple as it seems. I personally agree and hope it gets addressed in the future, but I don't see that happening in the short term.

This means that if I choose not to update home assistant or hassio / hassos, supervisor can get ahead, creating conflicts

That is actually not the issue, it is the other way around. The past is known, the future is not. Hence, the supervisor in the past is a big issue.

The auto-upgrading behaviour of supervisor is not (as far as I can tell) documented anywhere. That's not cool.

Feel free to contribute to our documentation, this is open source! 👍

For example: "I now have no idea how I am supposed to configure a local DNS server."

There are some assumptions here. You don't have to.

In fact, I can't find any documentation about CoreDNS or the CLI hassio dns command set.

All hassio command are documented, which is brought to your attention when you enter Hass.io by telling you to use hassio help.

I can't find any documentation to tell me whether NetworkManager config getting toasted is by design or is a bug (none of the current issues / PRs suggest to me that it is a bug).

This current issue has nothing to do with the NetworkManager, but I get the confusion. However, documentation is not for bugs, issues trackers are.

[SiliconAvatar]

@ntompson, 100% agreed.

As I see it, 3 major changes are needed to make this whole endeavor "mature", or at least not mickey mouse.

There needs to be a review process before major changes such as this. Similar to the Architecture repo for HA. This would allow the community to vet the validity and possible side effects of such a change.

HassIO Supervisor upgrades need to have changelogs with possible breaking changes same as standard HA releases.

And lastly, and most importantly, HassIO Supervisor releases can not, and should not, EVER auto-update. This should be an opt-in process to automatically update, and allow users to update when they desire. There should never be a situation where system breaks itself and all integrations without any user intervention. This latest issue revealed major point of fragility for this entire platform.

[SiliconAvatar]

All hassio command are documented, which is brought to your attention when you enter Hass.io by telling you to use hassio help.

@frenck , And how, exactly, were users to know such an option existed? The hassio dns CLI option never existed before, as far as I know. There was no documentation, release notes, or breaking change log which said this was now added, or that you might need to know about this now. I only learned about it from the github/forum comments. This isn't the right place to learn about such new options. This should be documented along with the upgrade, preferable before the user chose to upgrade.

[frenck]

There needs to be a review process before major changes such as this. Similar to the Architecture repo for HA. This would allow the community to vet the validity and possible side effects of such a change.

That is actually incorrect. CoreDNS is a replacement of existing functionality and not new.

[SiliconAvatar]

CoreDNS is a replacement of existing functionality and not new.

This might be correct (ok, it's 100% correct), the fact stands that this particular "upgrade" breaks a whole lot of things.

This replacement of an existing functionality is in many ways a full replacement of existing functionality, which has the possibility of multiple breaking changes. Due to this upgrade breaking everything for some people, I would consider it more than a transparent upgrade from one DNS method to another.

I understand that this is just a bug, in what should have been a 1:1 replacement, but the entire idea of replacing the DNS inside docker is a large architecture change IMHO. More so with hard coding in specific DNS servers and not respecting the DNS servers given out by my DHCP server.

[frenck]

This might be correct (ok, it's 100% correct), the fact stands that this particular "upgrade" breaks a whole lot of things.

I understand that this is just a bug, in what should have been a 1:1 replacement, but the entire idea of replacing the DNS inside docker is a large architecture change IMHO.

Bugs happen, unfortunately, not intentionally, and we are sorry. Did you participate in the beta to help out? I did, I ran both dev & beta versions, and I feel bad on missing this.

More so with hard coding in specific DNS servers

That sentence learned me how you are living in your own truth at this moment, making your comments unfundamented. Those are actually fallback servers, to keep your system going because of external reasons.

Thank you for your contributions.

[SiliconAvatar]

Did you participate in the beta to help out?

No. I was >1000 miles away on a business trip for the past few weeks, and my production server isn't for beta testing, though it seems I didn't have a choice since it auto-updates.

I understand bugs happen, and I do not generally mind them. My main grip isn't with the bug, it's with the automatic update when I was many miles from home and unable to fix it. I run beta versions when I have the time, but don't upgrade when I know I'm not able to deal with bugs or breaking changes. This experience broke that paradigm.

That sentence learned me how you are living in your own truth at this moment, making your comments unfundamented. Those are actually fallback servers, to keep your system going because of external reasons.

This is unnecessarily condescending. I wrote a technical response to it, but deleted it. It's not needed. There's no need to assume that you know better than the choices I make for my own network.

[frenck]

and my production server isn't for beta testing

We hear this a lot, unfortunately. However, we get a lot of complaints when a release goes south. At the same time, there are a very few actually helping out to get quality at a satisfying level.

This is unnecessarily condescending.

I'm sorry, it is not meant that way, however, it is incorrect and is a source of your answers.

There's no need to assume that you know better than the choices I make for my own network.

If that is the case, then none of this would apply to you. (Beside the bugs)

[mikemoyse]

That sentence learned me how you are living in your own truth at this moment, making your comments unfundamented. Those are actually fallback servers, to keep your system going because of external reasons.

The trouble is that for a fair number of people this did not keep their system going, quite the opposite in fact. Set defaults by all means but let those who can and want to, choose which servers to use. Many people run their own DNS server for various reasons usually along with a DHCP server which sets the DNS (obviously) which will set secondary DNS servers. While it's very nice of you to set up external DNS for failover, this is always going to fail for us so is useless and just causes confusion. If my system, breaks I know it's something I run, I'll fix it, I have my own redundancy and failover.

[KiLLeRRaT]

As with many here. It’s the auto update that gets me.

Bugs happen. It’s a fact of life, but either let us opt out of auto update or perhaps a better solution, let us pick a “stable” channel which perhaps lag the normal channels by a week or two? In that time, the problems may be ironed out and when our stable updates it would update to include any bug fixes that were made since the normal release. Perhaps the community isn’t large enough and too many people may jump on to stable heh.

Also, if it was documented then we’d know where to start looking.

We really appreciate your work and efforts, and sorry that it’s only when things go south, that you hear from the majority of us :)

[frenck]

"r perhaps a better solution, let us pick a “stable” channel which perhaps lag the normal channels by a week or two?"

We have 3 channels actually, stable, beta, and dev. They are used and none of the issues were picked up by any of the testers, including myself.

Which brings me to the fact that the number of users on our beta channels is really low. That is an issue.

[ntompson]

It's great to see such healthy discussion prompted by my comments.

@frenck: your point about lack of beta testers I absolutely get. This is a difficult problem and my approach isn't helping you at all. I suspect it comes with the territory: home automation affects the whole household, so the natural tendency is to treat the home assistant server as production. The family don't appreciate it when things go wrong (which happens often enough due to my own ineptitude). Having a second "dev" server doesn't really help: I can't virtualise all of the "things" that I am trying to control, so I can't imagine a setup that would be useful, without putting household harmony at risk.

Having said that, recognising the lack of beta testers - to me - makes a stronger argument for "opt-in" auto-updates. If we know testing is a problem, pushing out updates needs to be even more carefully orchestrated. Particularly problematic is the lack of consistency between supervisor (auto-update) and hassos / homeassistant (manual update) - this can only lead to confusion.

Coming back to the specific problem, I am still unclear what the intended behaviour is and how local DNS server IP addresses should be configured.

For example: "I now have no idea how I am supposed to configure a local DNS server."

There are some assumptions here. You don't have to.

In my case, I have configured my hassio RPi3 with static IP addressing. I do this using NetworkManager at the hassos level, and configure additional DNS servers - this seems to be the currently documented approach. One for my local DNS server, one for my router as fallback (which also has a local server) and then the usual fallbacks to google / cloudflare.

# cat /etc/NetworkManager/system-connections/my-network
[connection]
id=hassos-network
uuid=<redacted>
type=ethernet
permissions=
timestamp=1558765264

[ethernet]
mac-address-blacklist=

[ipv4]
address1=10.0.0.133/24,10.0.0.138
dns=10.0.0.157;10.0.0.138;1.1.1.1;8.8.8.8;
dns-search=
ignore-auto-dns=true
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

This method has worked reliably ever since I've been using HA (18 months roughly). I don't want to push my local DNS server by DHCP, as this would mean my HA server is dependent on the DNS / DHCP server being up, which would make the fallbacks rather pointless.

This method no longer works, and I still can't tell whether this is by design or by mistake. @frenck when you say I don't have to configure anything or that the current issue has nothing to do with NetworkManager, I am none the wiser.

What is the by-design method for configuring a local DNS server?

I have nothing but admiration and appreciation for the HA team. As a community, we are an ungrateful bunch who tend only to speak when something breaks. HA is simply awesome - there's nothing like it out there. I cringe every time I look at out of the box UIs that manufacturers provide with every "thing" I add to my home, ever thankful that I have HA to replace their crapware. But it is important that the community call out when something doesn't seem right.

[michalszymura]

It’s especially annoying when you run hassio supervisor on something like unraid, as it stops the container and basically does nothing, making the home assistant non functional until I manually update to the newest version. Being able to disable autoupdates is a must.

[ntompson]

So I can confirm that one or another of the most recent supervisor updates has resolved my underlying issue: my local DNS server configuration is now passed through from the NetworkManager config I have setup at the hassos level. Awesome and thank you to @frenck and the team for turning this around quickly.

But where are we at with automatic supervisor updates? In the community I've seen an outcry relating to automatic supervisor updates, largely triggered by this issue. It's OK to have issues - there will always be issues, but as a user, I have to be in control of when updates occur, so that I can manage the risk that the next update causes a new issue.

So what's the answer: how can we get in control of automatic supervisor updates?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.