Supervisor: [Thought] Adding Kubernetes support

[titilambert]

Description of problem:

This is not a problem, this is just a thought/idea. It could be a bad idea.
Reading the Hass.io architecture diagram, I see the hassio daemon as a container Orchestrator.
Today, Kubernetes leads the Containers Orchestrator world.

So my main question:
Have you thought about adding support of Kubernetes or K3s (https://k3s.io/ which is a lightweight Kubernetes) to hassio.

The main benefits will be:

• Network managed by Kubernetes
• Kubernetes ensure itself that containers a running
• You can use other Raspberry Pie/machines to add "workers" to the Kubernetes cluster
• With the previous feature you add HA to your Home Assistant server
• Docker will die in few years, you don't have to manage the transition
• Kubernetes API is becoming the mainstream containers API
• You use a Yaml file to deploy a container in the cluster

The main downsides:

• You have to learn a new technology
• You have add Kubernetes support in Hassio daemon
• It's more complex than a simple bare-metal installation

BTW, I'm reinstalling my Home Assistant server, I will try to run it on K3s. I will try to post here the manual steps to get it done (if it's possible)

Thanks for your answer.

[pvizeli]

Well, I use Kubernetes for other projects. So I love Kubernetes.

Like you say, Hass.io is a container orchestration and will going more closer to Home Assistant in future i.e. with an ingress like service. And there are other things on the roadmap.

I see that really heavy to make an orchestrator on top of another orchestrator. Next problem is, we can't break exists installation and a migration path is currently out of my perspective.

And at last, the benefit is not very high. We have a better DNS service and can better be clustering of the devices. Kubernetes is only an orchestrator and docker/containerd the runtime.

But I'm open for any PR that going in that direction.

[titilambert]

@pvizeli thanks for your answer,
It's really interesting ! Is the roadmap opened (I mean, readable for everybody) ?
Do you think Hass.io will be more like a custom scheduler of Kubernetes ?

Thanks !

[titilambert]

@pvizeli In fact Hassio should be an operator ! (https://coreos.com/operators/)
The operator/hassio will handle the creation/update/deletion of all containers about Hass.

[titilambert]

@pvizeli I want to start patching hassio but I don't understand what commands I have to run to build the image of Hassio:

• What does it mean: Set the developer modus on updater.json ? where is the file, what is content ?
• I don't see any docker build command to build the image.
• If I want to run docker build -t myhassio . command, I need to supply $BUILD_FROM argument. What is the value for linux_amd64 ?

Thanks for your answers !

[Codename-11]

👍 I would be interested in running Hassio inside of a K8's cluster. I have a Pi K8's cluster set up right now and Hassio is the only part of my home-lab setup I was unable to implement :( . I could run regular Home Assistant but I'm already having to manage all of my other services and applications so having Hassio do that for anything related to it would take a lot of time and effort out of the equation.

[titilambert]

Well, I just tried to patch hassio to get it working with K8s, It looks like I have to remove a lot of stuff that's Hassio is doing:

• Networking
• DNS port forwarfing
• Docker image management ???
• AppArmor ???
• I guess some others ?
  It looks like Hassio is really linked to docker :(

For now, I will deploy home-assistant and some addons (mqtt, mariadb, ..) manually on k8s and document everything I did.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[larstobi]

@titilambert Have you documented anything yet? I'm also keen on doing what you described. I have used this https://github.com/home-assistant/hassio-installer to get it installed on a docker host I have, but I'm not satisfied with this since it's very much tied to a single machine.

[titilambert]

I have it working for more than 9 weeks, I'm starting to write a documentation. I will submit it as a blog post. I will link the PR to this issue

[titilambert]

@larstobi https://github.com/home-assistant/home-assistant.io/pull/9653/files

[pvizeli]

Sure we can add kubernetes as second API support as module like we use docker inside Hass.io / Would be nice if someone will work on that 👍

[titilambert]

@pvizeli thanks for your answer ! Could you guide me a little bit ?
Where should I start ?

[pvizeli]

@titilambert copy the docker folder and write this function to Kubernetes API. At last we need a layer they route functions to correct end-point. It will be only sucessfully if we start with a MVP

[xhalo32]

Hi!
I'm currently trying to get hassio running in kubernetes and I'm getting quite astral problems including
CRITICAL (MainThread) [hassio.supervisor] Can't setup Supervisor Docker container!

I'll keep you updated (and release the configs) if I make progress
Cheers, Niklas

[pvizeli]

You need to update the code. That is not an only config thing, because you need to add kub API support to supervisor analog of docker API support. I think that is 1-2 month developer work

[xhalo32]

Hey thanks for replying!
I noticed that you spawn the hass container in the python program which probably has to change for the application to work with k8s...

[billimek]

for what it's worth, home-assistant (not hassio) is already a helm chart for Kubernetes: https://github.com/helm/charts/tree/master/stable/home-assistant

[surdy]

IIRC on one of the recent HA podcasts @bgulla mentioned that we was working on K8s integration for HassIO. It might still be in early days and he might not be ready to open source it yet. @xhalo32 see if Brandon wants to collaborate with you on this.

[bgulla]

@surdy not a lot to share right now unfortunately. Based on how hassio would spin up additional pods, I would think it would be best to write a custom Kubernetes Operator for hassio. I havent dedicated the time to jump in again but willing to support a team.

[pvizeli]

Yes, Hass.io need support the API for kubernetes and use the docker API oder Kubernetes API based on the backend

[aenertia]

+1 For an Operator - Helm is ok but probably easier to do it in Ansible or if you want to fancy things try the go sdk. OR refactor the helm chart into operator hub. There are already mqtt and promethus/grafana ones to consume

https://operatorhub.io/

Operators are Kubernetes distro agnostic and THE supported way to do complex deployments going forward. Doing it this way also will give you automatic HA pretty much.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pastukhov]

it's nott looking difficult.
We nedd put something like

runSupervisor() {
    docker rm --force hassio_supervisor || true

    # shellcheck disable=SC2086
    docker run --name hassio_supervisor \
        --privileged \
        $APPARMOR \
        --security-opt seccomp=unconfined \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v /var/run/dbus:/var/run/dbus \
        -v "${HASSIO_DATA}":/data \
        -e SUPERVISOR_SHARE="${HASSIO_DATA}" \
        -e SUPERVISOR_NAME=hassio_supervisor \
        -e HOMEASSISTANT_REPOSITORY="${HOMEASSISTANT}" \
        "${SUPERVISOR}"
}

to deployment

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alexdor]

(just commenting so the bot doesn't close the issue)

[r9labs]

I'd also love to see this functionality. I've got it running via the helm chart: https://github.com/helm/charts/tree/master/stable/home-assistant

but its missing hassio/supervisor, which would be a nice thing to have support for in the helm chart. Also, keeping it from going stale again.

[frenck]

We have discussed this, and decided not to move forward on this for now.

If you want to use things like k8, we suggest using Home Assistant Core.

👍

[r9labs]

I don't think this should be closed, how else are you supposed to install things from the marketplace as the application developers intended?

[aenertia]

My perspective is that the problem runs to the core of how HA is designed.
It's not function or microservice based. Components are quite tightly
interlinked and it would need a serious re-architecture to make it
'cloud-native' in that sense so that it can be easily consumed.

I agree that since it's really a monolithic app the best deployment
approach is to just dump it in a VM like a lift and shift workload and
figure out High availability using legacy crutch techniques until that
happens. Containerising and breaking it up just increases the risk a
component dies and brings everything else down with it. So the utility of
having an Operator to install HA is limited until such a rearch is
undertaken.

It's a shame because on the surface it looks like it might be the right
fit, but once you start poking it's really just a monolithic app like
OpenHab - just in Python rather than Java.

On Sat, 22 Feb 2020 at 12:57, r9labs [e-mail] wrote:

I don't think this should be closed, how else are you supposed to install
things from the marketplace as the application developers intended?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/home-assistant/supervisor/issues/980?email_source=notifications&email_token=AACF5LYRBOWU4O6X7ZJ3CSLREBS45A5CNFSM4HCAT4ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMUPDTI#issuecomment-589885901,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACF5L47J6PGV6CVLZJALUTREBS45ANCNFSM4HCAT4ZA
.

[bgulla]

My perspective is that the problem runs to the core of how HA is designed. It's not function or microservice based. Components are quite tightly interlinked and it would need a serious re-architecture to make it 'cloud-native' in that sense so that it can be easily consumed. I agree that since it's really a monolithic app the best deployment approach is to just dump it in a VM like a lift and shift workload and figure out High availability using legacy crutch techniques until that happens. Containerising and breaking it up just increases the risk a component dies and brings everything else down with it. So the utility of having an Operator to install HA is limited until such a rearch is undertaken. It's a shame because on the surface it looks like it might be the right fit, but once you start poking it's really just a monolithic app like OpenHab - just in Python rather than Java.
…
On Sat, 22 Feb 2020 at 12:57, r9labs @.*> wrote: I don't think this should be closed, how else are you supposed to install things from the marketplace as the application developers intended? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#980?email_source=notifications&email_token=AACF5LYRBOWU4O6X7ZJ3CSLREBS45A5CNFSM4HCAT4ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMUPDTI#issuecomment-589885901>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACF5L47J6PGV6CVLZJALUTREBS45ANCNFSM4HCAT4ZA .

I don't think it's a question of the right architecture and wether the devs should break it out to run on a distributed hardware infrastructure. At the end of the day, 99.99% of the home-assistant users are going to just throw hassio on a raspberry-pi or a nuc. The point of home-assistant is to simply home-automation, if we force the user-base to learn kubernetes, the community will run for the hills.

If we were to rewrite the core scheduler, I would recommend deploying a lightweight k8s like k3s. Then all single-node scheduling would utilize k3s as the supervisor/scheduler, much like hassio today with the docker socket. In theory, you could then add nodes to the k3s cluster to make it distributed.

Would it be cool if we could hack hassio to use Kubernetes? heck yeah. Should we refactor the core of hassio just to please a few hundred homelabbers... probably not. Trust me, I want it (a lot) but only if it's not disruptive to overall home-assistant development.

[pvizeli]

All users they have k8s did not need an supervisor because that is handelt by k8s. They just need run Home Assistant Core as a container in his environment.

I know add-ons are nice, but you can do the same thing on k8s. But add-ons are complete wrong and follow a different concept as k8s with pods.

[zaneclaes]

I use Kubernetes for the deployment of everything in my home, including HA. Personally I think there are even more good reasons to do so than listed here... but that's neither here nor there. I have generally not missed the add-on store, but I'm interested in this question — if only so that my HA install is "less unique."

Sure, re-architecting the HA scheduler or using operators sounds great. But I think there may be a more direct way to at least have feature-parity. Based upon my understanding of add-on architecture, it seems that the most direct path is to be able to convert a config.json from an add-on in to a Kubernetes deployment.yaml. I haven't looked deeply at every option afforded by config.json but it doesn't seem _too hard_.

If that were to be implemented, I think the rest is rather straightforward. A side-car added to the HA helm chart with the appropriate RBAC could easily deploy the resulting yamls. The only part I don't have much insight into is how to integrate with the HA UI, such that the standard add-on UI could be used to control the sidecar...

... does that sound reasonable enough? I'd be happy to take a crack at implementing the config.json -> deployment.yaml and requisite sidecar + helm changes (I've already contributed to the latter)...

I chatted with the folks on the Discord server, and they explained to me the reasons this would not work. Notably, the Docker containers are "not just Docker containers." The add-ons are tightly coupled with the Supervisor API. For example, the AdGuard add-on broadcasts itself to Home Assistant through the Supervisor API. Thus in order to actually support the add-ons, one would need to effectively re-create the entire Supervisor!

[kyounger]

I agree that keeping feature parity in a deployment would [hopefully] be fairly straight forward. I think doing the deployments with a sidecar might be a bit awkward, though.

Perhaps using a proper operator pattern? It could "listen" to HA and when it sees a different config.json, it would go into action to spin up the proper deployment in k8s.

That, or just handle it at the source level, and maintain separate helm charts for each app.

Edit: replied to this before I saw the edit. As for replicating the supervisor api... couldn't that just be exposed as cluster-local Service? Not sure why you'd need to reimplement it.

I gotta say, it's a real shame to see an entire container orchestration engine being built in HA when there's a perfectly good one that's already pretty universally accepted. k3s and k3OS even solve this well for edge/IoT. Maybe I can find some time to jump into the discord and try to understand it at a deeper level. I'm still a huge HA advocate, and will continue to use it; just worried that the architectural direction is fighting what, IMHO, is a clear standard.

[zaneclaes]

@kyounger I'll try to respond, despite my ignorance on the topic, to save the devs some hassle.

As you pointed out, the Supervisor is a container orchestrator. Therefore, its API is analogous to that of Kubernetes' API. So what you're proposing is a translation layer between the Supervisor API and Kubernetes API. From what I can see of the source code (and talking to the devs), it's also handling things like multicasting, audio streaming, and authentication with HA-Core. You'd furthermore need to handle the DNS resolution format, which differs from Kubernetes' own DNS format (i.e., bashio::services vs. bashio.default.svc.cluster.local). So in order to wrap the Supervisor into a Service, you'd need to rewrite the internals such that they exposed the same API interface... yet delegated the work to Kubernetes, making appropriate tweaks to the requests along the way. Basically every endpoint here would need to be translated into something that instead queried Kubernetes.

Then there's the UI itself. I won't pretend to understand how the Supervisor interacts with the HA UI. But if we were to containerize the Supervisor as a Service, it'd need to talk to HA-Core and manage that. This may or may not be easy, given that the Supervisor was not written to be a containerized service.

Finally, with all those changes, we would have effectively forked the Supervisor code-base and would then be responsible for keeping it in parity with the master Supervisor.

I share your sentiments about the orchestrator, TBH. If we could find a path forward which the devs were happy with, I'd gladly offer my time to implement it.

[frenck]

Looking at the above discussion from the sidelines... And I really don't get it.

I gotta say, it's a real shame to see an entire container orchestration engine being built in HA when there's a perfectly good one that's already pretty universally accepted.

Yeah, we have our own orchestration, with a lot of tightly coupled features. There is nothing wrong with that? It is an alternative, you don't have to use it.

k3s and k3OS even solve this well for edge/IoT.

If it did the way we do, we didn't have to build our own. Please, keep in mind, this is not just container orchestration... There is a lot more to it.

just worried that the architectural direction is fighting

☝️ That makes no sense at all and suggest things which are not true. Our system uses containers, which are available to anybody in any system you like. You can pick up the Core container and use it with any system.

The part I don't get; If you are personally capable of running an alternative container orchestration system. Then you don't need all the other parts. Just the Home Assistant Core. All other applications are available as containers in general. Using those will give you much more control, and way less brain damage.

🤷‍♂

Furthermore, I'd like to request to move the further discussion on this topic to a different place (e.g., the Community Forum). From the Home Assistant side, this issue is closed.

Thanks for understanding.

[zaneclaes]

Thanks @frenck. In the interest of addressing some of your points, I've moved the discussion here.