Supervisor: RFC: Updated build system Hass.io

[frenck]

Hi,

Let me get started by saying this is not actually an issue. I am trying to create a central point of discussion. Please do not take any of the following personally; I am trying to help the community and the project.

Background

In the past month, I have created a couple of Hass.io add-ons to contribute to the community. Those were well received by the community, for which I am thankful.

Soon I noticed weaknesses in the current setup provided by HA. I will sum up some of my concerns/frustrations.

Build environment:

• The enormous .sh scripts for building add-ons (and other images).
• A large amount of code duplication between all those build scripts and their minimal differences. Some of those differences where needed and some of those scripts seem to have evolved.
• The incompatibility of those scripts with MacOS. (e.g. sed -i does not work on vanilla MacOS)
• The file juggling in the build scripts (workdir copies) and the lack of error handling that might cause issues when Docker build fails.
• The magical 'search and replace' in Dockerfiles done in those scripts. (which was the only solution at the time)
• Build environment does not use all resources to collect information; it solely relies on config.json.

Generic add-on/images:

• Generic goto file is config.json. It is not actually configuring stuff, therefore wrongly named.
• Missing JSON schema for validation use.
• Lack of Docker label schemes.
• Hardcoded version numbers of the addons in configuration files (again config.json).

Other:

• The overall quality and use of coding standards in any bash scripts; not following any guidelines.
• The generic workflow and challenging implementation with external CI tools. (The add-on/images provided by HA do not have CI?!)

Not directly related to this discussion, but maybe solvable in the future:

• As a user, I am unable to choose which version of an addon to run. A user is only able to install and run the latest version. Even though there are enough version systems that could be leveraged to provide this? (Docker Hub, GIT tags).
• Add-ons (source) and repositories are the same thing. This makes it impossible to leverage git tags as a versioning system. Moreover, even more important, it has made separation of concerns quite hard. (e.g., running a repository and keeping the addon themselves in separate git repositories). I have worked around this issue, but it ain't pretty.
• Due to the current setup, it would be hard for Home Assistant to provide users a central interface to access/browse/search all community add-ons.
• Add-on topics in the forums are centered around add-on repositories, not specific add-ons. This makes the search for add-ons hard for the user (you will have to read/open each topic to find a list of provided add-ons). As a result, discussions in these topics are hard to follow, since they could be about any of the add-on provided by the repository.
• Missing requirement options (E.g., an add-on requires HA 0.80.0)

This list made me realize I need to change my own workflow in order to get more add-ons out.

So I started

So I started on working on a new build environment for my add-ons to solve a large part of the issues above. Soon I realized this could generally be used for all kinds of Hass.io images (not only add-ons).

The result of this is currently on GitHub: https://github.com/hassio-addons/build-env

Stuff I have added so far:

• Consolidated all build scripts into one single build script that resides inside the build environment image, thus removing the need for having build script everywhere.
• Set the build script as an entry point to the Docker container to allow direct execution.
• Leverages support for Docker build arguments, this removes the need for magic search and replacing stuff (requires a more recent Docker).
• Removed the need for local build workdirs/copies by streaming the build context into Docker build.
• Gather information from GIT
• Gather information from the Dockerfile
• Added support for the Docker label schema
• Added a shitload of error handling
• Adheres to the bash styling guides of Google and Dealerdirect.
• All shell scripts pass shellcheck.
• Keeping backward compatibility with existing add-ons and methods.

This allowed me to:

• Use CircleCI to validate add-on builds
• Use CircleCI to deploy build to Docker Hub
• Build images cross-platform without the need for specific shell scripts
• Leverage GIT version tags (if wanted)

Currently begin used on: https://github.com/hassio-addons/addon-base
CircleCI result example: https://circleci.com/gh/hassio-addons/addon-base/tree/master
See example PR results:

Showed my idea

At this point, I was getting confident this would be a great general addition to Home Assistant. So I have started contacting @pvizeli & @balloob. Their reactions were amazing, which drove me to improve things even more. (e.g., added support for building non-addon stuff).

Changes to Hass.io thus far

@pvizeli Started making changes to Hass.io. Things he is has done so far:

• Introduced build.json additional to config.json containing build information.
• Updating Docker version in Hass.io
• Updating Hass.io to support build arguments, thus removing the magical search and replace in Dockerfiles.
• Writing documentation.

https://github.com/home-assistant/hassio/pull/191
https://github.com/home-assistant/home-assistant.github.io/pull/3406

This all is great work! However, I think we need to do some more things before releasing all this into the public. Unfortunately, the PR was merged before I could raise my concerns and suggestions. Which is the origin of this GitHub issue.

Things I think we should do from this point

Remove config.json and build.json; introduce hassio.json

As stated earlier in my story, config.json is actually misnamed. There is nothing wrong with build.json, but it does introduce an extra file.

If you look at my build environment thus far, you will find that I use information from the current config.json (e.g., for the Docker label schema), but also the version of the add-on/image is in the config.json file. Adding it to build.json would simply duplicate information, which should be avoided in my humble opinion.

My suggestion would be to introduce a new file called hassio.json, to replace both config.json and build.json. This file would be the same as the current config.json however, extended with build requirements and information. Choosing this path would obviously remove the need for multiple files, duplicate data, and the misnaming issue.

Extend config.json (or even better hassio.json)

I would suggest extending the JSON configuration files on add-ons (and add them to other images).
The following format would be my suggestion.

{
    "name": "Homebridge",
    "slug": "folder",
    "version": "1.2",
    "type": "addon",

    "description": "Homebridge add-on for Hass.io",
    "maintainer": "Franck Nijhof <[email protected]>",
    "vendor": "Community Hass.io Addons",
    "url": "website with more information about add-on (ie a forum thread for support)",
    "source": "https://github.com/hassio-addons/addon-homebridge",
    "documentation": "https://github.com/hassio-addons/addon-homebridge/tree/master/README.md",

    "arch": ["amd64"],
    "from": "hassioaddons/base-{arch}",
    "arch_from": {
        "amd64": "homeassistant:amd64-base"        
    },
    "squashs": true,
    "args": {
        "my_build_arg": "xy"
    },

    "startup": "application",
    "boot": "auto",
    "webui": "http://[host]:[port:2839]/dashboard",

    "ports": {
      "123/tcp": 123
    },
    "host_network": true,
    "devices": [
        "/dev/ttyAMA0:/dev/ttyAMA0:rwm"
    ],
    "hassio_api": true,
    "privileged": "SYS_ADMIN",
    "map": ["config:rw", "ssl"],
    "environment": {
        "test": "henk"
    },
    "audio": true,
    "options": {},
    "schema": {},
    "image": "repo/{arch}-my-custom-addon",
    "timeout": 10,
    "tmpfs": true
  }

This would enable us:

• Add extra information to the Hass.io control panel (like maintainer, vendor, documentation & source code links).
• Provide information for Docker label schema on a build.
• Add extra Docker build arguments (suggested by @pvizeli!)
• Enable/disable Docker image squashing
• Specify specific images to build from on each architecture
• Allows you to specify the type of image (addon, core, supervisor, homeassistant, ...)

Create a JSON schema validation file

I have not created an example yet, but adding a JSON schema file would help developers. It could also be used in any build environment to validate the JSON file before starting the build process.

Update Hass.io build system

Update Hass.io build system to reflect above changes. The local build does not need to support all things. It should only be able to build for local use. (This approach is already the way @pvizeli currently works with).

Concerns

I do however have some concerns:

• The current change in the Hass.io build system lack backward compatibility. This does not have to be an issue, but I think I need to raise the concern.
• I think you should eat your own food, before serving it to others. We should migrate our own stuff on the new methods, before asking/presenting it to the users/add-on developers.
• Current build system change already on 'dev' branch. Maybe revert until we have found consensus? (conflicts with previous mentioned concern).

Other things to sort out

There are some other things to sort out:

• Provide extensive documentation (partially started on by @pvizeli 👍 )
• Provide updated example add-on, including Travis & CircleCI examples.
• Porting existing (core) add-ons? Adding automatic builds?
• Can we find a solution for versioning and rollback support?
• Update non-addon Dockerfile and images, including CI?
• Moving/re-branding/re-license the built environment under Home Assistant?
• Add support for custom build arguments on architecture level?
• Splitting add-on and the add-on repository? Maybe the repository should only reference add-ons?

Suggestions

Apart from all the things mentioned above, I would like to add one more suggestion:
Add backward compatibility to the Hass.io local build environment. This would give us a larger transition time and removes the need for a "big bang".

My personal to-do list regarding this all at this moment

• Build-env: Add support for additional build arguments
• Build-env: CircleCI trigger on release tag in GIT
• Build-env: Further testing with NON-addon Dockerfiles/images
• Build-env: Extensive documentation
• Example Addon: Create example addon
• Example Addon: TravisCI example
• Example Addon: CircleCI example
• At least port 1 addon to new structure

TL;DR

If this issue needs a TL;DR in your opinion, I guess you are not concerned enough with the problems I'm trying to solve. The main goal? Let's make Home Assistant more awesome. I would love to receive some feedback, and maybe we can sort things out to create a better base for Home Assistant and its developers.

[pvizeli]

The system is desine to use all time the last version. To allow install a old version with API is old degrade thing what will remove in future (maybe 0.65 or 0.66). The only hassio way to rollback is with a snapshot. There is a partial interface that is not implemented in the UI while we have no UI developer anymore. 60% of API feature will not supported on UI. That is the reason why I stop to add new feature until we found a new UI developer and they will add things that is not supported at the moment. The version problem will be solve soon. So you can only install actual addons and only move forward. I don't like system that run with old stuff. The auto-update will be also default enabled, since we can check if the new schema support the current and we can start the add-on after update.

I don't like to make big change in release. We move forward in small and fast steps. The merge windows for 0.64 is close and new PR go into 0.65. We make no wolpertinger.

Anyway we run an IoT device with less memory. The config.json hold all data they need on memory and need to have for run the images. build.json hold data that need only available for 10-15 minutes in a lifetime of a device. So I see no reason to hold this things also inside limited memory. Most of add-ons build by devoloper and push to a docker hub. We need never read this data or hold it present inside memory. Next is, we can not break stuff anymore only extend.

For run on IoT devices we need no magic like versions. This system don't look back it look only forward into feature. The snapshot feature is hard on limit for IoT device. Complexity make problems and simple things run persists.

But hassio is opensource, write code and make PRs. Instead to ask for some things, implement some things. Maybe your code can change some mindes 😃

The build scripts are not the best, yes. Hass.io are a very big project with a lot of parts. I work since feb on it and I can not have the fokus on every thing. It work and make my work easy for build stuff. I hope in future find time for work on this.

You can cook your own supe on your repository or try to find a consensus in the official repository. All ways are fine. You make a great work. But we can not merge stuff that have bad affect to end user.

The build-env or script build only a image and push it to docker hub. That is a small process of the hole circle. I can do the same also with docker CLI without any of ours build scripts. That is only a additional and helper to work with docker. We use our labels "io.hass" and that is what we need have. Other is also only a nice to have and not more.

The TravisCI or CircleCI don't allow to build and push stuff like we do. I think that is fine for small repository but we can not lost the favor for that tools on official homeassistant organisation. So we can not use that feature on the core repository.

[frenck]

Thank you for your extensive response.

So you can only install actual addons and only move forward. I don't like system that run with old stuff.

I do somewhat agree on this. But in the case of add-ons, you are dealing with third-party developers and all kinds of different quality. In my last month, I've seen examples of a bug introduced in a new add-on version release. The problem with this is that a user is unable to switch to the previous add-on version. Thus left with a broken add-on and broken functionality. From your perspective I agree, from the user perspective, I absolutely disagree.

I don't like to make big change in release. We move forward in small and fast steps. The merge windows for 0.64 is close and new PR go into 0.65. We make no wolpertinger.

Agreed, but those small and fast steps still can be part of a larger plan. Making wrong small steps can do just as harm as the large ones.

Anyway we run an IoT device with less memory. The config.json hold all data they need on memory and need to have for run the images. build.json hold data that need only available for 10-15 minutes in a lifetime of a device. So I see no reason to hold this things also inside limited memory. Most of add-ons build by devoloper and push to a docker hub. We need never read this data or hold it present inside memory. Next is, we can not break stuff anymore only extend.

This is a non-reason. Yes, we have to be sparse on the memory usage, but it would be easy to store only relevant information into memory. Carrying the contents of the whole JSON file in memory is, in my opinion, a bad design decision. There already is stuff in the JSON file that is not needed in memory all the time, so that needs fixing then I guess?

But hassio is opensource, write code and make PRs. Instead to ask for some things, implement some things. Maybe your code can change some mindes.

I'm not asking anything, I'm just opening discussion. My code did change some minds since you are implementing the basic idea, right? 😄

The build scripts are not the best, yes. Hass.io are a very big project with a lot of parts. I work since feb on it and I can not have the fokus on every thing. It work and make my work easy for build stuff. I hope in future find time for work on this.

I do respect the work done by you guys very much. 👍
Working first, pretty later.

You can cook your own supe on your repository or try to find a consensus in the official repository. All ways are fine. You make a great work.

As I already do. Thanks for the compliments though 👍

But we can not merge stuff that have bad affect to end user.

That is exactly my concern about the current changes and the reason for me to create this topic.

The build-env or script build only a image and push it to docker hub. That is a small process of the hole circle.

Agree. A lot of small process, however, make a big process.

The TravisCI or CircleCI don't allow to build and push stuff like we do. I think that is fine for small repository but we can not lost the favor for that tools on official homeassistant organisation. So we can not use that feature on the core repository.

I certainly do not agree on this. In terms of building CI/CD, this project is not that complex. Yes, it does have a lot of small parts. Approaching each small part individually will reduce the complexity of this effort.

[balloob]

I want to start with thanking everyone here for all their work on making Hass.io and Home Assistant the best home automation product out there. And I love the passion of trying to make it even better.

Let me introduce another angle into this discussion that has not been mentioned yet. We are all doing this in our spare time, trying to make something awesome. Yet we also have thousands of users that rely on it being a working system. So we have to juggle the time we have between the things we want to build vs the things that we have to maintain vs the contributions that we take. Each contribution takes time to review, test and, since functionality is getting introduced it makes for a more complex overall system. This increases the maintenance burden and makes future contributions harder.

And this is the reason why I support Pascal's vision on this. We have a working system today (and pretty straightforward) and we have kept it very minimal on purpose. This allows Pascal and others to focus on building out other new and exciting things.

Does this mean that this leaves no room for improvement of the current stack? No, there is. It just means that we have to be respectful of each other's time and focus right now on an evolutionary approach (making small improvements at a time) versus a revolutionary approach (doing a complete rewrite).

Hass.io is developed in the open. All work (including core devs) is contributed via pull requests to allow people to give feedback and raise concerns. So let's distill one improvement from this issue that you want to see, come up with use cases how this will help add-on development and let's talk about how this could be implemented as an incremental step to the current system.

---

Side note: I am 100% against more configuration options at all times unless there are really really good reasons. Each configuration option is going to introduce edge cases that we, the developers, have to deal with when people want to add even more features (which is happening all the time). At some point the system will be so big that no single developer can keep it in their minds and more bugs will be introduced.

Side note 2: For the Hass.io base system and images I also don't want to see auto push, as we'll need to do QA on it. These things are going out to thousands of people and a revert is not easy.

Side note 3: I like keeping our build and config files separate. By merging them, we're conflating concerns. It will lead to some properties becoming multipurpose which will eventually lead to us trying to expand the usage for one use case, forcing us to add workarounds to the other use cases that do not support the new usage.

[frenck]

Thank you for your answer @balloob. On most parts, I do agree.

Does this mean that this leaves no room for improvement of the current stack? No, there is. It just means that we have to be respectful of each other's time and focus right now on an evolutionary approach (making small improvements at a time) versus a revolutionary approach (doing a complete rewrite).

Hell yeah, I totally agree. My initial post isn't a "let us fix is all now!" post, it is about sharing a future vision. I'm all pro-"baby steps", nevertheless, I do think there should be a higher consensus about where this all is heading. This allows us to make those small steps in the "right" direction.

Side note 2: For the Hass.io base system and images I also don't want to see auto push, as we'll need to do QA on it. These things are going out to thousands of people and a revert is not easy.

I see your fear in this and maybe did not explain this feature on my end that well. I totally agree with your statement on this. The deploying mechanism created in my build environment is only occurring on explicit setting it (just like the current build scripts used by Home Assistant). So yeah...

Side note 3: I like keeping our build and config files separate. By merging them, we're conflating concerns. It will lead to some properties becoming multipurpose which will eventually lead to us trying to expand the usage for one use case, forcing us to add workarounds to the other use cases that do not support the new usage.

I think the decision of creating more configuration files (with possibly duplicate content, like the version number) conflicts with your own first side-note. Nevertheless, I guess I have to accept this veto. I will modify my environment to support the (additional) build.json. (I guess I'll have to in order to be compatible).

It seems like with this kinda "general" response, there is nothing left to discuss and I will, therefore, close this issue. Wishing you guys all the luck on the 0.64 release.