Core: RFC: Customization and attributes

[andrey-git]

Currently all types of attributes got into the attributes dictionary of each entity.

I would divide those attributes into 5 types:

(1) Variable attributes that describe part of entity state. For example battery_level of some sensor, or a brightness of a light.

(2) Constant informational attributes, that will never change: node_id, manufacturer, supported_features, etc.

(3) Mixed visual-semantic attributes. For example groups are visual, but the ability to turn the group on/off makes them semantic. hidden is mostly visual, but it causes entities to be excluded from history which makes it semantic.

(4) Backend-controlled visual attributes: entity_picture, icon, device_class. Those are used purely for visuals, but backend sets them directly.

(5) Purely-visual attributes for the sake of custom UI.

Questions

• Do we have more attributes of type (3)?

Thoughts

• It would be great to eliminate type (3) completely. This mix is confusing and complicates things. For example what if I have a sensor I don't want to see in the UI, but do want to see in history?
• We could split the attributes:
  
  
  
  • Type (1) stays as attributes.
  
  
  • Type (2) can be constant_attributes or properties.
  
  
  • Type (3) hopefully eliminated.
  
  
  • Type (4) either:
    
    
    
    
    
    • Move to a separate dictionary
    
    
    
    • Add device_subtype property and the frontend would set relevant properties on its own by domain + platform + subtype. This has the advantage of moving visuals out of the backend, and disadvantage that adding a custom platform will require adding frontend configuration. We could make this as simple as a json file however.
    
    
    
  
  
  • Type (5): Customization shouldn't be able to change type (1) & (2) attributes. It could change type (4) attribute or add a new one for custom UI sake.
  
  

[tinloaf]

Hi. I think it's a very good idea to clean up the attributes. Some thoughts:

• Please don't name group 2 properties. It's already hard to not confuse attributes with properties (those python methods annotated with @property. Which are technically attributes in Python, ironically.) If we now have another type of thing with the same name…
• For type 4:
  
  
  
  • What do you mean with "separate dictionary"? Something like attributes or constant_attributes?
  
  
  • If you take the device_subtype approach, then it's not possible to have dynamic symbols for entities, as it is currently the case with e.g. lights or batteries, is it? Would the entity change its subtype, e.g. from battery-70 to battery-60, to select between the different state-of-charge icons?
  
  
• I don't see the distinction between 4 and 5. Both are purely visual, customizations are allowed to change both…?

[andrey-git]

@tinloaf I agree regarding the name.

Yes, "separate dictionary" as visual_attributes or something.

Subtype is supposed to be constant. The "dynamics" would be implemented in the frontend, i.e. for a "battery" subtype - the frontend will pick an icon depending on battery_level attribute.

If (4) moves to a separate dictionary, then backend is still aware of things like "icon". Customization can override those or add new new attributes. If (4) moves to subtype, then customization can't override anything (there is nothing to override) but it can override frontend settings.

[OverloadUT]

/me puts on his Product Manager hat
The main thing I see missing here is an issue statement. I get the desire to separate the data out in to categories, but what problem does this solve. What issue are people facing right now that we want to solve, before we even think about the technical solution?

[andrey-git]

@OverloadUT For example there is discussion which attributes to show in the "more info" cards.
This approach makes a clear distinction between "types" of attributes.

[tinloaf]

@OverloadUT aside from what @andrey-git wrote, other things where I think a categorization of attributes makes sense:

• History platforms currently store a lot of useless data. We don't need to record the manufacturer of a sensor in the database every time the sensor changes. Neither do we need to store its symbol.
• The mqtt statestream platform, while being very useful, pushes a lot of useless stuff to MQTT - same as above. I don't need the manufacturer being published on MQTT if it never changes anyways.

[OverloadUT]

Thanks that's helpful.

I agree with your overall suggestion to move to 3 types of attributes: state attributes (which can change) and static attributes (which never change). The third and final category is all of the stuff related to how Hass uses and displays the entity, and perhaps could even be called "customization" or something similar.

As for hidden affecting history, to me that is a design issue with the history component itself, and should not be considered an issue with the hidden property. I would separate that issue from the one being solved here.

[tinloaf]

I agree. hidden should be split into hidden and not_recorded or something. This would have the added benefit that you could remove e.g. template sensors from history. Since their state at any time can be computed from the state of all other entities, recording them is mostly useless.

[happyleavesaoc]

I would call group (4) metadata.

[emlove]

Great write-up!

I'd throw out another name possibility of (1):state attributes and (2):entity attributes.

Another possible win is that by making this distinction we could make the websocket connection more intelligent wrt bandwidth usage. Category (2) attributes could be transmitted once when the state is first seen, then don't need to be transmitted with every state update. With this distinction it could be implemented in home-assistant-js-websocket transparently to polymer or other client consumers.

Also just bikeshedding, but I'd suggest that device_class belongs in category (2). Although it's currently only a visual distinction, it's not tightly coupled to our frontend implementation like icon/entity_picture are. It's just additional classification that describes the entity that clients can use how they choose.

[tinloaf]

To make sure nothing gets lost: please note https://github.com/home-assistant/home-assistant-polymer/pull/660#issuecomment-346162325 (just closed that PR)

[rytilahti]

While we are at it (making larger changes to attribute handling), would it make sense to explore ways to expose type (or unit) information for attributes in some form? Or would it be better just to enforce specific units for specific attributes?

A use-case is that currently there are some platforms (switch.tplink comes to my mind) which expose non-numeral values (see https://github.com/home-assistant/home-assistant/blob/dev/homeassistant/components/switch/tplink.py#L89) just for pretty-printing them in the UI, which forces one to do template tricks to get rid of the unit part.

Wrt metadata type (or 'constant attributes') proposal, would that be the place to expose information such as backend module name, its version, and bug reporting facilities? The use-case would be to allow adding "report a bug" / see platform information button, which would be allow prefilling issue descriptions (and thus allow automatic tagging of for component & platform owners), and making it feasible for end-users to check the version. This would also act as a sort of attribution and acknowledge the work done by backend lib developers.

[OverloadUT]

@rytilahti: I feel like this starts to get down a rabbit hole of attributes getting the features that states have to begin with, which says to me we have a design issue with the way attributes are being used in the first place.

Here's a crazy idea: What if we move "state attributes" (aka, attributes that change, like battery level, etc) to their own sensor entities, and allow entities to have _parents_ and _children_ for the purpose of UI grouping (and potentially even the way they are accessed via entity_id references).

So instead of:

• sensor.living_room_motion: detected
  ** (attribute) battery_level: 60%

We have:

• sensor.living_room_motion: detected
• sensor.living_room_motion_battery: 60

And then living_room_motion_battery can be configured (automatically when a platform is generating these or manually via configure) as a child entity of the main living_room_motion device. The UI would more or less display this the same way we do today with battery being a visible attribute when you look at the motion sensor device, BUT we gain all of the benefits of having the battery level being its own entity (such as unit of measurement designation, the thought that spawned this whole post.)

Many platforms _already_ work this way. Dark Sky creates a whole bunch of sensors rather than a single sensor with a ton of attributes. To me, I don't see a lot of consistency with regards to when a "sub-state" is its own sensor vs an attribute on a parent entity.

[OverloadUT]

To summarize the above, I guess there are only two changes in my suggestion:

• Allow sensors to be attached to parent entities, so that they show up a lot like attributes do today
• Start enforcing a design standard that any attributes that are really states themselves be split in to separate sensors and use the new parent feature

[andrey-git]

An issue no one touched yet is the group, which is both a semantic and a visual thing.
We could split it into "semantic" groups, where a service call on a group is propagated to all members, and "visual" groups that are configured at the frontend and the backend is not aware of them.

[OverloadUT]

Groups are their own entities; they aren't attributes.

But even if they are in some cases (?), I don't think making a distinction between semantic and visual properties has a lot of value. That's all "metadata" - stuff that Hass uses to do things (display or functionality, that's all just Hass stuff). That's why I suggested three types of Lists that can be on an entity:

• State Attributes (mutable, relevant to the state of the entity)
• Static Attributes (immutable info about the entity)
• Metadata (stuff Hass uses to do things: hidden, device_class)

[andrey-git]

Yes, groups are entities, but they also affect other entities presentation (entities in groups are excluded from default_view for example).
If I "just" want to "group" things visually I need to create an entity (which will tracks state, etc.)
This is separate from attributes, but in the same domain of semantic/visual mix.

[balloob]

_I realize that this has become more of a braindump then I wanted_

@andrey-git Great write up! I want to add a few minor updates to your list.

The first category we should call "State attributes": attributes that explain something extra about the current state. If a light is on, it will mention the color or brightness.

The second category is something that people like to add to their platforms and that we do a good job at not allowing. (I just removed the Tradfri ones that accidentally made it in #10399) However, I think that having something besides the state machine available for this would be a good idea. Changing this data would not fire any event nor would it be part of the serialized state object.

I would consider category 3 ("hidden") to be part of 4. These should indeed not change often. Things like icons representing the state of a device should all be dealt with in the frontend. I think that we can even do a better job here and not have the camera component change entity_picture every 15 seconds.

Category 5, "Purely-visual attributes". I prefer to refer to them as UI configuration attributes. I think that we have 1 as part of Home Assistant: control for the group entity. I don't like this category. It makes frontend development extremely hard because all of a sudden the UI can be in so many different states. I'm fine with them being used for custom UI as the authors of custom UI can choose themselves how complicated they will allow it to get.

Extra category 6: Directions for services that sync with Home Assistant: haaska_name, haaska_hidden. Built-in component that uses this is Alexa Smart Home. I'll make sure that this gets removed.

Thanks for putting the product manager hat on @OverloadUT, that's a good hat to carry around 👍

---

Home Assistant architecture is very easy to understand. By keeping it simple, we have been able to:

• allow developers to easily get started with development
• allow powerful extensions of the system like recorder and MQTT statestream without having to change anything about the architecture
• The architecture also supports replication (although we deprecated our implementation)
• get a whole bunch of dashboards, apps and other user interfaces for Home Assistant because it's so easy to integrate
• Want to try how something looks? Want to test an automation? Just change the state in the state machine.

I'm of the opinion that this simplicity has been one of the key drivers behind the success of Home Assistant. And simplicity is something that I will want to keep fighting for.

Simple systems are easy to understand, develop for and extend. However, the problem with each growing system is that it eventually will outgrow itself and we'll need a fundamental core change to be able to grow the next 10x. Past changes that fall in this category are the change from threaded to asyncio and on the frontend from NuclearJS + EventSource to WebSocket. Changing the way attributes work is however another magnitude bigger size change. This will impact everything that works/interfaces with Home Assistant. So let's be very thoughtful about this.

---

Would it be an idea that once we are able to decide on the categories that exist, we create a spreadsheet with known attributes and their categories?

[balloob]

btw, I wouldn't mind jumping on a voice call. Writing whole books is not always very helpful.

[andrey-git]

How are haaska_name and similar are used?

[OverloadUT]

How are haaska_name and similar are used?

For this example specifically, it is used to present a different name of the entity to voice interface services. You might want a group called "Living Room Front Track Lighting" inside Home Assistant, because that's what it is. But for _voice interface purposes_, you just want that called "Living Room Lights" because it's much easier and more natural to say.

[andrey-git]

I think control is the same as icon and entity_picture, i.e. something purely visual that the backend is aware of.

While type (2) shouldn't be part of state machine, I think it should be part of the API.

Here is the list of attributes that Entity extract from properties:
unit_of_measurement: Used by the backend for conversion and by frontend for display.
name, icon, entity_picture, hidden: visual properties (once we make hidden visual-only)
assumed_state: I'm not sure, does backend care about this or is it visual only.
supported_features, device_class: static attributes

[andrey-git]

@OverloadUT so it is not used by HA backend or HA frontend, just by external clients? So it fits in the same category as the rest of "for the sake of custom UI" attributes.

[emlove]

I'll throw out another thought for some brainstorming food:

We also have the case to deal with where we have smart home devices that create multiple entities in hass (i.e. zwave multisensor, etc.). The zwave component ended up moving the shared device state attributes (connection strength, etc.) into a completely separate entity in the zwave domain, so these weren't duplicated. Considering this use-case as well might help lead us to the right solution.

[tinloaf]

Another thought: What are the conceptual and technical differences between the string in state.state and the string in state.attributes[ATTR_WHATEVER]? The only real distinction I can come up with is that state.state is being used to give a short summary of what the entity is currently doing / measuring. But that's actually just a visual thing, isn't it? So, if we do a major refactoring anyway, why not get rid of state.state and move it into state.attributes[ATTR_SUMMARY] (or something). The UI then knows to use this attribute (if present) in state cards etc.

This would solve the problem with "physical things that have no state". For example, currently clickable devices ("buttons") are weird: Some platforms represent them as binary_sensor, some don't create entities for them at all but only fire some event when the button is pressed. In the latter case, you need an additional somewhat weird entity to report attributes like battery level, etc. If state.state was removed and state.attributes[ATTR_SUMMARY] was optional, these things could become entities like everything else and just not set ATTR_SUMMARY.

[balloob]

@andrey-git supported_features is dynamic. Based on the current state of a device, the features it support might change.

@tinloaf I am not a fan of that. Think about how confusing we're going to make it for users to get started. What is the most important attribute? With state.state it's clear that this is the actual state. It's the first thing you would say when describing a device. Light is on / off etc. We should keep it simple.

I guess my post was too long but the gist of the last part was this:

Let's go for an evolutionary approach versus a revolutionary approach.

We should not aim for a major refactor just to make the case of storing non-state data for an entity easier. The main purpose of Home Assistant is storing the state of entities. If we make that more complex, we're not doing the right thing.

I am a fan of a static data but they should not contain anything that would be needed for our frontend to render it. It would be extra information the user might be interested in (model, vendor, firmware).

Does that mean that custom UI config variables will be stored on the state machine? Yep but that's not an issue we have to solve for. This is a choice the custom UI authors make. They could have people put it as an object on the DOM too. (with this I mean that it should still be possible, just shouldn't have to perse be easier)

[andrey-git]

Custom UI customization can be huge and currently the only option of not putting it in the state machine is to use a custom component and put the config there instead of using customize.

[balloob]

Another option for customize UI is to have a config file that you load as an HTML import.

[andrey-git]

It will add an additional round trip.
Also writing configs in html is hard.

We could have a "customize" (another name needed) component that exports a config without putting it in the entities.

[balloob]

So I have toyed with that idea. We could move all Alexa/Google Assistant/Emulated Hue/Haaska config there too. My only concern is that now there are 2 places that hold configuration for a component. google_assistant: and homeassistant:/entity_config:

[andrey-git]

There can still be a single Customization config panel that will write to both places, so for UI config users the change would be transparent.

[tboyce021]

I would like to see some more consistency with what's shown as an attribute and what gets split into it's own entity, i.e. what's a "state attribute" vs a regular "state".

battery_level is an obvious one for me. Some components (e.g. roomba and owntracks) put battery_level as an attribute. For others (e.g. z-wave and my mqtt sensors from SmartThings) the battery level is it's own entity. The biggest issue currently with having it as an attribute is that it doesn't show up in the history and is slightly harder to work with in automations or show on the frontend in a separate "battery" view/group. The workaround for this is to create new template sensors that just extract the attribute into their own states.

Similarly, the nest component has config sections that do nothing but pull the attribute data into separate state entities for you. Unfortunately, with all of them enabled, it makes it difficult to see which sensors were created by nest since they're spread out over several different domains and don't all reference "nest". They only reference the device name, which may not contain the word "nest" and there can be several different devices all with completely different names. It would be nice to just have a single entry for each device with all the different "state attributes" grouped with that device.

Also, as was mentioned earlier, the "state attributes" don't currently have a good way to indicate the unit of measurement like regular states do. You either have to get it from another attribute or just decide what it is yourself. It's a bit odd to me that my climate.nest has a state of heat but the unit_of_measurement is °F. That unit of measurement is actually referring to the temperature attributes, not the state itself.

[OverloadUT]

@tboyce021's thoughts mirror my own. It's been days since I first mentioned it in a comment here, and I still feel that having the ability to have entities be "children" of other entities could be very nice. Then things like battery_level can ALWAYS be their own entities, but the UI can handle displaying them intuitively as, more or less, sub-entities of a main device.

[balloob]

It would totally be possible to add battery level to graphs if it is stored in an attribute. We already do this for climate entities. Although it is important not to forget the frontend, it should not drive any design decisions.

My issue with child entities is the complexity that we're adding.

[andrey-git]

The special hack for climate graph is a pain. I would prefer if we solved this without adding another hack.

I agree that the FE shouldn't drive BE design decisions, but the BE should expose enough data for _a_ frontend

[tboyce021]

I haven't looked at the code, but I would assume treating certain attributes as basically "less important" states would add it's own degree of complexity, but I have no idea which would add more.

The history and recorder components allow specifying a filter to include/exclude certain entities. How does this work for these state attributes? Does including the climate platform automatically include the operation mode, target temperature, current temperature, etc? If so, which ones does it include? It's not exactly clear (I honestly had no idea it even did this). And how would you go about filtering a subset of those out if you didn't want to save them in the database or show them in the history? Would we need to add configuration/logic to the history and recorder components to support this or is it somehow already supported?

I also would like these things to be presented more consistently. Is there a way to add an MQTT sensor that is a state attribute of an existing sensor? If all other components have battery level as an attribute, I would prefer to be able to do the same for my MQTT devices. Are we alright with adding something like an attributes config to all the MQTT components (and potentially others, e.g. template sensors) for specifying anything that would be considered an attribute?

binary_sensor:
- platform: mqtt
  state_topic: some_device/state
  state_attributes:
    battery_level:
      state_topic: some_device/battery
  static_attributes:
    firmware_version:
      state_topic: some_device/firmware

There's also still the issue of attributes needing to specify a unit of measurement, especially for components that may expose multiple state attributes with different units. Roomba, for example, has attributes for position, error, cleaning_time and cleaned_area. All of these could be useful for automation or history, but none of them specify the unit of measurement. I assume cleaning_time is in minutes and cleaned_area is either square feet or square meters, but there's nothing to indicate for sure.

I personally don't care what approach is used, I would just like to be able to have similar functionality and flexibility with state attributes as I do with regular states, and for things to be consistent across the various different platforms and components.

[balloob]

The recorder should always record every state change completely. If something changed that was not related to a state, it should not have fired a state change. Neither should it have been set via hass.states.set_state -> it shouldn't live on the state.

[BioSehnsucht]

Is it possible to expose data to the UI (such as via attributes) for display without recording it? For example, showing the hardware or software version of a device might be useful to display in the details of an entity, but might not be useful to log / graph.

[balloob]

So I think that for now I propose the following solution. This is a solution that will keep our API simple, which I think is one of our strengths.

• (1) Variable attributes will remain in the state
• (2) Will be moved to a new hass.info that contains static info for an entity that can be used for debugging or config (node_id, alexa_name). We can offer a shortcut on state objects that simply does self.hass.info.get(self.entity_id) so it's easily accessible in templates.
• (3) Mixed visual-semantic attributes will remain on the state machine.
• (4) Backend-controlled visual attributes will remain on the state machine
• (5) Custom UI attributes will remain on the state machine. Note that these are not _really_ our problem. The fact that custom UI uses states for configuration lives outside of HASS)

[andrey-git]

custom UI uses states for configuration lives outside of HASS because there is no built-in way to do it otherwise :)

[NovapaX]

About the entity and attributes (like battery levels) discussion above: I've always felt an entity should also be a 'real-life distinguishable' entity. So a battery level should always be an attribute of a entity (like a actual physical remote or sensor) and not be a separate entity/sensor. if the history/recorder does not handle that properly, that component should be changed.
I've mentioned my nest protect smoke-detectors earlier, 1 device generates 4 separate entities (sensors) and that is not desired behavior IMO.
But also a wall plug should always be one entity, the different attributes like 'power consumption', 'voltage', 'current' should never be separate sensors/entities.
I know this may be a separate issue, but having clear 'attribute guidelines' are part of the solution.
(the only issue I see here is that more configuration is needed to expose these relevant attributes in the front-end)

My solution for now would be: keep everything in the state machine, but start to group the different kind of attributes. If the need really arises it should be fairly easy to start extracting those groups into separate kinds of stores.
And make sure no unneccesary entities (and states) are created that actually should be attributes of an entity.
That should reduce a lot of clutter and noise in the state machine.

If we're talking about the event bus I think there are better ways to reduce bandwidth issues without extracting stuff out of the state machine. Like filtering unnessary attribute groups, not always sending both the full old and new states on a state-change, and extracting some very often triggered state-changes to seperate events which let the receivers pull the rest of the neccesary info out of the state-machine only if needed.
With regards to that last one, I'm thinking about the sun component, there should be better ways to use that then triggering a state change every 1 minute. (I actually don't really know why it does that, the state doesn't even change every minute.)

[tinloaf]

I fully agree with @NovapaX - I think if we try to improve how states / attributes work, we should try to achieve a consensus about when (and I vote for "never") to break out attributes as separate entities. I've seen many discussions about this in PRs (and been part of some), and I think coming to a general decision about this would also benefit the unification of the idea about what states / attributes are. Any thoughts on this, @balloob ?

[OverloadUT]

Now see, I feel exactly the opposite here.

I think requiring the user to know "what physical device is measuring UV index in the living room" in order to determine if it's an entity or an attribute is silly. It's a sensor, so it should be a sensor entity.

To put it another way: If I have two devices that track motion and vibration respectively in the room, and then I replace those two devices with a multisensor that tracks both things in a single device, _nothing should change_ in Hass. There were two things being measured before, and there are two things being measured now. Two sensors before, and two sensors after.

I really REALLY don't like the idea that power and energy readings would be attributes on a switch. The switch is a _switch_ for turning on and off. The energy and power readings are _sensors_ because they are sensing something and reporting the changes.

It would also create problems when you're trying to determine what exactly "real-life distinguishable" means. If you have a weather station that tracks wind speed and wind direction (one physical device) and temperature (a second physical device), under this policy wind speed would be an entity, with direction being an attribute, but temperature would be a separate sensor. To a user, this would be very confusing: why on earth are two of my weather readings under one entity, but the third is its own? Just because it's a discrete device on the roof?

All of these examples are saying the same thing: To me, the whole point of a big fancy controller like Hass is to abstract away the nitty gritty of how various HA platforms and sensors and whatnot work.

I would be more in favor of eliminating non-static attributes entirely than going the other way around.

[NovapaX]

That makes sense too.
But when I add my smoke detector to hass I expect to see one entity state card with a state being 'safe' and maybe some additional secondary info. I don't expect to see separate 'conectivity status' and 'battery level' sensor state cards for every device.

I agree things are arbitrary, the multisensor example is spot on. I would expect those to be two separate sensors indeed.

But when I add my smart meter, 9 (ungrouped) entities get added to the state machine.
That may not be the biggest problem for end users, but they are not grouped in any way so it takes quite some work to get that sorted out in the UI, and then still it takes a lot of space for info of which I (and probably almost every user) actually only want a summary state-card and more info in the more-info-panel.
And should a meter-update (every 10 seconds) really trigger 9 state-object-changes when it could (and probably should) just be one event?

Of course everybody can start creating custom UI for these kind of things, but I think it's just bad UX.

EDIT: I'm not saying it should be one way or the other. I'm just saying there should be clear guidelines as to when a separate entity is warranted and which things should just be attributes.
And what state.state should be like. (for which state.summary would be a better name then IMO)
For a smoke detectors I just expect a state of 'safe/ok' or 'alarm'
And for my smart meter I expect the state to be the calculated power consumption (positive or negative)

[pvizeli]

I would prefere the info data to merge it to state.info after it comes from statemachine.

[balloob]

So I'm more with @OverloadUT here.

@NovapaX we should not conflate the concerns of the user interface and the backend. It's totally possible for the backend to have it all split up and the frontend to still show it grouped. It also allows cool things like what we do with our generic thermostat. It takes in a switch that controls a heater and sensor and mimics a thermostat by turning the switch on when it gets too cold.

This reminds me of this great video that I saw the other day about 1st order and 2nd order synthetic sensors: https://youtu.be/aqbKrrru2co?t=217 (the whole video is worth a watch btw)

Alright, back to topic:

We have done a great job to make sure that the core is not aware of what it is used for. This has allowed the core to stay small and lean. Just a little over 1100 lines including an async and a sync API.

We're breaking from this when we want the core to become aware of which attributes it stores and what they mean. It also means that it will limit what custom components can achieve, as their things might not "known" by the core. That's not a good path to take.

With hass.info we are adding a new piece to the core where we can store static info for entities that are not related to the current state. Think manufacturer, firmware version or if an entity should be synced over to Google Assistant. This doesn't mean that the info can't get updated, it's just that the frontend or other consumers don't have to listen to all changes because they can assume it's static and not being up to date is not the end of the world.

[NovapaX]

Yes, I suppose you are right and I was trying to solve the issue from the wrong angle.
But for UX sake there must be guidelines and a little focus as to how things are exposed by default components and what UI is used.
We should just be able to assume state.battery_level is the battery level of the device (and a value between 0 and 100 or maybe a couple of keywords), and if its not there, there is no battery level to report in the UI or elsewhere.
Convention over configuration.

On the other hand, storing everything as separate entities is fine too and for the core doesn't really matter if a battery level is stored as a separate entity or an attribute. (if it's changed separately and acted upon a lot I suppose it could as well be a separate sensor.)
But it is a bit confusing you could have switch.kitchen_light and sensor.kitchen_light_switch_battery_level two kind of related entities completely uncoupled from each other in the state machine.
If there was some kind of grouping or relational info in the state machine it would be a lot easier for UI to group things accordingly. (and for future custom UI components/wizzard to automatically select related entities.
The platforms are still and always should be responsible for what is stored in state and info. With that I fully agree.

The core doesn't need to know what attributes it stores and what they mean. But there should be consistency and caution in the platforms about which, and how much, entities are created.
For all I care the hass.info could as well just be 'hass.state[entity].info'
Or maybe it should actually have been. hass.entities[id].state and hass.entities[x].info but that would be a major architectural and code change and we probably don't want to open that can of worms.

[tboyce021]

I agree with @novapaX mostly in the need for consistency. It's rather annoying trying to figure out what is considered a separate state and what is considered an attribute.

I rather like the approach SmartThings uses. The UI lists every devices, which could be handled by the frontend so no change needed for that. Under each device, there's a list of "current states", this could just be a list of every state/entity that gets created by the device, including battery level, switch status, temperature, etc. Everything that changes dynamically is considered a state, so there's no need to figure out if battery level or temperature is a state or attribute for each individual device.

With that, there's essentially no need for "attributes" on each state/entity (some of which are created by the same device, so things like firmware and battery level either get duplicated or only stored with one of the states and not the others). Instead, attributes would be device-specific. SmartThings handles this with a separate "data" section under each device, which stores things like manufacturer and model info. They also have a separate, "firmware" section, but for simplicity we could probably leave that as part of data/info/attributes/whatever we call it.

So I also agree with the last comment that something like hass.entities[id].states and hass.entities[id].info would make sense, or for even more clarity, hass.devices[id].states[state_id] and hass.devices[id].info[info_id]. Of course, we could still provide a way to access the same information via something like hass.states[device_id]['battery_level']. Taking that one step further, we could even leave the current API and just make entity_id a concatination of device_id and state_id and it would look identical to what we already have in many cases today: hass.states['device_id_battery_level'].

With that, it's obvious that anything that changes dynamically is a state and on the state machine and anything that is just informational is in the info. No need for this state vs. attribute stuff.

[BioSehnsucht]

I like the idea of a separate info for things which aren't relevant to the state.

In regards to the issue of multiple sensors/devices being related to one physical/logical device (such as the example given of kitchen light switch which also has a sensor for it's battery level), perhaps add a property / attribute related_entities (or whatever name works best) which is a Dict/List/whatever of all the entities related to the device? Then the UI could use this to find all the related and group them if desired, or not - but the core doesn't have to do anything about it, the Dict or whatever is managed by the platform/component that created the entities.

[OverloadUT]

_Some_ sort of relationship mapping between entities has been suggested several times (albeit in different forms) in this thread by different people. My early suggestion was to have "children" entities of a parent entity.

It really feels like something like this would be very beneficial, at least for UI purposes.

[rytilahti]

So some thoughts (I'm mostly agreeing with @OverloadUT & @tboyce021 here):

1. I would also like having children-entities ("sensors"), which should also be initializable by the platform in question. That alone is the reason switch.tplink does not currently expose sensor entities; it would have been too burdensome split it out to multiple pieces of code, while also making the code less maintainable. The current way of encoding the units of measurement into attribute names feels really hacky, especially when we have sensors which provide a proper interface. As others have mentioned already, this would also simplify building UIs (UI elements presenting entities, among them their sensors) by making integration of sensors from a specific device into one state-card a breeze.

2. The child-entities should be linked also to their parent. This will also simplify the API usage for external apps (I'm unsure how e.g. floorplan currently finds out what sensor entities belong to which entity, or is there no link at all?), as all it would require is to go through the child entities when looking for a battery sensor of "kitchen.sink". Or to put it other way around, it would allow going through all "battery" sensors and figuring out the parent device if needed (e.g. for notifying that it's time to change battery of entity X).

3. I think that the platform developers are more knowledgeable than the users to choose which sensors should be exposed (per default at least) to the state tracking, that is "template sensors" is "a hack" to fix the lack of proper ways for exposing this (implicit) knowledge, so here I disagree with @NovapaX's "prefer attributes over entities" -- we should be explicit about what _unit_ is a certain value is. The developer should have an explicit way for stating that the "total_consumption" of "kitchen.sink" is delivered as watts, leaving it to the presenter to do a conversion if wanted. Furthermore, elevating an attribute to a sensor to be tracked should be configurable, if someone really wants to track static "firmware version" information all along.

Short example, instead of the following in the config:

    kitchen_amps:
      value_template: '{{ states.switch.tplink_kitchen.attributes["current"].split(" ")[0] | float }}'
      unit_of_measurement: 'A'

I would prefer to have tplink_kitchen.current sensor any day. The author of switch.tplink should know, that current is reported in units of 'A'.

1. I don't like the idea of having a separate info living inside of hass namespace. This will further complicate the lives of developers (especially of those using the rest api) with no apparent reason -- I feel like those _are_ attributes of the entity, not some separately living properties. If info is really wanted, there should at least be a clear link between the entity and this information, so that the information can be linked without knowledge of internal workings of the hass daemon.

I'm really starting to like the idea of @OverloadUT about getting rid of non-static attributes (they should be sensors anyway, right?) and keeping the static attributes as part of the entities, as wild as it may sound. To put it shortly: I would be for making the attributes (where it makes sense) behave like sensors with measurement units and all, which would then be exposed for recording. This way the developer may choose to expose the units without converting it themselves and just letting homeassistant (or anyone else using this information) to handle the conversion as needed/wanted (this is something quite a lot of javascript-based frontends do e.g. with date information, right? The original value being stored as a UNIX timestamp, which gets converted by the UI to wanted timezone etc.).

tl;dr: 1) introduce linking between parent and children entities, useful for GUI and external API users, 2) make sensory information be real sensors with units, this could be done by introducing units to attributes (and allow elevation to real sensors if configured so), 3) make developers choose which attributes/sensors are exposed per default.

[BioSehnsucht]

Instead of info being attached to hass, it could just be a member of an entity (as we have attributes now which are tracked in history etc, it would be the "static" / untracked equivalent)? Perhaps I lack understanding of something core but wouldn't states.switch.tplink_kitchen.info["version"] work just fine if you added an info Dict to the switch.tplink implementation without making any changes to hass core ? So all you'd need to do is define this as the new standard way of storing static information, and adjust existing platforms/components going forward (breaking changes, but no need to rush into it either - could have both for a while and indicate that the static attributes would be deprecated - until then can simply add all the values from self.info to the Dict being returned in self.attributes, so as to not duplicate code) ? There might be some additional work to pass the info Dict along with attributes to the UI, but that is surely very minor, just copying the attributes code as changing it to info? Then implementing what to do with it on the UI side is a UI problem, separate from hass core.

For setting the default visibility of attributes/info entries in UI, could have optional Dicts attribute_visibility / info_visibility where they are basically a map of attributes/info key name as the key and true/false as to show by default. If they don't exist, assume all should be visible by default.

As for parent-child relationships, could perhaps add child entities to attributes and wherever necessary the logic would check whether the attribute is a simple attribute (i.e., bool, str, etc) or an entity object (in which case we know it's a child entity) ? Require that relevant magic methods be implemented on child entities and they can even naively appear to be regular values for any code that doesn't know how to handle this yet.

[balloob]

If I would have to summarize some of the thoughts after my proposal about hass.info, people would prefer if:

• each attribute that is an individual measurement to be extracted as a sensor
• entities be able to maintain a relationship (be able to group them per device)

These are all great ideas but none of these solve where we're going to store entity configuration for specific components (what hass.info is all about). So I think that we should split up this issue into two issues: where do we store entity config and how to deal with a device that measures/controls multiple things.

Attributes as entities topic can be continued in #11118

On the info topic:

The idea would be that info doesn't change and is used for debugging and configuring components. You would never have to fetch it from the API. If we store it to the state object, we are conflating concerns of configuration and state. We would have to create a dict on each state change with the current info, this is expensive.

[NovapaX]

So if I'm correct the entity.info would be the place to store things like:

• Device information (brand, model, firmware version)
• Friendly id, display name
• Device properties (supported features, min-max values for things like color, temperature, speed and whatever, device specific overrides for platforms?)
• UI/display settings (show or hide the history chart in the frontend, what tab was last opened)
• Component config (recorder/history, homebridge, emulated hue)
  Am I right here?
  (I'm noticing a lot of overlap with the current customization.yaml, maybe we should take customizations.yaml and merge that into entity.info)

But I guess quite often the entity.info will be needed when acting on the entity.state

• When I invoke the more-info-panel of an entity I would need to fetch both the state and the info
• When sending notifications based on a state-change I would want to use the friendly_id or display name.

I haven't though about the cost of generating that info Dict with every state change. But I still prefer all the info contained within the state object.
Maybe the state object should contain two parts, the info (static) and the attributes (dynamic) part. We only update the dynamic part regularly, and only set the static part if not yet filled. (clearing the static info would mean it gets recalculated by the platform on the next state change.)
This way when invoking the state object from the API you get everything you could possibly need. What gets sent over the event-bus is food for another discussion.

[balloob]

@NovapaX No, info is only for things that are component configuration (like should it sync with Google Assistant) or debug info (firmware, manufacturer).

All other things like min/max values, supported features are part of the state, as they can change with the state of the device. The frontend will always be able to render the full entity from a state without needing any lookup in hass.info.

[balloob]

@armills I've moved your comment to #11118

[emlove]

@balloob I don't think it's the wrong issue. That would eliminate the need to even add hass.info.

[balloob]

Oh, I guess you are right there, oops :no_mouth: .

I think in that case I still wouldn't want to store component entity configuration in the state machine.

[emlove]

Hah, no problem. Appreciate the effort to at least try and wrangle this conversation!

Definitely agree on the entity configuration, but I think if that's the only problem left we can come up with a better solution for that on its own.

If possible I'd really like to not introduce more concepts to core hass. This feels like it knocks out a lot of the problems, while still living within our current infrastructure.

[pvizeli]

With last statement what it should stored inside hass.info (from @balloob) I think we don't need merge it to state like entity.info.

For custom UI we need support a interface to fetch this data over the API.

I'm also fine with that but I'm feel also like @armills. It solve this problem but I'm not 100% sure if we have not a deeper concept problem.

Anyway maybe we can use hass.data['info'][entity_id] without make a new store and can use a helpers to handle it. So we can call this function with hass.helpers.info.yx

[rytilahti]

My opinion wrt. firmware versions and manufacturers is that entity attributes/properties (such as firmware version, manufacturer, etc.) should belong to that entity and not some external place. If the issue is polluting the recorder, maybe we could think a way to avoid exposing those attributes to the recorder? Storing homeassistant's internal entity configuration could very well go outside from the entities themselves, but wouldn't it be more clear if that would be a part of the entity (as it logically feels to be), but which also should be ignored for state tracking purposes?

Regarding to configuration elements of entities themselves, I think those should also "belong" to an entity. In the future I would like to see a way for the platform developer to expose platform's extra services in a nice way for both API and REST API users. These should not definitely be part of the state, but should be accessible through the entity interface.

As it stands, adding those services into a global namespace breaks the encapsulation both on the code (separating descriptions out from the platform) and on the logical level (the available, platform-specific services have just prefixed names with no real way finding more about them). Considering that I would be an UI developer, providing our users a way to access some extra functionality like setting a custom fan speed or reseting the "exchange filter" counters after a change, to achieve this for all possible devices is just waste of time.

However, considering that these types of actions would very well be useful for our users, letting the developer of the platform to expose services (again defined inside the "main platform"/"entity") with some meta-data (type: "range? slider?", min_value: 0, max_value: 100) would enable UI developers to add such simple services e.g. into a separate "extra actions" tab. This (and introspectable interfaces in general) would also be beneficial for the REST API users. If one wants to access services offered by some platform it is currently very cumbersome to do so in appdaemon.

Now that I have written that out, I feel I need to apologize for taking it towards something which may be worth of another RFC, but I think this shows that there is an inherent need to have some type of non-state related attributes not in a globally shared container but inside the entities themselves.

[pvizeli]

After some discussion with @balloob I write follow statment:

I only vote for this if we try to solve real problems and make not only a useless core change. Yes 1-5% usercase for a new stuff is not realy height.

That means:
1) customize should not able to write new attributes
2) hasd.info data need to be fetch able over API for external system like UI, hasska

Only if the new stuff support this, we have a nice feature they will bring realy usable cases.

[BioSehnsucht]

I don't really know core well, but have done some component work / am working on a component/platform for a new integration.

Is there a reason for hass.info versus adding an info Dict property to the entities (i.e., from a code perspective on the entity, same as attributes, just different name / purpose)? Or is this the same idea (I've assumed that accessing hass.states...some_entity.attributes is accessing some_entity.attributes directly, but maybe it's not?)

[balloob]

The idea would be that info doesn't live on the state object, as it is not part of the state of a device. So there is a difference between hass.states.get(entity_id) and hass.info.get(entity_id).

Something that I have been thinking about that is somewhat related to this (as it impacts attributes), is the Web Thing API. It's a draft standard API for the internet of things on the web. (It doesn't cover all the things I would expect from an API but it's a draft so we can have an impact) We should probably move talking about the Web Things API to a new topic if people are interested. (leave a 👍 on this comment to indicate interest)

[balloob]

Update:

I have removed configuration from customize from both Alexa and Google Assistant. Emulated Hue is still using customize.

Based on all the discussion here, I thought of a related idea: the entity registry. More info here: https://github.com/home-assistant/home-assistant/issues/11533

[andrey-git]

@balloob
How about creating a "component" for storing "customization" that won't go on the state machine?
It

This is for the sake of custom UI components or alternate clients altogether: Right now it requires each author to write and each user to install a custom component.

[balloob]

I prefer if we can keep it out of the state machine because it causes side effects, like people complaining the recorder is bad.

What if we have a special component in which you can store whatever data structure you want and we add special code to the frontend to load it once (inside <home-assistant>#connectionChanged via an API call if it sees that the specific component is loaded?

[tboyce021]

Somewhat related to this discussion but not sure if I should create a new issue or not:

Some components (I'm specifically looking at template_sensor) have config options to set various UI-related attributes. With all this talk of separating the concerns of frontend and backend, it seems like these should not be config options. In fact, a lot of them can already be achieved in other ways, specifically

• friendly_name can be set via customize. A name config option is really only needed if it affects the entity_id, not just the display name.
• template_icon can be done via custom UI. The main difference I see is that template_icon is done server side and with custom UI it's (I think) done client side (correct me if I'm wrong here).
• entity_picture_template I assume can also be done with custom UI.

Since these are all fairly trivial to accomplish using the alternative methods (custom UI stuff being trivial with @andrey-git's repo) , would it be good to start removing these config options so people can stop using them and switch to the alternative methods?

[c727]

I like @andrey-git 's idea about creating such a "customization" "component". I would name it customizer but this name is already blocked :laughing:

Loading the data clientside on <home-assistant>#connectionChanged also sounds great. <-- That's something I meant when I aksed where to store the data client side for custom cards for different cards (full/state/more-info)

My PR for the custom_card (here) component has some limitations as it allows only storing config for entities created by the component. I can modify it to do both, allow the creation of custom_cards and provide config for any entity...

Or are you already working on a (better) solution, Andrey?

maybe it should be part of frontend?

frontend:
  custom_config:
    "light.light1":
      key1: val1
  custom_card:
    "card":
      "full_card": test-card
      "state_card": xxx

[balloob]

I think that adding it to the frontend component is not a bad idea. And then in the Polymer part, when stuff is loaded, see if the data was set and if so, fetch it with API call.

[balloobbot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

[balloobbot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

[balloob]

I think that we can close this issue:

• With device registry, we want things like battery level to be their own entity.
• UI customizations can be added directly to Lovelace config, also for custom cards