Core: Filter platform not working after 0.97

[benquan]

Home Assistant release with the issue:

0.97

Last working Home Assistant release (if known):
0.96

Operating environment (Hass.io/Docker/Windows/etc.):

Hass.io

Component/platform:

https://www.home-assistant.io/components/filter/

Description of problem:
Filters are not working anymore.

and the raw sensor (unfiltered data) seems to be ok:

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

  - platform: filter
    name: "room_temperature"
    entity_id: sensor.RAW_room_temperature
    filters:
      - filter: outlier
        window_size: 4
        radius: 2.0

  - platform: filter
    name: "hall_temperature"
    entity_id: sensor.RAW_hall_temperature
    filters:
      - filter: outlier
        window_size: 4
        radius: 2.0

  - platform: filter
    name: "room_humidity"
    entity_id: sensor.RAW_room_humidity
    filters:
      - filter: outlier
        window_size: 4
        radius: 4.0

Traceback (if applicable):

Log Details (ERROR)
Wed Aug 07 2019 22:59:32 GMT-0600 (CST)

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 400, in _async_add_entity
    await entity.async_added_to_hass()
  File "/usr/src/homeassistant/homeassistant/components/filter/sensor.py", line 244, in async_added_to_hass
    history_list.extend([state for state in filter_history[self._entity]])
KeyError: 'sensor.raw_room_temperature'

Additional information:

[probot-home-assistant[bot]]

Hey there @dgomes, mind taking a look at this issue as its been labeled with a integration (filter) you are listed as a codeowner for? Thanks!

[aidbish]

try changing the RAW to lowercase in the entity_id

[benquan]

tried changing lower case and no change.

[GreenhouseAU]

Same problem here.

[dgomes]

filter hasn't changed in several versions, most likely related to the recorder component from which the filter sensor gets data from. Any changes on that ?

[benquan]

On client side, I didn't change anything. Just after the update, the filtered items disappeared. On the recorder, I have it setup as selective recording where I do not include the "unfiltered entities" like sensor.RAW_room_temperature, but this is how it was working just fine in 0.96. This is my recorder yaml:

recorder:
  purge_keep_days: 1
  purge_interval: 1
  include:
    domains:
      - device_tracker
      - switch
    entities:
      - binary_sensor.door_sensor
      - sensor.room_humidity
      - sensor.room_temperature
      - sensor.hall_temperature
      - sensor.hem_power_w
      - climate.room_ac
      - media_player.ben_cast
      - media_player.sala_cast
      - sensor.cpu_temperature
      - sensor.processor_use
      - sensor.memory_use
      - sensor.disk_use
      - sensor.dark_sky_precip_intensity

[dgomes]

well you just diagnosed the problem, since the raw values are not recorded the filter sensor will fail to load previous values :(

I could create a "fix" for the error, but the filter sensor will misbehave since it will always slow start

[ve6rah]

That explanation doesn't make sense. If it was due to not recording the raw values, why did it work in the last version?

I'm in the same boat, no changes on my end between versions, but it worked beforee and doesn't now.

[benquan]

How come it worked before?

[benquan]

If it could work without recorder data would be idea. But I do not know how deeply integrated it depends on it.

[DoctorOctagonapus]

Only thing I can find in the changelog regarding recording anythng is the bit about Apache Kafka. Probably unrelated. I too disabled recording state history on my raw sensors to keep my db size down. Worked fine in previous versions, broken since 0.97.x.

[tungmeister]

I'm experiencing the same issue. Raw sensors are still fine but all my filtered ones are dead. The raw values aren't excluded from my recorder so that can't be the issue.

[dgomes]

@DoctorOctagonapus and @tungmeister to you get the same error message ? (KeyError)

[2xdehelft]

Same problem here with 0.97.1 and 0.97.2...

[isurkov]

Issue exists for me also in 0.97.1

[dgomes]

There is already a Fix waiting to be included ASAP

[ve6rah]

Do we know what version this fix will be in?

[dgomes]

It should already be in the beta ;)

[ve6rah]

Now on 0.98.1 and I no longer see the errors, however I have one filter that's still just not working, always showing unknown when the underlying raw sensor is working properly. Any suggestions on how to troubleshoot to see why it's showing unknown?

- platform: filter
  name: "Basement Temperature"
  entity_id: sensor.raw_basement_temperature
  filters:
    - filter: outlier
      window_size: 5
      radius: 5

- platform: mqtt
  state_topic: "hestia/furnaceroom"
  name: "Raw Basement Temperature"
  unit_of_measurement: "°C"
  value_template: '{{ (value_json.temp | round(1)) + 1 }}'
  device_class: temperature

The MQTT shows the right value, and the filter sensor used to work properly before all these changes, but no longer does, now it just shows "unknown"

EDIT: Not sure why the formatting is messed up in the "code" tags, but the indenting and such is correct, as I said, it did work until this change.

[dgomes]

can you activate debug ?
```
logger:
default: warning
logs:
homeassistant.components.filter: debug

[ve6rah]

I enabled the debug, and see debug for other filter entities, but not for this one. The ones I'm seeing debug entries for are for a DHT sensor that is working fine, but not for the mqtt sensor I show above. The only other difference I see for the one having an issue is it has a value template, not sure if that's relevant.

[dgomes]

you must see something like:

2019-09-01 23:52:59 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.raw_basement_temperature=32.0) -> 32.0

this log message happens right after filter gets a value from the raw sensor, if you don't see it, it means no value is reaching the filter. Or sensor.raw_basement_temperature state is UNKNOWN or UNAVAILABLE.

[ve6rah]

That's the point, I'm not seeing that. I see lots of those messages, but none for that particular entity.

How do I see why it's not reaching the filter?

[dgomes]

check in https://HASS/developer-tools/state for all information related to the raw sensor.

I'm also available on Discord for quicker assistance with the same handle

[ve6rah]

Well now I'm thoroughly confused. 2 days ago when I posted this, HomeAssistant had been up for several hours with the raw sensor showing a value, but the filter sensor showing unknown and the debug making no reference to it at all.

I was at work and couldn't check back until now, but now everything is showing properly. with the filter sensor now showing a value as well.

So I really have no idea what happened, but all's well that ends well?

[dgomes]

I'm closing this now ;)

[dornquast]

Thanks for fixing this - I just ran into it too. Glad it's already fixed! :)