Core: Derivative sensor doesn't update correctly for non-changing values

[Spirituss]

The problem

I use derivative sensor to measure the water flow through water counters. When the flow is changing, derivative shows realistic values. But when the flow becomes zero, derivative is still showing the last measured value for a very long period (about some hours).
In the meanwhile, 'change' attribute of the statistics sensor becomes zero with zero flow after the last HA update 0.105 (the values keeping logic was changed).
Below is the measurements of derivative and statistics sensors, and the historical values of the water meter itself as a prove.

Environment

Home Assistant 0.105.1 (ex. Hass.io)
arch: x86_64

• Home Assistant release with the issue: 0.105
• Last working Home Assistant release (if known): new feature
• Operating environment (Hass.io/Docker/Windows/etc.): Hass.io/Docker/Synology DSM
• Integration causing this issue: Derivative
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/derivative/

Problem-relevant configuration.yaml

sensor: 
  - name: raw_water_flow # problem-relevant sensor
    platform: derivative
    source: sensor.raw_water # water meter sensor, updates every 15 sec
    round: 2
    unit_time: min

  - name: raw_water_stat # Seems to correctly working sensor
    platform: statistics
    entity_id: sensor.raw_water # water meter sensor, updates every 15 sec
    max_age: '00:00:30' 
    sampling_size: 12
  - platform: template
    sensors: 
      raw_water_flow_stat:
        unit_of_measurement: l/min
        value_template: "{{ 2 * (state_attr('sensor.raw_water_stat', 'change') | float ) | round (2) }}"

Traceback/Error logs

No error, but incorrect behaviour.

Additional information

Here is the water meter values, which became zero at 01:18

The derivative sensor (sensor.raw_water_flow) was still showing non-zero (0.12 l/min) value after 01:18

The statistics sensor (sensor.raw_water_flow_stat) showed zero at 01:18

[Velly56]

+1

[probot-home-assistant[bot]]

Hey there @afaucogney, mind taking a look at this issue as its been labeled with a integration (derivative) you are listed as a codeowner for? Thanks!

[to4ko]

having the same on my server

[Spirituss]

Any news regarding the issue? The component is in production but doesn't work.

[punzenbergerpeter]

same problem here

[afaucogney]

Does your sensor update its value even if it doesn't change ? I mean do you have a real values table with several item of the same value ? or just a single that state constant ?

Could you please reproduce the issue and send the values table, I can add this to a test and see what's happen.

I need to understand if this is an issue in the "derivative component" algorithm or because of its integration.

BTW have you tried to add a "time_window"

Get more info :
Issue : https://github.com/home-assistant/core/issues/31395
MR that adds the time_window attribut : https://github.com/home-assistant/core/pull/31397

I someone has any idea, feel free to comment ! @basnijholt @dgomes

[dgomes]

statistics sensor runs periodically regardless if there are any changes in the sensor. This also means that it doesn't track changes during the period.

derivate sensor (and integration sensor in which it is based) tracks changes in the source sensor. That means that if the source sensor doesn't change, the derivate sensor will keep it's value for long periods of time.

One possible solution is to combine both methods: track changes and periodically read the source sensor to detect "no changes".

[Spirituss]

derivate sensor (and integration sensor in which it is based) tracks changes in the source sensor. That means that if the source sensor doesn't change, the derivate sensor will keep it's value for long periods of time.

It DOES NOT work thus, as you can see on me initial screenshots. In case, when water meter keeps its value (on screenshot - 7, February, 1:18 AM and later on) what means that the first order derivative is zero, while HA continues to show 0.12 l/min.
When you tell that "derivate sensor will keep it's value" you possibly mean that source sensor keeps its value, but not the derivative sensor. Otherwise, it does work not as derivative.

[afaucogney]

@Spirituss Are you sur that "HA continue to show 0.12", or maybe this is the chart which draw a line between 2 points. This is the reason why I ask about value table ?

[afaucogney]

Otherwise, it does work not as derivative.

There are plenty way of implementing derivative when it come to digital. Unfortunately.

[Spirituss]

@Spirituss Are you sur that "HA continue to show 0.12", or maybe this is the chart which draw a line between 2 points. This is the reason why I ask about value table ?

How can I get if from HA? I physically switched off source sensor for my water flow derivative but HA still show 0.16 l/min in states list. No matter what does chart show.

[Spirituss]

Otherwise, it does work not as derivative.

There are plenty way of implementing derivative when it come to digital. Unfortunately.

It does not explain the issue. Digital calculation of derivative can cause its inaccurate calculation, but in case when there is no change it must show zero.

[afaucogney]

@Spirituss Are you sur that "HA continue to show 0.12", or maybe this is the chart which draw a line between 2 points. This is the reason why I ask about value table ?

How can I get if from HA? I physically switched off source sensor for my water flow derivative but HA still show 0.16 l/min in states list. No matter what does chart show.

This is the point, if you switch off the sensor, the value is not updated, and its value keep the same value, but its timestamp is not updated. So everything is normal from my side.

Did you try the time_window ? I'm sur this is what you are looking for !

[afaucogney]

Otherwise, it does work not as derivative.

There are plenty way of implementing derivative when it come to digital. Unfortunately.

It does not explain the issue. Digital calculation of derivative can cause its inaccurate calculation, but in case when there is no change it must show zero.

When you say 'there is no change' : What is the diff between "no change" and "waiting for the next value". How can the component know if before getting the new value:

• the value keep the same
• the value become 0
• the value became -infinity
• the value became +infinity

In your context it is maybe obvious, but I designed to provide derivate values indexed on sensor values, nothing else ! because my sensor do not have any update frequency (or I do not want to care about)
@basnijholt added the time_widow to discard a part of the issues caused by sampled sensor.

If your case doesn't work with time_widow, feel free to open a PR, we can look on that.

[Spirituss]

Did you try the time_window ? I'm sur this is what you are looking for !

Possibly, it is what I need. I read the manual but It's not clear how it works. What value should I use for the time_window? Is it a time delta that derivative uses for calculation of increment? In this case I think it's better to use the time interval that my sensor is being updated (15 sec).

When you say 'there is no change' : What is the diff between "no change" and "waiting for the next value". How can the component know if before getting the new value:

I'm not agree with you, since we are talking about physical conception 'derivative' what means the speed of value changing versus time line, no matter what is the reason of such changes, either "no change" or "waiting for the next value". This is the _nature_ of any derivative. But in case you start to matter regarding the reason of changes you mean statistics but not derivative. Home Assistant already has statistics sensor which does work exactly the way you tell about.

[Spirituss]

The irony is that after one of the issue made for statistics sensor its behaviour has been updated and now can work just as derivative, while the derivative component started to work as statistics.

[Spirituss]

If your case doesn't work with time_widow, feel free to open a PR, we can look on that.

I added time_window to my sensors and nothing has changed. Derivatives show the same value as before.

[Spirituss]

About a month there was no any news. Do you still support the component?

[afaucogney]

Hi @Spirituss, I still support the component, and of course PR are also welcomes.
IMO, there is no issue. You are looking for a perfect derivative mechanism in a sampled world, that's not possible. Every derivative of a sampled signal is an approximation, because the sampled signal is also an approximation.
Why don't you use stat component if it offers the expected behavior ? Because from you word, this is what you expect !

[afaucogney]

If your case doesn't work with time_widow, feel free to open a PR, we can look on that.

I added time_window to my sensors and nothing has changed. Derivatives show the same value as before.

Maybe you miss configure it, please post your configuration, and the output. Extract of the datatable would also be suitable.

[Spirituss]

Maybe you miss configure it, please post your configuration, and the output. Extract of the datatable would also be suitable.

Config:

sensor:
  - name: raw_water_drink_filter_kitchen_flow
    platform: derivative
    source: sensor.raw_water_drink_filter_kitchen
    round: 2
    unit_time: min
    time_window: "00:00:15"

The sensor sensor.raw_water shows nothing for long time:

But derivative sensor which physically means flow is still showing non-zero value:

It is definitely not the problem of the _approximation_, but the obvious mistake in the calculation algorithm realisation.

[basnijholt]

This happens because to calculate the derivate it uses the last known values and if new data comes in the older values are discarded (of time window). In your case your sensor didn’t admit any data for over a day so the derivative is based on that data from a day ago.

I am not sure whether we want to change this logic. If we did, the following happens, you have a time window of 15 seconds, and if data comes in every (let’s say) 20 seconds, the derivative would never be able to be calculated because you would have only one point.

[Spirituss]

This happens because to calculate the derivate it uses the last known values and if new data comes in the older values are discarded (of time window). In your case your sensor didn’t admit any data for over a day so the derivative is based on that data from a day ago.

What is the value of the time_window parameter in this case? In such way it looks ridiculous.

I am not sure whether we want to change this logic. If we did, the following happens, you have a time window of 15 seconds, and if data comes in every (let’s say) 20 seconds, the derivative would never be able to be calculated because you would have only one point.

This is the point! Of course, in case no data received during last 20 seconds with 15 sec update interval in terms of approximation definitely means that flow is zero!
Otherwise, it is just another realisation of old-known integration statistics sensor.

[divanikus]

@basnijholt @afaucogney I believe that the problem lies in the plotting. Since people usually use such sensors for plotting, it's highly desirable to see when change stops. It means that if the time window has exceeded and no new values are present, sensor should report 0, null, undef whatever, but not the last value. The statistics integration does this by issuing a timer for time window time and resetting sensor's value on it's exceeding.

[Spirituss]

I fully agree with @divanikus - the problem is that the function shows value at the current moment but the value is being calculated on the past sensor values. Ir order to automate the solution is the value out-of-date, the integration can use time window parameter.

[divanikus]

Seriosly? Without a solution?

[dgomes]

@afaucogney already posted :

IMO, there is no issue. You are looking for a perfect derivative mechanism in a sampled world, that's not possible. Every derivative of a sampled signal is an approximation, because the sampled signal is also an approximation.

[divanikus]

@dgomes I don't know how to explain even better, but max_age setting isn't working at all. That is this issue all about. It should reset sensor value after max_age of no new data, instead it just freezes on the last value, which is simply obsolete after max_age time.