Cartodb: Histograms change their data range when filtered

[ivanmalagon]

Context

When there is a map with several widgets, after applying a filter in one of them, the data returned by the other widgets adapts to the filter. It changes its data range.

We spotted this and opened the issue https://github.com/CartoDB/cartodb/issues/12480 to tackle corner cases with the filters.

After talking with @javisantana , this is not the behaviour that we had since the beggining. The data range should be constant. Original comment here.

This should be addressed as a hotfix.

[ivanmalagon]

Found the original issue that provoked this.

Comment: https://github.com/CartoDB/cartodb/pull/12275#issuecomment-308140315
PR: https://github.com/CartoDB/cartodb.js/pull/1672

[AbelVM]

:thinking: I'm not sure whether this behavior was wrong at first. If you don't change the data range and, let's say, the original range was 0-1M, but once filtered is 1000-1001. If you keep the original data range of the histogram, all the detail is lost. But with a data range from 1000 to 1001, you have full details about your data there (that you won't be able to get with 0-1M range, not even zooming in)

[ivanmalagon]

Bad news.

Thought that it was fixed with the current state of this PR but it's not the case. I'll ellaborate (long explanation ahead).

I'll refer to the change that made the data range to be modified as the change from now on.

We can't revert the change in a simple way because of two reasons that were not present when the change was applied:

• Now we can aggregate by date.
• Now time-series widget has the option show totals.

Bear in mind that we're talking here of histograms and time-series widgets, served by the API from a single endpoint.

When totals were requested before

Before the change every dataview requested its total data before any filter was applied, so the response from the API had all the data range.

From the response, we calculated the total data range (start and end). These values are used to keep the total range when requesting filtered data (by bounding box or filters).

We never requested anymore the total data. We didn't request total data for time-series. So that data range was valid for every request.

Because of this behaviour, there was a bug in Builder when changing a column of a histogram. We didn't request again the total data so the shown totals belonged to another column. This is the bug that was solved by the change and provoked the bug meant to be fixed by this PR.

When totals should be requested now

In histogram widgets

In Builder we should request total data for histograms when there is a change in:

• column: this instantiates the map and we have to request totals after the URL is changed.
• bins.

In time-series widgets

Bear in mind that the API sends the data depending on aggregation (days, weeks, months, ...) and a timezone offset.

We don't know beforehand the number of bins generated by the aggregation. We can ask for an automatic aggregation but even asking for a particular aggregation we don't know the number of bins that'll be returned.

So, when we should request totals in time-series?

In Builder we should request total data for time-series when there is a change in:

• column: this instantiates the map and we have to request totals after the URL is changed.
• aggregation: this instantiates the map and we have to request totals after the URL is changed.
• timezone.
• local timezone toggle.

In Embed we should request total data for time-series where there is a change in:

• local timezone toggle.

In both widgets

Apart from those changes, dataviews must respond to URL changes. Those change can be provoked by an own change (the ones described above) or because a filter was applied.

After applying a filter on any widget on the visualization, all widget URLs change, so we must request the data for every widget. Moreover, we don't know who triggered the map instantiation. As a dataview, we only know that our URL was changed.

So the fix is...

Saying all this, we notice that we have a big mess with what events trigger total data refresh, filtered data refresh, etc...

Now we have a scenario where we should request the total data in any point of the visualization lifetime, not only in the first load. And in any point, we can have an applied filter on any widget, so we must request total data specifying the original start and end. If not, we are in the same situation as the bug we pretend to solve, receiving a data range that is filtered.

So the easy fix is to request total data with a particular data range. But this won't work in time-series widget aggregated by date. Why? Because the data range doesn't depend on the data itself. Depends on the aggregation.

Example:

We have a colum with the min date being 21 NOV 2011 and the max date being 7 OCT 2015. The data range is:
days -> 21 NOV 2011 to 7 OCT 2015
weeks -> 21 NOV 2011 (monday) to 12 OCT 2015
months -> 1 NOV 2011 to 1 NOV 2015
quarters -> 1 OCT 2011 to 1 JAN 2016
year -> 1 JAN 2011 to 1 JAN 2016

and those timestamps depend also on the timezone!!

Summing it up

I propose to do this:

• Make more explicit in code when we have to request the total data: Now in master it's well done but you have to know the code. The request for total data it's done in all the above cases but we should be more explicit on why we request the total data. This will ease future developments but it's not urgent.
• Request total data with start and end: in histogram widgets it's simple because they won't change after the initial request. It'll be more complex in time-series case. It implies to calculate all possible values depending on aggregation and timezone to get a coherent start and end that won't change the data range.

Pinging @javisantana @rochoa @xavijam @noguerol to be aware of this.

[ivanmalagon]

@AbelVM

The thing is that this behaviour (changing data range) is not the original behaviour. The change was unexpected and introduced trying to fix another issue. The problem is that we haven't caught it before.

So there are two actions here:

• Back to the previous (and standard behaviour) as soon as possible.
• Think if we want to implement the other behaviour. This a product decission.

The problem with the data range changes appears when filters are applied. If we don't keep the same bucket divisions when applying filters, the existing filter limits don't make any sense anymore. There is a design ticket that explains and try to fix the problems. https://github.com/CartoDB/cartodb/issues/12480 . The bad news is that we discovered more corner cases while developing it. And with those corner cases we realised that the behaviour was not the standard.

For example, imagine:

• You have a filter in a histogram from values 10 to 20, that correspond to the second bucket on a total range of 0 - 100.
• Then you apply a filter in another histogram. This filter changes the data range of our first widget to 2 - 50. The bucket limits now are 2 - 6.8 - 11.6 - 16.4 - 21.2 ...
• Now, our filter from 10 - 20 falls in the middle of the new buckets. Visually is not easy to represent. Stats about buckets are meaningless. We don't know where falls exactly 10 or 20 because data is aggregated.

So the real problem with changing the data range is when we have several filters. And there's no an easy way to fix it.

[AbelVM]

I agree it's not a trivial problem, not from UX PoV, either functional of innards points of view. But, IMHO, we should have the discussion about the desired/expected behavior from the user perspective

[noguerol]

Then we go as planned, closing this and implementing https://github.com/CartoDB/design/issues/878

Thank you for the very detailed explanation, @ivanmalagon :*

[xavijam]

I totally agree with @AbelVM, and trust me, the design team has been working pretty hard finding a super nice solution but we should be pragmatic here. IMHO we should get back to the previous behaviour and then study&research&discuss if the other solution (out-of-bounds) makes sense and it is possible in the technical perspective.

[xavijam]

So, guys, after checking it. We have decided to take a step back and keep with the behaviour we had before doing last time-series changes. After that, we will revisit the out-of-bounds and automatic-range stuff. Thanks for all everybody 💋 , and sorry for the inconveniences.

[ivanmalagon]

The fix is ready for code review and acceptance.

The behaviour about data range has been reverted to the previous one: the data range is kept after filtering via widgets. The out of bounds filters is not a case that happens now.

But as I said in my comment above, we can't go to the previous behaviour 100% due to the existence now of aggregated time-series. We have after these last changes a new behaviour and two "bugs".

New behaviour.

• The totals now respond to the filters. When we had only totals in histograms, they didn't change after filtering. Now, since we're fetching them with an applied filter, the data returned by the API is affected by the filters. This is unavoidable since we need to fetch totals for histograms if we toggle the local timezone button.

"Bugs"

• Now we send start and end to fetch the totals. We have a calculation to get the most suitable values in UTC to send to the API. Sometimes when we switch between timezones, the API answers us with empty buckets at the end, both in total and in filtered data.

It's not critical and the widget works well. But it'd be good not to get empty buckets.

It doesn't cause problems with filter because we reset the filter when toggling timezone since the filter limits changes.

• The other one is quite related. The filter don't reflect the right data in the edges. Shown points are not all related to the histogram. This is mainly visible with low granularities (hours and days) and when have too many points in the edges of the buckets. It's unnoticeable in greater aggregations.

The filter we send from a time-series widget sends timezone translated data. I suspect that the API wants those values in UTC, but I've sent them with the offset adjusted and it doesn't work properly neither.

These two issues will be tackled in another issue, working with @dgaubert

API returns unnecessary empty buckets in histogram dataviews #732

Filtering doesn't reflect the timezone offset in aggregated time-series #12637

[xavijam]

In production!

[dgaubert]

@ivanmalagon I'm going to create an issue in Windshaft-cartodb to tackle empty bins when the timezone is changed.

Thanks for the write down!!

[dgaubert]

Ops! You already did it!

Thanks again!