Cartodb: AOI buffers disappear after running a second analysis

[ramiroaznar]

Steps

I have encountered a couple of issues while replicating the 2nd demo @cartodave made in #the-retraining:

• Create a map with points dataset (layer1).
• Run AOI analysis in layer1.

• Continue the workflow analysis in layer1. This time "Enrich from DO". I select United States as COUNTRY, Transportation as MEASUREMENT and Commuters by Subway or Elevated as SEGMENTS.

• The buffers disappear. In addition, the values from the DO are really low. According to @AbelVM it is because the values are normalized. But another explanation could be that the second analysis is using the point geometry instead of the buffer geometry to extract the DO data.

[saleiva]

as you see, the geometries are still polygons so this looks as a bug with the cache of layers and the style we apply.

[saleiva]

Does this happen always?

[ramiroaznar]

It always happens with "Enrich with DO" analysis.

[saleiva]

@xavijam feel free to assign to @viddo with priority tho...

[saleiva]

Happening to me as well when doing an "Intersect with second layer" analysis in a polygon layer with a previously analized points layer. As soon as you any property on the style form on the polygons layer, they get correctly rendered as polygons

cc/ @javisantana

[talos]

The really low column values were a result of numbers being calculated per sq m instead of per sq km. This should be resolved now.

[talos]

Also want to flag https://github.com/CartoDB/dataservices-api/pull/237 here, as once that's merged then we'll get different numbers again for polygon measures -- the absolute number instead of a rate per sq km. This is the original intended behavior, but there were several layers of problems thus the back and forth.

[javisantana]

ok, several things here:

• hey, I can't reproduce with my datasets, could you send the .carto so I can check?
• I think DO analysis should have an option to normalize by pop at UI level cc @saleiva

[ramiroaznar]

@javisantana Sent it via slack.

[saleiva]

@javisantana the original design contains a denominator.
also AFAIK, for polygons those variables come up already normalized by sqkm

[talos]

@saleiva that's actually in error. The documentation says we should normalize by sq km for points, but have absolute values for polygons. This is a known issue documented here (https://github.com/CartoDB/dataservices-api/issues/235) that will be resolved by https://github.com/CartoDB/dataservices-api/pull/237 when deployed.

[talos]

And 👍 surfacing the normalize by denominator option!

[viddo]

Been debugging @ramiroaznar's map together with @javisantana , what we found was that when the maps API returns status "ready", and as such we do a lookup of the data to get the geometry type, the SQL API call returns no data even if there should be. If checking a moment later the data is there, so definitely a race condition. Seems like the maps API returns ready too early, so fixing that should fix this issue.

There's also a 2nd issue, which is that we should probably not set simple styles if a geometry is not found, so will change that separately.
_edit: 👉 #9146_

[talos]

Ah, a 🐞 inside of a 🐞 ... just had one of those last week. Glad to hear this should be resolvable so easily.

[javisantana]

cc @rochoa read the last @viddo comment. My guess is it only happens on SaaS so it's likely a race condition thing. It's easily reproducible with the steps provided and this carto https://cartoteam.slack.com/files/ramiro/F1V4VHYCB/subwaystations_map__on_2016-07-26_at_12.58.07_.carto

I'd try to reproduce with a basecode before this pr (https://github.com/CartoDB/cartodb/pull/9146) which kind of workaround the problem

[rochoa]

I see queries to the SQL API to retrieve the geometry type before the node is set to ready status. Furthermore, those tables are not being cached, if you check the response headers you see X-Cache:MISS. I see the client is sending a cache_buster param to avoid being cached, but that should not be needed since https://github.com/CartoDB/CartoDB-SQL-API/pull/330.

Does the client cache the geometry type at any time?

[viddo]

I see the client is sending a cache_buster param to avoid being cached, but that should not be needed since CartoDB/CartoDB-SQL-API#330.

The cache bust was added as a temporary solution (#8749), I've created a separate issue to have it 🔥 then 👉 #9217

Does the client cache the geometry type at any time?

No, not yet, it's planned to be done soon enough though 👉 #8827

[viddo]

Debugged with @rochoa in-person:

I see queries to the SQL API to retrieve the geometry type before the node is set to ready status.

This was a false lead, separate request done by the table-view, it has no affect on how the layer is rendered.

Anyway, @javisantana @ramiroaznar we can't reproduce the issue anymore, and unfortunately we're not sure why. 😕 For now we'll close it, feel free to re-open if this happens again (and hopefully have some more data/STR to go on).

Note that an exported .carto file that have undesired layer styles (as the one sent offline for this issue) will always render the things wrong, removing/adding a node will update the styles accordingly though.