Openstreetmap-website: way/:id/full/history

[emacsen]

Trying to look at the history of the geometry of a way is difficult. Doing so means that a client would need to call a way's history, then retrieve node history for each node.

This means a lot of requests to the server.

If the server could return a full history that included all the versions of all the way, along with all the history of the nodes ever in that way, this would make generating a geometry history much easier and less expensive to the server in terms of reducing the number of calls.

[tomhughes]

Yes it will reduce the number of calls. By replacing them with one very expensive call. I fail to see how this is a win when this is not something people should be using the API for anyway.

[emacsen]

People are asking for geometry history in Changemonger before it hits 1.0. This seems the simplest way to provide this data. The alternative is a set of calls, and that will cost a lot of wall and CPU time in serialization.

[tomhughes]

I'm sure people are asking for herds of ponies as well - it doesn't mean we're able to magically deliver them.

[emacsen]

Can you offer any helpful suggestions?

[tmcw]

@tomhughes Can you allow for discussion before closing issues singlehandedly? This is not a productive way to work on a project that is a multiple contributor effort.

[gravitystorm]

This is the problem that Potlatch 2 faces for its history dialog, and indeed it fires off a bunch of node history requests to achieve the goal. Potlatch 1 has/had its own api as part of the AMF controller to acheive the result so didn't require N calls, but instead relied on the server doing more work.

In general expensive calls in the core API are avoided if there is a different approach that can be scaled decentrally - generally routes involving minutely replication and onto second-tier services like JXAPI. These history-of-way calls could be fulfilled using JXAPI (or whatever service is appropriate) but perhaps it's instructive that p2 queries the main API directly.

So rather than adding expensive non-critical calls to the core API, is there a mechanism to use decentralised services? What would be the current barriers against using decentralised responses? For a history call I don't think replication delay would be a problem, but I can see service reliability, and the lack of full-history-storing services being the main ones.

[emacsen]

There seem to be two issues here- first being the need for this call, and second being a question of infrastructure.

As Andy point out, this call is makable via AMF today, and doing it today the "official way" is even more expensive, as it requires even more calls, and that means more serialization, and more wall time, and Andy presents a second use case (one that is already in use).

Now to the question of "editing API" vs "history API"- for my purposes, an up to date history server is fine- since Changemonger by default already makes individual object requests off an XAPI server in order to reduce the number of calls to the main API server.

But there is no "history server", and thus no "history server api". If there were a history server, I'd agree that all history calls, including this one, should go to the history server, but the website provides /history calls, so this is just an extension of the existing calls.

[zerebubuth]

The queries necessary to generate this output are the same regardless of whether it's done by a sequence of one way/:id/history followed by many node/:id/history calls or a single way/:id/history/full call. The temptation would certainly be to do the node/:id/history calls in parallel, tying up a bunch of backends. To me, it seems preferable to have a single, if expensive, call - and one that we could potentially later optimise, if it becomes a significant drain.

My feeling is that this would be a useful call for editors to have (it is already for P1). For "analysis" type workloads it would be good to have it in a "history server" or jXAPI as well.

[mmd-osm]

Forthcoming Overpass API 0.7.55 has a new feature to return a way's full history (at least back to Sep 2012), see: https://github.com/drolbr/Overpass-API/issues/282#issuecomment-315516184

Example: http://overpass-turbo.eu/s/uTE

[tyrasd]

Forthcoming Overpass API 0.7.55 has a new feature to return a way's full history

as far as I can see, this still doesn't perfectly fulfill what's been requested in this thread:

[…] return a full history that included all the versions of all the way, along with all the history of the nodes ever in that way […]

While this new Overpass-API functionality "only" returns _all the versions of the way, along with resolved nodes of this way at the respective way-version-timestamps_.

[mmd-osm]

@tyrasd : yes, that's correct. If you just move around some nodes, the way won't get a new version, and consequently that change is not reflected in the query result.

I will check with Roland if timeline can also be used more generically by providing a list of nodes in an inputset, rather than via an explicit object id as in my example. Then you could use a nested timeline / foreach / retro query.

[mmd-osm]

@Zverik has brought up this topic again in http://shtosm.ru/all/zhelaem-togo-chto-imeem/ and https://2019.stateofthemap.org/sessions/V7NUWP/

This could be done by a single SQL statement (see https://gist.github.com/mmd-osm/fff7f5620def9affd5477da6aeb419b9 for how this could look like for ways). I don't know how expensive this is in production, and if we should restrict it to a range of version numbers rather than evaluating the full history.

The example query returns a flat list of node id/version numbers, which would be used as a basis to serve the respective node/way versions back to the client. This way, the amount of data to be transferred would be kept to an absolute minimum. It would be the responsibility of the client to reassemble the full history based on those node and way versions. This really shouldn't be an issue after all.

A bit of an explanation on how the query works:

1. Calculate timestamp_from and timestamp_to for each way version
2. Break result from step 1 down to the way_nodes. In case the way version is invisible (and hence no way_nodes are available), we resort to the previous way version's way nodes. This is important in order to make sure deleted node versions are also included in the result.
3. Calculate timestamp_from and timestamp_to for every node that was ever used as a way node
4. Intersect node time ranges (step 3) with way_node time ranges (step 2).

• In case no subsequent version number is available for an object, timstamp_to will be set to a large value in the future (01.01.3000).
• Intervals are defined as: timestamp_from is always inclusive, timestamp_to exclusive:
  [timestamp_from, timestamp_to(

Edit: This logic seems a bit too complex, returning the full node history of every node that has ever been part of a way should be sufficient. It would include node versions where the node wasn't (yet/anymore) part of a way, but an API consumer should be able to sort this out.

[mmd-osm]

Adding to https://github.com/openstreetmap/openstreetmap-website/pull/1448: if we restrict way/:id/full/history to the previous and current object version by introducing some additional filter criteria, we would automatically get all the relevant details to render a representation of a deleted object at the time it was deleted.

As there's no requirement to return every possible way and node version in this case, applying an object version range restriction would save some bandwidth.

[mmd-osm]

I put this up on the dev instance to get some early feedback. The endpoint URL is different from what has been proposed here, so we don't have to change any of the Apache rewrite rules.

A few test cases:

• https://upload.apis.dev.openstreetmap.org/api/0.6/way/180401/history?nodes
• https://upload.apis.dev.openstreetmap.org/api/0.6/way/272908/history?nodes
• https://upload.apis.dev.openstreetmap.org/api/0.6/way/272970/history?nodes

If you like you can create your own account on https://upload.apis.dev.openstreetmap.org and try for yourself.