Openstreetmap-website: GPX files not following standard

[bjoeni]

Followup of #2046

When downloading public GPX traces through the API, all waypoints of non-trackable traces are randomized (or rather sorted by lat/lon) and delivered as one trackSegment for privacy reasons.

That however is violating the GPX standard that clearly states:

A Track Segment holds a list of Track Points which are logically connected in order. To represent a single GPS track where GPS reception was lost, or the GPS receiver was turned off, start a new Track Segment for each continuous span of track data.

The API should return a valid GPX file, i.e. by using a custom extension, so that unordered waypoints are never in a trackSegment. That would also have avoided JOSM ticket #16963 (when JOSM tried to display all unordered waypoints as a continuous track), though I see how that situation required an immediate fix for privacy reasons, but it should imo not be a permanent solution.

Also related: JOSM ticket #17569 (JOSM now fails to display lines for local GPX tracks without timestamps because of the hack for the non-standard GPX tracks from OSM)

[tomhughes]

So your solution to a technically invalid but mostly working response to return a totally invalid response?

I don't think we will be doing that...

I see two solutions - either we change the MIME type and/or XML id to make clear that this is a custom OSM format and not a GPX file or we put each point in it's own segment.

[bjoeni]

So your solution [...] to return a totally invalid response?

Where exactly did I say that? In this ticket I just stated the problem, not a solution (yet).

In the JOSM ticket I suggested either the one-point-per-segment solution (but I have some concerns that the response would get quite huge) or an own extension that would allow waypoints to be unordered. So we'd still have the waypoint type and could use it inside some type that replaces "trackSegment" - therefore perfectly valid and unordered.

Something like this (with an appropriate XSD file linked)

<gpx>
  <!-- metadatastuff -->
  <trk>
    <trkseg>
      <!-- all the trackable traces / waypoints as is -->
    </trkseg>
  </trk>
  <extensions>
    <osm:points unordered="true">
      <!-- all non-trackable waypoints -->
    </osm:points>
  </extensions>
</gpx>

[tomhughes]

You suggested returning the points without an enclosing segment I thought? Which as I understand it would not be valid.

[bjoeni]

Well without an enclosing element of the type trkSegType, but in an enclosing element defined by an extension that specifically allows this. See the example above.

[bjoeni]

Also, that way software that doesn't support unordered points would simply ignore the extension and only display the ordered tracks.

[tomhughes]

Right, but the main problem is that every existing user of that API will be instantly broken because they will have no idea about that extension.

[bjoeni]

I agree. That's also what I wrote in the JOSM ticket

And the reason I wrote that I don't see this happening is because it would probably break all other software relying on the API - GPX extensions are not there to override the standard but to extend it. So the points couldn't be in a <trkseg> anymore. But I'll open a ticket there and we'll see.

I don't know what software relies on the API. My guess is that all software was essentially broken the moment trackpoints weren't ordered anymore and probably chose to ignore the unordered ones (like JOSM). So it probably would only break the editors again, other than that I don't know what software could actually use the unordered trackpoints.

But that's the point where I'm not involved enough, I stated the possibilities from my point of view - I don't really know about the consequences.

[simonpoole]

The simple solution https://wiki.openstreetmap.org/w/index.php?title=API_v0.6&type=revision&diff=1842418&oldid=1725374

[bjoeni]

Maybe it's just me, but I think saying that you're violating it doesn't exactly make it any better. Standards are there for a reason and imo especially important for open source projects.

My preferred solution would still be extracting it to another element defined by an extension, and as said above I think it would only break the editors again. Any software designed for the original sorted waypoints would probably anyways be better off without the affected waypoints, rather than with randomized ones that it's expecting to be in order.

Alternative suggestion:
If that's not an option at all, what about creating an extension for the trkType or trkSegType so that the user can at least distinguish between untagged tracks and the list of untrackable waypoints? Because that's what caused the latest JOSM bug mentioned in the first post.

<gpx>
  <!-- metadatastuff -->
  <trk>
    <trkseg>
      <!-- all the trackable traces / waypoints as is -->
    </trkseg>
    <trkseg>
      <!-- all non-trackable waypoints -->
      <extensions>
        <osm:trackable>false</osm:trackable>
        <!-- or ordered/sorted/whatever -->
      </extensions>
    </trkseg>
  </trk>
</gpx>

Would still violate the standard (and make it impossible to use the GPX file outside of a specifically designed editor, but not break current users) - so better than nothing I guess.