Cartodb: Support OSM Overpass API urls for URL imports

[iriberri]

As an example, this query to the Overpass API returns a response with:

Content-Type:application/osm3s+xml

but we're not taking into account this source/format here. The osm and osm2 url translators only seem to accept the OSM API directly.

[saleiva]

Hey @rafatower could you evaluate how easy would be to do this?

[rafatower]

My guesstimate from my ignorance: 2 days or so if it is just adjusting params and testing. 1+ week if there's a need to implement another (sub)format. Osm3 points to a new format for the content-type, but I might be wrong.

@Kartones what do you think?

[Kartones]

no idea of this OSM3 format... ignorance++ my only experience with OSM format was fixing the url-based downloading, so without checking what the overpass API provides, how we import OSM etc I can't help much :/

[mmd-osm]

I originally raised this issue in a GIS stackexchange comment. The query example given above is not ideal, as it returns a JSON-based format. By removing [out:json] or replacing it by [out:xml] you will get the well known OSM XML format.

IMHO, it would probably be sufficient to just support this OSM XML format (it is already supported today, albeit the Content-Type application/osm3s+xml causes some issues) and simply ignore JSON.

Here's an updated example link - as the one mentioned in my comment on GIS stackexchange is currently not working.

[atlefren]

Given that you already have support for GeoJSON, could you consider using something like this: http://tyrasd.github.io/osmtogeojson/ ?

[pnorman]

The format is one we support, and if you save the overpass output to a .osm file it uploads fine, but this prevents use with a sync table

An example URL is https://overpass-api.de/api/interpreter?data=%2F*%0AThis%20has%20been%20generated%20by%20the%20overpass-turbo%20wizard.%0AThe%20original%20search%20was%3A%0A%E2%80%9Chighway%3D*%20and%20type%3Away%E2%80%9D%0A*%2F%0A%5Bout%3Axml%5D%5Btimeout%3A25%5D%3B%0A%2F%2F%20gather%20results%0A%28%0A%20%20%2F%2F%20query%20part%20for%3A%20%E2%80%9Chighway%3D*%E2%80%9D%0A%20%20way%5B%22highway%22%5D%2841.65620060264075%2C-4.723037481307983%2C41.657322834579915%2C-4.720258712768555%29%3B%0A%29%3B%0A%2F%2F%20print%20results%0Aout%20body%3B%0A%3E%3B%0Aout%20skel%20qt%3B

[jsanz]

Since uploading manually the file is working fine because we support OSM XML file format, this looks like just setting the content type check for Content-Type:application/osm3s+xml as @mmd-osm pointed.

[jsanz]

Hey @rafatower any chance you can revisit this? :smile:

[iriberri]

@jsanz I did a quick test adding the new content type and it's working with URLs like this one, which return XML. The URL I put one year ago in the issue description is returning JSON, so it won't work. I don't remember how I got it for JSON specifically, but now the Overpass Turbo API gives me always XML in the "get data directly from API" option, so 👌.

[mmd-osm]

I don't remember how I got it for JSON specifically

If you put [out:json] in your query, you will get JSON, otherwise XML will be default.

[iriberri]

Thanks @mmd-osm! :)

[mmd-osm]

Small nitpick: correct wording would be "Overpass API". There's no such thing as "Overpass Turbo API". overpass turbo is the name of the web frontend, consuming data from Overpass API.

[iriberri]

I just renamed the PR, thank you very much again!

[jsanz]

Awesome Carla, thanks!!

On Mon, Sep 26, 2016 at 12:21 PM, Carla [email protected] wrote:

I just renamed the PR, thank you very much again!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/CartoDB/cartodb/issues/5042#issuecomment-249533309,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AALfaJS0cj-2a-N0HP10EcCAArT_FICIks5qt5y2gaJpZM4Fsmgl
.

Jorge Sanz
Solutions Engineer
CARTO - Predict through location