Streetcomplete: Maps no longer updating on new vector tile server, out-of-date with quests

[matkoniecz]

Yes, #835 is back and #747 continues to cause problems.

Firstly, thanks to @Akasch for his great work in setting up a new vector tile server for rescue StreetComplete from the demise of Mapzen.

I am opening this issue to mention that as the server is not updating at the moment (and I'm sure it is probably not trivial to do so easily or cheaply), and therefore the map is no longer up-to-date with the OSM data (and therefore StreetComplete quests).

Is it possible to restart updates or run cyclic updates (once a month? once 3 months?) or switch to an alternative background map to StreetComplete that pulls direct from osm.org rendered tiles?

Currently it is not fatal problem in areas that are not significantly changed but it will steadily get worse if nothing is changed.

[matkoniecz]

I tested it by reinstalling app (what purges data) and downloading area where I changed well-visible data over 10 days ago ( https://www.openstreetmap.org/way/134071005#map=17/50.07755/19.93830 in that case ).

From what I know data was supposed to be updated continuously, what worked for some time and stopped.

I am not sure is it possible that data update will return in any form (even limited one like reload every 3 months).

[rugk]

Hmm, it all looks as if we maybe need a better powerful server there.
But @westnordost still rejects any donations. Or another method here. Or you can e.g. apply for the Thomas-Krenn-Award, which not only brings publicity, but they also donate hardware…

[westnordost]

A server is not a problem, terrestris runs one and the person maintaining it said it would be ok if SC used it. But the format is a little different, i.e. the stylesheet would need to be changed.

[westnordost]

Is this still an issue? I think the server just had a hickup, maybe?

[dbdean]

Absolutely, the maps in my area haven't fully updated in at least 4 months.

[westnordost]

Hm, damn. The admin of the server also told me that he is quite busy right now (with other things), so no chance one can expect this will be solved anytime soon

On May 28, 2018 12:06:31 AM GMT+02:00, David Dean [e-mail] wrote:

Absolutely, the maps in my area haven't fully updated in at least 4
months.

[matkoniecz]

Since I created this issue (and weeks before) I have seen no updates and plenty of cases of outdated map.

[rugk]

A server is not a problem, terrestris runs one and the person maintaining it said it would be ok if SC used it. But the format is a little different, i.e. the stylesheet would need to be changed.

Well…, but if you run your own server, you could get everything in the format you like.

[davidak]

And when multiple trusted members would have access to the server, problems can be solved faster, even if one don't have the time. Documentation is also important, in best case directly in configuration management. Minimizing the bus-factor...

[Akasch]

I have deployed a test, now Germany should be a more recent (I think 23.4. catching up, and the old tiles will be cached between 7 and 14 Days), all other Data is more old.

If this test is working it would be possible to split the Server to multiple small ones. As small Server are a lot cheaper for combined capacity, and it is easy possible to use unused capacity on private virtual servers this could be a model for a community driven Tile-server. I know that at the moment there will be a Problem with the tiles on the Border, I will fix it in the next days.

I think that we should put the Discussion about the Tileserver into its own repo/issue tracker. I would put a repo with configs etc up in the next day at the https://github.com/Map-Data Organisation.

[rugk]

Well, a "community-driven" sever would have privacy problems, so is rather now who runs the server I get tiles from.

[matkoniecz]

would have privacy problems

What would be worse than it is now? Location of given user is identifiable anyway by analysis of changes made.

[Akasch]

At the Moment only the fronted Cache Server is getting the IP address from the User, the Server Producing the Tiles is only seeing the IP of the fronted web Servers, and at the moment the User Agent which is always "okhttp/3.5.0" for StreetComplete. I think that it is not possible to correlate users to requests.

The Structure at the moment is like this:

Cacheserver01 -----               |------------------> Tileserver Germany
                   |-----> Master Server
Cacheserver02 -----               |------------------> Tileserver World

The Master Server is routing the requests to the correct Tilesserver, and is serving the prerendered Tiles. For a better availability the master server could be duplicated.

[matkoniecz]

In that case there is almost no reason to consider privacy as an issue.

[rugk]

Ah yes, if users could just host the backend servers yeah. This would be useful.

To answer the previous question: users may/do not always answer quests, but when they have the app open tiles are always downloaded.

[westnordost]

Tiles seem to not load anymore, @Akasch

[Akasch]

@westnordost deployed a workaround, should work again, will look into the causes later

[dbdean]

Any movement on getting worldwide up-to-date maps happening again?

[cstackpole]

I was just in an area that I haven't been to in a long time. The tiles were all downloaded. Yet I had house quests without outlines. I too request a up-to-date map. Thank you for everyone helping out with this!

[francois2metz]

I would be happy to host the France tile server (or more depending of the load). The proposition to route each traffic country to a specific instance is nice IMHO.

But I have no idea how to setup a tile server and update it. Anyone to give me a link to some doc?

[pkoby]

I've moved to a new area and also have no information on my maps, despite having added a lot of the buildings to OSM a few months back. This is not really workable, as others have said, without the context for which building you're updating.

I understand the issues with servers and whatnot, but perhaps we could get an option to use offline maps? I use c:geo for geocaching, and it has this option. You can download maps from http://download.mapsforge.org/maps/, then point the app to use them instead of, say, Google maps or Mapnik via wifi or data.

The size of the maps is not trivial, but I'd like the ability to update the basemap as necessary by downloading the mapsforge option much more frequently (about every month they update).

EDIT: alternatively, Maps.me has a download page (http://direct.mapswithme.com/regular/daily/), but I'm not sure what the license would be on that.

[matkoniecz]

offline maps are discussed in #122 (tl;dr: "This is not possible yet with the map rendering library I use").

[westnordost]

Actually, it would be when implemented manually. A nested nested for-loop that downloads all tiles from zoom level A to B within a bounding box C,D,E,F and stores these tiles in a local SQLite database. Tangram-es can then be pointed at that database for offline usage. However, that is not an effort I am prepared to make, because I do not want to invest time and effort into the dealing with the map library.

(Needless to say, this would put quite a strain on the tileserver. I have the impression that even now, the map-data tileserver we use does not cope with the traffic)

[rugk]

I have the impression that even now, the map-data tileserver we use does not cope with the traffic

Only @Akasch can say for sure.

[ENT8R]

Is there something which speaks against using Mapbox Vector Tiles? I know that one has to adapt the stylesheet, but that should be no big problem... But at least those tiles are always up-to-date...

[matkoniecz]

The MapBox tileserver is only allowed to be used through the MapBox SDK. I initially decided against MapBox because they have some telemetry built-in and their terms and conditions require to have this by default on (only letting the user switch it off in the settings).

https://github.com/westnordost/StreetComplete/issues/747#issuecomment-354842669

[matkoniecz]

And at this point it would require significant(?) app rewrite to make that switch.

[matkoniecz]

Also

However, I set for Tangram-ES instead of MapBox early in development because of this. I have no control over what and when the map renderer calls home, which is adverse to my goal to make this app very light on data usage and able to work offline.
I asked at MapBox a few years ago whether as an app developer, I can select the "opt out" to be the default, the answer was no.

https://github.com/westnordost/StreetComplete/issues/40#issuecomment-289603003

[ENT8R]

I just did some testing but with Tangram Play I could retrieve and render Mapbox Vector tiles... I can do some further testing but I think if it works with Tangram (JS) it should also work with Tangram-ES, shouldn't it?

[ENT8R]

I converted nearly everything now... Some things are still missing but you should get an overview; https://ent8r.github.io/streetcomplete-mapstyle/mapbox/
What I don't understand is that it is working because they say in their Terms of service:

The "Mapping APIs" means the Maps API, Styles API, Static API, Static (Classic) API and Tileset API. You must access and use map tiles, static map images, style files, glyphs, and sprites that we provide to you (collectively "Map Assets”) only through our APIs.

Is it illegal to provide such a website/map then?

[westnordost]

There is another vector tile server maintained by terrestris this project could use for free but I did not get into contact with them yet because it seemed that the current solution works ok.

What is the reason you think a switch is necessary now?

[dbdean]

Because our maps are six months out of date.

On Tue., 7 Aug. 2018, 20:50 Tobias Zwick, [e-mail] wrote:

There is another vector tile server maintained by terrestris this project
could use for free but I did not get into contact with them yet because it
seemed that the current solution works ok.

What is the reason you think a switch is necessary now?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/westnordost/StreetComplete/issues/1011#issuecomment-411016152,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMIsPi4atNyN-VnoYWGx3xsCGuezwFfAks5uOXDygaJpZM4TM2N9
.

>

http://dbdean.com

[cstackpole]

Because our maps are six months out of date.

I second this. I've temporarily stopped recommending StreetComplete to the new mappers because they kept having issues/questions about why the map was "wrong". Instead of them getting a bad impression (of StreetComplete, OSM, ect) or getting frustrated, I just asked them to wait. :-(

[matkoniecz]

What is the reason you think a switch is necessary now?

Because maps are not updating since at least April (and I created this issue because maps were not updating for a long time), and I see no easier way to change that.

[pkoby]

Maybe we could have a basemap selection option? So default could stay Tangram, but allow users to select another source (like Mapbox), making clear the caveats with each layer if necessary. Then maybe include an option for pointing the app to a folder on the device where you can store mapsforge maps like c:geo does? I admit, I don't know the coding issues involved, but I think providing choice might solve some of these issues.

[ENT8R]

Outdated maps but also bad user reviews because of map issues were the main reason for me...

Then maybe include an option for pointing the app to a folder on the device where you can store mapsforge

Don't think this will work, Mapsforge is a completely different rendering library...

There is another vector tile server maintained by terrestris

Do you know the url of the documentation? I couldn't find anything except an example map: https://ows.terrestris.de/osmvectortiles/index.html

EDIT: I also found this page: https://ows.terrestris.de/dienste.html#openstreetmap-wms but is this already the complete documentation?

[westnordost]

Alright, I didn't know the maps were that outdated. So let's put this into v7.

I will get into contact with the terrestris-person in either case. The documentation can be expected to be tiny/nonexistent because it is one of those services that I think are essentially only public/free because it is not advertised and thus not (mis)used commercially (that much), instead only in OSM context. Is the page you linked enough to go by or should I ask for more information?

Also, for those that complain that the map is not updated anymore, what about nextzen? Has it been updated since Dec 2017?

[matkoniecz]

From what I remember nextzen was supposed to be static and not updated (that is why I ignored it so far).

[rugk]

A static map??

[ENT8R]

Yes, Nextzen still uses the data from December 2017: https://tile.nextzen.org/preview.html

[exploide]

Maybe this all highly depends on the region?! Changes I made on July, 1st are visible on the currently used SC map. (And the tile.nextzen.org map contains changes I made 9 days ago.)

But several weeks / few months ago, I also noticed that everything is very outdated (missing buildings, etc).

[ENT8R]

Because maps are not updating since at least April

I checked it now on my device and the data is younger than 2 months (saw a fence I added on 22.06.) in Germany. Is the age of the map/tiles that different throughout the world?

[exploide]

Maybe because of https://github.com/westnordost/StreetComplete/issues/1011#issuecomment-394478833

I have deployed a test, now Germany should be a more recent (I think 23.4. catching up, and the old tiles will be cached between 7 and 14 Days), all other Data is more old.

[cstackpole]

I hope this is helpful/related information. It was on the OSM chat a while back.
https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/new-osm-vector-basemap/

Their contact is osm at esri.com .

Could this be used?

[westnordost]

I find esri embracing osm is as dubious as Microsoft embracing open source. Also, the mentioned tileserver is labelled as _"beta release, meaning it’s available for you to use and test in your maps and apps but you shouldn’t use the map or related items in a production application"_.

[westnordost]

I haven't heard anything from @weskamm yet, the admin of the OpenStreetMap WMS at terrestris, so trying out/migrating to another vector tile server will probably not make it into the upcoming v7 (due this Sunday).

[westnordost]

I did get answer from him now:

The WMS is at https://ows.terrestris.de/osm-vectortiles/osm:osm_world_vector@mapbox-vector-tiles@pbf/{z}/{x}/{-y}.pbf.

There is no documentation, but from the source code of his demo application, one can see what layers there are etc.
Regarding the attributes of map features (tunnel=yes, building height,...), he wrote that they are usually what Imposm3 maps in its standard configuration.

@ENT8R Are you interested in adapting the style to be compatible with this?

[ENT8R]

Yes, of course! I will look into it the following days (when the weather is bad again :smile:).

[westnordost]

Maybe I'd give it a try because I want this to be part of the next beta which I want to release on the weekend - unless you also intended to do it then.

[ENT8R]

I will try to do this today. Probably I will post the first result in the afternoon...

[westnordost]

Cool!

Am 24. August 2018 08:04:46 MESZ schrieb ENT8R [e-mail]:

I will try to do this today. Probably I will post the first result in
the afternoon...

[ENT8R]

So, the first version is finished and can be found at https://ent8r.github.io/streetcomplete-mapstyle/ if you click at the layer button in the top right corner and select Light Style (Terrestris). The source can be found at https://github.com/ENT8R/streetcomplete-mapstyle/tree/terrestris
During converting the style I found some issues regarding the functions of the tile server:

• the coast lines effect does not work (maybe we have to remove the effect 😕) (code)
• elements specified as stream, ditch, drain are not shown (code)
• the tiles contain no information about the height of a building so a 3D feature is basically useless and this is also the reason why I set extrude to false in the code
• barriers like walls and fences are not shown too even though the tiles should contain that information (code)
• it seems like the tiles contain no information about trams, so this can't be displayed too (code)
• steps are also not displayed at all even though they should be contained in the tiles (code)
• house numbers are also not shown and I could not figure out yet whether the tiles contain that information or there is an issue in the style...
• swimming pools are also not shown though this should be covered by the waterareas layer...

[westnordost]

Cool, thank you!

Hmm, a lot of problems, doesn't sound so good. How did you find out that certain information do not exist in the layers? Maybe they are just named differently?

Am 24. August 2018 15:45:54 MESZ schrieb ENT8R [e-mail]:

So, the first version is finished and can be found at
https://ent8r.github.io/streetcomplete-mapstyle/ if you click at the
layer button in the top right corner and select Light Style (Terrestris). The source can be found at
https://github.com/ENT8R/streetcomplete-mapstyle/tree/terrestris>
During converting the style I found some issues regarding the functions
of the tile server:>

• the coast lines effect does not work (maybe we have to remove the
  effect 😕)
  (code)>
• elements specified as stream, ditch, drain are not shown
  (code)>
• the tiles contain no information about the height of a building so a
  3D feature is basically useless and this is also the reason why I set
  extrude to false in the
  code>
• barriers like walls and fences are not shown too even though the
  tiles should contain that information
  (code)>
• it seems like the tiles contain no information about trams, so this
  can't be displayed too
  (code)>
• steps are also not displayed at all even though they should be
  contained in the tiles
  (code)>
• house numbers are also not shown and I could not figure out yet
  whether the tiles contain that information or there is an issue in the
  style...>
• swimming pools are also not shown though this should be covered by
  the waterareas layer...

[ENT8R]

How did you find out that certain information do not exist in the layers? Maybe they are just named differently?

I looked at the source code of the demo website, the default config of imposm3 and mainly what is included in the vector tiles (the source). For example I've never seen something which says e.g. housenumber in the source of the vector tile...

[westnordost]

But how do you see what is included in the vector tiles? MVT/pbf is binary after all.

[westnordost]

Some notes:

• trams, steps, swimming pools and aeroways also do not show on the terrestris demo map, looks like this information is missing. Except for steps, this is not so important though.
• that the building height information is approximated from building:levels and delivered in vector tiles is actually a feature of the mapzen vector tile server, IIRC. Without this, this approximation would need to be made client-side. Though, the tiles do neither seem to contain height nor building:levels. This is too bad, after all, half the fun with vector tiles are the 3D-features.
• regarding barriers, the source of the terrestris demo map does not even mention the layer osm_barrierways, perhaps it is missing altogether
• the same with housenumbers, , the source of the demo map does not mention osm_housenumbers, perhaps the layer is not available
• streams do show for me. For example here

New:

• the roads of different classification overlap pretty ugly. I think this is because it is a Mapzen tile server feature that it delivers a z_order with every element, thus defines a "default" sort order of the elements. When this is missing, the order needs to be defined client-side (in the style). So, this is something we can fix.
• there seems to be a problem with coastlines. See Manhattan (this looks quite bad), or a small part of Bremerhaven
• the server seems to have problems delivering vector tiles with practically 0 users on it. Just zoomed in randomly to a city and tiles on higher zoom levels just won't load (also in their demo map not). (Tangram is much less forgiving when tiles are missing - the area will just be left blank. OpenLayers instead shows the tile from a lower zoom level blurred)

[westnordost]

I also sent @Akasch and @weskamm an email explaining that I want to try out the tileserver in the next beta. Though, perhaps this is a bit on a too short notice and I should wait until mid of next week, especially in light of the problems with the integration we have.

Most importantly, I don't know if this is just a hiccup right now, but the last point would be a dealbreaker. It wouldn't be usable like this. (Perhaps it is the weekly import/cache flush?)

[ENT8R]

streams do show for me

true, streams are shown but ditches not. (but they are not that important anyway...)

the server seems to have problems delivering vector tiles with practically 0 users on it

I also thought that it takes sometimes pretty long to load the tiles... (right now I can only zoom in until zoom level 8 to see the USA...) all other requests are failing with 400: Problem communicating with GeoServer

But how do you see what is included in the vector tiles? MVT/pbf is binary after all.

I couldn't see every detail but I made assumptions 😄...

[westnordost]

Oh, okay :-D

So there is a z_order, then the mentioned problem seems to stem from somewhere else.

Edit: Anyway, no sense in putting any more time in it until this 400-error is resolved.

[iandees]

Hi all. An update on Nextzen: we're actively working on speeding up the time it takes to build a tileset of the world so that we can get to the point of weekly updates. We're currently doing this privately but I'll see if we can kick off a build to get the Nextzen tileset more up-to-date.

[rugk]

BTW another maps service/tile API: https://www.mapcat.com/

[westnordost]

@iandees that sounds too good to be true

A short update here:

1. We planned to move to the terrestris tileserver. We had some problems with changing the stylesheet to be compatible with it. Also, it turns out that the terrestris tileserver is not usable for about a whole day and night each week, so this server would only be only eligible as-is in an emergency, i.e. if our current server would become inaccessible on a very short notice. But thank you, @ENT8R, for your effort to create the stylesheet so far!

2. The reason why our current tileserver has old tiles in many places is because it stopped updating correctly due to the database being in a partly corrupted state. To solve this problem, @Akasch is coordinating a complete reimport of the map which takes a few days up to a week, during which the tileserver is offline. To fill the gap, tiles from @iandees' nextzen tile service are served in the meantime (which are old, from Dec '17)
   As soon as the complete reimport finished successfully, the map is all-new again and the (bi?-)weekly update of the map should work again without problem.
   Disclaimer: Of course, as the currently used tileserver is a bit of a shoestring-server and only set up as an intermediate solution anyway, there is no guarantee that the problems we had with the tileserver will not repeat itself or even that the service is cancelled on a relatively short notice.

[daviewales]

Is there any chance we could add satellite imagery as an alternative?

I'm trying to add house numbers, but sometimes feel that satellite imagery in StreetComplete would give me a bit more confidence that I'm adding the right number to the right house. (This would also work as an interim solution to the tile-server not being updated.)

[arbie65]

Thank you everyone for all the work you are doing to fix this issue. I am asking I see very out of date tiles is it safe to keep using streetcomplete or is it possible that using it with old tiles can cause conflicts on OSM?

[ENT8R]

is it safe to keep using streetcomplete or is it possible that using it with old tiles can cause conflicts on OSM?

It is still safe to use StreetComplete. The map tiles have no effect of the created quests. In case you want to have some more information: the quests are downloaded with the Overpass API (which is usually up-to-date with an approximately delay of 3-5 minutes...)

[arbie65]

Is there anything we as non technical users of streetcomplete can do to help get this ticket closed? It's a great tool but when features are in sync with OSM as it looks daily instead of a snapshot of months ago

[westnordost]

Not really. A more reliable and current map would require using a commercial vector tile service like Thunderforest, Omniscale, OpenMapTiles or what other providers there are. One could tentatively ask them if they would be willing to sponsor this app by providing the service for free to StreetComplete but this is probably something that I should do myself.

[mattwelke]

@westnordost Have you considered hosting a tile server in the cloud and reaching out to an organization like CloudFront to see if they'd sponsor free CDN service wrapped around your tile server to minimize load on it? DigitalOcean in particular often sponsors nonprofit organizations (https://www.digitalocean.com/community/questions/discounts-or-support-for-nonprofit-organisation), which could provide the origin server.

[westnordost]

No, because I do not want to concern myself with administrating a live tile server, rather I spend my time on developing the app.

[iandees]

FYI Nextzen just finished updating the data behind our tiles. I think the date on them is December 5th or so.

[westnordost]

Cool! So, how long did it take? Is this now a regular update or did you manually trigger it?

[iandees]

how long did it take?

It takes approximately a week to rebuild the tiles.

Is this now a regular update or did you manually trigger it?

We still have to manually trigger it. We're working on funding right now, but we might be able to get to quarterly or monthly update cycles.

[Susurrus]

What's the process for requesting or helping with updating these tiles again? I've been making substantial edits in my area, and the quests are updating, but the map tiles are pretty out of date at this point.

[Akasch]

At the moment unfortunately I have not so much time. But I now have a collection of scripts to setup a tileserver serving a limited region. Together with some server components it is now possible to distribute the tileserver.

I will cleanup the scripts in the next days. Maybe put some components into a doker container for easier deployment.

A server will need something like 50GB harddisk, 2-4 CPU cores, 4 GB RAM, I think. With CPU and RAM I am not so sure at the moment. Required software is Python-virtualenv (Python2), bash and postgresql with hstor and gis extensions.

The 50 GB is for the bigger regions, there are multible smaller ones, the computing recurces are only needed at import time and for very short times while generating some tiles. So it should be possible to have two or more regions on a server if the harddisk is big enough.

At the moment I am testing it with the region Hamburg in Germany.

If there is someone who want to try hosting a region, write me, and we can try out the progress.

[Julientroploin]

New user of StreetComplete, I think I experience the bug described here, so I can not contribute to OSM with your app... An image of what I get while zooming : https://drive.google.com/file/d/1rArOT_YmfNMIU23lAiQPcCV5iHcZtBUo/view?usp=sharing
Reading a part of the discussion (and not being an english native-speacker), I think I understood that your stuck with a server problem.

I can not help but I wanted to thank all of You for your efforts.

[Akasch]

@Julientroploin can you give me the location of the error, then I can try to debug the error.

[Julientroploin]

Thank you for your fast answer, this affect a lots of tiles near my position (La rivière-de-corps, FRANCE 48° 17′ 19″ N, 4° 01′ 14″ E )
Hope you can help to debug this!

[Akasch]

Ok i can see the error, but I will need 2 or 3 days to fix it.

[Julientroploin]

Thank you @Akasch , I'll check this in a few days.

[Julientroploin]

@Akasch, It work now... I suppose you fix the problem, that's great ! Right in time to contribute under a fantastic spring sun !

[Akasch]

@Julientroploin yes, it was faster than i thought.

[NonnEmilia]

@Akasch also in my zone in Italy we have an old map not showing some recent (1 or 2 months old) roads. Could you kindly take a look?
Coordinates: Lat 44.76617 Lon 10.66540

[Akasch]

The scripts to host a regional part of the tileserver are finally mostly usable. So if you want your region to be updated you can contribute to it. The scripts can be found at: https://github.com/Map-Data/regiontileserver . You will need a Server with ~60 GB of free space, best on an SSD as the import will be faster (Hours instant of Days). Before starting the import please write me a short email so that we do not have unnecessary duplicates.

If you have questions ask me. If you do not want do do the administration yourself but sponsor a region I can do the administration on a provided server/virtual machine, contact me to discuss details.

A visualization of the hosted regions is available at https://map-data.de/#map=3/0/0&style=cinnabar&show_server . The red regions are served by http://nextzen.org with older data. The green ones are currently served by us, with data from the last few weeks. The yellow ones are normally served by us but due to maintenance currently not available so they fall back to nextzen.

[westnordost]

We have another ticket open for this, so I'll close this old one

[Kezxo]

We have another ticket open for this, so I'll close this old one

What ticket is that? I can't find any open ones linked here.

EDIT: is it #1700 ? It probably is.

[rugk]

Duplicate of https://github.com/westnordost/StreetComplete/issues/1700