Openstreetmap-carto: Low-zoom ocean shapefile not rendered correctly on some setups

[codebrane]

Using Mapnik 3.0.9 with openstreetmap-carto 2.39.0 produces tiles with a blue tint. The coastlines are not visible. Using the version from master, the land and inland water is fine but the sea isn't rendered. This tile shows the issue with 2.39.0. I'm using polytiles.py with osm.xml generated by carto from project.mml. If I use the default Mapnik osm.xml on the same data it renders fine but of course with the older OSM style.

[codebrane]

This is what I get using openstreetmap-carto master

and this is what it should look like

[matthijsmelissen]

Have you tried running ./get-shapefiles.sh? We switched from land-polygons to ocean-polygons, this might have something to do with this.

I also saw some blue tiles in Europe on the openstreetmap.org servers, so there might also be a data problem.

[codebrane]

yes. The flow was:

./get-shapefiles.sh
osm2pgsql -c -d gis --slim -C 1000 --flat-nodes flatnodes GB.pbf --style openstreetmap-carto.style
carto -l project.mml > osm.xml
polytiles.py -b -5.93 55.68 -2.29 58.79 -z 10 11 -s osm.xml -t tiles

if I use:
polytiles.py -b -5.93 55.68 -2.29 58.79 -z 10 11 -s osm.xml -t tiles
using the original Mapnik osm.xml the tiles are fine

[pnorman]

The shapefiles are not rendering on either 2.39.0 or master for you.

On master go to the same directory as project.mml and osm.xml and run

ogrinfo -so -al data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp
ogrinfo -so -al data/water-polygons-split-3857/water_polygons.shp

The results should be

INFO: Open of `data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp'
      using driver `ESRI Shapefile' successful.

Layer name: simplified_water_polygons
Geometry: Polygon
Feature Count: 6101
Extent: (-20037558.342789, -20037408.342789) - (20037558.342789, 20037558.342789)
Layer SRS WKT:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
FID: Real (11.0)

INFO: Open of `data/water-polygons-split-3857/water_polygons.shp'
      using driver `ESRI Shapefile' successful.

Layer name: water_polygons
Geometry: Polygon
Feature Count: 35214
Extent: (-20037508.342789, -20037508.342789) - (20037508.342789, 20037508.342789)
Layer SRS WKT:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
FID: Real (11.0)

Check that the feature count is approximately the same.

Also, post the result of ls -hl data/water-polygons-split-3857 data/simplified-water-polygons-complete-3857

[codebrane]

simplified_water_polygons.shp output is identical to above.
water_polygons.shp Feature Count = 35211 instead of 35214

ls -hl data/water-polygons-split-3857 data/simplified-water-polygons-complete-3857

data/simplified-water-polygons-complete-3857:
total 29M
-rw-r--r-- 1 vagrant vagrant    6 Mar  1 08:37 simplified_water_polygons.cpg
-rw-r--r-- 1 vagrant vagrant  72K Mar  1 08:37 simplified_water_polygons.dbf
-rw-rw-r-- 1 vagrant vagrant 169K Mar 30 12:15 simplified_water_polygons.index
-rw-r--r-- 1 vagrant vagrant  847 Mar  1 08:37 simplified_water_polygons.prj
-rw-r--r-- 1 vagrant vagrant  29M Mar  1 08:37 simplified_water_polygons.shp
-rw-r--r-- 1 vagrant vagrant  48K Mar  1 08:37 simplified_water_polygons.shx

data/water-polygons-split-3857:
total 565M
-rw-r--r-- 1 vagrant vagrant    6 Mar  1 08:38 water_polygons.cpg
-rw-r--r-- 1 vagrant vagrant 413K Mar  1 08:38 water_polygons.dbf
-rw-rw-r-- 1 vagrant vagrant 497K Mar 30 12:15 water_polygons.index
-rw-r--r-- 1 vagrant vagrant  847 Mar  1 08:38 water_polygons.prj
-rw-r--r-- 1 vagrant vagrant 564M Mar  1 08:38 water_polygons.shp
-rw-r--r-- 1 vagrant vagrant 276K Mar  1 08:38 water_polygons.shx

[codebrane]

I'm using this to import GB.pbf to the db:

osm2pgsql version 0.91.0-dev (64 bit id space)

[tds4u]

Problem is still there, even with ogr2ogr for all shape files to fix possible inconsistencies.

[matthijsmelissen]

I cannot reproduce this.

@codebrane Did you already find the cause of this problem by any chance?

[tds4u]

I can reproduce that:

mapnik-config -v
2.2.0

node -v
v0.10.29

Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessi

[matthijsmelissen]

@springmeyer Do you perhaps have any idea what might cause this rendering glitch? The screenshot above looks quite particular.

[imagico]

The image shown looks like an 180-degree-meridian reprojection error.

Keep in mind what i wrote in

http://blog.imagico.de/on-slicing-the-world-news-from-openstreetmapdata-com/

on the simplified water polygons - that you should not attempt to reproject them, not even a simple round trip conversion Mercator->Geographic->Mercator. This will fail since the polygons slightly extend over the 180-degree-meridian.

I assume if the coordinate system you have set in Mapnik and the coordinate system in the shapefile are formally different you get a coordinate system conversion and therefore a failure. I am not familiar enough with Mapnik to know if and under what conditions this happens.

[tds4u]

Projection is EPSG:3857 / WGS 84 Pseudo Mercator and in project file it is EPSG:900913 - so the google one. Should be no problem at all.

@springmeyer: Did you test it with planet import?

[tds4u]

QuantumGIS can render this correctly:

I think the shape file is broken, the following is working:

#  - id: "ocean-lz"
#    name: "ocean-lz"
#    class: "ocean"
#    geometry: "polygon"
#    <<: *extents
#    extent: *world
#    srs-name: "mercator"
#    srs: "+proj=merc +datum=WGS84 +over"
#    Datasource:
#      file: "data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp"
#      type: "shape"
#    advanced: {}
#    properties:
#      maxzoom: 9
  - id: "ocean"
    name: "ocean"
    class: "ocean"
    geometry: "polygon"
    <<: *extents
    Datasource:
      file: "data/water-polygons-split-3857/water_polygons.shp"
      type: "shape"
    properties:
#      minzoom: 10
    advanced: {}

[xan-der]

Can confirm this problem with Mapnik 3.0.11 and Mapnik 2.2 and the current openstreetmap-carto pulled earlier this week. All of the global level maps (through level 5 or so) are malformed. I see exactly the same image as reported by tds4u earlier. Note that OSMBright from MapBox works correctly using Mapnik 3.0.11, but it doesn't use water-polygons, it only uses simplified-land-polygons-complete-3857 and land-polygons-split-3857.

[matthijsmelissen]

Thanks for the confirmation. Certainly something we need to look at.

@pnorman Any idea why some users suffer from this and other don't?

[pnorman]

@pnorman Any idea why some users suffer from this and other don't?

No. I'll see if I can reproduce.

[xan-der]

Followup. I was able to get clean world level images by changing project.mml to use the ocean color for the background and land-polygons-split-3857/simplified-land-polygons-complete-3857. Have only tried the first few zoom levels so far. This was the only change made to the mml file - replacing background color and the two layers from water to land.

From this experiment, I have to conclude that there is something squirrely about the water polygons as they are processed by Mapnik.

[pnorman]

Having tried recent water polygons on several versions of Mapnik, I'm unable to reproduce, so I'm concluding that the issue is something local or a problem specific to some version of Mapnik, neither of which are fixable in this repo.

[matthijsmelissen]

Note that we have three people reporting the same problem, so I'd still like to keep an eye on this. I wouldn't like to see this problem suddenly showing up on production without us having an idea of what might cause it.

[springmeyer]

Those that are able to replicate: does the problem go away if you delete the .index file that comes along with the shapefiles?

[xan-der]

The problem did not go away after deleting the .index files for the water polygons. Attached are two images that show (1) what you get when using water color as the background and land polygons as the foreground and (2) what you get when using land color as the background with water polygons as the foreground . Besides the water/land inversion, the rest of the yaml/mml/xml files are identical.

The configuration is: Debian 8.3, Mapnik 3.0.11 (using GDAL 2.0.2, boost 1.60), simplified_water_polygons and water-polygons (split-3857) both dated 18 April 2016, Python 2.7.9.

While Mapproxy is being used as the front end, caching for the example layers is turned off, so the images are regenerated every time.

[springmeyer]

I don't know what would cause this.

[tds4u]

Let me think...

• Carto will convert YAML into CSS via NodeJS. So no problems so far, right?
• Mapnik parses XML and render stuff, equal if 2.2 or 3.x, right?
• Using MapProxy or other render tools is not important because all use Mapnik, right?

So the problem maybe is not inside Mapnik, maybe inside conversion or rendering tool chain. Maybe OGR or similar libraries? But how to check this?

[pnorman]

Mapnik doesn't read CartoCSS, it reads Mapnik XML. Carto converts CartoCSS to Mapnik XML. You could rule it out by building the XML on a known good machine, but I can't see any way for it to be a cartocss problem with that last picture of inverted water bands.

[tds4u]

So, I tested further. With my workaround (https://github.com/gravitystorm/openstreetmap-carto/issues/2101#issuecomment-211353096) I had some blocks on the 180° meridian. That happen if rendering was made till level 10 via MapProxy - rendering till level 8 won't generate such blocks!!! After that all is fine for rendering from 8 to maximum level.
So maybe it's a "feature" / bug on higher zoom levels with shape files?

[pnorman]

I don't know what would cause this.

As @imagico commented, could it be related to the reprojection? Is mapnik making any reprojections here? If so, is it using any libraries which might change the behavior depending on version?

---

@codebrane what zoom range did you observe problems on?

[codebrane]

zoom 10 shows the water problems. Below that there isn't any boundary between the missing water and the land (screenshot of level 5 attached)

[pnorman]

zoom 10 shows the water problems

If you're seeing them on zoom 10+, that means you're seeing problems with both the ocean-lz and ocean layers.

What OS are you using, as well as what geos and proj versions is your Mapnik linked against?

[codebrane]

Ubuntu 15.04
mapnik-config -v = 3.0.9
geos-config --version = 3.4.2

[pnorman]

@codebrane proj version? proj | head -n1 will get you it

[codebrane]

sorry about that
proj | head -n1
Rel. 4.8.0, 6 March 2012

[tds4u]

_Two different servers but same problem, here are the versions:_

Planet (businessmaps.info):

root@server1 / # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessie
root@server1 / # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessie
root@server1 / # proj
Rel. 4.8.0, 6 March 2012
usage: proj [ -beEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]
root@server1 / # node -v
v6.2.0
root@server1 / # mapnik-config -v
2.2.0
root@server1 / # geos-config --version
3.4.2
root@server1 / # gdalinfo --version
GDAL 1.10.1, released 2013/08/26
root@server1 / # osm2pgsql
osm2pgsql version 0.91.0-dev (64 bit id space)

D-A-CH (maps42.de):

root@server0:/# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessie
root@server0:/# proj
Rel. 4.9.1, 04 March 2015
usage: proj [ -beEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]
root@server0:/# node -v
v0.12.14
root@server0:/# mapnik-config -v
2.2.0
root@server0:/# geos-config --version
3.4.2
root@server0:/# gdalinfo --version
GDAL 1.10.1, released 2013/08/26
root@server0:/# osm2pgsql
osm2pgsql SVN version 0.88.0 (64bit id space)

Howto:

lsb_release -a
proj
node -v
mapnik-config -v
geos-config --version
gdalinfo --version
osm2pgsql

[pnorman]

@tds4u What software are you using to render the maps? (e.g. kosmtik, mapproxy, renderd)

[tds4u]

I use MapProxy 1.8.x for pre-rendering tiles (WMS, WMTS, TMS).

[pnorman]

Is anyone getting this with something other than MapProxy?

[StyXman]

Maybe this is related? Parts of Marseille are overwater!
https://www.openstreetmap.org/#map=18/43.29154/5.36511
https://www.openstreetmap.org/#map=17/43.29179/5.36164
https://www.openstreetmap.org/#map=16/43.2901/5.3592

Notice that here is ok:
https://www.openstreetmap.org/#map=15/43.2901/5.3592

[pnorman]

Maybe this is related? Parts of Marseille are overwater!

No, that's unrelated to this issue.

[pnorman]

Is anyone getting this with something other than MapProxy?

The original reported was using polytiles.py, but like MapProxy this uses Python.

It would be interesting if there were any reports from nodejs bindings (e.g. kosmtik, tilemill) or C++ bindings (e.g. renderd)

[matthijsmelissen]

And, for completeness' sake, any reports from MapProxy where this problem does not occur :)

[pnorman]

Can someone reproducing the bug try going into the produced JSON and changing "srs-name": "900913", to "srs-name": "3857", for the ocean-lz layer, regenerating the Mapnik XML from the JSON MML file, then clearing your cache and re-rendering?

Don't make any changes to the YAML.

[tds4u]

On my own server there is no difference in rendering when switching 900913 to 3857 in ocean-lz layer.

Maybe we can use the approach with TIFF files, using DEM data and colored by a shade ramp file?

[xan-der]

While it doesn't fix the current situation, there is a simple workaround. As I mentioned in an earlier post, simply use water color for the background and then use the land polygons shapefile instead of the water polygons shapefile. Conceptually, I'm not clear why the continents are being drawn as negative space in water anyway :)

FWIW, it isn't just MapProxy. A straight Mapnik render results in the same effect.

[pnorman]

While it doesn't fix the current situation, there is a simple workaround

We're interested in fixing the problem, not reverting #2066

FWIW, it isn't just MapProxy. A straight Mapnik render results in the same effect.

With what bindings?

[xan-der]

A non-MapProxy example, using python bindings, OGCServer (pulled today, with edits to change Envelope to Box2d)) via mod_wsgi as the front end. The request looked like this:
http://192.168.218.219/osm?LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3857&BBOX=-20037508.34,-20037508.34,20037508.3384,20037508.3384&WIDTH=256&HEIGHT=256

Standard map_factory.py and mapnix.xml as configuration files. Following the example above, the tool information is:

debian8# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.4 (jessie)
Release: 8.4
Codename: jessie
debian8# proj
Rel. 4.8.0, 6 March 2012
usage: proj [ -beEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]
debian8# node -v
bash: node: command not found
debian8# mapnik-config -v
3.0.11
debian8# geos-config --version
3.4.2
debian8# gdalinfo --version
GDAL 2.0.2, released 2016/01/26

The resulting image looked like this:

[pnorman]

I reduced it to a minimal testcase: https://github.com/pnorman/openstreetmap-carto/tree/onlyocean

Could someone who has the problem try that and verify that they still have problems? A low zoom is best, you should get just ocean/land rendering

[tds4u]

Here is the result:
4326: http://maps42.de:8444/service?LAYERS=OSMLIVE&FORMAT=image%2Fpng&SRS=EPSG%3A4326&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&BBOX=-212.6953125,-105.46875,212.6953125,105.46875&WIDTH=1210&HEIGHT=600

3857: http://maps42.de:8444/service?LAYERS=OSMLIVE&FORMAT=image%2Fpng&SRS=EPSG%3A3857&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&BBOX=-47354267.76322,-23481455.0892,47354267.76322,23481455.0892&WIDTH=1210&HEIGHT=600

[xan-der]

Perhaps narrowing down a bit.

The two attached files render the world very quickly and with no visible artifacts. The same backend, but rendering the same bounds via WMs through either OGCServer or MapProxy give the bad result that tds4u and I have been seeing.

When rendering via OGCServer or MapProxy, it takes a really long time (many minutes) whereas running the C++ or Python code renders in seconds.

mapniktests.zip

On Debian, I have to compile the C++ with this:
$ gcc -std=c++11 -I /usr/local/include/mapnik -L /usr/local/lib -lmapnik -lstdc++ -licuuc basic.cpp

[mboeringa]

Hi all,

I have downloaded the "simplified_water_polygons.shp" shapefile today (31th May) from http://openstreetmapdata.com/data/water-polygons and ran it through a Repair Geometry operation in ArcGIS. ArcGIS detected over 1200 "empty geometries" and maybe a dozen "self-intersections".

I have attached the repaired shapefile in a ZIP. It also includes the logging text file output of the repair tool showing the records in error ("ArcGIS_Repair_Geometry_simplified_water_polygons.txt").

Can someone try this version?
ArcGIS_Repair_Geometry_simplified-water-polygons-complete-3857.zip

I have also ran the shapefile of the simplified water polygons through QGIS 2.14.3 (Essen)'s Check geometry validaty tool. That reported a total of 2484 errors, but each error geometry seems to lead to a double error record in QGIS for this type of error, so the total number is actually quite consistent with ArcGIS's results. Also, at first glance, the error record numbers of the originally downloaded shapefile seem consistent between ArcGIS and QGIS, so that is another good sign of proper checks.

I have attached QGIS's log as ZIPed textfile
QGIS_Check_geometry_validaty.zip

[xan-der]

And here's another variant to narrow down the problem. In these examples, Node.js is used for invoking Mapnik, with a locally generated image coming out good and one generated via WMS server failing.

Using node.js 4.4.5 and node-mapnik (from https://github.com/mapnik/node-mapnik), executing locally with a command like, $ nodejs genimg.js, the result is good:

The code to generate the image is:

var mapnik = require('mapnik');
var fs = require('fs');

// register fonts and datasource plugins
mapnik.register_default_fonts();
mapnik.register_default_input_plugins();

var map = new mapnik.Map(256, 256);
map.load('/tools/openstreetmap-carto/mapnik.xml', function(err,map) {
    if (err) throw err;
    merc_bbox = [-20037508.34, -20037508.34, 20037508.34, 20037508.34 ]
    map.zoomToBox(merc_bbox);
    var im = new mapnik.Image(256, 256);
    map.render(im, function(err,im) {
      if (err) throw err;
      im.encode('png', function(err,buffer) {
          if (err) throw err;
          fs.writeFile('map.png',buffer, function(err) {
              if (err) throw err;
              console.log('saved map image to map.png');
          });
      });
    });
});

Using node-mapnik-wms (from https://github.com/plepe/node-mapnik-wms, the result is:

The call to node-mapnik-wms was: http://192.168.218.219:8001/?LAYERS=__all__&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3857&BBOX=-20037508.34,-20037508.34,20037508.34,20037508.34&WIDTH=256&HEIGHT=256

The WMS server was the demo code provide with node-mapnik-wms running with openstreetmap-carto XML: ./server.js /tools/openstreetmap-carto/mapnik.xml 8001

The module node-mapnik-wms loads node-mapnik to do the rendering. In this example, only the C++ interface is used (eliminating the Python interfaces as the only problem). However, as with the Python interfaces to Mapnik, the WMS server code probably is doing some transforms under the hood.

[mboeringa]

By the way, I also did a quick check on the non-simplified water polygons. That turned up just over 400 'self-intersections' but no 'empty geometries'. I can't attach the repaired shapefile here, as it is to big as you all know.

[tds4u]

I have attached the repaired shapefile in a ZIP. It also includes the logging text file output of the repair tool showing the records in error ("ArcGIS_Repair_Geometry_simplified_water_polygons.txt").

Can someone try this version?
ArcGIS_Repair_Geometry_simplified-water-polygons-complete-3857.zip

No changes in result :-(
You can see it here (click the links):

• maps42.de - 4326
• maps42.de - 3857

[mboeringa]

You can see it here (click the links):

maps42.de - 4326
maps42.de - 3857

I can't get these links to work?

Well, at least a source data problem as the sole cause of trouble now seems less likely... (although there might still be some really pesky problem that ArcGIS can't detect or repair, although it gives a clean bill of health for the repaired shapefile one).

[mboeringa]

@tds4u

As a double check, I have now ran QGIS's Check geometry validity again against the ArcGIS repaired shapefile.

It now turns out QGIS still detects three erroneous records (603,1224 and 1887). You can see the error records in the included image. @tds4u, could you try and _delete_ these records from the repaired shapefile I made available, and try that one again in your workflow? It is likely that you will loose some data now in the rendering, but it will help excluding data problems as the cause of the rendering issues. I did notice in QGIS that the affected three records were of high complexity shape forms (e.g. many inner holes).

[tds4u]

@mboeringa

I can't get these links to work?

Should now work. Test server was down.

PS: Can you give me a hint how to do that? I'm not very familiar with QGIS.
OR: Can you do that for me / us?
I'll test everything for now, but i can't managing geo data itself.

[xan-der]

Following up on earlier post, doing some debug prints, node-mapnik-wms is using srs='+init=epsg:3857', as requested in the URL. In the straight nodejs render, the map has srs=''+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over'.

Changing the code for node-mapnik-wms to force the srs to the explicit Mercator, right before the call to render, resulted in a clean image. It appears that the transform between 3857 and the longer definition is enough to trash the result. Similarly, explicitly setting map.srs and directly rendering using nodejs was enough to reproduce the error.

Taking that and modifying a short C++ program (attached), I was able to generate a bad image (or good) simply by setting the srs of the map right before rendering. This particular example code is a nice short path to Mapnik, without any of the WMS/Python/NodeJS stuff in between.

badimg.zip

[tds4u]

That works with EPSG 3857 :-)

    geometry: "polygon"
    srs: "+init=epsg:3857"
    <<: *extents
    Datasource:
      file: "data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp"

[xan-der]

That works with EPSG 3857 :-)

If I set <Map srs="+init=epsg:3857"... in my mapnik.xml, I get a bad result with either setting of map.srs in the .cpp example. Same with the osm.xml in the reduced/oceans-only version.

However, with a little more effort, it appears the definition of epsg:3857 in /usr/share/proj may have something to do with the problem. I added a definition of epsg:900913 to the end and was able to render w/o problems using 900913. Note the difference at the end of the line for the two definitions:

<3857> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs <>

<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over

It appears the "<>" may be a problem. Modifying mapnik.xml, replacing +over with <> (actually <>), or using an entry in the epsg file with <> causes me rendering problems.

Try changing the definition of 3857 in your /usr/share/proj/epsg so that the <> is replaced with +over and see what happens.

[pnorman]

However, with a little more effort, it appears the definition of epsg:3857 in /usr/share/proj may have something to do with the problem. I added a definition of epsg:900913 to the end and was able to render w/o problems using 900913. Note the difference at the end of the line for the two definitions:

900913 should be in /usr/share/proj/other.extra. On Ubuntu 14.04 it's defined as +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs <>

[pnorman]

proj.4 reference on +over

PROJ4 on the sahpefile is defined as +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs

[pnorman]

Can someone check https://github.com/pnorman/openstreetmap-carto/tree/onlyocean2?

[tds4u]

My current projection file is that:
proj.txt

There is always a "<>" at the end of each line.

Inside:

# WGS 84 / Pseudo-Mercator
<3857> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs <>
#Google
<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs <>

@pnorman
Look at my links with your new code. No improvement.

[mboeringa]

PS: Can you give me a hint how to do that? I'm not very familiar with QGIS.
OR: Can you do that for me / us?
I'll test everything for now, but i can't managing geo data itself.

@tds4u

I hope you still agree it is usefull to dig a little further and find out if this is not caused by a data problem in the first place. Changing the tool chain or its settings may accidentally "fix" the issue through some processing / reprojection step, but won't solve the root cause if it is bad data.

I have now attached a new shapefile that ditches the affected three geometries. QGIS's geometry checker no longer sees issues in this last version.

In the image, you can see which geometries were affected. The red arrows point them out. These geometries are now missing in this latest cleaned file. Interestingly, the upper left geometry is quite close to the latitude where the properly rendered small band is visible in the images posted here. This may indeed indicate a data problem as the root cause, but it is of course highly circumstantial evidence.

@tds4u, can you try this latest version in your original workflow?

ArcGIS_Cleaned_simplified-water-polygons_V2.zip

[tds4u]

@mboeringa
Sorry, no success. Reverted test environment to original shape file.

So, wrong geometries isn't the issue...

[mboeringa]

Sorry, only single color image without borders. Reverted test environment to original shape file.

@tds4u, I would still think it wise to use the cleaned file for further testing in your toolchain, even if it fails in your original workflow. Otherwise you are mixing two variables, and you will never know which one is to blame for the problems. I am pretty sure, even though it may fail in your workflow, that the cleaned data is in a much healthier state than the original data. Both ArcGIS and QGIS give a clean bill of health for the latest dataset, although I am the last to say - knowing from experience how pesky these problems can be - that the cleaned dataset is guaranteed to be without any remaining issue.

Bad geometries can cause a multitude of difficult to debug problems in software and GIS. You don't want that kind of heritage when trying to debug software or figure out a configuration problem.

[pnorman]

@pnorman
Look at my links with your new code. No improvement.

I don't see any links - the branch I linked is new.

[pnorman]

@tds4u, I would still think it wise to use the cleaned file for further testing in your toolchain, even if it fails in your original workflow. Otherwise you are mixing two variables, and you will never know which one is to blame for the problems. I am pretty sure, even though it may fail in your workflow, that the cleaned data is in a much healthier state than the original data. Both ArcGIS and QGIS give a clean bill of health for the latest dataset, although I am the last to say - knowing from experience how pesky these problems can be - that the cleaned dataset is guaranteed to be without any remaining issue.

Please don't use anything other than the file from openstreetmapdata.com when testing with what I'm providing.

[tds4u]

As I wrote, the code is original now - with default shape files.

Links:

• maps42.de - 4326: http://maps42.de:8444/service?LAYERS=OSMLIVE&FORMAT=image%2Fpng&SRS=EPSG%3A4326&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&BBOX=-212.6953125,-105.46875,212.6953125,105.46875&WIDTH=1210&HEIGHT=600
• maps42.de - 3857: http://maps42.de:8444/service?LAYERS=OSMLIVE&FORMAT=image%2Fpng&SRS=EPSG%3A3857&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&BBOX=-47354267.76322,-23481455.0892,47354267.76322,23481455.0892&WIDTH=1210&HEIGHT=600

[pnorman]

Any difference with https://github.com/pnorman/openstreetmap-carto/tree/onlyocean3?

[mboeringa]

Please don't use anything other than the file from openstreetmapdata.com when testing with what I'm providing.

I am slightly puzzled by this statement coming from someone who is - clearly - an experienced geographical database administrator considering all work done on osm2pgsql and Carto.

I would fully concur not introducing another variable in the mix _if the data was known to be good_. However, on the contrary, we now have data proven to have issues.

Continuing to use datasets with known issues, may make it excruciatingly hard to find the cause of the rendering issues.

[xan-der]

Any difference with https://github.com/pnorman/openstreetmap-carto/tree/onlyocean3?

It generates invalid XML when run through carto. Can't have "<>". Attempting to replace those characters with "<&gt"; caused other parsing problems.

[codebrane]

I gave https://github.com/pnorman/openstreetmap-carto/tree/onlyocean2 a try but I get the error:

carto: Millstone not found, required if localizing stylesheet resources. Module did not self-register.

[gravitystorm]

It appears to me to be a fairly straightforward problem - the +over in the projection of the map output is 100% required, and any system missing it out will run into difficulties. This has a long history of appearing within mapnik-related work, see for example some good description regarding +over within Tilemill from back in 2011.

If your software (mapproxy, hand-written scripts etc) uses a proj string without +over - either from the distribution's proj files or via shortcuts like +init=epsg:3857 then you'll get rendering problems with polygons being wrapped. It's most likely to appear with low zoom oceans because a) the shapefiles include polygons right at or even crossing the anti-meridian and b) the polygon which crosses the anti-meridian needs to be in the area that you're rendering, which usually isn't the case for e.g. zoom 16 London.

The true definitions of epsg:3857 and 900913 don't contain +over, so it's not a bug in the proj library. That's why we, along with all the other software using mapnik, use the verbose proj definitions and not the short ones.

[pnorman]

Keep in mind what i wrote in

http://blog.imagico.de/on-slicing-the-world-news-from-openstreetmapdata-com/

on the simplified water polygons - that you should not attempt to reproject them, not even a simple round trip conversion Mercator->Geographic->Mercator. This will fail since the polygons slightly extend over the 180-degree-meridian.

If simplified-water-polygons-complete-3857 extends past 180 degrees, it needs +over in its definition which it doesn't currently have.

[imagico]

If simplified-water-polygons-complete-3857 extends past 180 degrees, it needs +over in its definition which it doesn't currently have.

An argument can be made for that. I however consider the .prj file in the package to serve for documenting the coordinate system rather than facilitating reprojection. It is no promise the data in the file can be converted into other coordinate systems.

Also note while the +over in this case might cure the symptoms of the problem of a coordinate system conversion happening where it should not this only works because the Mercator projection is topologically identical to EPSG:4326. For map rendering in general it is necessary to be able to render also data that lies outside the area of validity of the projection or where conversion to geographic coordinates and back is inherently ambiguous - and no +over can help with that.

[pnorman]

An argument can be made for that. I however consider the .prj file in the package to serve for documenting the coordinate system rather than facilitating reprojection. It is no promise the data in the file can be converted into other coordinate systems.

As it stands right now the shapefile is invalid according to the proj4 string supplied with it.

[imagico]

As it stands right now the shapefile is invalid according to the proj4 string supplied with it.

No, as said the projection specified does not imply the data can actually be converted to another coordinate system. Changing this is an enhancement that would facilitate something useful but it is not a bug that prevents the file from fulfilling its purpose (that is being rendered in native coordinates).

[pnorman]

I can see three ways to fix this

1. Give up on ocean polygons
2. Use ocean polygons at high zoom and land polygons at low zoom
3. Fix the low zoom ocean polygons with ogr2ogr

I don't like 2, which leaves 1 and 3 as options.

[gravitystorm]

@pnorman I think we have different ideas what the problem is here. As far as I'm concerned, the problem is using software that has projection strings that are missing +over. There's nothing wrong with the shapefile, it just happens to be the canary that points out that the proj string used is incorrect.

[pnorman]

@pnorman I think we have different ideas what the problem is here. As far as I'm concerned, the problem is using software that has projection strings that are missing +over. There's nothing wrong with the shapefile, it just happens to be the canary that points out that the proj string used is incorrect.

The shapefile defines its projection as one without +over

Specifically,

$ grep PROJ4 data/simplified-water-polygons-complete-3857/simplified_water_polygons.prj
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],

@codebrane Could you try adding +over to the end of the PROJ4 string in ta/simplified-water-polygons-complete-3857/simplified_water_polygons.prj?

[tds4u]

Some news here?

[pnorman]

Some news here?

Still need someone to test what I asked in https://github.com/gravitystorm/openstreetmap-carto/issues/2101#issuecomment-224637513

[rrzefox]

I'm afraid I can only add to the confusion here, but my tiles (generated with tirex on Ubuntu 16.04, mapnik 3.0.9+ds-1) look just like codebranes in comment 2. No blue or weird artifacts like tds4u, just no sea at all in the low zoom levels (higher ones seem fine). Adding +over to the .prj file does not change anything, however using the files from mboeringas ArcGIS_Cleaned_simplified-water-polygons_V2.zip fixes the problem for me. Could it be that the simplified_water_polygons.prj from at least June 7 and June 15 are broken? (those were the two I tried)

[mboeringa]

Adding +over to the .prj file does not change anything, however using the files from mboeringas ArcGIS_Cleaned_simplified-water-polygons_V2.zip fixes the problem for me.

You probably realized it, but note that in the last V2 version of the files, I deliberately dropped three polygons that ArcGIS could not fix, but QGIS still reported as having an error. They will be missing from the rendering. I only created them for testing purposes.

Your experience really points in the direction of asking @imagico and @joto to try and come up with a version of the shapefiles that cleanly pass the QGIS geometry validation tests without any warnings and errors.

I know this may not be an easy task, as the coastline process is probably pretty complex, given the nature of the data, but I do think this option and the possibility for a clean valid dataset should be explored.

[imagico]

Just for info - the process for generating the simplified water polygons is open source, it is available on

https://github.com/imagico/poly_grid

This process currently does not remove all empty features from the database that remain after processing (because ST_Area(GEOMETRY) on an empty geometry is NULL and not zero) but this will not affect any application i know if in a significant way. This will be fixed on occasion but is not of high priority.

This of course has nothing to do with the subject of this issue.

[mboeringa]

@imagico: ArcGIS and QGIS also detected invalid shapes with other errors than just being empty.

Don't consider my remarks about the dataset's problems as critic. I am well aware of the effort you guys must have put into developing some form of working process for the creation of a usable coastline dataset from the inherently complex OSM datasource.

I just think that if there are options of tackling these issues on the data side, that it may be worth the investment. While empty shapes may not be a big issue for some software, polygon features with other types of problem, can cause a lot of issues. And as such, I do think it may be relevant to the issues posted here.

P.S. maybe a nice GSOC project?...

[rrzefox]

Up until now, I do not consider it to be proven that the repairing with Arcgis made the polygons work, it might just as well have been a coincidence - maybe the s-w-p version from May used as the basis for the repair would work unmodified too? There is no way to tell, or is there an archive of old s-w-p-versions somewhere?
Of course in the "repaired-v2" version some polygons are missing, but "coast looks broken in New Guinea" is still a vast improvement over "sea is missing on the whole planet".

I have since tried the first "repaired" version (ArcGIS_Repair_Geometry_simplified-water-polygons-complete-3857.zip), and it seems to work just as well. Even the area with the broken polygon off the coast of New Guinea that was missing from v2 looks just fine.

Looking at the Arcgis-log in the .zip, apart from many empty geometries which I'd assume to not hurt, the only repaired errors are "self intersections".

[mboeringa]

Up until now, I do not consider it to be proven that the repairing with Arcgis made the polygons work, it might just as well have been a coincidence

Agree, but using a dataset without geometric errors for testing, is likely to make debugging a potential other issue in the toolchain a lot more straightforward.

[matthijsmelissen]

I have reverted the change that causes this for now.

I still would like to merge this eventually, so I have created a branch ocean-polygons which can be used for testing the problem (this branch is not intended to be released, it is only there to make testing easier).

[imagico]

Just FYI - the changes announced above (adding +over to the projection definition and removing empty geometries) are now implemented in the shapefiles available.

[mboeringa]

@imagico and others,

I have run the new shapefiles as provided by imagico through ArcGIS's and QGIS's geometry validation checks.

@imagico , _are you absolutely sure_ you updated the shapefile links properly on the website making them available?

Because unfortunately, as to empty or Null shapes, both ArcGIS and QGIS still report the same results. They both detect again just over 1200 of them, and ArcGIS still reports 10 self intersections.

The self intersections ArcGIS reports are these records:

If I click on one the error records in QGIS Check Geometry Validity dialog, it throws an Python error referencing an issue with the "boundingBox" propery of a "NoneType" object instead of geometry. See below:

Traceback (most recent call last): File "C:\PROGRA~1\QGISES~1\apps\qgis\python\plugins\fTools\tools\doValidate.py", line 191, in zoomToError rect = mc.mapRenderer().layerExtentToOutputExtent(self.vlayer, ft.geometry().boundingBox()) AttributeError: 'NoneType' object has no attribute 'boundingBox'

I have used ArcGIS's Repair Geometry tool again to clean the shapefile. If I subsequently run QGIS's Check Geometry Validity, then, just like before, QGIS still detects a few erroneous records, this is consistent with previous results. See the attached image for the remaining issues.

I have also included the repaired shapefile again as a ZIP:

ArcGIS_Repair_Geometry_simplified-water-polygons-complete-3857_V3.zip

The number of records compared to the original has been going down from 6165 to 4923 in the repaired version.

[pnorman]

@codebrane or @tds4u, can either of you reproduce with current shapefiles and one of the ocean branches?

[codebrane]

not sure what combination to try. Should I clone master and use get-shapefiles.sh as normal and do 'something' with the ocean-polygons branch? If you could point me in the right direction that would be great thanks

[imagico]

You can fetch and checkout the ocean-polygons branch:

git fetch upstream ocean-polygons
git checkout ocean-polygons

use its get-shapefiles.sh to get the water polygons shapefiles and test it.

[codebrane]

'fraid it doesn't work. These are the steps I took, just to make sure I'm doing it right:

clone master
recreate gis database with osm2pgsql with GB.pbf and openstreetmap-carto.style from master
clone ocean-polygons branch to separate dir
copy get-shapefiles.sh from ocean-polygons to master/get-water.sh
run master/get-shapefiles.sh and master/get-water.sh
./polytiles.py -b -4.4 57.37 -3.8 57.63 -z 9 10 -s osm.xml -t tiles

should be like this:

but is like this:

[pnorman]

Unfortunately I haven't had a chance to check the steps above. Are you using polytiles.py from https://github.com/Zverik/polytiles?

[imagico]

Since this issue blocks a significant number of changes we have in mind (see #2507) i think it is time to make a decision here.

The decision to make is if we require all data we use, esp. shapefiles, to be compatible to round trip conversion Mercator->Geographic->Mercator. If Mapnik cannot be reliably forced to not perform such round trip conversion and we want to continue supporting applications that lead Mapnik to do this in some way even though technically this is clearly unnecessary we obviously would have to do this.

If such a decision is made there would still be the options to either modify the script used to invert the land polygons (which i linked to in https://github.com/gravitystorm/openstreetmap-carto/issues/2101#issuecomment-226549830) to generate such a shapefile or to post process the data we have specifically for osm-carto.

Needless to say i consider this pointless from a technical standpoint and it would probably be easier to just add support to Mapnik to specifically treat input data as being in native coordinates. But obviously with the current state of development of Mapnik this option is quite academic.

The alternatives would be to:

• drop support for applications that perform round trip coordinate conversion.
• stop using simplified coastline polygons at all and use the special low zoom preprocessed waterbody data for z0-z6 and the full detail water polygons for z7+. This is explained in #2507 with its implications.

I would also welcome any other alternatives being developed by others but given how long this whole subject has been around i don't have high hopes that new developments suddenly turn up.

[matthijsmelissen]

drop support for applications that perform round trip coordinate conversion.

Which software would that be?

[imagico]

Apparently the setups used by @codebrane and @tds4u. But since the discussion in this issue was not particularly goal oriented overall so far it is not really clear under what conditions exactly Mapnik performs a reprojection. It is possible that this can be prevented reliably (at least by making sure there isn't any other proj4 string involved than the one in this style) although some applications might be hardwired in a way that makes this impossible.

[matthijsmelissen]

I have reverted the change that causes this for now.

And therefore this issue can be closed.

[jeisenbe]

While this issue is currently closed, it's disappointing that we never confirmed the cause, though it is most likely related to round-trip reprojection of the simplified ocean polygon shape file.

Andy Alan seemed to think it was a problem with incorrect setup on the affected servers, since this reprojection should not be allowed:

"It appears to me to be a fairly straightforward problem - the +over in the projection of the map output is 100% required, and any system missing it out will run into difficulties." Full comment: https://github.com/gravitystorm/openstreetmap-carto/issues/2101#issuecomment-223933484

There were some thoughts that the shapefiles were a problem, but changing to shape files to add +over and to remove the empty geometries did not seem to fix the problem. However only one user seems to have tested after this change, and did not respond to a follow-up question a couple of months later: https://github.com/gravitystorm/openstreetmap-carto/issues/2101#issuecomment-266698640

@pnorman what are your thoughts? Do you agree with Andy's comments that this was due to inappropriate setups on the affected servers?

[rrzefox]

It has been a long time since this was closed, so it might be worth invastigating this again.
As one of those that saw this problem in the past, I can say that I have not seen this occour again after the german fork of this style switched to using water polygons (we render that style too).
As to WHY I did not see this again I have no clue, there are too many variables. The german style might use them in an other way that avoids the issue. We have updated to a new Ubuntu LTS release with significantly newer software versions. And last but not least, I never was convinced it wasn't the shapefiles that were to blame (at least partially), so whether you could see the error or not (also) depended on what day you downloaded them on. BTW, to be able to debug that by using a version from a specific date I have since set up https://ftp.fau.de/pub/openstreetmapdata.com-archive/ as openstreetmapdata.com itself only ever keeps the newest version.

[kocio-pl]

Thanks for the comment, I was about to ask you if you could test it. If water polygons are generally working elsewhere, I think we could take the risk to go this way too, because the stakes are high enough. As @Adamant36 said, it's possible to revert this if something will go wrong.

I would start with making test code branch and asking again affected users if they still see the problem. If not, we could merge it soon after release, so there will be plenty of time to do additional testing.

[imagico]

If this is done - and i don't want to prejudice the opinion of other maintainers in any way here - it would be important to come to a consensus how long to wait afterwards before changes that depend on ocean polygons are made (which would prevent a simple revert).

[jeisenbe]

@rrzefox thank you for the update, and for maintaining that archive! I believe if the German fork is rendering properly, we are likely to be able to implement the same.

how long to wait afterwards before changes that depend on ocean polygons are made

I'd suggest merging the changes to Master right after the next release, which will give current contributors and other interested parties several weeks for testing before the changes are released. Then we should wait 1 more month after the water polygons are in production before merging PRs that depend on this change.

Example: if the new release is tomorrow, we should submit a PR using the split and simplified water polygons; and hopefully we can merge this in a week or two. Then we will have 3 or 4 weeks with the water polygons merged to master, but not yet released. Assuming the next release after that is in mid-March, we would then wait till mid or late April before merging any PRs that depend on the change.

Does that sound like enough time for testing?

[jeisenbe]

We are considering changing to ocean polygons again, which risks reintroducing this issue. See PR #3694 - however @rrzefox has reported that he no longer has the problem.

@codebrane , @xan-der and @tds4u - would you be able to test the new branch from the PR to see if this will now work on your current system? The branch name is jeisenbe:ocean-polygons https://github.com/jeisenbe/openstreetmap-carto/tree/ocean-polygons

If anyone else had seen this bug back in 2016, please let us know if the shapefiles are now working for you.

[sommerluk]

On my local machine with a fresh download of all shape files, the ocean polygons do not render. I can fix this by deleting the .index files. I've done this also for the low-zoom boundaries (otherwise the rendering eats all the memory and crashs) and for ice shields.

What is the role of the .index files? Are there side-effects deleting them? If not, could we do this automatically in our shapefile download script?

[pnorman]

What is the role of the .index files? Are there side-effects deleting them? If not, could we do this automatically in our shapefile download script?

We should automatically be building new ones as part of the download.

[sommerluk]

When building new ones (with default options: shapeindex *.shp) rendering is broken again.

[imagico]

@StyXman pointed out in https://github.com/imagico/poly_grid/issues/1#issuecomment-498847357 that this might be specific to mapnik master and does not occur in released versions.

[StyXman]

It's also broken for mapnik-3.1.0, which is the one that kosmtik is using. See https://github.com/kosmtik/kosmtik/issues/295 and https://github.com/gravitystorm/openstreetmap-carto/issues/3717

Rendering those shapefiles with python-mapnik works for me.

[imagico]

Does it work fine without shapeindex as @sommerluk described?

[StyXman]

Indeed it does with mapnik-3.1.0. Interesting...

[sommerluk]

This could maybe be a missmatch of the Mapnik package of the Linux distribution (which provides also the program that generates the index files) and the node-mapnik version used by kosmtik?