Openstreetmap-carto: Provide installation script

[matkoniecz]

Version for Ubuntu 14.04: https://github.com/gravitystorm/openstreetmap-carto/issues/657#issuecomment-121279334

Current setup instructions require unnecessary figuring out what and how it should be done.

But it is possible to assume that user have certain Linux (for example in VM) and provide installation script. It would be possible to provide entire installation instruction also in form:

install Lubuntu 14.04, it may be a virtual machine
$ sudo apt-get install git
$ git clone openstreetmap-carto
$ sudo sh ./openstreetmap-carto/lubuntu14.04_installation_script.sh

script would include

• set up of PostGIS database
• installing listed dependencies and fonts
• running get-shapefiles.sh
• fetching some small area suitable for testing and loading this data into database

Note that currently some people even make pull requests without testing them (see for example #654 ).

I may make and test the installation script (it would be mostly copy of http://switch2osm.org/loading-osm-data/ ).

[matthijsmelissen]

@kocio-pl I think something like this should work. Let me know if anything is missing. I would like to have an install script, but I think we first need to have a tested procedural description.

• Install Tilemill
• Load OSM data. You can skip the Section 'Assorted software'.
  If you want to contribute to the stylesheet, first create a fork by clicking the Fork button on Github, and then in the 'Stylesheet' section, instead execute

git clone https://github.com/username/openstreetmap-carto.git

where 'username' is your Github username.

In the 'Loading with osm2pgsql' section, use instead the command

osm2pgsql -d gis ~/osm/liechtenstein-latest.osm.pbf --style openstreetmap-carto.style

• Install some dependencies:

sudo apt-get install mapnik-utils ogr2ogr python-yaml

• Run

./get-shapefiles.sh
ogr2ogr ne_10m_populated_places_fixed.shp ne_10m_populated_places.shp

• To install non-Western fonts, run

    sudo apt-get install ttf-dejavu fonts-droid ttf-unifont fonts-sipa-arundina fonts-sil-padauk fonts-khmeros \
    ttf-indic-fonts-core ttf-tamil-fonts ttf-kannada-fonts

[pnorman]

Install some dependencies:
sudo apt-get mapnik-utils ogr2ogr
pip install pyyaml

It's better to use the python-yaml package.

[matthijsmelissen]

Updated my original comment.

[gravitystorm]

Having an osm2pgsql database loaded with osm data is a common task and certainly not exclusive to this style. It can also be fairly platform-specific.

I'm wary of creating Yet Another Guide(s) to this repo which will, inevitably, bitrot. Could we not just link to an existing guide elsewhere?

[matthijsmelissen]

I think the description in my comment above is the minimum description we need to give, even when we link to existing guides whenever possible.

[matthijsmelissen]

@pnorman Would it be possible to drop the 'Stylesheet' section from http://switch2osm.org/loading-osm-data/? The section doesn't really fit within 'Loading OSM data', and is insufficient as it doesn't talk about fonts, shapefiles etc.

It would requiring removing the '--style ~/osm/openstreetmap-carto/openstreetmap-carto.style' part from the osm2pgsql line though.

[kocio-pl]

For now I just try to achieve the success, but I will try to make a clean room installation some other time to check if all instructions are correct. However I remember "pip" package should be replaced with "python-pip" and "ogr2ogr" with "gdal-bin".

I guess "Yet Another Guide(s) to this repo" is highly recommended, because (a) Ubuntu 14.04 is here to stay for 5 years, so bitrot won't get it too soon =} , (b) process of installing is long and consists of many different and unusual steps, so one can easy get distracted - I'm hardly a newbie in Linux/FLOSS world, but I still couldn't get the environment ready in one go today.

[kocio-pl]

First success is here: I managed to fulfill the database! Now the TileMill is awaiting...

One more thing: there should be some "cd" to the right directories in the instructions. I realized, where I should go to issue some command, by trial and error, but not everybody can effectively handle such guessing.

[kocio-pl]

So I try to create PostGIS project in TileMill and I have even found the correct "crashcourse" part, I guess (https://www.mapbox.com/tilemill/docs/guides/postgis-work/), but I don't know what is the db password, since (using tips from http://switch2osm.org/loading-osm-data/ ) I just created the user and no password was set by me:

sudo -u postgres createuser -s $USER

but when I omit the password, it can't connect to the database.

I don't know what the ID is (the same as the database name - "gis"?). I also don't know what tables should I import and how should I calculate the extent, given that I just get only one country (Poland).

[pnorman]

but when I omit the password, it can't connect to the database

Omit the user, password, hostname, and port, giving only the dbname. This will use local unix sockets to connect.

For extents use -20037508,-20037508,20037508,20037508. This is perhaps slightly sub-optimal for only poland, but will still work.

Also use 900913 for the SRS, not the custom string they gave.

Overall reading through https://www.mapbox.com/tilemill/docs/guides/postgis-work/ I see numerous problems with it.

My guide to setting up TileMill to work with OpenStreetMap Carto is https://hackpad.com/TileMill-I6rxRVszKMv but I haven't had time to finish it off.

[kocio-pl]

OK, so my next right step was adding layer like this:

1. Select PostGIS tab [I probably tried SQLite by mistake before, now the extents field is not here at all, and the extent field doesn't need to be touched]
2. ID field: carto [just sounds nice]
3. Connection field: dbname=gis [as you said!]
4. Table or subquery field: (select * from planet_osm_line where highway like 'path') as paths [example query from the hint under the field]
   ...and Save.

But I needed to add styling, so I just copied the #countries settings in style.mss as:

carto {

::outline {
line-color: #85c5d3;
line-width: 2;
line-join: round;
}
polygon-fill: #fff;
}

and after saving it worked!

So, now I have working database filled with current data and I can use it inside the TileMill at the basic level - double success! But what I really want is testing real carto style. Let's try it then...

[kocio-pl]

Yep:

ln -s ~/osm/openstreetmap-carto ~/Documents/MapBox/project/

did the trick! Now I have my forked style as a separate TileMill project and it starts loading...

However today I won't wait to see how long it takes and if my computer has enough of resources to work with entire country data. It's just a nice point to stop for now. Combined instructions and my guessing were OK, now let's make them simpler and no-guessing.

[pnorman]

You should be able to just run through the commands at https://hackpad.com/TileMill-I6rxRVszKMv, which will get you all the way to running TileMill, which depends on your setup, and opening the project from within TileMill, which is fairly simple.

I have no interest in writing a guide which runs through the steps without explaining, as then you have no idea why you've done what you've done, and if the _slightest_ thing changes the person has no knowledge of what's going on. I have shell scripts which run all the commands to set up an osm2pgsql database from scratch and install TileMill, but they're extremely fragile.

[kocio-pl]

Hm, I'm a bit confused now: this ticket is all about installation script - do you think it is completely unnecessary or you just have some minor/technical issues?

What parts are so fragile (for example I don't think package installation is such a part)?

Maybe the script should have some conditionalities with hints ("Which source do you chose to download data from: 1..., 2..., 3..., skip downloading [press Enter for default: 1]")?

[pnorman]

Given that switch2osm will be getting a guide on setting up the stylesheet when the writing is finished, I think it's unnecessary.

A script will bitrot unless written _very_ carefully. Writing robust scripts isn't trivial. This is based on experience, and answering support questions, as well as having my scripts break.

[pnorman]

@pnorman Would it be possible to drop the 'Stylesheet' section from http://switch2osm.org/loading-osm-data/? The section doesn't really fit within 'Loading OSM data', and is insufficient as it doesn't talk about fonts, shapefiles etc.

No. We need a style file from a stylesheet as one shouldn't expect the default.style to work with a particular stylesheet. I've chosen to go with the openstreetmap-carto.style file instead of one of the others.

[matkoniecz]

IMPORTANT: Current version resides in https://github.com/gravitystorm/openstreetmap-carto/issues/657#issuecomment-121279334

@math1985 - new version: edits in the beginning, working ogr2ogr installation, missing cd, symbolic link.

Needs testing, preferably by somebody new to developing OSM styles.

---

Instructions for installation on Ubuntu 14.04

• Install Tilemill

sudo add-apt-repository ppa:developmentseed/mapbox
sudo apt-get update
sudo apt-get install tilemill libmapnik nodejs

• Load OSM data as descibed on switch2osm. You may skip the Section 'Assorted software', 'Getting ready to load' and 'Tuning'. One should not attempt 'Loading the full planet' as start - 'Loading an extract' is clearly superior for the beginner.
• Open Tilemill and close it (it generates its files in ~/Documents)
• Add openstreetmap-carto to Tilemill projects using symbolic link

ln -s ~/osm/openstreetmap-carto ~/Documents/MapBox/project/

If you want to contribute to the stylesheet, first create a fork by clicking the Fork button on Github, and then in the 'Stylesheet' section in switch2osm guide, instead execute

git clone https://github.com/username/openstreetmap-carto.git

where 'username' is your Github username.

In the 'Loading with osm2pgsql' section, use instead the command

osm2pgsql -d gis ~/osm/liechtenstein-latest.osm.pbf --style ~/osm/openstreetmap-carto/openstreetmap-carto.style

• Install some dependencies:

sudo apt-get install mapnik-utils gdal-bin python-yaml

• Run

cd ~/osm/openstreetmap-carto
./get-shapefiles.sh
ogr2ogr ne_10m_populated_places_fixed.shp ne_10m_populated_places.shp

• To install non-Western fonts, run

    sudo apt-get install ttf-dejavu fonts-droid ttf-unifont fonts-sipa-arundina fonts-sil-padauk fonts-khmeros \
    ttf-indic-fonts-core ttf-tamil-fonts ttf-kannada-fonts

• It is now possible to open Tilemill and see rendered output. Changes in style definition files will result in a changed style of generated map.

[matkoniecz]

I tested and improved previous comment (mostly wording - and two instructions "Open Tilemill and close it" + final one).

[matthijsmelissen]

As far as I'm concerned, we add this to https://github.com/math1985/openstreetmap-carto/blob/master/INSTALL.md .

[pnorman]

I have had nothing but problems with ppa:developmentseed/mapbox causing conflicts with other versions.

[matkoniecz]

@pnorman

I have had nothing but problems with ppa:developmentseed/mapbox causing conflicts with other versions.

So what is the proper way of installing Tilemill? There are recommending this method on their website.

Or is there a suitable replacement for Tilemill?

[pnorman]

I'd prefer recommending Kosmtik over Tilemill.

• Kosmtik is better suited to large complex stylesheets. Tilemill tries to be a text editor, but can't deal with more than 3-4 .mss files
• Tilemill is essentially abandoned. Kosmtik is more of a community driven effort
• Kosmtik is smaller

The main differences with Kosmtik install and setup are

• You need to install it from source (@yohanboniface, any progress there?)
• You point it directly at the project.yaml file
• While running, it only deals with one project

For a new user creating new projects which will only involve 1 MSS file, I could see recommending Tilemill, as the built-in text editor makes it easier to start, but it just doesn't work on large projects.

[yohanboniface]

You need to install it from source (@yohanboniface, any progress there?)

Nope, sadly. Creating a Debian package is not really an issue in se, but I need to sit down a bit to find the best way to deal with Mapnik.

[matkoniecz]

Kosmtik sounds nice (I am not fan of Tilemill but I was unaware that alternative exists), lets hope that standard installation from source is straightforward.

BTW, is there any performance difference compared to Tilemill?

[kocio-pl]

I try to install from source on Ubuntu Utopic, but I get this error (I got node package from regular repositories) - is it the same as @yohanboniface was talking?:

$ npm install
npm WARN package.json [email protected] No README data
-
> [email protected] install /home/kocio/devel/kosmtik/node_modules/mapnik
> node-pre-gyp install --fallback-to-build

[pnorman]

BTW, is there any performance difference compared to Tilemill?

For map development, not really. You're still bottlenecked by Mapnik, which on machines with slow disks and low RAM means you're bottlenecked by the database. There's also the metatile size tradeoff between throughput and responsiveness. I can't speak to time to export tilesets or images.

It's probably marginally quicker in some ways, as it does less.

[pnorman]

Nope, sadly. Creating a Debian package is not really an issue in se, but I need to sit down a bit to find the best way to deal with Mapnik

I was thinking of via npm install kosmtik but I see that works now

[matthijsmelissen]

I also switched to kosmtik.

For me it's slightly faster with a stylesheet reload, which is the main bottleneck when I'm developing. On my computer it's about 23 seconds with tilemill, and about 17 seconds with kosmtik.

[matthijsmelissen]

I do have a bit more problems with cut labels on meta tile boundaries. Is there anything that can explain the difference between tilemill and kosmtik in this respect? CC @yohanboniface

[yohanboniface]

I do have a bit more problems with cut labels on meta tile boundaries. Is there anything that can
explain the difference between tilemill and kosmtik in this respect?

That should not be the case. Things that can have an impact:

• metatile size
• map buffer
• Mapnik version

Fist two points should be exactly the same when loading project on Tilemill or Kosmtik. So maybe you have not the same Mapnik version? Can you check the one you are using on your Tilemill install?

[yohanboniface]

Tilemill seems to be using latest mapnik version since October 2014 so if you are using the master you certainly are on the latest mapnik.

I wonder if we should do the same think on Kosmtik. Note that I'm already using the master locally, but by default kosmtik still require the (old) stable mapnik.

Also note that I don't think OSM servers are already on Mapnik 3.

Anyway, if you want to try to update mapnik on your Kosmtik install, just run npm install [email protected] from Kosmtik root, and see if you have the same rendering as when using Tilemill (which then may not be the same as on OSM servers).

[yohanboniface]

Suggestion: you can use kosmtik-map-compare plugin to compare your local version with the one currently in production.

[matthijsmelissen]

I use the last tagged version of Tilemill, which is 0.10.1. Does that mean that my Mapnik version is 0.7.15 in Tilemill?

[matthijsmelissen]

Which mapnik version do we run in production?

[matthijsmelissen]

And does the problem I experience might have to do with the different default of the clip setting across Mapnik versions?

[pnorman]

We run 2.2 in production. And there is some way to make Tilemill use a system mapnik

[matthijsmelissen]

@pnorman So you don't have problems with metatile boundaries in Kosmtik? My Kosmtik either ignores or cuts all labels stretching metatile boundaries, while Tilemill renders the same labels fine.

[pnorman]

I have lots of metatile boundary label problems with 2.2 and Kosmtik. I just don't debug them.

[kocio-pl]

@math1985:

in https://github.com/gravitystorm/openstreetmap-carto/issues/657#issuecomment-56023323 "install" is missing in:

sudo apt-get mapnik-utils ogr2ogr python-yaml

[matthijsmelissen]

@kocio-pl Thanks, added.

Sorry for little comments on other issues, I'm very busy at the moment.

[kocio-pl]

No problem. =}

I think the task of providing installation script (and updating it) is too big, because it takes us too long. So I have another idea - in my regular work I lately got to know Docker, which is quite interesting Linux virtualization software. It's not typical full-VM system, so it's much faster, but it isolates "containers" from the system and let them be connected in different ways, so one can have different software pieces and combine them (for example postgres-latest + tilemill-git + tile-server). Another nice Docker feature are "recipes" (Dockerfile) for building such containers, which are so readable, that are kind of self-documenting installation script. To my surprise, there are already some of OSM-related containers in the repository - here's a short recipe as an example:

https://registry.hub.docker.com/u/openfirmware/postgres-osm/dockerfile/

The downside is Docker requires 64-bit Linux as a host to run containers. I don't know what kind of people would be interested in osm-carto rendering, but worst case scenario is Windows and 32-bit Linux users can just use full-VM solution like VirtualBox to run 64-bit Linux with Docker containers inside. The performance overhead should be not much worse than full-VM alone (which is quite popular now, I guess), while we would have standard, easy to modify and combine rendering "bricks".

What do you think about this direction?

[pnorman]

What do you think about this direction?

It might be interesting, this isn't the place for it - see https://github.com/gravitystorm/openstreetmap-carto/pull/1342#issuecomment-76924304

[kocio-pl]

@pnorman: OK, I see.

@matkoniecz:

fetching some small area suitable for testing and loading this data into database

What areas should it be? Most probably we would need different areas for testing different features, so:

1. We have to have the list of interesting "testing" places (I remember you're trying to make such collection, am I right?).
2. The script should download all of them - or we can ask the user which ones does he want?

[matkoniecz]

@kocio-pl I am using my home city, it is at the same time well known to me (so it is easier to notice Bad Things Happening), with various features (not everything but quite close) and well mapped. There are even some areas where rendered features are fully mapped.

But always recommending to load well known area is not perfect (mapping may just start or it may be without variety of different features). Advising to always load Kraków (or other location) also would not be a food advise for most people. Add discussing good testing locations would be too long and distracting.

My list of varied testing places is on https://github.com/matkoniecz/CartoCSSHelper/blob/master/CartoCSSHelper.rb (current state: https://github.com/matkoniecz/CartoCSSHelper/blob/c03ff739972102aa552ee14e8524c098d3677bbc/CartoCSSHelper.rb ).

[kocio-pl]

So what do you recommend? I'm lost a bit. =}

[matkoniecz]

@kocio-pl In your case Warsaw is probably a good idea - AFAIK it fulfills all requirements (well known to you, well mapped, variety of features).

[kocio-pl]

Sure, but I was asking about how to solve this problem in general installation script.

[kocio-pl]

What is the meaning of --style option in:

osm2pgsql -d gis ~/osm/liechtenstein-latest.osm.pbf --style openstreetmap-carto.style

I was not able to read about it here nor there.

[yohanboniface]

http://wiki.openstreetmap.org/wiki/Osm2pgsql#Import_style should help

[kocio-pl]

I know it controls things, but I still don't know the meaning for the end user: is it for limiting the data (to get rid of not used in this style), making something faster or what?

[yohanboniface]

You may be confused by the name. Let's say it's called mapping instead of style, because it has nothing to do with rendering: it maps the data to an SQL schema. So it decides _how_ the data will be stored in the DB, which is important then for the carto project because it expects some columns and tables with given names.

[matkoniecz]

for limiting the data (to get rid of not used in this style)

This is part of its purpose (currently not everything is loaded into database). See also https://github.com/gravitystorm/openstreetmap-carto/milestones/3.x%20-%20Needs%20upgrade%20to%20openstreetmap-carto.style - things impossible to do as some tags are not stored in the database.

[kocio-pl]

OK, now I understand!

I was surprised it works on my machine even without this option (probably because I didn't know where to find openstreetmap-carto.style and just forgot to report this problem). The reason is it just happens that default osm2pgsql "style" is the same as what we use currently on osm-carto. However in the future they may be different, so probably we should run it with full path to .style file - in this case:

osm2pgsql -d gis ~/osm/liechtenstein-latest.osm.pbf --style ~/osm/openstreetmap-carto/openstreetmap-carto.style

[matkoniecz]

Instructions for installation on Ubuntu 14.04 - new version, without no longer needed ogr2ogr

---

Needs testing, preferably by somebody new to developing OSM styles.

---

Note about Windows:

In the past I used Windows for developing CartoCSS style and I used
Linux virtual machine (VirtualBox was fine). Later I switched to
dual-booted Linux.

Unfortunately I have no idea how to do this on pure Windows, but I can
only say that Linux turned out to be more user-friendly (and
especially programmer friendly) than I expected.

---

Instructions for installation on Ubuntu 14.04

• Install Tilemill

sudo add-apt-repository ppa:developmentseed/mapbox
sudo apt-get update
sudo apt-get install tilemill libmapnik nodejs

• Load OSM data as descibed on switch2osm. Follow instruction in all sections. It is OK to skip sections 'Assorted software', 'Getting ready to load' and 'Tuning'. One should not attempt 'Loading the full planet' as start - 'Loading an extract' is clearly superior for the beginner.
• Open Tilemill and close it (it generates its files in ~/Documents)
• Add openstreetmap-carto to Tilemill projects using symbolic link

ln -s ~/osm/openstreetmap-carto ~/Documents/MapBox/project/

If you want to contribute to the stylesheet, first create a fork by clicking the Fork button on Github, and then in the 'Stylesheet' section in switch2osm guide, instead execute

git clone https://github.com/username/openstreetmap-carto.git

where 'username' is your Github username.

In the 'Loading with osm2pgsql' section, use instead the command (TODO: why?)

osm2pgsql -d gis ~/osm/liechtenstein-latest.osm.pbf --style ~/osm/openstreetmap-carto/openstreetmap-carto.style

• Install some dependencies:

sudo apt-get install mapnik-utils gdal-bin python-yaml

• Run

cd ~/osm/openstreetmap-carto
./get-shapefiles.sh

• To install non-Western fonts, run

    sudo apt-get install ttf-dejavu fonts-droid ttf-unifont fonts-sipa-arundina fonts-sil-padauk fonts-khmeros \
    ttf-indic-fonts-core ttf-tamil-fonts ttf-kannada-fonts

• It is now possible to open Tilemill and see rendered output. Changes in style definition files will result in a changed style of generated map.

---

Instructions for installation on Ubuntu 16.04:

Install kosmstik

(blocked on https://github.com/switch2osm/switch2osm.github.io/issues/34)

improvements to https://switch2osm.org/loading-osm-data/

<RichardF> Mateusz: sure - you can ask anyone with access to the Wordpress site (which on here includes SomeoneElse, simonpoole, pnorman, Komzpa IIRC) and they can make changes. I'd offer but I'm a bit snowed under with work at the moment I'm afraid.

<RichardF> Mateusz: s2o.github.io isn't really "the new website" at the moment, it's "the stalled project to move it to github"... I don't think there's any work going into making it live though I might be wrong
<Mateusz> RichardF: So you recommend asking one of SomeoneElse, simonpoole, pnorman, Komzpa to make changes to wordpress version?
<RichardF> Mateusz: yep
<Mateusz> SomeoneElse: What is preferred format of suggestions for https://switch2osm.org/loading-osm-data/ ? Description of changes send over mail? Link to patch that can be applied to text? Something else?

[kocio-pl]

Thanks!

Is it ready now to be published or we should test it more?

What about putting this version to some permanent place (like INSTALL.md )? If we want to test the procedure further, I suggest announcing it to Talk or maybe OSM-dev mailing list, because I doubt anybody outside is aware it exists.

[pnorman]

So it decides how the data will be stored in the DB, which is important then for the carto project because it expects some columns and tables with given names

It also defines what is an area and what is a linear feature.

[kocio-pl]

Is anybody contributing here interested in showing potential new developers how do we do it?

[matthijsmelissen]

This is also still an open item, and related to #2291.

Would providing a docker image help here? There already exists a docker image which is quite close to what we need: https://github.com/geo-data/openstreetmap-tiles-docker/
I think we'd only need to add support for openstreetmap-carto and kosmtik.

I never worked with docker before, so I'm not quite sure how to approach this. Does anybody know what the standard workflow is for developing code in a docker image? I suppose keeping the git checkout within the docker instance is not a good idea, because it will be lost when destroying the instance. So should the git checkout be outside the instance, and somehow be called by docker? But than you lose the advantage of providing everything in one bundle. Any suggestions?

[kocio-pl]

Docker is fairly user friendly technology in my opinion. You can mount host directory (which is used for PostgreSQL data for example):

https://docs.docker.com/engine/tutorials/dockervolumes/

We could do the same for Kosmtik code, but it can be also included in the image and updated during image rebuilding. Docker instance (container) would just inherit this code, but we could just make the script at the boot time, which will download fresh Kosmtik and install it, since this is a public code and "destroying" it is not a big thing (if we believe the repo will not disappear and we don't care for possible additional time and network usage during boot).

I would make one bundle with Kosmtik and make an updating script at the container boot. Other tools used there are pulled at the time of image creation and probably don't try to update later, which is safer of course, but it means images should be recreated to catch up with development of these tools.

The image you mentioned could be updated to run Ubuntu 16.04 and newer PostgreSQL, by the way. Lack of license details is also a problem for me.

[Ircama]

Just (too) quick check to this installation script; I wonder whether the following packages are really needed for Kosmtik and openstreetmap-carto:

• postgresql-9.3-postgis-2.1 vs. more recent versions
• Mapnik library
• mod_tile and renderd
• Apache vs. node-js
• osm2pgsql compilation with different method than the README (it should be updated); tmp is not deleted also.

[matthijsmelissen]

@kocio-pl Yes, installing kosmtik during image building is probably best. But what about the openstreetmap-carto code?

[kocio-pl]

We could do the same. Image may contain latest release version and the script inside the container could update it to the latest git. This script could be also used for replacing the repo with the user's fork.

[matthijsmelissen]

postgresql-9.3-postgis-2.1 vs. more recent versions

More recent versions are preferred.

Mapnik library

Isn't this the package we need for shapeindex (in the installatino instructions) by any chance?

mod_tile and renderd

This is not necessary for development, only for running a tile server.

Apache vs. node-js

Not sure what you mean.

osm2pgsql compilation with different method than the README (it should be updated); tmp is not deleted also.

Also not sure what you mean.

[Ircama]

Isn't this the package we need for shapeindex (in the installatino instructions) by any chance?

Yes, Mapnik Utilities

$ sudo apt-get install mapnik-utils

Apache should not be needed.

I guess osm2pgsql should be compiled this way:

# Needed dependencies
sudo apt-get install make cmake g++ libboost-dev libboost-system-dev \
  libboost-filesystem-dev libexpat1-dev zlib1g-dev \
  libbz2-dev libpq-dev libgeos-dev libgeos++-dev libproj-dev lua5.2 \
  liblua5.2-dev

# Download osm2pgsql
cd /tmp
git clone git://github.com/openstreetmap/osm2pgsql.git 

# Prepare for compiling
cd osm2pgsql
mkdir build && cd build
cmake ..

# Compile
make

# Install
sudo make install

# Clean-out temporary files
cd ../..
rm -rf osm2pgsql

[kocio-pl]

Do we really need to compile it? There's a 0.88.1 for Ubuntu 16.04 (and 0.90.1 for yakkety):

http://packages.ubuntu.com/xenial/osm2pgsql

[pnorman]

Any install script/docker image/instructions should not involve compiling anything from source. There is no need.

[kocio-pl]

If geo-data image will not work for us (no open license, stale development), we could make our own image. I guess the hardest part would be proper installation of PostgreSQL, so I would find such image and base our set of tools on it (Docker allows this). However its official images are based on Debian, which would make it harder to maintain, because Ubuntu seems to be current standard.

[pnorman]

However its official images are based on Debian, which would make it harder to maintain, because Ubuntu seems to be current standard

It doesn't matter what OS the postgresql image is running since with docker you should have kosmtik and the db on different images.

[Ircama]

Do we really need to compile it? There's a 0.88.1 for Ubuntu 16.04 (and 0.90.1 for yakkety):

Not needed. I was just referring to this.

[matthijsmelissen]

Any install script/docker image/instructions should not involve compiling anything from source.

With the exception of kosmtik, right? Assuming you consider running nodejs as compiling.

It doesn't matter what OS the postgresql image is running since with docker you should have kosmtik and the db on different images.

This is currently not the case on the geo-data image, right?

[kocio-pl]

It doesn't matter what OS the postgresql image is running since with docker you should have kosmtik and the db on different images.

That would be along Docker philosophy (one service per container), but on the other hand typical Postgres image won't have PostGIS probably and we have to add it anyway, so I have thought it would be easier to maintain just one image.

However I've just found a popular PostGIS image on MIT license and with current software versions (also fresh 9.5+2.2 with current builds available), which could be just fine for us in case we would need to fork it:
https://github.com/appropriate/docker-postgis
https://hub.docker.com/r/mdillon/postgis/builds/

We also need a Kosmtik image - but there is also a few and this one is on Apache license and has fresh builds from git (latest build has some problems):
https://github.com/Joxit/docker-kosmtik
https://hub.docker.com/r/joxit/kosmtik/builds/

So we could just make a simple script running latest postgis and kosmtik containers. We just need to decide how to deal with data and style, which are to be swapped probably:

• do we want to download and import - let's say - Lichtenstein by default or leave it entirely to the user?
• what about osm2pgsql - do we need to create our own small container with it? or maybe we could have it added by other images we use?
• do we want to download standard osm-carto by default (latest release or just latest git) or leave it to the user to define her own fork somehow?

These are questions about where we draw the line between what we want to have by default and where we think it's better to not do it.

[pnorman]

That would be along Docker philosophy (one service per container), but on the other hand typical Postgres image won't have PostGIS probably and we have to add it anyway, so I have thought it would be easier to maintain just one image.

There's a fairly standard postgis image out there. We don't need to do anything to it

[pnorman]

do we want to download standard osm-carto by default (latest release or just latest git) or leave it to the user to define her own fork somehow?

Neither, we want to use the osm-carto in the current working directory

[kocio-pl]

I see. So the script for dockerized Kosmtik needs to be run with something like kosmtik serve ./openstreetmap-carto/project.yml.

And what about the data? I've found docker-osm2pgsql images, but it's rather old and has a common problem with lack of the license:

https://github.com/openfirmware/docker-osm2pgsql
https://hub.docker.com/r/openfirmware/osm2pgsql/

So maybe we could just install osm2pgsql from package, without container?

[matthijsmelissen]

Now I think of it, it would also be really cool if we could provide an image (or series of images? not sure how this works) that would provide a complete tile server. I think that would really stimulate people to run their own tile servers.

CC @systemed

[kocio-pl]

That's exactly why geo-data/openstreetmap-tiles-docker would be so interesting - if they have a license and continue to maintain this project - because this is exactly what they do and they even have nice tools for managing data.

But if they won't, we can use existing images and use them as a service boxes, or build upon them to add what we need - or mix these two methods. It's not a rocket science to create it, but it's better to think twice, because it's a different code than style itself and will need some care for quite a long time probably.

[pnorman]

link for kosmtik: https://github.com/Joxit/docker-kosmtik

We should be able to put together this, a postgis docker, and a small one with the style using docker-compose

[matthijsmelissen]

a small one with the style

I'm still not sure if I fully understand the docker philosophy. Would we need to be able to run get-shapefiles in this repository? In that case, we probably should make it a repository extending ubuntu or something like that, right?

[matthijsmelissen]

I created https://github.com/math1985/openstreetmap-carto-docker as a starting point for the discussion. Is this the right way to think about it, or am I doing something wrong? @pnorman @kocio-pl

[pnorman]

Is this the right way to think about it, or am I doing something wrong? @pnorman @kocio-pl

Have a look at https://github.com/openstreetmap/openstreetmap-website/pull/1290

[kocio-pl]

Default Ubuntu image is currently 14.04, so I would use rather FROM ubuntu:16.04. But we can also use minimal image, which is supposed to solve some Docker-specific issues, so probably FROM phusion/baseimage.

[kocio-pl]

There are also known best practices for using apt-get in Docker, related to its extensive caching. Basically - always use apt-get update in the same line as apt-get install.

If we want to always have fresh git content from osm-carto (every time the container is started, not only when building new image), we should update it through ENTRYPOINT.

And when we have all the needed containers, we have to start them in a coordinated way - it can be made with a simple script, but there also exists Vagrant, which is popular VM "orchestrating" software. See this example.

[matthijsmelissen]

@kocio DId you get the geo-data image to work? I repeatedly get '2016-10-14 20:42:37 UTC FATAL: could not access private key file "/etc/ssl/private/ssl-cert-snakeoil.key": Permission denied'.

[kocio-pl]

No, I didn't try it because of lack of the license - I don't like to waste the time with such uncertain project. I can test it however in a few days, if it would be useful to you in any way.

[kocio-pl]

I have some problems building, but you could also try with a fork which has quite fresh build available:

$ docker run xingfuryda/openstreetmap-tiles-docker

[pnorman]

Here's the dockerfile I have

FROM debian:testing

# Install style dependencies and Kosmtik dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
    nodejs nodejs-legacy npm \
    zip unzip curl git ca-certificates \
    fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont \
    && rm -rf /var/lib/apt/lists/*

RUN npm install kosmtik

RUN mkdir -p /openstreetmap-carto
WORKDIR /openstreetmap-carto

# Get the shapefiles
ADD get-shapefiles.sh ./
RUN ./get-shapefiles.sh

Next steps would be to make a docker-compose file which

• Maps . to /openstreetmap-carto
• Exposes port 6789
• Has a container for PostGIS
• Runs Kosmtik on start

The data loading also requires another dockerfile for osm2pgsql which will load the data into the PostGIS container.

I'm doing an OS upgrade so might not get back to this for a few days.

[kocio-pl]

I was not aware of Docker Compose, but it looks like a better way than Bash scripts or Vagrant rules - thanks!

[pnorman]

OS upgrade finished, and I have something that partially works

https://github.com/gravitystorm/openstreetmap-carto/compare/master...pnorman:docker starts two containers, one for the DB, the other for Kosmtik, and docker-compose up brings them up together. I'm still working on the import side.

to-do before opening a PR for discussion

• [ ] use a data volume for shapefiles
• [ ] get importing going
• [ ] add documentation

[kocio-pl]

Why don't you use Joxit/docker-kosmtik, which you mentioned yourself, or kosmtik/kosmtik for example?

[pnorman]

Why don't you use Joxit/docker-kosmtik, which you mentioned yourself, or kosmtik/kosmtik for example?

I'd have needed to do just as much work modifying it, or end up with another container which does the work of downloading the shapefiles.

[matthijsmelissen]

So currently the import process fails (amongst others) because it has no createdb executable. Should we add postgresql-client to the import Dockerfile? Still trying to understand how this works.

[pnorman]

So currently the import process fails (amongst others) because it has no createdb executable. Should we add postgresql-client-9.3 to the import Dockerfile? Still trying to understand how this works.

Yes. Well, not that version, postgresql-client is what should be used, but I've done this change locally and haven't pushed the changes.

[matthijsmelissen]

Thanks. Do you have any other local changes? Half implementations are fine.

[matthijsmelissen]

The issue I'm not having is:

import_1 | createdb: could not connect to database template1: FATAL: the database system is starting up

In other words, at the moment createdb is executed, the psql server is not yet fully running. Is there a way to handle this?

[pnorman]

In other words, at the moment createdb is executed, the psql server is not yet fully running. Is there a way to handle this?

Not sure.

[matthijsmelissen]

I added a sleep 10 now. There must be a better way though...

[pnorman]

https://docs.docker.com/compose/startup-order/

[matthijsmelissen]

Good point.

[matthijsmelissen]

Next problem: the import Docker file contains WORKDIR /openstreetmap-carto and RUN npm install kosmtik, but when doing a CMD, /openstreetmap-carto/node_modules does not exist. What am I missing?

[kocio-pl]

I would try to look at this, since I consider having easy toolset for developers to be quite important thing - getting to know some details takes me years, so it's good to have some developers to take care for some easier tasks, so we have more people ready to work with the code in the longer perspective.

[matthijsmelissen]

See also https://www.reddit.com/r/openstreetmap/comments/5i6tif/how_to_attract_new_mappers_for_osm/db6kupn/

[Ircama]

Sometimes this could be of help...

[matthijsmelissen]

Would https://shinesolutions.com/2016/12/30/generating-high-res-maps-with-mapnik-and-docker/ help us with the docker work?

[pnorman]

they're using a flag file to indicate if the DB is initialized and hard-coding the location of the data that gets imported. It doesn't help with what we still need to work out.

[nebulon42]

I have updated the branch (https://github.com/gravitystorm/openstreetmap-carto/compare/master...nebulon42:docker) and added some documentation. It basically works. The only thing I have not looked at is if we can prevent get-shapefiles.py from indexing every time and if there need to be other improvements with this script.

Another thing is that the shapefile data currently is owned by the user with UID 1000. First it was root which was even worse. We have to think of something for systems where 1000 is not the default user.

We also have to think about if we should build the containers and upload them to the registry to avoid that everybody has to re-build them.