Generating Maps With Mapnik and Docker

gowebing

I have been working on a project that is a real-estate web portal. One of the main features of the portal is maps. You can search for a property on the map using certain criteria. You can see what other objects – supermarkets, hospitals, kindergartens – are located near it, or you can drill into information about specific properties.

Initially, I chose to use OpenStreetMap to render maps, the main reason being the level of detail that it provides. For instance, you can see things like factories, industrial zones, suburb borders, schools, etc. This information is quite important when you are looking for a new place to live.

However, I ran into a problem when I got my first Retina MacBook Pro. Maps looked a bit blurry on it. It turned out that the problem was the Retina (hi-res) display.

What I needed were special tiles that rendered with a bigger scale factor to make them sharper. I was looking for a free, ready-to-use solution, but couldn’t find one. There are some commercial projects that you can use, but they cost money, which we didn’t really have for that project.

Furthermore, I needed to keep the tile format the same, so I don’t need to make any modifications on the front end.

Fortunately, OpenStreetMap not only serves up map tiles but also gives you access to the huge geo-database underneath, as well. So, I decided to render my own tiles from this database.

In this article, I’ll present my solution. Importantly, this solution will use Docker to build and run. Using Docker helps avoid problems running shared libraries across different platforms because most of the tools in that area written with C++. The hope is that you’ll be able to easily setup this solution in your own environment.

Let’s Get Started

To support Retina image display, we need to generate tiles with 2x resolution. The web client can then request these images and render them into the same sized area on a web page that was previously being used for the non-Retina images.

For instance, if we have an image foo.png with a size "128px x 128px," the client will implicitly give the tag an area of "128px x 128px." For a Retina-grade image, we would now have the client request an image with size 256px x 256px, but explicitly keep the display area at "128px x 128px."

So, instead of having the client request prerendered tiles from the OpenStreetMap service, we’ll have it request them from a custom back-end that can render high-resolution tiles. Also, rendering our own tiles can be handy if:

Generally speaking, if you need flexibility and you have compute resources, then rendering tiles yourself is the way to go. Let’s take a look at the solution!

Solution Overview

We will set up our own render server with a custom map style. That renderer will generate map tiles as png images with a scale factor of 2. We need a custom style mainly to change the font size because it’s too big for the new tiles (otherwise, objects would be rendered closer together, but the font size would remain the same). Also, it’s handy to be able to change a style to use our own color scheme or drop some object type that we don’t need.

As an example, our solution will render Melbourne city. It will render a map that will look like this:screen-shot-2016-12-28-at-12-06-36-pm

Which we can contrast to the original one from openstreetmap.org:screen-shot-2016-12-28-at-12-06-27-pm

Here are projects that the solution will use to render our tiles:

Best of all, everything is packaged up inside Docker so we’ll be able to reuse it in any environment.

Building It

Note: For this to work on your machine, you should have ports 8888,8080,8090 open and free on your host machine.

The project is hosted on GitHub and contains a Docker Compose environment that will run the Database + Gopnik cluster + Web example. First of all, let’s clone the repo:

$ git clone https://github.com/dooman87/melbourne-map.git

We also need some static files to download because they will be required to render some basic stuff (i.e., continent shapes) on low zoom. The files are quite big (the total is about 1 Gb), which is why we don’t download them as part of the Docker image-building step. To download these static files, go to your repository clone and run:

$ ./download.sh

Now let’s have a look at docker-compose.yml:

version: '2'
services:
  postgis:
    build: ./postgis
    volumes:
      - ./data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=gis
  renderer:
    build: ./renderer
    volumes:
      - ./renderer/map_data:/map_data
      - ./renderer/scripts:/scripts
    volumes_from:
      - postgis:ro
    ports:
      - "8080:8080"
      - "9090:9090"
    depends_on:
      - postgis
  web:
    image: nginx:1.11-alpine
    volumes:
      - ./web:/usr/share/nginx/html
    ports:
      - "8888:80"

You can see that we have three services. Let’s have a look at each a bit closer:

1. Postgis Service

This service runs a database that contains imported data for Melbourne city. This data has been extracted from OpenStreetMap by the guys at the mapzen project, and is also open source (you can find extracts for all big cities there). The Docker image itself is based on a postgis image that was created by mdillon. I added the osm2pgsql package because we depend on it to import map data into the database. The entire Dockerfile is quite small and simple:

FROM mdillon/postgis:9.5

WORKDIR /

RUN apt-get update && \
    apt-get install -y osm2pgsql git wget && \
    rm -rf /var/lib/apt/lists/*

RUN git clone https://github.com/dooman87/openstreetmap-carto.git && \
    wget https://s3.amazonaws.com/metro-extracts.mapzen.com/melbourne_australia.osm.pbf


#Overriding init script to add hstore extension that osm2pgsql requires
COPY ./initdb-postgis.sh /docker-entrypoint-initdb.d/postgis.sh

We clone the openstreetmap-carto project because it defines what objects we will render, as osm2pgsql will need to know what objects should be imported into the DB. The project contains a file called openstreetmap-carto.style that defines objects and has a simple structure of things that will be in DB. For example, here’s a fragment of the file:

# OsmType  Tag          DataType     Flags
node,way   access       text         linear
node,way   addr:housename      text  linear
node,way   addr:housenumber    text  linear
node,way   addr:interpolation  text  linear
node,way   admin_level  text         linear
node,way   aerialway    text         linear
node,way   aeroway      text         polygon
node,way   amenity      text         polygon
node       capital      text         linear
...
...

So, for example, if we don’t want to have labels on capitals, then we can simply remove the line with that type from the file.

Also, I had to customize the Docker entry point script to import map data upon first starting the container and create a flag-file to let the renderer know that the database was successfully imported (last two lines):

#!/bin/sh

set -e

# Perform all actions as $POSTGRES_USER
export PGUSER="$POSTGRES_USER"

# Create the 'template_postgis' template db
psql --dbname="$POSTGRES_DB" <<- 'EOSQL'
CREATE DATABASE template_postgis;
UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
EOSQL

# Load PostGIS into both template_database and $POSTGRES_DB
for DB in template_postgis "$POSTGRES_DB"; do
echo "Loading PostGIS extensions into $DB"
psql --dbname="$DB" <<-'EOSQL'
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;
CREATE EXTENSION fuzzystrmatch;
CREATE EXTENSION postgis_tiger_geocoder;
CREATE EXTENSION hstore;
EOSQL
done

#import Melbourne city
osm2pgsql --style /openstreetmap-carto/openstreetmap-carto.style -d gis -U postgres -k --slim /melbourne_australia.osm.pbf

touch /var/lib/postgresql/data/DB_INITED

2. Renderer Service

This is the renderer that generates tiles using Gopnik. The image below is showing how exactly rendering process is working. “Dispatcher” and “Render” are Gopnik’s components. We are using a normal filesystem as a cache. Mapnik is used as a library that “Render” component calls:

render

Source.

The Docker image for the renderer is based on an original gopnik image that I created and contributed back to the project. Here’s the Docker file:

FROM dpokidov/gopnik

RUN apt-get update && \
    apt-get install -y git curl unzip node-carto mapnik-utils fontconfig && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /
RUN wget https://github.com/adobe-fonts/source-sans-pro/archive/2.020R-ro/1.075R-it.tar.gz && \
    tar -xzvf 1.075R-it.tar.gz && \
    mkdir /usr/share/fonts/truetype/sourcepro-ttf/ && \
    cp source-sans-pro-2.020R-ro-1.075R-it/TTF/*.ttf /usr/share/fonts/truetype/sourcepro-ttf/ && \
    fc-cache && \
    rm 1.075R-it.tar.gz && \
    rm -rf source-sans-pro-2.020R-ro-1.075R-it

RUN git clone https://github.com/dooman87/openstreetmap-carto.git
WORKDIR /openstreetmap-carto
ADD shapes/ data/
RUN shapeindex --shape_files \
    data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp \
    data/water-polygons-split-3857/water_polygons.shp \
    data/antarctica-icesheet-polygons-3857/icesheet_polygons.shp \
    data/antarctica-icesheet-outlines-3857/icesheet_outlines.shp \
    data/ne_110m_admin_0_boundary_lines_land/ne_110m_admin_0_boundary_lines_land.shp

RUN mkdir /map_data
VOLUME /map_data

RUN mkdir /scripts
COPY scripts/compile_style.sh /scripts
COPY scripts/run_render.sh /scripts
COPY entrypoint.sh /

RUN chmod 755 /scripts/*.sh && \
    chmod 755 /entrypoint.sh

WORKDIR /
ENTRYPOINT /entrypoint.sh

Some key points about this file:

"Cmd": ["/gopnik/bin/gopnikslave", 
"-stylesheet", "/openstreetmap-carto/stylesheet.xml", 
"-pluginsPath", "/usr/lib/mapnik/2.2/input", 
"-fontsPath", "/usr/share/fonts", 
"-scaleFactor", "2"]

The entry point script of the Docker container for the renderer service compiles the map style and waits until the DB becomes accessible. We kick off the renderer process once the DB is ready:

#!/usr/bin/env bash

sh /scripts/compile_style.sh

while [ ! -e /var/lib/postgresql/data/DB_INITED ]
do
sleep 5
echo "Waiting while database is initializing..."
done

#Have to wait because once DB created then osm2pgsql restarting postgres.
#TODO: Using pg_isready
echo "DB successfully created, waiting for restart"
sleep 60

echo "Starting renderer"
sh /scripts/run_render.sh

3. Web Service

This is a Docker container that uses a ready-made Nginx image to serve up a simple example web page. It uses the leaflet library to show a map and position at Melbourne:

<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<link rel="stylesheet" href="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.css" />
<script type="text/javascript" src="http://cdn.leafletjs.com/leaflet-0.7.1/leaflet.js"></script>
<style type="text/css">
html, body, #map {
width: 100%;
height: 100%;
margin: 0 !important;
overflow: hidden;
}
</style>
</head>

<body>
<div id="map"></div>
</body>

<script type="text/javascript">
var map = L.map('map').setView([-37.8130, 144.9484], 14);
L.tileLayer('http://localhost:8080/{z}/{x}/{y}.png', {
attribution: 'Map data &copy; <a href="http://openstreetmap.org">OpenStreetMap</a> contributors, <a href="http://creativecommons.org/licenses/by-sa/2.0/">CC-BY-SA</a>, Imagery © <a href="http://cloudmade.com">CloudMade</a>',
maxZoom: 18
}).addTo(map);
</script>
</html>

Starting It

To run things, you need to run the following command in your repository clone:

$ docker-compose up

That’s really it. The first time it will take a while to build images and start up (up to 10 minutes) because we need to import the database. You can find out when all containers are started and ready to go by looking for log messages that look like this:

postgis_1   | LOG:  autovacuum launcher started
renderer_1  | Starting renderer
renderer_1  | 2016/12/27 23:34:30 app.go:266: [INFO] Serving debug data (/debug/vars) on %s... :9080
renderer_1  | 2016/12/27 23:34:30 app.go:267: [INFO] Serving monitoring xml data on %s... :9080
renderer_1  | 2016/12/27 23:34:30 renderselector.go:209: [DEBUG] ping error %v dial tcp 127.0.0.1:8090: getsockopt: connection refused
renderer_1  | 2016/12/27 23:34:30 renderselector.go:117: [DEBUG] '%v' is %v localhost:8090 Offline
renderer_1  | 2016/12/27 23:34:30 main.go:118: [INFO] Starting on %s... :8080
renderer_1  | 2016/12/27 23:34:30 app.go:266: [INFO] Serving debug data (/debug/vars) on %s... :9090
renderer_1  | 2016/12/27 23:34:30 app.go:267: [INFO] Serving monitoring xml data on %s... :9090
renderer_1  | 2016/12/27 23:34:35 main.go:95: [INFO] Done in %v seconds 4.84147165
renderer_1  | 2016/12/27 23:34:35 main.go:103: [INFO] Starting on %s... :8090

Testing it

Now you can open your favorite browser and go to:

http://localhost:8888

The initial rendering may take 10-30 seconds, after which you will see a fabulous map of Melbourne city.

Shipping It

How you ship this solution is really up to you. I’m using this in a project that has about 300 hits per day (what a big boy!). I’m running all the containers on one EC2 instance. Also, if you cache all tiles that are already rendered, then you won’t need huge CPU resources and – up to a point – may even have some resilience if the Gopnik renderers go down. You can even get the Gopnik renderers to prerender tiles if you want.

That said, strictly speaking, this is not a production-ready solution. From gopnik's docs:

Code is unstable. Current status of Gopnik is something like developer preview. Everything may change. Everything may be broken.

Regardless, it’s probably worth mentioning that uptime of my EC2 instance is currently 5412 hours (launched on May 18, 2016) and it’s still serving tiles.

Where Is  NEW YORK ,  PARIS ,  MOSCOW , MILAN?

It’s quite easy to render other cities. Just update the Postgis Dockerfile to download a different PBF file and update its name in the entry point script.

Here’s the line that you would change in the Dockerfile:

wget https://s3.amazonaws.com/metro-extracts.mapzen.com/melbourne_australia.osm.pbf

...and here’s the line you would change in initdb-postgis.sh:

osm2pgsql --style /openstreetmap-carto/openstreetmap-carto.style -d gis 
-U postgres -k --slim /melbourne_australia.osm.pbf

After you’ve changed these, you then clean and rebuild everything:

$ ./clean.sh
$ docker-compose build
$ docker-compose up

What’s Next?

So, that’s my solution for serving up Retina-quality map tiles, packaged up with Docker for easy distribution.

In the ideal world, it would be nice to use vector format like SVG to render tiles. That way, you wouldn’t have to worry about serving up particular resolutions. There are some things going on in that space, along with commercial solutions.

There is one project that I’d like to mention: osm2vectortiles. You can generate vector tiles from data extracts using it. So, the next step for my project would be to evaluate the current solution to generate vector tiles instead of raster.

 

 

 

 

Top