Distributed

bulk rendering

on the Amazon cloud

Dr. Raphael Volz

nogago GmbH, Gilching, Germany

State of the map, Girona, July 9, 2010

Another 6 years into the project, the planet will have 5 TB (compressed).

Distributed bulk rendering on the Amazon cloud

Hypothesis

July 2010

10 GB

July 2016

5 TB

OSM is power law distributed with top 10% covering only 1 ‰ of planet

Distributed bulk rendering on the Amazon cloud

Observation

Size

(Byt

es

GZ-

com

pre

sse

d)

6x6 splits by size

Top 100 6°x6° areas

# 1 / 1 GBWest 6 South 48

# 2 / 0,6 GBWest -84 South 30

Lower barriers of entry and increase the number of eyes !

Distributed bulk rendering on the Amazon cloud

How to achieve the 5 TB planet

Ride the smartphone wave !

Distributed bulk rendering on the Amazon cloud

Source: Nogago analysis of Juniper Research, Gartner Spring 2009 forecasts

Note: Assumption: Devices sold are in use for two years

Smartphones in use (in Mio.)

For example nogago outdoor is a smartphone appthat accompanies your outdoor activities

Planactivities

Enjoy theOutdoors !

Shareexperiences

• DownloadMaps

• Choose from400.000 tracks

• Plan newtracks

• View tracks

• View photos(geo-referenced)

• Share withfamily, friendsor the world !

• View maps andguides anytimeand offline

• Record andfollow Tracks

Distributed bulk rendering on the Amazon cloud

Our customers will download offline mapsfor regions of choice

Distributed bulk rendering on the Amazon cloud

Raster mapPOI DB

Ways DB

Max.200 MB

per 1°x1°

Generating offline maps takessignificant time and space

Distributed bulk rendering on the Amazon cloud

Using the OSM map rendering tool chain…

Getplanet

osm.bz2

Load intoDatabase

RenderMaps

XML PNG

wget osm2pgsql mapnik

… takes space and time …

Zoom Level 15(1:14000 @eq.)

1 BillionTiles

5 TB

621Days

Assumptions: 16 KB per Tile on average, 20 Tiles per Second to render on modern hardware

Zoom Level 18(1:1700 @eq.)

68 BillionTiles

350 TB

109Years

Basic approach to parallelization usingAmazon cloud

Distributed bulk rendering on the Amazon cloud

Ubuntu 10.04Apache

Postgres 8.4osm2pgsql

osmosismapnik

TimKay‘sawsmapnik-OSMNogago tools

RarelyChangingContent

Total: 3 GB

Render

Image Nogago.com

Amazon S3

Amazon EC2

Amazon EC2runs images

Amazon S3stores files

Render Image

Get

planetosm.bz2

Load intoDatabase

RenderMaps

Storeon S3

Start Instance

UserData

Msg. started

Shut off instance

Msg. finished

Note: UD = User Data, thanks to Eric Hammond (http://ec2-run-user-data.notlong.com)

To make things fit consider corner cases

Distributed bulk rendering on the Amazon cloud

Data to load into Postgres

(depends on zoom levels)

Area to be rendered

by mapnik

Example:

1 x1

to be rendered

at zoom level 10

1,352 x 1,352

to be loaded

into database

190% overheadat zoom level 10

Optimization: Do not always get planetPrepare chunks periodically

• Two step approach, first:

– Download planet, periodically

– Split into chunks

– Store chunks on EC2

• Then, many rendering images

– Get appropriate chunk from S3

– Load (parts of) chunk into Postgres

– Render from Postgres

Distributed bulk rendering on the Amazon cloud

Max.

3 hours

per

1 x1

1 Day

to process

and copy

on S3

What have we learned ?

Rendering the world on zoom level 15

in 28 hours instead of 621 days

( 2 ‰ of the time)

By distributing bulk renderingon the Amazon cloud

Distributed bulk rendering on the Amazon cloud

Thank

you !

Distributed bulk rendering on the Amazon cloud

EU FP7project partner

sought formapping-relatedproject proposal