r e the geofabrik download server s a o n w o d k b f o e ... · the geofabrik download server what...

24
G e o f a b r i k D o w n l o a d S e r v e r The Geofabrik Download Server What it does & how it works Frederik Ramm <[email protected]> SOTM-US, San Francisco 08 June 2013

Upload: others

Post on 28-Oct-2019

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

The Geofabrik Download Server

What it does & how it works

Frederik Ramm <[email protected]>

SOTM-US, San Francisco08 June 2013

Page 2: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

What it does

● cuts “planet file” in handy pieces● daily updates● Shapefiles

new:● regional diff updates● collection of older files for each region

Page 3: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Page 4: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Page 5: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Page 6: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Data held

● all continents● most countries● sub-country units for US, Canda,

Germany, France, UK

Page 7: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Why?

● easier for everyone to work with● makes Geofabrik a “household name” in

OSM circles

Page 8: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Page 9: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

(today: 2.5 GB)

Page 10: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

A couple numbers

● monthly traffic ~ 50 TB(20 billable, 5 Hetzner internal, 25 handledby GWDG FTP mirror)

● about 2k unique (downloading) visitors daily

● PBF data preferred 3:1 over bzip2● 5% most popular files account

for 90% of traffic

Page 11: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

A couple numbers

● monthly traffic ~ 50 TB(20 billable, 5 Hetzner internal, 25 handledby GWDG FTP mirror)

● about 2k unique (downloading) visitors daily

● PBF data preferred 3:1 over bzip2● 5% most popular files account

for 90% of traffic

Thank you GWDG!

Page 12: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

A couple numbers

● used to be computed on server at theGeofabrik office

● later moved bz2 generation to download server

● meanwhile all done directly on download server – office line had 10 MBit/s upstream and could not cope

Page 13: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

How does it work?

Page 14: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

How does it work?

planet

Page 15: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

How does it work?

planetnew

planetosmosis

OSM

Page 16: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

How does it work?

planetnew

planetosmosis

OSM

east

west

newplanet

Page 17: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

How does it work?

planetnew

planetosmosis

OSM

east

west

newplanet

osm-history-splitterosm-history-splitterosm-history-splittercontinentscontinentscontinents

Page 18: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

How does it work?

planetnew

planetosmosis

OSM

east

west

newplanet

osm-history-splitterosm-history-splitterosm-history-splittercontinentscontinentscontinents

osm-history-splitterosm-history-splitterosm-history-splittercontinentscontinentscoun

tries

Page 19: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

How does it work?

planetnew

planetosmosis

OSM

east

west

newplanet

osm-history-splitterosm-history-splitterosm-history-splittercontinentscontinentscontinents

osm-history-splitterosm-history-splitterosm-history-splittercontinentscontinentscoun

triesBig ugly Perl script!

Page 20: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Polygon files WTF?

● maproom.psu.edu/dcw ← their fault!● format later adopted by Osmosis● convert anything to poly files

(svn.openstreetmap.org/applications/utils/osm-extract/polygons)

● planet-history-splitter can use .osm files

Page 21: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Shortcomings

● cascading production causes problems● complete relations● large setup effort even for seldom-

required extracts● could small extracts be generated on the

fly?● no layers; extra shapes for rendering

Page 22: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

If you want to do this yourself...

● do you need “lossless” exctracts?● do you need the whole planet?● …● Geofabrik code is available but not

polished

Page 23: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Alternatives

● extract.bbbike.org● Overpass API● wiki.openstreetmap.org/wiki/Planet.osm

Page 24: r e The Geofabrik Download Server S a o n w o D k b f o e ... · The Geofabrik Download Server What it does & how it works Frederik Ramm  SOTM-US, San Francisco

Ge

ofa

bri

k D

ow

nl o

ad S

erv

er

Thank you!

Questions?

Frederik Ramm <[email protected]>

SOTM-US, San Francisco08 June 2013