using openstreetmap data - british computer society · osm has a web service api to allow access to...
TRANSCRIPT
What is OpenStreetMap?
OpenStreetMap[1] is a set of mapping data covering the whole worldLiberally licenced (CC-by-SA)Anyone can contributeRendered maps available, but the data is the important thing!See http://www.openstreetmap.org
Why OpenStreetMap?
Existing mapping data was not free (as in beer or speech)
Licencing fees necessary to use maps onlineGenerally only the rendered maps were available, not the underlying data
Data available to business, but at very high costThis restricted how you can use the data
It's the data!
OSM has a default “slippy” map but access to the raw data is what makes OSM
Developers can use this data in their own applications
OSM has a web service API to allow access to the raw dataRaw data also downloadable as XML files
Contributing
Survey roads, footpaths and POIs with a GPS deviceUse freely-available data sources (Ordnance Survey out-of-copyright/OpenData [2], Bing aerial imagery)Use an OSM editor (JOSM [3], Potlatch [4]) to draw highways on top of GPS traces or imagery
Overview of in-the-field mapping procedure
Survey using a GPS device (e.g. smartphone, Garmin GPS unit)Load GPS track and waypoints into an editor (web or standalone)Download existing OSM data into the editorDraw new roads, paths, points of interest in OSM editor using your GPS track/waypointsUpload changes to OSM
Editing the dataJOSM (josm.openstreetmap.de)
Standalone editor, written in JavaPowerful and flexibleCommunicates with API over HTTP
Potlatch (http://wiki.openstreetmap.org/wiki/Potlatch_2)
Online Flash EditorEasy to use
iD (ideditor.com)HTML5/JavaScript editor, in early development
Other sources of OSM data
In addition to direct surveys, the following sources can be used to add data to OSM:
Aerial imagery (Bing; special agreement)Out-of-copyright historical mapping (e.g. OS maps from 50+ years ago, i.e. 1962 or before)Ordnance Survey OpenData or other liberally-licenced official data sources
The editors can load in these data sources as a backdrop to the OSM data
The OpenStreetMap APIAPI to allow developers to make use of OSM dataREST-compliant web service providing data in OSM's own XML formatAllows creation, retrieval and modification of map dataAvailable athttp://www.openstreetmap.org/api/0.6/(operation)
Examples of operations on following slides
Nodes and Ways
OSM data consists of nodes and waysA node represents a point feature
e.g. a point of interest, or a constituent point of a line feature such as a road
A way represents a line or polygon featuree.g. a road, a stream, a wood, a lakeWays consist of two or more nodes
OpenStreetMap API -Examples
Retreive map data in a given bounding box (OSM XML format)/map?bbox=-1.5,50.9,-1.4,51.0
Retrieve a given way (here, way with ID 223)/way/223 (GET request, retrieves OSM XML)
Modify a way (here, way with ID 223)/way/223 (PUT request, send OSM XML to URL)
OpenStreetMap XML format<osm version="0.6"><osm version="0.6">
<node lat="50.9" lon="<node lat="50.9" lon="--1.4" id="12345">1.4" id="12345">
<tag k="place" v="city"/><tag k="place" v="city"/>
<tag k="name" v="Southampton"/><tag k="name" v="Southampton"/>
</node></node>
<way id="223"><way id="223">
<nd id="1234" /><nd id="1234" />
<nd id="1235" /><nd id="1235" />
<tag k="highway" v="motorway" /><tag k="highway" v="motorway" />
<tag k="ref" v="M27" /><tag k="ref" v="M27" />
</way></way>
......
OpenStreetMap XML -Explanation
The <node> tag represents a nodeIt has an ID, latitude and longitude attributes and a series of free-form tags describing the point
The <way> tag represents a wayA way is made up of constituent nodesThe XML consists of a series of node references, referenced by node ID (the <nd> tag) together with free-form tags describing the way
OSM tagging
Properties of map objects in OSM are described by a series of tagsEach tag has a key and valueThe key is the general class of property, the value is the actual value for that property
Examples of tags for linear ways
highway: the class of road (motorway, primary, secondary etc)maxspeed: maximum speed in km/href: the road reference numberoneway: is it a one way road? (boolean)
Examples of tags for nodes
naturalIf it's a natural feature, indicates the type (hill summit, etc)
man_madeIf it's a man made feature, indicates the type (water tower, mast etc)
amenityIf it's an amenity (e.g. a pub, restaurant, recycling centre etc) indicates the type
OSM way tagging example
A main road could be described by the following tags:
highway=primaryIndicates it's a “primary” class road (A road in UK)
name=High Streetref=A286
The road numbermaxspeed=64
Maximum speed in km/h
OSM node tagging examples
Hill / Mountain:natural=peak; name=Snowdon
Pub:amenity=pub; name=The Crown
Place:place=city; name=Southampton
Communications mast:man_made=mast
Examples of tags for areas
Areas (polygons) are treated as ways (which just so happen to be closed)e.g. landuse for managed land
landuse=forest or landuse=industrial or landuse=residential
or natural for natural land covernatural=wood or natural=water or natural=heath
OSM “planet” filesRequesting large amounts of data from API can place undue load on the OSM serverAPI is really designed for adding and editing data, not retrievingSo OSM also makes its data available as downloadable “planet” files in OSM XML format, containing OSM data for the entire planetWhole planet too large for most to use directly, so country or regional extracts available
e.g. Geofabrik (http://download.geofabrik.de)
Extracting OSM data with osmosis
Even with a country extract, chances are you are only interested in some of the dataFor example, a walkers' map might only need data of interest to walkersThe osmosis tool (wiki.openstreetmap.org/wiki/Osmosis) allows you to extract only the data you need
osmosis
Osmosis allows you to (amongst other things):Extract data by bounding box or polygonExtract all nodes or ways containing a given key, or key/value pairWrite OSM data to a MySQL databaseRead OSM data from a database and write to an OSM data file
osmosis
Examples of osmosis commands:
osmosis --rx file=uk.osm -–bounding-box top=52 bottom=50 left=-2 right=1 –-wx file=southern_england.osm
This reads in uk.osm, extracts a bounding box defined by 50N, 52N, 2W and 1E, saving the results in southern_england.osm
osmosis –-rx file=uk.osm -–tf reject-ways –-tf accept-nodes place=* --wx file=ukplaces.osm
This reads in uk.osm, rejects all ways and accepts only nodes with a place tag, writing the result to ukplaces.osm
General principles of osmosis
Has a set of commands specified as command-line options, e.g. --rx (read XML), --tf (tag filter)Commands typically filter the input streamSo the following command:
osmosis –-rx file=uk.osm –-tf reject-ways –-tf accept-nodes place=* --tf accept-nodes amenity=* --wx file=output.osm
will only give nodes with both a place tag and an amenity tagIf we want nodes with either a place tag or an amenity tag, we must use named pipes and stream merging, see the osmosis page for details
Overpass API
An alternative to downloading the OSM data as a flat fileThe Overpass API is a third-party live web API which allows OSM data (a local copy of the OSM database) to be queried and returned as XMLUses an XML-based query languageSee http://wiki.openstreetmap.org/wiki/Overpass_API
Setting up an OSM database
You may wish to setup your own copy of the OSM databasee.g. for rendering your own maps or developing a web service of some kindTypically, a PostgreSQL / PostGIS database is used for thisOSM data is imported into two tables planet_osm_point and planet_osm_line
Setting up PostgreSQL / PostGIS
PostgreSQL and its GIS-specific extensions, PostGIS, normally available as packages for Linux distributionsInstallation fairly straightforward on current Ubuntu or Debian distributionsDetailed installation instructions available at:
wiki.openstreetmap.org/index.php/Mapnik/PostGIS
Importing OSM dataHaving set up the PostGIS database, OSM data must be imported into itA custom tool, osm2pgsql, is used for thisThis can be downloaded from OSM's github repositoryOsm2pgsql allows us to control which features are imported (by tag), minimising the size of the databaseSee the osm2pgsql wiki page:
http://wiki.openstreetmap.org.index.php/Osm2pgsql
Map rendering
OK, you've set up a database containing OSM data, but how are you going to generate your own maps?Two broad approaches:
Server side rendering (e.g. Mapnik)Client side rendering (e.g kothic-js)
Mapnik: server-side rendering
Mapnik (mapnik.org) is a C++ library (with Python wrappers) for rendering high-quality maps from geographical dataMapnik is able to read data from (amongst other data sources) PostGIS databases and uses an XML style file to control the look-and-feel of the final mapDevelopers create programs in C++ or Python to render data using Mapnik from the PostGIS database
Helper tools for Mapnik
Mapnik is powerful but can be tricky to setup and useSo tools have been developed to ease creation of Mapnik mapsOne example is TileMill (mapbox.com/tilemill)
An easily-installable package which allows you to design and render maps locallyUses a locally-running webserver
Creating web maps
OK, we've got the maps rendered now, but what about creating a Google-style “slippy map”?JavaScript mapping libraries are used here
Leaflet: www.leafletjs.comOpenLayers: www.openlayers.org
You need to ensure you generate tiles in the “Google XYZ” format
Brief intro to Google XYZ
The tiling system used by most web mappingMap tiles (projected in “Spherical Mercator”) described by an x,y,z coordinate schemeThe x and y coordinates represent the tile's position in the world (x=0 is west; y=0 is north)The z coordinate represents the zoom level
Google XYZ zoom levels
Zoom level 0 contains the whole world on one 256x256 pixel tileZoom level 1 zooms in by a factor of two, so the whole world is contained by four 256x256 pixel tiles
… and so on for zoom levels 2, 3, etcSee www.free-map.org.uk/course/ewt/webmapping.xhtml
Google XYZ zoom levels
Zoom level 0 : whole world as 256x256 tile
Google XYZ zoom levelsZoom level 1: 4 tiles (2 rows, 2 cols) for world
Client-side rendering
Server-side rendering can be computationally expensive, and requires large storage space (cache) once renderedTherefore not always the optimal solutionClient-side, in-browser HTML5 canvas rendering possible with kothic-js (kothic.org/js)
kothic-jsJavaScript library developed as a port of the Python renderer kothicRenders GeoJSON data client-side using HTML5 Canvas
GeoJSON is a standard JSON-based format for representing geographical data, see geojson.org
Server-side scripts must be written to generate the GeoJSON from a database using the
“Google” XYZ tile systemA Python implementation is available, produced by the kothic developers
Other uses of OSM data
Location-based smartphone appsApps can include vector maps Apps can find nearby points of interest downloaded from OSM
Routing softwareOSM data can be converted to binary formats optimised for routing
Example site: FreemapFreemap (www.free-map.org.uk) is an example of a third-party site making use of OSM dataOSM site geared for UK walkers, showing rights-of-wayOSM data for parts of the UK loaded into PostGIS database with osm2pgsqlServer-side scripts generate GeoJSON from the databasekothic-js is used client-side to render the dataOS LandForm-PANORAMA data[5] used for contours
Example mobile app: OpenTrail
OpenTrail (www.free-map.org.uk/downloads/android/opentrail.apk) is Freemap's Android appIt shows vector maps generated from OSM data using the Mapsforge library (code.google.com/p/mapsforge)It also allows users to find nearby POIs
These are delivered from Freemap's web service, which is also used to generate the GeoJSON for Freemap itself
Novel uses of OSM data: example
Augmented reality is an interesting concept in mobile development
The general idea is that the real world (e.g. camera feed) is overlaid with virtual objects (e.g. POIs)
Hikar (see https://github.com/nickw1/Freemap/tree/master/java/hikar/) is a “researchware” app which overlays OSM data (provided by Freemap's API) on the phone's camera feed… and in doing so potentially offers a novel approach to navigation for walkers
References[1] OpenStreetMap, 2011 "OpenStreetMap" (online) Available: www.openstreetmap.org, last accessed 6/6/11
[2] Ordnance Survey, 2011 "OS OpenData" (Online) Available http://www.ordnancesurvey.co.uk/oswebsite/products/os-opendata.html, last accessed 6/6/11
[3] OpenStreetMap, 2011 "JOSM" (online) Available: josm.openstreetmap.de, last accessed 6/6/11
[4] OpenStreetMap, 2011 "Potlatch 2" (online) Available: wiki.openstreetmap.org/wiki/Potlatch_2, last accessed 6/6/11
[5] Ordnance Survey, 2011 "Land-Form PANORAMA - small-scale height data of Great Britain",(online) Available: http://www.ordnancesurvey.co.uk/oswebsite/products/land-form-panorama/index.html, last accessed 6/6/11
Thanks for listening
Any questions?