10/12/2010

AmericaView 2010 Fall Technical Meeting

Technical WMS

Dayne Broderson

Maddison WI October 12th, 2010

www.gina.alaska.edu

dayne@alaska.edu

Technical WMS services overview

AmericaView FTM 2010AlaskaView – Dayne Broderson

AlaskaView applications and architecture Data prep WMS overviewMapServerTile services Web mapping GISJAM

AlaskaView WMS

BDL – (best image layer)Blue Marble → Landsat → Highres

USGS ToposNOAA ChartsDEM HillshadeMODIS passes (near-realtime WMS)Many custom off by one layers

10/12/2010

The bdl_high_res layer contains data from a variety of sources and is intended to be viewed at scales from 25m per pixel to 1m per pixel. Images courtesy of Digital Globe, USDA-NRCS, USDA-FS, USGS, BLM, NPS, FAA, Alaska DNR, DMVA, DCED, DOT&PF

http://www.alaskamapped.org/data/wms-wfs-wcs-web-service-feeds

10/12/2010

NOAA ChartsThe Charts service is meant to provide a unified view of the RNC data, without the collar information. The service is designed to only display charts that are meaningful at the requested scale.

http://www.alaskamapped.org/data/gallery

10/12/2010

Shaded Relief NEDThis group has one layer which provides a shaped relief dataset derived from the National Elevation Dataset (NED).

10/12/2010

Realtime MODIS and AVHRRRealtime optical and infrared satellite data captured in Fairbanks and Barrow by UAF-GINA and NOAA-NESDIS FCDAS.

10/12/2010

Not just rasters

Gridded data sets such as DEMs can be served up in a very similar fashion

10/12/2010

2 arc second NEDStandard National Elevation Dataset (NED) for Alaska.

10/12/2010

ASTER GDEM – 1 arc second

10/12/2010

Using it for web applications

Alaska Statewide Digital Mapping Initiative's AlaskaMapped.org

http://browse.alaskamapped.orgNorth Slope Science Initiative's catalog

http://catalog.northslope.orgAlaskaViews user portals for MODIS and firepoint browsing

http://hub.gina.alaska.edu

10/12/2010

show-n-tell

Good place to see it all come together in one place: http://hub.gina.alaska.edusee latest MODIS passview firepoints

10/12/2010

WMS clients

Desktop clients, primarily ArcGIS and QGISServer clients

Alaska DNR's AlaskaMapperAlaska Ocean Observing SystemARMAPNRCS Soil Survey

10/12/2010

Behind the curtain (hardware/OS)

Open standards, open platform, optomized for users and clientsHardware

SuperMicro chassis+motherboardsJBOD RAID + FC RAID

Operating systemCentOS 5x / RHEL (Redhat Enterprise Linux)Trusted repositories (rpmforge+postgresql)

10/12/2010

Behind the curtain (software)

programming language: ruby and javascriptRails and extjs (sencha)

gdal, PROJ4, libgeos, mapserver, ...Postgresql + postgis for database and spatial engine… piles of supporting libraries and software

10/12/2010

Custom software

dans-gdal-toolsCustom tools developed to use in conjunction with GDAL - developed by Dan Stahlke at GINA

swathviewerCustom java application for browsing the data archive of GINA

many custom ruby, c, perl utilities to assist with data ingest and prep.Javascript libraries and REST APIs for simplifying web application development inside and outside of AlaskaView

WMS HA - High availability

Two primary servers, in separate machine rooms attached to their own independent mirror of the data.

High Availability - HA provided by using:two HA IPs shared between the serversDNS round robin between HA IPsHA IP failover

Allows us to bring down a WMS, do massive data update, and switch over to updated dataset w/o service interruption. secret sauce: Linux HA (heartbeat)

Tile Servers

Similar to WMS HA setup changed up to optimized for tile clients. A HA setup pointed at two load balancing proxy server

Apache → proxy → one of many instances of shiv

secret sauce: apache (and it's built in proxy capability)

Preparing Data

Some things to evaluate/figure out before shoving your data at a WMS

map product no data valuesmask/outline of valid dataprojection/datum of sourceresolution of source optimized projection/file-format/pyramiding of data

no-data | valid data

Knowing which pixels in a file are valid data pixels is very important for creating a layered WMS. What is the outline of your dataset? (mask)What are the correct 'nodata' value? (or values!)

For just looking at a single image no big deal - layering in a WMS having no-data pixels be opaque will block pixels from other layers you want - or having data pixels labeled as no-data will cause voids where lower layers can peek through (ugly artifacts) secret sauce: gdalinfo

gdalinfo - nodata

gdalinfo 55012391009071952112J0_gm.tif Driver: GTiff/GeoTIFF

Files: 55012391009071952112J0_gm.tifSize is 7450, 7390

----------(lots more info here)--------Band 1 Block=7450x1 Type=Byte, ColorInterp=Red

NoData Value=0

Band 2 Block=7450x1 Type=Byte, ColorInterp=Green

NoData Value=0

Band 3 Block=7450x1 Type=Byte, ColorInterp=Blue

NoData Value=0

Band 4 Block=7450x1 Type=Byte, ColorInterp=AlphNoData Value=0

gdalinfo - projection

PROJCS["NAD83 / Alaska zone 3", GEOGCS["NAD83", DATUM["North_American_Datum_1983", SPHEROID["GRS 1980",6378137,298.2572221010002, AUTHORITY["EPSG","7019"]], AUTHORITY["EPSG","6269"]], PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4269"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",54], PARAMETER["central_meridian",-146], PARAMETER["scale_factor",0.9999], PARAMETER["false_easting",1640416.666666667], PARAMETER["false_northing",0], UNIT["US survey foot",0.3048006096012192, AUTHORITY["EPSG","9003"]], AUTHORITY["EPSG","26933"]]

if only...

The gdalinfo trick only works if the stars align:image format selected allows for that sort of embedded metadatathe people generating the data used correct options allowing for the fields to be populated

Many data formats you have to look in an auxiliary metadata file. With the same issues as above.

Many older data products suffer from complete lack of required metadata -- Sherlock Holmes hat and pipe.

updating state: prepped -> create_images/data/scratch/tracker-prep/54542271009182120192J0Warping scene01/imagery.tif -> sdmi/54542271009182120192J0_aa.tif

running: >>> gdalwarp -dstnodata '0 0 0' -t_srs epsg:102006 -rb -co TILED=YES -co COMPRESS=LZW scene01/imagery.tif sdmi/54542271009182120192J0_aa.tif

Generating RGB sdmi/54542271009182120192J0_aa.tif -> sdmi/54542271009182120192J0_aa_rgb.tif

running: >>> gdal_translate -a_nodata '0 0 0' -b 1 -b 2 -b 3 -co TILED=YES -co COMPRESS=LZW sdmi/54542271009182120192J0_aa.tif sdmi/54542271009182120192J0_aa_rgb.tif

Generating pyramids for sdmi/54542271009182120192J0_aa_rgb.tif

running: >>> gdaladdo -r average sdmi/54542271009182120192J0_aa_rgb.tif 2 4 8 16 32 64 128 256 512 1024 2048 4096

Fetching metadataCreating tagger.ymlupdating state: create_tagger_yml -> prepping

What did we just see?

gdalwarp: Raw SPOT image rubber-sheeted info an Alaskan Albers projection in a 4 band geotif

gdal_translate: Converting the 4 band geotif to a RGB product in Alaskan Albers gdaladdo: Create internal pyramids/overviews for the RGB geotif (make it speedy for ArcGIS users and WMS).

running: >>> gdalwarp -dstnodata '0 0 0' -t_srs '+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +wktext +no_defs' -rb -co COMPRESS=DEFLATE scene01/imagery.tif /home/ingest/data/gm/54542271009182120192J0_gm.tif

generating cornersrunning: >>> gdal_list_corners /home/ingest/data/gm/54542271009182120192J0_gm.tif > 54542271009182120192J0_gm.yml Generating RGB /home/ingest/data/gmrunning: >>> gdal_translate -a_nodata '0 0 0' -b 1 -b 2 -b 3 /home/ingest/data/gm/54542271009182120192J0_gm.tif /home/ingest/data/gm/54542271009182120192J0_gm_rgb.tif

creating preview imagerunning: >>> tifftopnm /home/ingest/data/gm/54542271009182120192J0_gm_rgb.tif | pamscale -xyfit 1300 1300 | pnmtopng -transparent=black > /home/ingest/data/preview/54542271009182120192J0_gm_rgb.png

What was that?

That was doing a similar set of steps only this time the goal was a Google Mercator geotif.

Also saw a handy tool called gdal_list_corners being used. Like gdalinfo only output format is YAML making it easily parse-able by scripts.

Lastly we saw the conversion of the GM RGB to a resized 1300x1300 PNG using tifftopnm | pamscale | pnmtopng Why? http://photos.brute.gina.alaska.edu

-Data-

Know your data and it's attributes

Have that information in an easy to use format (human and scripts). -- YAML much easier than XML

Get confused and bang around on the command line but keep track of the commands and script them up for large job.

WMS features

Sam hopefully did this already

on the fly re-projection (server side)customized layers/mosaicsprovide access to vast archive w/o downloadupdate 'best available' and be able to update what that meansprovides a great back-end to tile based work

WMS limitations

Bandwidth and latency between server and clientsTimeouts by clients ESRI 'features'

no user control over image formatinternal timeouts w/o errorsuncontrolled caching by clientbad defaults for behavior re-projection and re-sampling algorithms

Above add up to limited large printing capability (plotting)Client side proxies/network setup can cause a nasty headache when debugging

WMS Server Options

Open SourceGeoServer - http://geoserver.org

java based - MapServer

ProprietaryESRI ArcGIS Server, ArcIMSOracle, LizardTech, ....GeoWebPublisher, ObjectFX, ...

MapServer

http://mapserver.org/

University of Minnesota "The MapServer Project Steering Committee (PSC), sanctioned by OSGeo, manages and administers the project which is maintained, improved, and supported by developers from around the world. MapServer is not a full-featured GIS system, nor does it aspire to be."

secret sauce: yum install mapserver (not actually but you could to get started)

MapServer Architecure

Ingredients for a WMS:

Pile of raster mapsPile of vectors (not covered here)Apache serverMapServerA CGI/URL Map file (organization/configuration for a WMS)

above shamelessly stolen/borrowed from mapserver's introduction documentation

What did we see there?

Data sources:raster and vector data on local machineWFS, WCS, WMS - remote servers

Mapfile configuration

list of what data sources to include Map server CGI (MapScript application ignored)

Apache server

Output - the WMS/WCS/WFS

Dirty hands - data store

bishop@salt:~/mapserv/data$ ls

AERIAL.UAF.MAY2007.UAF_AREA_2_12AERIAL.UAF.MAY2007.UAF_AREA_2_13bmSPOT.DNR.INTERIOR.TANANA_VALLEY

Mapfile - real world examples

BDL example

pile of examples (restricted URL)

mapfile - phase 1

MAP NAME "AView Example" # SIZE 600 400 # pixel width and height of the map EXTENT -180 -90 180 90 # geographic IMAGECOLOR 128 128 255 # default image background colorEND

map file - data layer

LAYER NAME "Blue_Marble" STATUS on DATA "/home/bishop/mapserv/data/bm/blue_marble_bright.tif" TYPE RASTER PROJECTION "init=epsg:102006" END END

the cgi

The following are some example CGI's used by AlaskaView:

simple real world example

the BDL cgi (more magic)

my aview example

#!/bin/sh MAPFILE="/home/bishop/mapserv/aview.map" MAPSERV="/usr/lib/cgi-bin/mapserv" if [ "${REQUEST_METHOD}" = "GET" ]; then if [ -z "${QUERY_STRING}" ]; then QUERY_STRING="SERVICE=WMS&map=${MAPFILE}" else QUERY_STRING="SERVICE=WMS&map=${MAPFILE}& STYLES=&${QUERY_STRING}" fi exec ${MAPSERV} else echo "Sorry, I only understand GET requests." fi exit 1

aview cgi

REQUEST=GetCapabilities

curl http://localhost/cgi-bin/aview?REQUEST=GetCapabilities

--Provide clients an XML description of what is offered.

File formats supportedLayers available...

curl http://wms.alaskamapped.org/extras?REQUEST=GetCapabilities

msProcessProjection(): Projection library error. no options found in &#39;init&#39; file root@salt:/usr/share/proj# grep 102006 esri<102006> +proj=aea +lat_1=55 +lat_2=65 +lat_0=50 +lon_0=-154 +x_0=0 +y_0=0 +ellps=GRS80 +datum=NAD83 +units=m no_defs <>

gah - that needs to be in /usr/share/proj/epsg grep 102006 esri >> epsg

Optimizing WMS

Watch how your clients are using the WMS

Avoid re-projection on the flyUse TIFFS with overviews/pyramidsminimize load-up time of the cgi/mapserver environmentuse 'ab' to benchmark your changesenable cachingMany different map files (lean/optimized) are better than one massive map file

optimizing rasters faq @ mapserver.org

Expert words

General tips for performance all cases:First and foremost is hardware. An extra GB of RAM will give your map performance increases beyond anything you’re likely to achieve by tweaking your data. With the price of RAM these days, it’s cheap and easy to speed up every map with one inexpensive upgrade.Use the scientific method. Change one thing at a time, and see what effect it had. Try disabling all layers and enabling them one at a time until you discover which layer is being problematic.Use shp2img program to time your results. This runs from the command line and draws an image of your entire map. Since it’s run from the command line, it is immune to net lag and will give more consistent measurements that your web browser.

http://mapserver.org/faq.html

Tile Generators

Tile generators take tile requests and map them to WMS requests

Maps a request for a tile: http://www.foo.com/tilesets/x/y/z

To a WMS request:http://www.foo.com/wms?layers=mylayer&styles=&service=WMS&format=image...

Why? Allows you to use a WMS to provide fast and efficient tiles to web mapping frameworks

Tile Generators

A Tile Generator shouldDo cachingProvide a way to "seed" the cacheGenerate several tiles from a single WMS request

ExtrasKML and ESRI Rest Style output

Tile Generator Options

Tile Cache http://tilecache.org/

ShivESRI Arc ServerProbably lots of other good options..

Shiv - overview

Written in Ruby Built by Jay Cable for AlaskaView/GINAProven to be very flexible and stable

shiv instances with 100+ day uptimeability to add/remove/revamp what tile layers are available w/o restarting of services can handle all major tile schemes

Shiv

featuresorganized tile cache structure

Tiles reside on disk in a tree structure pre-generates tiles optimizes requests to WMS

requests a large image that it chops upFlexible tile support (power of 2)

Such as Google Maps Generates the KML for Google Earth Speaks ESRI style REST

ExperimentalNot a recommended direction to go in

An Example

Shiv receives a request for the Google Maps style tile at 0,0,1 (x,yz) Having no pre-generated tile to serve up Shiv requests the following image from the WMS

An Example

Taking that large image Shiv generates these tiles..

An Example

then returns the tile, 0,0,1:

Nice - but how to use it?

GINA.mapTypes = { SDMI_BDL: new google.maps.ImageMapType({ name: 'GINA/SDMI Best Data Layer', alt: 'GINA/SDMI Best Data Layer', minZoom: 1, maxZoom: 21, tileSize: new google.maps.Size(256, 256), isPng: true, getTileUrl: function(a, z) { var tiles = 1 << z, X = (a.x % tiles); if(X < 0) { X += tiles; } return 'http://swmha.gina.alaska.edu/tilesrv/bdl/tile/' + X + '/' + a.y + '/' + z + '.png'; } }), ...

Tile URL

That was kinda hard to read - broken apart:

http://swmha.gina.alaska.edu /tilesrv/bdl/tile/' + X + '/' + a.y + '/' + z + '.png';

/tilesrv/bdl/tile/0/0/1.png

/tilesrv/bdl/tile/1/3/4.png

/tilesrv/bdl/tile/341/1161/12.png

...

Time to switch to the 'make a map' portion of this show.

Prepare yourself a text editor and have firefox ready to go to some URLs

http://gisjam.gina.alaska.edu/

gisjam slides