meeting the inspire mandate with fme
DESCRIPTION
Discover how you too can solve common INSPIRE-related data challenges using FME. You’ll see how to easily perform schema mapping and meet INSPIRE’s precise data model requirements. You’ll also learn about the various FME-based INSPIRE solutions available and how people are using them. Finally, the project leader of Luxembourg’s public Geoportal (Jeff Konnen) will share how they’re using FME to help reach their INSPIRE objectives.TRANSCRIPT
FME INSPIRE Webinar:Implementation in Luxembourg
July 2012
Jeff Konnen, Luxembourg National GeoportalDon Murray, Safe SoftwareDean Hintz, Safe Software
Agenda
Harmonization principles for INSPIRE SDIs Development stages and FME examples:
assembly, transformation, validation and publication
con terra’s INSPIRE Solution Pack for FME Solutions for Luxembourg by Jeff Konnen FME tools to support INSPIRE INSPIRE trends Consuming INSPIRE
Reading Building value added services
INSPIRE - Goals
Assist EU environmental management
Extend Member States’ SDIs using: Common data model Open standards
INSPIRE SDI should: Combine spatial data from different sources Share spatial data between public authorities
INSPIRE - Challenge
You want to meet INSPIRE data provision requirements, but your data is organized rather differently
?
Poll Question
What INSPIRE efforts have you worked on to date?
?
What we do …
We build data bridges
Poll Question
How long have you been using FME?
Harmonization Core Concepts
Harmonization: implied INSPIRE requirement.
Disparate sources must be mapped to a common destination data model to support integration
Core to the harmonization workflow is a process called schema mapping.
Delivered by services based on open standards
INSPIRE Harmonization Principles
Typical stages:1. Evaluation2. Assembly3. Transformation4. Validation5. Publication
Based on the Spatial ETL concept (Extract, Transform and Load), as applied to INSPIRE SDI’s
Evaluation
Assess destination schema and data requirements
Assess source datasets and schema Consider fundamental differences in
representation, resolution Closely inspect actual representative datasets
Data Assembly
Assess the diversity of source data types: vector, raster, CAD, GIS, database, text, XML, web,
3D, sensor and non-spatial Review format and semantic translation needs Decide how to perform necessary joins
ID joins, spatial relates, nearest neighbor, one to many relationships
Goal is to build a data structure that corresponds with INSPIRE requirements
Translation
FME supports reading from a wide array of data formats and types 275+ formats, with more added each year CAD, GIS, XML, raster, database, non-spatial, 3D
Lithuania Geographic Information Infrastructure - Highlights
Wide range of format support required by member agencies
Solution combines FME, ArcGIS and con terra tools
Conversion rules key to harmonization process
Centralized workflow and common data model simplified source to destination mapping
FME Server supports sharing for LGII partners, business, education and research institutions, NGOs
LGII provides real value with broader access to seamless data via a common INSPIRE schema
LGII – Lithuania Geospatial Portal Translation Support
File Formats: AutoCAD DXF / DWG ESRI Personal Geodatabase Shape MapInfo TAB Microstation Design V8 GML GIF / PNG JPEG TIFF ERDAS IMG ECW
Coordinate systems: LKS94 WGS84 ETRS89 1963 Pulkovo 1942 UTM (34, 35 zones)
Other data sources: WFS SDE Files
LGII Geoportal Central System Components
FME Server Data conversion for download Transformation and schema mapping between data schemes,
data models and formats
con terra’s SDI Suite Content and user management E-Commerce and accounting Reporting
ArcGIS Server WxS services (WMS, WFS, WCS) Catalog services (CAT) Metadata harvesting (CSW, WAF) Spatial data editing, redlining functionality
Data Transformation - Schema
Reshape source data to match required destination schema
Schema mapping feature type attribute name new attribute creation code lists conditional value mappings
Schema Mapping in FME
Feature Type Mapping in FME Workbench
Attribute Mapping in FME Workbench
Schema Mapping in FME
Value Mapping
FME Data Model Restructuring: Attribute Names & Values
FME SchemaMapper: INSPIRE geographic names
Name mapping
Name & value mapping
FME Workspace
INSPIRE Pilot:Swedish Protected Sites
Swedish EPA project with Metria of Sweden Read from 3 different data sources (pilot):
Swedish Protected Areas Helsinki data commission (Helcom) European Natura2000 habitats
Perform required joins, generate IDs Transform to INSPIRE schema Load INSPIRE-like staging database (PostGIS) Publish INSPIRE web services based on staging
database
Helsinki Commission Source Data
Results in Staging Database
* Data provided by Lantmäteriet, the Swedish mapping, cadastral and land registration authority www.lantmateriet.se , and Metria, Sweden www.metria.se.
Protected Sites Schema Mapping: NVR to INSPIRE
FilterAttribute
Filter Attribute Value
SourceAttribute Field_Inspire
Destination AttributeValue DestinationAttribute
FullINSPIRE_APPLICATIONSCHEMA
DID INSPIRE_LOCALID
SE INSPIRE_NAMESPACEVALID_FROM_DATE
INSPIRE_ LEGALFOUNDATIONDATE
DECISIONDATE INSPIRE_DATE
creationINSPIRE_DATETYPE
IUNC_CATEGORY0
Empty value INSPIRE_DESIGNATION
IUNC_CATEGORYIa strictNatureReserve
INSPIRE_DESIGNATION
IUNC_CATEGORYIb wildernessArea
INSPIRE_DESIGNATION
IUNC_CATEGORYII nationalPark
INSPIRE_DESIGNATION
IUNC_CATEGORYIII naturalMonument
INSPIRE_DESIGNATION
IUNC_CATEGORY null Empty value INSPIRE_DESIGNATION
OBJECTNAME INSPIRE_SITENAME
PROTECTIONTYPE 1 natureConservation INSPIRE_PROTECTIONCLASSIFICATION
PROTECTIONTYPE 2 natureConservation INSPIRE_PROTECTIONCLASSIFICATION
Transformation: Geometry
Non-spatial to spatial Geometry extraction (spatial to GML) Representation transform: CAD drawing lines with
labels to GIS polygonal features with attributes Coordinate System Reprojection (ED50 to ETRF89) Simple to complex geometry
Source point and polygon data to multiple geometric representations (city as point / area, river as line / area)
Generalization and interpolation Highly granular national and regional datasets often
require thinning to be usable on pan-European scales
Validation
INSPIRE schema validation (xsds) Data integrity
Unique IDs Geometric integrity (closed polygons) Null values (nullable?) Valid values: ranges and domain codes Data gaps Bounds Network integrity
Custom validity rules specific to domain Validation automation via FME Server upload
Ensure data quality throughout the data transformation process.
Swedish Transportation Administration: Validation
System supports propagation of municipal and regional road data to national dataset
Data model transformation and QC to translate 2.5 million road links into NVDB
Transform between NVDB and INSPIRE Workflows automated by FME and FME Server Validation key to support upload services
Publication
Produce INSPIRE compliant GML Provide discovery, view or download services, for
WxS, GML and other desired formats Publish with FME Server or integrate with your
geo web server of choice: ArcServer (ArcGIS for INSPIRE) Deegree GeoServer MapServer
Spatial Data Services
Publication
How to make data most accessible? Support for mandated open standards and OGC
services Support for defacto industry standards,
proprietary and legacy agency systems Possibilities for Invoke Spatial Data Services as a
bridge accepts user request queries required INSPIRE OGC services provides desired web service (KML, PDF)
Swedish Transportation Administration:
Invoked Services
Supports estimation of new road costs related to archeological and protected sites
Site potential using terrain model and historic coast lines
WFS Sources: The Swedish Transport Administration roads Swedish EPA Protected sites Swedish National Heritage Board
Archaeological findings Result: PDF showing site potential
Format translation Schema mapping String and list manipulation Data validation Database load and extract XML,GML,WFS: reading, validation, publication Web services: WFS, WMS, integration with others Metadata support Enterprise services with FME Server
FME Tools for INSPIRE
Poll Question
Specifically related to INSPIRE, in what areas have you used FME?
Harmonization Stages: Geographic Names Demo
Geographic Names: GML Result
Other INSPIRE Examples
Finnish National Land Survey GML to Shape conversion Data upload and validation WPS test pilot – FME Server / Deegree integration
Technical University Of Munich XMI mapping of UML schema transformation rules Maps between AAA and INSPIRE data models Uses FME to perform schema mapping based on XMI
Nature SDI: Datasiel, Liguria Region, Italy FME data harmonization -> loads staging database Generates protected sites GML for WFS
Croatian NSDI: Geoportal By State Geodetic Administration (SGA) Central register of base map, ortho, spatial units, cadastre FME used primarily in data migration and updates
Example INSPIRE Solutionsby FME Partners
INSPIRE Solution Pack from con terra (> 20 implementations across Europe)
Metria, Sweden (Protected Areas Pilot; Swedish Department of Transportation)
HNIT Lithuania (LGII) Technical University of Munich (multiple
INSPIRE projects) AED Sicad (NAS to INSPIRE conversions) Spatialworld, Finland (National Land Survey)
con terra – FME INSPIRE Solution Pack Highlights
Simplify data transformation and schema mapping for ArcGIS for INSPIRE
Predefined FME Workbench templates help jump start the data migration and Harmonization process
Connect your existing databases to the standardized ESRI INSPIRE Geodatabase
Automate data load and update processes
Quality assurance and metadata
Publish once and use many times
Internal information/storage systems
Consumers
European SDI
INSPIRE fusionCenter – Data models and web services
transformation & integration
2. Integration into business processes and
transformation of data to INSPIRE
2. Integration into business processes and
transformation of data to INSPIRE
1. Data Models for spatial data sets compliant to
INSPIRE data specifications (COTS product)
1. Data Models for spatial data sets compliant to
INSPIRE data specifications (COTS product)
3. Access via INSPIRE network services meeting the Quality-
of-Service requirements (COTS product)
3. Access via INSPIRE network services meeting the Quality-
of-Service requirements (COTS product)
con terra – FME INSPIRE Solution Pack: 3 Key Tasks
ETL Workflow descriptionimport source data
create destination schema
import destination schema
Load Data INSPIRE GDB
automatedautomated
automatedautomated
automatedautomated
FME INSPIRE Solution Pack
schema mapping
FME data import process
con terra - FME INSPIRE Solution Pack: Schema Mapping
Additional INSPIRE information Tutorial workspace (complete sample mapping [AdminUnits])
Additional HTML Workbench Help (description of INSPIRE GDB data model)
Direct access to the INSPIRE data specification (link to specific themes)
Additional functionality Template workspaces (destination schema and predefined workspaces)
Destination data schema for all Annex 1 themes (INSPIRE GDB of ArcGIS for
INSPIRE http://resources.arcgis.com/de/content//arcgis-inspire/1.0/about )
More than 100 additional INSPIRE transformers
INSPIRE specific value and attribute mapping (voidable values)
transformer for recurrent tasks (ID management, lifespan setter)
Components of the FME INSPIRE Solution Pack
Solution Pack Template Workspaces
one template Workspace per Annex I theme predefined datasets (INSPIRE GDB) predefined attribute-creation and settings predefined attribute-mapping integrated description for mapping file
completion
one template Workspace per Annex I theme predefined datasets (INSPIRE GDB) predefined attribute-creation and settings predefined attribute-mapping integrated description for mapping file
completion
INSPIRE Transformers (Attribute and Value Mapping)
Automated filling of obligated attribute legalStatus legalStatus_void
Possible values for obligated attributes (if not void) agreed notAgreed
Predefined void value reasons (if void) 0 = no reason given 1 = reason: unknown 2 = reason: unpopulated
AttributeRenamer functionality usable if attribute already
existing
FME – the pillar of an INSPIRE Infrastructure
Jeff KonnenAdministration du Cadastre et de la Topographie
Service GéoportailGrand-Duché de Luxembourg
Or ...
Thanks to FME, the red lion is juggling with INSPIRE
About the ACT
ACT : Administration du Cadastre et de la Topographie Administration depending on the Ministry of Finance 130 people Responsible for:
Cadastral documentation Administrative boundaries and frontiers Delimitation of properties Management of property-related and topographic databases Management of the national address register Topographic documentation / maps Management of geodetic networks Management and hosting of the national Geoportal
About the ACT: website
About the geoportal
Central SDI platform for the Luxembourgish State Visualization of national geodata
General viewer (http://map.geoportal.lu ) Thematic viewers (eg http://tourisme.geoportal.lu) Mobile Viewer (http://m.geoportal.lu )
Geo-Applications Routing (http://routing.geoportal.lu ) Distance map Coordinate conversion
Shop Maps Cadastral extracts Geodata etc
About Geoportal.lu
Map.geoportal.lu
Map.geoportal.lu with other sauces
map.geoportal.lu
About 130 different layers Very fast (everything is tiled and precalculated) Easy to use (Google Style) Based on Mapfish Technology Single Search Field searching in 1Mio different
entries (Addresses, Parcels, Localities, Rivers etc) Stats: 1300+ single users per day, tendency
growing API
About me
Jeff Konnen Project Leader at the ACT / Geoportal Long time FME User FME Professional & Trainer [email protected]
Geoportal vs INSPIRE
ACT and especially the Geoportal crew have been designated as INSPIRE Point of Contact (NPOC) for Luxembourg
The Geoportal crew has to manage metadata and Network Services for the Luxembourgish Node of INSPIRE
Since May 2010, Luxembourg is reporting and monitoring about its Network Services and since November 2011, the Discovery and View Services for Annex I & II Themes are available online (http://inspire.geoportal.lu/geoportal)
Since June 2012, a first set of Download services is available (IOC)
INSPIRE Portal
Geoportal vs INSPIRE
The Geoportal crew of Luxembourg consists of 5 people 1 Manager 1 Project Leader 2 Developers 1 IT Manager
There are not a lot ressources for INSPIRE A complete solution for serving INSPIRE relevant data
had to be found ArcGIS for INSPIRE
ArcGIS for INSPIRE
INSPIRE Datamodel for the Geodatabase We didn’t have to implement the INSPIRE datamodels by
ourselves INSPIRE Geoportal
Management of the INSPIRE Metadata and the INSPIRE Discovery Service
Caveat: Our main Metadata management is multilingual (4 languages). By using INSPIRE Geoportal we decided to go English only for the INSPIRE relevant metadata
INSPIRE Add-on for ArcGIS Desktop Create maps for INSPIRE based on the INSPIRE datamodel
INSPIRE Add-on for ArcGIS Server Creates INSPIRE View Services and Download Services with one
click
ArcGIS for INSPIRE .
ArcGIS for INSPIRE ..
What ArcGIS for INSPIRE does not do: Convert your data into the INSPIRE Datamodel That’s where we need FME
FME is the tool used to convert data from the production databases into the INSPIRE datamodel
The con terra ISP (INSPIRE Solution Pack) helps to understand the technical aspects of the mapping that has to be done and to streamline your efforts
The semantical mapping of the data has to be done by the thematic experts, no solution will do it for you
FME and ISP
Using FME and ISP, a set of scripts has been created These scripts can be executed regularly to fill the
INSPIRE Datamodel The INSPIRE View Services are based on the
content of the INSPIRE Datamodel Loading takes about 24 hours Executed once a year or more often
Themes: Annex I: AU,AD,CP,GN,TN,(PS),HY as INSPIRE View
Services Annex II: OI,EL,LC,GE as WMS 1.3.0 as A4I not ready
yet
Example: Mapping of the Annex I Theme AD
FME and INSPIRE : Lessons learned
FME is the right tool to transform your data ISP helps you to streamline your efforts The complicated part is not FME, it’s the semantic mapping (!)
Plan a lot of time for this part A lot of discussions / compromises to find
FME and INSPIRE : Next steps
Activate some more download services for Annex I (and II) until the end of the year Only « one click » but a lot of questions to clarify
(pricing) Do the mapping for Annexes II and III
Wait for the datamodels to be final Activate their view and download services
Permanently : Keep up with new data
Uses of FME Server: Shop
Uses of FME Server: Shop
Shop: Automatic Extraction of orders
Uses of FME Server: Shop (Orchestration with Syncserv)
Other Uses of FME
Weekly syncronization of production data in the geoportal databases STAR (CAD) -> ORACLE ORACLE -> PostGIS
Daily conversion of measurement files (STAR) to DWG
Quality control QC of Administrative Unit data
Statistics Count features issued by the different WFS services
Other uses of FME (Server)..
Online coordinate conversion (CSV File conversion from WGS84 -> LUREF (EPSG:2169) using FME Server
Other uses of FME (Server)..
Preview of DWG in files in the shop before ordering Streaming service doing a generic transformation
from DWG to PNG
Other uses of FME ..
Weekly generation of an address point database based on the centerpoints of the biggest building inside a parcel
Other uses of FME ..
Weekly transformation of heterogeneous data stored in production tables into one homegeneous search table for the Full-Text-Search used in the geoportal
Other uses of FME ..
One – shot analysis and export jobs Mosaicking and Production of ECW – Files for
background rasters
Other uses of FME:Grayscale conversion
About FME and its use
The general credo of the geoportal is to use a maximum of FOSS components: PostGIS Mapserver Mapfish (OpenLayers, ExtJS, GeoExt)
Exceptions: Use of ArcGIS:
Complex symbologies Data delivered by FGDB / MXD ArcGIS for INSPIRE
Use of FME FME is the only tool which allows us to really
MANIPULATE geodata, there is no alternative
Any questions?
Email: [email protected] Follow us on twitter: @geoportal_lux
INSPIRE Trends
Integration between proprietary and open standards – especially as deployment moves to regional and local agencies
Increased focus on consuming INSPIRE services and GML
More opportunities for spatial data and invoke services to bridge this gap
Examples: machine to machine communication to provide seamless support for KML and PDF streaming
Success = User isn’t aware they are accessing INSPIRE
INSPIRE Compliant GML – Now What?
<?xml version="1.0" encoding="UTF-8"?><gml:FeatureCollection xmlns="http://www.w3.org/2001/XMLSchema" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:ps-f="urn:x-inspire:specification:gmlas:ProtectedSitesFull:3.0" xmlns:ps-n2000="urn:x-inspire:specification:gmlas:ProtectedSitesNatura2000:3.0" gml:id="id-0">
<gml:boundedBy><gml:Envelope srsName="EPSG:4326" srsDimension="2">
<gml:lowerCorner>0 360</gml:lowerCorner><gml:upperCorner>0 360</gml:upperCorner>
</gml:Envelope> </gml:boundedBy> <gml:featureMembers> <ps-f:ProtectedSite xmlns:ps="urn:x-inspire:specification:gmlas:ProtectedSites:3.0" xmlns:gco="http://www.isotc211.org/2005/gco"
xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tn-a="urn:x-inspire:specification:gmlas:AirTransportNetwork:3.0" xmlns:base="urn:x-inspire:specification:gmlas:BaseTypes:3.2" xmlns:tn="urn:x-inspire:specification:gmlas:CommonTransportElements:3.0" xmlns:gn="urn:x-inspire:specification:gmlas:GeographicalNames:3.0" xmlns:net="urn:x-inspire:specification:gmlas:Network:3.2" gml:id="ID_0A16FE5BFD6C9CB51DE1DEE17804A757">
<ps:geometry><gml:MultiSurface gml:id="id-9e3bfd62-aa14-4f81-a3ef-9af6d5129518-0" srsName="EPSG:4326"
srsDimension="2"><gml:surfaceMember>
<gml:Surface gml:id="id-9e3bfd62-aa14-4f81-a3ef-9af6d5129518-1">
<gml:patches><gml:PolygonPatch>
<gml:exterior>
<gml:LinearRing>
<gml:posList>59.42 18.87 59.42... </gml:posList></
gml:LinearRing></gml:exterior><gml:interior>
<gml:LinearRing>
<gml:posList>59.41 18.89 59.47 ... </gml:posList></
gml:LinearRing></gml:interior>
</gml:PolygonPatch> …
Now What?
“How is anyone going to use INSPIRE?” FME allows you to consume INSPIRE compliant
GML as well as create it. Why not build value on the web services every
one else is investing in? Extract and flatten complex INSPIRE structures for
use within your GIS applicationsAdd spatial data services to transform INSPIRE into data streams suitable for widely accessible client applications (PDF, KML, GeoJSON)
XML vs. Relational
attribute1 attribute2 Active Date_From
Active Date_To
John Vancouver 11-22-99 12-11-09
June Surrey 02-25-05 9-15-10
Relational
Reading INSPIRE GML: with flattening
Output to PDF
INSPIRE Hydrography fromLuxembourg WFS on FME Viewer
Poll Question
Which areas would you like to see enhancements to FME’s support for INSPIRE?
Format translation Schema mapping String and list manipulation Data validation Database load and extract XML,GML,WFS: reading, validation, publication Web services: WFS, WMS, integration with others Metadata support Enterprise services with FME Server
FME Tools for INSPIRE
FME Tools for INSPIRE XML
GML reading and writing (3.2.1 and earlier) XML Reading – read virtually any XML and extract
features XML Processing – validate, format, and update XML Geometry Extraction (features > GML objects) XML Writing - Write XML or GML no matter how
complex XMLTemplater
FME merges attribute values into a template structure Template per feature type and dataset
XML Validation – validated to ensure compliance with INSPIRE schemas
Complex GML Geometry Support: CityGML (basis for Annex III Buildings)
FME Tools for INSPIRE
FME provides a full range of tools for INSPIRE, our partners provide complete solutions Data modeling interface makes it easier to handle
schema mapping - no coding required. Automated workflows save time in creating and
maintaining transformation processes. Self-documenting workspaces ease collaboration Easily integrates with INSPIRE-compliant services:
Esri ArcGIS Server Deegree, MapServer and GeoServer
Scalability and performance support quality of service requirements
Summary: Solution Approaches
Typical stages for SDI projects: Evaluation, assembly, transformation, validation and publication
FME provides tools that empower solutions for each step of the INSPIRE harmonization process
Our partners provide a range of solutions Conterra’s INSPIRE Solution Pack for FME
Luxembourg Implementation of by Jeff Konnen INSPIRE trends
consuming INSPIRE data need for spatial data services to support common clients
FME provides tools to support INSPIRE, our partners apply these to provide solutions
Poll Question
Interested in a free seat of online training?
Thank You!
Questions?
For more information:Jeff [email protected]
Dean [email protected] www.safe.com/inspire www.fmepedia.com (examples)
?