simplifying the complex: serving data from pipeline data models
TRANSCRIPT
Simplifying the Complex:Serving data from Pipeline Data ModelsPeter VeenstraTRC
Introduction
Peter Veenstra
Problem or Challenge?
GIS is difficult, users want data …Non-GIS Professionals need access to data to do their jobs in XLS
Industry Standard Pipeline Data Models are complex Linear Referencing
Data stored in Silos – hard to synchronize or coordinate …
Model, Load, As-Built – Re-buildSilo 1 Silo 2
Big Gnarly Mess1000’s if not 10’s of 10000’s miles of data overlaid on a single route.
Linear events (in 100’s of tables) related to each other by common position along the pipeline but without formal relates.
Potentially 7+ million segments to process.
X
Good
Well Drained
Test 1 - Current
MAOP
T
MFL
Type 1
Type 2b Type 2a
X X X X X X
Poor
Poorly Drained
Test 2 - Current
Solution
Build Toolboxes using FME
Move toolboxes to server.
Let the data be consumed.
ToolboxesFME Toolkit for Pipeline Data Models (PDM)
FME Toolkit for Linear Referencing (LRM)
FME Toolkit for Database Integration (DIM)
XYZM
PL-AAA
XYZM
XYZM
Sum
Late
st
Mer
ge
Move to FME Server
What is Server?
Why is it better than scheduled tasks and scripts?
Planned Migration (build, then optimize)
AggregationData Warehouse
FME ServerBuild a data ware-house aggregating data stored in different silos …- Shared coordinate
plane- Hierarchy to Hierarchy
translation- Dynamic Segmentation
into massive tablePODS Silo 1 Silo 2 Silo 3
X
Good
Well Drained
Test 1 - Current
MAOP
T
MFL
Type 1
Type 2b Type 2a
X X X X X X
Poor
Poorly Drained
Test 2 - Current
FME ServerServices BusQuery Engine
TableauHierarchy FinderData ExtractorData Delta Finder/SynchronizerData Health2016: Status and Progress Tracker
A lot to get your head around …
Subscribe, publish, consume … respond.
Services-oriented rather than task-oriented
Event-driven rather than procedural …
GIS is difficult, users want data …
"Good data leads to good knowledge, and good knowledge solves problems."
Good data eh!
