simplifying the complex: serving data from pipeline data models

Simplifying the Complex:Serving data from Pipeline Data ModelsPeter VeenstraTRC

Introduction

Peter Veenstra

Problem or Challenge?

GIS is difficult, users want data …Non-GIS Professionals need access to data to do their jobs in XLS

Industry Standard Pipeline Data Models are complex Linear Referencing

Data stored in Silos – hard to synchronize or coordinate …

Model, Load, As-Built – Re-buildSilo 1 Silo 2

Big Gnarly Mess1000’s if not 10’s of 10000’s miles of data overlaid on a single route.

Linear events (in 100’s of tables) related to each other by common position along the pipeline but without formal relates.

Potentially 7+ million segments to process.

X

Good

Well Drained

Test 1 - Current

MAOP

T

MFL

Type 1

Type 2b Type 2a

X X X X X X

Poor

Poorly Drained

Test 2 - Current

Solution

Build Toolboxes using FME

Move toolboxes to server.

Let the data be consumed.

ToolboxesFME Toolkit for Pipeline Data Models (PDM)

FME Toolkit for Linear Referencing (LRM)

FME Toolkit for Database Integration (DIM)

XYZM

PL-AAA

XYZM

XYZM

Sum

Late

st

Mer

ge

Move to FME Server

What is Server?

Why is it better than scheduled tasks and scripts?

Planned Migration (build, then optimize)

AggregationData Warehouse

FME ServerBuild a data ware-house aggregating data stored in different silos …- Shared coordinate

plane- Hierarchy to Hierarchy

translation- Dynamic Segmentation

into massive tablePODS Silo 1 Silo 2 Silo 3

X

Good

Well Drained

Test 1 - Current

MAOP

T

MFL

Type 1

Type 2b Type 2a

X X X X X X

Poor

Poorly Drained

Test 2 - Current

FME ServerServices BusQuery Engine

TableauHierarchy FinderData ExtractorData Delta Finder/SynchronizerData Health2016: Status and Progress Tracker

A lot to get your head around …

Subscribe, publish, consume … respond.

Services-oriented rather than task-oriented

Event-driven rather than procedural …

GIS is difficult, users want data …

"Good data leads to good knowledge, and good knowledge solves problems."

Good data eh!

Thank you!

Peter Veenstra, TRC

pveenstra@trcsolutions.com816-820-7841