automating data management with fme workbench(s) data mapping attr1 attr2 attr3 derived prod info...

Automating Data Management with FME

Washington State Counties

Data and UsesDerived Products: Address (points) Parcels (boundaries) Streets (centerlines) Owner Information (tabular)

Geocoding Land base

Challenges 38 sources All in different data schemas Source formats are different

Keeping data up to date

Safe Software• Desktop 2016/17• Server 2016/17

Microsoft• Management Studio 2008• Server 2012

ESRI• Desktop 10.5.1• AGS 10.5.1• Portal 10.5.1

Standardize Data

1 - Map Data2 - Scheduler

FME Workbench(s)

Data Mapping

Attr1Attr2Attr3

Derived Prod Info

Attr1Attr2Attr3

3 – Download DataCounty FTP

FME Workbench(s)Map data to StandardFME Workbench(s)

Check data

WAMAS Data

Service

Derived Product

Attr1Attr2Attr3

4 – Data Validation

FME Workbench – Address Points

1 – Call workbenches for each County

FME Workbench – Address Points

1 – Call workbenches for each County

FME Workbench – Address Points

1 – Call workbenches for each County

FME Workbench – Address Points

2 – Get attribute mapping for county schema

FME Workbench – Address Points

2 – Get attribute mapping for county schema

To next workbench@Value(attrValue)@Value(attrValue)@Value(attrValue)@Value(attrValue)

FME Workbench – Address Points

3 – Parse Address and validate

\\servername\\servername

FME Workbench – Address Points

3 – Parse Address and validate

FME Workbench – Address Points

3 – Parse Address and validate

@Value(attrValue)

FME Workbench – Address Points

3 – Parse Address and validate

FME Workbench – Address Points

4 – Validate and write to Prod

https://state-wamas.wa.gov/webdirectory/pagename.asmx/Getstandardizedaddress?address=@Value(mADDR)&address2=@Value(mADDR2)&city=@Value(mCITY)&zip=@Value(mZIP)&zip4=@Value(mPLUS4)&state=@Value(mSTATE)&company=&Consumer=@Value(mCONSUMER) HTTP/1.1

FME Workbench – HTML Reports

FME Workbench – HTML ReportsTransformers:HTMLReportGeneratorSorterHTMLLayouterWriter:HTMLWriter

FME Workbench – HTML Reports

FME Workbench – HTML Reports

FME Server – Schedule

Self Serve Portal

Parcel and Owner Information

Street Information

Address Point Information

Address Point Information (City)

Data Counts and UsageAddress Points – 2,167,403Parcels – 3,114,394Streets – 731,172Owner Information – 2,101,161PSAP Streets (911) – 566,910

Geocodes – 2.6mBatch Service – 1.5m

Apple – 14.5kAndroid – 10k

Issues• FME 2017 – FMEServerJobSubmitter @Value(xxxx)• FTP data moving to OpenData

Next Steps• Create a generic workbench to accept each derived product type• Email notifications

Take away• Schedule Workbench Multiple Workbenches• Store attributes/configuration in database• Create Universal Custom Transformers (re-usable)• Log processes and generate reports

Questions

Austin HildrethDORDLI.S.GISTeam@DOR.WA.GOV