automating data management with fme workbench(s) data mapping attr1 attr2 attr3 derived prod info...
TRANSCRIPT
Automating Data Management with FME
Washington State Counties
Data and UsesDerived Products: Address (points) Parcels (boundaries) Streets (centerlines) Owner Information (tabular)
Geocoding Land base
Challenges 38 sources All in different data schemas Source formats are different
Keeping data up to date
Safe Software• Desktop 2016/17• Server 2016/17
Microsoft• Management Studio 2008• Server 2012
ESRI• Desktop 10.5.1• AGS 10.5.1• Portal 10.5.1
Standardize Data
1 - Map Data2 - Scheduler
FME Workbench(s)
Data Mapping
Attr1Attr2Attr3
Derived Prod Info
Attr1Attr2Attr3
3 – Download DataCounty FTP
FME Workbench(s)Map data to StandardFME Workbench(s)
Check data
WAMAS Data
Service
Derived Product
Attr1Attr2Attr3
4 – Data Validation
FME Workbench – Address Points
1 – Call workbenches for each County
FME Workbench – Address Points
1 – Call workbenches for each County
FME Workbench – Address Points
1 – Call workbenches for each County
FME Workbench – Address Points
2 – Get attribute mapping for county schema
FME Workbench – Address Points
2 – Get attribute mapping for county schema
To next workbench@Value(attrValue)@Value(attrValue)@Value(attrValue)@Value(attrValue)
FME Workbench – Address Points
3 – Parse Address and validate
\\servername\\servername
FME Workbench – Address Points
3 – Parse Address and validate
FME Workbench – Address Points
3 – Parse Address and validate
@Value(attrValue)
FME Workbench – Address Points
3 – Parse Address and validate
FME Workbench – Address Points
4 – Validate and write to Prod
https://state-wamas.wa.gov/webdirectory/pagename.asmx/Getstandardizedaddress?address=@Value(mADDR)&address2=@Value(mADDR2)&city=@Value(mCITY)&zip=@Value(mZIP)&zip4=@Value(mPLUS4)&state=@Value(mSTATE)&company=&Consumer=@Value(mCONSUMER) HTTP/1.1
FME Workbench – HTML Reports
FME Workbench – HTML ReportsTransformers:HTMLReportGeneratorSorterHTMLLayouterWriter:HTMLWriter
FME Workbench – HTML Reports
FME Workbench – HTML Reports
FME Server – Schedule
Self Serve Portal
Parcel and Owner Information
Street Information
Address Point Information
Address Point Information (City)
Data Counts and UsageAddress Points – 2,167,403Parcels – 3,114,394Streets – 731,172Owner Information – 2,101,161PSAP Streets (911) – 566,910
Geocodes – 2.6mBatch Service – 1.5m
Apple – 14.5kAndroid – 10k
Issues• FME 2017 – FMEServerJobSubmitter @Value(xxxx)• FTP data moving to OpenData
Next Steps• Create a generic workbench to accept each derived product type• Email notifications
Take away• Schedule Workbench Multiple Workbenches• Store attributes/configuration in database• Create Universal Custom Transformers (re-usable)• Log processes and generate reports
