back to fme school - day 1: your data and fme
TRANSCRIPT
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks
Double, DoubleLess Toil, Less TroubleGIS professionals are finding themselves tasked with doing double the work in half the time!
Luckily, FME is designed to magically ease that burden.
A Quick List● Remodeled CSV Reading● Improved Attribute Management● Better List Handling● Enhanced Polygon Dissolving● Faster FeatureReader
Workflows● Empowered Reading
of Data From Web Sources
Steps1. Read a Geographical Names CSV Dataset
2. Filter the Geographical Names Data for a List of Cities Only
3. Read the Current List of FME World Tour Events
4. Merge the FMEWT Event with its Country Code
5. Match the FMEWT Event to its Host City
6. Ensure the Correct Location is chosen for each Event
2016
• Read Data: 1 minute 35 seconds• Read/Filter: 1 minute 36 seconds
2017
• Read Data: 8.3 seconds• Read/Filter: 14.1 seconds
Processing Times
SummaryPerformance in Creation• Attributes to Read• List Element Selection• HTML Table Reader• Select Source from Web• Modeless Parameter Dialogs• Quick Add Readers/Writers• User Parameter Copy/Paste• Open Containing Folder
Performance in Execution
• Attributes to Read• List Element Selection• Feature Table Formats• Feature Table Transformers• Faster Polygon Dissolving• Faster FeatureReader
Relational Databases
ID Name Position
1 Harry Potter Seeker
2 Ginny Weasley Chaser
3 Draco Malfoy Seeker
ID Name Colors
1 Gryffindor Red and gold
2 Slytherin Green and silver
TeamID PlayerID
1 1
1 2
2 3
Player
Team
Team_Players
Relational Databases
✔ Static schema ✔ Server Infrastucture ✔ Accuracy ✔ Complex SQL querying ✔ Control security for each table
NoSQL Databases
{
ID:1,
Title:"BilboBaggins’EleventyFirstBirthdayParty",
Location:Shire,
FireworksDisplay:{
Name:"GandalftheWizard",
Time:"halfpastsix",
}
}
AWS DynamoDB
FME 2017 – New Formats New Formats: • MBTiles (R/W) • OGC GeoPackage Raster (R/W) • MapInfo Extended TAB (R) • IBM dashDB (R/W)
OGC GeoPackage • Free and open source • Standards-based • Platform-independent • Portable • Compact
Supports raster and vector all in a SQLite Database! http://www.geopackage.org/
Step 2: Collect Data
Go Mobile!
Place your screenshot here
http://ngageoint.github.io/geopackage-mapcache-android/
Final Step: Merge all Updates into Master
FME Tips: 1. Detect updates 2. fme_db_operation 3. Unique ID field
Summary
• Choose the right database for the occasion – Pros and cons to relational,
NoSQL, and file-based databases
• Update master database using UpdateDetector, fme_db_operation, and a unique ID
Questions?
Learn more:
knowledge.safe.com Tutorial: Updating Databases
NoSQL blogs + webinar
safe.com/nosql
RCMP E DivisionHeidi Lee | Robert Shultz
Goal: Load GPS records into ArcGIS.
Problems:➔ Inconsistent date formats➔ Time zones➔ Daylight saving
Dates and times are complicated.
Formatting?
● YYYYMMDDHHMMSS.SS (FME)
● <d v="2016-12-13T00:00:00"/> (Excel)
● YYMMDD, HHMMSS, UTC
● Jun 2016
● ‘on Saturday, Jan 9th 2016, 01:00 am’ & ‘+0530’
● 2016-12-07 12:20:07.785403-05
● 20160313020000.000 (March 13 - Daylight Saving)
● YYYY-MM-DD hh:mm:ss[.nnnnnnn] (SQL Server ‘datetime2’ value)
Calculations?
● Date2-Date1 = How many days?
Southern CompanyJeff DeWitt
HOK Inc.David Baldacchino
Goals:
➔ Test for patterns in attribute values
➔ Extract substrings from attribute values
➔ Validate strings
2. Finding patternsNGI BelgiumJan Beyen
RCMP E Div.Heidi Lee
Southern CompanyProblem: Attribute value cleanup
- MONTANA * or Sales/Other (1)
HOK Inc.Problem: Extract Sheet numbers from file names
- MyProject - Sheet - A512 - PARTITION TYPES & … - G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...
NGIProblem: Validate address strings
- Rue Achille Masset 52A
Sheet number extraction
MyProject - Sheet - A512 - PARTITION TYPES & … G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...
^[\S\s]*?[-]?[ ]?([A-Z0-9]{1,5})[ ]+[-]+[\S\s]*$
Address validation
Rue Achille Masset 52A
^((([a-zA-Z]+) )+)([0-9]+)([a-zA-Z]*)$
Wave the wand of regex
Regex vs. String Functions Example
Code ABD3705337067
Regular Expression: ([A-Z]{3})([0-9]+)
String Functions:
Attribute String Function
alpha @Left(@Value(Code),3)
beta @Substring(@Value(Code),3,-1)
StringPairReplacer
Replace multiple values from a list, for example:
Niccolo' Giosue' would become Niccolò Giosuè
More String Replacement options...
TRC Inc.Peter Veenstra
Goals:
➔ Compare current and previous Excel rows.
➔ Sum attribute values with the previous row.
3. Time-travelling attributes.
FPInnovationsMatt Kurowski
Past, Present & Future Attributes
feature[-1].measure measure
feature[+1].measure
Where to find it:AttributeManagerAttributeCreator
Referencing the last feature’s valuesFor the attributes stationing, measure & odometer, calculate the cumulative values from the previous features
Summary1. DateTime transformers and Text Editor
functions help with:○ Date/time formatting○ Calculations○ Time zones
2. Regex and string functions help with patterns.
3. Work with current and previous attribute values in the AttributeManager.
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks