Time Machines and Attribute

Alchemy

Date & Time Attribute Manipulation

1. Dates and times

RCMP E Division Heidi Lee | Robert Shultz Goal: Load GPS records into ArcGIS. Problems: ➔  Inconsistent date formats ➔  Time zones ➔  Daylight saving

Dates and times are complicated.

Formatting?

●  YYMMDD, HHMMSS, UTC

●  Jun 2016

●  ‘on Saturday, Jan 9th 2016, 01:00 am’ & ‘+0530’

●  2016-12-07 12:20:07.785403-05

●  20160313020000.000 (March 13 - Daylight Saving)

●  <d v="2016-12-13T00:00:00"/> (Excel)

●  YYYY-MM-DD hh:mm:ss[.nnnnnnn] (SQL Server ‘datetime2’ value)

Calculations?

●  Date2-Date1 = How many days?

Improved FME Date Functions

Read Transform Parse Process Format Write

Read Transform Parse Process Format Write

Over to FME

Time Zones

FME 2017 UTC Offset e.g. -08:00

FME 2018 IANA Time Zone

e.g. America/Vancouver

Attribute Alchemy

Last Year

Attribute Management Quality Control

Southern Company Jeff DeWitt HOK Inc. David Baldacchino

Goals:

➔  Test for patterns in attribute values

➔  Extract substrings from attribute values

➔  Validate strings

2. Finding patterns NGI Belgium Jan Beyen RCMP E Div. Heidi Lee

Southern Company Problem: Attribute value cleanup

-  MONTANA * or Sales/Other (1) HOK Inc. Problem: Extract Sheet numbers from file names

-  MyProject - Sheet - A512 - PARTITION TYPES & … -  G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ...

NGI Problem: Validate address strings

-  Rue Achille Masset 52A

Sheet number extraction MyProject - Sheet - A512 - PARTITION TYPES & … G001 - GENERAL NOTES, ABBREVIATIONS, SYMBOLS, ... ^[\S\s]*?[-]?[ ]?([A-Z0-9]{1,5})[ ]+[-]+[\S\s]*$ Address validation Rue Achille Masset 52A ^((([a-zA-Z]+) )+)([0-9]+)([a-zA-Z]*)$

Wave the wand of regex

Regular Expression Editor Making regex easier since FME 2016

Alternatively: String functions

Regex vs. String Functions Example

Code ABD3705337067 Regular Expression: ([A-Z]{3})([0-9]+) String Functions:

Attribute String Function

alpha @Left(@Value(Code),3)

beta @Substring(@Value(Code),3,-1)

TRC Inc. Peter Veensta

Goals: ➔  Compare current and previous

Excel rows. ➔  Sum attribute values with the

previous row.

3. Time-travelling attributes.

FPInnovations Matt Kurowski

Past, Present & Future Attributes

feature[-1].measure measure

feature[+1].measure

Attribute Aggregation Challenge Rule: If T is 3 or less, aggregate this row with the row above.

Attribute Aggregation Challenge

Summary 1.  DateTime transformers and Text Editor

functions help with: ○  Date/time formatting ○  Calculations ○  Time zones

2.  Regex and string functions help with

patterns.

3.  Work with current and previous attribute values in the AttributeManager.

Story Time with the

AttributeManager

Automation: Schemas & Data Enhancement

for Dry Rot Insurance

Sigbjørn Tillerli Herstad

Goal: To automate and enhance the quality of daily/weekly imports of customer data to a common schema.

Problem: Source Data Mayhem ●  Codan Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates

●  DNB Forsikring – CSV – 1 file – 5 attributes

●  Eika Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates

●  Enter Forsikring – CSV – 2 files – 14 attributes

●  Frende Forsikring – XLS – 1 file – 10 attributes – missing AddressID, coordinates

●  Gjensidige – CSV– 2 file – 20 attributes – missing AddressID, coordinates

●  IF Skadeforsikring – XLS – 1 file – 10 attributes – missing owner

●  Jernbanepersonalets Forsikring – CSV – 3 files – 5 attributes

Achieving Automation •  Define schema to use for import to database (52+ attributes, one

feature type) •  AttributeFilter: Separate data streams for each company •  FeatureReader: Reads the actual data •  AttributeManager: Convert to common schema •  Can be run at scheduled intervals when a file arrives

Achieving Quality Custom transformers to improve and split data, reused on multiple files:

•  SplittFornavnEtternavn: Separate firstname and lastname into 2 different attributes.

•  SplitTelefonOgMobiltlf: Decide if number is a cellular or landline and create 2 different attributes.

•  SplitStreetNameNumberLetter: You have one attribute in which contains streetname, housenumber, houseletter. Output is 3 different attributes.

Achieving Quality Use existing services and databases to look up and verify values:

•  CheckAIDToOwner: Checks if this is the official owner of that property.

•  NorkartGeocoder: API to check the validity of an address, handles misspellings, validates postal number, municipality number, etc. Fresh data every day!

Achieving a Common Schema Translate each customer’s

schema to the common schema:

AttributeManager

One separate

AttributeManager for each company.

Questions?

Tutorial: fme.ly/datetime

AttributeManager documentation