overcoming poor data quality amidst a crisis
DESCRIPTION
TRANSCRIPT
![Page 1: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/1.jpg)
March 2008
Overcoming Poor Data Quality Amidst a Crisisv
Ulf MånssonSWECO
![Page 2: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/2.jpg)
Combined expertise in consulting engineering, environmental technology and architecture
So where does ULF constantlyinform (nag) about FME?
![Page 3: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/3.jpg)
SWECO
4
The Nordic region’s leading provider of consulting services in the fields of engineering, environmental technology and architecture
4,900 employees in 10 countries
3.9 billion SEK- annual net sales
24,000 assignments each year
7,000 clients
![Page 4: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/4.jpg)
Present Establishments
Sweden:2.560
Norway: 600
Denmark: 40
Finland:1.000
Estonia: 225
Lithuania: 175
Employees 20074.900
Russia: 110
Bulgaria
Czech rep: 270Slovakia
![Page 5: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/5.jpg)
Assignments Worldwide
Assignments in more than 100 countries since 1903
Increased international presence
Ongoing assignments in 75 countries
20% –
15% –
15% –
14% –
12% –
9% –
8% –
5% –
2% –
Industrial Engineering
Water and Environment
Transport and Civil Eng.
Energy Systems
Building Service Systems
Architecture
Structural Engineering
Project Management
Geographic Positioning
![Page 6: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/6.jpg)
Ulf Månsson
Modern organizations depend more on data quality than ever before.
Only a few spare parts in stock
Large centralized districs, no local offices
No extra staff
Modern Organizations
![Page 7: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/7.jpg)
Ulf Månsson
Modern Organizations
Organizations rely on systems and their data. Geographic data are crucial:
No local offices Poor knowledge of local geography
Where are my supplies/spare parts?
Where are my people? How can they find me?
![Page 8: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/8.jpg)
Ulf Månsson
C R I S I S
The importance of data quality becomes strikingly clear in a crisis management process.
FME is an important tool for managing ”ad-hoc” situations at a major crisis.
The hurricane Gudrun is the best Swedish example.
”Spontaneous ETL”
Instantly connect data from many systems to improve quality and derive the needed information output.
![Page 9: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/9.jpg)
Hurricane Gudrun 2005
Wind speed 10-46 m/s 20 000 km electrical cables damaged 2 500 people in workforce Out of supplies, gas, equipment,etc.
The – first map illustrating the situation. (2D-Pointreplacer)
Power outages lasted for weeks We used FME for weeks
![Page 10: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/10.jpg)
Services Provided for Our Client
Information to the crisis management teams Rescue workers and municipalities Daily delivery of maps and lists of persons/addresses
Data quality
Prognoses for re-connections
Providing maps for field-work
Maps for reconstruction
![Page 11: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/11.jpg)
”Spontaneous ETL”
Large power company – many IT systems Outage management systems Network schema systems Network Information Systems (NIS) Customer Information Systems (CIS)
Oracle Text files Excel Shape files Etc…
FME El Dorado !!!
![Page 12: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/12.jpg)
Challenge: Making a MapShowing Areas with Outages
![Page 13: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/13.jpg)
Connecting the Parts
OutageManagement
System (Oracle)
Customer IDOutage data
NetworkInformation
System(Oracle Spatial)
Customer IDX,Y
Customer Data(SAP->Txt files)
Customer IDAddresses, Zip-
Codes etc.
XLSLists of addresses
Bufferedareas with
outage data
Shp filesGIF files
FMEWorkbenches
![Page 14: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/14.jpg)
Key FME Features
The diversity of readers and writers GIF, XLS, Shape, etc… (today we would have used pdf)
Joiner-functionality
Buffer and dissolver
Scripting Today, we would have used the fanout-functionality
![Page 15: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/15.jpg)
It all worked gloriously (in theory) but bad data…
The mercyless ConvexHullAccumulator exposes bad quality.
Defining municipality by zip-code from CIS…
Solution Use municipality-
borders to clip data
![Page 16: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/16.jpg)
ConvexHullAccumulator - Good Data
![Page 17: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/17.jpg)
Ugly Data
”You should at least have used the
2DPointReplacer before deployment!”
![Page 18: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/18.jpg)
Connecting the Parts
OutageManagement
System (Oracle)
Customer IDOutage data
NetworkInformation
System(Oracle Spatial)
Customer IDX,Y
Customer Data(SAP->Txt files)
Customer IDAddresses, Zip-
Codes etc.
XLSLists of addresses
Bufferedareas with
outage data
Shp filesGIF files
FMEWorkbenches
![Page 19: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/19.jpg)
”To know what you not know”
Polygons representing reconnection forecasts A simple geometry manageable by each municipality The buffer radius correspond to the data quality Polygons marked as ”We simply don’t know…”
![Page 20: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/20.jpg)
Conclusions
Keep it simple GIF and PDF files are easy to distribute and print.
Spontaneous ETL Know your FME and it’s a great problem-solver in
unexpected situations.
FME Prototyping A great way to define and describe processes. These can
also be implemented in other systems.
![Page 21: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/21.jpg)
SWECO Vision
To be Europe’s most respected knowledge company in consulting engineering, environmental technology and architecture
![Page 22: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/22.jpg)
Recent artwork(or ”unexpected
results”)
![Page 23: Overcoming Poor Data Quality amidst a Crisis](https://reader033.vdocument.in/reader033/viewer/2022061117/5468483baf7959a2358b611e/html5/thumbnails/23.jpg)
More artwork