![Page 1: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/1.jpg)
Automating
Everything with FME
![Page 2: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/2.jpg)
Agenda
• Journey to Automation
• What is Automation?
• Data Loading Scenarios
• Working with Big Data
![Page 3: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/3.jpg)
![Page 4: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/4.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
![Page 5: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/5.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
![Page 6: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/6.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
Episode #3:
Validation Strikes Back
The data in these systems sucks. I need to filter out the
bad, fix it, and make sure only good stuff gets moved.
![Page 7: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/7.jpg)
The Journey of Data Superheroes
Episode #1:
A New Movement
I have systems that aren't connected and data lying
around. I need to start moving data.
Episode #2:
Transformation Menace
These systems are too different and I can’t easily move
data between them.
Episode #3:
Validation Strikes Back
The data in these systems sucks. I need to filter out the
bad, fix it, and make sure only good stuff gets moved.
Episode #4:
Automation Awakens
I want data to be moved automatically. It should
happen each night, as soon as data is available, or on-
demand if someone needs it right then.
![Page 8: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/8.jpg)
FME Server is Automation.
![Page 9: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/9.jpg)
What is Automation?
Running an
FME WorkspaceDelivering DataTriggering an event
![Page 10: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/10.jpg)
Your Top Automations*
1. Loading Data
2. Processing Big Data
* based on 5 years of survey data from FME Server Customers
![Page 11: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/11.jpg)
Loading DataTriggered
• By Schedule
• On-Demand
• In Real-Time
Top Automations: #1
![Page 12: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/12.jpg)
Example: Scheduled Replication
“FME Server is the caretaker that lets us pay attention to other things.”
![Page 13: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/13.jpg)
Example: Loading On-Demand
![Page 14: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/14.jpg)
Example: Loading in Real-Time
![Page 15: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/15.jpg)
Data Loading Tips
✓ Validate data before it’s loaded and
immediately report the quality of a user’s
submission.
✓ For duplicate systems, nightly replication
avoids disruptions during the day.
✓ Keeping systems connected in real-time is
better for insight/transparency.
![Page 16: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/16.jpg)
Processing Big Data
Top Automations: #2
![Page 17: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/17.jpg)
Big Data Symptoms
❏ Difficulty figuring out how to handle all your data.
❏ So much data but not enough ability to process it.
❏ Regular use of the term terabyte.
❏ Work with imagery, LiDAR, networks,
live streams.
❏ Performing tasks like modelling, spatial
relationships, change detection, large-
scale map production.
![Page 18: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/18.jpg)
Example: Big Networks (10,000 route queries!)
![Page 19: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/19.jpg)
Example: Big Maps
“The volume of data required to be processed needed an automation workspace run on FME Server.”
![Page 20: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/20.jpg)
Big Data Tips✓ Look to improve performance in your FME workspaces.
✓ Use 64-bit FME for full access to memory resources.
✓ Use a dedicated FME Server machine.
✓ Use multiple FME Engines for concurrent processing.
✓ Spread processing across multiple machines.
✓ Estimate the total processing time required and the
frequency of processing to decide if automation is
worth it.
![Page 21: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/21.jpg)
Example: Big Maps in the Cloud
![Page 22: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/22.jpg)
Tip: Put Big Data in the Cloud.✓ Scalable
� On-demand processing power.
✓ Reliable� 99.99%+ uptime.
✓ Cost effective� Pay only for what you need.
![Page 23: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/23.jpg)
22 Minutes to
Destination
Example: Real-Time Networks
Brake pressure
OK
Fuel Level at
60%
JMS
![Page 24: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/24.jpg)
“Fast Data” Tips
✓ Keep FME Engines always running.
✓ Use message queue software (JMS, SQS).
✓ Use multiple FME Engines for scalability.
![Page 25: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/25.jpg)
● FME Server and FME Cloud were created
for Automation.
● Every data superhero's journey leads to automation.
● Automations are triggered by schedules, on-demand
requests and real-time events.
● Loading data and processing big data are the most
common automations.
![Page 26: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/26.jpg)
More info about all of the stories seen here:
● On-Demand Data Loading example (Swedish EPA): http://www.safe.com/fmeuc/automated-quality-controls-with-fme/
● Scheduled Replication example (San Jose): http://www.safe.com/customers/san-jose-water-company/
● Data Loading in Real Time example (San Jose): http://www.safe.com/presentation/oracle-database-triggers-and-fme-server-for-
task-tracking/
● Big Networks example (London Fire Brigade): http://www.1spatial.com/wp-
content/uploads/2015/11/London_Fire_Case_Study_FINAL.pdf
● Big Maps example: http://www.safe.com/presentation/national-map-production-with-mapnik-rasterizer/ ;
http://www.imgs.ie/ordance-survey-ireland
● Big maps in the cloud example (Weather): http://www.slideshare.net/SafeSoftware/fme-around-the-world-46744668/53 ;
http://blog.safe.com/2015/02/event-driven-architecture-aws-lambda-s3/
● “Fast data” railway example: http://blog.safe.com/2014/12/125000-mph-railways/
![Page 27: Automating Everything with FME - Excellence Center for FME · Automating Everything with FME. Agenda • Journey to Automation ... requests and real-time events. Loading data and](https://reader030.vdocument.in/reader030/viewer/2022021511/5b150b437f8b9a8f548d73ba/html5/thumbnails/27.jpg)
Thank you!
Questions?