ultimate real-time — monitor anything, update anything
TRANSCRIPT
![Page 1: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/1.jpg)
Ultimate Real-Time Monitor Anything, Update Anything
![Page 2: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/2.jpg)
FME Server Notification Service lets you act on events as they happen, and send information as it becomes available. Publications and Subscriptions let you publish to and monitor different systems and data.
![Page 3: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/3.jpg)
Notifications What they are for
What they are not for
![Page 4: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/4.jpg)
Notifications What they are for
ü A brief message, usually to trigger an action.
What they are not for x Transmitting large amounts of
spatial data.
![Page 5: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/5.jpg)
Notifications What they are for
ü A brief message, usually to trigger an action.
ü Triggering an FME Server response to an event that happened outside of FME.
What they are not for x Transmitting large amounts of
spatial data. x Triggering an FME Server
response to a continuous series of messages (many per second).
![Page 6: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/6.jpg)
Notifications What they are for
ü A brief message, usually to trigger an action.
ü Triggering an FME Server response to an event that happened outside of FME.
ü Sending a message about something that happened in FME.
What they are not for x Transmitting large amounts of
spatial data. x Triggering an FME Server
response to a continuous series of messages (many per second).
x Sending more than one message per second about what’s happening in FME.
![Page 7: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/7.jpg)
FME Server Notification Service
See the list of what built-in publications and subscribers can be: http://fme.ly/protocols
![Page 8: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/8.jpg)
Workflows for “Ultimate Real-Time”
Poll it Push it Update it
![Page 9: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/9.jpg)
Poll it “Are we there yet?”
![Page 10: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/10.jpg)
Polling Workflow 1. Change detection
○ GeoRSS reader ○ Caching ○ Timestamps
2. Run the workspace on a schedule
Timestamp
![Page 11: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/11.jpg)
Scheduling (FME Server)
![Page 12: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/12.jpg)
GeoRSS Reader
![Page 13: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/13.jpg)
GeoRSS Reader
• Contains change detection logic
• Option to only read new features
![Page 14: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/14.jpg)
Caching
• Whole datasets or timestamps
• ChangeDetector
• SQL - match on ID
![Page 15: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/15.jpg)
ChangeDetector Example
![Page 16: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/16.jpg)
SQL to find different features: @Value(uuid) = "uuid" AND @Value(aqi) != "aqi"
SQL Example
![Page 17: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/17.jpg)
Timestamps
● Cache the timestamp
● Last updated attribute
● Filter features newer than timestamp
● Add a timestamp attributes to your dataset
![Page 18: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/18.jpg)
Reads last cached time and replaces with current time
![Page 19: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/19.jpg)
Checks against timestamp and filters out old features
![Page 20: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/20.jpg)
Pros and Cons Caching whole datasets
ü Easy to set up and detect changes.
x Need to store entire dataset - processing time?
Caching timestamps
ü No storage needed. x Harder to set up -
variables, datetimes.
Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?
![Page 21: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/21.jpg)
Story: Real-Time Traffic
![Page 22: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/22.jpg)
![Page 23: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/23.jpg)
![Page 24: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/24.jpg)
✓ Poll Waze feed
✓ Parse XML
✓ Filter
✓ Transform
✓ Validate
✓ Email notifications
![Page 25: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/25.jpg)
Workflows for “Ultimate Real-Time”
Poll it Push it Update it
![Page 26: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/26.jpg)
Push It
![Page 27: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/27.jpg)
Pushing Methods • Database triggers • Webhooks
In FME Server:
• Direct URL • REST API
![Page 28: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/28.jpg)
Database Triggers
● Databases can trigger events.
● Functions with HTTP
requests can push data to FME Server.
![Page 29: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/29.jpg)
Database Triggers
![Page 30: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/30.jpg)
Webhooks
● i.e. Give systems/services a URL that will respond.
● Send the notification to
FME Server. ● Queue the received
notifications and process the requests on a separate thread.
![Page 31: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/31.jpg)
Pros and Cons Database Triggers
ü Pushes right to FME Server topic.
ü Can push lots of changes to a holding table.
ü Perfect for real-time database changes.
x Database permissions.
Webhooks
ü No need to waste effort polling.
ü Truly real time. x Complex to set up. x Not all systems provide
or accept webhooks.
![Page 32: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/32.jpg)
Direct URL Systems can use HTTP to push data to topics or trigger workspaces to run.
![Page 33: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/33.jpg)
![Page 34: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/34.jpg)
Topic POST URL
![Page 35: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/35.jpg)
REST API In-depth interaction
with FME Server
● Send/receive notifications ● Run jobs ● Manage connections ● Manage users ● Licensing ● Backups ● Scheduling ● Etc!
![Page 36: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/36.jpg)
Workflows for “Ultimate Real-Time”
Poll it Push it Update it
![Page 37: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/37.jpg)
To Poll or Push?
![Page 38: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/38.jpg)
Poll vs. Push Considerations ● Simplicity vs. “real-time”ness ● Can the system push to FME Server? ● Are jobs too long for polling? ● API restrictions for polling?
![Page 39: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/39.jpg)
Tip: Use FME Server for internal systems and FME Cloud for external systems.
![Page 40: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/40.jpg)
Update It
![Page 41: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/41.jpg)
Updating Methods
• FME Writers o Bulk drop/load/overwrite o Update option
• SQLExecuter • HTTPCaller
![Page 42: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/42.jpg)
FME Writers: UPDATE operation
![Page 43: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/43.jpg)
INSERT, UPDATE or DELETE in the middle of a workflow.
![Page 44: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/44.jpg)
POST data or changes to systems.
![Page 45: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/45.jpg)
Summary
• Polling workflow: o Change detection + scheduling
• Pushing options: o Database triggers, webhooks o FME Server: URL or REST API
• Updating: o Operations in writer parameters o SQLExecutor o HTTPCaller
![Page 46: Ultimate Real-Time — Monitor Anything, Update Anything](https://reader031.vdocument.in/reader031/viewer/2022030306/58d10b531a28ab823e8b5b15/html5/thumbnails/46.jpg)
Questions?
Database triggers tutorial: fme.ly/dbtriggers
Webhooks etc:
fme.ly/web
REST API documentation: fme.ly/restv3