an open-source streaming machine learning and real-time...
TRANSCRIPT
![Page 1: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/1.jpg)
1
William Markito@william_markito
Fred Melo@fredmelo_br
An Open-Source Streaming Machine Learning and Real-Time Analytics Architecture
Using an IoT example
(incubating) (incubating)
![Page 2: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/2.jpg)
2
Traditional Data Analytics - Limitations
HDFS
Data Lake
Store Analytics
Hard to change Labor intensive
Inefficient
No real-time information ETL based Data-source specific
![Page 3: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/3.jpg)
3
Stream-based, Real-Time Closed-Loop Analytics
HDFSData LakeExpert System /
Machine Learning
In-Memory Real-Time Data
Continuous Learning Continuous
Improvement Continuous Adapting
Data Stream Pipeline
Multiple Data Sources Real-Time Processing Store Everything
![Page 4: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/4.jpg)
4
A Streaming Machine Learning for IoT Example
Sensor Data
Smart System
Learns with HISTORICAL TRENDS
"How were the temperature and vibration sensors reading when the latest failures happened? "
Live data becomes historical over time
Real-Time
Evaluates LIVE DATA“According to historical trends, there’s an 80% chance this equipment would fail in the next 12 hours"
Historical
Predictive Maintenance Scenario
![Page 5: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/5.jpg)
5
Info
Analysis
Look at past trends (for similar input)
Evaluate current input
Score / Predict
Machine Learning
Streaming Machine Learning
![Page 6: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/6.jpg)
6
Info
Analysis
Filter
[ json ]
Machine Learning
Streaming Machine Learning
![Page 7: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/7.jpg)
7
Info
Analysis
Filter Enrich Machine Learning
Streaming Machine Learning
![Page 8: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/8.jpg)
8
Info
Analysis
Filter Enrich Transform Machine Learning
Streaming Machine Learning
![Page 9: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/9.jpg)
9
Info
Analysis
Filter Enrich TransformML Model
Streaming Machine Learning
![Page 10: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/10.jpg)
10
Info
Analysis
Filter Enrich Transform
Transform
ML Model
Streaming Machine Learning
![Page 11: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/11.jpg)
11
In-Memory Data Grid
Front-end
Update Push
ML Model
Streaming Machine Learning
![Page 12: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/12.jpg)
12
Neural Network
In-Memory Data GridReal-time scoring
Train
Supervised Learning ExampleStreaming Machine Learning
![Page 13: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/13.jpg)
13
Ingest Transform SinkSpringXD
Store / Analyze
Fast Data
Distributed Computing
Predict / Machine Learning
Other Sources and Destinations
JMS
A Streaming Machine Learning Reference Architecture
![Page 14: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/14.jpg)
Indoors Localization - Applied Example
14
![Page 15: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/15.jpg)
Trilateration and its limitations
15
Noisy Data
Physical Barriers
Large Overlap Areas
Moving Targets
Innacuracy
Large Overlap Areas
![Page 16: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/16.jpg)
Particle Filters - Calculating the optimum solution
16
![Page 17: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/17.jpg)
Particle Filters - Calculating the optimum solution
17
![Page 18: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/18.jpg)
The Solution
18
1. Capture signal strength 2. Calculate distance from
antenna 3. Trilaterate different sensors
to predict location in real-time 4. Show on a map with live
updates
![Page 19: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/19.jpg)
Architecture Overview
19
Ingest
SpringXD
Groovy
JSON HTTP
+ Distance
Transform Sink
Calculate Device Distance Predict
Location
Spring Boot
Application Platform
GUI
![Page 20: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/20.jpg)
20
• Cache • Configurable through XML, ,Java
• Region • Distributed j.u.Map on steroids • Highly available, redundant
• Member • Locator, Server, Client
• Callbacks • Listener, Writer, AsyncEventListener, Parallel/Serial
Geode Basic Concepts
![Page 21: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/21.jpg)
Introduction to SpringXD
21
Runs as a distributed application or as a single node
![Page 22: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/22.jpg)
Spring XD
22
A stream is composed from modules. Each module is deployed to a container and its channels are bound to the transport.
![Page 23: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/23.jpg)
Demo
![Page 24: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/24.jpg)
24
Why have we selected those projects
• Iterative & Exploratory model
• Web based REPL • Multiple Interpreters
• Apache Geode • Apache Spark • Markdown • Flink • Python…
• In-memory & Persistent • Highly Consistent • Extreme transaction
processing • Thousands of concurrent
clients • Reliable event model
• Productivity • Built-in connectors • Cloud Agnostic • Highly Scalable • Easy to setup • Streams without coding
![Page 25: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/25.jpg)
25
https://github.com/Pivotal-Open-Source-Hub/WifiAnalyticsIoTSource code and detailed instructions available at:
25
William Markito@william_markito
Fred Melo@fredmelo_br
Follow us on GitHub!
![Page 26: An Open-Source Streaming Machine Learning and Real-Time ...events17.linuxfoundation.org/sites/events/files/slides...1 William Markito @william_markito Fred Melo @fredmelo_br An Open-Source](https://reader036.vdocument.in/reader036/viewer/2022071513/6134a052dfd10f4dd73bd9b8/html5/thumbnails/26.jpg)
26
26
William Markito@william_markito
Fred Melo@fredmelo_br
Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode (incubating), R and Spring XD
Room: Tohotom - 14:30, Sep 30 Fred Melo, Pivotal, William Markito, Pivotal