scaling data distribution service (dds) to internet scale with vortex cloud and vortex fog
TRANSCRIPT
AngeloCorsaro,PhDCTO,ADLINKTech.Inc.Co-Chair,[email protected]
Taking DDS to Internet Scale with Vortex Cloud and Fog
DDS Applications
Autonomous Vehicles
Transport Management Systems Complex Medical Devices
Smart CitiesIndustrial Automation & Robotics
Smart Energy Grid
Integrated Modular Vetronics Training & Simulation Systems Naval Combat Systems
Air Traffic Control & Management Unmanned Air Vehicles Aerospace Applications
DDS provides a virtual data space abstraction that hides the network connectivity and topology details.
Conceptually, this abstraction works well to coordinate applications at
any scale — from a single host to an Internet scale application
But how can we make sure that it scales?
dDS Abstraction
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
No single point of failure or bottleneck
Decentralised
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
TopicDQoS
TopicDQoS
TopicAQoS
Data-Space
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
Scale Challenges
DDS dynamic discovery is an essential element in providing isolation from and
adaptation to the connectivity and topology details and overall location
transparency
To bring DDS application to Internet scale, discovery data needs to be
properly managed
Discovery information has to be constrained to limit resource usage and
avoid scalability bottlenecks
Discovery
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
When scaling DDS application to internet scale we have to face
network heterogeneity w.r.t. the supported features, i.e. lack of
multicast in some segments, as well as the differences in latencies and bandwidth
This heterogeneity has to be dealt without compromising the
performance of edge systems
Network Heterogeneity
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
Once we open “edge” systems or “edge” devices to exchange data
with other systems/devices, security becomes a concerns
Additionally NAT and Firewall introduce connectivity limitations
that have to be dealt-with
Security
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
The QoS required on edge systems may not be the
same that can be sustained end-to-end
Thus some form of QoS adaptation are required
QoS
DDS Global Data Space
...
Data Writer
Data Writer
Data Writer
Data Reader
Data Reader
Data Reader
Data Reader
Data Writer
TopicAQoS
TopicBQoS
TopicCQoS
TopicDQoS
Vortex Cloud and Fog
Device implementations optimised for OT, IT and
consumer platforms
Native support for Cloud and Fog Computing
Architectures
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdk
VORTEX Cloud provides a universally accessible “Internet Service” for
sharing data between DDS-enabled applications
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdkVortex Cloud
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdkVortex Cloud
‣ Elastic and Fault-Tolerant
‣ Public/Private Clouds Deployments
‣ Reliable and Best-Effort Data Sharing
‣ Last value Reliability
‣ Unicast and Multicast Communication
‣ Support for enterprise, embedded, and mobile platforms
‣ Source Filtering
‣ Customisable Load-Balancing
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdkVortex Cloud
‣ Efficient Binary Protocol (DDSI)
‣ Multiple Transports:
‣ UDP/IP
‣ TCP/IP
‣ WebSockets
‣ Connectivity to MQTT, AMQP, etc., via VORTEX Gateway
VORTEX Fog provides a transparent, elastic and
secure mechanism to expand connectivity of
edge-systems
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdkVortex Fog
VORTEX Fog creates a logical boundary around a subsystem and expose it to the “external
world” as if it was a single application
For this “equivalent” application we can control the Readers/
Writers to expose as well as the locators, the protocols and the
security
Vortex Fog
Rx
Outside View
Inside View
RyRzWx Wy
WzRx Wx
White List
- Read(x)- Write(x,z)
This slides have been crafted by Angelo Corsaro
Any use of these slides that does include me as Author/Co-Author is plagiary
location transparency
Cloud Computing
Fog Computing
Device-to-Cloud Communication
Device-to-Device Communication
Fog-to-Cloud Communication
Cloud-to-Cloud Communication
Device-to-Device Communication
Collect | Store | Analyse | Share
Collect | Store | Analyse | Share
Fog ComputingFog Computing
Vortex SecurityData-Level security
with Pluggable
Authentication Access Control and Crypto
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdk
Default Plug-ins
X.509 Public Key Infrastructure (PKI) based
authentication
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdk
Default Plug-ins
Access Control List available at a trusted/
authenticated URI
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdk
Default Plug-ins
Crypto based on TLS Cipher Suite
Device-2-DeviceDevice-2-Cloud
Fog-2-Cloud
Device-2-Fog
Cloud-2-Cloud
Fog-2-Fog
infra
structure
sdk
Boundary Security
Boundary security support is enabled by Vortex-Fog
Device-to-Cloud Communication
Peer-to-Peer (Broker-less)
Device-to-Device Communication
Fog Computing Fog ComputingFog Computing
TLS
TLS
Access Control
Boundary Security
Separates security concerns at different scales and controls
what information is exposed
Device-to-Cloud Communication
Peer-to-Peer (Broker-less)
Device-to-Device Communication
Fog Computing Fog ComputingFog Computing
TLS
TLS
Access Control
Example Deployments
Cop
yrig
ht P
rism
Tech
, 201
5
Connected CarsPowering Infotainment,
and driver assistance
This slides have been crafted by Angelo Corsaro
Any use of these slides that does include me as Author/Co-Author is plagiary
Vortex device, such as Lite, Café, and Web are used to share data between different kinds of applications within a car
Café and Web are typically used Android / HTML5 based infotainment
Lite in typically used in ECU, sensors and onboard analytics
This slides have been crafted by Angelo Corsaro
Any use of these slides that does include me as Author/Co-Author is plagiary
Vortex Fog is used to transparently (for in car apps) decouple and control the data sharing within and across the car
Vortex Fog also helps defining security boundaries and policies
This slides have been crafted by Angelo Corsaro
Any use of these slides that does include me as Author/Co-Author is plagiary
Vortex Fog efficiently and securely deals with car to car communication
Vortex Fog allows to decouple the transport used for in-car communication and that used for car-to-car communication
This slides have been crafted by Angelo Corsaro
Any use of these slides that does include me as Author/Co-Author is plagiary
Vortex Fog efficiently and securely deals with cloud connectivity adapting traffic flows and protocols
Device-to-Cloud Communication
Cloud Analytics
Fog AnalyticsFog Analytics
Fog Analytics
This slides have been crafted by Angelo Corsaro
Any use of these slides that does include me as Author/Co-Author is plagiary
Cloud Analytics
Fog AnalyticsFog Analytics
Fog Analytics
Device-to-Cloud Communication
Location TransparencyTechnology Abstraction
Data Federation (instances)
Data DeliveryQuery
History
Vortex virtualises data…
Cop
yrig
ht P
rism
Tech
, 201
5
Connected Boulevard — Nice, FranceVortex enables is the data sharing
platform for the City Operating System
Tens of thousands of devices connected by Vortex
Cop
yrig
ht P
rism
Tech
, 201
4
The smartest city in the world
IoE is Helping Nice Realize its Plan to Become a Smart City
Click to learn more:
Architecture
Autonomous Vehicles
coordination of fast moving autonomous
vehicles
intermittent connectivity
dynamic pairing of devices
Cop
yrig
ht P
rism
Tech
, 201
4
Architecture
Harvesters, combiners and other kinds of machinery communicate peer-to-peer to (1) exchange position to avoid crashing into each other, (2) agree on the division of labor to optimally harvest the field
Some telemetry data is pushed to Vortex Cloud to enable preventive maintenance other kinds of business intelligence
Cop
yrig
ht P
rism
Tech
, 201
4
Smart Factory0.5 TB of data produced per
day
Cop
yrig
ht P
rism
Tech
, 201
4
VORTEX provides ACME with a single technology for addressing OT as well as the IT requirements
ACME has been able to seamlessly integrate applications across Field, Process, Plant and Enterprise level eliminating the IT/OT integration challenges and promoting agile and extensible architectures
ACME Power Generation Platform
Cop
yrig
ht P
rism
Tech
, 201
4
ACME leverages Vortex Real-Time Cloud Messaging Technology to achieve horizontal integration
ACME Power Generation Platform
Additionally, ACME exploits VORTEX support for Fog and Cloud computing to ensures that the systems are minimally impacted by changes in connectivity
Cop
yrig
ht P
rism
Tech
, 201
4
ACME Deployment
Cloud Computing
Fog-to-Cloud
Communication
Cloud-to-Cloud
CommunicationDevice-to-Device
Communication
Collect | Store | Analyse | Share
Collect | Store | Analyse | Share
Fog Computing
Fog Computing
Fog Computing
Cloud and Fog Architecture
Cop
yrig
ht P
rism
Tech
, 201
4
At its core, Vortex-Cloud is made by two building blocks:
Discovery Service: Deals with transparently mediating and scaling the match between applications as well as establishing connectivity
Routing Service: Efficiently and scalable mediates the communication between DDS applications
Discovery and Routing Services
Discovery Service
Routing Service
Cop
yrig
ht P
rism
Tech
, 201
4
The Discovery service, is an elastic and fault tolerant service that:
Controls access to information
Matches DataReaders and DataWriters while minimising the discovery information propagated to end applications
Establish the optimal communication path between communicating end-points
Manages and load-balance the Routing Service instances
For a generic DDS application, the Discovery Service is completely transparent
Discovery Service
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
App1
DWX
DRY
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
LBApp1
DWX
DRY
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
App1
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
(DWX,DRY )
App1
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
(GUID1, DWX,DRY )App1
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
App2
DWY
DRX
App1 (GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
App1LB
App2(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
App2App1 (GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
(DWY,DRX ) App2App1 (GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
(GUID2, DWY,DRX )
App2App1 (GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
Match!
Match!
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
App2
DWA
DRB
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App2
LB
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App2(DWA,DRB )
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App3
(GUID3, DWA,DRB )App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App3
(GUID3, DWA,DRB )App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App3
(GUID3, DWA,DRB )App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App3
(GUID3, DWA,DRB )App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
LBApp1
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App3
(GUID3, DWA,DRB )App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )App1
(DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Discovery Service
Discovery Service
DWX
DRY
DWY
DRX
DWA
DRB
App3
(GUID3, DWA,DRB )App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )App1
Cop
yrig
ht P
rism
Tech
, 201
4
The Routing service, is an elastic and fault tolerant service that:
Efficiently routes information from producer to consumers
Provides configurable decoupling between producing and consuming applications
For a generic DDS application, the Routing Service is completely transparent
Routing Service
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
Match!
Match!
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
App1App2
(RS2,2, GUID1, GUID2)
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
(RS2,2)(RS2,2)App1
App2
(RS2,2, GUID1, GUID2)
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
(RS2,2, GUID1, GUID2)
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
(RS2,2, GUID1, GUID2)
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
(RS2,2, GUID1, GUID2)
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
(RS2,2, GUID1, GUID2)
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
(RS2,1, GUID1, GUID2)
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
(RS2,1, GUID1, GUID2)
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
(RS2,1)(RS2,1)
Cop
yrig
ht P
rism
Tech
, 201
4
Routing Service
Discovery Service Routing Service
DWX
DRY
DWY
DRX
App1App2
(GUID2, DWY,DRX )(GUID1, DWX,DRY )
(RS2,2, GUID1, GUID2)
Vortex Cloud and Fog allow to seamless grow existing DDS application to Internet Scale
For new applications, the Vortex platform provides extremely powerful abstractions at massive scale, thus it should be highly considered as the connectivity fabric
In Summary
