brian knox, principal architect, talksum - 21st bdl meetup
DESCRIPTION
"Cross-domain Data Management" Brian Knox is the principal architect at Talksum where he focuses on real-time data routing and analysis. Brian has over 20 years of experience as a systems/data architect with expertise in high-speed messaging, service delivery, operations management, systems integration, and distributed processing. Most recently, Brian has served in the roles as principal architect and database engineer for large-scale infrastructures and service delivery companies, including AOL. In addition, he has also focused on large-scale filtering technologies and data management for other successful start-up companies such as SurfControl and others.TRANSCRIPT
![Page 1: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/1.jpg)
1 Confidential Information of Talksum, Inc.
I’m Brian!
Who Is This Guy?
![Page 2: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/2.jpg)
2 Confidential Information of Talksum, Inc.
• Principal Architect at Talksum
•Open Source Contributor ZeroMQ Rsyslog
Who Is This Guy?
![Page 3: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/3.jpg)
3 Confidential Information of Talksum, Inc.
Talksum Data Stream Router
Next Age of Data Management
November 2013
![Page 4: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/4.jpg)
4 Confidential Information of Talksum, Inc.
The Talksum Data Stream Router
Apache Common Logging – FilesSNMP - UDP
Unix Logs – RFC3164 UDP/TCPNetflow – UDP – NG v.5, 8, 9, 10
Patient Records (HL7) XML/ASN.1Transportation (BSM) SAE J2735
I2C, CAN, SNMP, Serial
XML, JSON, File, HTTP REST
Twitter, RSS, CAP (Weather Alerts)
Refined Data Stream
Refined Data Stream
Refined Data Stream
Indexed, Mapped, ReducedOrdered, Sorted Data Streams
Bulk Data Streams(Lightly Ordered
and Filtered)
TalksumData Stream
Router(TDSR)
• Data Normalization• Parsers• Filters• Metrics and
Counts• Inline ETL/PTL• Asynchronous
Outputs• Protocol
Verification
Customer A:Summarized Data
SystemLogs
ApplicationData
Sensor andIndustrial
Data
3rd Party DataB2B/M2M
Social andPublic Data
Customer B:Aggregated Data
Customer C:Dynamic Stream
ApplicationLogs
• SQL Warehouse• Bulk Data Stores• File Storage
• Object Data Stores• Indexed Data Caches• NoSQL Data
Warehouses
![Page 5: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/5.jpg)
5 Confidential Information of Talksum, Inc.
We turn alphabet soup into coherent streams of information so that you can get to work solving problems.
In Other Words…
![Page 6: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/6.jpg)
6 Confidential Information of Talksum, Inc.
Cross Domain Routing with ZeroMQ
November 2013
![Page 7: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/7.jpg)
7 Confidential Information of Talksum, Inc.
“A territory over which rule or control is exercised”
“A sphere of activity, concern, or function”
- freedictionary.com
Domains
![Page 8: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/8.jpg)
8 Confidential Information of Talksum, Inc.
•Energy
•Transportation
•Biomedical
•Information Systems
Some Domains!
![Page 9: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/9.jpg)
9 Confidential Information of Talksum, Inc.
•Application Threads
•Application Processes
•Systems on a network
•Networks themselves
Some More Domains!
![Page 10: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/10.jpg)
10 Confidential Information of Talksum, Inc.
ØMQ \zeromq\:
• A network socket library that acts as a concurreny framework
• Carries messages across multiple transports
• Enables networking patterns
• Provides Asyncronous I/O
• Large and active open source community
• Bindings for 40+ programming languages
• Works on most operating systems
What is ZeroMQ
![Page 11: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/11.jpg)
11 Confidential Information of Talksum, Inc.
What is ZeroMQ
WHAT IF I TOLD YOU
WE CAN HAVE LOCK-FREE IN MEMORYMESSAGE PASSING BETWEEN THREADS?
Here there be dragons
![Page 12: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/12.jpg)
12 Confidential Information of Talksum, Inc.
What Is ZeroMQ To Us
ZeroMQ is a tool which facilitates the construction of cross domain communication patterns
![Page 13: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/13.jpg)
13 Confidential Information of Talksum, Inc.
When we can apply a pattern from one domain to a different domain, we can do more with less work.
Patterns!
![Page 14: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/14.jpg)
14 Confidential Information of Talksum, Inc.
Network Monitoring & Optimization
Unix Logs – RFC3164 UDP/TCPNetflow – UDP – NG v.5, 8, 9, 10
Refined Data Stream
Refined Data Stream
Indexed, Mapped, ReducedOrdered, Sorted Data Streams
Bulk Data Streams(Lightly Ordered
and Filtered)
TalksumData Stream
Router(TDSR)
• Data Normalization• Parsers• Filters• Metrics and
Counts• Inline ETL/PTL• Asynchronous
Outputs• Protocol
Verification
Existing BI Tools
SystemLogs
NOC Alerting
• SQL Warehouse• Bulk Data Stores• File Storage
• Object Data Stores• Indexed Data Caches• NoSQL Data
Warehouses
Customer: Large European ISP/Email Communications ProviderUse Case: Ingest Netflow data, parse and aggregate in real time, monitors and alerts, optimize network topology Status: Deploying beta appliance
![Page 15: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/15.jpg)
15 Confidential Information of Talksum, Inc.
Financial
XML, JSON, File, HTTP REST
Twitter, RSS, CAP (Weather Alerts)
Refined Data Stream
Refined Data Stream
Indexed, Mapped, ReducedOrdered, Sorted Data Streams
Bulk Data Streams(Lightly Ordered
and Filtered)
TalksumData Stream
Router(TDSR)
• Data Normalization• Parsers• Filters• Metrics and
Counts• Inline ETL/PTL• Asynchronous
Outputs• Protocol
Verification
Alerting & Notification
3rd Party DataTrading Desks
Social andPublic Data
Market Dashboard
• SQL Warehouse• Bulk Data Stores• File Storage
• Object Data Stores• Indexed Data Caches• NoSQL Data
Warehouses
Customer: Major Financial Stock ExchangeUse Case: Ingest unstructured financial market data, parse and filter for quality, aggregate, integrate with existing data warehouseStatus: Acquiring data sample for POC
![Page 16: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/16.jpg)
16 Confidential Information of Talksum, Inc.
Request Reply
![Page 17: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/17.jpg)
17 Confidential Information of Talksum, Inc.
Round Robin Load Balancing
![Page 18: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/18.jpg)
18 Confidential Information of Talksum, Inc.
Unicast
![Page 19: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/19.jpg)
19 Confidential Information of Talksum, Inc.
Multicast
![Page 20: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/20.jpg)
20 Confidential Information of Talksum, Inc.
Fair Queuing
![Page 21: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/21.jpg)
21 Confidential Information of Talksum, Inc.
Explicit Routing
![Page 22: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/22.jpg)
22 Confidential Information of Talksum, Inc.
•Within a thread
• Between threads
• Between processes on the same system
• Between processes on different systems
Inter Domain Messaging
![Page 23: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/23.jpg)
23 Confidential Information of Talksum, Inc.
Why Talksum Contributes to Zeromq
• We are all facing a lot of problems. • We believe in the value of empirical evidence.• Solutions emerge from all of us working together
across our various domain boundaries.• Participate!
![Page 24: Brian Knox, Principal Architect, Talksum - 21st BDL meetup](https://reader038.vdocument.in/reader038/viewer/2022103013/5404edaa8d7f729e768b499a/html5/thumbnails/24.jpg)
24 Confidential Information of Talksum, Inc.
• http://www.zeromq.org/ - Main ZeroMQ Web Site
• http://zguide.zeromq.org/ - ZeroMQ User Guide
• http://github.com/zeromq - ZeroMQ Git Repository
• http://api.zeromq.org/ - ZeroMQ Reference Manual
• http://zeromq.org/whitepapers:architecture – Architecture Whitepaper
• http://czmq.zeromq.org/ - ZeroMQ C Binding
• http://zeromq.github.io/pyzmq/ - ZeroMQ Python Binding
Resources