how to get monitoring right for streaming & fast data systems built with spark, mesos, akka,...
TRANSCRIPT
WEBINAR
How To Get Monitoring Right For Streaming And Fast Data Systems Built With Spark,
Mesos, Akka, Cassandra and KafkaPaul Jasek, Senior Director of Global Solution Architects
Agenda1. Fast Data & Streaming Applications 2. The Challenges of Monitoring Fast Data Applications3. What To Look For In a Fast Data Application 4. Intelligent End-To-End Monitoring from Lightbend5. Live Demo 6. Questions
reactivemanifesto.org
Reactive Underpinnings: Fast Data and streaming applications often incorporate, or are based on, Reactive principles
• Real-timepersonalization
• Real-timedecision-making
• IoTdataprocessing
• Legacybatchprocessing
modernization
Growing Number Of Use Cases Across Industries
• Serveexistingcustomersbetterandreducechurn
• Attractnewonesanddrivegrowth
• Launchnewproductsmoreeasily
• Enternewmarketsmorequickly.
• Rapidly Evolving Ecosystem • Understanding the Data Pipeline • Dynamic Architectures • Intricately Interconnected • Distributed And Clustered
The Challenges of Monitoring Fast Data Applications
Apache Spark, As An Illustrative ExampleThe Challenges of Monitoring Fast Data Applications
Concern QuestionsToAsk
DataHealth(foragivenapplication)
• Throughput:isdataprocessingoccurringattheexpectedrate?• Latency:isdataprocessingoccurringwithintheexpectedtimeframe?• Error/quality:arethereproblemswiththedatabeingproduced?• Inputdata:areinputdatastreamsflowingintoSparkbehavingnormally?Forinstance,whatarethe
throughputratesforKafkatopicsfeedingintotheSparkjob?
DependencyHealth • Arethesystemsfeedinginputintothestormjob(suchasKafka)healthy?• Arethesystemsthattheapplicationisdependenton,suchasMemcacheorotherAPIendpoints,
healthy?
ServiceHealth • IstheSparkmasteroperatingnormally?Ifnot,engineeringwillbeunabletore-balanceworkloadsorrestartjobs.
ApplicationHealth • AretheapplicationKPIswithinnormaloperatingparameters?
TopologyHealth • ArethereresourcesassignedtothegivenSparktopology?• •AretheSparktasksandexecutorswell-distributedamongsttheSparkcluster?• •Aretheperformancecounters(emitted,failed,latency,etc.)forthegivenSparktopologynormal?
NodeSystemHealth • Arethekeysystemmetrics(load,CPU,memory,net-i/o,disk-i/o,diskfree)operatingnormally?
Can traditional monitoring tools help?
Why traditional monitoring tools won’t help you
• Builttomonitormonolithicapplications
• Canonlybeusedtoextractmetricsandtraceinformationbasedonasynchronousflow
• Notbuiltforasynchronousflows(i.e.inFastDataandstreamingapplications)
• Cannoteasilyhandlestreamingsystemsrunningondistributedclusters
• Deep Telemetry
• Domain Expertise
• Automated Discovery
• Real-Time Topology Visualization
• Intelligent, Rapid Troubleshooting
What users need to effectively monitor Fast Data and streaming applications
• Lightbend Monitoring takes a modern approach to instrumenting and
visualizing distributed streaming systems
• Helps users not just in production but also in development (so they can
build their applications right from Day 1)
• Shows the end-to-end status of applications, data frameworks, and the
associated infrastructure in a single view.
Intelligent, End-To-End Monitoring
• Deep Telemetry
• Domain Expertise
• Intelligent Anomaly
Detection
• Fine-Grained
Visibility, with Drill-
Down Capabilities
Data-Science Driven Anomaly Detection
• Automated Topology
Discovery
• Automatic Metric
Collection
• Real-Time Topology
Visualization
Automated Discovery, Configuration & Topology Visualization
• Single Pane of Glass
Visibility
• Rapid Root Cause
Analysis
• Reduced Mean-Time-
To-Repair (MTTR)
Intelligent, Rapid Troubleshooting
• Dramatically reduce the time and cost to identify and remediate issues across application life-cycle.
• Create happier, more satisfied customers – and lower churn
• Lower HW/infrastructure costs and reduce concerns about chargebacks & SLA penalties
• Deliver rapid time to value because everything you need for monitoring is packaged into an easy-to-use solution
Benefits for your business
On to the demo…
Upgrade your grey matter!Get the free O’Reilly book by Dr. Dean Wampler,
VP of Fast Data Engineering at Lightbend
bit.ly/lightbend-fast-data
End-To-End Monitoring For Your Fast Data And Streaming Applications From Lightbend
SET UP A 20-MIN DEMO