monitoring at scale over diverse data sources @ paypal ... · monitoring at scale over diverse data...
TRANSCRIPT
Monitoring at Scale over diverse data sources @ PayPal – Druid, TSDB,
Hadoop Grafanacon, December 1, 2016
Senthil Pandurangan - Platform Architect
Abhishek Sant – Senior Software Engineer
Agenda
©2016 PayPal Inc. Confidential and proprietary.
• Introduction
• Monitoring Platform Architecture
• Grafana Druid Plugin
• Q & A
PayPal Command Center
Powered by Monitoring Team
3
PayPal Monitoring Ecosystem - Sherlock
4
APIs
LogsProcessing
MetricProcessing
InsightsEngine
Alerts
Visualiza7on
EventsStream
MetricsStream
Topology
LogsStream
Pools&Hosts
PaaS
IaaS
Network
Databases
CAL(Logging)
Remedia7onOrchestra7onConfigura7on
Sherlock is a unified monitoring, alerting and logging platform that exposes real time data and insights, enabled by an extensible visualization platform.
Developers
Analysts
CommandCenter
SRE
Tools & Technologies
©2016 PayPal Inc. Confidential and proprietary.
Storage
Processing
Visualization
Ingest Queue Process Store Query
PayPal Monitoring Platform - Architecture
6
Message B
us
App
Logs
App
A
pp
Servi ces
DB
LB
N
O
SQ L
Met
rics
Stream processing
Met
rics
Logs
Log Views
Hadoop Reports
Custom Visualizations
Alerts
Agenda
©2016 PayPal Inc. Confidential and proprietary.
• Introduction
• Monitoring Platform Architecture
• Grafana Druid Plugin
• Q & A
Grafana Druid Plugin
https://grafana.net/plugins/abhisant-druid-datasource/installation
©2016 PayPal Inc. Confidential and proprietary.
Grafana - Adding Druid Data source
©2016 PayPal Inc. Confidential and proprietary.
Grafana Druid Plugin – Query Editor
©2016 PayPal Inc. Confidential and proprietary.
Druid Query Types
©2016 PayPal Inc. Confidential and proprietary.
Time Series
Top N
Group by
Select
Multiple Data sources
©2016 PayPal Inc. Confidential and proprietary.
Time Series
Top N
Group by
Select
TSDB
Druid
Hadoop Reports
Thank You
©2016 PayPal Inc. Confidential and proprietary.