How Comcast Turns Big Data into Real-Time Operational InsightsRaanan Dagan, Big Data Solutions, Splunk

Patrick Shumate, CDN Engineering, Comcast

Copyright © 2012 Splunk Inc.

What We’ll Talk About

Supporting the Anytime, Anywhere Network

Splunk and Big Data

Comcast’s Universal Database Initiative

Going for Gold – the London Olympics

2

• Company– Founded 2004, first software release in

2006– HQ: San Francisco, CA– Regional HQs: Hong Kong, London– Over 600 employees, in 8 countries

• 4,400+ Enterprise Customers– Customers in over 80 countries– 54 of the Fortune 100

• One of nation's leading providers of entertainment, information & communications products and services

The Comcast Cable Team

4

ProductEngineering

ProductApplication

Services

VideoSystemServices

CDNEngineering

CDN Engineering: Software Development, Selection and Management Across Services

Search VSS: Centralized machine data collector for real-time monitoring, analytics, event correlation, reporting and dashboards

5

Supporting an Anytime, Anywhere Network

6

The Challenge

Comcast – UDB Before Splunk

7

Turning This

8

To These

Requirements for Universal Database

9

Universal Database(UDB)

• High volume of data from many systems along a complex workflow

• Developers expressing artistic prerogative on log formats

• Many different data sources and formats

• Drive operational intelligence• Improve user experience• Troubleshooting, root cause analysis• Track and measure success• Reports, alarms

Caller ID Metadata Distribution

STB Menus

MenuEntitlement

Input Requirements Output Requirements

Big Data Comes from MachinesVolume | Velocity | Variety | Variability

GPS,RFID,

Hypervisor,Web Servers,

Email, MessagingClickstreams, Mobile,

Telephony, IVR, Databases,Sensors, Telematics, Storage,

Servers, Security Devices, Desktops

Machine-generated data is one of the fastest growing, most complex

and most valuable segments of big data

10

What Does Machine Data Look Like?

11

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Machine Data Contains Critical Insights

12

Order ID

Customer’s Tweet

Time Waiting On Hold

Product ID

Company’s Twitter ID

Sources

Twitter

Care IVR

Middleware Error

Order Processing

Order ID

Customer ID

Twitter ID

Customer ID

Customer ID

Splunk: The Platform for Machine Data

13

Insight and Visualizations for Executives

Statistical Analysis

Proactive Monitoring

Search and Investigation

Machine Data Operational Intelligence

Splunk storage - Hadoop

Customer Facing Data

Outside the Datacenter

ApplicationsWeb logsLog4J, JMS, JMX.NET eventsCode and scripts

NetworkingConfigurationssyslogSNMPnetflow

DatabasesConfigurationsAudit/query logsTablesSchemas

Virtualization & Cloud

HypervisorGuest OS, AppsCloud

Linux/UnixConfigurationssyslogFile systemps, iostat, top

WindowsRegistryEvent logsFile systemsysinternals

Logfiles Configs Messages Traps Alerts

Metrics Scripts TicketsChanges

Click-stream dataShopping cart dataOnline transaction data

Manufacturing, logistics…CDRs & IPDRsPower consumptionRFID dataGPS data

Splunk Collects and Indexes Machine DataNo upfront schema. No RDBMS. No custom connectors.

14

• Refine transactions into readable logs

• 10s TBs of multi event, multi-line transactions

Universal Database Use Case

Forwarder

Splunk visualize and report on Hadoop dataUDB

15

Before Splunk• 100G of data - monitoring and responding to errors cumbersome

and prone to false positives• KPI extraction near impossible

16

UDB After Splunk

17

“Universal Database” Video back office

Pipe the access logs into SplunkFind the errorsBuild the alarmsDefine the KPIBuild the dashboards!

Splunk Has Four Primary Functions

Searching and Reporting (Search Head)

Indexing and Search Services (Indexer)

Local and Distributed Management (Deployment Server)

Data Collection and Forwarding (Forwarder)

A Splunk install can be one or all roles…

18

Splunk Components and Scalability

Send data from 1000s of servers using combination of Splunk Forwarders, syslog, WMI, message queues, or other remote protocols

Auto load-balanced forwarding to as many Splunk Indexers as you need to index terabytes/day

Offload search load to Splunk Search Heads

19

Analyzing Heterogeneous Data

No data normalizationAutomatically handles timestampsParsers not requiredIndex every term & pattern “blindly”No attempt to “understand” up front

Normalization as it’s neededFaster implementationEasy search languageMultiple views into the same data

Knowledge applied at search-timeNo brittle schema to work aroundMultiple views into the same dataFind transactions, patterns and trends

Universal Indexing

Late Structure Binding

Analysis and Visualization

Rapid time-to-deploy: hours or days

20

Real-time Analytics

DataPa

rsin

g Q

ueue

Parsing Pipeline• Source, event typing• Character set

normalization• Line breaking• Timestamp identification• Regex transforms

Indexing Pipeline

Real-time Buffer

Raw dataIndex Files

Real-time Search Process

Monitor Input

Inde

x Q

ueue

TCP/UDP Input

Scripted Input SplunkIndex

21

Splunk Search Processing LanguageLots of random “hypothetical examples” from our Mugs

22

Operational Intelligence for IT and Business UsersWeb Intelligence

Application Management Business Analytics

Security & Compliance

LOB Owners/Executives

CustomerSupport

SystemAdministrator

IT Operations Management

OperationsTeams

SecurityAnalysts

IT Executives

Development Teams

Auditors

Website/BusinessAnalysts

23

Better Interoperability Drives Time-to-value

24

Splunk Hadoop ConnectReliable Data ExportImport Hadoop Data

Splunk App for HadoopOpsEnd-to-end monitoring, troubleshooting , analysis of Hadoop environment

>>>>

Real-time Collection and

Analysis

Dashboards, Reports,

Access Controls

>>

25

Splunk Hadoop Connect

Delivers reliable integration between Splunk and Hadoop

Export events collected and aggregated in Splunk to HDFSExplore and browse HDFS directories and filesImport and index data from HDFS for secure searching, reporting, analysis and visualizations in Splunk

Splunk App for HadoopOps

26

End-to-end monitoring and troubleshooting for Hadoop

Monitoring of entire Hadoop environment (Network, Switch, Operating System and Database)

Integrated alerting to track and respond to activities from MapReduce to the individual node in the cluster

Centralized real-time view of Hadoop nodes using intuitive heatmap display

Splunk Big Data Solution

Product-basedSolution

Performance at scale

Integrated and End-to-end

Easy to download and deploy

Pre-integrated, end-to-end functionality

Enterprise-grade features

Proven at multi-terabyte scale per day

Upwards of PB under management

4,000+ customers

Collects data from tens of thousands of sources

Advanced real-time and historical analysis of data

Fast, custom visualizations for IT and business users

Developer APIs SDKs

27

Splunking NBC Olympics Coverage

28

24x7Coverage

1,700 Assets

245Event Replays

219M Americans watched NBC's Olympics coverage

27.5MVOD Views

Data Splunked 24 hours a day for 21 Days during Olympics

Search VSS: Primary fault detection, alarming and reporting console for all Olympic content

NBC Olympics - Results

29

Content Management Team

NBC Olympics - ResultsOn Demand-Online• Real-time watch lists for active content

– How many customers watching what– Impact of Editorial promotion– “viral” content

• CDN Management– Finding, reporting, monitoring vendor

bugs• CDN Capacity Planning

– Monitoring throughput– Cache capacity evaluation– Time-to-serve monitoring

30

Combine technologies to deliver better results – faster

Use Hadoop for batch processingUse Splunk for real-time processing

31

Comcast – Key Takeaways

Summary - Splunk Big Data Solution

Product-basedsolution

Performance at scale

Integratedend-to-endreal-time

32

Come to the Splunk booth to see a demo of new Splunk-Hadoop integrations

Copyright © 2012 Splunk Inc.

Thank You

splunk.com/bigdata