exclusive verizon employee webinar: getting more from your cdr data
TRANSCRIPT
Verizon & Big Data: Getting More from CDR Data
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
The program will begin shortly. Please listen to the webinar with your computer speakers turned on.
John Michael Brack, Manager - Northeast Region, Pentaho Bo Borland, Head of System Engineers, Pentaho
Agenda
• Project Profile
• Pentaho Overview
• Big Wireless Analytics
• Call Detail Records (CDR) dashboard and analysis
• Retail Sales Reporting, IR, Mobile and Analysis
• DW Optimization
• Rescuing CDR data from tape archive with Hadoop
• Pentaho for Hadoop (ingestion, map reduce, orchestration, analysis)
2 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
Verizon Project Profile
Audience: Business, Executives, Management, IT, Architects Company: Verizon Communications Inc. was founded in 1983 and is based in New York, New York and has over 180,000 associates and 115 B in revenue. Verizon Communications Inc., through its subsidiaries, provides communications, information and entertainment products and services to consumers, businesses, and governmental agencies worldwide. Its Verizon Wireless segment offers access to various wireless voice and data services comprising Internet access through smart phones and basic phones, and notebook computers and tablets; messaging services, which enable customers to send and receive text, picture, and video messages; and consumer-focused and business-focused multimedia applications. . Goals:
• Self Service Business Analytics • Better TCO • Data Integration • Self Service Powerful Reporting • Usability and connectivity • Self Service Dashboards • Time to Market • Access to Big Data and Analytical DB
3 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
A modern, unified embeddable platform built for the future of analytics, including big data and cloud-ready analytics
4 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
CENTRAL ADMINISTRATION, AUDITING & MONITORING
DELIVER When & Where Users Need It
STREAMLINE Information Delivery
VISUALIZE & Report Information In Any Style
ACCESS All Enterprise Data Sources
ISV & Packaged Applications
SaaS / Cloud Applications
EMBEDDED
Web
Mobile
STANDALONE
‣ Advanced & Predictive Analytics
DATA MINING
‣ Interactive ‣ Operational
‣ Enterprise
REPORTING
‣ Ad hoc Exploration ‣ Multi-Dimensional
ANALYSIS
‣ Interactive Metrics ‣ Rich Visualizations
DASHBOARDS
ERP / CRM / Enterprise Apps (e.g. SAP, Oracle)
Hadoop & NoSQL Data
Unstructured & semi-structured (XML, Excel, Files, etc.)
Relational Data Sources
Cloud (e.g. Salesforce, Amazon, Dell)
‣ Direct Access
‣ Data Integration
‣ Hadoop Clustering
‣ Graphical ETL Designer
‣ Enterprise Scalability
INTEGRATE, CLEANSE, & ENRICH DATA
‣ In Memory Caching
‣ High Performance
‣ Relational OLAP Cubes
METADATA LAYER
Over 1500 Customers Across All Industries
5 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
Skype – Sales & financial reporting with plans to analyze user stats to improve performance monitoring.
Comcast – Data integration for “single version of the truth” and new BI initiative to enable self service reporting/analytics for business analysts.
Nuance – Use Pentaho to join and transform data from mobile device usage logs to perform complex data analysis.
Bell Canada – Optimize project and consulting resources by being able to view all project activity across multiple teams (and acquisitions –all different data sources).
Pentaho for Hadoop Large Financial Institution
6 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 6 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
Why Pentaho • Ability to load disparate data sources into
HDFS and Hbase
• Ability to load post-processed data into DB2
• Ability to interface with caching and message queue technologies via customer-specific Java libraries
Business Challenge To gain competitive advantage through intraday balance reporting for commercial customers.
Pentaho Benefits • Lowers technical barriers by providing an easy to use ETL
environment for designing MapReduce jobs without having to write code
• Provides a graphical orchestration environment for Hadoop, HBase and DB2 data integration workloads
• Processes Client, Account, Reference, Transaction and Balance information at the lowest level of granularity possible
Forrester Enterprise Hadoop Solutions Wave Highest-Scored Analytics Vendor
7 © 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
Pentaho BIG Data & Data Integration Walkthrough
Bo Borland
Head of System Engineers
8 © 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
Pentaho Platform Design Drivers
1. Big data is changing the world
2. Open systems are more innovative
3. Subscriptions models reduce cost and risk
4. Simplicity empowers the masses
5. Pluggable java architectures enables flexibility and competitive advantage
6. Enterprise-wide integration reduce cost and complexity
7. Predictive technologies are next big thing in analytics
9
Big Wireless
10 © 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
3 Calling Plans
• Nationwide • PAYG • Prepaid 50
2 Business
Units
• B2B • B2C
7 Retail Stores
7 Product Lines
3 Websites
Big Wireless– Wireless Carrier
• San Francisco • Boston • NYC • Paris • Tokyo • Sydney • London
• Smartphones • Home Phones • Wifi Devices • Modems • Notebooks • Tablets • Accessories
• Ecommerce Site • Reseller Portal • Manufacturer Portal
Store Managers
Executives & Product Managers
Operations and Store Employees
Marketing & Customer Support
B2B Sales Organization
Databases
Call Detail Records
Retail Sales
Website Clickstream
Website User Registration
2013 Performance Goals
12
Increase subscription revenue
Improve store profitability
Eliminate inventory stock outs
Leverage big data to maximize profits
Profile and target profitable customers
Improve supply chain visibility for partners
2013 Performance Goals
13
Goals Objectives Enablers Increase
subscription revenue
Analyze call data to upsell PAYG customers to subscriptions
Improve store profitability
Hold store managers accountable by pushing store income statements to
Eliminate inventory stock outs
Empower store employees with iPads and real-time inventory reports
Profile and target profitable customers
Profile mobile plan customers with high average call duration
Leverage big data to maximize profits
Analyze e-commerce clickstream data in MongoDB to profile purchasing
users and predict users propensity to purchase.
Improve supply chain visibility for
partners
Give phone manufacturers and resellers web access to secure sales
reports
3 Calling Plans
• Nationwide • PAYG • Prepaid 50
2 Business
Units
• B2B • B2C
7 Retail Stores
7 Product Lines
3 Websites
Enterprise-Wide Analytics
10 Resellers
10 Phone Manf
Red River Mobile
• San Francisco • Boston • NYC • Paris • Tokyo • Sydney • London
• Smartphones • Home Phones • Wifi Devices • Modems • Notebooks • Tablets • Accessories
• Ecommerce Site • Reseller Portal • Manufacturer Portal
EXTERNAL INTERNAL
IFrame Integration
Custom Widget Embedding
Big Data
15 © 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555
Mobile Network Provider
Call Detail Records (CDR) • Mobile networks generate vast amounts of daily call data • CDR tracks every voice, SMS, or location service • 2 years of detailed CDR records in DW • Archived to tape after 2 years
Data Sources Data Warehouse Architecture
Data Warehouse (Master & Transactional Data)
ERP
CRM
CDR
Analytic Data Mart(s)
Analytic Data Mart(s)
Analytic Data Mart(s)
Tape Archive
Current Data Warehouse Architecture
Data Sources Data Warehouse Architecture
Data Warehouse (Master & Transactional Data)
ERP
CRM
CDR
Analytic Data Mart(s)
Analytic Data Mart(s)
Analytic Data Mart(s)
Tape Archive
With Current EDW Architecture With Hadoop
EDW stores only 2 years of data à Hadoop active archive for all history
Infrastructure at capacity à Frees EDW capacity for high value data
Expensive to scale à Lowers cost and inexpensive to scale
ETL process complex and slow à Streamlined ingestion of raw data
Only analyze 2 years of data à Analyze 10 years of data
Data Warehouse Optimization
Data Sources Big Data Architecture
Data Warehouse (Master & Transactional Data)
ERP
CRM
CDR
Analytic Data Mart(s)
Analytic Data Mart(s)
Analytic Data Mart(s)
Logs Logs
Other Data
Raw Data
Parsed Data
Analytic Datasets
Master Data
Tape Archive
ORCHESTRATE
ERP DW
Processing
CRM
Pig, Oozie, Flume, Hive, Hbase, Sqoop
Raw Data
Parsed Data
Analytic Datasets
Pentaho Analytics for Hadoop
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 19
Master Data
Analysis & Reporting
ANALYZE
Unstructured Data
Structured Data
INGEST
Ingestion
VISUAL MAP REDUCE
Data Integration Analytics
Raw Data
Ingest Raw and Master Data
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 20
Master Data Unstructured
Data
Structured Data
INGEST
Ingestion
Processing
Raw Data
Parsed Data
Analytic Datasets
Visual Map Reduce
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 21
Master Data
VISUAL MAP REDUCE
1. Map Reduce Input – calling data
2. Calculate Month, Day, Day of Week
3. Extract 3 digit area code
4. Lookup geo master data in HDFS
5. Filter for weekend and US only calls
6. Create “Value” field for Key-Value Pair
7. Create “Key “ field for Key-Value Pair
8. Map Reduce Output – Key-Value Pair
Java Programing
Data Agnostic & Data Orchestration
© 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 22
Pentaho Data Integration
Parsed Data
Analytic Datasets
Hadoop Data Analysis
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 23
Analysis
ANALYZE
• Analyze 10 years of call data by geography, time zone for US only calls made on the weekend. – Understand annual growth rates
– Which geographies are driving highest call volume growth rates?
ORCHESTRATE
Raw Data
Parsed Data
Analytic Datasets
Pentaho Big Data Demonstration
© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 24
Master Data
ANALYZE
INGEST
VISUAL MAP REDUCE Ingest CDR data into Hadoop 1
Execute Map Reduce to enrich CDR data 2
Create and load a Hive table with the map
reduce results 3
Analyze 10 years of call data 4
© 2012, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 25
Thank You
Join the conversation. You can find us on:
blog.pentaho.com @Pentaho Facebook.com/Pentaho Pentaho Business Analytics