monetizing big data at telecom service providers
DESCRIPTION
TRANSCRIPT
Monetizing Big Data at Telecom Service Providers
Juergen UrbanskiTech Alpha
Hadoop Makes Shareholders Happy
The World's Largest Telcos are Driving Business Performance with Hadoop at the Center of an Enterprise-Wide Modern Data Architecture
Juergen Urbanski CEO, Tech AlphaBoard Member Big Data & Analytics, BITKOM (German IT Industry Association)
Agenda
• Telco Data Management Challenges
• Hadoop Business Value
• Data Lake Business Value
• Data Lake Reference Architecture
• 21 Telco Use Cases for Hadoop
– Network Infrastructure
– Service and Security
– Sales and Marketing
– New and Adjacent Business
3
Enterprise Data Management Challenges
Limited Insight:• Schema On Write• Data In Silos
Limited Scale:• Not Designed to Scale• Not Affordable at
ScalePhysicalInfrastructure
Presentation &Application
Data Access
Data Management
EngineeredSystems
Shared Storage Systems
OLTP OLAPTraditionalAnalytics
=
=
– 4 –
Business Value of Hadoop
Data Access Layer
Data Management Layer
Hadoop Core Capabilities:
Broader Insights:• Allows simultaneous access by and
timely insights for all your users across all your data
• Irrespective of the processing engine, analytical application or presentation
• Enabled by schema on read and enterprise-wide pool of data
Unlimited Scale:• Allows to acquire all data in its original
format and store it in one place, cost effectively and for an unlimited time
• Affordable and performing well into the 100+ petabyte scale
=
=
– 5 –
A New Approach for Broader Insights
HADOOPIterate over structure
Transform and analyze
Hadoop Approach• Apply schema on read• Support range of access patterns to
data stored in HDFS: polymorphic access
Batch Interactive Real-time
Right Engine, Right Job
In-memory
Traditional Approach• Apply schema on write• Heavily dependent on IT
Determine list of questions
Design solution
Collect structured data
Ask questions from list
Detect additional questions
Single Query EngineSQL
– 6 –
Compelling Economics Allow Scale
0 5 10 15 20 25 30 35 40
SAN
EDW / MPP
Engineered System*
NAS
HADOOP
Cloud Storage
Min
Max
Fully Loaded Cost per Raw TB Deployed
US$ ‘000s
Hadoop Provides Highly Scalable Data Storage at 5% of the Cost of Alternatives
36 to 180
20 to 80
12 to 18
10 to 20
0.250 to 1
0.1 to 0.3
* E.g., Oracle Exadata
– 7 –
5 Capabilities of Hadoop 2.x Enable the Data Lake
– 8 –
Data Integration & Governance
Integrate with existing systems.
Move data into, within and out of the environment
Security
Provide layered approach to
security
Operations
Deploy and manage a
multi-tenant, environment easily, using existing tools
where possible
Environment and Deployment Model
Run anywhere
Data Lake Functional Requirements
1 32
4
Data Access = Insight
…ask questions later (or in the moment)
Data Management = Scale
Store first…
Presentation & ApplicationEnable existing and new
applications
5
Data Lake Reference Architecture
– 9 –
DeploymentModel
Environment
Data Integration & Governance
Data Access
Security Operations
Data Management
Storage: HDFS (Hadoop Distributed File System)
Multitenant Processing: YARN(Hadoop Operating System)
Online
HBaseAccumulo
Real-Time
Storm
Others
Commodity HW
Linux Windows
Appliance
On Premise Virtualize
Cloud/Hosted
AuthenticationAuthorizationAccountabilityData Protection
acrossStorage: HDFS
Resources: YARNAccess: Hive, … Pipeline: Falcon
Cluster: Knox
Provision, Manage & MonitorAmbari
SchedulingOozie
Data WorkflowData Lifecycle
Falcon
Real-time and Batch Ingest
FlumeSqoop
WebHDFSNFS
Batch
MapReduce
Script
Pig
SQL
Hive
In-memory
Spark
Metadata ManagementHCatalog
Presentation & Application
Multiple Use Cases and Tools Run on Hadoop as a Shared Service
– 10 –
Hadoop 2.x: Shared Service = Data Lake
Hadoop 1.x:Dedicated Project Silos = Data Ponds
BU2 BU3BU1
Customer Intimacy
HbaseOpera-tional
Excellence
Lucene New BusinessStorm
Risk Manage-
ment
Map-Reduce
BU4
Customer Intimacy
Hbase
Opera-tional
Excellence
Lucene
New Business
Storm
Risk Manage-
ment
Map-Reduce
Enterprise-wide
• Poor resource management
• Limited governance
• Batch processing, no streams
Shared service operational benefits similar to infrastructure cloud Speed of provisioning and de-provisioning for capacity and users Fast learning curve and reduced operational complexity Consistent enforcement of data security, privacy and governance Optimal capital efficiency driven by scale and load balancing
Value grows exponentially as data from more applications lands in one Hadoop 2.x data lake Marginal cost of retaining data is less than marginal value Able to run a broader range of analyses More data in one place usually leads to better answers Results is order-of-magnitude better insights
Data Lake Business Rationale
– 11 –
Technical and Business Drivers
– 12 –
Foundation for a modern data architecture
New data types
Sensors
Machine Generated
Geolocation
Documents, Email,
Voice to Text
Social Networks
Web Logs,Click Streams
Operational excellence E.g., Network Maintenance
Compliance & Risk Mgt.E.g., Fraud Reduction
Customer Intimacy
E.g., 360o View of Customer
New BusinessE.g., Data as a
Product
Business drivers
Network capacity planning Network upgrades Network maintenance Network performance management Network traffic shaping
21 Telco Use Cases for Hadoop
– 13 –
Use Case
Network Infrastructure
Function
Customer experience analytics Contact center productivity Field service productivity Data protection and compliance End-user device security
Service and Security
360-degree view of customer value Personalized marketing campaigns Upselling and cross-selling Next-product-to-buy (NPTB) Churn reduction
Sales and Marketing
New product development Actionable intelligence serving:
Advertisers Merchants/retailers Payment processors Federal governments Local governments
New and Adjacent Business
NetworkCareSales
New Biz
Hadoop in Network Infrastructure
– 14 –
Business Problem
Network capacity planning Network upgrades Network maintenance Network performance management Network traffic shaping
Hadoop is used to optimize the rollout of 4G coverage in time and space to match the likely pick-up in service revenue, allowing an operator to defer more than 10% of capex for the same resulting revenue.
Hadoop helped detect that only a small number of congested cable network nodes were responsible for the majority of churn, and could thus be prioritized for maintenance and upgrades.
Network function virtualization, software defined networking and unified all IP networks vastly increase the amount of machine and log data relevant for trouble shooting. Hadoop helps with root cause analysis and may even be used to reason on the data in real-time.
Value Realized
NetworkCareSales
New Biz
Network Infrastructure – Network Capacity Planning
– 15 –
Business Problem
The consumption of services and resulting bandwidth in a particular neighborhood may be out of sync with a telco’s plans to build new towers or transmission lines in that same neighborhood.
This leads to a mismatch between expensive infrastructure investments and the actual revenue from those investments.
Examples:
4G (LTE)
FTTC (fiber to the curb)
FTTH (fiber to the home)
One European carrier used Hadoop to optimize the rollout of 4G coverage in time and space to match the likely pick-up in service revenue, based on detailed cell tower traffic data of the last few years.
With their prior, less informed approach, they would have had to spend 10% more capex for the same outcome.
Value Realized
NetworkCareSales
New Biz
Network Infrastructure – Network Upgrades
– 16 –
Business Problem
Hadoop is used for targeted network maintenance and upgrades by cable companies.
One large US cable MSO was unsure how cable network congestion affects churn, and where exactly network upgrades produce the most incremental revenue.
The result was that only a small number of nodes were responsible for the majority of the negative customer experience, and could therefore be prioritized for upgrades.
Value Realized
NetworkCareSales
New Biz
Hadoop in Network Infrastructure – Network Upgrades Improve the Customer Experience
– 17 –
• Correlate network congestion and customer experience
• 11 different data sources
• 4m subscriber records, 12m work orders, 9m calls, 42m IPDRs, 20m Tivoli NPMs
• Finding: Only a few nodes responsible for most of the negative customer experience
Network Node
TNMP CMTS
Performance
Network
Sensors
IPDR Cable
Modem Usage
CompetitiveSpendData
HouseholdHousehold
Master Subscriber
Record
Marketing Demo-
graphics
Caller Experience
Work Orders
Mobile Devices
Customer
Premise
Equipment
Online
Transactions
Social Media
Interactions
SOURCE DATA
NetworkCareSales
New Biz
Network Infrastructure – Network Maintenance
– 18 –
Business Problem
Radio access networks provide the air interface between a mobile provider and the end user mobile devices.
Maintenance and repair of radio access networks poses substantial logistical challenges. In most countries, mobile networks cover more than 95% of a country’s surface area.
Many transmission towers are in remote and difficult to access locations.
In high-density areas, pico- and femto-cells optimize local coverage, but in turn require coordination with the building owner for maintenance.
Hadoop improves a provider’s ability to service equipment proactively, which is always cheaper and less disruptive than the replacement of equipment that has already failed.
Value Realized
NetworkCareSales
New Biz
Network Infrastructure – Network Performance Management
– 19 –
Business Problem
Existing network management platform meant to diagnose poor cellular service such as dropped calls or poor audio quality.
Overwhelmed by data volume, ingesting 10 million messages per second
Each analysis was limited to a 24-hour time window and only one-fiftieth the surface area of the United States.
Same customer issue may generate multiple support calls, but the operator’s team cannot see relationships between multiple variables across time.
Is the problem with the customer’s device? Is it their neighborhood or proximity to a tower? Is it because of how they use their phone?
With more history, they are able to explore root causes that they have never been able to identify by reviewing just one day’s data, allowing them to to improve cell phone service.
Value Realized
NetworkCareSales
New Biz
Hadoop in Service and Security
– 21 –
Business Problem
Customer experience analytics based on call detail records (CDRs)
Contact center productivity Field service productivity Data protection and compliance End-user device security
With Hadoop, one operator detected that 25% of callers were contacting the call center merely to have their late fees on the monthly bill waived. Clearly a case for call deflection to interactive voice recognition and online self-service.
Contact center agents had insufficient ways of diagnosing what was wrong with customers, leading to many unnecessary truck rolls. Hadoop helped avoid these.
3% of smartphones account for 10-15% of traffic because of malware (notably on Android phones) and some fair use violations. Hadoop helps detect that so operators can take remedial action.
Value Realized
NetworkCareSales
New Biz
Service and Security – Customer Experience Analytics Based on Call Detail Records (CDRs)
– 22 –
Business Problem
A typical mobile service provider generates >1 billion CDRs per day, ingesting millions of CDRs per second.
System holds >100 billion records, half a petabyte added every month!
Due to the cost of existing solutions, the data expires after 60 days
CDRs need to be analyzed and archived for compliance, billing and congestion monitoring.
Example: forensics on dropped calls and poor sound quality.
High volume makes pattern recognition and root cause analysis difficult.
Often those need to happen in real-time, with a customer waiting for answers.
With Hadoop the carrier can to retain some data for up to three years
Hadoop provides both a cost advantage – Hadoop provides storage 20x cheaper than enterprise-grade storage – and better insights.
Better analysis to continuously improve call quality, customer satisfaction and servicing margins.
Value Realized
NetworkCareSales
New Biz
Service and Security –Contact Center Productivity
– 23 –
Business Problem
A US-based mobile provider struggled with a combination of high costs but low customer satisfaction related to customer care.
An increasing share of support cases are related to mobile data usage and associated charges.
Traditionally, contact center agents did not have granular insights into a particular customer’s data usage, hence were unable to provide effective call resolution.
With Hadoop, one operator detected that 25% of callers were contacting the call center merely to have their late fees on the monthly bill waived.
The provider was able to off-load these cases to online self-service and interactive voice recognition.
Frees up the agents to focus on more valuable customer interactions.
The provider is now extending this solution to focus on issue resolution.
Value Realized
NetworkCareSales
New Biz
Service and Security – Field Service Productivity
– 24 –
Business Problem
A provider’s contact center agents had insufficient ways of diagnosing what was wrong with customers, leading to many unnecessary truck rolls.
In particular, the agents were not able to triage network vs. home-based problems accurately enough.
Therefore, technicians were dispatched to the customer premises for problems that reside within the network.
The provider was able to avoid a large number of “false positive” truck rolls.
With each truck roll costing about $150 fully loaded, the provider was able to save several million dollars already in the first year.
Value Realized
NetworkCareSales
New Biz
Service and Security –End User Device Security
– 26 –
Business Problem
A mobile operator needed to identify real-time malware threats from non-trusted application stores and contain their impact on customers.
3% of smartphones account for 10-15% of traffic because of malware (notably on Android phones) and some fair use violations.
Hadoop helps detect that so operators can take remedial action, thus eliminating a disproportionate share of network tonnage.
Options ranged from notifying an affected customer all the way to blocking certain URLs for the whole network.
Value Realized
NetworkCareSales
New Biz
Hadoop in Sales and Marketing
– 27 –
Business Problem
360-degree view of customer value Personalized marketing campaigns Upselling and cross-selling Next-product-to-buy (NPTB) Churn reduction
Telesales revenue increase by 50% by tracking competitors web-sites visited and counter offers to products searched
+20% conversion rate increase by optimizing and personalizing the path-to-transaction
$1.65 ARPU increase for 1 million customers boosts topline by $20 million per year.
Reducing cable subscriber churn (“cord cutting”). Every 100,000 subscribers equates to customer lifetime value of $1 billion
Churn model quality increase Price related churn down by 40%
Value Realized
NetworkCareSales
New Biz
Sales and Marketing – 360 Degree View of Customer Value
– 28 –
Business Problem
Telcos and cable companies interact with customers across many channels and points in time.
Data about those interactions is stored in silos.
Difficult to correlate data about customer purchases, marketing campaign results, and online browsing behavior.
Problem is exacerbated by recent acquisitions and a proliferation in the volume and type of customer data.
Merging that data in a relational database structure is slow, expensive and technically difficult.
Enterprise-wide data lake of several petabytes
360-degree unified view of the customer (or household) life time value based on usages patterns across time, products and channels.
Value Realized
NetworkCareSales
New Biz
Sales and Marketing – Personalized Marketing Campaigns
– 29 –
Business Problem
Marketers have long sought ways to tailor their marketing campaigns to the needs of each individual customer.
Telcos are uniquely positioned to deliver on that goal because mobile phones not only follow their owners everywhere, but also reveal a lot about their owners’ interests through browsing behavior and the applications present on the phone.
Telcos are looking for ways to mine that information.
Provider risked losing substantial revenue as prepaid customers were starting to switch to a competitor as a result of a particularly effective marketing campaign.
The provider used Hadoop to pinpoint those individual customers most at risk of churning, and then built a highly targeted campaign to retain the remaining customers in that segment.
A churn alarm system was established and revenue leakage was minimized.
Telesales revenue increase by 50% by tracking competitors web-sites visited and counter offers to products searched
+20% conversion rate increase by optimizing and personalizing the path-to-transaction
$1.65 ARPU increase for 1 million customers boosts topline by $20 million per year.
Value Realized
NetworkCareSales
New Biz
Sales and Marketing – Up-selling and Cross-selling
– 30 –
Business Problem
The provider needed to find an approach to upsell smart phones into a user base that was still largely on legacy feature phones.
The operator converted many hundred thousand feature phone users to smart phones with associated data plans.
Value Realized
NetworkCareSales
New Biz
Sales and Marketing –Next Product to Buy (NPTB)
– 31 –
Business Problem
As telco product portfolios grow more complex, there are ever more opportunities to sell additional services to the same customer base.
Many sales reps however are overwhelmed with that complexity and struggle to translate the breadth of the product portfolio into incremental sales.
Confident NPTB recommendations, based on data from all its customers, empower sales associates and improve their interactions with customers pre-transaction.
Value Realized
NetworkCareSales
New Biz
Sales and Marketing – Churn Reduction
– 32 –
Business Problem
A North American provider faced the following challenge: 50% of new customers churned off within 6 months of acquisition.
The average customer life time in this segment was 13 months, well short of the 18 months needed to break even.
The provider increased the “right” customer acquisitions by 27% and decreased subsequent churn in this segment by 50%.
Price related churn down by 40% Reducing cable subscriber churn (“cord
cutting”). Every 100,000 subscribers equates to customer lifetime value of $1 billion
Value Realized
NetworkCareSales
New Biz
Hadoop in New and Over-the-Top / Adjacent Businesses
– 33 –
Business Problem
New product development Actionable intelligence serving:
Advertisers
Merchants/retailers
Payment processors
Federal governments
Local governments Hadoop-as-a-Service
Telcos are well positioned to provide big data as a service to retail, hospitality and logistics customers. This can generate $50-100m in annual revenue for each medium-sized country.
Value Realized
NetworkCareSales
New Biz
New and Adjacent Businesses –New Product Development
– 34 –
Business Problem
Mobile devices produce large amounts of data about where, when, how and why they are used.
This data is extremely valuable for product managers, yet much of it is out of reach. Either it is never captured or never converted into business insight. Its volume and variety make it difficult to ingest, store and analyze at scale.
One provider who logged 27m devices with more than 1bn events per month has developed more than 20 projects and pilots within 18 months after launch, leading to increased revenue and profitability.
Value Realized
NetworkCareSales
New Biz
New and Adjacent Businesses –Actionable Intelligence Serving Advertisers
– 35 –
Business Problem
Europe’s leading real estate marketplace Scout24 – a subsidiary of Deutsche Telekom – features more than one million properties for rent or sale at any given time, and has facilitated more than 20 million property transactions over the last few years.
The company wanted to drive more market share to Scout24 by offering advertisers – typically real estate agents and brokers – an even better service.
A small team consisting of a product manager, a data scientist and a few developers was able to make a meaningful contribution to revenue growth.
Value Realized
NetworkCareSales
New Biz
Big Data as a Product: ImmobilienScout (Deutsche Telekom)
– 36 –
NetworkCareSales
New Biz
New and Adjacent Businesses –Actionable Intelligence Serving Merchants
– 37 –
Business Problem
A French mobile service provider is a great example for how location information per customer segments can be used to optimize promotions and point-of-sale locations of bricks-and-mortar retailers.
The retailers were able to increase their reported same-store-sales through better campaign management and in-store optimizations. They also gained valuable insights to optimize their store network.
Value Realized
NetworkCareSales
New Biz
New and Adjacent Businesses –Actionable Intelligence Serving Payment Processors
– 38 –
Business Problem
Credit card issuers experience increasing fraud when their card members are travelling abroad.
95% of travelers opted into the SMS alerting service, resulting in a substantial decrease in fraud related to card use in foreign countries.
Value Realized
NetworkCareSales
New Biz
New and Adjacent Businesses –Actionable Intelligence Serving Federal Governments
– 39 –
Business Problem
The Eastward expansion of the European Union has resulted in a longer and more porous border to non-EU member states.
This has made it more difficult to protect the EU against a stream of illegal goods and refugees, which often travel over land from the EU’s Eastern and South-Eastern neighbors.
Law enforcement agencies are able to target their scarce resources much more effectively, for instance choosing to intercept suspicious cars traveling in certain directions at speeds above 130km/h.
This radically increases their hit rate per mission.
Value Realized
NetworkCareSales
New Biz
New and Adjacent Businesses –Actionable Intelligence ServingLocal Governments
– 40 –
Business Problem
In a large French city, traffic to large events regularly caused massive congestion on the city’s streets and highways.
The city identified and implemented dozens of specific traffic management measures, relieving congestion around major events.
They are also exploring how to use these insights for environmental impact studies, city planning and disaster management.
Value Realized
NetworkCareSales
New Biz
• Makes capital investments more efficient
• Leads to a better customer experience
• Lowers churn
• Increases conversions
• Strengthens security
• Opens up new markets
Hadoop Drives Business Outcomes for the World’s Telcos and Cable Companies!
– 41 –
Questions?
Email [email protected] for a copy of the presentation.
LinkedIn: juergenurbanski
Download 200-page BITKOM / Forrester Guide to Big Data Technologies (in German):
http://www.bitkom.org/files/documents/BITKOM_Leitfaden_Big-Data-Technologien-Wissen_fuer_Entscheider_Febr_2014.pdf