big data analytics - from generating big data to deriving business value

©2015 IBM Corporation

Analytics : Key to go from generating

big data to deriving business valuePiyush Malik

WW Big Data Analytics CoE Leader

IBM Global Business Services, USA

@pmalik1

(Based on a paper published by IEEE co-authored in collaboration with

Dr Deepali Arora, University of Victoria, Canada)

https://www.linkedin.com/company/166687?trk=prof-exp-company-name

https://www.linkedin.com/company/166687?trk=prof-exp-company-name

©2015 IBM Corporation|

Talk Outline

1. Why is big data

important?

2. Use of big data to derive

business value

3. How is analytics useful in

deriving value

4. Five patterns of business

intelligence from big data

5. Current challenges

6. Future directions

2009

800,000 petabytes

2020

35 zettabytes

as much Data and Content

44x

80%

*Source: IDC


Use of big data to derive business value


What we used to call “Big Data” is becoming the

norm in industry conversations

4

2012 2013 2014 2015

Volume

Velocity

Variety

Veracity

Confluence of

Social, Mobile,

Cloud, Big Data, and

Analytics

Systems of Insight

Data Transforming

Industries

Data will disrupt IT

in the every

industry

Mobile Social

Cloud

Internet of Things

Data


1 in 2business leaders do not have access to

data they need

83%of CIO’s cited Business

Intelligence (BI) and analytics as part of their

visionary plan

5.4Xmore likely that top

performers use business analytics

80%of the world’s data today is unstructured

90%of the world’s data was created in the

last two years

20%of available data can

be processed by traditional systems

Source: GigaOM, IBM Software Group, IBM Institute for Business Value"

Let’s consider the realities associated with data


How is analytics useful in deriving business value

Big data is analyzed using advanced techniques including machine learning algorithms


How is analytics useful in deriving business value

7

Supervised Unsupervised

Linear Nonlinear

Single Combined

Easy to Interpret Hard to Interpret

Linear

RegressionLogistic

Regression

Perceptron

Bagging Boosting Random

Forest

Decision Rule

Trees Learning

Naïve k-Nearest

Bayes NeighboursMulti-Layer SVM

Perceptron

K-Means EM Self-

Organizing

Maps

Common Machine Learning Algorithms

Machine learning algorithms commonly used to analyze big data are:


Real-time traffic flow optimization

Fraud and risk detection

Accurate and timely threat detection

Predict and act on intent to purchase

Understand and act on customer

sentiment

Low-latency network analysis

Industry Examples of Applied Big Data Analytics


Five use cases of business intelligence from big data

Case 1: Sentiment analysis in social networks

With an IBM Social Analytics you can decode the psychological genotype

of your customer to achieve unprecedented customer intimacy

9

Psychological profile

Personality

Needs

Values

Activity profiles

IBM FOAK‘s with …

Two retailers

Three hotel chains

Two airlines

Two governmental

departments

Followers analyzed

200+ million Tweets

300K+ users analyzed




Recently a Telecom company used cloud-based social media analytics to

proactively retain customers before they have decided to leave

Business challenge: Customers continue to shift to mobile and social

channels in the way they converse about brands. Social channels are

often the first way to express grievances and doubts. In real time, negative

sentiments can quickly proliferate and influence existing and prospective

customers. How can we leverage these social channels to connect with

customers in a way that suits their preferences in order to build loyalty

and reduce attrition?

The socially aware solution: The Telco uses customer sentiment derived

from Twitter postings along with other social data and internal company

records to understand customer preferences and predict customers at risk

of attrition. Resource is being shifted from staffing call centres to social

media engagements in order to empower customers and respond to them

via their preferred communication medium.

30% reductionin customer attrition rate

Increased revenues 10% increase in Call

Centre Agent revenues

25% increase in cross sell

and upsell opportunities

Higher Customer

SatisfactionRating improved from 1.5 to 2.7

on a five point scale

Powered by Real time Twitter feed and other

Social Media data

IBM SMA 1.3 (DB2, Cognos)

IBM Big Match

IBM SPSS Modeler

IBM Psycholinguistics

Next Best Action .


12

xDRs

Billing

CRM

Location

Account

Internet

Network

Millions of events

per second

Dropped CallsOutgoing International Calls

Call Duration

Extra Call

Contract Expiration

Entered new cell

New Top-Up

5 minutes left on pre-paid

Invoice Issued

Congested Cells

Invoice Paid

Acquired new products

Change contracts

Brand Reputation

Customer Sentiment

Customer is roaming

Customer is at home

3 dropped calls in 10 minutes

Customer is close to a store

Customer enters a shopping area

Invoice paid + ‘liked’ competitor

Smart phone browsing pattern

Customer is watching an OTT

video

Streams of

intelligence

from Social network

Changed Home Location

Broadband Saturation

Where should I

INVEST to

gain/retain more

high-value

customers?

Who is THIS

customer and what

do THEY

want/need?

What should I be

OFFERING specific

customers to

improve individual

ARPU/profitability?

Actionable

insight

MDM EDW/ADW

Big Data Analytics Value proposition Scenario : Combine Network, Billing, Subscriber, Call Records, etc to gain new, valuable and actionable insights, prevent churn and improve outcomes

Case 2: Preventing customer churn in telecommunication sector



Case 2: Preventing customer churn in telecommunication sector


A Malaysian Telco leveraging Big Data in Motion

Data in motion

CDRs

Data

Location

Reload

Provisioning

Others

(future)

8000+

Events

per

second

Microsecon

d Latency

Dropped Calls

Outgoing International

CallsCall Duration

Extra Call

Dongle purchase

Entered new cell

New Top-Up

5 minutes left on pre-paid

RTP Platform

3 Dropped Calls in the last hour

In location of interest

5 Outgoing international Calls in the last day

Data Aggregated at

Single Customer Level

Filtering of high-throughput TPS to

fewer selected events of interest

RSS Feeds

Web sites

GPS Location

13


Case 3: Enhancing customers’ online shopping experience

a) Enhanced Social Shopping



b) In-Store Presence Zones

Intelligent location-based technology to gain deep insight into customer in-store behavior

Enables retailers to integrate the physical and digital experience to facilitate an ongoing dialogue, create loyalty and deliver an exceptional in-store shopping experience

Presence Zones Sensors



15


c) Hyper-personalization and contextual shopping

Enables retailers to microsegment customer base, target based on demographic and psychographic criteria for 1:1 marketing and lift sales using social commerce,

gamification and big data




Smart Meter

Analytics

Condition

Based

Maintenance

Smart Meter

Analytics

Smart Meter

Analytics

Distribution

Load

Forecasting &

Scheduling

Improve Generation

Performance

Transform Customer

Operations

Condition

Based

Maintenance

Transform the Utility

Network

Distribution

Load

Forecasting &

Scheduling

Smart Meter Analytics is a Common Element Across a Smarter Energy

Value Chain, Differentiation Comes through Leveraging Big Data

Customer

Insight

Case 4: Generating value from smart utility meters


17




How IBM Assisted an Electric Utility in Leveraging Smart Metering/Big Data to create Business Value

Load, manage, and analyze information from smart meters, the smart grid and customer

information - use that data to gain customer and operational insights.

What Data?

• Meter readings

• Grid data

• Customer

information

• Meter generated

alerts and

power quality

indicators

• Meter

connection

status

What Capability?

• Load interval meter

readings (Time Series

Data)

• Track & evaluate

energy losses

• Micro customer

segmentation

• Secure Analytics

Warehouse for

operational insight

• Analyze meter failures

and power outages

What Outcome?

• Better understand

and manage outages

• Detect energy theft

• Drive customer

usage portal

• Prepare customer

specific alerts and

communications

• Customer

segmentation and

insights

Smart

Meter Data

Grid Data

Customer

Information

18


Analytics and

Reporting

Sentiment Analysis

Call Center Analysis

Offering Management

DistributionGeneration Transmission

EmployeesMaintenance

Suppliers Orders

Marketing GIS

Customers

Smart Meters

Trading

Regulations

Social Media

Sensors

Imp

roved A

naly

tics

Unstructured

Exploration/DiscoveryQueryable Archive

InfoSphereBigInsights

Unstructured

Streaming Structured or Unstructured

Unstr

uctu

red

Imp

roved A

naly

tics

Str

uctu

red

Smart Grid Analytics

Distribution Grid Monitoring

Root Cause Failure Analysis

IBM InfoSphere

Streams

Real Time Scoring and Response

Analytics and

Reporting

ETLIBMData Warehouse

Analytics and

Reporting

Meter Data Management

Demand Forecasting

Maintenance Scheduling



How IBM Assisted an Electric Utility in Leveraging Smart Metering/Big Data to create Business Value


Smarter Planet = More Connected, More Vulnerable

Case 5: Improving Security


20


Threats are evolving

Attacker generic

Malware / Hacking / DDoS IT Infrastructure

Before . .

Advanced

Persistent

Threat

Critical data /

infrastructure

Attacker

!




IBM QRadar Platform: Taking in data from wide spectrum of feeds and continually adding context for increased accuracy

Security Intelligence Feeds

Internet ThreatsGeo Location Vulnerabilities




Applying Adaptive Analysis & Classification

Antennae (Directional,

Omnidirectional)+ Digitization

Direction Finding

De-interleaving

Chain Analysis

Classification

Tracking

Stream Computing

As fast, low latency

Signal Processing Functions

Adaptive

History

hadoop technologiesOffline AnalysisBuild Models & PatternsCondition Real Time Processing

Pulse Data Analysis



Security/Intelligence Extension

Lower risk, detect fraud and monitor cyber security in real-time

23


Big Data Processing

• Long-term, multi-PB storage

• Unstructured and structured

• Distributed Hadoop infrastructure

• Preservation of raw data

• Enterprise Integration

Big Data

Platform

Analytics and Forensics

• Advanced visuals and interaction

• Predictive & decision modeling

• Ad hoc queries

• Interactive visualizations

• Collaborative sharing tools

• Pluggable, intuitive UI

Security Intelligence

Platform

Real-time Processing

• Real-time data correlation

• Anomaly detection

• Event and flow normalization

• Security context & enrichment

• Distributed architecture

Security Operations

• Pre-defined rules and reports

• Offense scoring & prioritization

• Activity and event graphing

• Compliance reporting

• Workflow management

Integrated analytics and exploration in a new architecture

Integrated

IBM

Solution



24


Data ingest

Insights

IBM Security QRadar

• Hadoop-based

• Enterprise-grade

• Any data / volume

• Data mining

• Ad hoc analytics

• Data collection and

enrichment

• Event correlation

• Real-time analytics

• Offense prioritization

Big Data Platform

Custom AnalyticsAdvanced Threat Detection

Traditional data sources

IBM InfoSphere BigInsights

Non-traditional

Security Intelligence Platform

How? By integrating QRadar with Hadoop-based solution



25


Current challenges and future directions

1. Finding the right kind of data to use

2. Managing large datasets

3. Selecting algorithms to extract meaningful information for

different domains

4. Security and Privacy

5. Finding skilled people with good understanding of analytics

• Contextual

• Cognitive

• Public-Private Data Partnerships

• Data Marketplaces

Current challenges

Future Directions

26


Summary & Conclusion

Questions?

27

big data analytics - from generating big data to deriving business value

Data & Analytics

big data important

ibm social analytics

available data

business analytics80

business valuelets

business leaders

ibm institute

business value3