5733 a deep dive into ibm watson foundation for csp (wfc)

5733 - A Deep Dive into Watson Foundations for CSPs (WFC) Architecture Dr. Arvind Sathi [email protected] Richard Harken [email protected] Tommy Eunice [email protected] Mathews Thomas [email protected] Wed 29/Oct, 04:30 PM - 05:45 PM

© 2014 IBM Corporation

Content

•  Watson Foundation for CSPs (WFC)

•  Discovery and Predictive Modeling using SPSS

•  Detection and Real-time Analytics using InfoSphere Streams

•  Component Integration

•  Conclusions

Real Time Analytical Processing Data Warehouse

Real-Time Analysis and Event Processing

(RTAP)

Input Data

OSS (Landing Area)

Reference Data (from EDW)

Network

Netezza / ISAS / … BSS

Mediation

Our Telco Evolu,on – Ini,ally Just Media,on and Complex Event Detec,on

Real Time Analytical Processing Data Warehouse


(RTAP)

Our Telco Evolu,on – Adding on Marke,ng and Predic,ve Analy,cs

Input Data

OSS (Landing Area)


Network

DWH, Analytics Foundation

(ELT & In-Database Analytics)

BSS

Mediation

Analytical Source Systems Monetization Platforms

Marketing Campaign System

Mobile Advertising

Real-Time Event Triggers

Predictive Analytics (SPSS, etc.)

PMML Model Deployent

Real Time Analytical Processing Network Analytics


EDW

Our Telco Evolu,on – Adding DPI and Network Analy,cs


Mobile Advertising


BI & Visualization

Customer Experience Management (Data Model)

Other BI Dashboards & Reports

Network Analytics Insights

(Dashboards)

Analytics Foundation (ELT & In-Database

Analytics)

CRM

Predictive Analytics

TNF


(RTAP)

Input Data

OSS (Landing Area)


Network

BSS

Mediation

Deep Packet Inspection

SPSS-Streams Toolkit

TNF

Real Time Analytical Processing Network Analytics


EDW

Looking Ahead-‐-‐ More Analy,cs , Cybersecurity, BigInsights


Mobile Advertising


BI & Visualization

Customer Experience Management (Data Model)

Other BI Dashboards & Reports

Network Analytics Insights

(Dashboards)

Analytics Foundation (ELT & In-Database

Analytics)

CRM

Predictive Analytics

TNF


(RTAP)

Input Data

OSS (Landing Area)


Network

BSS

Mediation

Deep Packet Inspection

Big Insights

Hadoop Distributed File System

(HDFS )

Text Analytics Machine Learning

Large Scale Analytics

Unstructured Data

Social Media Applications

Web Log Analytics Sentiment Analytics

Topic-based Influencers

Geo-Spatial Analytics Cyber-security Analytics

Web Analytics

TNF

SPSS-Streams Toolkit

Drive Apply the results of inves1ga1on to take ac1on by interac1ng with the subscribers in real-‐1me. Collect feedback from ac1on for future analysis.

Discover Collect historical behavioral data, past acts, and success rates. Analyze historical data to formulate pa?erns and changes required to detect, and inves1gate steps

Decide Gather data on targeted customers from a variety of sources over 1me to establish behavioral pa?erns and iden1fy how to respond to an emerging pa?ern.

Detect Detect in real 1me if a transac1on, request, applica1on, document, etc. is required for targe1ng. Flag the selected dataset and ignore the rest.

WFC uses D4 for tight integration across four analytics components

Detect observa,ons about a target

Take ac,on in real ,me – when it

ma8ers

Find new targets by analyzing historical

data

Iden,fy pa8erns over ,me and ac,ons required

Drive

Detect

Discover

Decide

Target Subscriber

7

Drive Interact with the customer to seek permission to use loca1on informa1on and send campaign, record interac1on and results.

Discover Collect historical behavioral data, past acts, and success rates. Analyze historical data to formulate pa?erns and changes required to detect, and inves1gate steps

Decide Use background informa1on, past campaigns, privacy preferences, customer reac1on to past campaigns, purchase intent, preferences expressed in social media to design campaign.

Detect Detect in real 1me if a transac1on relates to targeted subscribers. Iden1fy, align, score, and send for further processing (e.g., a targeted customer driving towards mall)

Smarter Campaigns using D4



ma8ers


data


Drive

Detect

Discover

Decide

Target Subscriber

8

Drive Take appropriate ac1on to minimize losses due to fraud. Record ac1ons and results for future analysis.

Discover Collect historical fraudulent pa?erns. Analyze historical data to formulate pa?erns and changes required to detect, and inves1gate steps

Decide Use background informa1on, past usage, loca1ons, subscrip1on, bill payments to find if the fraudulent transac1ons are associated with a subscrip1on. Seek more data as needed. Raise an alarm.

Detect Detect in real 1me if a transac1on relates to a fraudulent subscrip1on (e.g., inconsistent geography or usage in consecu1ve transac1ons). Send alert for further Inves1ga1on.

Fraud Analytics and Management using D4



ma8ers


data


Drive

Detect

Discover

Decide

Target Subscriber

9

WFC Use Cases

10

Business Capability Use Case Name

Business Intelligence

Centralized Business Intelligence

EDW/BI Transformation Competitive Monitoring

Customer Experience

Management

Effective Customer Care

Apologize for Poor NW exp

Enhance Consumer Billing Reports

CX for Roamers

Personalized Experience Best Video Experience Customer Care for VIP Quality of Experience for Apps

Service Trouble Shooting at C-Center

Cross-channel Optimization

Channel Optimization

Lead Management Sales and Support Integration Shopping Carts and Lists Management

Insight Analytics

Data usage Patterns Multi-Sim Behavior Online Behavior – Trending Behaviors Online Behavior – website Analysis

Roamer Behavior Voice/SMS Apps impact on Traditional voice and sms services

Tethering Behaviors


Data Monetization

Online Market Analysis Enterprise M2M Proposition Family Online Protection Sports Analytics Effective Advertising Media Metrics Monetize market data 3rd Party Advertising Networks

Device Management

Device Analysis Device SW version Upgrade Notifications Device Migrations Traffic Analysis

Voice and Data Performance Fault detection/notification/NFF

Dynamic Pricing & Service Models

Advanced Thetering on Shared Billing Accounts

Proactive High Speed Access at Partnered Locations B-width onDemand at Strategic Hotspots

Field Service Management

Agenda Optimization & Resource Activation at Customer Premises

Fraud Management

Equipment Fraud Subscription Fraud Identity Theft VoIP Hacking Dealer Fraud


Smarter Campaigns

Improve Conversion rates by Timing offers effectively Contract Renewal/Retention/Up/X-sell

Intelligent Data Services

Content Based Advertising Content Based third-party Advertising based on Location Content provider Advertising QoS Charging for Content Providers

Location-based Services

Contextual marketing Co-Presence Dynamic AdHoc group Geo-fence Mktg Propositions Massive Events Mobility patterns Hang out Occupancy/Traversal/Anomaly

Tariff plan innovation / optimization

App/Location/Time- based Plans

Tariff Planning

Plans and Add-on Impact on App Usage

Network and Service

Optimization

Service Usage & Usage Location

Customer Centric NW Monitoring 3G Locked Subscribers IB roaming performance analysis Intelligent network policies Capacity Planning

Subscriber and Service

Management

Interactive multimedia ticketing

Mobile Real Time Social Campagn New Business Models- Loyalty Self Care Multi-Device Family Accounts

Content •  Watson Foundation for CSPs (WFC)




•  Conclusions

Introduction to Subscriber Dimensions from Mobility Analytics

Usage Style l Heavy Voice l SMS Mostly l No Data

Interests l From DPI l Webpage analytics l e.g. Golf, Betting

Quality of Service l From xDR l Network l The Now Factory

Demographics l Based on usage patterns l Websites l Buddies

Lifestyle l Commuter l Homebody l Night Owl

Preferences l OTT Messaging l Travel, Games l Handset prefs

Preferred Locations l Hangouts l Home Work l Mode of Travel

Best Buddies l Who calls who l Who hangs out with Who?

How to turn streaming noisy Telco Location data into meaningful location, then discover customer insights

Call Detail Records

SMS Voice

GPS Tracking

Cell Tower Wifi AP Maps

GIS, POI

Special Service Numbers

e.g bank, 1-800

Reference Data

Stream data

subscriberId: Timestamp: Position: latitude + longitude Precision: 0~2 km Direction: nullable Speed: nullable Activity : nullable

Analyzable Location Event Data Who, when, where and what

Meaningful Location

subscriberId: home: Work: POIs & period … Sequence of meaningful Locations… Commute means: car/subway/bus

Micro segmentaton Business traveler Regular commuter Heavy driver Social Butterfly Mom …..

ü Every Sunday noon, Bob goes to xxx mall to shopping and has lunch ü Every Thursday afternoon, Bob goes to customer site at XXX ü …..

Location Patterns on Individual and Group level

Mobile Location Data Processing: Map mapping,

Business rules et. Big Data

Integration Spatio-Temporal Event Association Analysis

Wifi off load

Location Pattern Analytics

Discovery using structured data •  A typical discovery uses statistical tools to identify pattern in data. •  Discovery may contribute new derived attributes for further analysis or reporting.

Night Owls at Night

Delivery People During the Day

Quiet Weekday peoplego for dinner on weekends

Almost no Homebodies any time

Mobility Lifestyles (developed by IBM)

* from the Television show, “Cheers”. Norm was an accountant who went to the same pub every night

Mobility Lifestyles •  How do the lifestyles of subscribers vary by location and time

of day •  Why do lifestyles matter for Retailers?

§  Certain lifestyles tend follow habits much more than others: Daily Grinders and Homebodies go to the same locations often and are predictable. The other lifestyles tend to be less predictable

Mobility and Usage Lifestyles Very distinct patterns, this level of differential is high

Handsets by Lifestyle

Location Analytics - Dashboard

Screen shots from Cognos

Predictive Modeling and Scoring example





•  Conclusions

Streams Data Processing in Telco Environment

Streams Telco Realtime Processing

CDRs

Logs

Event Data

Performance Data

Configuration Data

Telc

o N

etw

ork

Elem

ents

Source data format: ASN.1, XML, ASCII, binary Standardized or proprietary, via edge adapters

Output into dashboards, databases, files Statistics, monitoring, archiving

Decoding, filtering, aggregation, correlation, summation, transformation, formatting, ...

Tap into Message Transfer

Telco Solutions

Continuous Ingestion Continuous Queries / Analytics of data in motion

Visual Representation A New Paradigm: In-Motion analytics for High throughput and Ultra-low latencies

Data Tuple Operator

Streams Application

Data Sink

Data Sources

InfoSphere Streams Overview

From Vast Data to Actionable Insights InfoSphere Streams addresses the challenge for CSPs is to turn the vast amounts of customer data they collect into usable and actionable insight

CDRs

Billing

CRM

Location

Account Mgt

Internet

Network

Millions of events per second Dropped Calls

Outgoing Interna,onal Calls Call Dura,on Extra Call

Contract Expira,on

Entered new cell

New Top-‐Up 5 minutes leM on pre-‐paid

Invoice Issued

Congested Cells

Invoice Paid

Acquired new products Change contracts

Brand Reputa,on Customer Sen,ment

Customer is roaming Customer is at home

3 dropped calls in 10 minutes

Customer is close to a store

Customer enters a shopping area

Invoice paid + ‘liked’ compe,tor

Smart phone browsing pa8ern

Customer is watching a video

Streams of Intelligence

Microsecond Latency Required

from Social network

Changed Home Loca,on

Broadband Satura,on

Who is THIS customer and what does S/HE want?

Ac,onable Insight

MDM, EDW

IBM CONFIDENTIAL

Possible Architecture (for live system) Network/Internet Analytics 2 Data Collection 1

TAP Filter

(Brocade, Gigamon, etc) Load Balance

(Brocade, Gigamon, etc)

ISP

Blade 1

Blade 2

Blade 3

Blade 4

Blade 5

Blade 6

Blade 7

Blade 8

Blade 9

Blade 10

Blade …

Each line represents a link

to a physical network interface

on a blade. It carries data for one or more pre-

specified protocols. All

packets belonging to one session are sent

on one link.

Netezza

1. Data Collection and Pattern identification. 2. The offline modeling step- using the SPSS Modeler- creates analytic models based on labeled training data. The data can be hosted on any platform example: data warehouse, Pure Data Systems for Analytics, Hadoop. 3. The intermediate integration step - There are two alternatives to deploy SPSS models in streams. One is to generate the PMML model. There is a limited set of models that generate the PMML format. The PMML model is then deployed in the Streams mining toolkit. The other approach is to publish the model to generate the .pim, .par and .xml files which are supported by the SPSS Analtyics Toolkit for Streams. These files are then configured on the SPSS Modeler Solution Publisher. 4. The on-line phase– Using Sreams SPL (Streams Processing Language), streams developer further uses appropriate operators and input output definitions of the models. This enables realtime analytics. 5. The action is triggered for instance executing a mobile campaign when a defined threshold is interpreted in real time.

Integration with Discovery

InfoSphere Streams

NETEZZA

IBM SPSS Modeler Solution

Publisher In-‐database mining

Published .pim, .par, .xml files generated from SPSS Scoring node SPSS

During the offline phase, SPSS Modeler accesses the training data residing in the Pure Data System for Analytics and creates the Model Nugget. The Modeler ODBC node can access the database table’s definitions as well as the data, and retrieves the relevant training data according to the selection criteria. Once trained, the Modeler creates a Model Nugget which can be published to Streams.

Publish the Model

DB2

InfoSphere Streams

UNICA Campaign

polling

Worklight Server Mobile

App

IBM SPSS Modeler Solution

Publisher

Model execution with Streaming data





•  Conclusions

WFC Application Architecture A

B

C

D G

AAP Capabilities High Performance Historical analysis Model Based Predictive Analytics Real-time scoring, classification, detection and action

Visualize, explore, investigate, search and report

High Performance Unstructured Data analysis Discovery Analytics Take action on analytics

F

Information Interaction

Analytics Engine

Prediction / Policy Engine

Sense, Identify,

Align

Reports

Geo/Semantic Mapping

Dashboards

Simulation

Outcome Optimization

Model Creation

Semi Structured

Data

Dat

a R

epos

itorie

s

Network Events

Network Policies C

ontin

uous

Fee

d S

ourc

es

XDR

Batch Data

Data for Historical Analysis

Deploy Model

Streaming Engine

Streaming Data Categorize, Count, Focus

Score, Decide

Historical Data Models

In Database Mining

Reports & Dashboards

Ad-hoc Queries

Actions

Event Execution

Policy Mgmt

Ext

erna

l D

ata Social

3rd party

High Velocity

High Volume

Open API

Customer Activities

A

B

C

D G

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Campaign Mgmt.

Pro-active Customer

Experience Management

Pro-active Network Mgmt

Real time Scoring & Decision Mgmt.

...

Deploy Model

Policy Management

Data Integration ETL

Deduplicate

Standardize

Identity Resolution

Network Topology

Data

Application & Usage

Data Customer

Data

Capture Changes

Un-Structured

Data Hadoop

E

E

Structured Data

Insight F Search, Pattern Matching, Quantitative, Qualitative

EDW

Advanced Analytics Platform

Create & Deliver Smarter Services Transform Operations Build Smarter

Networks Personalize Customer Engagements

Database Server

A

B

C

D G

AAP Capabilities High Performance Historical analysis Model Based Predictive Analytics Real-time scoring, classification, detection and action

Visualize, explore, investigate, search and report

High Performance Unstructured Data analysis Discovery Analytics Take action on analytics

F

Information Interaction

Analytics Engine

Prediction / Policy Engine

Sense, Identify,

Align

Reports

Geo/Semantic Mapping

Dashboards

Simulation

Outcome Optimization

Model Creation

Semi Structured

Data

Dat

a R

epos

itorie

s

Network Events

Network Policies C

ontin

uous

Fee

d S

ourc

es

XDR

Batch Data

Data for Historical Analysis

Deploy Model

Streaming Engine

Streaming Data Categorize, Count, Focus

Score, Decide

Historical Data Models

In Database Mining

Reports & Dashboards

Ad-hoc Queries

Actions

Event Execution

Policy Mgmt

Ext

erna

l D

ata Social

3rd party

High Velocity

High Volume

Open API

Customer Activities

A

B

C

D G

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Marketing

Customer Care

Users

NOC/SOC

Network Planning

...

Campaign Mgmt.

Pro-active Customer

Experience Management

Pro-active Network Mgmt

Real time Scoring & Decision Mgmt.

...

Deploy Model

Policy Management

Data Integration ETL

Deduplicate

Standardize

Identity Resolution

Network Topology

Data

Application & Usage

Data Customer

Data

Capture Changes

Un-Structured

Data Hadoop

E

E

Structured Data

Insight F Search, Pattern Matching, Quantitative, Qualitative

Enterprise Data Warehouse

Advanced Analytics Platform

Create & Deliver Smarter Services Transform Operations Build Smarter

Networks Personalize Customer Engagements

InfoSphere Streams

SPSS

ODM, Optim, Open Pages

PDA

Social Media Analytics

Watson Explorer

Cognos

InfoSphere BigInsights

IBM (Unica)

Campaign

ODM

PDOA

SPSS Database Server

BPM

TNF SourceWorks TNF Smart Works

Watson Analytics

WFC Application Architecture using IBM products

InfoServer

EA

Analytics Capabilities •  Reporting

§  Structured, Unstructured, Ad hoc

•  Discovery §  Structured, Unstructured

•  Predictive Modeling •  Identity Resolution •  Customer Profiling •  Real-time Filtering

§  Static, Dynamic

•  Real-time Scoring •  Simulation •  Feedback and Machine Learning •  Visualization

32





•  Conclusions

Reading Material

•  IBM Developer Works §  Explore the advanced analytics platform, Part 1: Support your business requirements using big

data and advanced analytics

§  Explore the advanced analytics platform, Part 2: Explore use cases that cross multiple industries using the advanced analytics platform

§  Explore the advanced analytics platform, Part 3: Analyze unstructured text using patterns

§  Explore the advanced analytics platform, Part 4: Analyze location data to determine movement patterns using a mobility profile pattern

§  Explore the advanced analytics platform, Part 5: Deep dive into discovery and visualization

§  Explore the advanced analytics platform, Part 6: Dive into orchestration with a combination of SPSS, Operational Decision Management (ODM), and Streams using care and fraud management case studies

•  IBM Data Magazine

§  Mining Data in a High-Performance Sandbox - Fulfill data analysts’ dreams with data warehouse appliances for in-database analytics and data mining

§  Target Behavior in Real Time for Effective Outcomes: Part 1 - How real-time, adaptive architectures can drive management decisions for specific use cases

§  Target Behavior in Real Time for Effective Outcomes: Part 2 Drive marketing and business management decisions using a real-time, adaptive architecture

•  Books

§  Big Data Analytics: Disruptive Technologies for Changing the Game

§  Engaging Customers Using Big Data: How Marketing Analytics Are Transforming Business

We Value Your Feedback! •  Don’t forget to submit your Insight session and speaker feedback!

Your feedback is very important to us – we use it to continually improve the conference.

•  Access the Insight Conference Connect tool to quickly submit your surveys from your smartphone, laptop or conference kiosk.

35

Acknowledgements and Disclaimers Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

© Copyright IBM Corporation 2014. All rights reserved.

— U.S. Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

— Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2,Maximo, Clearcase, Lotus, etc

IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at • “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml

• If you have mentioned trademarks that are not from IBM, please update and add the following lines:[Insert any special 3rd party trademark names/attributions here] • Other company, product, or service names may be trademarks or service marks of others.

36

Thank You

5733 a deep dive into ibm watson foundation for csp (wfc)

Documents