ensuring big data is supporting financial analytics gaining a thorough understanding of big data in...
TRANSCRIPT
Ensuring big data is supporting financial analytics
• Gaining a thorough understanding of big data in order to understand analytics
• Transforming unstructured data into structured intelligence
• Using big data to predict client behaviour
Bhavani Raskutti @ Pacific Brands
Ensuring big data is supporting financial analytics
• Gaining a thorough understanding of big data in order to understand analytics
• Transforming unstructured data into structured intelligence
• Using big data to predict client behaviour
Bhavani Raskutti @ Pacific Brands
Agenda
• Big data
• Unstructured data
• Framework for embedding financial analytics – data analysis leading to decisions that
– impact company financials
Bhavani Raskutti @ Pacific Brands
Big Data“Big data” is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making.
-- Gartner Inc. 2001
• Volume: Number of entries (rows) – NOT attributes (columns)
• Velocity: Rate of change, usually rate of arrival of rows– Impacts volume
• Variety: attributes (columns) of entries– Structured:
• Transaction data: value, timestamp, type, location, …• Customer data: gender, age, occupation, …
– Unstructured: • Text: Customer interactions, blogs & twitters about the company …• Image: Customer signature, photo, ….
Big Data
Predictive Analytics to Make DecisionsPredict fraudulent credit card transactions
1. Learn a model from historical data
Know
n Ca
tego
ries
NO
Kno
wn
Cate
gorie
s
Classification
Clustering Classification
2. Categorise new data to make decisions
Back-end process Time consuming Built from sampled data
Volume NOT an issue
Models simple Quick process
Velocity & Volume NOT an issueN
ew d
ata
Big Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Structuring Unstructured Data Unstructured Data
Get freeform
source text
Load sentiment word list
love, best, … hate, worst, …
Score sentiment:+1 for positive words-1 for negative words Normalise if needed
1. Sentiment Analysis 2. Topic Detection
Use in Business:For model buildingCompetitor comparisonsMood change over timeAddressing negative scores
Get freeform
source text
Load stop words list:the , on, of, a, an, by, …
Create clusters with term frequency matrix
Use in Business:For model buildingMajor customer issuesImpact of initiatives
Determine clusters of interest and label
Learn cluster models for classification
Business input
Sentiment Analysis OutputUnstructured Data
No sarcasm detection
Topic Detection OutputUnstructured Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Variety NOT an issue Financial Analytics NOT impacted by Big Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Impact of Variety on AnalyticsIndustry Structured Data Unstructured Data Analytics
MaturityFinance • Transactions
• Customer & Product• Call centre
• Twitter/blogs/complaints
Wholesale / Retail
• Inventory, sales, POS• Customer & Product• Sourcing & supply chain
• Twitter/blogs/complaints
Utilities (gas, electricity, …)
• Usage data• Customer & Product• Sensor data
• Twitter/blogs/complaints
Telecom • Call records• Customer & Product• Call centre
• Twitter/blogs/complaints• Inbound emails/SMS• GPS data
Web services • Usage• Product & subscriber• Call centre
• Twitter/blogs/complaints• Inbound emails• Click stream transactions
Benchmark
Big Data
Do BIG analytics with enough data!!
Decision
• Big $ impact• Many people• Timely
Framework for Embedding Analytics Initiative
Insight
Pilot
Data
Deploy
• Just the facts needed for decision making
• Prioritise entries• Support actioning
• Sell to whom, what & $$• Ordered by $$
Framework
Right People
$
Decision Support Initiative for Wholesale Sales
Demand
In-s
tock
%
· R1· R2
Demand
Sell
Rate
Example of Additional Support for ActioningFramework
Sell rate vs Consumer Demand plot • Each point is a store• R1 & R2 are comparable retailers• Values for the same product
Possible reasons for difference• Competing product at R2• Pricing at R2 vs R1• Lack of stock at R2
Decision
• Big $ impact• Many people• Timely
Framework for Embedding Analytics Initiative
Insight
Pilot
Data
Deploy
• Just the facts needed for decision making
• Prioritise entries• Support actioning
• Sell to whom, what & $$• Ordered by $$
Framework
• Automated feed• Objective Data• Most specific • Complete
• POS feed from retailers• SKU & store master
• Pick champions early in the process
• Develop pilot• Validate outputs from
pilot• Iterate pilot with
champions until it is accepted
• Automate • Helpdesk• Training
Right People
$
Decision Support Initiative for Wholesale Sales
Conclusion• Big data is not an impediment to analytics
• Unstructured data can be structured using sentiment analysis & topic detection
• Key success factors for doing BIG analytics is to have the right people to choose–Right decisions–Right insights–Right data –Right process to pilot & deploy
Questions?