wso2con usa 2015: keynote - the future of real-time analytics and iot
TRANSCRIPT
The Future Of Analytics And IoT It Can’t Happen Without You
Mike Gualtieri, Principal Analyst
November 3, 2015 San Francisco
Twitter: @mgualtieri
#Priority
© 2015 Forrester Research, Inc. Reproduction Prohibited 3
52%
53%
53%
54%
58%
64%
64%
65%
66%
73%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Better leverage big data and analytics in business decision-making
Create a comprehensive strategy for addressing digital technologies like mobile, social & smart products
Create a comprehensive digital marketing strategy
Better comply with regulations and requirements
Improve differentiation in the market
Increase influence and brand reach in the market
Address rising customer expectations
Improve our ability to innovate
Reduce costs
Improve our products /services
Improve the experience of our customers
Customer experience is a top priority for business leaders
› Base: 3,005 global data and analytics decision-makers › Source: Global Business Technographics Data And Analytics Online Survey, 2015
For you For all For segments For you
Demographic Relationships
Hyper-Personal, Real-Time
Relationships
Personal Relationships
Mass Relationships
Cus
tom
er E
xper
ienc
e
1800 1900 1950 2000 2015
#Celebrity
Customers want and increasingly expect to be treated like celebrities.
• Learn individual customer characteristics and behaviors
• Detect customer needs and desires in real-time
• Adapt applications to serve an individual customer
Celebrity experiences must:
#Analytics
© 2015 Forrester Research, Inc. Reproduction Prohibited 9
Learn Model Detect Adapt
True BI means having four kinds of analytics
Predictive Analytics
Streaming Analytics
Descriptive Analytics
(Advanced Analytics)
Prescriptive Analytics
Batch Real-time
Most firms invest here They must invest here too
© 2015 Forrester Research, Inc. Reproduction Prohibited 10
Source: Forrester Research
Advanced analytics is rightly surging “What is your firm's/business unit's current use of the following technologies?”
Source: Forrester's Global Business Technographics Data And Analytics Survey, 2015 and 2014 Base: 1805 (2015), 1063 (2014)
19%
19%
24%
31%
34%
22%
22%
35%
31%
43%
53%
54%
50%
50%
69%
39%
42%
42%
42%
42%
43%
43%
46%
48%
52%
54%
55%
56%
57%
69%
Non modeled data exploration and discovery
Search/interactive discovery
Streaming analytics
Metadata generated analytics
OLAP
Advanced visualization
Text analytics
Location analytics
Predictive analytics
Process analytics
Embedded analytics
Web analytics
Dashboards
Performance analytics
Reporting
2015
2014
Sweet! Most of your competitors still haven’t
started!
#Data
Some say that data is the new oil.
It’s more like the Sun – virtually limitless.
© 2015 Forrester Research, Inc. Reproduction Prohibited 14
Every industry is graced with more data › Richer transactional data from portfolio of hundreds of
business applications
› Usage and behavior data from web and mobile apps
› Social media data
› Log data
› IoT device sensor and event data
› Data economy – firms buying and selling data
Using your best estimate, what is the size of all data stored within your company?
Source: Forrester Research, September 2015 Base: 100 US Managers and above currently using Hadoop for processing and analyzing data.
Enterprises have plenty of data from both internal and external sources
10-49 Terabytes
5% 50-99 Terabytes
12%
100-500 Terabytes
54%
Greater than 500
Terabytes 29%
Internal business
data 49%
External source data
51%
What % of the data available is from internal business applications (ERP and business
applications) versus external sources (social, IoT)?
All data is born fast!
1100
1001
1011
001
0100
1001
1011
001
0100
1100
1101
101
0100
1001
1011
001
Cus
tom
er D
ata
Tran
sact
ions
Dat
a W
areh
osue
IoT
But, analytics is usually done much later.
#WhyWait
Perishable insights can have exponentially more value than after-the-fact traditional historical
analytics.
#Perishable
How can you prevent this dude from fleecing you right now?
What are movers and shakers saying about equities that we cover right now?
How can you know if your baby is sleeping soundly or if something is wrong right now?
How can you warn other drivers that the road is slippery to avoid a crash right now?
What offers should you make to your customer if they are within proximity of your store right now?
What music should you play if your customer is jogging right now?
#
© 2015 Forrester Research, Inc. Reproduction Prohibited 28
Most firms struggle make insights actionable
Insights for lines of business:
Source: October 16, 2014, “The Customer-Activated Enterprise” Forrester report
Applications are blind – IoT can make them see.
© 2015 Forrester Research, Inc. Reproduction Prohibited 30
Learn Model Detect Adapt
Build analytics into your applications
Predictive Analytics
Streaming Analytics
Descriptive Analytics
(Advanced Analytics)
Prescriptive Analytics
Batch Real-time
Most firms invest here They must invest here too
#Predictive
© 2015 Forrester Research, Inc. Reproduction Prohibited 32
Top 3 most read Forrester research reports in Q3 2015.
#1 Does Customer Experience Really Drive Business Success?
#2 Forrester Wave: Big Data Predictive Analytics, Q2 2015
#3 The US Customer Experience Index, Q12015
ANALYTICS
PREDICTIVE Techniques, tools, and technologies that use
data to find models – models that can anticipate outcomes with a significant
probability of accuracy.
© 2015 Forrester Research, Inc. Reproduction Prohibited 34
Ways to create predictive models › Handcraft business rules or program code to
create a predictive model based on human experts.
› Use big data predictive analytics tools to build models that analyze data with machine learning algorithms to build a predictive model.
› Sometimes you need both.
© 2015 Forrester Research, Inc. Reproduction Prohibited 35
Predictive models can be very powerful and profitable, but understand that:
› Predictive models are about probabilities, not absolutes • E.g. 78% chance you will like Better Call Saul
› Accurate predictive models may not exist for every question
• E.g. Economists have a very poor record
› Prediction models are not necessarily and probably not causative
Correlation does not imply causation.
Data scientists use a combination of statistical
and machine learning algorithms to find
patterns and predictive models.
© 2015 Forrester Research, Inc. Reproduction Prohibited 38
Are you a data scientist?
38
K-means clustering Association rules Boosting trees CHAID Cluster analysis Feature selection Independent components analysis Kohonen Networks (SOFM) Neural networks Social network analysis (SNA)
Random forests Mars regression splines Linear and logistic regression Naïve Bayesian classifiers Optimal binning Partial least squares Response Optimization Root cause analysis Support vector machines Natural language processing
© 2013 Forrester Research, Inc. Reproduction Prohibited 39
Big data is the fuel and machine learning is the engine › Classifiers
• Predict a specific event, characteristic, or behavior
› Recommenders • Make a recommendation
› Clusters • Find groups that share common characteristics
What customers are likely to plan a vacation in the next 6 weeks?
#Prescriptive
Adapt and act in real-time.
ANALYTICS
PRESCRIPTIVE Tools, techniques, and technologies that are used to determine the next best decision or
action using a combination of methods including business rules, descriptive/predictive/streaming
analytics, and optimization.
How can you influence swing voters to vote for you?
Image source: iStockphoto
45 © 2015 Forrester Research, Inc. Reproduction Prohibited
You must support both analytical and intuitive decision makers.
Analytical Intuitive
Automated Decision can be modeled Information is available Algorithm can be defined
Expert Decision is not modeled Unknown or fuzzy information Experience and judgment
Image sources: Cryteria, Eurogamer.net, Business Insider, CBS Entertainment, Cinema Blend
46 © 2015 Forrester Research, Inc. Reproduction Prohibited
Prescriptive analytics relies on a combination of approaches • Business rules and/or program code based on
human expertise and experience. • Use previously built predictive models. • Use previously solved mathematical
optimizations. You will likely need a combination of all of these to make the best possible decision.
#Streaming
Streaming analytics can detect and act on perishable insights.
DEFINITION
FORRESTER Streaming analytics filter, aggregate, enrich, and analyze a high throughput of data from
disparate live data sources to identify patterns, detect urgent situations, and automate
immediate actions in real-time.
© 2015 Forrester Research, Inc. Reproduction Prohibited 50
Real-time means business-time › A customer walks into a shopping mall
› A shopper clicks on an online add
› A temperature sensor spikes
› A stock price rises
› A customer uses a credit card
› A customer wakes up
© 2015 Forrester Research, Inc. Reproduction Prohibited 51
Thinking in streams is different…
› Ingest
› Filter
› Transform
› Normalize
› Link
› Enrich
› Correlate
› Location/motion (geofencing)
› Time windows
› Temporal pattern detection
› Business logic/rules execution
› Action interfaces
Continuous ETL Continuous Analytics
How can an online retailer sell more motorcycle helmets and optimize
profits?
› Temporal pattern detection
› Time windows
› Business logic/rules execution
› Action interfaces
© 2015 Forrester Research, Inc. Reproduction Prohibited 53
Streaming analytics enables an entirely new real-time selling model
› Analytic: When has this user viewed at least three motorcycle safety products including at least one helmet?
› Action: Display most profitable motorcycle helmets.
› Analytic: What is the real-time daily total sales of motorcycle helmets?
› Action: If sales trending lower than usual, then dynamically lower price.
Temporal Pattern Detection Time Window
© 2015 Forrester Research, Inc. Reproduction Prohibited 54
Try doing that with plain-old, conventional SQL
SELECT SUM(CustomerViews.view) WHERE CustomerViews.productCategory = ‘MCSafety’ AND …
#
Hadoop is designed for volume.
Spark is designed for speed.
Spark on Hadoop minimizes the latency of loading HDFS data into memory, and DevOps can use
YARN to tune performance for simultaneous jobs.
© 2015 Forrester Research, Inc. Reproduction Prohibited 59
Spark and Hadoop can coexist in the same cluster.
Hadooponomics makes batch analytics at scale feasible
SELECT * FROM Hadoop;
Pure SQL for Hadoop
Boosted SQL for Hadoop
• Apache Hive • Apache Drill • Apache Phoenix
(for HBase) • Cloudera Impala • Presto • Spark SQL
(through Spark)
• Actian Vortex • HP Vertica SQL
on Hadoop • IBM Big SQL • JethroData • Pivotal HAWQ
• Microsoft Polybase
• Oracle Big Data SQL
• Teradata QueryGrid
Database+ SQL for Hadoop
Spark’s directed acyclic graph (DAG) engine maximizes parallelization for batch jobs.
© 2015 Forrester Research, Inc. Reproduction Prohibited 63
Spark also includes a growing number of specialized tools
#Lambda
© 2015 Forrester Research, Inc. Reproduction Prohibited 65
Learn Model Detect Adapt
Only the analytical enterprise can compete and win in the age of celebrity
Predictive Analytics
Streaming Analytics
Descriptive Analytics
(Real-time)
Prescriptive Analytics
(Continuous Batch)
þ þ þ þ Invest here Invest here Invest here
© 2015 Forrester Research, Inc. Reproduction Prohibited 66
Combine application middleware and analytics to create industrial strength “lambda” applications
#
1100
1001
1011
001
0100
1001
1011
001
0100
1100
1101
101
0100
1001
1011
001
His
toric
al
Tran
sact
ions
Cus
tom
er d
ata
Sec
urity
Silos must disappear or appear to disappear.
Performance of analytics and services must be blazing fast.
Scale to handle any amount of data.
Fault-tolerance is non-negotiable.
Confidential information must be secure.
Solutions and/or platforms must fit seamlessly into existing architectures.
Embed and act on real-time analytics in existing and new applications.
Diverse, community-based rapid innovation.
#You
Your customers are ready to be
treated like celebrities!
What kind of celebrity customer experiences can you create with a
great open source platform?
© 2015 Forrester Research, Inc. Reproduction Prohibited 79
Think like a venture capitalist to find opportunities in your business.
1. Walk through critical or challenging business processes - At each step of the business process ask how analytics could improve the process
2. Walk through customer experience to improve customer experience
- At each step of the customer journey, ask how analytics could help create celebrity customer experiences
Build real-time, predictive apps to make it happen.