the analytics continuum

1

The Analytics Continuum

Rob Marano

7 May 2014

5/7/14© 2014 The Hackerati, Inc. This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

© 2014 The Hackerati, Inc. This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 2

“What’s measured improves.”

Peter F. Drucker

5/7/14


“Knowledge has to be improved, challenged, and increased constantly, or it vanishes.”

Peter F. Drucker

5/7/14


“When you develop your opinions on the basis of weak evidence, you will have difficulty interpreting subsequent information that

contradicts these opinions, even if this new information is obviously more accurate.”

Nassim Nicholas Taleb

The Black Swan: The Impact of the Highly Improbable

5/7/14


Agenda

• Execution vs. search• Balancing the “knowns” & “unknowns”• Data here, there, everywhere …• Machine learning as foundation to analytics• Visualization as action to analytics• Imminent opportunities

5/7/14


History of Analytics

Source: Economic Time of India

What drives the progression?5/7/14

http://economictimes.indiatimes.com/features/corporate-dossier/big-data-personal-analytics-going-beyond-sports-into-realms-of-wealth-life-satisfaction/articleshow/31942693.cms


Why Consider Such an Investment?

• Like any innovation, right?• Enable the business to gain

– Competitive advantage– Cost cutting via productivity or automation– Compliance

• But what about all that tech we already have?

Is change good to the bottom line?5/7/14


Why Consider Such an Investment?

• Machine learning is used in– Web search– Spam filters– Recommender systems– Ad placement– Credit scoring– Fraud detection– Stock trading– Drug design– and much more

5/7/14


Impact of “Startup Culture”

• The most successful of businesses have perfected execution

• They run operations with the highest level of efficiency and effectiveness for the business

• Like any auto-assist or fully automated system, the operations are modeled perfectly

Change is not considered a constant or asset5/7/14



• The most successful of starts have perfected change as its advantage to search for its niche

• Startups build solutions that anticipate change, especially on how to use data to pivot

• Data & analytics form core to manage change

Startups value change inherently 5/7/14



• The startup community continues to be the vendor of choice behind all modern analytics

• Google, Yahoo, Facebook, Twitter, etc … the list goes on

• Google started this “analytics age” – open source now dominates it

Any business has access to modern analytics5/7/14


Knowns & Unknowns

• Knowledge & business strategy– “Known knowns”– “Known unknowns”– “Unknown unknowns”

• Operations & strategy depend upon evidence• Timely get the right info to the right person

5/7/14


(Big) Data Here, There, Everywhere

• Data operates every process but not collected• The more online, the more potential• Advantages

– Competitive– Productivity/efficiency– Compliance

Wisdom

Knowledge

Info

Data

5/7/14


How Big is “Big Data?”

5/7/14

What’s big for your department? Company?Source: InfoChimps, “[Infographic] Taming Big Data from Wikibon”

http://blog.infochimps.com/2012/11/28/infographic-taming-big-data-wikibon/






Foundation of Analytics

• Historically rigid data dictionaries provided advantages via SQL and RDBMS

• As compute/storage reduced in cost & deployment complexities, more data processed

• Cost of infrastructure kept rising; state-of-the-art not keeping pace

Big Data enables commodity analytics5/7/14


Analytics Core

• Big Data– Commodity computation & storage– Modern computation framework– Open, loose-coupling of components

• Machine learning– Commodity knowledge discovery

• Delivered as a cost-effective service

5/7/14


IT Transition to Big Data Analytics

• Startup advantages lead to cost-effective analysis of large quantities of data

• Traditional data warehouse solutions do not effectively scale in cost nor productivity

• Growth of open source delivers both

New “open” vendors leading the way5/7/14


Big Data as Enabler

Source: VMware Blog, “4 Key Architecture Considerations for Big Data Analytics”5/7/14

http://blogs.vmware.com/vfabric/2012/08/4-key-architecture-considerations-for-big-data-analytics.html


Apache Hadoop as Epicenter

5/7/14

Dat

a In

tegr

ation

(Flu

me,

Chu

kwa,

Sqo

op)

Scripting(Pig)

Distributed Storage(HDFS)

Syst

ems

Man

agem

ent &

Mon

itorin

g(A

mba

ri, Z

ooke

eper

)

Wor

kflow

& S

ched

ulin

g(O

ozie

)

Dat

abas

e(H

base

, Cas

sand

ra)

Distributed Compute(MapReduce)

Meta Data Services(HCatalog)

Query(Hive)

Mac

hine

Lea

rnin

g(M

ahou

t)

Source: Hortonworks, “About Hortonworks Data Platform”

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0-Win/bk_getting-started-guide/content/ch_about-hortonworks-data-platform.html




The Hadoop Ecosystem• Ambari Deployment, configuration and monitoring• Flume Collection and import of log and event data• HBase Column-oriented database scaling to billions of rows• HCatalog Schema and data type sharing over Pig, Hive and MapReduce• HDFS Distributed redundant file system for Hadoop• Hive Data warehouse with SQL-like access• Mahout Library of machine learning and data mining algorithms• MapReduce Parallel computation on server clusters• Pig High-level programming language for Hadoop computations• Oozie Orchestration and workflow management• Sqoop Imports data from relational databases• Whirr Cloud-agnostic deployment of clusters• Zookeeper Configuration management and coordination

5/7/14Source: Edd Dumbill, “What is Apache Hadoop?”

http://strata.oreilly.com/2012/02/what-is-apache-hadoop.html




So, what is Machine Learning?

• Non-trivial process of finding and communicating “valid, novel, potentially useful and understandable patterns in data.”1

• Delivers the engineering behind the science of automated classification, categorization, and recommendation without being explicitly programmed

• Allows data to be transformed with relative ease into actionable knowledgeML powers today’s internet economies

1: Ciro Donalek, “Supervised & Unsupervised Learning”5/7/14

http://www.astro.caltech.edu/~george/aybi199/Donalek_Classif.pdf


Machine Learning as Enabler

• Open source, cloud computing, & startup culture powered rise of analytics

• Delivers powerful processing & results• Figures out how to perform a particularly

manual task by generalizing from examples

Tactics & strategy require evidence that learns5/7/14


Learning – Human or Machine

• Learning an iterative process to converge• The ML “space” is huge and growing, but get a

handle on the intended mission objectives– Representation

• Which group of classifiers will “it” learn; which features

– Evaluation• Distinguish good from bad classifiers

– Optimization• Which is the highest scoring classifier

1: Pedro Domingos, “A Few Useful Things to Know about Machine Learning”5/7/14

http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf


Analytics Starts With Data

5/7/14

Ingestion

Conversio

n

Upload

Image Source: Research Live, “Order from Chaos”

websites + web svcs

http://www.research-live.com/magazine/order-from-chaos/4004808.article


and It Ends with Knowledge

5/7/14

Aggregati

on

Analysis

Visualization

Image Source: Visualize This by Nahan Yau

Wisdom

Knowledge

Info

Data


Taxonomy of ML

• ML converts data trends into logic to automate data processing

• Based upon pattern recognition• Basic goal is generalization• Built upon two key techniques

– Supervised learning– Unsupervised learning

5/7/14


Supervised Learning

• ML technique which takes a training data set with specific features that result in a model

• The model is used to assess whether an input is of a pre-defined class

• Key to supervised learning remains feature set extraction

• Popular examples include– Regression– Classification– Outliers detection

5/7/14


Unsupervised Learning

• ML technique to group data according to similar features, or characteristics

• Such technique does not require a model to be generated, rather similarity is calculated

• Popular examples include– Clustering– Density estimation– Visualization by projection

5/7/14


Most Important Step in ML

• “Know thine data like thyself”– Know features about your data in order to narrow

the algorithm selection process– Are the features nominal or continuous?– Are there missing values in the features?– If missing values, where are they missing?– Are there outliers in the data?– Are you looking for something that occurs very

infrequently?

5/7/14


Choosing the ML Algorithm

• Know your data inside out & back again• Consider the goal• Use unsupervised unless need to predict certain

target values, then use supervised• Choose a set of algos matched to goal/data• Try each algorithm, assess and compare• Adjust and combine optimization techniques• Choose, operate, and continually measure• Repeat

5/7/14


Generalized ML Application Steps

• Collect data• Prepare the input data• Analyze input data & features• Train the algorithm (if supervised)• Test the algorithm with fresh data• Operate ML• Detect subtle changes to data (cycles,seasons)• Measure for performance• Repeat as frequently needed

5/7/14

Portions sourced: Machine Learning in Action by Peter Harrington, Manning Publications


Highlights of Supervised Algos

• Generalized Linear Models– Bayesian Regression– Ordinary least squares (regression)

• Support Vector Machines• K Nearest Neighbors• Naïve Bayes• Decision Trees• Neural Networks• Ensemble Methods

5/7/14

Portions sourced: “Supervised Learning” from scikit-learn.org

http://scikit-learn.org/stable/supervised_learning.html


Highlights of Unsupervised Algos

• Clustering• K-means• DBSCAN• Hidden Markov Models

• Density Estimation• Neural Networks (restricted Boltzmann)

5/7/14

Portions sourced: “Supervised Learning” from scikit-learn.org

http://scikit-learn.org/stable/supervised_learning.html


Learning -> Evaluation

5/7/14

• The Classifier Evaluation Framework

1 2 : Knowledge of 1 is necessary for 2

1 2 : Feedback from 1 should be used to adjust 2

Choice of Learning Algorithm(s)

Datasets Selection

Error-Estimation/ Sampling Method

Performance Measure of Interest Statistical Test

Perform Evaluation

Source: “Performance Evaluation of Machine Learning Algorithms” by Mohak Shah & Nathalie Japkowicz

http://www.ecmlpkdd2013.org/wp-content/uploads/2013/04/Performance-Evaluation-of-Machine-Learning-Algorithms.pptx



Overview of Performance Measures

5/7/14

All Measures

Additional Info (Classifier Uncertainty Cost ratio

Skew)

Confusion Matrix Alternate Information

Deterministic Classifiers Scoring Classifiers Continuous and Prob. Classifiers (Reliability

metrics)

Multi-class Focus

Single-class Focus

No Chance Correction

Chance Correction

Accuracy Error Rate

Cohen’s Kappa Fielss Kappa

TP/FP Rate Precision/Recall Sens./Spec. F-measure Geom. Mean Dice

Graphical Measures

Summary Statistic

Roc Curves PR Curves DET Curves Lift Charts Cost

Curves

AUCH Measure

Area under ROC- cost curve

Distance/Error measures

KL divergence K&B IR BIRRMSE

InformationTheoretic Measures

Interestingness Comprehensibility Multi-

criteria





Confusion Matrix-BasedPerformance Measures

5/7/14

• Multi-Class Focus:– Accuracy =

(TP+TN)/(P+N)

• Single-Class Focus: – Precision = TP/(TP+FP)– Recall = TP/P– Fallout = FP/N– Sensitivity = TP/(TP+FN)– Specificity =

TN/(FP+TN)

True class

Hypothesized class

Pos Neg

Yes TP FP

No FN TN

P=TP+FN N=FP+TN

Confusion Matrix





Tying It All Together

5/7/14


Visualization as Action

5/7/14


Imminent Opportunities

• Any business with high volume of data– Look at processes, human-machine interfaces– Sentiment; Customer Experience; Campaigns– Infosec; Network Services; Customer Churn

• Sectors coming analytics-ready– Healthcare; Government; Retail– Manufacturing; Utilities

• Imagine a world of Internet-of-Things?

Can you imagine keeping all data? Analyze it?5/7/14


Analytics

• Big Data– Commodity compute & storage

• Analytics– Commodity intelligence

• Big Data Analytics– Store everything– Analyze everything– Do it everyday

Cost effectively manage “unknown unknowns”5/7/14


“Know the enemy and know yourself; in a hundred battles you will never be in peril.”

Sun Tzu

5/7/14


“It’s no longer hard to find the answer to a given question; the hard part is finding the right question, and as questions evolve, we gain

better insight into our own ecosystem and our business.”

Kevin Weil

Director of Product for RevenueTwitter

5/7/14


The Analytics Continuum

Rob [email protected]

7 May 2014

5/7/14

mailto:[email protected]

the analytics continuum

Data & Analytics

international license

commodity analytics

data analytics form

foundation of analytics

knowns unknowns data

analytics visualization

modern analytics google

history of analytics