apache spark machine learning decision trees

© 2016 MapR Technologies 10-1© 2016 MapR Technologies

Machine Learning with Apache SparkCarol McDonald, Solution ArchitectStrata + Hadoop World March 2016

© 2016 MapR Technologies 10-3

Agenda

• Brief overview of • Classification• Clustering• Collaborative Filtering

• Predicting Flight Delays using a Decision Tree


Spark SQL

• Structured Data• Querying with

SQL/HQL• DataFrames

Spark Streaming

• Processing of live streams

• Micro-batching

MLlib

• Machine Learning• Multiple types of

ML algorithms

GraphX

• Graph processing• Graph parallel

computations

RDD Transformations and Actions

• Task scheduling• Memory management• Fault recovery• Interacting with storage systems

Spark Core

What is MLlib?


MLlib Algorithms and Utilities

Algorithms and Utilities Description

Basic statistics Includes summary statistics, correlations, hypothesis testing, random data generation

Classification and regression Includes methods for linear models, decision trees and Naïve Bayes

Collaborative filtering Supports model-based collaborative filtering using alternating least squares (ALS) algorithm

Clustering Supports K-means clustering

Dimensionality reduction Supports dimensionality reduction on the RowMatrix class; singular value decomposition (SVD) and principal component analysis (PCA)

Feature extraction and transformation

Contains several classes for common feature transformations


Examples of ML Algorithms

Supervised• Classification

– Naïve Bayes– SVM– Random Decision

Forests

• Regression– Linear– Logistic

Machine Learning

Unsupervised• Clustering

– K-means

• Dimensionality reduction– Principal Component

Analysis– SVD





Forests


Machine Learning


– K-means


Analysis– SVD



Machine Learning


– K-means


Analysis– SVD



Forests



Three Categories of Techniques for Machine Learning

Collaborative Filtering (Recommendation)

Classification Clustering


Machine Learning: ClassificationClassification

Identifies category for item


Classification: DefinitionForm of ML that:• Identifies which category an item belongs to • Uses supervised learning algorithms

– Data is labeled

FRAUD

Sentiment


If it Walks/Swims/Quacks Like a Duck …… Then It Must Be a Duck

swims

walks

quacks

Features:

walks

quacksswims

Features:

ducksnot ducks


Building and Deploying a Classifier Model

Image reference O’Reilly Learning Spark

Spam:free money now!get this moneyfree savings $$$

Training Data

Non-spam:how are you?that Spark joblunch plans




++ ;+

; ;

Feature Vectors

FeaturizationSpam:

free money now!get this moneyfree savings $$$

Training Data





++ ;+

; ;

Feature Vectors Model

Featurization TrainingSpam:free money now!get this moneyfree savings $$$

Training Data


++ ;+

; ;




++ ;+

; ;

Feature Vectors Model

Featurization TrainingModel

Evaluation

Best Model

Spam:free money now!get this moneyfree savings $$$

Training Data


++ ;+; ;

++ ;+

; ;

++ ;+

; ;

++ ;+

; ;


Machine Learning: ClusteringClassification Clustering


Clustering: Definition• Unsupervised learning task• Groups objects into clusters of high similarity


Clustering: Definition• Unsupervised learning task• Groups objects into clusters of high similarity

– Search results grouping– Grouping of customers– Anomaly detection– Text categorization


Clustering: Example• Group similar objects• Use MLlib K-means algorithm

1. Initialize coordinates to center of clusters (centroid)

2. Assign all points to nearest centroid

3. Update centroids to center of points

4. Repeat until conditions met


Three Categories of Techniques for Machine Learning

Collaborative Filtering (Recommendation)

Classification Clustering


Collaborative Filtering with Spark• Recommend items

– (Filtering)

• Based on user preferences data

– (Collaborative)

4 5 5

5 5

5 ?

Ted

Carol

Bob

A B C

User Item Rating Matrix


Train a Model to Make Predictions

Ted and Carol like movies B and C

Bob likes movie B, what might he like?

Bob likes movie B, predict C

TrainingData ModelAlgorithm

New Data PredictionsModel

4 5 5

5 5

5 ?

Ted

Carol

Bob

A B C

User Item Rating Matrix


Predict Flight Delays


Use Case: Flight Data• Predict if a flight is going to be delayed• Use Decision Tree for prediction

• Used for Classification and Regression• Represents tree with nodes, Binary decision at each node


Flight Data


// Define the schemacase class Flight(dofM: String, dofW: String, carrier: String, tailnum: String, flnum: Int, org_id: String, origin: String, dest_id: String, dest: String, crsdeptime: Double, deptime: Double, depdelaymins: Double, crsarrtime: Double, arrtime: Double, arrdelay: Double, crselapsedtime: Double, dist: Int)

def parseFlight(str: String): Flight = { val line = str.split(",") Flight(line(0), line(1), line(2), line(3), line(4).toInt, line(5), line(6), line(7), line(8), line(9).toDouble, line(10).toDouble, line(11).toDouble, line(12).toDouble, line(13).toDouble, line(14).toDouble, line(15).toDouble, line(16).toInt)}// load file into a RDDval rdd = sc.textFile(”flights.csv”)// create an RDD of Flight objectsval flightRDD = rdd.map(parseFlight).cache()//Array(Flight(1,3,AA,N338AA,1,12478,JFK,12892,LAX 900.0,914.0,14.0,1225.0,1238.0, 13.0,385.0,2475)

Parse Input



++ ;+

; ;

Feature Vectors

FeaturizationDelayed:

FridayLAXAA

Training Data

Not Delayed:WednesdayBNADelta


Classification Learning Problem - Features

• Label delayed and not delayed - delayed if delay > 40 minutes

• Features {day_of_month, weekday, crsdeptime, crsarrtime, carrier, crselapsedtime, origin, dest}


// create map of airline -> numbervar carrierMap: Map[String, Int] = Map()var index: Int = 0flightsRDD.map(flight => flight.carrier).distinct.collect.foreach( x => { carrierMap += (x -> index); index += 1 } )carrierMap.toString// String = Map(DL -> 5,US -> 9, AA -> 6, UA -> 4...)

// create map of destination airport -> numbervar destMap: Map[String, Int] = Map()var index2: Int = 0flightsRDD.map(flight => flight.dest).distinct.collect.foreach( x => { destMap += (x -> index2); index2 += 1 })destMap.toString// Map(JFK -> 214, LAX -> 294, ATL -> 273,MIA -> 175 ...

Transform non-numeric features into numeric values


Classification Learning Problem - Features

• Label delayed and not delayed - delayed if delay > 40 minutes

• Features {day_of_month, weekday, crsdeptime, crsarrtime, carrier, crselapsedtime, origin, dest}

• MLLIB Datatypes:• Vector: Contains the feature data points• LabeledPoint: Contains feature vector and label


// Defining the features arrayval mlprep = flightsRDD.map(flight => { val monthday = flight.dofM.toInt - 1 // category val weekday = flight.dofW.toInt - 1 // category val crsdeptime1 = flight.crsdeptime.toInt val crsarrtime1 = flight.crsarrtime.toInt val carrier1 = carrierMap(flight.carrier) // category val crselapsedtime1 = flight.crselapsedtime.toDouble val origin1 = originMap(flight.origin) // category val dest1 = destMap(flight.dest) // category val delayed = if (flight.depdelaymins.toDouble > 40) 1.0 else 0.0 Array(delayed.toDouble, monthday.toDouble, weekday.toDouble, crsdeptime1.toDouble, crsarrtime1.toDouble, carrier1.toDouble, crselapsedtime1.toDouble, origin1.toDouble, dest1.toDouble)})mlprep.take(1)//Array(Array(0.0, 0.0, 2.0, 900.0, 1225.0, 6.0, 385.0, 214.0, 294.0))val mldata = mlprep.map(x => LabeledPoint(x(0),Vectors.dense(x(1),x(2),x(3),x(4), x(5),x(6), x(7), x(8))))mldata.take(1)// Array[LabeledPoint] = Array((0.0,[0.0,2.0,900.0,1225.0,6.0,385.0,214.0,294.0]))

Define the features, Create LabeledPoint with Vector


Build ModelSplit data into:• Training data RDD (80%) • Test data RDD (20%)

Data Build Model

TrainingSet

TestSet


// Randomly split RDD into training data RDD (80%) and test data RDD (20%)val splits = mldata.randomSplit(Array(0.8, 0.2))

val trainingRDD = splits(0).cache()val testRDD = splits(1).cache()testData.take(1)//Array[LabeledPoint] = Array((0.0,[18.0,6.0,900.0,1225.0,6.0,385.0,214.0,294.0]))

Split Data


Build Model

Training Set with Labels, Build a model

Data Build Model

TrainingSet

TestSet


Use Case: Flight Data• Predict if a flight is going to be delayed• Use Decision Tree for prediction

• Used for Classification and Regression• Represents tree with nodes• Binary decision at each node


// set ranges for categorical features var categoricalFeaturesInfo = Map[Int, Int]()categoricalFeaturesInfo += (0 -> 31) //dofM 31 categoriescategoricalFeaturesInfo += (1 -> 7) //dofW 7 categoriescategoricalFeaturesInfo += (4 -> carrierMap.size) //number of carrierscategoricalFeaturesInfo += (6 -> originMap.size) //number of origin airportscategoricalFeaturesInfo += (7 -> destMap.size) //number of dest airportsval numClasses = 2val impurity = "gini"val maxDepth = 9val maxBins = 7000

// call DecisionTree trainClassifier with the trainingData , which returns the modelval model = DecisionTree.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo,

impurity, maxDepth, maxBins)

Build Model


// print out the decision treemodel.toDebugString// 0=dofM 4=carrier 3=crsarrtime1 6=origin res20: String = DecisionTreeModel classifier of depth 9 with 919 nodes If (feature 0 in {11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0,20.0,21.0,

22.0,23.0,24.0,25.0,26.0,27.0,30.0}) If (feature 4 in {0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,13.0}) If (feature 3 <= 1603.0) If (feature 0 in {11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0}) If (feature 6 in {0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,10.0,11.0,12.0,13.0...

Build Model


Get Predictions

TestData

Without label

PredictDelay or NotModel


// Get Predictions,create RDD of test Label, test Predictionval labelAndPreds = testData.map { point => val prediction = model.predict(point.features) (point.label, prediction)}labelAndPreds.take(1)// Label, Prediction//Array((0.0,0.0))

Get Predictions


// get instances where label != predictionval wrongPrediction =(labelAndPreds.filter{ case (label, prediction) => ( label !=prediction) })

val wrong= wrongPrediction.count()res35: Long = 11040

val ratioWrong=wrong.toDouble/testData.count()

ratioWrong: Double = 0.3157443157443157

Test Model


To Learn More:• Download example code

– https://github.com/caroljmcdonald/sparkmldecisiontree• Read explanation of example code

– https://www.mapr.com/blog/apache-spark-machine-learning-tutorial

• Engage with us!– https://www.mapr.com/blog/author/carol-mcdonald

– https://community.mapr.com

https://github.com/caroljmcdonald/sparkmldecisiontree

https://github.com/caroljmcdonald/sparkmldecisiontree

https://www.mapr.com/blog/apache-spark-machine-learning-tutorial

https://www.mapr.com/blog/apache-spark-machine-learning-tutorial

https://www.mapr.com/blog/author/carol-mcdonald


https://community.mapr.com/




Q & A@mapr


Engage with us!

mapr-technologies

apache spark machine learning decision trees

Software