scaling up machine learning algorithms for classification

Scaling up Machine Learning algorithms for classification

Department of Mathematical InformaticsThe University of Tokyo

Shin Matsushima

How can we scale up Machine Learning to Massive datasets?

• Exploit hardware traits– Disk IO is bottleneck– Dual Cached Loops– Run Disk IO and Computation simultaneously

• Distributed asynchronous optimization (ongoing)– Current work using multiple machines

2

LINEAR SUPPORT VECTOR MACHINES VIA DUAL CACHED LOOPS

3

• Intuition of linear SVM

– xi: i-th datapoint

– yi: i-th label. +1 or -1

– yi w ･ xi : larger is better, smaller is worse

4

××

×

×

×

×

×

×× ×: yi = +1

×: yi = -1

• Formulation of Linear SVM

– n: number of data points– d: number of features– Convex non-smooth optimization

5

• Formulation of Linear SVM – Primal

– Dual

6

Coordinate descent7

8

9

10

11

12

13

14

• Coordinate Descent Method– For each update we solve one-variable optimization

problem with respect to the variable to update.

15

• Applying Coordinate Descent for Dual formulation of SVM

16

17

• Applying Coordinate Descent for Dual formulation of SVM

Dual Coordinate Descent [Hsieh et al. 2008]

18

Attractive property

• Suitable for large scale learning– We need only one data for each update.

• Theoretical guarantees– Linear convergence （ cf. SGD ）

• Shrinking[Joachims 1999]

– We can eliminate “uninformative” data:

cf.

19

Shrinking [Joachims 1999]

• Intuition: a datapoint far from the current decision boundary is unlikely to become a support vector

20

×

×

×

×

○

○

Shrinking [Joachims 1999]

• Condition

• Available only in the dual problem

21

Problem in scaling up to massive data

• In dealing with small-scale data, we first copy the entire dataset into main memory

• In dealing with large-scale data, we cannot copy the dataset at once

22

Read

Disk

Memory

Data

ReadData

• Schemes when data cannot fit in memory1. Block Minimization [Yu et al. 2010]– Split the entire dataset into blocks so that each

block can fit in memory

Train RAM

• Schemes when data cannot fit in memory1. Block Minimization [Yu et al. 2010]– Split the entire dataset into blocks so that each

block can fit in memory

ReadData

• Schemes when data cannot fit in memory1. Block Minimization [Yu et al. 2010]– Split the entire dataset into blocks so that each

block can fit in memory

Train RAM

• Schemes when data cannot fit in memory1. Block Minimization [Yu et al. 2010]– Split the entire dataset into blocks so that each

block can fit in memory

Block Minimization[Yu et al. 2010]

27

ReadData

• Schemes when data cannot fit in memory2. Selective Block Minimization [Chang and Roth 2011]

– Keep “informative data” in memory

Train RAM

Block

• Schemes when data cannot fit in memory2. Selective Block Minimization [Chang and Roth 2011]

– Keep “informative data” in memory

Train RAM

Block

• Schemes when data cannot fit in memory2. Selective Block Minimization [Chang and Roth 2011]

– Keep “informative data” in memory

ReadData

• Schemes when data cannot fit in memory2. Selective Block Minimization [Chang and Roth 2011]

– Keep “informative data” in memory

Train RAM

Block

• Schemes when data cannot fit in memory2. Selective Block Minimization [Chang and Roth 2011]

– Keep “informative data” in memory

Train RAM

Block

• Schemes when data cannot fit in memory2. Selective Block Minimization [Chang and Roth 2011]

– Keep “informative data” in memory

Selective Block Minimization[Chang and Roth 2011]

34

• Previous schemes switch CPU and DiskIO– Training (CPU) is idle while reading– Reading (DiskIO) is idle while training

35

• We want to exploit modern hardware1. Multicore processors are commonplace2. CPU(Memory IO) is often 10-100 times

faster than Hard disk IO

36

1.Make reader and trainer run simultaneously and almost asynchronously.

2.Trainer updates the parameter many times faster than reader loads new datapoints.

3.Keep informative data in main memory.(=Evict uninformative data primarily from main memory)

37

Dual Cached Loops

ReaderThread

TrainerThread

Parameter

Dual Cached Loops

RAM

Disk

Memory

Data

38

ReaderThread

TrainerThread

Parameter

Dual Cached Loops

RAM

Disk

Memory

Data

39

Read

Disk

Memory

Data

W: working 　index set

40

Train

ParameterMemory

41

Which data is “uninformative”?

• A datapoint far from the current decision boundary is unlikely to become a support vector

• Ignore the datapoint for a while.

42

××

×

×

×

○

○○

Which data is “uninformative”?

– Condition

43

• Datasets with Various Characteristics:

• 2GB Memory for storing datapoints • Measured Relative Function Value

45

• Comparison with (Selective) Block Minimization (implemented in Liblinear)

– ocr ： dense, 45GB

46

47

• Comparison with (Selective) Block Minimization (implemented in Liblinear)

– dna ： dense, 63GB

48

Comparison with (Selective) Block Minimization (implemented in Liblinear)

– webspam ： sparse, 20GB

49

Comparison with (Selective) Block Minimization (implemented in Liblinear)

– kddb ： sparse, 4.7GB

• When C gets larger (dna C=1)

51

• When C gets larger(dna C=10)

52

• When C gets larger(dna C=100)

53

• When C gets larger(dna C=1000)

54

• When memory gets larger(ocr C=1)

55

• Expanding Features on the fly– Expand features explicitly when the reader thread

loads an example into memory.• Read (y,x) from the Disk• Compute f(x) and load (y,f(x)) into RAM

Read

Disk

Data

12495340( )x R

x=GTCCCACCT…

56

2TB data

16GB memory

10hrs

50M examples

12M featurescorresponding to

2TB

57

• Summary– Linear SVM Optimization when data cannot fit in

memory– Use the scheme of Dual Cached Loops– Outperforms state of the art by orders of magnitude– Can be extended to• Logistic regression• Support vector regression• Multiclass classification

58

DISTRIBUTED ASYNCHRONOUS OPTIMIZATION (CURRENT WORK)

59

Future/Current Work

• Utilize the same principle as dual cached loops in multi-machine algorithm– Transportation of data can be efficiently done without

harming optimization performance– The key is to run Communication and Computation

simultaneously and asynchronously– Can we do more sophisticated communication

emerging in multi-machine optimization?

60

• (Selective) Block Minimization scheme for Large-scale SVM

61

Move data Process Optimization

HDD/ File 　system

One machine

One machine

• Map-Reduce scheme for multi-machine algorithm

62

Move parameters Process Optimization

Master node

Workernode

Workernode

63

≈

64

65

Stratified Stochastic Gradient Descent [Gemulla, 2011]

66

67

68

• Map-Reduce scheme for multi-machine algorithm

69

Move parameters Process Optimization

Master node

Workernode

Workernode

Asynchronous multi-machine scheme70

Parameter Communication

Parameter Updates

NOMAD71

NOMAD72

73

74

75

76

Asynchronous multi-machine scheme

• Each machine holds a subset of data• Keep communicating a potion of parameter from

each other• Simultaneously run updating parameters for

those each machine possesses

77

• Distributed stochastic gradient descent for saddle point problems– Another formulation of SVM (Regularized Risk

Minimization in general)– Suitable for parallelization

78

How can we scale up Machine Learning to Massive datasets?

• Exploit hardware traits– Disk IO is bottleneck– Run Disk IO and Computation simultaneously

• Distributed asynchronous optimization (ongoing)– Current work using multiple machines

79