Machine Learning with WEKA

6/12/2015 University of Waikato 1

6/12/2015 University of Waikato 2

WEKA: the bird

Copyright: Martin Kramer (mkramer@wxs.nl)

6/12/2015 University of Waikato 3

WEKA: the software

Machine learning/data mining software written in

Java (distributed under the GNU Public License)

Used for research, education, and applications

Complements “Data Mining” by Witten & Frank

Main features:

Comprehensive set of data pre-processing tools,

learning algorithms and evaluation methods

Graphical user interfaces (incl. data visualization)

Environment for comparing learning algorithms

6/12/2015 University of Waikato 4

WEKA: versions

There are several versions of WEKA:

WEKA 3.0: “book version” compatible with

description in data mining book

WEKA 3.2: “GUI version” adds graphical user

interfaces (book version is command-line only)

WEKA 3.3: “development version” with lots of

improvements

This talk is based on the latest snapshot of WEKA

3.3 (soon to be WEKA 3.4)

6/12/2015 University of Waikato 5

@relation heart-disease-simplified

@attribute age numeric

@attribute sex { female, male}

@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}

@attribute cholesterol numeric

@attribute exercise_induced_angina { no, yes}

@attribute class { present, not_present}

@data

63,male,typ_angina,233,no,not_present

67,male,asympt,286,yes,present

67,male,asympt,229,yes,present

38,female,non_anginal,?,no,not_present

...

WEKA only deals with “flat” files

6/12/2015 University of Waikato 6

@relation heart-disease-simplified

@attribute age numeric

@attribute sex { female, male}

@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}

@attribute cholesterol numeric

@attribute exercise_induced_angina { no, yes}

@attribute class { present, not_present}

@data

63,male,typ_angina,233,no,not_present

67,male,asympt,286,yes,present

67,male,asympt,229,yes,present

38,female,non_anginal,?,no,not_present

...

WEKA only deals with “flat” files

6/12/2015 University of Waikato 7

6/12/2015 University of Waikato 8

6/12/2015 University of Waikato 9

6/12/2015 University of Waikato 10

Explorer: pre-processing the data

Data can be imported from a file in various

formats: ARFF, CSV, C4.5, binary

Data can also be read from a URL or from an SQL

database (using JDBC)

Pre-processing tools in WEKA are called “filters”

WEKA contains filters for:

Discretization, normalization, resampling, attribute

selection, transforming and combining attributes, …

6/12/2015 University of Waikato 11

6/12/2015 University of Waikato 12

6/12/2015 University of Waikato 13

6/12/2015 University of Waikato 14

6/12/2015 University of Waikato 15

6/12/2015 University of Waikato 16

6/12/2015 University of Waikato 17

6/12/2015 University of Waikato 18

6/12/2015 University of Waikato 19

6/12/2015 University of Waikato 20

6/12/2015 University of Waikato 21

6/12/2015 University of Waikato 22

6/12/2015 University of Waikato 23

6/12/2015 University of Waikato 24

6/12/2015 University of Waikato 25

6/12/2015 University of Waikato 26

6/12/2015 University of Waikato 27

6/12/2015 University of Waikato 28

6/12/2015 University of Waikato 29

6/12/2015 University of Waikato 30

6/12/2015 University of Waikato 31

6/12/2015 University of Waikato 32

Explorer: building “classifiers”

Classifiers in WEKA are models for predicting

nominal or numeric quantities

Implemented learning schemes include:

Decision trees and lists, instance-based classifiers,

support vector machines, multi-layer perceptrons,

logistic regression, Bayes’ nets, …

“Meta”-classifiers include:

Bagging, boosting, stacking, error-correcting output

codes, locally weighted learning, …

6/12/2015 University of Waikato 33

6/12/2015 University of Waikato 34

6/12/2015 University of Waikato 35

6/12/2015 University of Waikato 36

6/12/2015 University of Waikato 37

6/12/2015 University of Waikato 38

6/12/2015 University of Waikato 39

6/12/2015 University of Waikato 40

6/12/2015 University of Waikato 41

6/12/2015 University of Waikato 42

6/12/2015 University of Waikato 43

6/12/2015 University of Waikato 44

6/12/2015 University of Waikato 45

6/12/2015 University of Waikato 46

6/12/2015 University of Waikato 47

6/12/2015 University of Waikato 48

6/12/2015 University of Waikato 49

6/12/2015 University of Waikato 50

6/12/2015 University of Waikato 51

6/12/2015 University of Waikato 52

6/12/2015 University of Waikato 53

6/12/2015 University of Waikato 54

6/12/2015 University of Waikato 55

6/12/2015 University of Waikato 56

6/12/2015 University of Waikato 57

6/12/2015 University of Waikato 58

6/12/2015 University of Waikato 59

6/12/2015 University of Waikato 60

6/12/2015 University of Waikato 61

6/12/2015 University of Waikato 62

6/12/2015 University of Waikato 63

6/12/2015 University of Waikato 64

6/12/2015 University of Waikato 65 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

6/12/2015 University of Waikato 66 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

6/12/2015 University of Waikato 67 QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

6/12/2015 University of Waikato 68

6/12/2015 University of Waikato 69

6/12/2015 University of Waikato 70

6/12/2015 University of Waikato 71

6/12/2015 University of Waikato 72

6/12/2015 University of Waikato 73

6/12/2015 University of Waikato 74

6/12/2015 University of Waikato 75

Quic k Time™ and a TIFF (LZW) dec ompres s or are needed to s ee this pic ture.

6/12/2015 University of Waikato 76

6/12/2015 University of Waikato 77

6/12/2015 University of Waikato 78

6/12/2015 University of Waikato 79

6/12/2015 University of Waikato 80

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

6/12/2015 University of Waikato 81

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

6/12/2015 University of Waikato 82

6/12/2015 University of Waikato 83

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

6/12/2015 University of Waikato 84

6/12/2015 University of Waikato 85

6/12/2015 University of Waikato 86

6/12/2015 University of Waikato 87

6/12/2015 University of Waikato 88

6/12/2015 University of Waikato 89

6/12/2015 University of Waikato 90

6/12/2015 University of Waikato 91

6/12/2015 University of Waikato 92

Explorer: clustering data

WEKA contains “clusterers” for finding groups of

similar instances in a dataset

Implemented schemes are:

k-Means, EM, Cobweb, X-means, FarthestFirst

Clusters can be visualized and compared to “true”

clusters (if given)

Evaluation based on loglikelihood if clustering

scheme produces a probability distribution

6/12/2015 University of Waikato 93

6/12/2015 University of Waikato 94

6/12/2015 University of Waikato 95

6/12/2015 University of Waikato 96

6/12/2015 University of Waikato 97

6/12/2015 University of Waikato 98

6/12/2015 University of Waikato 99

6/12/2015 University of Waikato 100

6/12/2015 University of Waikato 101

6/12/2015 University of Waikato 102

6/12/2015 University of Waikato 103

6/12/2015 University of Waikato 104

6/12/2015 University of Waikato 105

6/12/2015 University of Waikato 106

6/12/2015 University of Waikato 107

6/12/2015 University of Waikato 108

Explorer: finding associations

WEKA contains an implementation of the Apriori

algorithm for learning association rules

Works only with discrete data

Can identify statistical dependencies between

groups of attributes:

milk, butter bread, eggs (with confidence 0.9 and

support 2000)

Apriori can compute all rules that have a given

minimum support and exceed a given confidence

6/12/2015 University of Waikato 109

6/12/2015 University of Waikato 110

6/12/2015 University of Waikato 111

6/12/2015 University of Waikato 112

6/12/2015 University of Waikato 113

6/12/2015 University of Waikato 114

6/12/2015 University of Waikato 115

6/12/2015 University of Waikato 116

Explorer: attribute selection

Panel that can be used to investigate which

(subsets of) attributes are the most predictive ones

Attribute selection methods contain two parts:

A search method: best-first, forward selection,

random, exhaustive, genetic algorithm, ranking

An evaluation method: correlation-based, wrapper,

information gain, chi-squared, …

Very flexible: WEKA allows (almost) arbitrary

combinations of these two

6/12/2015 University of Waikato 117

6/12/2015 University of Waikato 118

6/12/2015 University of Waikato 119

6/12/2015 University of Waikato 120

6/12/2015 University of Waikato 121

6/12/2015 University of Waikato 122

6/12/2015 University of Waikato 123

6/12/2015 University of Waikato 124

6/12/2015 University of Waikato 125

Explorer: data visualization

Visualization very useful in practice: e.g. helps to

determine difficulty of the learning problem

WEKA can visualize single attributes (1-d) and

pairs of attributes (2-d)

To do: rotating 3-d visualizations (Xgobi-style)

Color-coded class values

“Jitter” option to deal with nominal attributes (and

to detect “hidden” data points)

“Zoom-in” function

6/12/2015 University of Waikato 126

6/12/2015 University of Waikato 127

6/12/2015 University of Waikato 128

6/12/2015 University of Waikato 129

6/12/2015 University of Waikato 130

6/12/2015 University of Waikato 131

6/12/2015 University of Waikato 132

6/12/2015 University of Waikato 133

6/12/2015 University of Waikato 134

6/12/2015 University of Waikato 135

6/12/2015 University of Waikato 136

6/12/2015 University of Waikato 137

6/12/2015 University of Waikato 138

Performing experiments

Experimenter makes it easy to compare the

performance of different learning schemes

For classification and regression problems

Results can be written into file or database

Evaluation options: cross-validation, learning

curve, hold-out

Can also iterate over different parameter settings

Significance-testing built in!

6/12/2015 University of Waikato 139

6/12/2015 University of Waikato 140

6/12/2015 University of Waikato 141

6/12/2015 University of Waikato 142

6/12/2015 University of Waikato 143

6/12/2015 University of Waikato 144

6/12/2015 University of Waikato 145

6/12/2015 University of Waikato 146

6/12/2015 University of Waikato 147

6/12/2015 University of Waikato 148

6/12/2015 University of Waikato 149

6/12/2015 University of Waikato 150

6/12/2015 University of Waikato 151

6/12/2015 University of Waikato 152

The Knowledge Flow GUI

New graphical user interface for WEKA

Java-Beans-based interface for setting up and

running machine learning experiments

Data sources, classifiers, etc. are beans and can

be connected graphically

Data “flows” through components: e.g.,

“data source” -> “filter” -> “classifier” -> “evaluator”

Layouts can be saved and loaded again later

6/12/2015 University of Waikato 153

6/12/2015 University of Waikato 154

6/12/2015 University of Waikato 155

6/12/2015 University of Waikato 156

6/12/2015 University of Waikato 157

6/12/2015 University of Waikato 158

6/12/2015 University of Waikato 159

6/12/2015 University of Waikato 160

6/12/2015 University of Waikato 161

6/12/2015 University of Waikato 162

6/12/2015 University of Waikato 163

6/12/2015 University of Waikato 164

6/12/2015 University of Waikato 165

6/12/2015 University of Waikato 166

6/12/2015 University of Waikato 167

6/12/2015 University of Waikato 168

6/12/2015 University of Waikato 169

6/12/2015 University of Waikato 170

6/12/2015 University of Waikato 171

6/12/2015 University of Waikato 172

6/12/2015 University of Waikato 173

Conclusion: try it yourself!

WEKA is available at

http://www.cs.waikato.ac.nz/ml/weka

Also has a list of projects based on WEKA

WEKA contributors:

Abdelaziz Mahoui, Alexander K. Seewald, Ashraf M. Kibriya, Bernhard

Pfahringer , Brent Martin, Peter Flach, Eibe Frank ,Gabi Schmidberger

,Ian H. Witten , J. Lindgren, Janice Boughton, Jason Wells, Len Trigg,

Lucio de Souza Coelho, Malcolm Ware, Mark Hall ,Remco Bouckaert ,

Richard Kirkby, Shane Butler, Shane Legg, Stuart Inglis, Sylvain Roy,

Tony Voyle, Xin Xu, Yong Wang, Zhihai Wang