Department of Computer Science,University of Waikato, New Zealand

Eibe Frank

� WEKA: A MachineLearning Toolkit

� The Explorer• Classification and

Regression• Clustering• Association Rules• Attribute Selection• Data Visualization

� The Experimenter� The Knowledge

Flow GUI� Conclusions

Machine Learning withWEKA

10/19/2007 University of Waikato 2

WEKA: the bird

Copyright: Martin Kramer (mkramer@wxs.nl)

10/19/2007 University of Waikato 3

WEKA: the software� Machine learning/data mining software written in

Java (distributed under the GNU Public License)� Used for research, education, and applications� Complements “Data Mining” by Witten & Frank� Main features:

� Comprehensive set of data pre-processing tools,learning algorithms and evaluation methods

� Graphical user interfaces (incl. data visualization)� Environment for comparing learning algorithms

10/19/2007 University of Waikato 4

WEKA: versions� There are several versions of WEKA:

� WEKA 3.0: “book version” compatible withdescription in data mining book

� WEKA 3.2: “GUI version” adds graphical userinterfaces (book version is command-line only)

� WEKA 3.3: “development version” with lots ofimprovements

� This talk is based on the latest snapshot of WEKA3.3 (soon to be WEKA 3.4)

10/19/2007 University of Waikato 5

@relation heart-disease-simplified

@attribute age numeric@attribute sex { female, male}@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}@attribute cholesterol numeric@attribute exercise_induced_angina { no, yes}@attribute class { present, not_present}

@data63,male,typ_angina,233,no,not_present67,male,asympt,286,yes,present67,male,asympt,229,yes,present38,female,non_anginal,?,no,not_present...

WEKA only deals with “flat” files

10/19/2007 University of Waikato 6

@relation heart-disease-simplified

@attribute age numeric@attribute sex { female, male}@attribute chest_pain_type { typ_angina, asympt, non_anginal, atyp_angina}@attribute cholesterol numeric@attribute exercise_induced_angina { no, yes}@attribute class { present, not_present}

@data63,male,typ_angina,233,no,not_present67,male,asympt,286,yes,present67,male,asympt,229,yes,present38,female,non_anginal,?,no,not_present...

WEKA only deals with “flat” files

10/19/2007 University of Waikato 7

10/19/2007 University of Waikato 8

10/19/2007 University of Waikato 9

10/19/2007 University of Waikato 10

Explorer: pre-processing the data� Data can be imported from a file in various

formats: ARFF, CSV, C4.5, binary� Data can also be read from a URL or from an SQL

database (using JDBC)� Pre-processing tools in WEKA are called “filters”� WEKA contains filters for:

� Discretization, normalization, resampling, attributeselection, transforming and combining attributes, …

10/19/2007 University of Waikato 11

10/19/2007 University of Waikato 12

10/19/2007 University of Waikato 13

10/19/2007 University of Waikato 14

10/19/2007 University of Waikato 15

10/19/2007 University of Waikato 16

10/19/2007 University of Waikato 17

10/19/2007 University of Waikato 18

10/19/2007 University of Waikato 19

10/19/2007 University of Waikato 20

10/19/2007 University of Waikato 21

10/19/2007 University of Waikato 22

10/19/2007 University of Waikato 23

10/19/2007 University of Waikato 24

10/19/2007 University of Waikato 25

10/19/2007 University of Waikato 26

10/19/2007 University of Waikato 27

10/19/2007 University of Waikato 28

10/19/2007 University of Waikato 29

10/19/2007 University of Waikato 30

10/19/2007 University of Waikato 31

10/19/2007 University of Waikato 32

Explorer: building “classifiers”� Classifiers in WEKA are models for predicting

nominal or numeric quantities� Implemented learning schemes include:

� Decision trees and lists, instance-based classifiers,support vector machines, multi-layer perceptrons,logistic regression, Bayes’ nets, …

� “Meta”-classifiers include:� Bagging, boosting, stacking, error-correcting output

codes, locally weighted learning, …

10/19/2007 University of Waikato 33

10/19/2007 University of Waikato 34

10/19/2007 University of Waikato 35

10/19/2007 University of Waikato 36

10/19/2007 University of Waikato 37

10/19/2007 University of Waikato 38

10/19/2007 University of Waikato 39

10/19/2007 University of Waikato 40

10/19/2007 University of Waikato 41

10/19/2007 University of Waikato 42

10/19/2007 University of Waikato 43

10/19/2007 University of Waikato 44

10/19/2007 University of Waikato 45

10/19/2007 University of Waikato 46

10/19/2007 University of Waikato 47

10/19/2007 University of Waikato 48

10/19/2007 University of Waikato 49

10/19/2007 University of Waikato 50

10/19/2007 University of Waikato 51

10/19/2007 University of Waikato 52

10/19/2007 University of Waikato 53

10/19/2007 University of Waikato 54

10/19/2007 University of Waikato 55

10/19/2007 University of Waikato 56

10/19/2007 University of Waikato 57

10/19/2007 University of Waikato 58

10/19/2007 University of Waikato 59

10/19/2007 University of Waikato 60

10/19/2007 University of Waikato 61

10/19/2007 University of Waikato 62

10/19/2007 University of Waikato 63

10/19/2007 University of Waikato 64

10/19/2007 University of Waikato 65QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

10/19/2007 University of Waikato 66QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

10/19/2007 University of Waikato 67QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

10/19/2007 University of Waikato 68

10/19/2007 University of Waikato 69

10/19/2007 University of Waikato 70

10/19/2007 University of Waikato 71

10/19/2007 University of Waikato 72

10/19/2007 University of Waikato 73

10/19/2007 University of Waikato 74

10/19/2007 University of Waikato 75

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

10/19/2007 University of Waikato 76

10/19/2007 University of Waikato 77

10/19/2007 University of Waikato 78

10/19/2007 University of Waikato 79

10/19/2007 University of Waikato 80

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

10/19/2007 University of Waikato 81

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

10/19/2007 University of Waikato 82

10/19/2007 University of Waikato 83

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

10/19/2007 University of Waikato 84

10/19/2007 University of Waikato 85

10/19/2007 University of Waikato 86

10/19/2007 University of Waikato 87

10/19/2007 University of Waikato 88

10/19/2007 University of Waikato 89

10/19/2007 University of Waikato 90

10/19/2007 University of Waikato 91

10/19/2007 University of Waikato 92

Explorer: clustering data� WEKA contains “clusterers” for finding groups of

similar instances in a dataset� Implemented schemes are:

� k-Means, EM, Cobweb, X-means, FarthestFirst� Clusters can be visualized and compared to “true”

clusters (if given)� Evaluation based on loglikelihood if clustering

scheme produces a probability distribution

10/19/2007 University of Waikato 93

10/19/2007 University of Waikato 94

10/19/2007 University of Waikato 95

10/19/2007 University of Waikato 96

10/19/2007 University of Waikato 97

10/19/2007 University of Waikato 98

10/19/2007 University of Waikato 99

10/19/2007 University of Waikato 100

10/19/2007 University of Waikato 101

10/19/2007 University of Waikato 102

10/19/2007 University of Waikato 103

10/19/2007 University of Waikato 104

10/19/2007 University of Waikato 105

10/19/2007 University of Waikato 106

10/19/2007 University of Waikato 107

10/19/2007 University of Waikato 108

Explorer: finding associations� WEKA contains an implementation of the Apriori

algorithm for learning association rules� Works only with discrete data

� Can identify statistical dependencies betweengroups of attributes:� milk, butter ⇒ bread, eggs (with confidence 0.9 and

support 2000)� Apriori can compute all rules that have a given

minimum support and exceed a given confidence

10/19/2007 University of Waikato 109

10/19/2007 University of Waikato 110

10/19/2007 University of Waikato 111

10/19/2007 University of Waikato 112

10/19/2007 University of Waikato 113

10/19/2007 University of Waikato 114

10/19/2007 University of Waikato 115

10/19/2007 University of Waikato 116

Explorer: attribute selection� Panel that can be used to investigate which

(subsets of) attributes are the most predictive ones� Attribute selection methods contain two parts:

� A search method: best-first, forward selection,random, exhaustive, genetic algorithm, ranking

� An evaluation method: correlation-based, wrapper,information gain, chi-squared, …

� Very flexible: WEKA allows (almost) arbitrarycombinations of these two

10/19/2007 University of Waikato 117

10/19/2007 University of Waikato 118

10/19/2007 University of Waikato 119

10/19/2007 University of Waikato 120

10/19/2007 University of Waikato 121

10/19/2007 University of Waikato 122

10/19/2007 University of Waikato 123

10/19/2007 University of Waikato 124

10/19/2007 University of Waikato 125

Explorer: data visualization� Visualization very useful in practice: e.g. helps to

determine difficulty of the learning problem� WEKA can visualize single attributes (1-d) and

pairs of attributes (2-d)� To do: rotating 3-d visualizations (Xgobi-style)

� Color-coded class values� “Jitter” option to deal with nominal attributes (and

to detect “hidden” data points)� “Zoom-in” function

10/19/2007 University of Waikato 126

10/19/2007 University of Waikato 127

10/19/2007 University of Waikato 128

10/19/2007 University of Waikato 129

10/19/2007 University of Waikato 130

10/19/2007 University of Waikato 131

10/19/2007 University of Waikato 132

10/19/2007 University of Waikato 133

10/19/2007 University of Waikato 134

10/19/2007 University of Waikato 135

10/19/2007 University of Waikato 136

10/19/2007 University of Waikato 137

10/19/2007 University of Waikato 138

Performing experiments� Experimenter makes it easy to compare the

performance of different learning schemes� For classification and regression problems� Results can be written into file or database� Evaluation options: cross-validation, learning

curve, hold-out� Can also iterate over different parameter settings� Significance-testing built in!

10/19/2007 University of Waikato 139

10/19/2007 University of Waikato 140

10/19/2007 University of Waikato 141

10/19/2007 University of Waikato 142

10/19/2007 University of Waikato 143

10/19/2007 University of Waikato 144

10/19/2007 University of Waikato 145

10/19/2007 University of Waikato 146

10/19/2007 University of Waikato 147

10/19/2007 University of Waikato 148

10/19/2007 University of Waikato 149

10/19/2007 University of Waikato 150

10/19/2007 University of Waikato 151

10/19/2007 University of Waikato 152

The Knowledge Flow GUI� New graphical user interface for WEKA� Java-Beans-based interface for setting up and

running machine learning experiments� Data sources, classifiers, etc. are beans and can

be connected graphically� Data “flows” through components: e.g.,

“data source” -> “filter” -> “classifier” -> “evaluator”� Layouts can be saved and loaded again later

10/19/2007 University of Waikato 153

10/19/2007 University of Waikato 154

10/19/2007 University of Waikato 155

10/19/2007 University of Waikato 156

10/19/2007 University of Waikato 157

10/19/2007 University of Waikato 158

10/19/2007 University of Waikato 159

10/19/2007 University of Waikato 160

10/19/2007 University of Waikato 161

10/19/2007 University of Waikato 162

10/19/2007 University of Waikato 163

10/19/2007 University of Waikato 164

10/19/2007 University of Waikato 165

10/19/2007 University of Waikato 166

10/19/2007 University of Waikato 167

10/19/2007 University of Waikato 168

10/19/2007 University of Waikato 169

10/19/2007 University of Waikato 170

10/19/2007 University of Waikato 171

10/19/2007 University of Waikato 172

10/19/2007 University of Waikato 173

Conclusion: try it yourself!� WEKA is available at

http://www.cs.waikato.ac.nz/ml/weka� Also has a list of projects based on WEKA� WEKA contributors:

Abdelaziz Mahoui, Alexander K. Seewald, Ashraf M. Kibriya, BernhardPfahringer , Brent Martin, Peter Flach, Eibe Frank ,Gabi Schmidberger ,Ian H.Witten , J. Lindgren, Janice Boughton, Jason Wells, Len Trigg, Lucio deSouza Coelho, Malcolm Ware, Mark Hall ,Remco Bouckaert , RichardKirkby, Shane Butler, Shane Legg, Stuart Inglis, Sylvain Roy, Tony Voyle,Xin Xu, Yong Wang, Zhihai Wang