statistical learning and data mining...

Statistical Learning and Data Mining Stat557

Statistical Learning and Data MiningStat557

Jia Li

Department of StatisticsThe Pennsylvania State University

Email: [email protected]://www.stat.psu.edu/∼jiali

Jia Li http://www.stat.psu.edu/∼jiali


General Information

I Course homepage:http://www.stat.psu.edu/˜jiali/stat557

I Prerequisite:I Elementary probability theoryI Conditional distribution, expectationI C, Matlab, or S-plus programming



I Text books:I Required: The Elements of Statistical Learning, by T. Hastie,

R. Tibshirani, and J. Friedman(ElemStatLearn).

I Optional:

1. Classification and Regression Trees by L. Breiman, J. H.Friedman, R. A. Olshen, and C. J. Stone

2. Pattern Recognition and Neural Networks by B. Ripley3. Principles of Data Mining by H. Mannila, P. Smyth and D. J.

Hand4. Data Mining: Concepts and Techniques by J. Han and M.

Kamber



What Is Data Mining?

Data mining: tools, methodologies, and theories for revealingpatterns in data—a critical step in knowledge discovery.Driving forces:

I Big data:I Enormous volumeI High complexity: dimension, structureI Dynamic

I Explosive growth of data in a great variety of fieldsI Cheaper storage devices with higher capacityI Faster communicationI Better database manage systems

I Rapidly increasing computing power: distributed and parallelplatforms

I Make data to work for us



Research fields

I Statistics

I Machine learning

I Pattern recognition

I Signal processing

I Database



Applications

I BusinessI Wal-Mart data warehouseI Credit card companies

I GenomicsI Human genome project: DNA sequencesI Microarray data

I Information retrievalI Terrabytes of data on the internetI Multimedia information

I Communication systemsI Speech recognitionI Image analysis

I Many other scientific fields



Problems Focused: Prediction



Terminology

Notation

I Input X : X is often multidimensional. Each dimension of X isdenoted by Xj and is referred to as a feature, predictor, orindependent variable/variable.

I Output Y : response, dependent variable.

CategorizationI Supervised learning vs. unsupervised learning

I Is Y available in the training data?

I Regression vs. ClassificationI Is Y quantitative or qualitative?I For qualitative Y , it is also denoted by

G ∈ G = {1, 2, ...,K}.



Examples

Email spam: (ElemStatLearn)

I Goal: predict whether an email is a junk email, i.e., “spam”.

I Raw data: text email messages.

I Input X : relative frequencies of 57 of the most commonlyoccurring words and punctuation marks in the email message.

I Training data set: 4601 email messages with email typeknown (supervised learning).



Examples

Handwritten digit recognition:(ElemStatLearn)

I Goal: identify single digits 0 ∼ 9 based on images.I Raw data: images that are scaled segments from five digit

ZIP codes.I 16× 16 eight-bit grayscale mapsI Pixel intensities range from 0 (black) to 255 (white).

I Input data: a 256 dimension vector, or feature vectors withlower dimensions.



Examples

Image segmentation:

I Goal: segment images into regions of different types, e.g.,man-made vs. natural in aerial images, graph and picture vs.text in document images.

I Raw data: grayscale images represented by matrices of sizem × n, or color images represented by 3 such matrices.



Aerial images. Left: Original image of size 512× 512 with pixel intensity

ranging from 0 to 255, Right: Hand-labeled classified images. White:

man-made, Gray: natural.



I Input data:I Divide images into blocks of pixels or form a neighborhood

around each pixel.I Compute statistics using pixel intensities in each block.I An image is converted to an array of input vectors.

I Methodologies:I Assume the feature vectors are independent.I Employ spatial models to capture dependence among the

vectors.



Examples

Speech recognition:

I Goal: identify words spoken according to speech signalsI Automatic voice recognition systems used by airline companiesI Automatic stock price reporting

I Raw data: voice amplitude sampled at discrete time spots (atime sequence).



I Input data: speech feature vectors computed at the samplingtime.

I Methodology:I Estimate an Hidden Markov Model (HMM) for each word,

e.g., State College, San Francisco,Pittsburgh.

I For a new word, find the HMM that yields the maximumlikelihood.

I Identify the word as the one associated with the HMM.



ExamplesDNA Expression Microarray:

I Goal: identify disease or tissue types

I Raw data: for each sample taken from a tissue of a particulardisease type, the expression levels of a large collection ofgenes are measured.

I Input data: cleaned-up gene expression dataI NormalizationI Denoising.I Ample literature on the topic of cleaning microarray data

I Example data set: 4026 genes, 96 samples taken from 9classes of tissues.

I Challenges:I very high dimensional dataI very limited number of samples



Examples

DNA sequence classification:

I Goal: distinguish “junk” segments from coding segments.

I Raw data: sequences of letters, e.g., A,C,G,T for DNAsequences.

I Input data: likelihood ratio statistics computed fromstochastic models.

I Supervised learning: estimate stochastic models, selectmodels.



Supervised Learning

Two types of learning:

I Regression: the response Y is quantitative.

I Classification: the response Y is qualitative, or categorical.

Two aspects in learning:

I Fit the data well.

I Robust

Equivalent concepts:

I Training error vs. testing error

I Bias vs. variance

I Fitting vs. overfitting

I Empirical risk vs. model complexity (capacity)



Learning Spectrum



Regression

Overview:I Linear models:

I The mean response is a linear function of the independentvariables.

I Generalized linear modelsI Expand basis:

I Splines (polynomials)I Reproducing Kernel Hilbert SpacesI Wavelet smoothing

I Kernel methods



Classification: A graphic View



Outlines

I Linear regression

I Linear methods for classification

I Prototype methods

I Classification and regression tree (CART)

I Mixture discriminant analysis

I Hidden Markov models and its applications


statistical learning and data mining...

Documents