scalable machine learning cmsc 491 hadoop-based distributed computing spring 2015 adam shook
TRANSCRIPT
Scalable Machine Learning
CMSC 491Hadoop-Based Distributed Computing
Spring 2015Adam Shook
But What is Machine Learning
• “Machine Learning is programming computers to optimize a performance criterion using example data or past experience”
• Given a data set X, can we effectively predict Y by optimizing Z?
Intro. to Machine Learning by E. Alpaydin
Supervised vs. Unsupervised
• Algorithms trained on labeled examples– I know these images are of cats and these are of
dogs, tell me if this image is a cat or a dog• Algorithms trained on unlabeled examples– Group these images together by similarity, i.e.
some kind of distance function
Use Cases
• Collaborative Filtering– Takes users' behavior, and from that try to find items users might
like• Clustering
– Take things and put them into groups of related things• Classification
– Learn from existing categories to determine what things in a category look like, and assign unlabeled things the (hopefully) correct category
• Frequent Itemset Mining– Analyzes items in a groups and identifies which items frequently
appear together
Clustering
• Dirichlet Processing Clustering– Bayesian mixture modeling
• K-Means Clustering– Partition n observations into k clusters
• Fuzzy K-Means– Soft clusters where a point can be in more than one
• Hierarchical Clustering• Hierarchy of clusters from bottom-up or top-down
• Canopy Clustering– Preprocess data before K-Means or Hierarchical
More Clustering• Latent Dirichlet Allocation
– Cluster words into topics and documents into mixtures of topics
• Mean Shift Clustering– Finding modes or clusters in 2-
dimensional space, where number of clusters is unknown
• Minhash Clustering– Quickly estimate similarity
between two data sets
• Spectral Clustering– Cluster points using eigenvectors
of matrices derived from data
Collaborative Filtering
• Distributed Item-based Collaborate Filtering– Estimates a user’s preference for one item by looking at
preference for similar items
• Collaborate Filtering using a Parallel Matrix Factorization– Among a matrix of items that a user has not yet seen, predict
which items the user might prefer
Classification
• Bayesian– Classify objects into
binary categories
• Random Forests– Method for classification
and regression by constructing a multitude of decision trees
Dog
Cat
Frequent Itemset Mining
• Parallel FP Growth Algorithm– Analyzes items in a group and then identifies
which items appear together
Algorithm Examples
• K-Means Clustering– Using Mahout
• Alternating Least Squares (Recommender)– Using Spark Mllib
APACHE MAHOUT
ma·hout -\mə-ˈha t\ - noun - A keeper and driver of an elephantu̇�
Overview• Build a scalable machine learning library, in both data volume and
processing• Began in 2008 as a subproject of Apache Lucene, then became a top-
level Apache project in 2010• No longer accepting Java MapReduce implementations in favor of
Spark MLlib
• Address issues commonly found in ML libraries:– Lack community, scalability, documentation/examples, Apache licensing– Not well-tested– Not research oriented– Not built on existing production-quality projects– Active Community
Technical Requirements
• Linux• Java 1.6 or greater• Maven• Hadoop– Although, not all algorithms are implemented to
work on Hadoop clusters
Building Mahout for Hadoop 2
• Check out Mahout trunk with gitgit clone https://github.com/apache/mahout.git
• Build with Maven, giving it the proper Hadoop and HBase versions
cd gitmvn install -DskipTests \
-Dhadoop2 -Dhadoop2.version=2.6.0 \-Dhbase.version=1.0.0
cd ../mv mahout /usr/share/491s15# Edit .bashrc/.bash_profile to add a $MAHOUT_HOME variable, # $MAHOUT_HOME/bin to the path, and # export HADOOP_CONF_DIR=/usr/share/491s15/hadoop/etc/hadoop
c1
c2
c3
K-Means Clustering
c1
c2
c3
K-Means Clustering
c1
c2
c3
K-Means Clustering
c1
c2
c3
c1
c2
c3
K-Means Clustering
c1
c2
c3
K-Means Clustering
K-Means ClusteringExample
• Let’s cluster the Reuter’s data set together– A bunch (21,578 to be exact) of hand-classified
news articles from the greatest year created, 1987
• Steps!1. Generate Sequence Files from data2. Generate Vectors from Sequence Files3. Run k-means
K-Means ClusteringConvert dataset into a Sequence File
• Download and extract the SGML files$ wget http://www.daviddlewis.com/resources/testcollections/reuters21578/reuters21578.tar.gz$ mkdir reuters-sgm$ tar -xf reuters21578.tar.gz -C reuters-sgm/
• Extract content from SGML to text file$ mahout org.apache.lucene.benchmark.utils.ExtractReuters \
reuters-sgm/ reuters-out/$ hdfs dfs -put reuters-out . # Takes a while...
• Use seqdirectory tool to convert text file into a Hadoop Sequence File
$ mahout seqdirectory -i reuters-out \-o reuters-out-seqdir -c UTF-8 -chunk 5
Tangent: Writing to Sequence Files// Say you have some documents array
Configuration conf = new Configuration();FileSystem fs = FileSystem.get(conf); Path path = new Path("testdata/part-00000"); SequenceFile.Writer writer = new SequenceFile.Writer(fs,
conf, path, Text.class, Text.class); for (int i = 0; i < MAX_DOCS; ++i) {
writer.append(new Text(documents[i].getId()),new Text(documents[i].getContent()));
}writer.close();
Original File
$ cat reut2-000.sgm-30.txt26-FEB-1987 15:43:14.36
U.S. TAX WRITERS SEEK ESTATE TAX CURBS, RAISING 6.7 BILLION DLRS THRU 1991
Now, in Sequence File
/reut2-000.sgm-30.txt 26-FEB-1987 15:43:14.36
U.S. TAX WRITERS SEEK ESTATE TAX CURBS, RAISING 6.7 BILLION DLRS THRU 1991
Key Value*
* Contains new line characters
• Steps1. Compute Dictionary2. Assign integers for words3. Compute feature weights4. Create vector for each document using word-integer
mapping and feature-weight
• Or simply run $ mahout seq2sparse
$ mahout seq2sparse \ -i reuters-out-seqdir/ \ -o reuters-out-seqdir-sparse-kmeans
K-Means ClusteringGenerate Vectors from Sequence Files
Document to Integers to Vector
26-FEB-1987 15:43:14.36
U.S. TAX WRITERS SEEK ESTATE TAX CURBS, RAISING 6.7 BILLION DLRS THRU 1991
14.36 273715
29621991 396026
540543
83616.7
10882billion 15528curbs 19078dlrs 20362estate 21578feb
22224raising
33629seek 35909tax
38507u.s
39687writers 41511
{3960:1.0,21578:1.0,33629:1.0,41511:1.0,8361:1.0,10882:1.0,5405:1.0,22224:1.0,15528:1.0,38507:2.0,39687:1.0,2737:1.0,35909:1.0,2962:1.0,19078:1.0,20362:1.0}
One document of many!
After seq2sparse
/reut2-000.sgm-30.txt {3960:1.0,21578:1.0, 33629:1.0,41511:1.0,8361:1.0,10882:1.0,5405:1.0,22224:1.0,15528:1.0,38507:2.0,39687:1.0,2737:1.0 ,35909:1.0,2962:1.0,19078:1.0,20362:1.0}
Key Value
K-Means ClusteringRun the kmeans program
$ mahout kmeans \-i reuters-out-seqdir-sparse-kmeans/tfidf-vectors/ \-c reuters-kmeans-clusters \-o reuters-kmeans \-dm
org.apache.mahout.common.distance.CosineDistanceMeasure \-cd 0.1 -x 10 -k 20
• Key Parameters– dm: Distance measure– cd: Convergence delta– x: Number of iterations– k: Creating assignments
Inspect clusters$ bin/mahout clusterdump \ -i reuters-kmeans/clusters-*-final \ -d reuters-out-seqdir-sparse-kmeans/dictionary.file-0 \ -dt sequencefile -b 100 -n 10
:{"identifier":"VL-316","r":[{"00":0.497},{"00.14":0.408},{"00.18":0.408},{"00.56Top Terms:
president => 3.4944214993103375chief => 3.3234287659025012executive => 3.16472187060367officer => 3.143776322498974chairman => 2.5400053276308587vice => 1.9913627557428164named => 1.9851312619198411said => 1.9030630459350324company => 1.782354193948521names => 1.4052995438811444
FAQs• How to get rid of useless words?
– Increase minSupport and or decrease dfPercent– Use StopwordsAnalyzer
• How to see documents to cluster assignments? – Run clustering process at the end of centroid generation using –cl
• How to choose appropriate weighting?– If its long text, go with tf-idf. Use normalization if documents different in
length• How to run this on a cluster?
– Set HADOOP_CONF directory to point to your hadoop cluster conf directory• How to scale?
– Use small value of k to partially cluster data and then do full clustering on each cluster.
FAQs
• How to choose k?– Figure out based on the data you have. Trial and error– Or use Canopy Clustering and distance threshold to figure it
out– Or use Spectral clustering
• How to improve Similarity Measurement?– Not all features are equal– Small weight difference for certain types creates a large
semantic difference– Use WeightedDistanceMeasure– Or write a custom DistanceMeasure
Recommendations
• Help users find items they might like based on historical preferences
Based on example by Sebastian Schelter in “Distributed Itembased Collaborative Filtering with Apache Mahout”
Recommendations
Alice
Bob
Peter
5 1 4
2 5
4 3 2
?
Recommendations
• Algorithm– Neighborhood-based approach– Works by finding similarly rated items in the user-
item-matrix (e.g. cosine, Pearson-Correlation, Tanimoto Coefficient)
– Estimates a user's preference towards an item by looking at his/her preferences towards similar items
Recommendations
• Prediction: Estimate Bob's preference towards “The Matrix”1. Look at all items that
– a) are similar to “The Matrix“ – b) have been rated by Bob
=> “Alien“, “Inception“
2. Estimate the unknown preference with a weighted sum
Recommendations
• MapReduce phase 1– Map – Make user the key
(Alice, Matrix, 5)
(Alice, Alien, 1)
(Alice, Inception, 4)
(Bob, Alien, 2)
(Bob, Inception, 5)(Peter, Matrix, 4)
(Peter, Alien, 3)
(Peter, Inception, 2)
Alice (Matrix, 5)
Alice (Alien, 1)
Alice (Inception, 4)
Bob (Alien, 2)
Bob (Inception, 5)Peter (Matrix, 4)
Peter (Alien, 3)
Peter (Inception, 2)
Recommendations
• MapReduce phase 1– Reduce – Create inverted index
Alice (Matrix, 5)
Alice (Alien, 1)
Alice (Inception, 4)
Bob (Alien, 2)
Bob (Inception, 5)Peter (Matrix, 4)
Peter (Alien, 3)
Peter (Inception, 2)
Alice (Matrix, 5) (Alien, 1) (Inception, 4)
Bob (Alien, 2) (Inception, 5)
Peter (Matrix, 4) (Alien, 3) (Inception, 2)
Recommendations
• MapReduce phase 2– Map – Isolate all co-occurred ratings (all cases
where a user rated both items)
Matrix, Alien (5,1)
Matrix, Alien (4,3)
Alien, Inception (1,4)
Alien, Inception (2,5)
Alien, Inception (3,2)Matrix, Inception (4,2)
Matrix, Inception (5,4)
Alice (Matrix, 5) (Alien, 1) (Inception, 4)
Bob (Alien, 2) (Inception, 5)
Peter(Matrix, 4) (Alien, 3) (Inception, 2)
Recommendations
• MapReduce phase 2– Reduce – Compute similarities
Matrix, Alien (5,1)
Matrix, Alien (4,3)
Alien, Inception (1,4)
Alien, Inception (2,5)
Alien, Inception (3,2)Matrix, Inception (4,2)
Matrix, Inception (5,4)
Matrix, Alien (-0.47)
Matrix, Inception (0.47)
Alien, Inception(-0.63)
Recommendations
• Calculate Weighted sum
(-.47*2 + .47*5) / (.47+.47) = 1.5
Recommendations
Alice
Bob
Peter
5 1 4
2 5
4 3 2
1.5
Implementation in Spark
• Alternating Least Squares (ALS)• Accepts a tuple of (user, product, rating) to
train data• Accepts a tuple of (user, product) to predict
their rating• Example:
https://spark.apache.org/docs/latest/mllib-collaborative-filtering.html
Implementations in Mahout
• ItemSimilarityJob– Computes all item similarities– Various configuration options:• Similarity measure to use (cosine, Pearson-Correlation,
etc.)• Maximum number of similar items per item• Maximum number of co-occurences to consider
– Input: CSV file (userId, itemID, value)– Output: Pairs of itemIDs with associated similarity
Implementations in Mahout
• RecommenderJob– Distributed Itembased Recommender– Various configuration options:• Similarity measure to use• Number of recommendations per user• Filter out some users or items
– Input: CSV file (userId, itemID, value)– Output: UserIds with recommended itemIDs and
their scores
References
• http://mahout.apache.org• http://spark.apache.org• http://
isabel-drost.de/hadoop/slides/collabMahout.pdf
• http://www.slideshare.net/OReillyOSCON/hands-on-mahout#
• http://www.slideshare.net/urilavi/intro-to-mahout