The Longest Common Substring
Problema.k.a Long Repeat
by Donnie Demuth
Sections
1. MapReduce and Hadoop2. Map and Reduce3. Mappers and Reducers4. Using Tools (Amazon)5. Conclusions
1. MapReduce and Hadoop
• What is it?• And how do I get it?
Google MapReduce
• Circa 2003• Based on Map and Reduce (go figure)– and Functional Programming!
• Proprietary
Apache Hadoop
• Circa 2006, released 2009• Named after an Elephant Toy• Seconds, maybe a minute, to install
Installing Hadoop on OSX
• Single Cluster setup is a piece of cake• Download the archive (tar.gz)• Modify conf/hadoop-env.sh:
– # export JAVA_HOME=/usr/lib/j2sdk1.6-sun – export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/
• Modify bin/hadoop:– JAVA=$JAVA_HOME/bin/java– JAVA=$JAVA_HOME/Commands/java
• Just run bin/hadoop with arguments
STOP!
• Actually, installing Hadoop wasn’t necessary• We can write parallel code without it
2. Map and Reduce
• What is it?– Quick Primer to Functional Programming• Higher-Order Functions• Alonzo Church (Lamba Calculus)
• Haskell Curry (Spicy Food)
• How do I use it?
(x (y x*x + y*y))(5)(2)↦ ↦
Code w/ Side-Effects
>>> thing = {'name':'Donald'}>>> def change_name(object): object['name'] = 'Donnie'... >>> change_name(thing)>>> thing{'name': 'Donnie'}
Pure Code, Side-effect Free
>>> thing = {'name':'Donald'}>>> def change_name(object): ... new_obj = {'name': 'Donnie'}... # copy any other values... return new_obj... >>> thing = change_name(thing)>>> thing{'name': 'Donnie'}
Benefits of Pure Code / FP
• easy to understand– Local vars = easy– Global vars + side-effects = hard
• it’s easy to parallelize– We only care about what we know RIGHT NOW
Map
1
2
3
1
4
6
f(x)
Map in Python
• Use the map(<function>, <list>) built-in
>>> map(lambda x: x*x, range(1,100))[1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961, 1024, 1089, 1156, 1225, 1296, 1369, 1444, 1521, 1600, 1681, 1764, 1849, 1936, 2025, 2116, 2209, 2304, 2401, 2500, 2601, 2704, 2809, 2916, 3025, 3136, 3249, 3364, 3481, 3600, 3721, 3844, 3969, 4096, 4225, 4356, 4489, 4624, 4761, 4900, 5041, 5184, 5329, 5476, 5625, 5776, 5929, 6084, 6241, 6400, 6561, 6724, 6889, 7056, 7225, 7396, 7569, 7744, 7921, 8100, 8281, 8464, 8649, 8836, 9025, 9216, 9409, 9604, 9801]
Reduce
1
2
3
0f(x, y)
f(x, y)
f(x, y) = 6
Reduce in Python
• Use the map(<function>, <list>, <unit>) built-in
>>> reduce(lambda x, y: x+y, [1,2,3], 0)6
>>> reduce(lambda x, y: x+y, (map(lambda x: x*x, range(1,100)), 0)
328350
3. Mappers and Reducers
• How do I write them?– Word Count (Hello World for Distrib. Comp.)– Longest Repeat
• Show me how to pipe them
Mappers
• Pseudo-Code– Take some input– Process it– And emit a Key – Value pair
Word Count Mapper
• For some input:– Donald Demuth Donald Draper
• The output should be:– Donald 1– Demuth 1– Donald 1– Draper 1
Word Count Mapper Code
• wordcount/mapper.py
#!/usr/bin/env pythonimport sys, re
word_re = re.compile('[a-zA-Z]+')for line in sys.stdin: line = line.strip().lower() for word in word_re.findall(line): print '%s\t%s' % (word, 1)
Reducers
• Dependant on the Mapper’s emissions• Pseudo-Code for word count– Read an emission from the mapper– Find the key and the value– Store the key in a dictionary with it’s value• But if the key already exists, add the value with the
pre-existing value!
– Emit the dictionary
Word Count Reducer Code
• wordcount/reducer.py#!/usr/bin/env pythonimport sys
counts = {}for line in sys.stdin: line = line.strip() word, count = line.split('\t', 1) count = int(count) counts[word] = counts.get(word, 0) + count
for word, count in counts.items(): print '%s\t%s'% (word, count)
Unix Pipes
• Does this really work??
$ cat books/*.txt | wordcount/mapper.py | wordcount/reducer.py | sort | heada 10526ab 3aback 1abaft 2abaht 1abandon 2abandoned 10abandonment 1abasement 1abash 1
Longest Repeat (LCS)
• Many problems can be solved with a series of Maps and Reduces
• However, Hadoop Streaming is a single Map and Reduce step
• After much trial and error my solution involves a pre-processing step
Pre-processing
• fasta_to_line.py
• gen_suffixes.py
ecoli.fasta.line
ecoli.fasta.line.0
ecoli.fasta.line.100000
ecoli.fasta.line.200000
4.6 megs
4.5 megs
4.4 megs
4.3 megs
ecoli.fasta ecoli.fasta.line
LCS Mapper
• Pseudo-code– Read a line from a suffix file– Determine the index (first chars)– Cycle through the first 100,000 positions• Cycle through possible lengths (10 3000)
– Emit the Length (Key) and the Position (Val)
• Emit (-1) and (-1) to STAY ALIVE
LCS Reducer
• Pseudo-Code– Simple– Find the largest KEY emitted by any mapper– Display it
LCS w/ Murmur.txt$ cat murmur.txt.line.0 | lcs/mapper.py | lcs/reducer.py length(63) pos(128)
$ python>>> text = open('murmur.txt.line').read()>>> text[128:128+63]'Dance the cha chaOr the can canShake your pom pomTo Duran Duran'
>>> seq = text[128:128+63]>>> text.index(seq)128>>> text[129:].index(seq) + 1291777>>> text[128:128+63] == text[1777:1777+63]True>>> text[1777:1777+63]'Dance the cha chaOr the can canShake your pom pomTo Duran Duran'
4. Using Tools, Amazon
• Harness the power of many machines at once– Easy to use 20
• Need to sign up for:– Amazon Elastic MapReduce Service (EMS)– Amazon Elastic Compute Cloud (EC2)– Amazon Simple Storage Service (S3)– Amazon SimpleDB
Deploying Data/Code
• First you’ll need to upload it to S3– Create a new bucket (or global folder) named ecoli-lcs
– Create a new path named input, ecoli-lcs/input– Upload all of the generated suffixes to the input
folder– Upload mapper.py and reducer.py to ecoli-lcs
Creating a Job (Flow)
Creating a Job Flow (…)
RESULTS!
• Need to download the output$ cd output$ cat * | sort (...)length(2815) pos(4166641)
$ python>>> text = open('ecoli.fasta.line').read()>>> seq = text[4166641:4166641+2815]>>> text.index(seq)4166641>>> text[4166642:].index(seq) + 41666424208043>>> text[4166641:4166641+2815] == text[4208043:4208043+2815]
5. Conclusions
• Costs– It’s about 3 cents an hour for a “medium” VM– One run took 840 instance hours (20+ actual)
• Approx. $25– Used about 2000 instance hours in total
• Hadoop Streaming is EASY– Though requires many (easy) tools– But costly if you have “bugs”
A Better Solution?
• Jeff Parker’s program used the following approach:– Cycle through the sequence and find all repeats of
a given size– Emit the location– Increase the size and use the previously known
locations to find larger matches
• Looks good for MapReduce (Core)
