how to parallelize an algorithm
DESCRIPTION
How to Parallelize an Algorithm. Lecture 2. Today’s Outline. Quiz Functional Programming Review Algorithm Parallelization Announcements Projects Readings. Quiz. This one is graded (unlike the last one). Fold & Map review. Fold: foldl f z [] = z foldl f z (x:xs) = foldl f (f z x) xs - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/1.jpg)
How to Parallelize an Algorithm
Lecture 2
![Page 2: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/2.jpg)
Today’s Outline• Quiz• Functional Programming Review• Algorithm Parallelization• Announcements
– Projects– Readings
![Page 3: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/3.jpg)
Quiz• This one is graded (unlike the last one)
![Page 4: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/4.jpg)
Fold & Map reviewFold:
foldl f z [] = zfoldl f z (x:xs) = foldl f (f z x) xs[foldr f z (x:xs) = f x (foldr f z xs)]– Applies a function to every element in the list– Each iteration can access the result of the previous
How would you parallelize Map? Fold?
Map:map f [] = []map f (x:xs) = f x : map xs– Applies a function to every element in the list– Each iteration is independent of all others
![Page 5: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/5.jpg)
Group Exercises• See handout
![Page 6: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/6.jpg)
Answers to Group ExercisesConcat:
concat xss = foldr (++) [] xss– Given a list of lists, concats all sublists
Group:group xss = foldl group_func [] xssgroup_func (result) (k,v) = if (has (k,v) result) then(map (update (k,v)) result)else (k,v) :: resultupdate (k1,v1) (k2,v_list) = if (EQ k1 k2) then (k1, v1::vlist) else (k2,v_list)
![Page 7: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/7.jpg)
Why Parallelize?
Reasons to parallelize:– Your reasons here
What issues are there in parallelization?
Reasons to not parallelize:– Your reasons here
![Page 8: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/8.jpg)
Implicit Serialization Example
DocInfo:f = read_file("file.txt”)
words = count_uniq_words(f)spaces = count_spaces(f)
f = capitalize(f)f = remove_punct(f)
words2 = count_uniq_words(f)
puts(“unique words: ” + words)puts(“num spaces: ” + spaces)puts(“unique scrubbed words: ” + words2)
Which statements can be reordered?
![Page 9: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/9.jpg)
Data Dependency Graph
f = read_file("file.txt”)
words = count_uniq_words(f)spaces = count_spaces(f)
f = capitalize(f)f = remove_punct(f)
words2 = count_uniq_words(f)
puts(“unique words” + words)puts(“num spaces” + spaces)puts(“unique scrubbed words” + words2)
words2
f0
spaces
f1
f2words
read_file
count_spaces
coun
t_un
iq_w
ords
capitalize
rem
ove_
punc
t
coun
t_un
iq_w
ords
console0
console1
console2
puts
puts
puts Which operations can be done in parallel?
![Page 10: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/10.jpg)
Distributing Computation
words2
f0
spaces
f1
f2words
read_file
count_spaces
coun
t_un
iq_w
ords
capitalize
rem
ove_
punc
t
coun
t_un
iq_w
ords
console0
console1
console2
puts
puts
puts
Cpu1
Cpu2
Cpu3
Cpu4
Storage
Ram
![Page 11: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/11.jpg)
Eliminating Dependencies 1 of 3
Synchronization points?– f[0,1,2]
– console[0,1,2]
words2
f0
spaces
f1
f2words
read_file
count_spaces
coun
t_un
iq_w
ords
capitalize
rem
ove_
punc
t
coun
t_un
iq_w
ords
console0
console1
console2
puts
puts
puts
Ideas for removing them:– Your ideas here
![Page 12: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/12.jpg)
Eliminating Dependencies 2 of 3
f = read_file("file.txt”)scrubbed_f = scrub_words(f)
words = count_uniq_words(f)spaces = count_spaces(f)
words2 = count_uniq_words(scrubbed_f)
puts(“unique words” + words)puts(“num spaces” + spaces)puts(“unique scrubbed words” + words2)
captialize, remove_punct can be combined and run first to create a copy of the data before “counting”.
DocInfo 2.0:
![Page 13: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/13.jpg)
Dependency Graph 2.0
f = read_file("file.txt”)scrubbed_f = scrub_words(f)
words = count_uniq_words(f)spaces = count_spaces(f)
words2 = count_uniq_words(scrubbed_f)
puts(“unique words” + words)puts(“num spaces” + spaces)puts(“unique scrubbed words” + words2)
words2
f
spaceswords
read_file
count_spaces
count_uniq_words
coun
t_un
iq_w
ords
console0
console1
console2
puts
puts
puts
scrubbed_f
scrub_words
![Page 14: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/14.jpg)
Distributing Computation 2.0
Cpu1
Cpu2
Cpu3
Cpu4
Storage
Ram
words2
f
spaceswords
read_file
count_spaces
count_uniq_words
coun
t_un
iq_w
ords
console0
console1
console2
puts
puts
puts
scrubbed_f
scrub_words
![Page 15: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/15.jpg)
Eliminating Dependencies 3 of 3
f = read_file("file.txt”)
words = count_uniq_words(f)spaces = count_spaces(f)
words2 = count_uniq_scrubbed_words(f)
puts(“unique words” + words)puts(“num spaces” + spaces)puts(“unique scrubbed words” + words2)
captialize, remove_punct only needs to be applied to each word (not the whole file) before “counting”.
DocInfo 3.0:
![Page 16: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/16.jpg)
Dependency Graph 3.0
f = read_file("file.txt”)
words = count_uniq_words(f)spaces = count_spaces(f)
words2 = count_uniq_scrubbed_words(f)
puts(“unique words” + words)puts(“num spaces” + spaces)puts(“unique scrubbed words” + words2)words2
f0
spaceswords
read_file
count_spacescount_uniq_scrubbed_words
coun
t_un
iq_w
ords
console0
console1
console2
puts
puts
puts
![Page 17: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/17.jpg)
Distributing Computation 3.0
words2
f0
spaceswords
read_file
count_spacescount_uniq_scrubbed_words
coun
t_un
iq_w
ords
console0
console1
console2
puts
puts
puts
Cpu1
Cpu2
Cpu3
Cpu4
Storage
Ram
![Page 18: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/18.jpg)
Parallelization Summary
Why avoid data dependencies?– lowers complexity– makes parallelization possible
How do you avoid data dependencies?– avoid stateful algorithms– avoid side effects (clone state instead of modifying)– avoid global variables and member variables
Parallelization tradeoff:Good:better scalability
Bad:less algorithmic flexibility, higher complexityNeutral:optimizes for large input over small input
![Page 19: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/19.jpg)
Parallelizing Map
What’s required to parallelize map?– function needs to be stateless– function available to each computation unit– input accessible by all computation units– output ordering isn’t important
Definition of map:map f [] = []map f (x:xs) = f x : map xs
![Page 20: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/20.jpg)
Parallelizing Fold
What’s required to parallelize fold?You can’t.
Why can’t you parallelize fold?Each step depends on the result of the previous.
Definition of fold:foldl f z [] = zfoldl f z (x:xs) = foldl f (f z x) xs
How is fold useful in parallel computing then?
![Page 21: How to Parallelize an Algorithm](https://reader035.vdocument.in/reader035/viewer/2022062217/56815a8d550346895dc80275/html5/thumbnails/21.jpg)
MapReduceMapReduce:
mapreduce fm fr l =map (reducePerKey fr) (group (map fm l))
reducePerKey fr (k,v_list) = (k, (foldl (fr k) [] v_list))
– Assume map here is actually concatMap.– Argument l is a list documents– The result of first map is a list of key-value pairs– The function fr takes 3 arguments key, context, current.
With currying, this allows for locking the value of “key” for each list during the fold.
MapReduce maps a fold over the sorted result of a map!