umass lowell computer science 91.503 analysis of algorithms prof. giampiero pecelli fall, 2010...
TRANSCRIPT
UMass Lowell Computer Science 91.503
Analysis of Algorithms Prof. Giampiero Pecelli
Fall, 2010
UMass Lowell Computer Science 91.503
Analysis of Algorithms Prof. Giampiero Pecelli
Fall, 2010
Paradigms for Optimization ProblemsParadigms for Optimization ProblemsDynamic Programming Dynamic Programming
& & Greedy AlgorithmsGreedy Algorithms
Optimization Optimization
This, generally, refers to classes of problems that possess This, generally, refers to classes of problems that possess multiple solutions at one level, and where we have a real-multiple solutions at one level, and where we have a real-valued function defined on the solutions.valued function defined on the solutions.
ProblemProblem: find a solution that minimizes or maximizes the : find a solution that minimizes or maximizes the value of this function.value of this function.
NoteNote: there is no guarantee that such a solution will be : there is no guarantee that such a solution will be unique and, moreover, there is no guarantee that you will unique and, moreover, there is no guarantee that you will find it (local maxima, anyone?) unless the search is over a find it (local maxima, anyone?) unless the search is over a small enough search space or the function is restricted small enough search space or the function is restricted enough.enough.
Optimization Optimization
QuestionQuestion: are there classes of problems for which you can : are there classes of problems for which you can guarantee an optimizing solution can be found?guarantee an optimizing solution can be found?
AnswerAnswer: yes. BUT you also need to find such a solution in a : yes. BUT you also need to find such a solution in a "reasonable" amount of time."reasonable" amount of time.
We are going to look at two classes of problems, and the We are going to look at two classes of problems, and the techniques that will succeed in constructing their solutions in techniques that will succeed in constructing their solutions in a "reasonable" (i.e., low degree polynomial in the size of the a "reasonable" (i.e., low degree polynomial in the size of the initial data) amount of time.initial data) amount of time.
Optimization Optimization
We begin with a rough comparison that contrasts a method We begin with a rough comparison that contrasts a method you are familiar with (divide and conquer) and the method you are familiar with (divide and conquer) and the method (still unspecified) of Dynamic Programming (developed by (still unspecified) of Dynamic Programming (developed by Richard Bellman in the late 1940's and early 1950's).Richard Bellman in the late 1940's and early 1950's).
For some history and other ideas, see:For some history and other ideas, see:http://en.wikipedia.org/wiki/Dynamic_programming
Two Algorithmic Models: Two Algorithmic Models:
Divide &Conquer
DynamicProgramming
View problem as collection ofsubproblems“Recursive” nature
Independent subproblems
Number of subproblems depends onpartitioning
factors
typically small
PreprocessingCharacteristic running time typically log
function of ndepends on numberand difficulty ofsubproblems
Primarily for optimizationproblemsOptimal substructure:optimal solution to problemcontains within it optimalsolutions to subproblems
Overlapping subproblems
Dynamic ProgrammingDynamic Programming
Example: Rod Cutting (text)Example: Rod Cutting (text)
You are given a rod of length You are given a rod of length nn ≥ 0 ( ≥ 0 (nn in inches) in inches)
A rod of length A rod of length ii inches will be sold for inches will be sold for ppii dollars dollars
Cutting is free (simplifying assumption)Cutting is free (simplifying assumption)
ProblemProblem: given a table of prices : given a table of prices ppii determine the maximum determine the maximum
revenue revenue rrnn obtainable by cutting up the rod and selling the obtainable by cutting up the rod and selling the
pieces.pieces.Length i
Price pi
1 2 3 4 5 6 7 8 9 10
1 5 8 9 10 17 17 20 24 30
Example: Rod CuttingExample: Rod Cutting
We can see immediately (from the values in the table) that We can see immediately (from the values in the table) that
nn ≤ ≤ ppnn ≤ 3 ≤ 3nn. .
This is not very useful because:This is not very useful because:
The range of potential revenue is very largeThe range of potential revenue is very large
Our finding quick upper and lower bounds depends on Our finding quick upper and lower bounds depends on
finding quickly the minimum and maximum finding quickly the minimum and maximum ppii//ii ratios ratios
(one pass through the table), but then we are back to the (one pass through the table), but then we are back to the
point above…. point above….
Example: Rod Cutting Step 1: Characterizing an Optimal SolutionExample: Rod Cutting Step 1: Characterizing an Optimal Solution
QuestionQuestion: in how many different ways can we cut a rod of length : in how many different ways can we cut a rod of length nn??For a rod of length 4:For a rod of length 4:
224 - 14 - 1 = 2 = 233 = 8 = 8
For a rod of length For a rod of length nn: 2: 2nn-1-1. . ExponentialExponential: we cannot try all possibilities for : we cannot try all possibilities for nn "large". The obvious exhaustive approach won't work."large". The obvious exhaustive approach won't work.
Example: Rod Cutting Step 1: Characterizing an Optimal SolutionExample: Rod Cutting Step 1: Characterizing an Optimal Solution
QuestionQuestion: in how many different ways can we cut a rod of length : in how many different ways can we cut a rod of length nn??
Proof DetailsProof Details: a rod of length : a rod of length nn can have exactly can have exactly nn-1 possible cut positions – -1 possible cut positions – choose 0 ≤ choose 0 ≤ kk ≤ ≤ nn-1 actual cuts. We can choose the -1 actual cuts. We can choose the kk cuts (without repetition) cuts (without repetition) anywhere we want, so that for each such anywhere we want, so that for each such kk the number of different choices is the number of different choices is
When we sum up over all possibilities (When we sum up over all possibilities (kk = 0 to = 0 to kk = = nn-1):-1):
For a rod of length For a rod of length nn: 2: 2nn-1-1. .
€
n −1
k
⎛
⎝ ⎜
⎞
⎠ ⎟
k= 0
n−1
∑ =(n −1)!
k!(n −1− k)!k= 0
n−1
∑ = 1+1( )n−1
= 2n−1.€
n −1
k
⎛
⎝ ⎜
⎞
⎠ ⎟
Example: Rod Cutting Characterizing an Optimal SolutionExample: Rod Cutting Characterizing an Optimal Solution
Let us find a way to solve the problem recursively (we might be able to modify the Let us find a way to solve the problem recursively (we might be able to modify the solution so that the maximum can be actually computed): assume we have cut a solution so that the maximum can be actually computed): assume we have cut a rod of length rod of length nn into 0 ≤ into 0 ≤ k k ≤ ≤ nn pieces of length pieces of length ii11, …, , …, iikk,,
nn = = ii11 +…+ +…+ iikk, ,
with revenuewith revenuerrnn = = ppii11 + … + + … + ppikik
Assume further that this solution is optimal.Assume further that this solution is optimal.
How can we construct it?How can we construct it?
AdviceAdvice: when you don’t know what to do next, start with a simple example and : when you don’t know what to do next, start with a simple example and hope something will occur to you…hope something will occur to you…
Example: Rod Cutting Characterizing an Optimal SolutionExample: Rod Cutting Characterizing an Optimal Solution
We begin by constructing (by hand) the optimal solutions for We begin by constructing (by hand) the optimal solutions for ii = 1, …, 10: = 1, …, 10:rr11 = 1 = 1 from sln. 1 = 1 (no cuts)from sln. 1 = 1 (no cuts)
rr22 = 5 = 5 from sln. 2 = 2 (no cuts)from sln. 2 = 2 (no cuts)
rr33 = 8 = 8 from sln. 3 = 3 (no cuts)from sln. 3 = 3 (no cuts)
rr44 = 10 = 10 from sln. 4 = 2 + 2from sln. 4 = 2 + 2
rr55 = 13 = 13 from sln. 5 = 2 + 3from sln. 5 = 2 + 3
rr66 = 17 = 17 from sln. 6 = 6 (no cuts)from sln. 6 = 6 (no cuts)
rr77 = 18 = 18 from sln. 7 = 1 + 6 or 7 = 2 + 2 + 3from sln. 7 = 1 + 6 or 7 = 2 + 2 + 3
rr88 = 22 = 22 from sln. 8 = 2 + 6from sln. 8 = 2 + 6
rr99 = 25 = 25 from sln. 9 = 3 + 6from sln. 9 = 3 + 6
rr1010 = 30 = 30 from sln. 10 = 10 (no cuts)from sln. 10 = 10 (no cuts)
Length i
Price pi
1 2 3 4 5 6 7 8 9 10
1 5 8 9 10 17 17 20 24 30
Example: Rod Cutting Characterizing an Optimal SolutionExample: Rod Cutting Characterizing an Optimal Solution
Notice that in some cases Notice that in some cases rrnn = = ppnn, while in other cases the optimal revenue , while in other cases the optimal revenue rrnn is is
obtained by cutting the rod into smaller pieces. obtained by cutting the rod into smaller pieces. In ALL cases we have the recursionIn ALL cases we have the recursion
rrnn = max( = max(ppnn, , rr11 + + rrnn-1-1, , rr22 + + rrnn-2-2, …, , …, rrnn-1-1 + + rr11))
exhibiting exhibiting optimal substructure optimal substructure (meaning?)(meaning?)A slightly different way of stating the same recursion, which avoids repeating A slightly different way of stating the same recursion, which avoids repeating some computations, issome computations, is
rrnn = max = max1≤1≤ii≤≤nn((ppii + + rrnn--ii))
And this latter relation can be implemented as a simple top-down recursive And this latter relation can be implemented as a simple top-down recursive procedure:procedure:
Example: Rod Cutting Characterizing an Optimal SolutionExample: Rod Cutting Characterizing an Optimal Solution
Time Out:Time Out: How to justify the step from: How to justify the step from:rrnn = max( = max(ppnn, , rr11 + + rrnn-1-1, , rr22 + + rrnn-2-2, …, , …, rrnn-1-1 + + rr11))
totorrnn = max = max1≤1≤ii≤≤nn((ppii + + rrnn--ii))
NoteNote: every optimal partitioning of a rod of length : every optimal partitioning of a rod of length nn has a first cut – a segment of, has a first cut – a segment of, say, length say, length ii. The optimal revenue, . The optimal revenue, rrnn, must satisfy , must satisfy rrnn = = ppii + r + rn-in-i, where , where rrn-in-i is the is the
optimal revenue for a rod of length optimal revenue for a rod of length nn – – ii. If the latter were not the case, there would . If the latter were not the case, there would be a better partitioning for a rod of length be a better partitioning for a rod of length nn – – ii, giving a revenue , giving a revenue r’r’n–in–i > > rrn-in-i and a and a
total revenue total revenue r’r’nn = = ppnn + + r’r’n-in-i > > ppii + r + rn-i n-i == r rnn. .
Since we do not know which one of the leftmost cut positions provides the largest Since we do not know which one of the leftmost cut positions provides the largest revenue, we just maximize over all the possible first cut positions.revenue, we just maximize over all the possible first cut positions.
Example: Rod Cutting Characterizing an Optimal SolutionExample: Rod Cutting Characterizing an Optimal Solution
We can also notice that all the items we choose the maximum of are optimal in We can also notice that all the items we choose the maximum of are optimal in their own right: each substructure (max revenue for rods of lengths 1, …, their own right: each substructure (max revenue for rods of lengths 1, …, nn-1) is -1) is also optimal (again, also optimal (again, optimal substructure propertyoptimal substructure property).).
Nevertheless, we are still in trouble: computing the recursion leads to recomputing Nevertheless, we are still in trouble: computing the recursion leads to recomputing a number (= overlapping subproblems) of values – how many?a number (= overlapping subproblems) of values – how many?
Example: Rod Cutting Characterizing an Optimal SolutionExample: Rod Cutting Characterizing an Optimal Solution
Let’s call Cut-Rod(p, 4), to see the effects on a simple case:Let’s call Cut-Rod(p, 4), to see the effects on a simple case:
The number of nodes for a tree corresponding to a rod of size The number of nodes for a tree corresponding to a rod of size nn is: is:
€
T 0( ) =1, T(n) =1+ T( j)j= 0
n−1
∑ = 2n, n ≥1.
Example: Rod Cutting Beyond Naïve Time ComplexityExample: Rod Cutting Beyond Naïve Time Complexity
We have a problem: “reasonable size” problems are not solvable in “reasonable We have a problem: “reasonable size” problems are not solvable in “reasonable time” (but, in this case, they are solvable in “reasonable space”). time” (but, in this case, they are solvable in “reasonable space”).
SpecificallySpecifically::• Note that navigating the whole tree requires 2Note that navigating the whole tree requires 2nn stack-frame activations. stack-frame activations.• Note also that no more than Note also that no more than nn + 1 stack-frames are active at any one time and that + 1 stack-frames are active at any one time and that no more than no more than nn + 1 different values need to be computed or used. + 1 different values need to be computed or used.
Can we exploit these observationsCan we exploit these observations??A standard solution method involves saving the values associated with each A standard solution method involves saving the values associated with each TT((jj), ), so that we compute each value only once (called “so that we compute each value only once (called “memoizingmemoizing” = writing yourself a ” = writing yourself a memo).memo).
Example: Rod Cutting Naïve CachingExample: Rod Cutting Naïve Caching
We introduce two procedures:We introduce two procedures:
Example: Rod Cutting More Sophisticated CachingExample: Rod Cutting More Sophisticated Caching
We now remove some unnecessary complications:We now remove some unnecessary complications:
Example: Rod Cutting Time ComplexityExample: Rod Cutting Time Complexity
Whether we solve the problem in a top-down or bottom-up manner the asymptotic Whether we solve the problem in a top-down or bottom-up manner the asymptotic time is Θ(time is Θ(nn22), the major difference being recursive calls as compared to loop ), the major difference being recursive calls as compared to loop iterations.iterations.Why??Why??
Example: Longest Common Subsequence (LCS): MotivationExample: Longest Common Subsequence (LCS): Motivation
Strand of DNA: string over finite set {A,C,G,T}Strand of DNA: string over finite set {A,C,G,T} each element of set is a base: each element of set is a base: adenine, guanine, cytosine or thymineadenine, guanine, cytosine or thymine
Compare DNA similaritiesCompare DNA similarities SS1 1 = = ACCACCGGGGTCGTCGAGAGTTGGCGCGCCGGGGAAGCCGGCCGAAAAGCCGGCCGAA
SS2 2 = = GTCGTGTCGTTTCGGAACGGAATTGCCGGCCGTTTTGCGCTTCCTTGGTTAAAAAA
One measure of similarity:One measure of similarity: find the longest string Sfind the longest string S33 containing bases that also appear (not containing bases that also appear (not
necessarily necessarily consecutivelyconsecutively) in S) in S11 and S and S22
SS3 3 = = GTCGTCGGAAGCCGGCCGAAGTCGTCGGAAGCCGGCCGAA
source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.
Example: LCS Definitions
Example: LCS Definitions
The sequenceThe sequence is a is a subsequencesubsequence of of if if (strictly (strictly increasing indices of increasing indices of XX) such that) such that example: example: is a subsequence of is a subsequence of
with index sequence with index sequence
ZZ is is common subsequencecommon subsequence of of XX and and YY if if ZZ is is subsequence of both subsequence of both X X and and YY example:example:
common subsequence but not common subsequence but not longestlongest common subsequence. Longest?common subsequence. Longest?
Longest Common Subsequence ProblemLongest Common Subsequence Problem: Given 2 sequences : Given 2 sequences X,X, YY, find maximum-length common subsequence , find maximum-length common subsequence ZZ..
source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.
€
Z = z1,z2,K ,zk
€
X = x1, x2,K xm
€
∃ i1,i2,K ik
€
∀ j =1,2,K k,
€
x i j= z j
€
Z = B,C,D,B
€
X = A,B,C,B,D, A,B
€
2,3,5,7
€
X = A,B,C,B,D, A,B
€
Y = B,D,C, A,B, A
€
B,C,A
€
B,C,B,A
Example: LCS Step 1: Characterize an LCS
Example: LCS Step 1: Characterize an LCS
THM 15.1: Optimal LCS Substructure THM 15.1: Optimal LCS Substructure Given sequences:Given sequences:For any LCSFor any LCS of of XX and and YY: :
1 1 if if xxmm = = yynn then then zzkk = = xxmm = = yynn and and ZZkk-1-1 is an LCS of is an LCS of XXmm-1-1 and and YYnn-1-1
2 2 if if xxmm ≠ ≠ yynn then then zzkk ≠ ≠ xxmm ZZ is an LCS of is an LCS of XXmm-1-1 and and YY
3 if 3 if xxmm ≠ ≠ yynn then then zzkk ≠ ≠ yynn Z Z is an LCS of is an LCS of XX and and YYnn-1-1
PROOF: based on producing PROOF: based on producing contradictionscontradictions1 a) 1 a) Suppose Suppose zzkk ≠ ≠ xxmm. Appending . Appending xxmm = = yynn to to Z Z contradicts contradicts longest longest nature ofnature of Z Z..
b) To establish b) To establish longest longest nature of nature of ZZkk-1-1, suppose common subsequence , suppose common subsequence W W of of XXmm-1-1 and and YYnn-1-1 has has
length length > > kk-1-1. Appending . Appending xxmm to to WW yields common subsequence of length yields common subsequence of length > > kk = contradiction.= contradiction.
2 2 Common subsequence Common subsequence WW of of XXmm-1-1 and and YY of length of length > > kk would also be common subsequence of would also be common subsequence of XXmm, , YY, contradicting , contradicting longest longest nature of nature of ZZ..
3 3 Similar to proof of (2)Similar to proof of (2)
source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.
€
X = Xm = x1, x2,K , xm
€
Y = Yn = y1,y2,K ,yn
€
Z = Zk = z1,z2,K ,zk
Example: LCS Step 2: A Recursive Solution
Example: LCS Step 2: A Recursive Solution
Implications of Thm 15.1:Implications of Thm 15.1:
??yesyes nono
Find LCS(Xm-1, Yn-1) Find LCS(Xm-1, Y) Find LCS(X, Yn-1)
LCS(X, Y) = LCS(Xm-1, Yn-1) + xm LCS(X, Y) = max(LCS(Xm-1, Y), LCS(X, Yn-1))
LCS(X, Y)
Example: LCS Step 2: A Recursive Solution (continued)
Example: LCS Step 2: A Recursive Solution (continued)
Overlapping subproblem structure:Overlapping subproblem structure:
Recurrence for length of optimal solution:Recurrence for length of optimal solution:
Conditions of problem can exclude some subproblems!
),(),( 111 YXLCSYXLCS mnm −−− ⊆),( 1−⊆ nYXLCS
),( YXLCS⊆
c[i,j]= c[i-1,j-1]+1 if i,j > 0 and xi=yj
max(c[i,j-1], c[i-1,j]) if i,j > 0 and xi=yj
0 if i=0 or j=0
(mn) distinct (mn) distinct subproblemssubproblems
source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.
Example: LCS Step 3: Compute Length of an LCSExample: LCS Step 3: Compute Length of an LCS
c tablec table
(represent b table)(represent b table)source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.
Example: LCS Step 4: Construct an LCSExample: LCS Step 4: Construct an LCS
source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.
Example: LCS Improve the Code
Example: LCS Improve the Code
Can eliminate Can eliminate bb table table cc[[ii,,jj]] depends only on 3 other depends only on 3 other cc table entries: table entries:
cc[[ii-1,-1,jj-1], -1], cc[[ii-1,-1,jj], ], cc[[ii,j,j--1]1] given value of given value of cc[[ii,,jj]], can pick the one in O(1) time, can pick the one in O(1) time reconstruct LCS in O(reconstruct LCS in O(mm++nn) time similar to PRINT-LCS) time similar to PRINT-LCS same same ((mnmn) space, but ) space, but ((mnmn) was needed anyway...) was needed anyway...
Asymptotic space reductionAsymptotic space reduction leverage: need only 2 rows of leverage: need only 2 rows of cc table at a time table at a time
row being computedrow being computed previous rowprevious row
can also do it with ~ space for 1 row of can also do it with ~ space for 1 row of cc table table but does not preserve LCS reconstruction databut does not preserve LCS reconstruction data
source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.
Algorithmic Paradigm ContextAlgorithmic Paradigm Context
Divide &Conquer
DynamicProgramming
GreedyAlgorithm
View problem as collection ofsubproblems“Recursive” natureIndependent subproblems overlapping typically
sequentialdependence
Number of subproblems depends onpartitioning
factors
typically small
Preprocessing typically sortCharacteristic running time typically log
function of ndepends on numberand difficulty ofsubproblems
often dominatedby nlogn sort
Primarily for optimizationproblemsOptimal substructure:optimal solution to problemcontains within it optimalsolutions to subproblemsGreedy choice property:locally optimal producesglobally optimalHeuristic version useful forbounding optimal value