lecture 5 6_7 - divide and conquer and method of solving recurrences
TRANSCRIPT
Lecture 5 : Divide & Conquer Approach & Methods of Solving
Recurrences
Jayavignesh T
Asst Professor
SENSE
Problems – Big Oh, Big Omega, Big Thetha
• First two functions are linear and hence have a lower order of growth than g(n) = n2, while the last one is quadratic and hence has the same order of growth as n2
• Functions n3 and 0.00001n3 are both cubic and hence have a higher order of growth than n2, and so has the fourth-degree polynomial n4 + n + 1
Problems – Big Oh, Big Omega, Big Thetha
• Ω(g(n)), stands for the set of all functions with a higher or same order of growth as g(n) (to within a constant multiple, as n goes to infinity).
• Θ(g(n)) is the set of all functions that have the same order of growth as g(n) (to within a constant multiple, as n goes to infinity). Every quadratic function an2 + bn + c with a > 0 is in Θ(n2).
Divide-and-Conquer
• The most-well known algorithm design strategy:
1. Divide instance of problem into two or more smaller instances
2. Solve smaller instances recursively
(Base case: If the sub-problem sizes are small enough, solve the sub-problem in a straight forward or direct manner).
3. Obtain solution to original (larger) instance by combining these solutions
• Type of recurrence relation
Example
Algorithm : Largest Number Input : A non-empty list of numbers L Output : The largest number in the list L Comment : Divide and Conquer If L.size == 1 then return L.front Largest 1 <- LargestNumber (L.front .. L.mid) Largest 2 <- LargestNumber (L.mid … L.back) If (Largest 1 > Largest 2) then Largest <- Largest 1 Else Largest <- Largest 2 Return Largest
Recurrence Relation
• Any problem can be solved either by writing recursive algorithm or by writing non-recursive algorithm.
• A recursive algorithm is one which makes a recursive call to itself with smaller inputs.
• We often use a recurrence relation to describe the running time of a recursive algorithm.
• Recurrence relations often arise in calculating the time and space complexity of algorithms.
Recurrence Relations contd..
• A recurrence relation is an equation or inequality that describes a function in terms of its value on smaller inputs or as a function of preceding (or lower) terms.
1. Base step:
– 1 or more constant values to terminate recurrence.
– Initial conditions or base conditions.
2. Recursive steps:
– To find new terms from the existing (preceding) terms.
– The recurrence compute next sequence from the k preceding values .
– Recurrence relation (or recursive formula).
– This formula refers to itself, and the argument of the formula must be on smaller values (close to the base value).
Recurrence Formula : Ex 1 Fibonacci Sequence
• Recurrence has one or more initial conditions and a recursive formula, known as recurrence relation.
• Fibonacci sequence f0,f1,f2…. can be defined by the recurrence relation
• (Base Step) – The given recurrence says that if n=0 then f0=1 and if n=1
then f1=1. – These two conditions (or values) where recursion does
not call itself is called a initial conditions (or Base conditions).
Ex : Fibonacci Sequence contd..
• (Recursive step): This step is used to find new terms f2,f3….from the existing (preceding) terms, by using the formula
• This formula says that “by adding two previous sequence (or term) we can get the next term”.
Recurrence Relation for Factorial Computation
• M(n)
– denoted the number of multiplication required to execute the n!
• Initial condition
– M(1) = 0 ; BASE Step
• n > 1
– Performs 1 Multiplication + FACT recursively called with input n-1
Example 3
• Let T(n) denotes recurrence relation - number of times the statement x=x+1 is executed in the algorithm
x+1 to be executed T(n/2) additional times
Example 3
Performs TWO recursive calls each with the parameter at line 4, and some constant number of basic operations
22
Recurrences and Running Time
• An equation or inequality that describes a function in terms of
its value on smaller inputs.
T(n) = T(n-1) + n
• Recurrences arise when an algorithm contains recursive calls to
itself
• What is the actual running time of the algorithm?
• Need to solve the recurrence
– Find an explicit formula of the expression
– Bound the recurrence by an expression that involves n
23
Recurrent Algorithms - BINARY-SEARCH
• for an ordered array A, finds if x is in the array A[lo…hi]
Alg.: BINARY-SEARCH (A, lo, hi, x)
if (lo > hi)
return FALSE
mid (lo+hi)/2
if x == A[mid]
return TRUE
if ( x < A[mid] )
BINARY-SEARCH (A, lo, mid-1, x)
if ( x > A[mid] )
BINARY-SEARCH (A, mid+1, hi, x)
12 11 10 9 7 5 3 2
1 2 3 4 5 6 7 8
mid lo hi
24
Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
– lo = 1 hi = 8 x = 7
mid = 4, lo = 5, hi = 8
mid = 6, A[mid] = x Found! 11 9 7 5 4 3 2 1
11 9 7 5 4 3 2 1
1 2 3 4 5 6 7 8
8 7 6 5
25
Another Example
• A[8] = {1, 2, 3, 4, 5, 7, 9, 11}
– lo = 1 hi = 8 x = 6
mid = 4, lo = 5, hi = 8
mid = 6, A[6] = 7, lo = 5, hi = 5 11 9 7 5 4 3 2 1
11 9 7 5 4 3 2 1
1 2 3 4 5 6 7 8
11 9 7 5 4 3 2 1 mid = 5, A[5] = 5, lo = 6, hi = 5 NOT FOUND!
11 9 7 5 4 3 2 1
low high
low
low high
high
26
Analysis of BINARY-SEARCH
Alg.: BINARY-SEARCH (A, lo, hi, x) if (lo > hi) return FALSE mid (lo+hi)/2 if x = A[mid] return TRUE if ( x < A[mid] ) BINARY-SEARCH (A, lo, mid-1, x) if ( x > A[mid] ) BINARY-SEARCH (A, mid+1, hi, x)
• T(n) = c +
– T(n) – running time for an array of size n
constant time: c2
same problem of size n/2
same problem of size n/2
constant time: c1
constant time: c3
T(n/2)
27
Methods for Solving Recurrences
• Iteration method
–(unrolling and summing)
• Substitution method
• Recursion tree method
• Master method
• Iteration Method : – Converts the recurrence into a summation and then relies
on techniques for bounding summations to solve the recurrence.
• Substitution Method : – Guess a asymptotic bound and then use mathematical
induction to prove our guess correct.
• Recursive Tree Method : – Graphical depiction of the entire set of recursive
invocations to obtain guess and verify by substitution method.
• Master Method : – Cookbook method for determining asymptotic solutions to
recurrences of a specific form.
Method of Solving Recurrences
29
The Iteration Method
• Convert the recurrence into a summation and
try to bound it using known series
– Iterate the recurrence until the initial condition is
reached.
– Use back-substitution to express the recurrence in
terms of n and the initial (boundary) condition.
Binary Search – Running Time T(n) = c + T(n/2)
T(n) = c + T(n/2)
= c + c + T(n/4)
= c + c + c + T(n/8)
Assume n = 2k
T(n) = c + c + … + c + T(1) = c log2 n + T(1)
= O(log n)
k times
Example Recurrences
• T(n) = T(n-1) + n – Recursive algorithm that loops through the input to eliminate one item • T(n) = T(n/2) + c – Recursive algorithm that halves the input in one step • T(n) = T(n/2) + n – Recursive algorithm that halves the input but must examine every item in the input • T(n) = 2T(n/2) + 1 – Recursive algorithm that splits the input into 2 halves and does a constant amount of other work • T(n) = T(n/3) + T(2n/3) + n
Divide and Conquer – Recurrence form
T(n) – running time of problem of size n.
If the problem size is small enough (say, n ≤ c for some constant c), we have a base case.
The brute-force (or direct) solution takes constant time: Θ(1)
Divide into “a” sub-problems, each 1/b of the size of the original problem of size n.
Each sub-problem of size n/b takes time T(n/b) to solve
Total time to solve “a” sub-problems = spend aT(n/b)
D(n) is the cost(or time) of dividing the problem of size n.
C(n) is the cost (or time) to combine the sub-solutions.
Iteration Method
Unroll (or substitute) the given recurrence back to itself until a regular pattern is obtained (or series).
Steps to solve any recurrence:
1. Expand the recurrence
2. Express the expansion as a summation by plugging the recurrence back into itself until you see a pattern.
3. Evaluate the summation by using the arithmetic or geometric summation formulae
Recursion Tree Method
A convenient way to visualize what happens when a recurrence is iterated.
Pictorial representation of how recurrences is divided till boundary condition
Used to solve a recurrence of the form
Steps for solving a recurrence using recursion Tree:
Step1: Make a recursion tree for a given recurrence as follow:
a) Put the value of f(n) at root node of a tree and make “a” no of child node of this root value f(n).
c) Expand a tree one more level (i.e. up to (at
least) 2 levels)
Steps for solving a recurrence using recursion Tree:
Recursive Tree method
Step2: (a) Find per level cost of a tree
Per level cost = Sum of the cost of each node at that level
Ex : Per level cost at level 1 = Row Sum
Total (final) cost of the tree = Sum of costs of all these levels. – Column Sum
Per level cost = Sum of cost at each level = Row sum
Ex: Depth 2
Total cost is the sum of the costs of all levels (called Column sum), which gives the solution of a given Recurrence
Example 2
• Solve recurrence
using recursive tree method
We always omit floor & ceiling function while
solving recurrence. Thus given recurrence can
be written
Example 3
• To move n disks (n > 1) from peg A to C
– Move (n-1) disks recursively from peg A to peg B using peg C as auxillary = M(n-1) moves
– Move the nth disk directly (last) from peg A to peg C = 1 Move
– Move (n-1) disk recursively from peg B to peg C using peg A as auxillary = M(n-1) moves
Recurrence Relation for the Towers of Hanoi
Given: T(1) = 1
T(n) = 2 T( n-1 ) +1
N No.Moves
1 1
2 3
3 7
4 15
5 31
T(n) = 2 T( n-1 ) +1
T(n) = 2 +1
T(n) = 2 [ 2 T(n-2) + 1 ] +1
T(n) = 2 [ 2 + 1 ] +1
T(n) = 2 [ 2 [ 2 T(n-3) + 1 ]+ 1 ] +1
T(n) = 2 [ 2 [ 2 [ 2 T( n-4 ) + 1 ] + 1 ]+ 1 ] +1
. . .
T(n) = 24 T ( n-4 ) + 15
T(n) = 2k T ( n-k ) + 2k - 1
Since n is finite, k -> n. Therefore,
lim T(n) k -> n = 2n - 1
Tower of Hanoi - Recursion
TOWER(n, A, C, B) {
TOWER(n-1, A, B, C);
Move(A, C);
TOWER(n-1, B, C, A) }
if n<1 return;
Quick Sort
Divide:
• A [p. . r] is partitioned (rearranged) into A [p..q-1] and A [q+1,..r],
• Each element in the left subarray A[p…q-1] is ≤ A [q] and
• A[q] is ≤ each element in the right subarray A[q+1…r]
• PARTITION procedure (Divide Step); returns the index q, where the array gets partitioned.
Conquer:
• These two subarray A [p…q-1] and A [q+1..r] are sorted by recursive calls to QUICKSORT.
Combine:
• Since the subarrays are sorted in place, so there is no need to combine the subarrays.
Pseudo Code of Quick Sort
• QUICK SORT (A, p, r)
{
If (p < r) /* Base Condition
{
q ← PARTITION (A, p, r) /* Divide Step*/
QUICKSORT (A, p, q-1) /* Conquer
QUICKSORT (A, q+1, r) /* Conquer
}
}
Pseudo Code of Quick Sort
PARTITION (A, p, r) {
x ← A[r] /* select last element as pivot */
i ← p – 1 /* i is pointing one position before than p initially */
for j ← p to r − 1 do
{
if (A[j] ≤ x)
{
i ← i + 1
swap(A[i],A[j])
}
}/* end for */
swap(A [i + 1] ,A[r])
return ( i+1)
} /* end module */
QUICK-SORT
Fastest known Sorting algorithm in practice
Running time of Quick-Sort depends on the nature of its input data
Worst case (when input array is already sorted) O(n2)
Best Case (when input data is not sorted) Ω(nlogn)
Average Case (when input data is not sorted & Partition of array is not unbalanced as worst case) : Θ(nlogn)
91
Master’s method
• “Cookbook” for solving recurrences of the form:
where, a ≥ 1, b > 1, and f(n) > 0
Idea: compare f(n) with nlogba
• f(n) is asymptotically smaller or larger than nlogba by a
polynomial factor n
• f(n) is asymptotically equal with nlogba
)()( nfb
naTnT
93
Master’s method
• “Cookbook” for solving recurrences of the form:
where, a ≥ 1, b > 1, and f(n) > 0
Case 1: if f(n) = O(nlogba -) for some > 0, then: T(n) = (nlog
ba)
Case 2: if f(n) = (nlogba), then: T(n) = (nlog
ba lgn)
Case 3: if f(n) = (nlogba +) for some > 0, and if
af(n/b) ≤ cf(n) for some c < 1 and all sufficiently large n, then:
T(n) = (f(n))
)()( nfb
naTnT
regularity condition
94
Examples
T(n) = 2T(n/2) + n
a = 2, b = 2, log22 = 1
Compare nlog2
2 with f(n) = n
f(n) = (n) Case 2
T(n) = (nlgn)
95
Examples
T(n) = 2T(n/2) + n2
a = 2, b = 2, log22 = 1
Compare n with f(n) = n2
f(n) = (n1+) Case 3 verify regularity cond.
a f(n/b) ≤ c f(n)
2 n2/4 ≤ c n2 c = ½ is a solution (c<1)
T(n) = (n2)
96
Examples (cont.)
T(n) = 2T(n/2) +
a = 2, b = 2, log22 = 1
Compare n with f(n) = n1/2
f(n) = O(n1-) Case 1
T(n) = (n)
n
97
Examples
T(n) = 2T(n/2) + nlgn
a = 2, b = 2, log22 = 1
• Compare n with f(n) = nlgn
– seems like case 3 should apply
• f(n) must be polynomially larger by a factor of n
• In this case it is only larger by a factor of lgn
98
Examples
T(n) = 3T(n/4) + nlgn
a = 3, b = 4, log43 = 0.793
Compare n0.793 with f(n) = nlgn
f(n) = (nlog4
3+) Case 3
Check regularity condition:
3(n/4)lg(n/4) ≤ (3/4)nlgn = c f(n),
c=3/4
T(n) = (nlgn)
Substitution Method
• Step1
– Guess the form of the Solution.
• Step2
– Prove your guess is correct by using Mathematical Induction
Mathematical Induction
• Proof by using Mathematical Induction of a given statement (or formula), defined on the positive integer N, consists of two steps:
1. (Base Step): Prove that S(1) is true
2. (Inductive Step): Assume that S(n) is true, and prove that S(n+1) is true for all n>=1
Substitution method
• Guess a solution – T(n) = O(g(n)) – Induction goal: apply the definition of the
asymptotic notation • T(n) ≤ c g(n), for some c > 0 and n ≥ n0 – Induction hypothesis: T(k) ≤ c g(k) for all k < n
(strong induction) • Prove the induction goal – Use the induction hypothesis to find some
values of the constants c and n0 for which the induction goal holds
Substitution Method
• T(n) = 2 if 1<=n<3
3T(n/3)+n if n>=3
• Guess the solution is T(n) = O(nlogn)
• Prove by mathematical induction
To prove : T(n) = O(nlogn)
T(n) <=cnlogn , n>=n0
Induction hypothesis : Let n > n0 and assume k < n
T(k) <=cklog(k)
Substitution method
• Let’s take k = (n/3), T(n/3) <= c(n/3) log (n/3)
• To show T(n) <=cnlogn
T(n) = 3T(n/3) + n ( By recurrence for T)
T(n) = 3c(n/3)log(n/3) + n (By Induction hypothesis)
T(n) = cn(log n -1) + n
T(n) = cnlogn – cn + n
• To obtain T(n) <=cnlogn we need to have –cn+n <=0 , so c >=1 Induction step is cleared
• To determine n0, base step T(n0) <=cn0(logn0)
Advantages of Divide and Conquer
• Solving difficult problems
• Algorithm Efficiency
– Size n/b at each stage
• Parallelism
– Sub problems – multiprocessor
• Memory Access
– Make efficient use of memory cache