csc401: analysis of algorithms 5-2-1 csc401 – analysis of algorithms chapter 5 – 2....

CSC401: Analysis of AlgorithmsCSC401: Analysis of Algorithms 5-2-5-2-11

CSC401 – Analysis of Algorithms CSC401 – Analysis of Algorithms

Chapter 5 – 2. Chapter 5 – 2.

Divide-and-ConquerDivide-and-ConquerObjectives:Objectives: Introduce the Divide-and-conquer paradigm and Introduce the Divide-and-conquer paradigm and

variants: Decrease-and-Conquer and Transform-and-variants: Decrease-and-Conquer and Transform-and-ConquerConquer

Review the Merge-sort algorithmReview the Merge-sort algorithm Solve recurrence equationsSolve recurrence equations

– Iterative substitutionIterative substitution– Recursion treesRecursion trees– Guess-and-testGuess-and-test– The master methodThe master method

Discuss integer and matrix multiplicationsDiscuss integer and matrix multiplications


Divide-and-ConquerDivide-and-ConquerDivide-and conquerDivide-and conquer is a general algorithm design is a general algorithm design paradigm:paradigm:– DivideDivide: divide the input data : divide the input data SS in two or more disjoint in two or more disjoint

subsets subsets SS11, S, S22, …, …– RecurRecur: solve the subproblems recursively: solve the subproblems recursively– ConquerConquer: combine the solutions for : combine the solutions for SS11,, SS22, …, into a solution , …, into a solution

for for SS

The base case for the recursion are subproblems of The base case for the recursion are subproblems of constant sizeconstant sizeAnalysis can be done using Analysis can be done using recurrence equationsrecurrence equations


Merge-Sort ReviewMerge-Sort ReviewMerge-sort on an input Merge-sort on an input sequence sequence SS with with nn elements consists of elements consists of three steps:three steps:– DivideDivide: partition : partition SS into into

two sequences two sequences SS11 and and SS22 of about of about nn22 elements elements eacheach

– RecurRecur: recursively sort : recursively sort SS11 and and SS22

– ConquerConquer: merge : merge SS11 and and SS2 2 into a unique sorted into a unique sorted sequencesequence

Algorithm mergeSort(S, C)Input sequence S with n

elements, comparator C Output sequence S sorted

according to Cif S.size() > 1

(S1, S2) partition(S, n/2)

mergeSort(S1, C)

mergeSort(S2, C)

S merge(S1, S2)7 2 9 4 2 4 7 9

7 2 2 7 9 4 4 9

7 7 2 2 9 9 4 4


Recurrence Equation AnalysisRecurrence Equation AnalysisThe conquer step of merge-sort consists of merging The conquer step of merge-sort consists of merging two sorted sequences, each with two sorted sequences, each with nn22 elements and elements and implemented by means of a doubly linked list, takes implemented by means of a doubly linked list, takes at most at most bnbn steps, for some constant steps, for some constant bb..Likewise, the basis case (Likewise, the basis case (nn < 2) < 2) will take at will take at bb most most steps.steps.Therefore, if we let Therefore, if we let TT((nn) ) denote the running time of denote the running time of merge-sort:merge-sort:

We can therefore analyze the running time of merge-We can therefore analyze the running time of merge-sort by finding a sort by finding a closed form solutionclosed form solution to the above to the above equation.equation.– That is, a solution that has That is, a solution that has TT((nn) ) only on the left-hand side.only on the left-hand side.

2if)2/(2

2if )(

nbnnT

nbnT


Iterative SubstitutionIterative SubstitutionIn the iterative substitution, or “plug-and-chug,” In the iterative substitution, or “plug-and-chug,” technique, we iteratively apply the recurrence technique, we iteratively apply the recurrence equation to itself and see if we can find a patternequation to itself and see if we can find a pattern

Note that base, T(n)=b, case occurs when 2Note that base, T(n)=b, case occurs when 2ii=n. =n. That is, i = log n. That is, i = log n. So,So,Thus, T(n) is O(n log n).Thus, T(n) is O(n log n).

ibnnT

bnnT

bnnT

bnnT

bnnbnT

bnnTnT

ii

)2/(2

...

4)2/(2

3)2/(2

2)2/(2

))2/())2/(2(2

)2/(2)(

44

33

22

2

nbnbnnT log)(


The Recursion TreeThe Recursion TreeDraw the recursion tree for the recurrence relation Draw the recursion tree for the recurrence relation and look for a pattern:and look for a pattern:

depthdepth T’sT’s sizesize

00 11 nn

11 22 nn22

ii 22ii nn22ii

…… …… ……

2if)2/(2

2if )(

nbnnT

nbnT

timetime

bnbn

bnbn

bnbn

……

Total time = bn + bn log n(last level plus all previous levels)


Guess-and-Test MethodGuess-and-Test MethodIn the guess-and-test method, we guess a closed In the guess-and-test method, we guess a closed form solution and then try to prove it is true by form solution and then try to prove it is true by induction:induction:

Guess: T(n) < cnlogn.Guess: T(n) < cnlogn.

Wrong: we cannot make this last line be less than Wrong: we cannot make this last line be less than cn log ncn log n

nbncnncn

nbnncn

nbnnnc

nbnnTnT

loglog

log)2log(log

log))2/log()2/((2

log)2/(2)(

2iflog)2/(2

2if )(

nnbnnT

nbnT


Guess-and-Test Method: Guess-and-Test Method: Part 2Part 2Recall the recurrence equation:Recall the recurrence equation:

Guess #2: T(n) < cn logGuess #2: T(n) < cn log22 n. n.

– if c > b.if c > b.

So, T(n) is O(n logSo, T(n) is O(n log22 n). n).In general, to use this method, you need to have a In general, to use this method, you need to have a good guess and you need to be good at induction good guess and you need to be good at induction proofs.proofs.

ncn

nbncnncnncn

nbnncn

nbnnnc

nbnnTnT

2

2

2

2

log

loglog2log

log)2log(log

log))2/(log)2/((2

log)2/(2)(

2iflog)2/(2

2if )(

nnbnnT

nbnT


Master MethodMaster MethodMany divide-and-conquer recurrence Many divide-and-conquer recurrence equations have the form:equations have the form:

The Master Theorem:The Master Theorem:

dnnfbnaT

dncnT

if)()/(

if )(

.1 somefor )()/( provided

)),((is)(then),(is)(if 3.

)log(is)(then),log(is)(if 2.

)(is)(then),(is)(if 1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb


Master Method, ExamplesMaster Method, ExamplesThe form:The form:

The Master The Master Theorem:Theorem:

dnnfbnaT

dncnT

if)()/(

if )(





log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

nnTnT )2/(4)(Example 1:Example 1:– Solution: a=4, b=2, logba=2, f(n)=n, so case 1

says T(n) is O(n2).

Example 2:– Solution: a=b=2, logba=1, f(n)=nlogn, so case 2

says T(n) is O(n log2 n).

nnnTnT log)2/(2)(


Master Method, ExamplesMaster Method, Examples

The form:The form:


dnnfbnaT

dncnT

if)()/(

if )(





log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

Example 3:Example 3:– Solution: a=1, b=3, logba=0, f(n)=nlogn, so case 3 says

T(n) is O(nlogn).

Example 4:– Solution: a=2, b=8, logba=3, f(n)=n2, so case 1 says

T(n) is O(n3).

2)2/(8)( nnTnT

nnnTnT log)3/()(


Master Method, ExamplesMaster Method, ExamplesThe form:The form:


1)2/()( nTnT

dnnfbnaT

dncnT

if)()/(

if )(





log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

Example 5:Example 5:– Solution: a=9, b=3, logba=2, f(n)=n3, so case 3 says

T(n) is O(n3).Example 6: (binary search)– Solution: a=1, b=2, logba=0, f(n)=1, so case 2 says

T(n) is O(log n).Example 7: (heap construction)– Solution: a=2, b=2, logba=1, f(n)=logn, so case 1

says T(n) is O(n).

3)3/(9)( nnTnT

nnTnT log)2/(2)(


Iterative “Proof” of the Master TheoremIterative “Proof” of the Master TheoremUsing iterative substitution, let us see if we can Using iterative substitution, let us see if we can find a pattern:find a pattern:

We then distinguish the three cases asWe then distinguish the three cases as– The first term is dominantThe first term is dominant– Each part of the summation is equally dominantEach part of the summation is equally dominant– The summation is a geometric seriesThe summation is a geometric series

1)(log

0

log

1)(log

0

log

2233

22

2

)/()1(

)/()1(

. . .

)()/()/()/(

)()/()/(

))/())/((

)()/()(

n

i

iia

n

i

iin

b

b

b

b

bnfaTn

bnfaTa

nfbnafbnfabnTa

nfbnafbnTa

bnbnfbnaTa

nfbnaTnT


Integer MultiplicationInteger MultiplicationAlgorithm: Multiply two n-bit integers I and Algorithm: Multiply two n-bit integers I and J.J.– Divide step: Split I and J into high-order and low-order Divide step: Split I and J into high-order and low-order

bitsbits

– We can then define I*J by multiplying the parts and We can then define I*J by multiplying the parts and adding:adding:

– So, T(n) = 4T(n/2) + n, which implies T(n) is O(nSo, T(n) = 4T(n/2) + n, which implies T(n) is O(n22).).– But that is no better than the algorithm we learned in But that is no better than the algorithm we learned in

grade school.grade school.

ln

h

ln

h

JJJ

III

2/

2/

2

2

lln

hln

lhn

hh

ln

hln

h

JIJIJIJI

JJIIJI

2/2/

2/2/

222

)2(*)2(*


An Improved Integer Multiplication AlgorithmAn Improved Integer Multiplication Algorithm

Algorithm: Multiply two n-bit integers I and J.Algorithm: Multiply two n-bit integers I and J.– Divide step: Split I and J into high-order and low-order bitsDivide step: Split I and J into high-order and low-order bits

– Observe that there is a different way to multiply parts:Observe that there is a different way to multiply parts:

– So, T(n) = 3T(n/2) + n, which implies T(n) is O(nSo, T(n) = 3T(n/2) + n, which implies T(n) is O(nloglog22

33), by the ), by the Master Theorem.Master Theorem.

– Thus, T(n) is O(nThus, T(n) is O(n1.5851.585).).

ln

h

ln

h

JJJ

III

2/

2/

2

2

lln

hllhn

hh

lln

llhhhlhhlllhn

hh

lln

llhhhllhn

hh

JIJIJIJI

JIJIJIJIJIJIJIJI

JIJIJIJJIIJIJI

2/

2/

2/

2)(2

2])[(2

2]))([(2*


Matrix Multiplication AlgorithmMatrix Multiplication AlgorithmProblem:Problem: Given n×n matrices X and Y, find their Given n×n matrices X and Y, find their product Z=X×Y, product Z=X×Y,

General algorithmGeneral algorithm: O(n: O(n33))

Strassen’s AlgorithmStrassen’s Algorithm::– Rewrite the matrix product as: Rewrite the matrix product as: – Where Where I=AE+BG, J=AF+BHI=AE+BG, J=AF+BH

K=CE+DG, L=CF+DHK=CE+DG, L=CF+DH– Define 7 submatrix products:Define 7 submatrix products:

S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E)S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E)

S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F)S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F)– Construct: Construct:

I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7-S3-I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7-S3-S5S5

– Running time: T(n) = 7T(n/2)+bnRunning time: T(n) = 7T(n/2)+bn22 for some constant b, for some constant b, which gives: O(nwhich gives: O(nlog7log7), better than O(n), better than O(n33))

1

0

],[],[],[n

k

jkYkiXjiZ

HG

FE

DC

BA

LK

JI


Decrease and ConquerDecrease and Conquer Reduce problem instance to smaller Reduce problem instance to smaller

instance of the same problem and extend instance of the same problem and extend solutionsolution

Solve smaller instanceSolve smaller instance Extend solution of smaller instance to Extend solution of smaller instance to

obtain solution to original problemobtain solution to original problem Also referred to as Also referred to as inductiveinductive or or

incrementalincremental approach approach


Examples of Decrease and ConquerExamples of Decrease and ConquerDecrease by one:Decrease by one:– Insertion sortInsertion sort– Graph search algorithms:Graph search algorithms:

DFSDFSBFSBFSTopological sortingTopological sorting

– Algorithms for generating permutations, subsetsAlgorithms for generating permutations, subsets

Decrease by a constant factorDecrease by a constant factor– Binary search Binary search – Fake-coin problemsFake-coin problems– multiplication multiplication à la russeà la russe– Josephus problemJosephus problem

Variable-size decreaseVariable-size decrease– Euclid’s algorithmEuclid’s algorithm– Selection by partitionSelection by partition


Decrease by OneDecrease by OneGenerating all subsetsGenerating all subsets– Given n elements in the setGiven n elements in the set– Take one element from the setTake one element from the set– Generate all subsets of remaining elementsGenerate all subsets of remaining elements– Insert the element taken to the subsets generatedInsert the element taken to the subsets generated– Base case: empty set only has a single subset – itselfBase case: empty set only has a single subset – itself– Time efficiency?Time efficiency?

Generating permutationsGenerating permutations– Given n symbolsGiven n symbols– Take one symbol from the setTake one symbol from the set– Generate all permutations of remaining symbolsGenerate all permutations of remaining symbols– Insert the symbol taken to the permutations generated Insert the symbol taken to the permutations generated – Base case: a single symbol has only one permutation -- Base case: a single symbol has only one permutation --

itself itself – Time efficiency?Time efficiency?


Variable-size-decrease: Variable-size-decrease: Binary search treesBinary search trees

Arrange keys in a binary tree with Arrange keys in a binary tree with the the binary search tree propertybinary search tree property::

k

<k >k • What about repeated keys?What about repeated keys?

Example 1:Example 1: 5, 10, 3, 1, 7, 12, 9

Example 2:Example 2: 4, 5, 7, 2, 1, 3, 6


Searching and insertion in binary Searching and insertion in binary search treessearch trees

Searching – straightforwardSearching – straightforward

Insertion – search for key, insert at leaf where Insertion – search for key, insert at leaf where search terminatedsearch terminated

All operations: worst case # key comparisons = All operations: worst case # key comparisons = hh + 1+ 1

log log nn ≤ ≤ h h ≤ ≤ nn – 1 with average (random files) – 1 with average (random files) 1.41log 1.41log nn

Thus all operations have:Thus all operations have:– worst case: worst case: ΘΘ((nn) ) – average case: average case: ΘΘ(lg(lgnn) )

Bonus:Bonus: inorder traversal produces sorted list inorder traversal produces sorted list ((treesorttreesort))


Transform and ConquerTransform and ConquerSolve problem by transforming into:Solve problem by transforming into:

a more convenient instance of the same problem a more convenient instance of the same problem ((instance simplificationinstance simplification))– presortingpresorting– Gaussian eliminationGaussian elimination

a different representation of the same instance a different representation of the same instance ((representation changerepresentation change))– balanced search treesbalanced search trees– heaps and heapsortheaps and heapsort– polynomial evaluation by Horner’s rulepolynomial evaluation by Horner’s rule– Fast Fourier TransformFast Fourier Transform

a different problem altogether (a different problem altogether (problem reductionproblem reduction))– reductions to graph problemsreductions to graph problems– linear programming linear programming


Instance simplification - PresortingInstance simplification - Presorting

Solve instance of problem by transforming into Solve instance of problem by transforming into another simpler/easier instance of the same another simpler/easier instance of the same problemproblem

Presorting:Presorting:Many problems involving lists are easier when list is Many problems involving lists are easier when list is

sorted.sorted.

searching searching

computing the median (selection problem)computing the median (selection problem)

computing the modecomputing the mode

finding repeated elementsfinding repeated elements


Selection ProblemSelection ProblemFind the kFind the kth th smallest element in A[1],…,A[smallest element in A[1],…,A[nn]. ].

Special cases:Special cases:– minimumminimum: k : k = 1= 1– maximummaximum: k : k = = nn– medianmedian: k : k = = nn/2/2

Presorting-based algorithm Presorting-based algorithm – sort listsort list– return A[return A[kk]]

Partition-based algorithm (Variable decrease & conquer):Partition-based algorithm (Variable decrease & conquer):– pivot/split at A[pivot/split at A[ss] using partitioning algorithm from quicksort] using partitioning algorithm from quicksort– if if ss==k k return A[ return A[ss] ] – else if else if s<k s<k repeat with sublist A[repeat with sublist A[ss+1],…A[+1],…A[nn]. ]. – else if s>k else if s>k repeat with sublist A[1],…A[repeat with sublist A[1],…A[ss-1]. -1].


Notes on Selection ProblemNotes on Selection Problem

Presorting-based algorithm: Presorting-based algorithm: ΩΩ((nnlglgnn) + ) + ΘΘ(1) = (1) = ΩΩ((nnlglgnn) ) Partition-based algorithm (Variable decrease & Partition-based algorithm (Variable decrease & conquer):conquer):– worst case: T(worst case: T(nn) =T() =T(nn-1) + (-1) + (nn+1) -> +1) -> ΘΘ((nn22) ) – best case: best case: ΘΘ((nn) ) – average case: T(average case: T(nn) =T() =T(nn/2) + (/2) + (nn+1) -> +1) -> ΘΘ((nn) ) – Bonus:Bonus: also identifies the also identifies the kk smallest elements (not just smallest elements (not just

the the kkthth))

Special cases max, min: better, simpler linear Special cases max, min: better, simpler linear algorithm (brute force)algorithm (brute force)Conclusion:Conclusion: Presorting does Presorting does notnot help in this case. help in this case.


Finding repeated elementsFinding repeated elementsPresorting-based algorithm: Presorting-based algorithm: – use mergesort (optimal): use mergesort (optimal): ΘΘ((nnlglgnn) ) – scan array to find repeated scan array to find repeated adjacentadjacent elements: elements:

ΘΘ((nn) )

Brute force algorithm: Brute force algorithm: ΘΘ((nn22) )

Conclusion:Conclusion: Presorting yields Presorting yields significantsignificant improvementimprovement

Similar improvement for modeSimilar improvement for mode

What about searching?What about searching?

ΘΘ((nnlglgnn))


Taxonomy of Searching AlgorithmsTaxonomy of Searching Algorithms

Elementary searching algorithmsElementary searching algorithms– sequential searchsequential search– binary searchbinary search– binary tree searchbinary tree search

Balanced tree searchingBalanced tree searching– AVL treesAVL trees– red-black treesred-black trees– multiway balanced trees (2-3 trees, 2-3-4 trees, B trees)multiway balanced trees (2-3 trees, 2-3-4 trees, B trees)

HashingHashing– separate chainingseparate chaining– open addressingopen addressing

csc401: analysis of algorithms 5-2-1 csc401 – analysis of algorithms chapter 5 – 2....

Documents