algorithm design and analysis - sjtuyuxi/teaching/lectures/theory... · 2014. 12. 29. · algorithm...

Algorithm Design and Analysis

Yuxi Fu

BASICS, Department of Computer ScienceShanghai Jiaotong University

December 29, 2010

Part A. Introduction

I. Basic Concepts in Algorithmic Analysis

Algorithm

An algorithm is a procedure that consists of a finite set ofinstructions which, given an input from some set of possible inputs,enables us to obtain an output through a systematic execution ofthe instructions that terminates in a finite number of steps.

Theorem proving is in general not algorithmic.

Theorem verification is often algorithmic.

Quotation from Donald E. Knuth

Computer Science is the study of algorithms.

Remark on Algorithm

The word ‘algorithm’ is derived from the name of Muhamma ibnMūsā al-Khwārizm̄i (780?-850?), a Muslim mathematician whoseworks introduced Arabic numerals and algebraic concepts toWestern mathematics. The word ‘algebra’ stems from the title ofhis book Kitab al jahr wa’l-muqābala. (American HeritageDictionary)

Algorithm vs. Program

A program is an implementation of an algorithm, or algorithms.

A program does not necessarily terminate.

What is Computer Science?

Computer Science is the study of problem solving usingcomputing machines. The computing machines must be physicallyfeasible.

I. Theory of Computation is to understand the notion ofcomputation in a formal framework.

Some well known models are: the general recursive functionmodel of Gödel and Church, Church’s λ-calculus, Post systemmodel, Turing machine model, RAM, etc.

II. Computability Theory studies what problems can be solved bycomputers.

III. Computational Complexity studies how much resource isnecessary in order to solve a problem.

IV. Theory of Algorithm studies how problems can be solved.

The problem to start with: Search and Ordering.

Linear Search, First Example of an Algorithm

Algorithm 1.1 LinearSearchInput: An array A[1..n] of n elements and an element x .Output: j if x = A[j ], 1 ≤ j ≤ n, and 0 otherwise.

1. j ← 12. while j < n and x 6= A[j ]3. j ← j + 14. end while5. if x = A[j ] then return j else return 0

Binary Search

Algorithm 1.2 BinarySearchInput: An array A[1..n] of n elements sorted in nondecreasingorder and an element x .Output: j if x = A[j ], 1 ≤ j ≤ n, and 0 otherwise.

1. low ← 1; high← n; j ← 02. while low ≤ high and j = 03. mid ← b(low + high)/2c4. if x = A[mid ] then j ← mid5. else if x < A[mid ] then high← mid − 16. else low ← mid + 17. end while8. return j

Analysis of BinarySearch

Suppose x ≥ 35. A run of BinarySearch on A[1..14] (see below) is

1 4 5 7 8 9 10 12 15 22 23 27 32 35

↓

12 15 22 23 27 32 35

↓

27 32 35

↓

35

Analysis of BinarySearch

The complexity of the algorithm is the number of comparison.

The number of comparison is maximum if x ≥ A[n].The number of comparisons is the same as the number ofiterations.

In the second iteration, the number of elements inA[mid + 1..n] is exactly bn/2c.In the j-th iteration, the number of elements in A[mid + 1..n]is exactly bn/2j−1c.The maximum number of iteration is the j such thatbn/2j−1c = 1, which is equivalent toj − 1 ≤ log n < j .Hence j = blog nc+ 1.

Merging Two Sorted Lists

Algorithm 1.3 MergeInput: An array A[1..m] of elements and three indices p, q and r .with 1 ≤ p ≤ q < r ≤ m, such that both the subarray A[p..q] andA[q + 1..r ] are sorted individually in nondecreasing order.Output: A[p..r ] contains the result of merging the two subarraysA[p..q] and A[q + 1..r ].Comment: B[p..r ] is an auxiliary array

Merging Two Sorted Lists

1. s ← p; t ← q + 1; k ← p2. while s ≤ q and t ≤ r3. if A[s] ≤ A[t] then4. B[k]← A[s]5. s ← s + 16. else7. B[k]← A[t]8. t ← t + 19. end if

10. k ← k + 111. end while12. if s = q + 1 then B[k ..r ]← A[t..r ]13. else B[k ..r ]← A[s..q]13. end if13. A[p..r ]← B[p..r ]

Analysis of Merge

Suppose A[p..q] has m elements and A[q + 1..r ] has n elements.The number of comparisons done by Algorithm Merge is

at least min{m, n}; andat most m + n − 1.

Selection Sort

Algorithm 1.4 SelectionSortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.

1. for i ← 1 to n − 12. k ← i3. for j ← i + 1 to n4. if A[j ] < A[k] then k ← j5. end for6. if k 6= i then interchange A[i ] and A[k]7. end for

Analysis of SelectionSort

The number of comparisons carried out by Algorithm SelectionSortis precisely

n−1∑i=1

(n − i) = n(n − 1)2

Insertion Sort

Algorithm 1.5 InsertionSortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.

1. for i ← 2 to n2. x ← A[i ]3. j ← i − 14. while j > 0 and A[j ] > x5. A[j + 1]← A[j ]6. j ← j − 17. end while8. A[j + 1]← x9. end for

Analysis of InsertionSort

The number of comparisons carried out by Algorithm InsertionSortis at least

n − 1

and at mostn∑

i=2

(i − 1) = n(n − 1)2

Bottom-Up Merge Sort

Algorithm 1.6 BottomUpSortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.

1. t ← 12. while t < n3. s ← t; t ← 2s; i ← 04. while i + t ≤ n5. Merge(A, i + 1, i + s, i + t)6. i ← i + t7. end while8. if i + s < n then Merge(A, i + 1, i + s, n)9. end while

Analysis of BottomUpSort

Suppose that n is a power of 2, say n = 2k .

The outer while loop is executed k = log n times.

Step 8 is never invoked.

In the j-th iteration of the outer while loop, there are2k−j = n/2j pairs of arrays of size 2j−1.

The number of comparisons needed in the merge of two sortedarrays in the j-th iteration is at least 2j−1 and at most 2j − 1.The number of comparisons in BottomUpSort is at least

k∑j=1

(n

2j)2j−1 =

k∑j=1

n

2=

n log n

2

The number of comparisons in BottomUpSort is at most

k∑j=1

(n

2j)(2j − 1) =

k∑j=1

(n − n2j

) = n log n − n + 1

Time Complexity

Computational Complexity evolved from 1960’s, flourished in1970’s and 1980’s.

Time is the most precious resource.

Important to human.

Order of Growth

Our main concern is about the order of growth.

Our estimates of time are relative rather than absolute.

Our estimates of time are machine independent.

Our estimates of time are about the behavior of the algorithmunder investigation on large input instances.

So we are measuring the asymptotic running time of thealgorithms.

Order of Growth

Our main concern is about the order of growth.

Our estimates of time are relative rather than absolute.

Our estimates of time are machine independent.

Our estimates of time are about the behavior of the algorithmunder investigation on large input instances.

So we are measuring the asymptotic running time of thealgorithms.

The O-Notation

The O-notation provides an upper bound of the running time; itmay not be indicative of the actual running time of an algorithm.

Definition (O-Notation)

Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be O(g(n)),written f (n) = O(g(n)), if

∃c .∃n0.∀n ≥ n0.f (n) ≤ cg(n)

Intuitively, f grows no faster than some constant times g .

The Ω-Notation

The Ω-notation provides a lower bound of the running time; it maynot be indicative of the actual running time of an algorithm.

Definition (Ω-Notation)

Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be Ω(g(n)),written f (n) = Ω(g(n)), if

∃c .∃n0.∀n ≥ n0.f (n) ≥ cg(n)

Clearly f (n) = O(g(n)) if and only if g(n) = Ω(f (n)).

The Θ-Notation

The Θ-notation provides an exact picture of the growth rate of therunning time of an algorithm.

Definition (Θ-Notation)

Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be Θ(g(n)),written f (n) = Θ(g(n)), if both f (n) = O(g(n)) andf (n) = Ω(g(n)).

Clearly f (n) = Θ(g(n)) if and only if g(n) = Θ(f (n)).

Examples

10n2 + 20n = O(n2).

log n2 = O(n).

log nk = Ω(log n).

n! = O((n + 1)!).

Examples

Consider the series∑n

j=1 log j . Clearly,

n∑j=1

log j ≤n∑

j=1

log n = n log n

On the other hand,

n∑j=1

log j ≥bn/2c∑j=1

log(n

2) = bn/2c log(n

2) = bn/2c log n − bn/2c

That isn∑

j=1

log j = Ω(n log n)

Examples

log n! =∑n

j=1 log j = Ω(n log n).

2n = O(n!).

The o-Notation

Definition (o-Notation)

Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be o(g(n)),written f (n) = o(g(n)), if

∀c .∃n0.∀n ≥ n0.f (n) < cg(n)

The ω-Notation

Definition (ω-Notation)

Let f (n) and g(n) be functions from the set of natural numbers tothe set of nonnegative real numbers. f (n) is said to be ω(g(n)),written f (n) = ω(g(n)), if

∀c .∃n0.∀n ≥ n0.f (n) > cg(n)

Definition in Terms of Limits

Suppose limn→∞ f (n)/g(n) exists.

limn→∞ f (n)/g(n) 6=∞ implies f (n) = O(g(n)).limn→∞ f (n)/g(n) 6= 0 implies f (n) = Ω(g(n)).limn→∞ f (n)/g(n) = c implies f (n) = Θ(g(n)).

limn→∞ f (n)/g(n) = 0 implies f (n) = o(g(n)).

limn→∞ f (n)/g(n) =∞ implies f (n) = ω(g(n)).

A Helpful Analogy

f (n) = O(g(n)) is similar to f (n) ≤ g(n).f (n) = o(g(n)) is similar to f (n) < g(n).

f (n) = Θ(g(n)) is similar to f (n) = g(n).

f (n) = Ω(g(n)) is similar to f (n) ≥ g(n).f (n) = ω(g(n)) is similar to f (n) > g(n).

Complexity Classes

An equivalence relation R on the set of complexity functions isdefined as follows: fRg if and only if f (n) = Θ(g(n)).

A complexity class is an equivalence class of R.

The equivalence classes can be ordered by ≺ defined as follows:f ≺ g iff f (n) = o(g(n)).

Complexity Classes

Space Complexity

The space complexity is defined to be the number of cells (workspace)) needed to carry out an algorithm, excluding the spaceallocated to hold the input.

The exclusion of the input space is to make sense the sublinearspace complexity.

Space Complexity

The space complexity is defined to be the number of cells (workspace)) needed to carry out an algorithm, excluding the spaceallocated to hold the input.

The exclusion of the input space is to make sense the sublinearspace complexity.

Space Complexity

It is clear that the work space of an algorithm can not exceed therunning time of the algorithm. That is S(n) = O(T (n)).

Trade-off between time complexity and space complexity.

Space Complexity

It is clear that the work space of an algorithm can not exceed therunning time of the algorithm. That is S(n) = O(T (n)).

Trade-off between time complexity and space complexity.

Optimal Algorithm

In general, if we can prove that any algorithm to solve problem Πmust be Ω(f (n)), then we call any algorithm to solve problem Π intime O(f (n)) an optimal algorithm for problem π.

HOW do we estimate time complexity?

Counting the Iterations

Algorithm 1.7 Count1Input: n = 2k , for some positive integer k .Output: count = number of times Step 4 is executed.

1. count ← 0;2. while n ≥ 13. for j ← 1 to n4. count ← count + 15. end for6. n← n/27. end while8. return count

Algorithm 1.7 Count1Input: n = 2k , for some positive integer k .Output: count = number of times Step 4 is executed.

1. count ← 0;2. while n ≥ 13. for j ← 1 to n4. count ← count + 15. end for6. n← n/27. end while8. return count

while is executed k + 1 times; for is executed n, n/2, . . . , 1 times

k∑j=0

n

2j= n

k∑j=0

1

2j= n(2− 1

2k) = 2n − 1 = Θ(n)

Algorithm 1.8 Count2Input: A positive integer n.Output: count = number of times Step 5 is executed.

1. count ← 0;2. for i ← 1 to n3. m← bn/ic4. for j ← 1 to m5. count ← count + 16. end for7. end for8. return count

Algorithm 1.8 Count2Input: A positive integer n.Output: count = number of times Step 5 is executed.

1. count ← 0;2. for i ← 1 to n3. m← bn/ic4. for j ← 1 to m5. count ← count + 16. end for7. end for8. return count

The inner for is executed n, bn/2c, bn/4c, . . . , bn/nc times

Θ(n log n) =n∑

i=1

(n

i− 1) ≤

n∑i=1

bnic ≤

n∑i=1

n

i= Θ(n log n)

Counting the Frequency of Basic Operations

Definition

An elementary operation in an algorithm is called a basic operationif it is of highest frequency to within a constant factor among allother elementary operations.

Worst Case Analysis

Consider the following algorithm:

1. if n is odd then k ← BinarySearch(A, x)2. else k ← LinearSearch(A, x)

In the worst case, the running time is Ω(log(n)) and O(n).

Average Case Analysis

Take Algorithm InsertionSort for instance. Two assumptions:

A[1..n] contains the numbers 1 through n.

All n! permutations are equally likely.

The number of comparisons for inserting element A[i ] in its properposition, say j , is on average the following

i − 1i

+i∑

j=2

i − j + 1i

=i − 1

i+

i−1∑j=1

j

i=

i

2− 1

i+

1

2

The average number of comparisons performed by AlgorithmInsertionSort is

n∑i=2

(i

2− 1

i+

1

2) =

n2

4+

3n

4−

n∑i=1

1

i

Amortized Analysis

In amortized analysis, we average out the time taken by theoperation throughout the execution of the algorithm, and refer tothis average as the amortized running time of that operation.

Amortized analysis guarantees the average cost of the operation,and thus the algorithm, in the worst case.

Amortized Analysis

In amortized analysis, we average out the time taken by theoperation throughout the execution of the algorithm, and refer tothis average as the amortized running time of that operation.

Amortized analysis guarantees the average cost of the operation,and thus the algorithm, in the worst case.

Amortized Analysis

1. for j ← 1 to n2. x ← A[j ]3. Append x to the list4. if x is even then5. while pred(x) is odd do delete pred(x)6. end if7. end for

The total number of elementary operations of insertions anddeletions is between n and 2n− 1. So the time complexity is Θ(n).

It follows that the time used to delete each element is O(1)amortized time.

Amortized Analysis

Input Size and Problem Instance

Suppose that the following integer

21024 − 1

is a legitimate input of an algorithm. What is the size of the input?

Algorithm 1.9 FIRSTInput: A positive integer n and an array A[1..n] with A[j ] = j for1 ≤ j ≤ n.Output:

∑nj=1 A[j ].

1. sum← 0;2. for j ← 1 to n3. sum← sum + A[j ]4. end for5. return sum

The input size is n. The time complexity is O(n). It is linear time.

Algorithm 1.10 SECONDInput: A positive integer n.Output:

∑nj=1 j .

1. sum← 0;2. for j ← 1 to n3. sum← sum + j4. end for5. return sum

The input size is k = blog nc+ 1. The time complexity is O(2k). Itis exponential time.

Exercise

1.5, 1.9, 1.13, 1.16, 1.17, 1.25, 1.31, 1.32, 1.33, 1.37

II. Mathematical Preliminaries

Mathematical Induction

Consider the fibonacci sequence 1, 1, 2, 3, 5, 8, . . ., defined by

f (n)def=

1, if n = 11, if n = 2f (n − 1) + f (n − 2), if n > 2

Let φ = 1+√

52 . Prove that f (n) ≤ φ

n−1 for all n.

Notice that φ2 = φ+ 1. So the induction may go as follows:

f (n) = f (n − 1) + f (n − 2) ≤ φn−2 + φn−3 = φn−3(φ+ 1) = φn−1

Mathematical Induction

Consider the fibonacci sequence 1, 1, 2, 3, 5, 8, . . ., defined by

f (n)def=

1, if n = 11, if n = 2f (n − 1) + f (n − 2), if n > 2

Let φ = 1+√

52 . Prove that f (n) ≤ φ

n−1 for all n.

Notice that φ2 = φ+ 1. So the induction may go as follows:

f (n) = f (n − 1) + f (n − 2) ≤ φn−2 + φn−3 = φn−3(φ+ 1) = φn−1

The Pigeonhole Principle

This principle is easy, powerful and indispensable.

Example. If n balls are distributed into m boxes, then(i) one box must contain at least dn/me balls, and(ii) one box must contain at most bn/mc balls.

Proof. If no box contains dn/me balls or more, then the total ballsin all the boxes are at most

m(d nme − 1) ≤ m( n

m+

m − 1m

− 1) < n

If all boxes have greater than bn/mc balls, then the total balls inall the boxes are at least

m(b nmc+ 1) ≥ m( n

m− m − 1

m+ 1) > n

The Pigeonhole Principle

This principle is easy, powerful and indispensable.

Example. If n balls are distributed into m boxes, then(i) one box must contain at least dn/me balls, and(ii) one box must contain at most bn/mc balls.

Proof. If no box contains dn/me balls or more, then the total ballsin all the boxes are at most

m(d nme − 1) ≤ m( n

m+

m − 1m

− 1) < n

If all boxes have greater than bn/mc balls, then the total balls inall the boxes are at least

m(b nmc+ 1) ≥ m( n

m− m − 1

m+ 1) > n

Logarithms

Suppose that b is a positive real number, x a real number and y apositive real number such that y = bx . Then x is called thelogarithm of y to the base b, denoted by

x = logb y

Here b is referred to as the base of the logarithm. We shall writelog x for log2 x and ln x for loge x , the natural logarithm, where

e = limn→∞

(1 +1

n)n = 1 +

1

1!+

1

2!+

1

3!+ · · · = 2.7182818 . . .

It is well known that

ln x =

∫ x1

1

tdt

Logarithms

Suppose that b is a positive real number, x a real number and y apositive real number such that y = bx . Then x is called thelogarithm of y to the base b, denoted by

x = logb y

Here b is referred to as the base of the logarithm. We shall writelog x for log2 x and ln x for loge x , the natural logarithm, where

e = limn→∞

(1 +1

n)n = 1 +

1

1!+

1

2!+

1

3!+ · · · = 2.7182818 . . .

It is well known that

ln x =

∫ x1

1

tdt

Logarithms

Facts:

logb xy = logb x + logb y

logb cy = y logb c

logb x =loga x

loga b

x loga y = y loga x

Floor and Ceiling Functions

The floor of x , denoted by bxc, is the greatest integer less than orequal to x .

The ceiling of x , denoted by dxe, is the least integer greater thanor equal to x .

Facts

bx/2c+ dx/2e = xb−xc = −dxed−xe = −bxc

Facts

Theorem

Let f (x) be a continuous and monotonically increasing functionsuch that x is an integer whenever f (x) is an integer. Then

bf (bxc)c = bf (x)cdf (dxe)e = df (x)e

Proof: bf (x)c = f (x ′) for some integer x ′. Hence x ′ ≤ bxc. Thusbf (x)c = f (x ′) ≤ f (bxc). It follows that bf (x)c ≤ bf (bxc)c.

Theorem

Let f (x) be a continuous and monotonically increasing functionsuch that x is an integer whenever f (x) is an integer. Then

bf (bxc)c = bf (x)cdf (dxe)e = df (x)e

Proof: bf (x)c = f (x ′) for some integer x ′. Hence x ′ ≤ bxc. Thusbf (x)c = f (x ′) ≤ f (bxc). It follows that bf (x)c ≤ bf (bxc)c.

For example

d√dxee = d

√xe

blogbxcc = blog xcddxe/ne = dx/nebbxc/nc = bx/nc

Summation

Arithmetic Series:

n∑j=1

j =n(n + 1)

2= Θ(n2)

Geometric Series:

n∑j=0

c j =cn+1 − 1

c − 1= Θ(cn)

Summation

Arithmetic Series:

n∑j=1

j =n(n + 1)

2= Θ(n2)

Geometric Series:

n∑j=0

c j =cn+1 − 1

c − 1= Θ(cn)

Approximation of Summations by Integration

Suppose that f (x) is continuous and monotonically increasing.Then ∫ n

m−1f (x)dx ≤

n∑j=m

f (j) ≤∫ n+1m

f (x)dx

Suppose that f (x) is continuous and monotonically decreasing.Then ∫ n+1

mf (x)dx ≤

n∑j=m

f (j) ≤∫ nm−1

f (x)dx

Consider the summation∑n

j=1 jk . Since jk is increasing, we have∫ n

0xkdx ≤

n∑j=1

xk ≤∫ n+1

1xkdx

That isnk+1

k + 1≤

n∑j=1

xk ≤ (n + 1)k+1 − 1

k + 1

Hencen∑

j=1

xk = Θ(nk+1)

Consider the Harmonic Series

Hn =n∑

j=1

1

j

Since 1x is monotonically decreasing, we have∫ n+11

1

xdx ≤

n∑j=1

1

j= 1 +

n∑j=2

1

j≤ 1 +

∫ n1

1

xdx

That is

ln(n + 1) ≤n∑

j=1

1

j≤ 1 + ln n

Hence

Hn =n∑

j=1

1

j= Θ(ln n) = Θ(log n)

Consider the seriesn∑

j=1

log j

Since log x is monotonically increasing, we have∫ n1

log(x)dx ≤n∑

j=2

log j =n∑

j=1

log j ≤ log n +∫ n

1log(x)dx

That is

n log n− n log e + log e ≤n∑

j=1

log j ≤ log n + n log n− n log e + log e

Hencen∑

j=1

log j = Θ(n log n)

Recursively Defined Functions

The complexity functions for recursive algorithms are oftenrecursively defined. This is why solving recursive functionequations are of crucial importance to algorithm analysis.

Solution of Linear Homogeneous Recurrence

Consider the following recursively defined function

f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)

Substituting xn for f (n) in (1) yields

xn = a1xn−1 + a2x

n−2 + . . .+ akxn−k

Dividing both sides by xn−k yields

xk = a1xk−1 + a2x

k−2 + . . .+ ak

or equivalently

xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)

(2) is called the characteristic function of (1).

f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)

xn = a1xn−1 + a2x

n−2 + . . .+ akxn−k

xk = a1xk−1 + a2x

k−2 + . . .+ ak

or equivalently

xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)

f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)

xn = a1xn−1 + a2x

n−2 + . . .+ akxn−k

xk = a1xk−1 + a2x

k−2 + . . .+ ak

or equivalently

xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)

f (n) = a1f (n − 1) + a2f (n − 2) + . . .+ ak f (n − k) (1)

xn = a1xn−1 + a2x

n−2 + . . .+ akxn−k

xk = a1xk−1 + a2x

k−2 + . . .+ ak

or equivalently

xk − a1xk−1 − a2xk−2 − . . .− ak = 0 (2)

Let x1, . . . , xk be the solutions to (2). Then f (n) = xni is a solution

to (1) when n ≥ k .

Suppose the general solution to (1) is

f (n) = c1xn1 + . . .+ ckx

nk .

In order for f (n) to be a solution when n < k, we solve thefollowing linear equation

x1c1 + . . .+ xkck = f (0),...

xk−11 c1 + . . .+ xk−1k ck = f (k − 1).

f (n) = c1xn1 + . . .+ ckx

nk .

x1c1 + . . .+ xkck = f (0),...

xk−11 c1 + . . .+ xk−1k ck = f (k − 1).

f (n) = c1xn1 + . . .+ ckx

nk .

x1c1 + . . .+ xkck = f (0),...

xk−11 c1 + . . .+ xk−1k ck = f (k − 1).

Example 1. The sequence 1, 4, 16, 64, 256, . . . is generated by

f (n) = 3f (n − 1) + 4f (n − 2)

The solution is f (n) = 4n.

Example 2. Consider the Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, . . .which can be computed by the recurrence

f (n)def=

0, if n = 01, if n = 1f (n − 1) + f (n − 2), if n ≥ 2

The characteristic function is x2 − x − 1 = 0, the two roots ofwhich are

1 +√

5

2and

1−√

5

2

Example 2. Consider the Fibonacci sequence 0, 1, 1, 2, 3, 5, 8, . . .which can be computed by the recurrence

f (n)def=

0, if n = 01, if n = 1f (n − 1) + f (n − 2), if n ≥ 2

The characteristic function is x2 − x − 1 = 0, the two roots ofwhich are

1 +√

5

2and

1−√

5

2

Let the solution be

f (n) = c1

(1 +√

5

2

)n+ c2

(1−√

5

2

)n

Then

c1 + c2 = 0

c11 +√

5

2+ c2

1−√

5

2= 1

One has that

c1 =1√5

and c2 = −1√5

So the solution is

f (n) =1√5

(1 +√

5

2

)n− 1√

5

(1−√

5

2

)n

Let the solution be

f (n) = c1

(1 +√

5

2

)n+ c2

(1−√

5

2

)nThen

c1 + c2 = 0

c11 +√

5

2+ c2

1−√

5

2= 1

One has that

c1 =1√5

and c2 = −1√5

So the solution is

f (n) =1√5

(1 +√

5

2

)n− 1√

5

(1−√

5

2

)n

Let the solution be

f (n) = c1

(1 +√

5

2

)n+ c2

(1−√

5

2

)nThen

c1 + c2 = 0

c11 +√

5

2+ c2

1−√

5

2= 1

One has that

c1 =1√5

and c2 = −1√5

So the solution is

f (n) =1√5

(1 +√

5

2

)n− 1√

5

(1−√

5

2

)n

Let the solution be

f (n) = c1

(1 +√

5

2

)n+ c2

(1−√

5

2

)nThen

c1 + c2 = 0

c11 +√

5

2+ c2

1−√

5

2= 1

One has that

c1 =1√5

and c2 = −1√5

So the solution is

f (n) =1√5

(1 +√

5

2

)n− 1√

5

(1−√

5

2

)n

Solution of Inhomogeneous Recurrence, I

Consider the inhomogeneous recurrence

f (n) = f (n − 1) + g(n) (3)

It is easy to see that the solution to (3) is

f (n) = f (0) +n∑

i=1

g(i)

Solution of Inhomogeneous Recurrence, I

f (n) = f (n − 1) + g(n) (3)

f (n) = f (0) +n∑

i=1

g(i)

Solution of Inhomogeneous Recurrence, II

f (n) = g(n)f (n − 1) (4)

f (n) = g(n)g(n − 1) . . . g(1)f (0)

Solution of Inhomogeneous Recurrence, II

f (n) = g(n)f (n − 1) (4)

f (n) = g(n)g(n − 1) . . . g(1)f (0)

Solution of Inhomogeneous Recurrence, III

f (n) = g(n)f (n − 1) + h(n) (5)

where g(n) > 0 for all n > 0.

Let f ′(0) satisfy the following equations:

f (0) = f ′(0)

f (n) = g(n)g(n − 1) . . . g(1)f ′(n) if n > 0

It follows from (5) that

f ′(n) = f ′(n − 1) + h(n)g(n)g(n − 1) . . . g(1)

for n > 1.

Solution of Inhomogeneous Recurrence, III

f (n) = g(n)f (n − 1) + h(n) (5)

where g(n) > 0 for all n > 0.

Let f ′(0) satisfy the following equations:

f (0) = f ′(0)

f (n) = g(n)g(n − 1) . . . g(1)f ′(n) if n > 0

It follows from (5) that

f ′(n) = f ′(n − 1) + h(n)g(n)g(n − 1) . . . g(1)

for n > 1.

Solution of Inhomogeneous Recurrence, IV

f (n) = nf (n − 1) + n! (6)

Let f (n) = n!f ′(n), assuming that f ′(0) = 0. It follows from (6)that

n!f ′(n) = n(n − 1)!f ′(n − 1) + n!

Thereforef ′(n) = f ′(n − 1) + 1

So f ′(n) = n. Hence

f (n) = n!f ′(n) = nn!

Solution of Inhomogeneous Recurrence, IV

f (n) = nf (n − 1) + n! (6)

Let f (n) = n!f ′(n), assuming that f ′(0) = 0. It follows from (6)that

n!f ′(n) = n(n − 1)!f ′(n − 1) + n!

Thereforef ′(n) = f ′(n − 1) + 1

So f ′(n) = n. Hence

f (n) = n!f ′(n) = nn!

Solution of Inhomogeneous Recurrence, V

f (0) = 0

f (n) = 2f (n − 1) + n, if n > 0

Let f (0) = f ′(0) and f (n) = 2nf ′(n). Then

2nf ′(n) = 2(2n−1f ′(n − 1)) + n

which simplifies to

f ′(n) = f ′(n − 1) + n2n

The rest should now be routine.

Solution of Inhomogeneous Recurrence, V

f (0) = 0

f (n) = 2f (n − 1) + n, if n > 0

Let f (0) = f ′(0) and f (n) = 2nf ′(n). Then

2nf ′(n) = 2(2n−1f ′(n − 1)) + n

which simplifies to

f ′(n) = f ′(n − 1) + n2n

The rest should now be routine.

Solution of Divide-and-Conquer Recurrence

The recurrences appeared in the divide-and-conquer algorithm takethe following form

f (n) =

{d if n ≤ n0a1f (n/c1) + . . .+ apf (n/cp) + g(n) if n > n0

We shall discuss the most common techniques for solving suchrecurrences.

Solution of Divide-and-Conquer Recurrence

The recurrences appeared in the divide-and-conquer algorithm takethe following form

f (n) =

{d if n ≤ n0a1f (n/c1) + . . .+ apf (n/cp) + g(n) if n > n0

We shall discuss the most common techniques for solving suchrecurrences.

Expanding the Recurrence

Consider

f (n) =

{d if n = 1af (n/c) + bnx if n > 1

where a, c are nonnegative integers, b, d , x are nonnegativeconstants, and n = ck . Then

f (n) =

{bnx logc(n) + dn

x if a = cx(d + bc

x

a−cx)

nlogc (a) −(

bcx

a−cx)

nx if a 6= cx

Expanding the Recurrence

f (n) = af (n/c) + bnx

= a(af (n/c2) + b(n/c)x) + bnx

= a2f (n/c2) + ab(n/c)x + bnx

...

= ak f (n/ck) + (a/cx)k−1bnx + . . .+ (a/cx)1bnx + bnx

= dalogc n + bnxk−1∑j=0

(a/cx)j

= dnlogc a + bnxk−1∑j=0

(a/cx)j

Substitution

In this method, we guess a solution and try to prove it byappealing to mathematical induction. Unlike the common naturalinduction, here the induction is carried out with some unknownconstants. The constants will be chosen in such a way that theinduction is not affected.

Substitution

Consider

f (n) =

{d if n = 1f (bn/2c) + f (dn/2e) + bn if n > 1

where b, d are nonnegative constants.

When n is a power of 2, we already know that

f (n) = bn log n + dn

We are led to guess that

f (n) = cbn log n + dn

for some c > 0.

Substitution

Consider

f (n) =

{d if n = 1f (bn/2c) + f (dn/2e) + bn if n > 1

where b, d are nonnegative constants.

When n is a power of 2, we already know that

f (n) = bn log n + dn

We are led to guess that

f (n) = cbn log n + dn

for some c > 0.

Substitution

Now

f (n) = f (bn/2c) + f (dn/2e) + bn≤ cbbn/2c logbn/2c+dbn/2c+ cbdn/2e logdn/2e+ddn/2e+ bn≤ cbbn/2c logdn/2e+ cbdn/2e logdn/2e+ dn + bn≤ cbn logdn/2e+ dn + bn≤ cbn log((n + 1)/2) + dn + bn= cbn log(n + 1)− cbn + dn + bn

It is sufficient to require that

cbn log(n + 1)− cbn + dn + bn ≤ cbn log n + dn

That is c log(n + 1)− c + 1 ≤ c log n. We can then fix some valuefor c .

Substitution

Now

f (n) = f (bn/2c) + f (dn/2e) + bn≤ cbbn/2c logbn/2c+dbn/2c+ cbdn/2e logdn/2e+ddn/2e+ bn≤ cbbn/2c logdn/2e+ cbdn/2e logdn/2e+ dn + bn≤ cbn logdn/2e+ dn + bn≤ cbn log((n + 1)/2) + dn + bn= cbn log(n + 1)− cbn + dn + bn

It is sufficient to require that

cbn log(n + 1)− cbn + dn + bn ≤ cbn log n + dn

That is c log(n + 1)− c + 1 ≤ c log n. We can then fix some valuefor c .

Exercise

2.10, 2.21

III. Data Structures

Graph

A graph G = (V ,E ) consists of a set of vertices V = {v1, . . . , vn}and a set of edges E . G is either directed or undirected.

An undirected graph is said to be complete if there is an edgebetween each pair of its vertices. A complete undirected graphwith n vertices is denoted by Kn.

An undirected graph G = (V ,E ) is said to be bipartite if V can bepartitioned into two disjoint subsets X and Y such that each edgehas one end in X and the other end in Y . Let m = |X | andn = |Y |. If there is an edge between each vertex x ∈ X and y ∈ Y ,then it is called a complete bipartite graph, and is denoted by Km,n.

Graph

Two Representations of Graph

A graph G = (V ,E ) can be represented by a boolean matrix M,called the adjacency matrix of G , defined as M[i , j ] = 1 if and onlyif (vi , vj) is an edge of G . The space complexity of such arepresentation is Θ(|V |2).

A graph G = (V ,E ) can be represented by an adjacency list. Thespace complexity of such a representation is Θ(|E |+ |V |).

Two Representations of Graph

A graph G = (V ,E ) can be represented by a boolean matrix M,called the adjacency matrix of G , defined as M[i , j ] = 1 if and onlyif (vi , vj) is an edge of G . The space complexity of such arepresentation is Θ(|V |2).

A graph G = (V ,E ) can be represented by an adjacency list. Thespace complexity of such a representation is Θ(|E |+ |V |).

Planar Graph

A graph G = (V ,E ) is planar if it can be embedded in the planewithout edge crossing. The Euler’s formula for the planar graphssays that v − e + r = 2,

n is the set of vertices,

e is the set of edges, and

r is the set of regions.

Moreover e ≤ 3v − 6.

Tree

A tree is a connected undirected graph that contains no cycles. IfT is a tree with n vertices, then

Any two vertices of T are connected by a unique path;

T has exactly n − 1 edges;The addition of one more edge to T creates a cycle.

When analyzing the time and space complexity in the context oftrees, the number of edges is insignificant.

Rooted Tree

A rooted tree T is a tree with a distinguished vertex r called theroot of T .

parent, child, sibling, leaf, internal vertex, ancestor, descendant,subtree, depth, height.

Rooted Tree

A rooted tree T is a tree with a distinguished vertex r called theroot of T .

parent, child, sibling, leaf, internal vertex, ancestor, descendant,subtree, depth, height.

Binary Tree

A binary tree is full if all its internal vertices have exactly twochildren.

A full binary tree is complete if all its leaves have the same depth.

A tree is almost complete if it is complete except that possibly oneor more leaves that occupy the rightmost positions may be missing.

Binary Tree

Facts about Binary Tree

In a binary tree, the number of vertices at level j is at most 2j .

Let n be the number of vertices in a binary tree T of height k .Then

n ≤k∑

j=0

2j = 2k+1 − 1

If T is almost complete, then

2k ≤ n ≤ 2k+1 − 1

The height of an almost complete tree is blog nc.

In a full binary tree, the number of leaves is equal to the numberof internal vertices plus one.

n ≤k∑

j=0

2j = 2k+1 − 1

2k ≤ n ≤ 2k+1 − 1

n ≤k∑

j=0

2j = 2k+1 − 1

2k ≤ n ≤ 2k+1 − 1

Binary Tree

A (almost) complete tree with n vertices can be representedefficiently by an array A[1..n] that lists its vertices according to thefollowing relationship:

The left and the right children, if any, of a vertex stored inA[j ] are stored in A[2j ] and A[2j + 1] respectively.

The parent of a vertex stored in A[j ] is stored in A[bj/2c].

Binary Search Tree

A binary search tree is a binary tree in which the vertices arelabeled with elements from a linearly ordered set in the followingmanner:

All elements stored in the left subtree of a vertex v are lessthan the element stored in v .

All elements stored in the right subtree of a vertex v aregreater than the element stored in v .

IV. Heaps and the Disjoint Sets Data Structures

Heap

A heap is a data structure that supports two operations:

inserting an element, and

finding the element of maximum value.

Formally, a heap is an almost complete binary tree with verticessatisfying the heap property: If v and p(v) are a vertex and itsparent respectively, then the key of the item stored in p(v) is notless than the key of the item stored in v .

Heap

A heap is a data structure that supports two operations:

inserting an element, and

finding the element of maximum value.

Formally, a heap is an almost complete binary tree with verticessatisfying the heap property: If v and p(v) are a vertex and itsparent respectively, then the key of the item stored in p(v) is notless than the key of the item stored in v .

Operations on Heap

delete-ma[H]: Delete and return an item of maximum key from anonempty heap H.

insert[H, x ]: Insert item x into heap H.

delete[H, i ]: Delete the i-th item from heap H.

makeheap[A]: Transform array A into a heap.

Operations on Heap

Sift-Up

Algorithm 3.1 SiftUpInput: An array H[1..n] and an index i between 1 and n.Output: H[i ] is moved up, if necessary, so that it is not lager thanits parent.

1. done ← false2. if i = 1 then exit3. repeat4. if key(H[i ]) > key(H[bi/2c])5. then interchange H[i ] and H[bi/2c]6. else done ← true7. i ← bi/2c8. until i = 1 or done

Sift-Down

Algorithm 3.2 SiftDownInput: An array H[1..n] and an index i between 1 and n.Output: H[i ] is percolated down, if necessary, so that it is notsmaller than its children.

1. done ← false2. if 2i > n then exit3. repeat4. i ← 2i5. if i + 1 ≤ n and key(H[i + 1]) > key(H[i ])6. then i ← i + 17. if key(H[bi/2c]) < key(H[i ])8. then interchange H[i ] and H[bi/2c]9. else done ← true

10. until 2i > n or done

Heap Insertion

Algorithm 3.3 InsertInput: A heap H[1..n] and a heap element x .Output: A new heap H[1..n + 1] with x being one of its elements.

1. n← n + 12. H[n]← x3. SiftUp(H, n)

The time required to insert one element into a heap of size n isO(log n).

Heap Deletion

Algorithm 3.4 DeleteInput: A nonempty heap H[1..n] and an index i between 1 and n.Output: A new heap H[1..n − 1] after H[i ] is removed.

1. x ← H[i ]2. y ← H[n]3. n← n − 14. if i = n + 1 then exit5. H[i ]← y6. if key(y) ≥ key(x) then SiftUp(H, i)7. else SiftDown(H, i)8. end if

The time required to insert one element into a heap of size n isO(log n).

DeleteMax

Algorithm 3.5 DeleteMaxInput: A heap H[1..n].Output: An element x of maximum key is retutrned and deletedfrom the heap.

1. x ← H[1]2. Delete(H, 1)3. return x

Creating a Heap

Algorithm 3.6 MakeHeapInput: An array A[1..n] of n elements.Output: A[1..n] is transformed into a heap.

1. for i ← bn/2c downto 12. SiftDown(A, i)3. end for

Complexity of MakeHeap

Let T be the almost complete tree corresponding to A[1..n].

The height of T is k = logbnc.The total number of iterations is bounded by

k−1∑i=0

= 20(k) + 21(k − 1) + . . .+ 2k−1(1)

=k∑

i=1

i2k−i

= 2kk∑

i=1

i/2i

≤ 2n

Heapsort

Algorithm 3.7 HeapsortInput: An array A[1..n] of n elements.Output: Array A[1..n] sorted in nondecreasing order.

1. MakeHeap(A)2. for j ← n downto 23. interchange A[1] and A[j ]4. SiftDown(A[1..j − 1], 1)5. end for

Disjoint Sets Data Structures

A set S of n distinct elements, partitioned into disjoint sets.Initially all disjoint sets are singleton sets.

Each set has a distinguished representative.

There are two operations on the distinct sets

Find(x): Find and return the representative of the setcontaining x .

Union(x , y): Replace the two sets containing x and y by theirunion. The representative of the union is either that of x orthat of y , to be determined later.

The goal is to design efficient algorithms for the two operations.

Representations of the Disjoint Sets Data Structures

Tree representation: The representatives are the roots; a childpoints to its parent; roots have null pointers.

Array representation: array representation of the trees. Thefollowing is an array representation. What is its treerepresentation?

1 2 3 4 5 6 7 8 9 10 11

0 3 0 8 2 2 1 0 0 1 1

Representations of the Disjoint Sets Data Structures

Tree representation: The representatives are the roots; a childpoints to its parent; roots have null pointers.

Array representation: array representation of the trees. Thefollowing is an array representation. What is its treerepresentation?

1 2 3 4 5 6 7 8 9 10 11

0 3 0 8 2 2 1 0 0 1 1

Algorithm

Straightforward algorithm

Find(x): Follow the path to the root.

Union(x , y): Let the pointer of root(x) points to root(y).

There is a problem:

A tree may degenerate. Suppose we start with{1, }, {2}, . . . , {n} and perform the following sequence ofactions: Union(1, 2),Union(2, 3), . . . ,Union(n − 1, n) followedby Find(1),Find(2), . . . ,Find(n). The cost of n Findoperations is Θ(n2).

Algorithm

Straightforward algorithm

Find(x): Follow the path to the root.

Union(x , y): Let the pointer of root(x) points to root(y).

There is a problem:

A tree may degenerate. Suppose we start with{1, }, {2}, . . . , {n} and perform the following sequence ofactions: Union(1, 2),Union(2, 3), . . . ,Union(n − 1, n) followedby Find(1),Find(2), . . . ,Find(n). The cost of n Findoperations is Θ(n2).

The Union by Rank Heuristic

Each vertex of a tree is attached with a rank (the height of thevertex). When performing Union(x , y), we do

If rank(x) < rank(y), we make y the parent of x .

If rank(x) > rank(y), we make x the parent of y .

If rank(x) = rank(y), we make y the parent of x and increasethe rank of y by 1.

Fact: The number of vertices in a tree with root x is at least2rank(x).Fact: The time complexity of a sequence of m interspersed Unionand Find operations is O(m log n).

The Union by Rank Heuristic

Each vertex of a tree is attached with a rank (the height of thevertex). When performing Union(x , y), we do

If rank(x) < rank(y), we make y the parent of x .

If rank(x) > rank(y), we make x the parent of y .

If rank(x) = rank(y), we make y the parent of x and increasethe rank of y by 1.

Fact: The number of vertices in a tree with root x is at least2rank(x).Fact: The time complexity of a sequence of m interspersed Unionand Find operations is O(m log n).

Path Compression

In a Find(x) operation, after the root y is found, we traverse thepath from x to y once again and change the parent pointers of allvertices along the path to point directly to y .

Find

Algorithm 3.8 FindInput: A vertex x .Output: root(x), the root of the tree containing x .

1. y ← x2. while p(y) 6= null do y ← p(y) end while3. root ← y4. y ← x5. while p(y) 6= null6. w ← p(y)7. p(y)← root8. y ← w9. end while

10. return root

Union

Algorithm 3.9 UnionInput: Two elements x , y .Output: The union of the two trees containing x and yrespectively.

1. u ← Find(x); v ← Find(y)2. if rank(u) ≤ rank(v) then3. p(u)← v4. if rank(u) = rank(v) then rank(v)← rank(v) + 15. else p(v)← u6. end if

Example

Let S be {1, 2, 3, 4, 5, 6, 7, 8, 9}. Demonstrate the procedure ofperforming the following operations:Union(1, 2),Union(3, 4),Union(5, 6),Union(7, 8),Union(2, 4),Union(8, 9),Union(6, 8),Find(5),Union(4, 8),Find(1).

Part B. Techniques Based on Recursion

This part is concerned with a particular class of algorithm, calledrecursive algorithms.

Recursion is the process of dividing a problem into subproblemsthat are identical in structure to the original problem. Combiningthe solutions of these subproblems gives rise to the solution of theoriginal problem.

This part is concerned with a particular class of algorithm, calledrecursive algorithms.

Recursion is the process of dividing a problem into subproblemsthat are identical in structure to the original problem. Combiningthe solutions of these subproblems gives rise to the solution of theoriginal problem.

V. Induction

Induction is based on the well known proof technique ofmathematical induction.

Given a problem with parameter n, designing an algorithm byinduction is based on the fact that if we know how to solve theproblem when presented with a parameter less than n, called theinduction hypothesis, then our task reduces to extending thatsolution to include those instances with parameter n.

Induction is based on the well known proof technique ofmathematical induction.

Given a problem with parameter n, designing an algorithm byinduction is based on the fact that if we know how to solve theproblem when presented with a parameter less than n, called theinduction hypothesis, then our task reduces to extending thatsolution to include those instances with parameter n.

Selection Sort by Induction

Algorithm 5.1 SelectionSortRecInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.

1. sort(1)

Procedure sort(i)

1. if i < n then2. k ← i3. for j ← i + 1 to n4. if A[j ] < A[k] then k ← j5. end for6. if k 6= i then interchange A[i ] and A[k]7. sort(i + 1)8. end if

Analysis of SelectionSortRec

Let C (n) denote the number of element comparisons performed bysort. Then

C (n) =

{0 if n = 1C (n − 1) + (n − 1) if n ≥ 2

The running time is proportional to the number of elementcomparisons. Therefore

C (n) =n(n − 1)

2= Θ(n2)

Analysis of SelectionSortRec

Let C (n) denote the number of element comparisons performed bysort. Then

C (n) =

{0 if n = 1C (n − 1) + (n − 1) if n ≥ 2

The running time is proportional to the number of elementcomparisons. Therefore

C (n) =n(n − 1)

2= Θ(n2)

Insertion Sort by Induction

Algorithm 5.2 InsertionSortRecInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.

1. sort(n)

Procedure sort(i)

1. if i > 1 then2. x ← A[i ]3. sort(i − 1)3. j ← i − 14. while j > 0 and A[j ] > x5. A[j + 1]← A[j ]6. j ← j − 17. end while8. A[j + 1]← x9. end if

Radix Sort

Let L = {a1, . . . , an} be a list of numbers each consisting ofexactly k digits. For instance the left most column is a list of 8numbers of 4 digits.

7467 6792 9134 9134 12391247 9134 1239 9187 12473275 3275 1247 1239 32756792 4675 7467 1247 46759187 7467 3275 3275 67929134 1247 4675 7467 74674675 9187 9187 4675 91341239 1239 6792 6792 9187

Radix Sort

Algorithm 5.3 RadixSortInput: A linked list of numbers L = {a1, . . . , an} and k , thenumber of digits.Output: L sorted in nondecreasing order.

1. for j ← 1 to k2. Prepare 10 empty lists L0, . . . , L93. while L is not empty4. a← next element in L; Delete a from L5. i ← j-th digit in a; Append a to list Li6. end while7. L← L08. for i ← 1 to 99. L← L, Li

10. end for11. end for12. return L

Radix Sort

The time complexity is Θ(kn).

Integer Exponentiation

Algorithm 5.4 ExpRecInput: A real number x and a nonnegative integer n.Output: xn.

1. power(x , n)

Procedure power(x ,m)

1. if m = 0 then y ← 12. else3. y ← power(x , bm/2c)4. y ← y 25. if m is odd then y ← xy6. end if7. return y

The Number of Multiplications

Let C (x , n) be the number of multiplications performed byExpRec . Then clearly

C (x , 0) = 0

C (x , bn/2c) + 1 ≤ C (x , n) ≤ C (x , bn/2c) + 2

It follows that blog nc ≤ C (x , n) ≤ 2blog nc. So the timecomplexity is Θ(log n).

The Number of Multiplications

Let C (x , n) be the number of multiplications performed byExpRec . Then clearly

C (x , 0) = 0

C (x , bn/2c) + 1 ≤ C (x , n) ≤ C (x , bn/2c) + 2

It follows that blog nc ≤ C (x , n) ≤ 2blog nc. So the timecomplexity is Θ(log n).

Integer Exponentiation

Algorithm 5.5 ExpInput: A real number x and a nonnegative integer n.Output: xn.

1. y ← 12. Let n be dkdk−1 . . . d0 in binary notation.3. for j ← k downto 04. y ← y 25. if dj = 1 then y ← xy6. end for7. return y

Evaluating Polynomials (Horner’s Rule)

Algorithm 5.6 HornerInput: A sequence of n + 2 real number a0, a1, . . . , an and x .Output: Pn(x) = anxn + an−1xn−1 + . . . a1x + a0.

1. p ← an2. for j ← 1 to n3. p ← xp + an−j4. end for5. return p

Horner costs n multiplications and n additions.

Generating Permutations

Algorithm 5.7 Permutations1Input: A positive integer n.Output: All permutations of the numbers 1, 2, . . . , n.

1. for j ← 1 to n2. P[j ]← j3. end for4. perm1(1)

Algorithm 5.7 Permutations1Procedure perm1(m)

1. if m = n then output P[1..n]2. else3. for j ← m to n4. interchange P[j ] and P[m]5. perm1(m + 1)6. interchange P[j ] and P[m]7. comment: At this point P[m..n] = m,m + 1, . . . , n.8. end for9. end if

Analysis of Permutations1

Consider perm1(1).

1 Step 1 is executed n! times. Altogether it takes nn! to output.

2 The number of the iterations of the for loop in Step 3 is

f (n) =

{0 if n = 1nf (n − 1) + n if n ≥ 2

Let h(n) be such that n!h(n) = f (n). Therefore

h(n) = h(n − 1) + 1(n − 1)!

=n−1∑j=1

1

j!<∞∑j=1

1

j!= e − 1

Hence f (n) = n!h(n) = Θ(n!).

Algorithm 5.8 Permutations2Input: A positive integer n.Output: All permutations of the numbers 1, 2, . . . , n.

1. for j ← 1 to n2. P[j ]← 03. end for4. perm2(n)

Algorithm 5.8 Permutations2Procedure perm2(m)

1. if m = 0 then output P[1..n]2. else3. for j ← 1 to n4. if P[j ] = 0 then5. P[j ]← m6. perm2(m − 1)7. P[j ]← 08. end if8. end for9. end if

The number of the iterations of the for loop is

f (m) =

{0 if m = 0mf (m − 1) + n if m ≥ 1

Let m!h(m) = f (m). Then

h(m) = h(m − 1) + n(m − 1)!

= nm−1∑j=1

1

j!< n

∞∑j=1

1

j!= (e − 1)n

Hence f (n) = Θ(nn!).

The number of the iterations of the for loop is

f (m) =

{0 if m = 0mf (m − 1) + n if m ≥ 1

Let m!h(m) = f (m). Then

h(m) = h(m − 1) + n(m − 1)!

= nm−1∑j=1

1

j!< n

∞∑j=1

1

j!= (e − 1)n

Hence f (n) = Θ(nn!).

Finding the Majority Element

Let A[1..n] be a sequence of integers. An integer a in A is calledthe majority of A if it appears more than bn/2c times in A.

Observation: If two different elements in the original sequence areremoved, then the majority in the original sequence remains themajority in the new sequence.

Let A[1..n] be a sequence of integers. An integer a in A is calledthe majority of A if it appears more than bn/2c times in A.

Observation: If two different elements in the original sequence areremoved, then the majority in the original sequence remains themajority in the new sequence.

Algorithm 5.9 MajorityInput: An array A[1..n] of n elements.Output: The majority element if it exists; otherwise none.

1. c ← candidate(1)2. count ← 03. for j ← 1 to n4. if A[j ] = c then count ← count + 15. end for6. if count > bn/2c then return c7. else return none

Procedure candidate(m)

1. j ← m; c ← A[m]; count ← 12. while j < n and count > 03. j ← j + 14. if A[j ] = c then count ← count + 15. else count ← count − 16. end while7. if j = n then return c8. else return candidate(j + 1)

Exercise

5.8, 5.12, 5.14, 5.33

VI. Divide and Conquer

The terminology “Divide and Conquer” has been given to apowerful algorithm design technique that is used to solve a varietyof problems.

In its simplest form, a divide-and-conquer algorithm divides theproblem instance into a number of subinstances (in most cases 2),recursively solves each subinstance separately, and then combinesthe solutions to the subinstances to obtain the solution to theoriginal problem instance.

The terminology “Divide and Conquer” has been given to apowerful algorithm design technique that is used to solve a varietyof problems.

In its simplest form, a divide-and-conquer algorithm divides theproblem instance into a number of subinstances (in most cases 2),recursively solves each subinstance separately, and then combinesthe solutions to the subinstances to obtain the solution to theoriginal problem instance.

A Simple Example

The following algorithm finds the minimum and the maximumelements in the array A[1..n].

1. x ← A[1]; y ← A[1]2. for i ← 2 to n3. if A[i ] < x then x ← A[i ]4. if A[i ] > y then y ← A[j ]5. end for6. return (x , y)

The number of element comparisons is 2n − 2.

A Divide and Conquer Solution

Using the divide-and-conquer approach we may have a solution asfollows:

Algorithm 6.1 MinMaxInput: An array A[1..n] of n integers, where n is a power of 2.Output: (x , y): the minimum and the maximum integers in A.

1. minmax(1, n)

A Divide and Conquer Solution

Procedure minmax(low , high)

1. if high − low = 1 then2. if A[low ] < A[high] then return (A[low ],A[high])3. else return (A[high],A[low ])4. end if5. else6. mid ← b(low + high)/2c7. (x1, y1)← minmax(low ,mid)8. (x2, y2)← minmax(mid + 1, high)9. x ← min{x1, x2}

10. y ← max{y1, y2}11. return (x , y)12. end if

Analysis of MinMax

Assuming n = 2k , the number of comparisons is calculated asfollows:

C (n) =

{1 if n = 22C (n/2) + 2 if n > 2

Hence

C (n) = 2C (n/2) + 2

= 2(2C (n/4) + 2) + 2

= 2k−1C (n/2k−1) + 2k−1 + 2k−2 + . . .+ 2

= n/2 + 2k − 2= 3n/2− 2

Analysis of MinMax

Assuming n = 2k , the number of comparisons is calculated asfollows:

C (n) =

{1 if n = 22C (n/2) + 2 if n > 2

Hence

C (n) = 2C (n/2) + 2

= 2(2C (n/4) + 2) + 2

= 2k−1C (n/2k−1) + 2k−1 + 2k−2 + . . .+ 2

= n/2 + 2k − 2= 3n/2− 2

Binary Search via Recursion

Algorithm 6.2 BinarySearchRecInput: An array A[1..n] of n elements sorted in nondecreasingorder and an element x .Output: j if x = A[j ] for some j ∈ {1..n}, and 0 otherwise.

1. binarysearch(1, n)

Procedure binarysearch(low , high)

1. if low > high then return 02. else3. mid ← b(low + high)/2c4. if x = A[mid ] then return mid5. else if x < A[mid ] then return

binarysearch(low ,mid − 1)6. else return binarysearch(mid + 1, high)7. end if

Analysis of BinarySearchRec

The number of comparisons is calculated as follows:

C (n) ≤{

1 if n = 11 + C (bn/2c) if n ≥ 2

Let 2k−1 ≤ n < 2k for some k . Then

C (n) ≤ (k − 1) + C (bn/2k−1c) = k = blog nc+ 1

Analysis of BinarySearchRec

The number of comparisons is calculated as follows:

C (n) ≤{

1 if n = 11 + C (bn/2c) if n ≥ 2

Let 2k−1 ≤ n < 2k for some k . Then

C (n) ≤ (k − 1) + C (bn/2k−1c) = k = blog nc+ 1

Mergesort

Algorithm 6.3 MergesortInput: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.

1. mergesort(A, 1, n)

Procedure mergesort(A, low , high)

1. if low < high then2. mid ← b(low + high)/2c3. mergesort(A, low ,mid)4. mergesort(A,mid + 1, high)5. MERGE (A, low ,mid , high)6. end if

Analysis of Mergesort

The number of comparisons needed to merge two sorted subarraysis between n/2 and n − 1.

The minimum number of comparisons:

C (n) ≤{

0 if n = 12C (n/2) + n/2 if n ≥ 2

The maximum number of comparisons:

C (n) ≤{

0 if n = 12C (n/2) + n − 1 if n ≥ 2

C (n) ≤{

0 if n = 12C (n/2) + n/2 if n ≥ 2

C (n) ≤{

0 if n = 12C (n/2) + n − 1 if n ≥ 2

C (n) ≤{

0 if n = 12C (n/2) + n/2 if n ≥ 2

C (n) ≤{

0 if n = 12C (n/2) + n − 1 if n ≥ 2

The Divide and Conquer Paradigm

Divide Step: In this step of the algorithm, the input is partitionedinto p ≥ 1 parts, each of which is strictly less than n, the size ofthe original instance.

Conquer Step: This step consists of performing p recursive call(s)if the problem size is greater than some predefined threshold n0.

Combine Step: In this step, the solution to the p recursive call(s)are combined to obtain the desired output.

The Divide and Conquer Algorithm

If the size of the instance I is “small”, then solve the problemusing a straightforward method and return the answer. Otherwise,continue to the next step.

Divide the instance I into p subinstances I1, . . . , Ip ofapproximately the same size.

Recursively call the algorithm on each subinstance Ij , 1 ≤ j ≤ p, toobtain p partial solutions.

Combine the results of the p partial solutions to obtain the solutionto the original instance I . Return the solution of instance I .

Selection: Finding the Median and the k-th Element

The median of a sequence of n numbers A[1..n] is the middleelement, or the dn/2e-th smallest element.

A straightforward method of finding the median, or the k-thelement, is to sort all elements and pick the k-th element. Thistakes Ω(n log n) times.

Selection: Finding the Median and the k-th Element

The median of a sequence of n numbers A[1..n] is the middleelement, or the dn/2e-th smallest element.

A straightforward method of finding the median, or the k-thelement, is to sort all elements and pick the k-th element. Thistakes Ω(n log n) times.

The Idea of Selection

1. The input array A[1..n] is broken into two sets A1 and A3.

2. In order for the Conquer step to be possible, A1 and A3 mustsatisfy the condition ∀x ∈ A1.∀y ∈ A3.x < y .

3. To guarantee the good performance of the algorithm, both A1and A3 should have sufficient number of elements.

Selection

Algorithm 6.4 SelectInput: An array A[1..n] of n elements and an integer k , 1 ≤ k ≤ n.Output: The k-th smallest element in A.

1. select(A, 1, n, k)

Selection

Procedure select(A, low , high, k)

1. p ← high − low + 12. if p < 44 then sort A and return (A[k])3. Let q = bp/5c. Divide A into q groups of 5 elements each. If

5 does not divide p, then discard the remaining elements.4. Sort each of the q groups individually and extract its median.

Let the set of the medians be M.5. mm← select(M, 1, q, dq/2e)6. Partition A[low ..high] into three arrays:

A1 = {a | a < mm}; A2 = {a | a = mm}A3 = {a | a > mm}

7. case|A1| ≥ k : return select(A1, 1, |A1|, k)|A1| < k and |A1|+ |A2| ≥ k : return mm|A1|+ |A2| < k : return select(A3, 1, |A3|, k − |A1| − |A2|)

8. end case

Selection

Steps 1-5 make sure that A1 and A3 are not too small.

Analysis of Select

Let A1 be defined by {a | a ≥ mm}. Clearly |A1|+ |A1| = n.Observe that

|A1| ≥ 3dbn/5c/2e ≥3

2bn/5c

Hence

|A1| ≤ n −3

2bn/5c ≤ n − 3

2(

n − 45

) = 0.7n + 1.2

Similarly |A3| ≤ 0.7n + 1.2.

Analysis of Select

Let A1 be defined by {a | a ≥ mm}. Clearly |A1|+ |A1| = n.Observe that

|A1| ≥ 3dbn/5c/2e ≥3

2bn/5c

Hence

|A1| ≤ n −3

2bn/5c ≤ n − 3

2(

n − 45

) = 0.7n + 1.2

Similarly |A3| ≤ 0.7n + 1.2.

Analysis of Select

Steps 1 and 2 cost Θ(1) time each.

Step 3 costs Θ(n) time.

Step 4 costs Θ(n) time.

Step 5 costs T (bn/5c) time.Step 6 costs Θ(n) time.

Step 7 costs at most T (0.7n + 1.2) time. Assume that0.7n + 1.2 ≤ b0.75nc, which is subsumed by0.7n + 1.2 ≤ 0.75n + 1, which is in turn subsumed by n ≥ 44.It follows that

T (n) ≤{

c if n < 44T (bn/5c) + T (b3n/4c) + cn if n ≥ 44

for some c . Hence T (n) = Θ(n).

Example

For the sake of illustration, let’s temporarily change the thresholdfrom 44 to, say 6. Suppose we want to find the median of thefollowing 25 numbers:

8, 33, 17, 51, 57, 49, 35, 11, 25, 37, 14, 3, 2, 13, 52, 12, 6, 29, 32,54, 5, 16, 22, 23, 7

Selection

What if each group consists of 7 elements, or 9 elements?

The Quick Sort Algorithm

Quick sort is a popular and efficient sorting algorithm. Oneadvantage of the algorithm is that it does not need additionalstorage.

The Quick Sort Algorithm

We say that an element A[j ] is in its proper position or correctposition if it is neither smaller than the elements in A[1..j − 1] norlarger than the elements in A[j + 1..n].

An Auxiliary Algorithm

Algorithm 6.5 SplitInput: An array of elements A[low ..high].Output: A with its elements rearranged and w the new position ofthe splitting element A[low ].

1. i ← low2. x ← A[low ]3. for j ← low + 1 to high4. if A[j ] ≤ x then5. i ← i + 16. if i 6= j then interchange A[i ] and A[j ]7. end if8. end for9. interchange A[low ] and A[i ]10. w ← i11. return A and w

Time Complexity of Split

The number of comparisons is precisely n − 1.

Quicksort

Algorithm 6.6 QuickSortInput: An array A[1..n] of n elements.Output: The elements in A sorted in nondecreasing order.

1. QuickSort(A, 1, n)

Procedure QuickSort(A, low , high)

1. if low < high then2. Split(A[low ..high],w)3. QuickSort(A, low ,w − 1)4. QuickSort(A,w + 1, high)5. end if

Analysis of QuickSort

The worst case complexity:

f (n) = (n − 1) + (n − 2) + . . .+ 1 + 0 = Θ(n2)

If we always select the median as the pivot element, then roughly

C (n) ≤{

0 if n = 12C (n/2) + Θ(n) if n ≥ 2

whose solution is C (n) = Θ(n log n).

Analysis of QuickSort

The worst case complexity:

f (n) = (n − 1) + (n − 2) + . . .+ 1 + 0 = Θ(n2)

If we always select the median as the pivot element, then roughly

C (n) ≤{

0 if n = 12C (n/2) + Θ(n) if n ≥ 2

whose solution is C (n) = Θ(n log n).

Practical Performance of QuickSort

If the input is ordered randomly, QuickSort is fast

Otherwise we could choose the pivot randomly.

Practical Performance of QuickSort

If the input is ordered randomly, QuickSort is fast

Otherwise we could choose the pivot randomly.

Average Case Behavior of QuickSort

Assume that the input numbers are 1, 2, . . . , n and that eachnumber is equally likely to be the first of the input.

Now

C (n) = (n − 1) + 1n

n∑w=1

(C (w − 1) + C (n − w))

= (n − 1) + 2n

n∑w=1

C (w − 1)

Then

nC (n) = n(n − 1) + 2n∑

w=1

C (w − 1)

Replacing n by n − 1

(n − 1)C (n − 1) = (n − 1)(n − 2) + 2n−1∑w=1

C (w − 1)

Subtracting and rearranging

C (n)

n + 1=

C (n − 1)n

+2(n − 1)n(n + 1)

Let

D(n) =C (n)

n + 1

Then

D(1) = 0

D(n) = D(n − 1) + 2(n − 1)n(n + 1)

Clearly

D(n) = 2n∑

j=1

j − 1j(j + 1)

2n∑

j=1

j − 1j(j + 1)

= 2n∑

j=1

2

(j + 1)− 2

n∑j=1

1

j

= 4n+1∑j=2

1

j− 2

n∑j=1

1

j

= 2n∑

j=1

1

j− 4n

n + 1

= 2 ln n −Θ(1)

=2

log elog n −Θ(1)

≈ 1.44 log n

So C (n) = (n + 1)D(n) ≈ 1.44n log n.

Multiplication of Large Integers

Assume that u, v are two n digits integers, where for simplicity it isassumed that n is a power of 2. Then

u = w2n/2 + x

v = y2n/2 + z

The product of u and v can be computed as

uv = wy2n + (wz + xy)2n/2 + xz

The time complexity is

T (n) =

{d if n = 14T (n/2) + bn if n > 1

So T (n) = Θ(n2).

u = w2n/2 + x

v = y2n/2 + z

T (n) =

{d if n = 14T (n/2) + bn if n > 1

So T (n) = Θ(n2).

u = w2n/2 + x

v = y2n/2 + z

T (n) =

{d if n = 14T (n/2) + bn if n > 1

So T (n) = Θ(n2).

But observe that

wz + xy = (w + x)(y + z)− wy − xz

Namely

uv = wy2n + ((w + x)(y + z)− wy − xz)2n/2 + xz

T (n) =

{d if n = 13T (n/2) + bn if n > 1

So T (n) = Θ(nlog 3) = Θ(n1.59).

But observe that

wz + xy = (w + x)(y + z)− wy − xz

Namely

T (n) =

{d if n = 13T (n/2) + bn if n > 1

So T (n) = Θ(nlog 3) = Θ(n1.59).

But observe that

wz + xy = (w + x)(y + z)− wy − xz

Namely

T (n) =

{d if n = 13T (n/2) + bn if n > 1

So T (n) = Θ(nlog 3) = Θ(n1.59).

Matrix Multiplication: The Traditional Algorithm

Matrix multiplication: a11 . . . a1n...an1 . . . ann

b11 . . . b1n...

bn1 . . . bnn

= c11 . . . c1n...

cn1 . . . cnn

where

cij =n∑

k=1

aikbkj

Clearly the complexity of multiplying two n × n matrices isT (n) = Θ(n3).

Matrix Multiplication: The Recursive Version

Assume that n = 2k . If n ≥ 2 then the matrix can be partitionedinto four matrices of dimensions n/2× n/2(

A11 A12A21 A22

)(B11 B12B21 B22

)=

(C11 C12C21 C22

)where(

C11 C12C21 C22

)=

(A11B11 + A12B21 A11B12 + A12B22A21B11 + A22B21 A21B12 + A22B22

)

Time Complexity:

T (n) =

{m if n = 18T (n/2) + 4(n/2)2a if n ≥ 2

Therefore T (n) = Θ(n3).

Time Complexity:

T (n) =

{m if n = 18T (n/2) + 4(n/2)2a if n ≥ 2

Therefore T (n) = Θ(n3).

Matrix Multiplication: Strassen’s Algorithm

Let

D1 = (A11 + A22)(B11 + B22)

D2 = (A21 + A22)B11

D3 = A11(B12 − B22)D4 = A22(B21 − B11)D5 = (A11 + A22)B22

D6 = (A21 − A11)(B11 + B12)D7 = (A12 − A22)(B21 + B22)

Then(C11 C12C21 C22

)=

(D1 + D4 − D5 + D7 D3 + D5D2 + D4 D1 + D3 − D2 + D6

)

Time Complexity:

T (n) =

{m if n = 17T (n/2) + 18(n/2)2a if n ≥ 2

Therefore T (n) = Θ(nlog 7) = Θ(n2.81).

Time Complexity:

T (n) =

{m if n = 17T (n/2) + 18(n/2)2a if n ≥ 2

Therefore T (n) = Θ(nlog 7) = Θ(n2.81).

The Closest Pair Problem

Given a set S of n points in the plane, find the pair p1 = (x1, y1)and p2 = (x2, y2) such that the Euclidean distance between p1 andp2 is minimal among all the pairs of points.

The Divide Step

1. Divide the points of S into two sets Sl , Sr .

2. This is achieved by sorting the points of S according to theirx-coordinates, and then divide S by the median x0.

The Conquer Step

1. Apply the algorithm to Sl , Sr .

2. Let δl , δr be the values returned by the recursive calls.

The Combine Step

1. Let δ be min{δl , δr}.

2. Find the distance δ′ between δl and δr .

3. Return min{δ, δ′}.

The Combine Step: How to Calculate δ′?

1. Sort the elements of S according to their y -coordinates.

2. Construct T whose elements are within δ-distance to thevertical line passing x0.

3. Find the closest pair in T using a gliding method.

ClosestPair

Algorithm 6.7 ClosestPairInput: A set of n points in the plane.Output: The minimum separation realized by two points in S .

1. Sort the points in S in nondecreasing order of theirx-coordinates.

algorithm design and analysis - sjtuyuxi/teaching/lectures/theory... · 2014. 12. 29. · algorithm...

Documents