analysis of algorithms an algorithm is a step-by-step procedure for solving a problem in a finite...

Analysis of AlgorithmsAn algorithm is a step-by-step procedure for

solving a problem in a finite amount of time.

Input OutputMerge Sort

Quick Sort

• Which one is better?• What are the criteria?

How to evaluate algorithms?

Problem: Find Maximum from Array

2

Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A

currentMax A[0]for i 1 to n 1 do

if A[i] currentMax thencurrentMax A[i]

return currentMax

• Find max from array A: A[] = [5 8 2 8 9 1 3 7 4 6 …]

How good is algorithm arrayMax()?

3

Problem: Prefix AveragesNumbers X = 8 4 3 5 9 3 7 6 5 4 2 1 …Prefix averages A = 8 6 5 5 …

Which algorithm is better?

Analysis of Algorithms

• Data structure– A systematic way of organizing and accessing data

• Algorithm– A step-by-step procedure for performing some task in

a finite amount of time

• Running time of algorithms and data structure operations is a natural measure of “goodness”

4




return currentMax

• Find max: A[0] … A[n-1]

• How good is algorithm arrayMax()?

A[] = [5 8 2 8 9 1 3 …]

Summations and Geometric Sums

5

n

i

nni

n

i

nni

d

i

dd

ii

n

i

b

ai

a

aaaaa

nananaananf

nnni

bfafafafif

0

1

0

12

0

2210

1

122...84212 : Example

1

1...1:Sum Geometric

...)(:Polynomial

2/)1(...21 : Example

)(...)2()1()()(:Summation

Running Time •Most algorithms transform input objects into output objects.• The running time of an algorithm typically grows with the input size.• Average case time is often difficult to determine.•We focus on the worst case running time.– Easier to analyze– Crucial to applications such as games, finance

and robotics

0

20

40

60

80

100

120

Runnin

g T

ime

1000 2000 3000 4000

Input Size

best caseaverage caseworst case

6

Experimental Studies• Write a program implementing the algorithm• Run the program with inputs of varying size and

composition• Use a method like System.currentTimeMillis( ) to get an accurate

measure of the actual running time• Plot the results

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 50 100

Input Size

Tim

e (

ms)

7




return currentMax

Limitations of Experiments• It is necessary to implement the algorithm, which

may be difficult• Results may not be indicative of the running time on

other inputs not included in the experiment. • In order to compare two algorithms, the same

hardware and software environments must be used

8




return currentMax

Theoretical Analysis• Uses a high-level description of the algorithm

instead of an implementation• Takes into account all possible inputs• Allows us to evaluate the relative efficiency of

any two algorithms in a way that is independent from the hardware/software environment

• Characterizes running time as a function of the input size, n.

9

Pseudocode

• High-level description of an algorithm• More structured than English

prose• Less detailed than a program• Preferred notation for

describing algorithms• Hides program design issues

10




return currentMax

Example: find max element of an array

A[] = [5 8 2 8 9 1 3 …]

• Consider all possible inputs• Independent of

hardware/software• Characterizes running time as

a function of the input size, n.

Pseudocode Details

• Control flow– if … then … [else …]

– while … do …

– repeat … until …

– for … do …

– Indentation replaces braces

• Method declarationAlgorithm method (arg [, arg…])

Input …

Output …

• Method callvar.method (arg [, arg…])

• Return valuereturn expression

• Expressions Assignment

(like in Java) Equality testing

(like in Java)n2 Superscripts and other

mathematical formatting allowed

11

Primitive Operations

• Basic computations performed by an algorithm

• Identifiable in pseudocode• Largely independent from

the programming language• Exact definition not

important (we will see why later)

• Running time– Assumed to take a constant amount

of time in the RAM model

Examples:• Evaluating an expression• Assigning a value to a

variable• Indexing into an array• Calling a method• Returning from a method

12

currentMax 0currentMax A[0]return result

Counting Primitive Operations

By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size

13

Estimating Running Time• Algorithm arrayMax executes 8n 2 primitive

operations in the worst case. Define:a = Time taken by the fastest primitive operationb = Time taken by the slowest primitive operation

• Let T(n) be worst-case time of arrayMax. Thena (8n 2) T(n) b(8n 2)

• Hence, the running time T(n) is bounded by two linear functions

14

Algorithm arrayMax(A, n)

# operations

currentMax A[0] 2for i 1 to n 1 do 2n

if A[i] currentMax then 2(n 1)currentMax A[i] 2(n 1)

{ increment counter i } 2(n 1)return currentMax 1

Total 8n 2

Growth Rate of Running Time• Changing the hardware/software environment – Affects T(n) by a constant factor, but– Does not alter the growth rate of T(n)

• The linear growth rate of the running time T(n) is an intrinsic property of algorithm arrayMax

15

a(8n 2) T(n) b(8n 2)

Nested Loops and Quadratic Function

• Appears in nested loops, where the inner loop performs a linear number of operations and the outer loop is performed a linear number of times; i.e., nn = n2

• Also arise in the context of nested loops where the first iteration of a loop uses one operation, the second uses two operations, the third uses three operations, and so on.i.e., 1+2+3+…+(n-2)+(n-1)+n = n(n+1)/2

16

Algorithm arrayMax(A, n)Input array A of n × n integersOutput maximum element of A

currentMax A[0, 0]for i 0 to n 1 do

for j 0 to n – 1 do if A[i, j] currentMax then

currentMax A[i, j]return currentMax

3735

496

18

2

7

39

568

A

i

j

for i 0 to n 1 do for j 0 to i do …

Lower-Order Terms and Constant Factors

• The growth rate is not affected by– lower-order terms or– constant factors

• ExamplesQuadratic function:

– 105n2 108n

– 105n2

Linear function:

– 102n 105

– 102n

Different linear function:

– 3n

– 2n+10

– n

17

Big-Oh Notation• Given functions f(n) and

g(n), we say that f(n) is O(g(n)) if there are positive constantsc and n0 such that

f(n) cg(n) for n n0

• Example: 2n 10 is O(n)

2n 10 cn

(c 2) n 10

n 10(c 2)

Pick c 3 and n0 10

18

2n 10 n for n 10

Big-Oh Example

• Example: the function n2 is not O(n)– n2 cn

– n c– The above inequality

cannot be satisfied since c must be a constant

19

f(n) is O(g(n)):f(n) cg(n) for n

n0

20More Big-Oh Examples 7n-2

7n-2 is O(n)need c > 0 and n0 1 such that 7n-2 c•n for n n0

this is true for c = 7 and n0 = 1

3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 c•n3 for n

n0

this is true for c = 4 and n0 = 213 log n + 53 log n + 5 is O(log n)need c > 0 and n0 1 such that 3 log n + 5 c•log n for

n n0

this is true for c = 8 and n0 = 2not affected bylower-order terms or constant

factors

f(n) is O(g(n)):f(n) cg(n) for n

n0

Big-Oh Rules• Not affected by lower-order terms and

constant factor• If f(n) is a polynomial of degree d, then f(n) is

O(nd), i.e.,1. Drop lower-order terms2. Drop constant factors

• Use the smallest possible class of functions– Say “2n is O(n)” instead of “2n is O(n2)”

• Use the simplest expression of the class– Say “3n 5 is O(n)” instead of “3n 5 is O(3n)”

21

Big-Oh and Growth Rate• The big-Oh notation gives an upper bound on

the growth rate of a function• “f(n) is O(g(n))” means “the growth rate of f(n) is

no more than the growth rate of g(n)”• We can use the big-Oh notation to rank

functions according to their growth rate

22

f(n) is O(g(n)) g(n) is O(f(n))

g(n) grows more

Yes No

f(n) grows more

No Yes

Same growth Yes Yes

Asymptotic Algorithm Analysis• The asymptotic analysis of an algorithm

determines the running time in big-Oh notation• To perform the asymptotic analysis

1. We find the worst-case number of primitive operations executed as a function of the input size

2. We express this function with big-Oh notation

Example of Algorithm arrayMax():1. We determine that algorithm arrayMax executes at most 8n 2

primitive operations2. We say that algorithm arrayMax “runs in O(n) time”

Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard them when counting primitive operations

23

Example: Computing Prefix Averages

• The i-th prefix average of an array X is average of the first (i 1) elements of X:

A[i] X[0] X[1] … X[i])/(i+1)

• Computing the array A of prefix averages of another array X has applications to financial analysis

24

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7

X

A

X = 21 23 25 31 20

…

A = 21 22 23 25 24

…

25

Algorithm1: Prefix Averages (Quadratic)

The following algorithm computes prefix averages in quadratic time by applying the definition

X = 8 4 3 5 9 3 7 6 5 4 2 1 … A = 8 6 5 5 …

• Ignor constant factors and lower-order terms• The running time of prefixAverages1 is O(1 2 …n)

O(1 2 …n) = O(n(n 1) 2) = O(n2)

• Thus, algorithm prefixAverages1 runs in O(n2) time

26

Algorithm2: Prefix Averages (Linear)The following algorithm computes prefix averages in linear time by keeping a running sum

X = 8 4 3 5 9 3 7 6 5 4 2 1 … A = 8 6 5 5

…Algorithm prefixAverages2(X, n)

Input array X of n integersOutput array A of prefix averages of X #operationsA new array of n integers ns 0 1for i 0 to n 1 do n

s s X[i] nA[i] s (i 1) n

return A 1Algorithm prefixAverages2 runs in O(n) time

Seven Used Functions• Polynomial functions: f(n) = a0+a1n+a2n2+a3n3+…+adnd

• Exponential function: an

27

29Relatives of Big-OhBig-Omega f(n) is (g(n)) if there is a constant c > 0

and an integer constant n0 1 such that

f(n) cg(n) for n n0

3nlogn+2n is (nlogn) Since: 3nlogn+2n 3nlogn, for n 2

Big-Theta f(n) is (g(n)) if there are constants c’ > 0 and

c’’ > 0 and an integer constant n0 1 such that

c’g(n) f(n) c’’g(n) for n n0

3nlogn+4n+5logn is (nlogn) Since: 3nlogn 3nlogn+4n+5logn (3+4+5)nlogn for n

2

f(n) is O(g(n)):f(n) cg(n) for n n0

f(n)

cg(n)

Intuition for Asymptotic Notation

30

Big-Oh f(n) is O(g(n)) if f(n) is asymptotically less than or equal

to g(n)

Big-Omega f(n) is (g(n)) if f(n) is asymptotically greater than or

equal to g(n)

Big-Theta f(n) is (g(n)) if f(n) is asymptotically equal to g(n)

f(n) is O(g(n)) f(n) is (g(n))

31Example Uses of the Relatives of Big-Oh

f(n) is (g(n)) if it is (n2) and O(n2). We have already seen the former, for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0

Let c = 5 and n0 = 1

5n2 is (n2)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0

let c = 1 and n0 = 1

5n2 is (n)



5n2 is (n2)

32Example Uses of the Relatives of Big-Oh

3nlogn 3nlogn + 4n + 5 (3+4+5) nlogn for n2 3nlogn+4n+5 is (nlogn)



5n2 is (n)



3nlogn+2n is (nlogn)

5n2+3nlogn+2n+5 is O(n2)5n2+3nlogn+2n+5 (5+3+2+5) n2 = cn2 for c =15, n0 = 2

A Recursive Algorithm for Computing Powers

• Power function p(x,n) = xn

– A linear recursion definition:

– Employ a squaring technique

• 27=2(26)=2(2(25))=…=128• 27=21+(6/2)2 =2(26/2)2=2(23)2=2((2(2)2)2)=128• What is running-time complexity of each

method?

33

otherwise )1,(

0 if 1),(

nxpx

nnxp

even is 0 if )2/,(odd is 0 if )2/)1(,(

0 if 1

),(2

2

nnxpnnxpx

n

nxp

A Recursive Algorithm for Computing Powers

Algorithm Power(x,n)Input: A number x and integer n 0Output: The value xn

if n = 0 then return 1if n is odd then y Power(x, (n-1)/2) return x • y • yelse y Power(x,n/2) return y • y

34

Math you need to Review

Logarithms and Exponents• properties of logarithms:

logb(xy) = logbx + logbylogb (x/y) = logbx - logbylogbxa = alogbxlogba = logxa/logxb

• properties of exponentials:a(b+c) = aba c

abc = (ab)c

ab /ac = a(b-c)

b = a loga

b

bc = a c*loga

b

35

SummationsLogarithms and ExponentsProof techniquesBasic probability

Simple Justification Techniques

• By example• The “Contra” attack– Contrapositive– Contradiction

• Induction and loop invariants

36

By example

Claim:“Every element x in a set S has property P”

• To falsify the claim, we need to only produce a counterexample

• Every number of the form 2i-1 is a prime, when i is an integer greater than 1.– A counterexample: 24-1 = 15

37

Contrapositive• “If p is true, then q is true” is equivalent to

“If q is not true, then p is not true”• Let a and b be integers.

If ab is even, then a is even or b is even.Contrapositive: …?Justify by contrapositive: Suppose a = 2i+1 and b = 2j+1, for some integer i and j. Then ab = 4ij+2i+2j+1 = 2(2ij+i+j)+1; hence, ab is odd.

38

Contradition• To establish a statement q is true

First supposing q is false and then showing that this assumption leads to a contradiction

• Let a and b be integers. If ab is odd, then a is odd and b is odd.

Proof by contraditionThe opposite: suppose a is even or b is even

Assume that a is even. Then a = 2i for some integer i. Hence, ab = (2i)b = 2(ib), that is, ab is even. Contradiction: ab cannot simultaneously be odd and even. Therefore, a is odd and b is odd.

39

Induction• To say some statement q(n) is true for all n1 by

showing “if q(i) is true for i<n, then q(n) is true.”• Consider the Fibonacci function F(n), where we

define F(1)=1, F(2)=2, and F(n)=F(n-1)+F(n-2) for n>2. We claim that F(n)<2n.

Base cases: (n 2). F(1) = 1<2=21 and F(2) = 2<4=22.Induction step: (n>2). Suppose our claim is true for n’<n. Consider F(n). Since n>2, F(n) =F(n-1)+F(n-2). Moreover, since n-1<n and n-2<n, apply the inductive assumption to imply that F(n)<2n-1+2n-2, since 2n-1+2n-2<2n-1+2n-1=2 ． 2n-

1=2n.

analysis of algorithms an algorithm is a step-by-step procedure for solving a problem in a finite...

Documents