Download - Algorithm Complexity lalit bansal
-
7/29/2019 Algorithm Complexity lalit bansal
1/52
Chapter 2
Fundamentals of the Analysis
of Algorithm Efficiency
Copyright 2007 Pearson Addison-Wesley. All rights reserved.
-
7/29/2019 Algorithm Complexity lalit bansal
2/52
Analysis of algorithms
Issues: correctness
time efficiency
space efficiency
optimality
Approaches:
theoretical analysis
empirical analysis
-
7/29/2019 Algorithm Complexity lalit bansal
3/52
Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the number ofrepetitions of the basic operationas a function ofinput size
Basic operation: the operation that contributes most
towards the running time of the algorithm
T(n) copC(n)
running time execution time
for basic operation
Number of times
basic operation is
executed
input size
-
7/29/2019 Algorithm Complexity lalit bansal
4/52
Input size and basic operation examples
Problem I nput size measure Basic operation
Searching for key in a
list ofnitems
Number of lists items,
i.e. nKey comparison
Multiplication of two
matrices
Matrix dimensions or
total number of elements
Multiplication of two
numbers
Checking primality of
a given integer nnsize = number of digits
(in binary representation) Division
Typical graph problem #vertices and/or edgesVisiting a vertex or
traversing an edge
-
7/29/2019 Algorithm Complexity lalit bansal
5/52
Empirical analysis of time efficiency
Select a specific (typical) sample of inputs
Use physical unit of time (e.g., milliseconds)
or
Count actual number of basic operations executions
Analyze the empirical data
-
7/29/2019 Algorithm Complexity lalit bansal
6/52
Best-case, average-case, worst-case
For some algorithms efficiency depends on form of input:
Worst case: Cworst(n)maximum over inputs of size n
Best case: Cbest(n) minimum over inputs of size n
Average case: Cavg(n)average over inputs of size n
Number of times the basic operation will be executed on typical input
NOT the average of worst and best case
Expected number of basic operations considered as a random variable
under some assumption about the probability distribution of all
possible inputs
-
7/29/2019 Algorithm Complexity lalit bansal
7/52
Example: Sequential search
Worst case
Best case
Average case
-
7/29/2019 Algorithm Complexity lalit bansal
8/52
Types of formulas for basic operations count
Exact formulae.g., C(n) = n(n-1)/2
Formula indicating order of growth with specific
multiplicative constante.g., C(n) 0.5 n2
Formula indicating order of growth with unknown
multiplicative constant
e.g., C(n) cn2
-
7/29/2019 Algorithm Complexity lalit bansal
9/52
The Big-Oh Notation:
given functions f(n) and g(n), we
say that f(n) is O(g(n) ) if and only
if there are positive constants c
and n0such that f(n) c g(n)for
n n0
-
7/29/2019 Algorithm Complexity lalit bansal
10/52
Example
(n) =2n +6For functionsf(n)and g(n) (to the
right) there are
positive constants c
and n0
such that:
f(n) c g(n)
forn n0
conclusion:
2n+6 is O(n).
-
7/29/2019 Algorithm Complexity lalit bansal
11/52
Another Example
On the other hand
n2 is not O(n) because
there is no c and n0 suchthat: n2 c n fornn0
(As the graph to the right
illustrates, no matter howlarge a c is chosen there is
an n big enough that
N2 > c n ) .
-
7/29/2019 Algorithm Complexity lalit bansal
12/52
EXAMPLE
Consider 1/3 n2 5n
The dominating term is n
2
Therefore it should be of O(n2)
Given a positive constant c , a positive integern0 to be found such that
1/3 n2 - 5 n c n2
-
7/29/2019 Algorithm Complexity lalit bansal
13/52
Dividing the inequality throughout by n2
1/3 - 5/n c
Therefore if n0 1 , choosing c 1/3 theinequality will never be violated
Hence the expression is indeed of O(n2)
-
7/29/2019 Algorithm Complexity lalit bansal
14/52
Relatives of the Big-Oh
(f(n)): Big Omega--asymptoticlowerbound
(f(n)): Big Theta--asymptotic t ightbound
-
7/29/2019 Algorithm Complexity lalit bansal
15/52
Categories of algorithm efficiency
Efficiency
Constant
Big O
O(1)
Logarithmic O(log n)
Linear O(n)
Linear logarithmic O(n log n)
Quadratic O(n2)
Polynomial O(nk)
Exponential O(cn)
Factorial O(n!)
-
7/29/2019 Algorithm Complexity lalit bansal
16/52
-notation
(g(n)) = {f(n): +veconstantsc
1
, c2
, and n0
such
that 0 c1g(n) f(n)c2g(n),
nn0}
For functiong(n), (g(n)) is
given by:
g(n) is an asymptotically tight boundfor f(n).
Intuitively: Set of all functions that
have the samerate of growth asg(n).
-
7/29/2019 Algorithm Complexity lalit bansal
17/52
O -notat ion
O(g(n)) = {f(n): +veconstantsc and n0 such that 0
f(n)cg(n), n n0}
For functiong(n), O(g(n)) is
given by:
g(n) is an asymptotic upper boundfor f(n).
Intuitively: Set of all functions
whose rate of growthis the same as
or lower than that ofg(n).
f(n) = (g(n)) f(n) = O(g(n)).(g(n)) O(g(n)).
-
7/29/2019 Algorithm Complexity lalit bansal
18/52
Relat ions Between, O,
-
7/29/2019 Algorithm Complexity lalit bansal
19/52
Practical Complexity
0
250
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n
-
7/29/2019 Algorithm Complexity lalit bansal
20/52
Problem size
10 102 103 104
log2n 3.3 6.6 10 13.3
n 10 102 103 104
nlog2n 0.33x10 0.7x103 104 1.3x105
n2 102 104 106 108
2n 1024 1.3x103 >10100 >10100
n! 3x106 >10100 >10100 >10100
-
7/29/2019 Algorithm Complexity lalit bansal
21/52
Linear loop
1 i=12 Loop(I
-
7/29/2019 Algorithm Complexity lalit bansal
22/52
Logarithm Loops
Multiply loops Divide loops
1 I = 1 1 I = n
2 Loop (I < n) 2 loop( i>= 1)
1 application code 1 application code
2 I=I/2
2 I = i*2
F(n) = [log n] F(n) = [log n]
-
7/29/2019 Algorithm Complexity lalit bansal
23/52
Nested loop- linear logarithmic
1 I = 1 2 loop(I
-
7/29/2019 Algorithm Complexity lalit bansal
24/52
Dependent Quadratic
1 I = 12 loop ( I < = n)
1 j = 1
2 loop( j < = i)
1 application code2 j = j + 1
3 I = I + 1
no of iterations in the body of the inner loop is
1 + 2 + 3 + 4 + + 10 = 55 I.e. n(n +1)/2
On an average = ( n+1/)2
thus total no of iterations = n (n+1)/2
-
7/29/2019 Algorithm Complexity lalit bansal
25/52
Quadratic
1 i=12 Loop (I < = n)
1 j = 1
2 Loop( j < = n)
1 application code
2 j = j+1
3 I = i+1
F(n) = n2
-
7/29/2019 Algorithm Complexity lalit bansal
26/52
Order of growth
Most important: Order of growth within a constant multipleas n
Example:
How much faster will algorithm run on computer that istwice as fast?
How much longer does it take to solve problem of double
input size?
-
7/29/2019 Algorithm Complexity lalit bansal
27/52
Values of some important functions as n
-
7/29/2019 Algorithm Complexity lalit bansal
28/52
Asymptotic order of growth
A way of comparing functions that ignores constant factors andsmall input sizes
O(g(n)): class of functions f(n) that grow no faster than g(n)
(g(n)): class of functions f(n) that grow at same rate as g(n)
(g(n)): class of functions f(n) that grow at least as fast as g(n)
-
7/29/2019 Algorithm Complexity lalit bansal
29/52
Establishing order of growth using the definition
Definition: f(n) is in O(g(n)) if order of growth of f(n) orderof growth ofg(n) (within constant multiple),
i.e., there exist positive constant cand non-negative integer
n0 such that
f(n) c g(n) for every nn0
Examples:
10nis O(n2)
5n+20 is O(n)
-
7/29/2019 Algorithm Complexity lalit bansal
30/52
Some properties of asymptotic order of growth
f(n)
O(f(n))
f(n) O(g(n)) iffg(n) (f(n))
Iff(n)
O(g(n)) and g(n)
O(h(n)) , then f(n)
O(h(n))
Note similarity with ab
Iff1
(n) O(g1
(n)) and f2
(n) O(g2
(n)) , then
f1(n) + f2(n) O(max{g1(n), g2(n)})
E bli hi d f h i li i
-
7/29/2019 Algorithm Complexity lalit bansal
31/52
Establishing order of growth using limits
limT(n)/g(n) =
0 order of growth ofT(n)< order of growth ofg(n)
c> 0 order of growth ofT(n)= order of growth ofg(n)
order of growth ofT(n)> order of growth ofg(n)
Examples:
10n vs. n2
n(n+1)/2 vs. n2
n
-
7/29/2019 Algorithm Complexity lalit bansal
32/52
LHpitals rule and Stirlings formula
LHpitals rule: Ifl im
nf(n
) =l im
ng(n
) =
andthe derivatives f, gexist, then
Stirlings formula: n! (2n)1/2 (n/e)n
f(n)
g(n)l imn
=f (n)
g(n)l imn
Example: log nvs. n
Example: 2nvs. n!
-
7/29/2019 Algorithm Complexity lalit bansal
33/52
Orders of growth of some important functions
All logarithmic functions loga
nbelong to the same class(log n) no matter what the logarithms base a> 1 is
All polynomials of the same degree kbelong to the same class:akn
k+ ak-1nk-1+ + a0(nk)
Exponential functions anhave different orders of growth fordifferent as
order log n 0) < order an < order n! < order nn
-
7/29/2019 Algorithm Complexity lalit bansal
34/52
Basic asymptotic efficiency classes
1 constant
log n logarithmic
n linear
nlog n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial
-
7/29/2019 Algorithm Complexity lalit bansal
35/52
Time efficiency of nonrecursive algorithms
General Plan for Analysis
Decide on parameter nindicating input size
Identify algorithms basic operation
Determine worst, average, and bestcases for input of size n
Set up a sum for the number of times the basic operation isexecuted
Simplify the sum using standard formulas and rules (seeAppendix A)
-
7/29/2019 Algorithm Complexity lalit bansal
36/52
Useful summation formulas and rules
liu1 = 1+1++1 = u- l+ 1In particular, liu1 = n- 1 + 1 = n(n)
1ini= 1+2++n= n(n+1)/2 n2/2 (n2)
1ini2 = 12+22++n2 = n(n+1)(2n+1)/6 n3/3 (n3)
0inai = 1+ a++ an = (an+1 - 1)/(a- 1) for any a 1In particular, 0in2i = 20 + 21 ++ 2n = 2n+1 - 1 (2n)
(aibi) = aibi cai = cai liuai = limai+ m+1iuai
-
7/29/2019 Algorithm Complexity lalit bansal
37/52
-
7/29/2019 Algorithm Complexity lalit bansal
38/52
Example 2: Element uniqueness problem
-
7/29/2019 Algorithm Complexity lalit bansal
39/52
Example 3: Matrix multiplication
-
7/29/2019 Algorithm Complexity lalit bansal
40/52
Example 4: Gaussian elimination
AlgorithmGaussianElimination(A[0..n-1,0..n])
//Implements Gaussian elimination of an n-by-(n+1) matrixA
fori 0 ton -2 do
forj i + 1 to n - 1 do
fork i ton do
A[j,k] A[j,k] -A[i,k] A[j,i] /A[i,i]
Find the efficiency class and a constant factor improvement.
-
7/29/2019 Algorithm Complexity lalit bansal
41/52
Pl f A l i f R i Al ith
-
7/29/2019 Algorithm Complexity lalit bansal
42/52
Plan for Analysis of Recursive Algorithms
Decide on a parameter indicating an inputs size.
Identify the algorithms basic operation.
Check whether the number of times the basic op. is executedmay vary on different inputs of the same size. (If it may, the
worst, average, and best cases must be investigatedseparately.)
Set up a recurrence relation with an appropriate initialcondition expressing the number of times the basic op. is
executed.
Solve the recurrence (or, at the very least, establish itssolutions order of growth) by backward substitutions oranother method.
-
7/29/2019 Algorithm Complexity lalit bansal
43/52
Example 1: Recursive evaluation ofn!
Definition: n! = 1 2 (n-1) nfor n1 and 0! = 1
Recursive definition ofn!: F(n) = F(n-1) nfor n1 and
F(0) = 1
Size:
Basic operation:
Recurrence relation:
S l i th f M( )
-
7/29/2019 Algorithm Complexity lalit bansal
44/52
Solving the recurrence for M(n)
M(n) = M(n-1) + 1, M(0) = 0
-
7/29/2019 Algorithm Complexity lalit bansal
45/52
Example 2: The Tower of Hanoi Puzzle
1
2
3
Recurrence for number of moves:
S l i f b f
-
7/29/2019 Algorithm Complexity lalit bansal
46/52
Solving recurrence for number of moves
M(n) = 2M(n-1) + 1, M(1) = 1
-
7/29/2019 Algorithm Complexity lalit bansal
47/52
-
7/29/2019 Algorithm Complexity lalit bansal
48/52
Example 3: Counting #bits
-
7/29/2019 Algorithm Complexity lalit bansal
49/52
Fibonacci numbers
The Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21,
The Fibonacci recurrence:
F(n) = F(n-1) + F(n-2)F(0) = 0
F(1) = 1
General 2nd order linear homogeneous recurrence with
constant coefficients:
aX(n) + bX(n-1) + cX(n-2) =0
S l i X( ) bX( 1) X( 2) 0
-
7/29/2019 Algorithm Complexity lalit bansal
50/52
Solving aX(n) + bX(n-1) + cX(n-2) =0
Set up the characteristic equation (quadratic)
ar2 + br+ c=0
Solve to obtain roots r1 and r2
General solution to the recurrence
ifr1 and r2 are two distinct real roots: X(n) = r1n+r2
n
ifr1 =r2 = rare two equal real roots: X(n) = rn+nr
n
Particular solution can be found by using initial conditions
Application to the Fibonacci numbers
-
7/29/2019 Algorithm Complexity lalit bansal
51/52
Application to the Fibonacci numbers
F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0
Characteristic equation:
Roots of the characteristic equation:
General solution to the recurrence:
Particular solution for F(0) =0, F(1)=1:
-
7/29/2019 Algorithm Complexity lalit bansal
52/52
Computing Fibonacci numbers
1. Definition-based recursive algorithm
2. Nonrecursive definition-based algorithm
3. Explicit formula algorithm
4. Logarithmic algorithm based on formula:
F(n-1) F(n)
F(n) F(n+1)
0 1
1 1=
n
for n1, assuming an efficient way of computing matrix powers.