asymtotic analysis
TRANSCRIPT
Runtime Efficiencydand
Asymptotic AnalysisAsymptotic Analysis
M. Shoaib Farooq
Algorithm Efficiency
There are often many approaches (algorithms) to solve a problem.solve a problem.
How do we choose between them?
Two goals for best design practices:
1. To design an algorithm that is easy to understand code debugunderstand, code, debug.
2. To design an algorithm that makes efficient use of the computer’s resources.
M. Shoaib Farooq
Algorithm Efficiency (cont)Algorithm Efficiency (cont)
Goal (1) is the concern of SoftwareGoal (1) is the concern of Software Engineering.
G l (2) i th f d t t tGoal (2) is the concern of data structures and algorithm analysis.
When goal (2) is important, how do we measure an algorithm’s cost?measure an algorithm s cost?
M. Shoaib Farooq
How to Measure Efficiency?How to Measure Efficiency?
1. Empirical comparison (run programs)p p ( p g )2. Asymptotic Algorithm Analysis
Critical resources:Critical resources:
Factors affecting running time:g g
For most algorithms, running time depends “ i ” f th i ton “size” of the input.
Running time is expressed as T(n) for some f ti T i t i
M. Shoaib Farooq
function T on input size n.
Efficiency (Complexity)• The rate at which storage or time grows as a function of the problem size.
• Two types of efficiency:
• Time efficiency
• Space efficiencySpace efficiency
• There are always tradeoffs between these twoefficiencies.
• Example: singly vs. doubly linked list
M. Shoaib Farooq
p g y y
Space/Time Tradeoff PrincipleSpace/Time Tradeoff Principle• There are always tradeoffs between these two
efficienciesefficiencies.One can often reduce time if one is willing to
sacrifice space or vice versasacrifice space, or vice versa.– Example:
• singly vs. doubly linked list
Disk-based Space/Time Tradeoff Principle: The smaller you make the disk storage requirements, the faster your program will run.
M. Shoaib Farooq
Time Efficiency• How do we improve the time efficiency of a program?
• The 90/10 Rule90% of the execution time of a program is spent in90% of the execution time of a program is spent inexecuting 10% of the code
• So, how do we locate the critical 10%?
• software metrics toolssoftware metrics tools• global counters to locate bottlenecks (loop executions, function calls)
M. Shoaib Farooq
Time Efficiency ImprovementsPossibilities (some better than others!)
• Move code out of loops that does not belong there(just good programming!)
• Remove any unnecessary I/O operations (I/O operationsare expensive timewise)
• Code so that the compiled code is more efficient
• Replace an inefficient algorithm (best solution)
Moral - Choose the most appropriate algorithm(s) BEFORE
M. Shoaib Farooq
Moral - Choose the most appropriate algorithm(s) BEFOREprogram implementation
Real Time Vs Logical units
• To Evaluate an algorithm efficiency real time unith i d d d h ld bsuch as microsecond and nanosecond should no be
used.
• Logical unit that express a relationship between the size n of data and the amount of time t required to process the data should be used.
M. Shoaib Farooq
Complexity Vs Input Size
• Complexity function that express relationship between n and t is usually much more complexbetween n and t is usually much more complex, and calculating such a function only in regard to large amount of data.g
• Normally, we are not so much interested in thetime and space complexity for small inputs.
• For example, while the difference in time l it b t li d bi h icomplexity between linear and binary search is
meaningless for a sequence with n = 3, it is more significant for for n = 230
M. Shoaib Farooq
significant for for n = 230.
Example
• N2 is less than 2N
- N=2 22=4 2 2= 4 Both are sameN 2 2 4 2 4 Both are same- N=3 32=9 23= 8 N2 is greater than 2N
N 4 42 16 24 16 b h- N=4 42=16 24= 16 both are same- N=5 52=25 25=32 2N is greater than N2
N2 < 2N for all N >=No where No =4
M. Shoaib Farooq
Complexity
• Comparison: time complexity of algorithms A and B
Algorithm AAlgorithm A Algorithm BAlgorithm BInput SizeInput Sizenn1010
5,000n5,000n50,00050,000
⎡⎡1.11.1nn⎤⎤33
1001001 0001 000
,,500,000500,000
5 000 0005 000 000 2 52 5⋅⋅10104141
13,78113,7811,0001,000
1,000,0001,000,0005,000,0005,000,000
55⋅⋅101099
2.52.5 10104.84.8⋅⋅10104139241392
M. Shoaib Farooq
Complexity
•This means that algorithm B cannot be used for large inputs while running algorithm A is stilllarge inputs, while running algorithm A is still feasible.
•So what is important is the growth of the complexity functionscomplexity functions.
•The growth of time and space complexity with•The growth of time and space complexity with increasing input size n is a suitable measure for the comparison of algorithms
M. Shoaib Farooq
the comparison of algorithms.
Best, Worst, Average CasesBest, Worst, Average Cases
Not all inputs of a given size take the same ti t
p gtime to run.
Sequential search for K in an array of ni tq yintegers:
• Begin at first element in array and look at each element in turn until K is foundeach element in turn until K is found
Best case:Worst case:Average case:
M. Shoaib Farooq
Average case:
Best Case
• Best case is defined as which input ofBest case is defined as which input of size n is cheapest among all inputs of size n.size n.
• Example1 Al d t d i I ti S t1- Already sorted array in Insertion Sort2- Key element exist at first position in Array for linear
searchsearch
M. Shoaib Farooq
Worst Case
• Worst case refers to the worst input from among the choices for possible inputs of a given size.
• Example1- Linear Search
key element is the last element in array2- Insertion Sort
Array is in reverse order sorted
M. Shoaib Farooq
Average Caseg• Average case appears to be the fairest
measure. • Data is equally likely to occur at any q y y y
position in any order• It may be difficult to determineIt may be difficult to determine• Probabilistic Analysis are used
E l• Example1- Randomly choose n numbers and apply insertion sort
M. Shoaib Farooq
Asymptotic Analysis (Runtime Analysis)
• Independent of any specific hardware or software
• Expresses the complexity of an algorithm in terms of its relationship to some known function
• The efficiency can be expressed as “proportional to” or“on the order of” some function
• A common notation used is called Big-Oh:g
T(n) = O(f(n))
M. Shoaib Farooq
Said: T of n is “on the order of” f(n)
Asymptotic Analysis
• This idea is incorporated in the "Big Oh" notation for asymptotic performance.
• Definition: T(n) = O(f(n))Definition: T(n) O(f(n))• if and only if there are constants c0 and n0
such that T(n) <= c0 f(n) for all n >= n0such that T(n) <= c0 f(n) for all n >= n0.
M. Shoaib Farooq
Example Algorithm and Corresponding Big-OhAlgorithm:
• prompt the user for a filename and read it(400 “ti it ”)(400 “time units”)• open the file (50 “time units”)• read 15 data items from the file into an array (10 “timeunits” per read)
• close the file (50 “time units”)F l d ibi l ith ffi iFormula describing algorithm efficiency:
500 + 10n where n=number of items read (here, 15)
Which term of the function really describes the growth patternas n increases?
M. Shoaib Farooq
Therefore, the Big-Oh for this algorithm is O(n).
Example (Con’t)F i ill b l h h• Function n will never be larger than the function 500 + 10 n, no matter how large n getsgets.
• However, there are constants c0 and n0 such that 500 + 10n <= c0 n when n >= n0 Onethat 500 + 10n <= c0 n when n >= n0. One choice for these constants is c0 = 20 andn0 = 50n0 = 50.
• Therefore, 500 + 10n = O(n).Oth h i f 0 d 0 F l• Other choices for c0 and n0. For example, any value of c0 > 20 will work for n0 = 50.
M. Shoaib Farooq
Common Functions in Big-Oh(Most Efficient to Least Efficient)(Most Efficient to Least Efficient)
• O(1) or O(c)Constant growth. The runtime does not grow at all as afunction of n. It is a constant. Basically, it is any operationthat does not depend on the value of n to do its job Hasthat does not depend on the value of n to do its job. Hasthe slowest growth pattern (none!).
E lExamples:
• Accessing an element of an array containing n elements• Adding the first and last elements of an array containing• Adding the first and last elements of an array containingn elements
• Hashing
M. Shoaib Farooq
Common Functions in Big-Oh (con’t)
• O(lg(n))L ith i th Th ti th i ti l tLogarithmic growth. The runtime growth is proportional tothe base 2 logarithm (lg) of n.
Example:
• Binary search
M. Shoaib Farooq
Common Functions in Big-Oh (con’t)
• O(n)Linear growth. Runtime grows proportional to the value ofn.
Examples:
• Sequential (linear) search• Any looping over the elements of a one-dimensionalarray (e.g., summing, printing)
M. Shoaib Farooq
Common Functions in Big-Oh (con’t)
• O(n lg(n))n log n growth. Any sorting algorithm that uses comparisonsbetween elements is O(n lg n), based on divide an conquer
happroach.
Examples
• Merge sort• Quicksort
M. Shoaib Farooq
Common Functions in Big-Oh (con’t)
• O(nk)( )Polynomial growth. Runtime grows very rapidly.
E lExamples:
• Bubble sort (O(n2)) • Selection (exchange) sort (O(n2))• Selection (exchange) sort (O(n2))• Insertion sort (O(n2))
M. Shoaib Farooq
Common Functions in Big-Oh (con’t)
• O(2n)( )Exponential growth. Runtime grows extremely rapidlyas n increases. Are essentially useless except for very
ll l fsmall values of n.
• Others• O(sqrt(n))• O(n!)
M. Shoaib Farooq
Practical Complexity25050
f(n) = nf(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2f(n) n 2f(n) = n 3f(n) = 2 n
01 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
M. Shoaib Farooq
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Practical Complexity500500
f(n) = nf(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2f(n) n 2f(n) = n 3f(n) = 2 n
01 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
M. Shoaib Farooq
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Practical Complexity1000000
f(n) = nf(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2f(n) n 2f(n) = n 3f(n) = 2 n
01 3 5 7 9 11 13 15 17 19
M. Shoaib Farooq
1 3 5 7 9 11 13 15 17 19
Practical Complexity5000
4000
5000
f( )
3000
f(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2
1000
2000f(n) n 2f(n) = n 3f(n) = 2 n
0
1000
5 5
M. Shoaib Farooq
1 3 5 7 9 11 13 15 17 19
Practical Complexity10000000
100000
1000000
0000000
10000
100000
100
1000
1
10
1 4 16 64 256 1024 4096 16384 65536
M. Shoaib Farooq
1 4 16 64 256 1024 4096 16384 65536
Asymptotic Notations• These notations are used to describe the asymptotic running time of an
algorithm, are defined in terms of function whose domain are set of natural number N=0,1,2,……
Asymptotic Time Complexity• The limiting behavior of the execution time of an algorithm when theThe limiting behavior of the execution time of an algorithm when the
size of the problem goes to infinity.
Asymptotic Space Complexity• The limiting behavior of the use of memory space of an algorithm
when the size of the problem goes to infinitywhen the size of the problem goes to infinity.
M. Shoaib Farooq
Asymptotic Tight BoundΘ- Notation
f( ) Θ( ( )) h Θ( ( )) th t f• f(n) = Θ(g(n)) where Θ(g(n)) are the set of functions
• A function f(n) is Θ(g(n)) if there exist positive constants c1, c2, and n0 such that
0≤ c1 g(n) ≤ f(n) ≤ c2 g(n) for all n ≥ n0
• Theorem– f(n) is Θ(g(n)) if f(n) is both O(g(n)) and Ω(g(n))
M. Shoaib Farooq
( ) (g( )) ( ) (g( )) (g( ))
Θ (Cont.)- A function f(n) belongs to the set Θ(g(n)) if there existA function f(n) belongs to the set Θ(g(n)) if there exist
positive constants c1 and c2 such that it can be “sandwiched” between c1g(n) and c2g(n) for allsandwiched between c1g(n) and c2g(n), for all large n.
Example: 3n+2 = Θ(n)Example: 3n+2 = Θ(n) as 3n+2 ≥3n for all n ≥2, and 3n+2 ≤4n for all n ≥2 so c1=3, c2=4 and n0=2.3n +3 = Θ (n)
M. Shoaib Farooq
-
asymptotic upper boundbig-O notation
• An asymptotic bound, as function of the size of the input, on the worst (slowest, most amount of space used, etc.) an algorithmwill do to solve a problem.
• That is, no input will cause the algorithm to use more resources than the boundthan the bound
– f(n) is O(g(n)) if there exist positive constants c and n0 such that f(n) ≤ c ⋅ g(n) for all n ≥ n0n0 such that f(n) ≤ c g(n) for all n ≥ n0
– Formally, O(g(n)) = f(n): there exist positive constants c and n0 such that0
– f(n) ≤ c ⋅ g(n) for all n ≥ n0
E l 3 +2 O( ) 3 +2 < 4 f l > 2 i 3 +3 O( )
M. Shoaib Farooq
Example 3n+2 = O(n) as 3n+2 <= 4n for al n>=2 i.e. 3n+3=O(n)
Big-Oh (Cont.)•The idea behind the big-O notation is to establish an upper boundary for the growth of a function f(n) for large .
hi b d i ifi d b f i ( ) h i ll•This boundary is specified by a function g(x) that is usually much simpler than f(x).
h C i h i•We accept the constant C in the requirement•f(n) ≤ C⋅g(n) whenever n > n0,
•We are only interested in large x, so it is OK iff(x) > C⋅g(x) for x ≤ k.( ) g( )•The relationship between f and g can be expressed by stating either that g(n) is an upper bound on the value of f(n)
h i h l f f
M. Shoaib Farooq
or that in the long run , f grows at most as fast as g.
Big-Oh (Cont.)
•Question: If f(x) is O(x2), is it also O(x3)?
•Yes. x3 grows faster than x2, so x3 grows also faster than f(x).
•Therefore, we always have to find the smallest simple function g(x) for which f(x) is O(g(x)).
M. Shoaib Farooq
Useful Rules for Big-O•For any polynomial f(x) = anxn + an-1xn-1 + … + a0, where a0, a1, …, an are real numbers,•f(x) is O(xn)•f(x) is O(xn).
•If f1(x) is O(g1(x)) and f2(x) is O(g2(x)), then 1( ) (g1( )) 2( ) (g2( )),(f1 + f2)(x) is O(max(g1(x), g2(x)))
f f ( ) i O( ( )) d f ( ) i O( ( )) h•If f1(x) is O(g(x)) and f2(x) is O(g(x)), then(f1 + f2)(x) is O(g(x)).
•If f1(x) is O(g1(x)) and f2(x) is O(g2(x)), then (f1f2)(x) is O(g1(x) g2(x)).
M. Shoaib Farooq
( 1 2)( ) (g1( ) g2( ))
Asymptotic Lower boundΩ notation
• An asymptotic bound, as function of the size of the input, on thebest (fastest, least amount of space used, etc.) an algorithm can
ibl hi t l blpossibly achieve to solve a problem.
• Ω notation provides an asymptotic lower bound on a function
• That is, no algorithm can use fewer resources than the bound
f( ) i Ω( ( )) if th i t iti t t d– f(n) is Ω(g(n)) if there exists positive constants c and n0such that 0 ≤ c g(n) ≤ f(n) for all n ≥ n– 0 ≤ c⋅g(n) ≤ f(n) for all n ≥ n0
• Theoremf( ) i ( ( )) if f( ) i b h ( ( )) d ( ( ))
M. Shoaib Farooq
– f(n) is Θ(g(n)) if f(n) is both O(g(n)) and Ω(g(n))
M. Shoaib Farooq
o-notation (small o)•A theoretical measure of the execution of an algorithm, usually the time or memory needed, given the problem size n, which is usually the number of items. •Informally saying some equation•Informally, saying some equationf(n) = o(g(n)) means f(n) becomes insignificant relative to g(n) as n approaches infinity. The notation is read, "f of n is little oh of g of n". pp y g
Formal Definition: f(n) = o(g(n)) means for all c > 0 there exists 0 0 h h 0 f( ) ( ) f ll 0some n0 > 0 such that 0 <=f(n) < cg(n) for all n>n0.
Example : 2n= o(n2) BUT 2n != O(n2)Example : 2n= o(n2), BUT 2n != O(n2)
M. Shoaib Farooq
ω-notation
• f(n) = ω (g(n)) means g(n) becomes insignificantrelative to f(n) as n goes to infinity.
Formal Definition: f(n) = ω (g(n)) means that for anyFormal Definition: f(n) = ω (g(n)) means that for any positive constant c>, there exists a constant n0>0, such that 0 <= cg(n) < f(n) for all n>=n0such that 0 <= cg(n) < f(n) for all n>=n0.
Example : n2/2= ω(n) but n/2 != ω(n2)
M. Shoaib Farooq
Complexity Examples•What does the following algorithm compute?•procedure who_knows(a1, a2, …, an: integers)
0•m := 0•for i := 1 to n-1• for j := i + 1 to nj• if |ai – aj| > m then m := |ai – aj|•m is the maximum difference between any two numbers in th i t the input sequence•Comparisons: n-1 + n-2 + n-3 + … + 1• = (n 1)n/2 = 0 5n2 0 5n• = (n – 1)n/2 = 0.5n2 – 0.5n
•Time complexity is O(n2).
M. Shoaib Farooq
Complexity Examples•Another algorithm solving the same problem:•procedure max_diff(a1, a2, …, an: integers)•min := a1•max := a1•for i := 2 to n• if ai < min then min := ai• else if ai > max then max := ai•m := max min•m := max - min•Comparisons: 2n + 2
•Time complexity is O(n)•Time complexity is O(n).
M. Shoaib Farooq
Big Oh Does Not Tell the Whole Story • Two Algorithms A and B with same asymptotic
performance
• Why select one over the other, they're both the same, right?
• They may not be the same There is this small matter of theThey may not be the same. There is this small matter of the constant of proportionality.
• Suppose that A does ten operations for each data item, butSuppose that A does ten operations for each data item, but algorithm B only does three.
• It is reasonable to expect B to be faster than A even though p gboth have the same asymptotic performance. The reason is that asymptotic analysis ignores constants of proportionality.
M. Shoaib Farooq
Big Oh Does Not Tell the Whole Story (cont)
Example: An algorithm A set up the algorithm, taking 50 time units; read in n elements into array A; /* 3 units per element */
for (i = 0; i < n; i++) do operation1 on A[i]; /* takes 10 units */ do operation1 on A[i]; /* takes 10 units */
do operation2 on A[i]; /* takes 5 units */ do operation3 on A[i]; /* takes 15 units */
M. Shoaib Farooq
TA(n) = 50 + 3*n + (10 + 5 + 15)*n = 50 + 33*n
Big Oh Does Not Tell the Whole Story (cont)g y ( )Example: An algorithm Bset up the algorithm, taking 200 time units; read in n elements into array A; /* 3 units per element */
for (i = 0; i < n; i++)for (i 0; i n; i ) do operation1 on A[i]; /* takes 10 units */ d ti 2 A[i] /* t k 5 it */do operation2 on A[i]; /* takes 5 units */
M. Shoaib Farooq
TB(n) =200 + 3*n + (10 + 5)*n = 200 + 18*n
Big Oh Does Not Tell the Whole Story (cont)
• Both algorithms have time complexity O(n). • Algorithm A sets up faster than B, but does more
operations on the data • Algorithm A is the better choice for small values
of n.• For values of n > 10, algorithm B is the better
choicechoice
M. Shoaib Farooq
Big Oh Does Not Tell the Whole Story (cont)
Algo AAlgo A
AlgoBAlgoB
M. Shoaib Farooq
Loop Invariant p• used to check the correctness of algorithm
It i diti ( ) hi h t h ld b f• It is a condition(s) which must hold before during and after the execution of any loop.
• If Can be written as - Descriptive Statement(s)- Mathematical Expression(s)
Note: In order to write Loop invariant we must follow actual implementation of Algorithm
M. Shoaib Farooq