Algorithm Analysis Part 2Algorithm Analysis Part 2

Complexity AnalysisComplexity Analysis

IntroductionIntroduction

Algorithm Analysis measures the efficiency Algorithm Analysis measures the efficiency of an algorithm, or its implementation as a of an algorithm, or its implementation as a program, as the input size becomes largeprogram, as the input size becomes largeActually, an estimation technique and Actually, an estimation technique and does not tell anything about the relative does not tell anything about the relative merits of two programsmerits of two programsHowever, it does serve as a tool for us to However, it does serve as a tool for us to determine whether an algorithm is worth determine whether an algorithm is worth consideringconsidering

Growth RateGrowth Rate

The The growth rategrowth rate for an algorithm is for an algorithm is the rate at which the cost of the the rate at which the cost of the algorithm grows as the size of its algorithm grows as the size of its input growsinput grows

Linear – grows as a straight lineLinear – grows as a straight line

Quadratic – grows as a quadraticQuadratic – grows as a quadratic

Exponential – grows exponentiallyExponential – grows exponentially

Best, Worst and Average CasesBest, Worst and Average CasesBest Case – If the data is arranged in such Best Case – If the data is arranged in such a way that when the algorithm runs it a way that when the algorithm runs it looks at the fewest data itemslooks at the fewest data itemsWorst Case – If the data is arranged in Worst Case – If the data is arranged in such a way that when the algorithm runs it such a way that when the algorithm runs it looks at the most data itemslooks at the most data itemsAverage Case – If the algorithm is run Average Case – If the algorithm is run many times, with random ordering of the many times, with random ordering of the data, the average number of data items data, the average number of data items the algorithm examinesthe algorithm examines

Asymptotic AnalysisAsymptotic Analysis

Asymptotic AnalysisAsymptotic Analysis refers to the refers to the study of an algorithm as the input study of an algorithm as the input size reaches a limitsize reaches a limit

We use some simplifying notionsWe use some simplifying notions– Big OBig O– Big OmegaBig Omega– ΘΘ Notation Notation

Upper BoundUpper BoundThe upper bound for an algorithm is used The upper bound for an algorithm is used to indicate the upper or highest growth to indicate the upper or highest growth raterate

We will measure this upper bound with We will measure this upper bound with respect to the best, worst or average caserespect to the best, worst or average case

We say “this algorithm has an upper We say “this algorithm has an upper bound to its growth rate of bound to its growth rate of ƒƒ(n) in the (n) in the average case”average case”

Or we say the algorithm is “in Or we say the algorithm is “in ƒƒ(n)”(n)”

Precise DefinitionPrecise DefinitionT(n) represents the running time of T(n) represents the running time of the algorithmthe algorithmƒƒ(n) is some expression for the upper (n) is some expression for the upper boundboundFor T(n) a non-negatively valued For T(n) a non-negatively valued function, T(n) is in set O(function, T(n) is in set O(ƒƒ(n)) if there (n)) if there exists two positive constants c and nexists two positive constants c and n00 such that T(n) such that T(n) ≤≤ c cƒƒ (n) for all n > n (n) for all n > n00

Lower BoundsLower Bounds

The lower bound for an algorithm is The lower bound for an algorithm is used to indicate the lowest growth used to indicate the lowest growth raterate

We will measure this lower bound We will measure this lower bound with respect to the best, worst or with respect to the best, worst or average caseaverage case

This is know as big Omega or just This is know as big Omega or just OmegaOmega

Precise DefinitionPrecise DefinitionT(n) represents the running time of T(n) represents the running time of the algorithmthe algorithmƒƒ(n) is some expression for the lower (n) is some expression for the lower boundboundFor T(n) a non-negatively valued For T(n) a non-negatively valued function, T(n) is in set function, T(n) is in set ΩΩ((ƒƒ(n)) if there (n)) if there exists two positive constants c and nexists two positive constants c and n00 such that T(n) such that T(n) ≥≥ c cƒƒ(n) for all n > n(n) for all n > n00

ΘΘ Notation NotationAn algorithm is said to be An algorithm is said to be ΘΘ((hh(n)) if it is in (n)) if it is in O(O(hh(n)) and if it is (n)) and if it is ΩΩ((hh(n)).(n)). ΘΘ notation is a stronger statement to say notation is a stronger statement to say because it requires information about both because it requires information about both the upper and lower boundsthe upper and lower boundsIt should be used when information about It should be used when information about both the upper and lower bounds is both the upper and lower bounds is availableavailableSometimes it is easier to express Sometimes it is easier to express information about a particular instantiation information about a particular instantiation of an algorithm than the algorithm itselfof an algorithm than the algorithm itself

Simplifying RulesSimplifying Rules1.1. If some function is an upper bound for your cost If some function is an upper bound for your cost

function, then its upper bound are also upper function, then its upper bound are also upper bounds for your functionbounds for your function

2.2. You can safely ignore any multiplicative You can safely ignore any multiplicative constantsconstants

3.3. When given two parts of a program run in When given two parts of a program run in sequence, you need consider only the more sequence, you need consider only the more expensive partexpensive part

4.4. If an action is in a loop, and each repetition has If an action is in a loop, and each repetition has the same cost, then the total cost is the cost of the same cost, then the total cost is the cost of the action multiplied by the number of times the action multiplied by the number of times the action takes placethe action takes place

Calculating the Running Time of a Calculating the Running Time of a ProgramProgram

a = b;a = b;– ΘΘ(1)(1)

sum = 0;sum = 0;for (i=1; i<=n; i++)for (i=1; i<=n; i++)

sum += n;sum += n;– ΘΘ(n)(n)

sum = 0;sum = 0;for (j=1; j<=n; j++)for (j=1; j<=n; j++)

for (i=1; i<=j; i++)for (i=1; i<=j; i++)sum++;sum++;

for (k=0; k<=n; k++)for (k=0; k<=n; k++)A[k] = k;A[k] = k;– ΘΘ(n(n22))

Common MisunderstandingsCommon Misunderstandings

Upper and lower bounds are only Upper and lower bounds are only interesting when you have interesting when you have incomplete knowledge about the incomplete knowledge about the thing being measuredthing being measured

The upper bound is not the same as The upper bound is not the same as the worst case.the worst case.

We don’t want to tie best and worst We don’t want to tie best and worst cases to input size.cases to input size.

Space BoundsSpace Bounds

Primary purpose of a data structure Primary purpose of a data structure is…is…– To store data!!!To store data!!!

Often you must store additional Often you must store additional information in the data structure to information in the data structure to allow access to the dataallow access to the data

This additional information is called This additional information is called overheadoverhead

The Space/Time TradeoffThe Space/Time TradeoffThe space/time tradeoff principle says that The space/time tradeoff principle says that one can often achieve a reduction in time one can often achieve a reduction in time if one is willing to sacrifice space or vice if one is willing to sacrifice space or vice versa.versa.Lookup tables allow the program to pre-Lookup tables allow the program to pre-store information that it will need often for store information that it will need often for easy lookup at the expense of needing to easy lookup at the expense of needing to store the tablestore the tableLet’s not even consider if this information Let’s not even consider if this information is stored on disk, talk about a tradeoff in is stored on disk, talk about a tradeoff in space and time to access a diskspace and time to access a disk