big oh representation used in time complexities

Fall 2002 CMSC 203 - Discrete Structures 1

Enough Mathematical Enough Mathematical Appetizers! Appetizers!

Let us look at something more interesting:Let us look at something more interesting:

AlgorithmsAlgorithms


Algorithms Algorithms

What is an algorithm?What is an algorithm?

An algorithm is a finite set of precise An algorithm is a finite set of precise instructions for performing a computation or instructions for performing a computation or for solving a problem.for solving a problem.

This is a rather vague definition. You will get to This is a rather vague definition. You will get to know a more precise and mathematically know a more precise and mathematically useful definition when you attend CS420. useful definition when you attend CS420.

But this one is good enough for now…But this one is good enough for now…


Algorithms Algorithms

Properties of algorithms:Properties of algorithms:

• InputInput from a specified set, from a specified set,• OutputOutput from a specified set (solution), from a specified set (solution),• DefinitenessDefiniteness of every step in the computation, of every step in the computation,• CorrectnessCorrectness of output for every possible input, of output for every possible input,• FinitenessFiniteness of the number of calculation steps, of the number of calculation steps,• EffectivenessEffectiveness of each calculation step and of each calculation step and• GeneralityGenerality for a class of problems. for a class of problems.


Algorithm ExamplesAlgorithm Examples

We will use a pseudocode to specify algorithms, We will use a pseudocode to specify algorithms, which slightly reminds us of Basic and Pascal.which slightly reminds us of Basic and Pascal.

Example:Example: an algorithm that finds the maximum an algorithm that finds the maximum element in a finite sequenceelement in a finite sequence

procedureprocedure max(a max(a11, a, a22, …, a, …, ann: integers): integers)max := amax := a11

forfor i := 2 i := 2 toto n nifif max < a max < aii thenthen max := a max := aii

{max is the largest element}{max is the largest element}



Another example:Another example: a linear search algorithm, a linear search algorithm, that is, an algorithm that linearly searches a that is, an algorithm that linearly searches a sequence for a particular element.sequence for a particular element.

procedureprocedure linear_search(x: integer; a linear_search(x: integer; a11, a, a22, …, , …, aann: : integers) integers)i := 1i := 1while while (i (i n and x n and x a aii))

i := i + 1i := i + 1ifif i i n n thenthen location := i location := ielseelse location := 0 location := 0{location is the subscript of the term that {location is the subscript of the term that equals x, or is zero if x is not found}equals x, or is zero if x is not found}



If the terms in a sequence are ordered, a If the terms in a sequence are ordered, a binary search algorithm is more efficient than binary search algorithm is more efficient than linear search.linear search.

The binary search algorithm iteratively The binary search algorithm iteratively restricts the relevant search interval until it restricts the relevant search interval until it closes in on the position of the element to be closes in on the position of the element to be located.located.



a c d f g h j l m o p r s u v x za c d f g h j l m o p r s u v x z

binary search for the letter ‘j’binary search for the letter ‘j’

center center elementelement

search search intervalinterval



a c d f g h j l m a c d f g h j l m o p r s u v x zo p r s u v x z






a c d f ga c d f g h j l m h j l m o p r s u v x zo p r s u v x z






a c d f ga c d f g h j h j l ml m o p r s u v x zo p r s u v x z






a c d f ga c d f g h h j j l ml m o p r s u v x zo p r s u v x z




found !found !


Algorithm ExamplesAlgorithm Examplesprocedureprocedure binary_search(x: integer; a binary_search(x: integer; a11, a, a22, …, , …, aann: : integers) integers)i := 1 i := 1 {i is left endpoint of search interval}{i is left endpoint of search interval}j := n j := n {j is right endpoint of search interval}{j is right endpoint of search interval} while while (i < j)(i < j)beginbegin

m := m := (i + j)/2(i + j)/2ifif x > a x > amm thenthen i := m + 1 i := m + 1elseelse j := m j := m

endendifif x = a x = aii thenthen location := i location := ielseelse location := 0 location := 0{location is the subscript of the term that {location is the subscript of the term that equals x, or is zero if x is not found}equals x, or is zero if x is not found}


ComplexityComplexity

In general, we are not so much interested in In general, we are not so much interested in the time and space complexity for small the time and space complexity for small inputs.inputs.

For example, while the difference in time For example, while the difference in time complexity between linear and binary search complexity between linear and binary search is meaningless for a sequence with n = 10, it is meaningless for a sequence with n = 10, it is gigantic for n = 2is gigantic for n = 23030..



For example, let us assume two algorithms A For example, let us assume two algorithms A and B that solve the same class of problems.and B that solve the same class of problems.

The time complexity of A is 5,000n, the one The time complexity of A is 5,000n, the one for B is for B is 1.11.1nn for an input with n elements. for an input with n elements.

For n = 10, A requires 50,000 steps, but B For n = 10, A requires 50,000 steps, but B only 3, so B seems to be superior to A.only 3, so B seems to be superior to A.

For n = 1000, however, A requires 5,000,000 For n = 1000, however, A requires 5,000,000 steps, while B requires 2.5steps, while B requires 2.510104141 steps. steps.



This means that algorithm B cannot be used This means that algorithm B cannot be used for large inputs, while algorithm A is still for large inputs, while algorithm A is still feasible.feasible.

So what is important is the So what is important is the growthgrowth of the of the complexity functions.complexity functions.

The growth of time and space complexity with The growth of time and space complexity with increasing input size n is a suitable measure increasing input size n is a suitable measure for the comparison of algorithms. for the comparison of algorithms.



Comparison:Comparison: time complexity of algorithms A and B time complexity of algorithms A and B

Algorithm AAlgorithm A Algorithm BAlgorithm BInput SizeInput Size

nn

1010

100100

1,0001,000

1,000,0001,000,000

5,000n5,000n

50,00050,000

500,000500,000

5,000,0005,000,000

55101099

1.11.1nn33

2.52.510104141

13,78113,781

4.84.810104139241392



This means that algorithm B cannot be used This means that algorithm B cannot be used for large inputs, while running algorithm A is for large inputs, while running algorithm A is still feasible.still feasible.

So what is important is the So what is important is the growthgrowth of the of the complexity functions.complexity functions.

The growth of time and space complexity with The growth of time and space complexity with increasing input size n is a suitable measure increasing input size n is a suitable measure for the comparison of algorithms. for the comparison of algorithms.


The Growth of FunctionsThe Growth of Functions

The growth of functions is usually described The growth of functions is usually described using the using the big-O notationbig-O notation..

Definition:Definition: Let f and g be functions from the Let f and g be functions from the integers or the real numbers to the real integers or the real numbers to the real numbers.numbers.We say that f(x) is O(g(x)) if there are We say that f(x) is O(g(x)) if there are constants C and k such thatconstants C and k such that

|f(x)| |f(x)| C|g(x)| C|g(x)|

whenever x > k.whenever x > k.



When we analyze the growth of When we analyze the growth of complexity complexity functionsfunctions, f(x) and g(x) are always positive. , f(x) and g(x) are always positive.

Therefore, we can simplify the big-O Therefore, we can simplify the big-O requirement torequirement to

f(x) f(x) C Cg(x) whenever x > k.g(x) whenever x > k.

If we want to show that f(x) is O(g(x)), we only If we want to show that f(x) is O(g(x)), we only need to find need to find oneone pair (C, k) (which is never pair (C, k) (which is never unique).unique).


The Growth of FunctionsThe Growth of FunctionsThe idea behind the big-O notation is to The idea behind the big-O notation is to establish an establish an upper boundaryupper boundary for the growth for the growth of a function f(x) for large x.of a function f(x) for large x.

This boundary is specified by a function g(x) This boundary is specified by a function g(x) that is usually much that is usually much simplersimpler than f(x). than f(x).

We accept the constant C in the requirementWe accept the constant C in the requirement

f(x) f(x) C Cg(x) whenever x > k,g(x) whenever x > k,

because because C does not grow with x.C does not grow with x.

We are only interested in large x, so it is OK ifWe are only interested in large x, so it is OK iff(x) > Cf(x) > Cg(x) for x g(x) for x k. k.



Example:Example:

Show that f(x) = xShow that f(x) = x22 + 2x + 1 is O(x + 2x + 1 is O(x22).).

For x > 1 we have:For x > 1 we have:

xx22 + 2x + 1 + 2x + 1 x x22 + 2x + 2x22 + x + x22

xx22 + 2x + 1 + 2x + 1 4x 4x22

Therefore, for C = 4 and k = 1:Therefore, for C = 4 and k = 1:

f(x) f(x) Cx Cx22 whenever x > k. whenever x > k.

f(x) is O(xf(x) is O(x22).).



Question: If f(x) is O(xQuestion: If f(x) is O(x22), is it also O(x), is it also O(x33)?)?

Yes.Yes. x x33 grows faster than x grows faster than x22, so x, so x33 grows also grows also faster than f(x).faster than f(x).

Therefore, we always have to find the Therefore, we always have to find the smallestsmallest simple function g(x) for which f(x) is simple function g(x) for which f(x) is O(g(x)). O(g(x)).



““Popular” functions g(n) arePopular” functions g(n) aren log n, 1, 2n log n, 1, 2nn, n, n22, n!, n, n, n!, n, n33, log n, log n

Listed from slowest to fastest growth:Listed from slowest to fastest growth:• 11• log nlog n• nn• n log nn log n• nn22

• nn33

• 22nn

• n!n!



A problem that can be solved with polynomial A problem that can be solved with polynomial worst-case complexity is called worst-case complexity is called tractabletractable..

Problems of higher complexity are called Problems of higher complexity are called intractable.intractable.

Problems that no algorithm can solve are Problems that no algorithm can solve are called called unsolvableunsolvable..

You will find out more about this in CS420.You will find out more about this in CS420.


Useful Rules for Big-OUseful Rules for Big-O

For any For any polynomialpolynomial f(x) = a f(x) = annxxnn + a + an-1n-1xxn-1n-1 + … + + … + aa00, where a, where a00, a, a11, …, a, …, ann are real numbers, are real numbers,f(x) is O(xf(x) is O(xnn).).

If fIf f11(x) is O(g(x) is O(g11(x)) and f(x)) and f22(x) is O(g(x) is O(g22(x)), then (x)), then (f(f11 + f + f22)(x) is O(max(g)(x) is O(max(g11(x), g(x), g22(x)))(x)))

If fIf f11(x) is O(g(x)) and f(x) is O(g(x)) and f22(x) is O(g(x)), then(x) is O(g(x)), then(f(f11 + f + f22)(x) is O(g(x)).)(x) is O(g(x)).

If fIf f11(x) is O(g(x) is O(g11(x)) and f(x)) and f22(x) is O(g(x) is O(g22(x)), then (x)), then

(f(f11ff22)(x) is O(g)(x) is O(g11(x) g(x) g22(x)).(x)).


Complexity ExamplesComplexity Examples

What does the following algorithm compute?What does the following algorithm compute?

procedureprocedure who_knows(a who_knows(a11, a, a22, …, a, …, ann: integers): integers)m := 0m := 0forfor i := 1 to n-1 i := 1 to n-1

forfor j := i + 1 to n j := i + 1 to nifif |a |aii – a – ajj| > m | > m thenthen m := |a m := |aii – a – ajj||

{m is the maximum difference between any two {m is the maximum difference between any two numbers in the input sequence}numbers in the input sequence}Comparisons: n-1 + n-2 + n-3 + … + 1Comparisons: n-1 + n-2 + n-3 + … + 1 = (n – 1)n/2 = 0.5n= (n – 1)n/2 = 0.5n22 – 0.5n – 0.5n

Time complexity is O(nTime complexity is O(n22).).


Complexity ExamplesComplexity Examples

Another algorithm solving the same problem:Another algorithm solving the same problem:

procedureprocedure max_diff(a max_diff(a11, a, a22, …, a, …, ann: integers): integers)min := a1min := a1max := a1max := a1forfor i := 2 to n i := 2 to n

ifif a aii < min < min thenthen min := a min := aii

elseelse if a if aii > max > max thenthen max := a max := aii

m := max - minm := max - min

Comparisons: 2n - 2Comparisons: 2n - 2

Time complexity is O(n).Time complexity is O(n).

big oh representation used in time complexities

Documents

binary search algorithm

linear search algorithm

letter j search interval

x ai i

growth of complexity

time complexity of algorithms

relevant search interval

b input size n