Big-O Analysis of Algorithms

Copyright © 2011 by Maria Litvin, Gary Litvin, and Skylight Publishing. All rights reserved.

Java MethodsJava MethodsObject-Oriented Programming

and Data Structures

Maria Litvin ● Gary Litvin

2nd AP edition with GridWorld

19-2

Objectives:

• Learn the big-O definition and notation

• Review big-O for some common tasks and algorithms studied earlier

• Review several popular sorting algorithms and their average, best, and worst case big-O

19-3

Evaluation of Algorithms

• Practice: benchmarks

• Theory: asymptotic analysis,

big-O

t

n

19-4

Analysis of Algorithms — Assumptions• Abstract computer model with “unlimited”

RAM

• Unlimited range for integers

• Basic operations on integers (addition, multiplication, etc.) take fixed time regardless of the values of the operands

19-5

Big-O Analysis

• Studies space and time requirements in terms of the “size” of the task

• The concept of “size” is somewhat informal here: The size of a list or another collection The dimensions of a 2-D array The argument of a function (for example,

factorial(n) or fibonacci(n)) The number of objects involved (for example,

n disks in the Tower of Hanoi puzzle)

19-6

Big-O Assumptions

• Here our main concern is time

• Ignores a constant factor measures the time in terms of the number

of some abstract steps

• Applies to large n, studies asymptotic behavior

19-7

Example 1: Sequential Search

for (int i = 0; i < n; i++) if (a[ i ] == target)

break;

( ) / 2init itert n t t n

( )t n A n B

( )1

t n B

A n A n

Small when n

The average time is approximately A·n — linear time (for large n)

t

n

t(n) = An

n/2 iterations on average

19-8

Example 2: Binary Searchn The Number of Comparisons13 27 3... ...n log2(n)

2( ) loginit itert n t t n

The average time is approximately C·log2 n — logarithmic time (for large n)

t

n

t(n) = C log n

19-9

Sequential vs. Binary Search• The logarithmic curve increases without

bound when n increases

t

n

• But... log grows much more slowly than any linear function.

• No matter what the constants are, the linear “curve” eventually overtakes the logarithmic curve.

19-10

Order of Growth

• For large enough n, a binary search on the slowest computer will finish earlier than a sequential search on the fastest computer.

• It makes sense to talk about the order of growth of a function.

• Constant factors are ignored.

For example, the order of growth of any linear function exceeds the order of growth of any logarithmic function

19-11

Big-O

Given two positive functions t(n) and g(n), we say that

t(n) = O(g(n))

if there exist a positive constant A and some number N such that

t(n) A g(n)

for all n > N.

19-12

Big-O (cont’d)

• Big-O means that asymptotically t(n) grows not faster than g(n).

• In analysis of algorithms, it is common practice to use big-O for the tightest possible upper bound.

• Example: Sequential Search is O(n) Binary Search is O(log n)

like “t(n) g(n)”

like “t(n) = g(n)”

19-13

Big-O (cont’d)

• For a log, the base is not important:

from the Change of Base Theorem

where

log log b n K n

1

logK

b

19-14

Big-O (cont’d)

• For a polynomial

11 1 0( ) ...kk

k kP n a n a a n a

( ) ( )kP n O n

the big-O is determined by the degree of the polynomial:

19-15

Big-O (cont’d)

• For “triangular” nested loops

The total number of iterations is:

2( 1)1 2 ... ( 1) ( )

2

n nn O n

for (int i = 0; i < n; i++) for (int j = i; j < n; j++) ...

19-16

Big-O Examples

• O(1) — constant time Finding a median value in a sorted array Calculating 1 + 2 + ... + n using the formula for the

sum of an arithmetic sequence Push and pop operations in an efficiently

implemented stack; add and remove operations in a queue (Chapter 22)

Finding a key in a lookup table or a sparsely populated hash table (Chapter 25)

19-17

Big-O Examples

• O(log n) — logarithmic time Binary search in a sorted list of n elements Finding a target value in a binary search

tree with n nodes (Chapter 24) add and remove operations in a priority

queue, implemented as a heap, with n nodes (Chapter 26)

19-18

Big-O Examples

• O(n) — linear time Traversing a list with n elements, (for

example, finding max or min) Calculating n factorial or the n-th Fibonacci

number iteratively Traversing a binary tree with n nodes

(Chapter 24)

19-19

Big-O Examples

• O(n log n) — “n log n” time Mergesort and Quicksort of n elements Heapsort (Chapter 26)

19-20

Big-O Examples

• O(n2) — quadratic time More simplistic sorting algorithms, such as

Selection Sort of n elements Traversing an n by n 2-D array Finding duplicates in an unsorted list of n

elements (implemented with a nested loop)

19-21

Big-O Examples

• O(an) (a > 1) — exponential time Recursive Fibonacci implementation

(a 3/2; see Chapter 23) Solving the Tower of Hanoi puzzle

(a = 2; see Section 23.5) Generating all possible permutations

of n symbols

19-22

Big-O SummaryO(1)  <  O(log n)  <  O(n)  <  O(n log n)  <  O(n2)  <  O(n3)  <  O(an)

Exponential:

t = kan Quadratic:

t = Cn2

n log n

log n

Linear

Constant

t

n

19-23

Sorting

• O(n2)

By counting Selection Sort Insertion Sort

• O(n log n)

Mergesort Quicksort Heapsort

If you are limited to “honest” comparison sorting, any sorting algorithm, in its worst case scenario, takes at least O(nlog n) steps

19-24

Sorting by Counting — O(n2)• For each value in the list, count how many values are

smaller (or equal with a smaller index).

• Place that value into the position indicated by the count.

for (int i = 0; i < n; i++) { count = 0; for (int j = 0; j < n; j++) if (a [ j ] < a [ i ] | | (a [ j ] == a [ i ] && j < i ) ) count++; b [count ] = a [ i ]; }

Always n2 comparisons

19-25

Selection Sort — O(n2)• Find the largest value and swap it with the last

element.

• Decrement the “logical” size of the list and repeat while the size exceeds 1.

while (n > 1) { iMax = 0; for (int i = 1; i < n; i++) if (a [ i ] > a [ iMax ]) iMax = i;

swap (a, iMax, n-1); n--; }

Always n(n-1)/2 comparisons

19-26

Insertion Sort — O(n2)• Keep the beginning of the list sorted.

• Insert the next value in order into the sorted. beginning segment.

for (int i = 1; i < n; i++) { temp = a [ i ]; for (int j = i; j > 0 && a [ j-1 ] > temp; j--) a [ j ] = a [ j-1 ]; a [ j ] = temp; }

O(n2) comparisons on average, but only O(n) when the list is already sorted

19-27

Mergesort — O(n log n)• Split the list down the middle into two lists of

approximately equal length.

• Sort (recursively) each half.

• Merge the two sorted halves into one sorted list.

private void sort (double a [ ], int from, int to) { if (to == from) return;

int middle = (from + to) / 2; sort(a, from, middle); sort(a, middle + 1, to); merge (a, from, middle, middle + 1, to); }

Each merge takes O(n) comparisons

19-28

Quicksort — O(n log n)• Choose a “pivot” element.

• Partition the array so that all the values to the left of the pivot are smaller than or equal to it, and all the elements to the right of the pivot are greater than or equal to it.

• Sort (recursively) the left-of-the-pivot and right-of-the-pivot pieces.

Proceed from both ends of the array as far as possible; when both values are on the wrong side of the pivot, swap them

O(n log n) comparisons, on average, if partitioning splits the array evenly most of the time

19-29

Heapsort — O(n log n)

• An elegant algorithm based on heaps (Chapter 26)

19-30

Sorting Summary Best case Average case Worst case

Selection Sort O(n2) O(n2) O(n2)

Insertion Sort

O(n) — array already sorted

O(n2) O(n2)

Mergesort

O(n log n), or O(n) in a slightly modified version when the array is sorted

O(n log n) O(n log n)

Quicksort

O(n log n) O(n log n) O(n2) — pivot is consistently chosen far from the median value, e.g., the array is already sorted and the first element is chosen as pivot

Heapsort O(n log n) O(n log n) O(n log n)

19-31

Review:

• What assumptions are made for big-O analysis?

• What O(...) expressions are commonly used to measure the big-O performance of algorithms?

• Is it true that n(n-1)/2 = O(n2)?

• Give an example of an algorithm that runs in O(log n) time.

• Give an example of an algorithm (apart from sorting) that runs in O(n2) time.

19-32

Review (cont’d):

• Name three O(n log n) sorting algorithms.

• How many comparisons are needed in sorting by counting? In Selection Sort?

• What is the main idea of Insertion Sort?

• What is the best case for Insertion Sort?

• What is the main idea of Quicksort?

• What is the worst case for Quicksort?