analysis of algorithms aaron tan tantc/cs1101.html

Analysis of Algorithms

Aaron Tan

http://www.comp.nus.edu.sg/~tantc/cs1101.html

2Analysis of Algorithms

Introduction to Analysis of Algorithms

After you have read and studied this chapter, you should be able to

Know what is analysis of algorithms (complexity analysis)

Know the definition and uses of big-O notation

How to analyse the running time of an algorithm


Introduction (1/2)

+

Two aspects on writing efficient codes: Programming techniques

Implementation of algorithms Practitioner’s viewpoint

Asymptotic analysis (“big-O” notation, etc.) Analysis and design of algorithms Theoretician’s viewpoint

Asymptotic analysis keeps the student’s head in the clouds, while attention to implementation details keeps his feet on the grounds.


Introduction (2/2)

+

Programming techniques versus Algorithm design:

int f1 (int n) { int a, sum=0;

for (a=1; a<=n; a++) sum += a; return sum;}


Sum of Two Elements (1/5) Given this problem:

A sorted list of integers list and a value is given. Write a program to find the indices of (any) two distinct elements in the list whose sum is equal to the given value.

Example:list: 2, 3, 8, 12, 15, 19, 22, 24

sum: 23

answer: elements 8 (at subscript 2) and 15 (at subscript 4)


Sum of Two Elements (2/5) Algorithm A:

list: 2, 3, 8, 12, 15, 19, 22, 24

sum: 23

answer: elements 8 (at subscript 2) and 15 (at subscript 4)

n = size of list

for x from 0 to n – 2

for y from x + 1 to n – 1

if ((list[x] + list[y]) == sum)

then found! (answers are x and y)


Sum of Two Elements (3/5) Code for algorithm A:import java.util.*;

class SumOfTwoA {

public static void main(String[] args) { Scanner scanner = new Scanner(System.in);

int[] list = { 2, 3, 8, 12, 15, 19, 22, 24 }; int n = list.length;

System.out.print("Enter sum: "); int sum = scanner.nextInt();

boolean found = false; for (int x = 0; x < n-1 && !found; x++) for (int y = x+1; y < n && !found; y++) if (list[x] + list[y] == sum) { System.out.println("Indices at " + x + " and " + y); found = true; } }}


Sum of Two Elements (4/5) Algorithm A used nested loop and scans some

elements many times. Algorithm B: can you use a single loop to

examine each element at most once? If this can be done, it will be more efficient than

Algorithm A.


Sum of Two Elements (5/5) Code for algorithm B:

+


The Race

Who will reach the finishing line first?

100 metres ahead


Complexity Analysis (1/2) Complexity analysis: to measure and predict the

behavior (running time, storage space) of an algorithm.

We will focus on running time here.

Inexact, but provides a good basis for comparisons.

We want to have a good judgment on how an algorithm will perform if the problem size gets very big.


Complexity Analysis (2/2) Problem size is defined based on the problem

on hand.

Examples of problem size: Number of elements in an array (for sorting

problems). Length of the strings in an anagram problem. Number of discs in the Tower of Hanoi problem. Number of cities in the Traveling Salesman Problem

(TSP).


Definition (1/4) Assume problem size is n and T(n) is the running time.

Upper bound – Big-O notationDefinition:

T(n) = O(f(n)) if there are constants c and n0

such that T(n) c*f(n) when n n0

We read the equal sign = as “is a member of” (), because O(f(n)) is a set of functions.

We may also say that T(n) is bounded above by f(n).


Definition (2/4)

0

3000

2000

1000

1 500 1000 1500 2000

n-axis

f(n)

K * g(n)

n0

Graphical Meaning of big-O Notation

A(n)

c*B(n)A(n) = O(B(n))


Definition (3/4) The functions’ relative rates of growth are compared.

For instance, compare f(n) = n2 with g(n) = 1000n. Although at some points f(n) is smaller than g(n), f(n) actually

grows at a faster rate than g(n). (Hence, an algorithm with running time complexity of f(n) is slower than another with running time complexity of g(n) in this example.)

Hence, g(n) = O(f(n)).

The definition says that eventually there is some point n0 past which c*f(n) is always larger or equal to g(n). Here, we can make c = 1 and n0 = 1000.


Definition (4/4) Besides the big-O (upper bound) analysis, there are the

Omega (lower bound) analysis, the Theta (tight bounds) analysis, and others.


Exercises (1/4) f(n) = 1 + 2 + 3 + … + n

Show that f(n) = O(n2)

Proof:

1 + 2 + 3 + … + n = n(n+1)/2 = n2/2 + n/2 n2/2 + n2/2 = n2

The above is the running time of basic sorting algorithms such as bubblesort, insertion sort, selection sort.


Exercises (2/4) f(n) = 17 + n + n/3

Show that f(n) = O(n)

Proof: 17 + n + n/3 3n = O(n)

f(n) = n4 + n2 + 20n + 100

Show that f(n) = O(n4)

Proof: n4 + n2 + 20n + 100 4n4 = O(n4)

From the two examples above, it can be seen that an expression is dominated by the term of the highest degree.


Exercises (3/4) Tower of Hanoi

Algorithm:Tower(n, source, temp, dest) {

if (n > 0) {tower(n-1, source, dest, temp);move disc from source to dest;tower(n-1, temp, source, dest);

}}

Let T(n) = number of move to solve a tower of n discs.


Exercises (4/4) Tower of Hanoi (cont.)

Let T(n) = number of move to solve a tower of n discs.

Prove that T(n) = 2n – 1. T(0) = 0

T(n) = T(n – 1) + 1 + T(n – 1)

= 2 T(n – 1) + 1

= 2 (2n–1 – 1) + 1

= 2 2n–1 – 2 +1

= 2n – 1

Hence T(n) = O(2n)


Some Common Series

122 1

0

nn

i

i1 + 2 + 4 + 8 + 16 + …+ 2n

2

)1(

1

nni

n

i

1 + 2 + 3 + 4 + 5 … + n

6

)12)(1(

1

2

nnni

n

i

12 + 22 + 32 + 42 + 52 … + n2


The Conversation Boss: Your program is too slow! Rewrite it!

You: But why? All we need to do is to buy faster computer!

Is this really the solution?


Complexity Classes (1/4) There are some common complexity classes.

In analysis of algorithm, log refers to log2, or sometimes written simply as lg.

Notation Name

O(1) Constant.

O(log n) Logarithmic.

O(n) Linear.

O(n log n) Linearithmic, loglinear, quasilinear or supralinear.

O(n2) Quadratic.

O(nc), c > 1 Polynomial, sometimes called algebraic.

Examples: O(n2), O(n3), O(n4).

O(cn) Exponential, sometimes called geometric.

Examples: O(2n), O(3n).

O(n!) Factorial, sometimes called combinatorial.

n is problem size.


Complexity Classes (2/4) Algorithms of polynomial running times are desirable.

Algorithm A stops in f(n) microseconds

f(n) n = 2 n = 16 n = 256 n = 1024 n = 1048576

1 1 1 1 1.00 x 100 1.00 x 100

log2 n 1 4 8 1.00 x 101 2.00 x 101

n 2 1.6 x 101 2.56 x 102 1.02 x 103 1.05 x 106

n log2 n 2 6.4 x 101 2.05 x 103 1.02 x 104 2.10 x 107

n2 4 2.56 x 102 6.55 x 104 1.05 x 106 1.10 x 1012

n3 8 4.10 x 103 1.68 x 107 1.07 x 109 1.15 x 1018

2n 4 6.55 x 104 1.16 x 1077 1.80 x 10308 6.74 x 10315652

Table 1. Running Times for Different Complexity Classes


Complexity Classes (3/4)

Table 2. Running Times for Algorithm A in Different Time Units.

f(n) n = 2 n = 16 n = 256 n = 1024 n = 1048576

1 1 sec 1 sec 1 sec 1 sec 1 sec

log2 n 1 sec 4 sec 8 sec 10 sec 20 sec

n 2 sec 16 sec 256 sec 1.02 ms 1.05 sec

n log2 n 2 sec 64 sec 2.05 ms 10.2 ms 21 sec

n2 4 sec 25.6 sec 65.5 ms 1.05 sec 1.8 wk

n3 8 sec 4.1 ms 16.8 sec 17.9 min 36,559 yr

2n 4 sec 65.5 ms 3.7x1063 yr 5.7x10294 yr 21x10315659 yr


Complexity Classes (4/4)

Table 3. Size of Largest Problem that Algorithm A can solve if solution is computed in time <= T at 1 micro-sec per step.

Number ofsteps is

T = 1 min T = 1 hr T = 1 day T = 1 wk T = 1 yr

n 6 x 107 3.6 x 109 8.64 x 1010 6.05 x 1011 3.15 x 1013

n log2 n 2.8 x 106 1.3 x 108 2.75 x 109 1.77 x 1010 7.97 x 1011

n2 7.75 x 103 6.0 x 104 2.94 x 105 7.78 x 105 5.62 x 106

n3 3.91 x 102 1.53 x 103 4.42 x 103 8.46 x 103 3.16 x 104

2n 25 31 36 39 44

10n 7 9 10 11 13


Analysing Simple Codes (1/4)

Some rules.

Basic operations are those that can be computed in O(1) or constant time. Examples are assignment statements, comparison

statements, and simple arithmetic operations.


Analysing Simple Codes (2/4) Code fragment 1:

temp = x;x = y;y = temp;

Running time: 3 statements = O(1)

Code fragment 2:

p = list.size();for (int i = 0; i < p; ++i) { list[i] += 3;}

Running time: 1 + p statements = O(p)



if (x < y) { a = 1; b = 2; c = 3;}else { a = 2; b = 4; c = 8; d = 13; e = 51;}

Running time: max{3, 5} statements = 5 = O(1)

In code fragments 2 and 3, we consider only assignment statements as our basic operations. Even if we include the loop test operation (i < p) and update operation (++i) in fragment 2, and the if test operation (x < y) in fragment 3, it will not affect the final result in big-O notation, since they are each of constant time.



sum = 0.0;for (int k = 0; k < n; ++k) sum += array[k];avg = sum/n;

Running time: 1 + n + 1 statements = n + 2 = O(n)

Code fragment 5:

for (int i = 0; i < n; i++) for (int j = 0; j < i; ++j) sum += matrix[i][j];

Running time: 0 + 1 + 2 + … + (n-1) = n(n-1)/2 = O(n2)


General Rules (1/3) Rule 1: Loops

The running time of a loop is at most the running time of the statements inside the loop times the number of iterations.

for (...) { …; …; …;}

m statements

n iterationsn m statements

Example: if m = 3, then running time is 3n or O(n).


General Rules (2/3) Rule 2: Nested loops

Analyse these inside out. The total running time of a statement inside a group of nested loops is the running time of the statement multiplied by the product of the sizes of all the loops.

for (…) { for (...) { …; …; …; }}

O(n)

k iterations

O(kn)


General Rules (3/3) Rule 3: Selection statements

For the fragment

if (condition) S1;else S2;

the running time of an if-else statement is never more than the running time of the condition test plus the larger of the running times of S1 and S2.

if (…) { …;}else { …;}

m statements

max{m, n}

n statements


Worst-case Analysis We may analyze an algorithm/code based on the best-

case, average-case and worst-case scenarios.

Average-case and worst-case analysis are usually better indicators of performance than best-case analysis.

Worst-case is usually easier to determine than average-case.


Running Time of Some Known Algorithms The following are worst-case running time of some

known algorithms on arrays. The problem size, n, is the number of elements in the array.

Sequential search (linear search) in an array: O(n). Binary search in a sorted array: O(lg n). Simple sorts (bubblesort, selection sort, insertion sort): O(n2). Mergesort: O(n lg n).


Sequential Search vs Binary Search (1/2)

Sequential/linear search: Start from first element, visit each element to see if it matches the search item.

public static int linearSearch(int[] list, int searchValue) { for (int i = 0; i < list.length; i++) { if (list[i] == searchValue) return i; } return -1; }


Sequential Search vs Binary Search (2/2)

Binary search: Works for sorted array. Examine middle element, and eliminate half of the array.

public static int binarySearch(int[] list, int searchValue) { int left = 0; int right = list.length - 1; int mid;

while (left <= right) { mid = (left + right)/2; if (list[mid] == searchValue) return mid; else if (list[mid] < searchValue) left = mid + 1; else right = mid - 1; } return -1; }


Analysis of Sequential Search Assume:

An array with n elements. Basic operation is the comparison operation.

Best-case: When the key is found at the first element. Running time: O(1).

Worst-case: When the key is found at the last element, or when the key is not

found. Running time: O(n).

Average case: Assuming that the chance of every element that matches the key

is equal, then on average the key is found after n/2 compare operations. Running time: O(n).


Analysis of Binary Search Assume:

An array with n elements. Basic operation is the comparison operation.

Best-case: When the key is found at the middle element. Running time:

O(1).

Worst-case: Running time: O(lg n). Why? If you start with the value n, how many times can you half it until

it becomes 1? Examples: Starting with 8, it takes 3 halving to get it to 1; starting

with 32, it takes 5 halving; starting with 1024, it takes 10 halving.


Analysis of Sort Algorithms For comparison-based sorting algorithms, the basic

operations used in analysis is The number of comparisons, or The number of swaps (exchanges).

Worst-case analysis: All the three basic sorts – selection sort, bubble sort, and

insertion sort – have worst-case running time of O(n2), where n is the array size.

What is the worst-case scenario for bubble sort? For selection sort? For insertion sort?


Maximum Subsequence Sum (1/6)

Given this problem: Given (possibly negative) integers a0, a1, a2, …, an-1,

find the maximum value of

(For convenience, the maximum subsequence sum is 0 if all the integers are negative.)

Example:list: -2, 11, -4, 13, -5, -2

answer: 20 (a1 through a3).

Many algorithms to solve this problem.

j

ik ka


Maximum Subsequence Sum (2/6) Algorithm 1

public static int maxSubseqSum(int[] list) { int thisSum, maxSum;

maxSum = 0; for (int i = 0; i < list.length; i++)

for (int j = 0; j < list.length; j++) {

thisSum = 0; for (int k = i; k <= j; k++) thisSum += list[k]; // count this line

if (thisSum > maxSum) maxSum = thisSum; }

return maxSum; }


Maximum Subsequence Sum (3/6) Algorithm 1: Analysis

How many times is line thisSum += list[k]; executed?

)(

6

23

)1(

1

3

23

1 2)2)(1(

1

1

nO

nnn

ij

n

i

inin

n

i

n

ij

n

i

n

ij

j

ik



+


Maximum Subsequence Sum (5/6) Algorithm 2: Analysis

Algorithm 2 avoids the cubic running time O(n3) by removing the inner-most for-k loop in algorithm 1.

New running-time complexity is O(n2).

)(

2

1

)1(

1

2

2

2)1(2

111

1

1

nO

nn

nn

in

in

nn

n

i

n

i

n

i

n

i

n

i

n

ij



+


End of file

analysis of algorithms aaron tan tantc/cs1101.html

Documents