eleg667/2001-f/topic-1a 1 a brief review of algorithm design and analysis

eleg667/2001-f/Topic-1a 1

A Brief Review of

Algorithm Design and Analysis


Outline

Basic concepts; Case study; Analyzing algorithms Comparing algorithms Problem complexity


Algorithm - Definition

Sequence of steps to carry out a particular task

Examples: Recipe; Assembly instructions; ……


Characteristics of Algorithms

Algorithms are precise. Each step has a clearly defined meaning;

Algorithms are effective. The task is always done as required;

Algorithms have a finite number of steps; Algorithms must terminate.


Any algorithm can be stated using the following logic constructs

Sequence: steps must follow each other in a logical sequence;

Decision: there may be alternative steps that may be taken subject to a particular condition;

Repetition: certain steps may be repeated while, or until, a certain condition is true.


Why study algorithms?

A motivating example…


Example: sorting playing cards

Start with one card on the left hand and

the others cards facing down on the table;

while there are cards left on the table do

{ take one card from the table;

compare it with each of the cards

already in the hand, from left to right,

inserting it into the right position;

}

Sequence

Decision

Repetition

Insertion Sort


Expressing computer algorithms

It should be expressed in a language more precise, less ambiguous, and more compact than a “natural language” such as English;

Algorithms are usually written in a pseudocode language and later translated to a real programming language.


Insertion Sort – An Intuitive Picture

j Q K5

…

Q

i

1 1

5252

key

A A1

…


Insertion Sort in Pseudocode

Insertion-Sort(A) ; A is an array of numbers for j = 2 to length(A) key = A[j] i = j - 1 while i > 0 and A[i] > key A[i+1] = A[i] i = i - 1 A[i+1] = key

SequenceRepetition

Decision


Algorithms and Machine Models

Sequential algorithms and machine

models

Parallel algorithms and machine models


Analyzing algorithms

Estimate the running time of algorithms;

= F(Problem Size)

= F(Input Size)

= number of primitive operations used (add, multiply, compare etc)


Analysis for Insertion Sort

Insertion-Sort(A) Cost Times (Iterations) 1 for j = 2 to length(A) { c1

2 key = A[j] c2

3 i = j – 1 c3 4 while i > 0 and A[i] > key c4

5 A[i+1] = A[i] c5

6 i = i – 1 c6

7 A[i+1] = key c7 }

n

n - 1

n - 1

n - 1


Insertion Sort Analysis (cont.)

Best Case:

Array already sorted, tj = 1 for all j



Worst Case:

Array in reverse order, tj = j for all j

Note that



Average Case:

Check half of array on average,

tj = j/2 for all j

T(n) = a.n2 + bn + c (quadractic in n)

We are usually interested in the worst-case running timeWe are usually interested in the worst-case running time


Growth rates

constant growth rate: T(n) = c linear growth rate : T(n) = c*n logarithmic growth rate : T(n) = c*log n quadratic growth rate : T(n) = c*n2

exponential growth rate : T(n) = c*2n


Asymptotic Analysis

Ignoring constants in T(n) Analyzing T(n) as n "gets large"

The running time grows roughly on the order of

Notationally, =

" n

T n O n

3

3

"

( ) ( )

As grows larger, is MUCH large than , , and ,so it dominates

n n n n nT n

n 23 log( )

T n n n n n n( ) log 13 42 2 43 2Example:

The big-oh (O) Notation


Formally…

T(n) = O(f(n)) if there are constants c and n such that T(n) < c.f(n) when n n0

c.f(n)

T(N)

n0 n

f(n) is an upper bound for T(n)


O ( big-oh)

Describes an upper bound for the running time of an algorithm

Upper bounds for Insertion Sort running times:

•worst case: O(n2) T(n) = c1.n2 + c2.n + c3

•best case: O(n) T(n) = c1.n + c2

•average case: O(n2) T(n) = c1.n2 + c2.n + c3

Time Complexity


Some properties of the O notation

Fastest growing function dominates a sum O(f(n)+g(n)) is O(max{f(n), g(n)})

Product of upper bounds is upper bound for the product If f is O(g) and h is O(r) then fh is O(gr)

f is O(g) is transitive If f is O(g) and g is O(h) then f is O(h)

Hierarchy of functions O(1), O(logn), O(n1/2), O(nlogn), O(n2), O(2n), O(n!)


Times on a 1-billion-steps-per-second computer


Simple statement sequence s1; s2; …. ; sk

O(1) as long as k is constant

Simple loops for(i=0;i<n;i++) { s; } where s is O(1) Time complexity is n O(1) or O(n)

Analyzing an Algorithm


Analyzing an Algorithm (cont.)

Nested loops for(i=0;i<n;i++) for(j=0;j<n;j++) { s; }

Complexity is n O(n) or O(n2)

Recursion Search(n) if middle_element = key return it

else if it’s greater search_left(n/2) else search_right(n/2) Solve recurrence equation: T(n) = T(1) + T(n/2) Complexity is O(log2n)


Reasonable and Unreasonable Algorithms

Polynomial Time algorithms An algorithm is said to be polynomial if it is

O( nc ), c >1 Polynomial algorithms are said to be reasonable

They solve problems in reasonable times!

Exponential Time algorithms An algorithm is said to be exponential if it is

O( rn ), r > 1 Exponential algorithms are said to be unreasonable


Another example: Quick Sort

Uses recursion to repetitively divide list in 1/2 and sort the two halves

QuickSort partitions the list into a left and a right sublist, where all values in the left sublist are less than all values in the right sublist. Then the QuickSort is applied recursively to the sublists until each sublist has only one element.


Quick Sort (cont.)

9 1 25 4 15 4 1 9 25 15

becomes

4 1

25 15 becomes

becomes 1

4 15 25

1 4 15 25


Quick Sort (cont.)

QuickSort (list(0,n)):

If length (list) > 1 then

partition into 2 sublists from

0..split-1 and from split + 1..n

QuickSort (list (0,split-1))

QuickSort (list (split + 1, n)


1. Pick an array value, pivot.2. Use two indices, i and j.

a. Begin with i = left and j =right + 1b. As long as i is less than j do 3 steps:

i. Keep increasing i until we come to an element A[i] >= pivot.ii. Keep decreasing j until we come to an element A[j] <= pivot.iii. Swap A[i] and A[j].

Quick Sort – Partition phase


The Operation of Quicksort


Quick Sort – Complexity

Assume the list size (n) is a power of 2 and that pivot is always chosen such that it divides the list into two equal pieces

At each step we are cutting in half the size to be sorted

O(nlog2n)


Quick Sort demo…


Comparing the Sort Algorithms…


Problem complexity

problems

sort

. . .Insertion Shellsort QuicksortAlgorithms:


Definitions

A problem’s upper bound refers to the best that we know how to do. It is determined by the best algorithmic solution that we have found for a given problem.

Example: For the sorting problem O(n.logn) is the upper bound.


Definitions (cont.)

A problem’s lower bound refers to the best algorithmic solution that is theoretically possible.

Example: It can be proved that sorting an unsorted list cannot be done in less than O(n.log n), unless we make assumptions about the particular input data. Thus O(n.log n) is the lower bound for sorting problems.


Definitions (cont.)

Upper bounds tell us the best we’ve been able to do so far;

Lower bounds tell us the best we can hope to do.

For a given problem, if the upper and lower bounds are the same, then we refer to that problem as a closed problem.


Tractable and Intractable problems

Tractable problems have both upper and lower bounds polynomial – O(logn), O(n), O(n.logn), O(nk);

Intractable problems have both upper and lower bounds exponential - O(2n), O(n!), O(nn);


Problems that cross the line

We have found only exponential solutions (i.e. from the standpoint of our algorithms, it appears to be intractable);

We cannot prove the necessity of an exponential solution (i.e. from the standpoint of our proofs, we cannot say that it is intractable).


NP-complete problems

The upper bound suggests that the problem is intractable;

No proof exists that shows these problems are intractable;

Once a solution is found, it can be checked in polynomial time;

They are all reducible to one another;


Example

Traveling salesman: give a weighted graph (e.g., cities with distances between them) and some distance k (e.g., the maximum distance you want to travel), is there some tour that visits all the points (e.g., cities) and returns home such that distance k?

…


P = NP ?

Win the Turing Award

and

get a $1 million prize!http://www.claymath.org/prize_problems/p_vs_np.htm

eleg667/2001-f/topic-1a 1 a brief review of algorithm design and analysis

Documents

j slide

analysis slide

a1a1 slide

study algorithms

ai key c

1a11 algorithms

required algorithms

computer algorithms