Software DesignAnalysis of Algorithms

i206Fall 2010

John ChuangSome slides adapted from Glenn Brookshear, Marti Hearst, or Goodrich & Tamassia

John Chuang 2

Analysis of Algorithms

Bits & BytesBinary Numbers

Number Systems

Gates

Boolean Logic

Circuits

CPU Machine Instructions

Assembly Instructions

Program Algorithms

Application

Memory

Data compression

Compiler/Interpreter

OperatingSystem

Data Structures

Analysis

I/O

Memory hierarchy

Design

Methodologies/Tools

Process

Truth tableVenn DiagramDeMorgan’s Law

Numbers, text,audio, video, image, …

Decimal, Hexadecimal, Binary

AND, OR, NOT, XOR, NAND, NOR,etc.

Register, CacheMain Memory,Secondary Storage

Context switchProcess vs. ThreadLocks and deadlocks

Op-code, operandsInstruction set arch

Lossless v. lossyInfo entropy & Huffman code Adders, decoders,

Memory latches, ALUs, etc.

DataRepresentation

Data

Data storage

Principles

ALUs, Registers,Program Counter, Instruction Register

Network

Distributed Systems Security

Cryptography

Standards & Protocols

Inter-processCommunication

Searching, sorting,Encryption, etc.

Stacks, queues,maps, trees, graphs, …

Big-O

UML, CRC

TCP/IP, RSA, …

ConfidentialityIntegrityAuthentication…

C/S, P2PCaching

sockets

Formal models

Finite automataregex

John Chuang 3

Traveling Salesman Problem

http://xkcd.com/399/

John Chuang 4

Which Algorithm is Better?What do we mean by “Better”?

Sorting algorithms- Bubble sort- Insertion sort- Shell sort- Merge sort- Heapsort- Quicksort- Radix sort- …

Search algorithms- Linear search- Binary search- Breadth first search

- Depth first search

- …

John Chuang 5

Analysis

Characterizing the running times of algorithms and data structure operations

Secondarily, characterizing the space usage of algorithms and data structures

John Chuang 6

Running Time

In general, running time increases with input size

Running time also affected by:- Hardware environment (processor, clock rate, memory, disk, etc.)

- Software environment (operating system, programming language, compiler, interpreter, etc.) (n)

John Chuang 7

Quantifying Running Time

Experimentally measure running time- Need to fully implement and execute

Analysis of pseudo-code

(n)

John Chuang 8

Pseudo-code Analysis Example

Algorithm: arrayMax(A,n):Input: An array A storing n >= 1 integersOutput: the maximum element in A

currentMax = A[0]for i=1 to (n-1) doif currentMax < A[i] then currentMax = A[i]

Return currentMax

John Chuang 9

The Seven Common Functions

Constant Linear Quadratic Cubic Exponential Logarithmic n-log-n

Polynomial

John Chuang 10

The Seven Common Functions

Constant: f(n) = c- E.g., adding two numbers, assigning a value to some variable, comparing two numbers, and other basic operations

Linear: f(n) = n- Do a single basic operation for each of n elements, e.g., reading n elements into computer memory, comparing a number x to each element of an array of size n

John Chuang 11

The Seven Common Functions

Quadratic: f(n) = n2

- E.g., nested loops with inner and outer loops

- E.g., insertion sort algorithm

Cubic: f(n) = n3

More generally, all of the above are polynomial functions:- f(n) = a0 + a1n + a2n2 + a3n3 + … + annd

- where the ai’s are constants, called the coefficients of the polynomial

Brookshear Figure 5.19

John Chuang 12

The Seven Common Functions

Exponential: f(n) = bn

- Where b is a positive constant, called the base, and the argument n is the exponent

- In algorithm analysis, the most common base is b=2

- E.g., loop that starts with one operation and doubles the number of operations with each iteration

- Similar to the geometric sum:- 1+2+4+8+…+2n-1 = 2n – 1

- BTW, 2n – 1 is the largest integer that can be represented in binary notation using n bits

John Chuang 13

The Seven Common Functions

Logarithmic: f(n) = logbn- Intuition: number of times we can divide n by b until we get a number less than or equal to 1

- E.g., the binary search algorithm has a logarithmic running time

- The base is omitted for the case of b=2, which is the most common in computing:- log n = log2n

Brookshear Figure 5.20

John Chuang 14

The Seven Common Functions

n-log-n function: f(n) = n log n- Product of linear and logarithmic- E.g., quicksort algorithm has n log n running time on average (but n2 in worst case)

John Chuang 15

Comparing Growth Rates

1 < log n < n1/2 < n < n log n < n2 < n3 < bn

http://www.cs.pomona.edu/~marshall/courses/2002/spring/cs50/BigO/

John Chuang 16

Polynomials

Only the dominant terms of a polynomial matter in the long run.  Lower-order terms fade to insignificance as the problem size increases.

http://www.cs.pomona.edu/~marshall/courses/2002/spring/cs50/BigO/

John Chuang 17

Example: Counting Primitive Operations

Algorithm: arrayMax(A,n):Input: An array A storing n >= 1 integersOutput: the maximum element in A

currentMax = A[0]for (i=1; i<=n-1; i++)if currentMax < A[i] then currentMax = A[i]

Return currentMax

Two operations: indexing into array; assign value to variable

Two operations repeated n times: subtract, compare

One operation: assign value to variable

Four or six operations repeated (n-1) times: - index and compare (for the if statement); - index and assign (for the then statement if necessary)- addition and assign (for the increment)

One operation: return value of variable

Total: 2 + 1 + 2n + 4(n-1) + 1 = 6n (best case)Total: 2 + 1 + 2n + 6(n-1) + 1 = 8n – 2 (worst case)

John Chuang 18

Big-O Notation

Asymptotic analysis- As n becomes large and approaches infinity

Let f(n) and g(n) be functions mapping non-negative integers to real numbers

Then f(n) is O(g(n)) or “f(n) is big-Oh of g(n)”

If there is a real constant c > 0 and an integer constant n0 >=1 such that- f(n) <= cg(n) for n >= n0

Example: the function f(n)=8n–2 is O(n) The running time of algorithm arrayMax is O(n)

John Chuang 19

Example 1: Search

Given:- A physical phone book

- Organized in alphabetical order- A name you want to look up- An algorithm in which you search through the book sequentially, from first page to last: Sequential Search or Linear Search

- What is the order of:- The best case running time?- The worst case running time?- The average case running time?

- What is:- A better algorithm?- The worst case running time for this algorithm?

John Chuang 20

Example 1: Search

This better algorithm is called Binary Search

What is its running time?- First you look in the middle of n elements- Then you look in the middle of n/2 = ½*n elements- Then you look in the middle of ½ * ½*n elements- …- Continue until you find the target element, or until there is only 1 element left

- Say you did this m times: ½ * ½ * ½* …*n- Then the number of repetitions is the smallest integer m such that 1

2

1=nm

John Chuang 21

Binary Search Pseudo-codeBrookshear Figure 5.14

John Chuang 22

Analyzing Binary Search- In the worst case, the number of repetitions is the smallest integer m such that

- We can rewrite this as follows:

mn

n

n

m

m

==

=

log

2

121

Multiply both sides by m2

Take the log of both sides

Since m is the worst case time, the algorithm is O(log n)

12

1=nm

John Chuang 23

Example 2: Insertion Sort is O(n2)

Brookshear Figure 5.19

Brookshear Figure 5.10

Brookshear Figure 5.11

John Chuang 24

Example 3: Prefix Average

Compute running average of a sequence of numbers

5 10 15 20 25 30

5/1 15/2 30/3 50/4 75/5 105/6

There are two straightforward algorithms:The first is easy but wasteful.The second is more efficient, but requires insight into the problem.

j=0 X[j]i

i+1A[i] =

John Chuang 25

1st Algorithm

Algorithm: prefixAverages1(X):Input: An n-element array X of numbersOutput: An n-element array A of numbers such that A[i] is the average of elements X[0],…X[i]

Let A be an array of n numbersFor i 0 to n-1 doa 0For j 0 to i do

a a+X[j]A[i] a/(i+1)

Return array A

j=0 X[j]i

i+1A[i] =

John Chuang 26

2nd Algorithm

Algorithm: prefixAverages2(X):Input: An n-element array X of numbersOutput: An n-element array A of numbers such that A[i]

is the average of elements X[0],…X[i]

Let A be an array of n numberss 0For i 0 to n-1 do

s s + X[i]A[i] s/(i+1)

Return array A

A useful tool: store intermediate results in a variable!Uses space to save time. The key – don’t divide s.Eliminates one for loop – always a good thing to do.

John Chuang 27

Summary: Analysis of Algorithms

A method for determining the asymptotic running time of an algorithm- Here asymptotic means “as n gets very large”

Useful for comparing algorithms Useful also for determining tractability- E.g., Exponential time algorithms are usually intractable.

John Chuang 28

Traveling Salesman Problem

http://xkcd.com/399/