analysis of algorithms asymptotic performance. review: asymptotic performance asymptotic...
TRANSCRIPT
Analysis of Algorithms
Asymptotic Performance
Review: Asymptotic Performance
• Asymptotic performance: How does algorithm behave as the problem size gets very large?
o Running time
o Memory/storage requirements
Remember that we use the Hypothetical machine (Random Access Machine):
o All memory equally expensive to access
o No concurrent operations
o All reasonable instructions take unit time (ci)
Review: Running Time
• Number of primitive steps that are executed Except for time of executing a function call most
statements roughly require the same amount of time
We can be more exact if need be
Example
int a, b, larger; c1
scanf(“%d %d”, &a, &b); c2
larger = b; c3
if (a > b) c4
larger = a; c5
printf(“Larger number is %d\n”, larger); c6
Running time T=c1+c2+c3+c4+c5+c6=C (constant)
Sum of first N natural numbers
void main() { c1 int N, count, sum; c2 scanf (“%d”, &N) ; c3 sum = 0; c4 count = 1; c5 while (count <= N) { c6 sum = sum + count; c7 count = count + 1; c8 } printf (“Sum = %d\n”, sum) ; c9}
T(n)=c1+c2+c3+c4+c5+c6*(n+1)+c7*n+c8*n+c9 =C’+c6*(n+1)+c7*n+c8*n
2-D Figure: with for loop
* * * * *
* * * * *
* * * * *
....for (row=1; row<=n; ++row) { (n+1) for (col=1; col<=n; ++col) { n*(n+1) printf(“* ”); n*n } printf(“\n”); n}
Problem A
Algorithm 1 Algorithm 2
T1(n)=n+nn+……….
T2(n)=n+n2+6/n……….
Which algorithm is better to solve problem A?
T(n)=> unknown function
Map this to a known function g(n)
Behavior of g(n) is known to us
Known function g(n)
0
250
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
n^32^n
n^2
nlogn
nlogn
Growth function g(n)
Upper Bound Notation
Read O as “Big-O” (you’ll also hear it as “order”)
• In general a functionf(n) is O(g(n)) if there exist positive constants c and n0 such that f(n) c g(n) for all n n0
• FormallyO(g(n)) = { f(n): positive constants c and n0 such that f(n) c g(n) n n0
11
f(n) = O( g(n))n
n0
f(n)
cg(n)
There exist positive constants csuch that there is a positive constant n0
such that …
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 10 40 20
1 2 3 4
i = j = key = A[j] = A[j+1] =
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 10 40 20
1 2 3 4
i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 0 key = 10A[j] = A[j+1] = 30
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
30 30 40 20
1 2 3 4
i = 2 j = 0 key = 10A[j] = A[j+1] = 30
David Luebke 19 04/21/23
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 2 j = 0 key = 10A[j] = A[j+1] = 10
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 0 key = 10A[j] = A[j+1] = 10
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 0 key = 40A[j] = A[j+1] = 10
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 0 key = 40A[j] = A[j+1] = 10
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
David Luebke 28 04/21/23
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 20
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 40 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 30 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 20 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20
An Example: Insertion Sort
InsertionSort(A, n) {for i = 2 to n {
key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {
A[j+1] = A[j]j = j - 1
}A[j+1] = key
}
}
10 20 30 40
1 2 3 4
i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20
Done!
Analysis of Insertion Sort
INSERTION-SORT(A) cost times 1. for j = 2 to length[A] c1 n2. do key A[j] c2 n-13. //insert A[j] to sorted sequence A[1..j-1] 0 n-14. i j-1 c4 n-15. while i >0 and A[i]>key c5 j=2
n tj
6. do A[i+1] A[i] c6 j=2n(tj –1)
7. i i-1 c7 j=2n(tj –1)
8. A[i+1] key c8 n –1
(tj is the number of times the while loop test in line 5 is executed for that value of j)The total time cost T(n) = sum of cost times in each line =c1n + c2(n-1) + c4(n-1) + c5j=2
n tj+ c6j=2n (tj-1)+ c7j=2
n (tj-1)+ c8(n-1)
Analyzing Insertion Sort
• What can T be? Best case -- inner loop body never executed Worst case -- inner loop body executed for all
previous elements Average case
o ???
44
Analysis of Insertion Sort (cont.)
• Best case cost: already ordered numbers tj=1, and line 6 and 7 will be executed 0 times T(n) = c1n + c2(n-1) + c4(n-1) + c5(n-1) + c8(n-1)
=(c1 + c2 + c4 + c5 + c8)n – (c2 + c4 + c5 + c8) = cn + c‘
• Worst case cost: reverse ordered numbers
tj=j, so j=2
n tj = j=2n j =n(n+1)/2-1, and j=2
n(tj –1) = j=2n(j –1) = n(n-1)/2, and
T(n) = c1n + c2(n-1) + c4(n-1) + c5(n(n+1)/2 -1) + + c6(n(n-1)/2 -1) + c7(n(n-1)/2)+ c8(n-1) =((c5 + c6 + c7)/2)n2 +(c1 + c2 + c4 +c5/2-c6/2-c7/2+c8)n-(c2 + c4 + c5 + c8) =an2+bn+c
• Average case cost: random numbers in average, tj = j/2. T(n) will still be in the order of n2, same as the worst case.
Insertion Sort Is O(n2)
• Proof Suppose runtime is an2 + bn + c
o If any of a, b, and c are less than 0 replace the constant with its absolute value
an2 + bn + c (a + b + c)n2 + (a + b + c)n + (a + b + c)
3(a + b + c)n2 for n 1 Let c’ = 3(a + b + c) and let n0 = 1
Big O Fact
• A polynomial of degree k is O(nk)
• Proof: Suppose f(n) = bknk + bk-1nk-1 + … + b1n + b0
o Let ai = | bi |
f(n) aknk + ak-1nk-1 + … + a1n + a0
ki
kk
i
ik cnan
n
nan
1. g(n) = n and f(n) = 2n + 3 f(n)=O(n)?
c=3, n0=3
2. 100n = O(n2)? c=1, n0=100
3. n2=O(100n)?
n2<c*100n
100c>n can not fix c!!!
Linear search
// Searches an unordered array of integers
int search(int data[], int n, int value){ /
for(int index = 0; index < n; index++){
if(data[index] == value)
return index;
}
return -1;
}T(n)=O(n)
Bubble sort
void bubbleSort ( int A[] , int n )
{
for (i=0; i<n; i++)
{
for (j=i+1; j<n; ++j) [n, n-1,….,2]
{
if (A[j] > A[j+1]) [n-1, n-2,….,1]
{
t = A[j]; A[j] = A[j+1]; A[j+1] = t;
}
}
}
}
T(n)=c1[n+(n-1)+(n-2)+…2]+c2[(n-1)+(n-2)..+1]=an2+bn+c=O(n2)
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lo himid
Ex. Binary search for 33.
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lo hi
Ex. Binary search for 33.
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lo mid hi
Ex. Binary search for 33.
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lo hi
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lo himid
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lohi
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lohi
mid
Binary Search
821 3 4 65 7 109 11 12 14130
641413 25 33 5143 53 8472 93 95 97966
lohi
mid
Non recursive
int binSearch ( int A[] , int n , int x )
{
int L, R, M;
L = 0; R = n-1; c1
while (L < R) (k+1)*c2
{
M = (L + R) / 2; c3*k
if (x > A[M]) c4*k
L = M+1; c5*k
else R = M; c6*k
}
return (A[L] == x); c7
}
n=2k
T(n)=c1+(k+1)*c2+(c3+c4+c5+c6)*k+c7 =k*C’+C’’=C’*log n+C’’=O(log n)
Recursive function
• Running time can be expressed as the recurrence relation
• First compute the recurrence relation
• Next, solve the recurrence relation using substitution method
Recursive
int binary(int a[],int n,int m,int l,int u){
int mid,c=0;
if(l<=u){
mid=(l+u)/2;
if(m==a[mid]){
c=1;
}
else if(m<a[mid]){
return binary(a,n,m,l,mid-1);
}
else
return binary(a,n,m,mid+1,u);
}
else
return c;
}
T(n)=T(n/2)+c
Recurrence relation
• Solving recurrences Substitution method
T(n/2)=T(n/4)+c
T(n/4)=T(n/8)+c
…………..
T(n/2(k-1))=T(n/2k)+c=T(1)+c
T(n)=T(n/2)+c
n=2k
T(n)=c+c+c…….+c + T(1)
=kc+c1=c*log(n)+c1=O(log(n))
k times
Recursive
Recurrence relation
Factorial
int factorialIter ( int n ) cost times
{
int prod, i; c1 1
if (n <= 1) c2 1
return 1; c3 1
prod = 1; c4 1
for (i=2; i<=n; ++i) c5 n
prod *= i; c6 n-1
return prod; c7 1
}
T(n)=c1+c2+..+n*c5+(n-1)*c6+c7=an+b=O(n)
Recursive version
int factorialRec ( int n )
{ int x;
if (n <= 1)
return 1;
x=n * factorialRec(n-1);
return x
}
T(0)=C1;
T(1)=C1
T(n)= C2+T(n-1) n>1
Recurrence relation
Solving recurrences
T(n)= C2+T(n-1)
T(n-1)=C2+T(n-2)
T(n)=C2+C2+T(n-2)
T(n-2)=C2+T(n-3)
T(n)=C2+C2+T(n-2)
=C2+C2+C2+T(n-3)
After n-1 number of steps
T(n)=(n-1)C2+T(1)
=(n-1)C2+C1
=an+b
=O(n)
Mergesort
Basic Idea
• Divide the array into two halves• Sort the two sub-arrays• Merge the two sorted sub-arrays into a single sorted
array• Step 2 (sorting the sub-arrays) is done recursively
(divide in two, sort, merge) until the array has a single element (base condition of recursion)
67
Merging Two Sorted Arrays
3 4 7 8 9
2 5 7
2
3 4 7 8 9
2 5 7
2 3
3 4 7 8 9
2 5 7
2 3 4
3 4 7 8 9
2 5 7
Problem: Two sorted arrays A and B are given. We are required to
produce a final sorted array C which contains all elements of A and B.
68
2 3 4 5
3 4 7 8 9
2 5 7
2 3 4 5 7
3 4 7 8 9
2 5 7
2 3 4 5 7 7
3 4 7 8 9
2 5 7
2 3 4 5 7 7 8
3 4 7 8 9
2 5 7
69
Merge Code
2 3 4 5 7 7 8 9
3 4 7 8 9
2 5 7
void
merge (int A[], int B[], int C[], int m,int n)
{
int i=0,j=0,k=0;
while (i<m && j<n)
{
if (A[i] < B[j]) C[k++] = A[i++];
else C[k++] = B[j++];
}
while (i<m) C[k++] = A[i++];
while (j<n) C[k++] = B[j++];
}
70
Merge Sort: Sorting an array recursively
void mergesort (int A[], int n)
{
int i, j, B[max];
if (n <= 1) return;
i = n/2;
mergesort(A, i);
mergesort(A+i, n-i);
merge(A, A+i, B, i, n-i);
for (j=0; j<n; j++) A[j] = B[j];
free(B);
}
T(1) = C1, T(n) = 2 T(n/2) + cn + d
Recurrence relation
Solving recurrences
T(n) = 2 T(n/2) + cn + d
T(n/2)=2T(n/4)+cn/2+d
T(n)=2(2T(n/4)+cn/2+d)+cn+d
=4T(n/4)+cn+2d+cn+d
=4T(n/4)+2cn+3d
After k steps
T(n)=2kT(n/2k) + kcn+C’d
=nT(1)+nlog(n)+C2’
=C1*n+nlog(n)+C2’=O(nlogn)
n=2k