![Page 1: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/1.jpg)
Data Structures and Algorithms
Lecture 1: Asymptotic Notations
![Page 2: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/2.jpg)
Basic Terminologies
• Algorithm– Outline– Essence of a computational procedure– Step by step instructions
• Program– Implementation of an Algorithm in some programming
language• Data Structure– Organization of Data needed to solve the problem
effectively– Data Structure you already know: Array
![Page 3: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/3.jpg)
Algorithmic Problem
Specification of Input
?Specification
of Output as a function of
Input
• Infinite number of input instances satisfying the specification
• E.g.: Specification of Input:• A sorted non-decreasing sequence of natural numbers
of non-zero, finite length:• 1, 20,908,909,1000,10000,1000000• 3
![Page 4: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/4.jpg)
Algorithmic Solution
• Algorithm describes the actions on the input instances
• Infinitely many correct algorithms may exist for the same algorithmic problem
Input instance adhering to
Specification
Algorithm
Output related to the
Input as required
![Page 5: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/5.jpg)
Characteristics of a Good Algorithm
• Efficient– Small Running Time– Less Memory Usage
• Efficiency as a function of input size– The number of bits in a data number– The number of data elements
![Page 6: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/6.jpg)
Measuring the running time
• Experimental Study– Write a program that implements the algorithm– Run the program with data sets of varying size and
composition– Use system defined functions like clock() to get an
measurement of the actual running time
![Page 7: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/7.jpg)
Measuring the running time
• Limitations of Experimental Study– It is necessary to implement and test the
algorithm in order to determine its running time.– Experiments can be done on a limited set of
inputs, and may not be indicative of the running time on other inputs not included in the experiment
– In order to compare 2 algorithms same hardware and software environments must be used.
![Page 8: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/8.jpg)
Beyond Experimental Study
• We will develop a general methodology for analyzing the running time of algorithms. This approach– Uses a high level description of the algorithm
instead of testing one of its implementations– Takes into account all possible inputs– Allows one to evaluate the efficiency of any
algorithm in a way that is independent of the hardware and software environment.
![Page 9: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/9.jpg)
Pseudo - Code
• A mixture of natural language and high level programming concepts that describes the main ideas behind a generic implementation of a data structure or algorithm.
• E.g. Algorithm arrayMax(A,n)– Input: An array A storing n integers– Output: The maximum element in A. currMax A[0] for i 1 to n-1 do if currMax < A[i] then currMax A[i] return currMax
![Page 10: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/10.jpg)
Pseudo - Code
• It is more structured that usual prose but less formal than a programming language
• Expressions:– Use standard mathematical symbols to describe
numeric and Boolean expressions– Use ‘ ’ for assignment instead of “=” – Use ‘=’ for equality relationship instead of “==”
• Function Declarations:– Algorithm name (param1 , param2)
![Page 11: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/11.jpg)
Pseudo - Code
• Programming Constructs:– Decision structures: if…then….[else….]– While loops: while…..do…..– Repeat loops: Repeat………until…….– For loop: for……do………..– Array indexing: A[i], A[I,j]
• Functions:– Calls: return_type function_name(param1 ,param2)– Returns: return value
![Page 12: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/12.jpg)
Analysis of Algorithms
• Primitive Operation: – Low level operation independent of programming
language. – Can be identified in a pseudo code.
• For e.g. – Data Movement (assign)– Control (branch, subroutine call, return)– Arithmetic and Logical operations (e.g. addition
comparison)– By inspecting the pseudo code, we can count the number
of primitive operations executed by an algorithm.
![Page 13: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/13.jpg)
Example: Sorting
Sort
INPUTSequence of numbers
OUPUTPermutation of the sequence of numbers in non-decreasing order
a1,a2,a3,a4,….an b1,b2,b3,b4,….bn
2, 5 , 4, 10, 7 2, 4 , 5, 7, 10Correctness (Requirements for the output)For any given input the algorithm halts with the output:• b1 < b2 < b3 < b4 < ….. < bn
• b1, b2, b3, b4,…. bn is a permutation of a1, a2, a3, a4, …. an
Running Time depends on• No. of elements (n)• How sorted (partial) the numbers are? • Algorithm
![Page 14: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/14.jpg)
Insertion Sort• Used while playing cards
3 4 6 8 9 7 2 5 1A1 n
i jStrategy• Start “empty handed”• Insert a card in the right position of the already sorted hand• Continue until all cards are inserted or sorted
INPUT: An array of Integers A[1..n] OUTPUT: A permutation of A such that A[1] <= A[2] <= A[3]<= ..<=A[n]For j 2 to n do
key A[j]Insert A[j] into sorted sequence A[1…j-1]i j-1while i >0 and A[i] >key
do A[i+1] A[i]i- -
A[i+1] key
![Page 15: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/15.jpg)
Total Time = n(c1 + c2 + c3 + c7) + (c4 + c5 + c6)–( c2 +c3 +c5+c6+c7)
tj counts the number of times the values have to be shifted in one iteration
Analysis of Insertion Sort
• For j 2 to n do• key A[j]• Insert A[j] into sorted
sequence A[1…j-1]• i j-1• while i >0 and A[i]
>key• do A[i+1] A[i]• i- - • A[i+1] key
Cost Times c1 n c2 n-1
c3 n-1
c4 c5 c6 c7 n-1
![Page 16: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/16.jpg)
Best / Average/ Worst Case: Difference made by tj
• Total Time = n(c1 + c2 + c3 + c7) + (c4 + c5 + c6)–( c2 +c3 +c5+c6+c7)
• Best Case– Elements already sorted– tj = 1– Running time = f(n) i.e. Linear Time
• Worst Case– Elements are sorted in inverse order– tj = j– Running time = f(n2) i.e. Quadratic Time
• Average Case– tj = j/2– Running Time = f(n2) i.e. Quadratic Time
![Page 17: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/17.jpg)
Best / Average/ Worst Case
• For a Specific input size say n
Input instance
Runn
ing
Tim
e (s
ec)
A B C D E F G H I J K L M N
5
4
3
2
1
Best Case
Worst Case
Average Case
![Page 18: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/18.jpg)
Best / Average/ Worst Case
• Varying input size
Input Size
Runn
ing
Tim
e (s
ec)
50
40
30
20
10
10 20 30 40 50 60
Worst case
Average Case
Best Case
![Page 19: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/19.jpg)
Best / Average/ Worst Case
• Worst Case:– Mostly used– It is an Upper bound of how bad a system can be.– In cases like surgery or air traffic control knowing
the worst case complexity is of crucial importance.• Average case is often as bad as worst case.• Finding an average case can be very difficult.
![Page 20: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/20.jpg)
Asymptotic Analysis
• Goal: Simplify analysis of running time by getting rid of details which may be affected by specific implementation and hardware– Like rounding 1000001 to 1000000– 3n2 to n2
• Capturing the essence: How the running time of an algorithm increases with the size of the input in the limit– Asymptotically more efficient algorithms are best for
all but small inputs.
![Page 21: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/21.jpg)
Asymptotic Notation
• The “big Oh” (O) Notation– Asymptotic Upper Bound– f(n) = O(g(n)), if there exists
constants c and n0 such that f(n) ≤ cg(n) for n ≥ n0
– f(n) and g(n) are functions over non negative non-decreasing integers
• Used for worst case analysis
n0f(n) = O(g(n))
n
f(n)
c. g(n)
![Page 22: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/22.jpg)
Examples• E.g. 1:
– f(n) = 2n + 6– g(n) = n– c = 4– n0 = 3– Thus, f(n) = O(n)
• E.g. 2:– f(n) = n2
– g(n) = n– f(n) is not O(n) because there exists no
constants c and n0 such that f(n) ≤ cg(n) for n ≥ n0
• E.g. 3:– f(n) = n2 + 5– g(n) = n2
– c = 2– n0 = 2– Thus, f(n) = O(n2)
![Page 23: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/23.jpg)
Asymptotic Notation
• Simple Rules:– Drop lower order terms and constant factors• 50 nlogn is O(nlogn)• 7n – 3 is O(n)• 8n2logn + 5n2 + n is O(n2logn)
– Note that 50nlogn is also O(n5) but we will consider it to be O(nlogn) it is expected that the approximation should be of as lower value as possible.
![Page 24: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/24.jpg)
Comparing Asymptotic Analysis of Running Time
• Hierarchy of f(n)– Log n < n < n2 < n3 < 2n
• Caution!!!!!– An algorithm with running time of 1,000,000 n is
still O(n) but might be less efficient than one running in time 2n2, which is O(n2) when n is not very large.
![Page 25: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/25.jpg)
Examples of Asymptotic Analysis
• Algorithm: prefixAvg1(X)• Input: An n element array X of numbers• Output: An n element array A of numbers such that
A[i] is the average of elements X[0]…X[i].for i 0 to n-1 do
a 0 for j 0 to i do a a+X[j] A[i] a/(i+1)return A
• Analysis: Running Time O(n2) roughly.
1 Stepi iterations i = 0 , 1, …., i-1
n iterations
![Page 26: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/26.jpg)
Examples of Asymptotic Analysis
• Algorithm: prefixAvg2(X)• Input: An n element array X of numbers• Output: An n element array A of numbers such that A[i]
is the average of elements X[0]…X[i].s 0
for i 0 to n do s s + X[i]
A[i] s/(i+1) return A• Analysis: Running time is O(n)
![Page 27: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/27.jpg)
Asymptotic Notation (Terminologies)
• Special cases of Algorithms– Logarithmic : O(log n)– Linear: O(n)– Quadratic: O(n2)– Polynomial: O(nk), k ≥ 1– Exponential: O(an), a>1
• “Relatives” of Big Oh– Big Omega (Ω(f(n)) : Asymptotic Lower Bound– Big Theta (Θ(f(n)): Asymptotic Tight Bound
![Page 28: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/28.jpg)
Asymptotic Notation
• The big Omega (Ω) Notation– Asymptotic Lower bound– f(n) = Ω(g(n), if there exists
constants c and n0 such that c.g(n) ≤ f(n) for n > n0
• Used to describe best case running times or lower bounds of asymptotic problems– E.g: Lower bound of searching
in an unsorted array is Ω(n).
![Page 29: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/29.jpg)
Asymptotic Notation
• The Big Theta (Θ) notation– Asymptotically tight bound– f(n) = Θ(g(n)), if there exists
constants c1, c2 and n0 such that c1.g(n) ≤ f(n) ≤ c2.g(n) for n > n0
– f(n) = Θ(g(n)), iff f(n) = O(g(n)) and f(n) = Ω(g(n),
– O(f(n)) is often misused instead of Θ(f(n))
![Page 30: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/30.jpg)
Asymptotic Notation
• There are 2 more notations– Little oh notation (o), f(n) = o(g(n))
• For every c there should exist a no. n0 such that f(n) ≤ o(g(n) for n > n0.
– Little Omega notation (ω)• Analogy with real numbers
– f(n) = O(g(n)) , f≤g– f(n) = Ω(g(n)), f≥g– f(n) = Θ(g(n)), f = g– f(n) = o(g(n)), f<g– f(n) = ω(g(n)), f>g
• Abuse of Notation: – f(n) = O(g(n)) actually means f(n) є O(g(n))
![Page 31: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/31.jpg)
Comparison of Running times
![Page 32: asymptotic analysis and insertion sort analysis](https://reader035.vdocument.in/reader035/viewer/2022062220/55619f4ad8b42ae1538b484d/html5/thumbnails/32.jpg)
Assignment
• Write a C program to implement i) Insertion sort and ii) Bubble Sort.
• Perform an Experimental Study and graphically show the time required for both the program to run.
• Consider varied size of inputs and best case and worst case for each of the input sets.