waterfordinstituteoftechnology march1,2016 johnfitzgerald › modules › jf-programming-2016 ›...
TRANSCRIPT
![Page 1: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/1.jpg)
Algorithms and their complexityLecture 12
Waterford Institute of Technology
March 1, 2016
John Fitzgerald
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 1/33
![Page 2: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/2.jpg)
Presentation Outline
Estimated duration presentationQuestions at end presentationTopics discussed:
• Methods to measure program complexity• Example algorithms
• Sorting• Searching
• Relative performance of algorithms
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 2/33
![Page 3: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/3.jpg)
AlgorithmDescription
What is a software algorithm?
• Program that solves problem• Algorithm examples:
• Sorting data• Searching data• Graphs (social networks)• Cryptography• Image processing• From labs:
• Generate PIN• Validate PIN
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 3/33
![Page 4: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/4.jpg)
AlgorithmRequirements
In developing an algorithm we areinterested in:
• Does it produce correct answer for allvalid inputs?
• How long to solve task?• How much computer memory used?• Is code easy to understand & thus
maintain?• Conceptual complexity• Computational complexity
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 4/33
![Page 5: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/5.jpg)
Algorithm outputExpect correct result for all valid inputs
Easier said than doneExamples of incorrect output:
• Timsort• Extensively used on Android• Bug recently discovered
• Binary search• Small & conceptually simple• Yet took years to eliminate all bugs
for all inputs
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 5/33
![Page 6: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/6.jpg)
Algorithm performanceHow long to run?
Goal not measurement run timebecause of variations in
• Computer speeds• Language implementation• Inputs
Instead, determine inherentcomplexity
• Classes of complexity exist• Provide comparison of
growth of numbercomputing operations
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 6/33
![Page 7: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/7.jpg)
Linear search algorithmHow long to run?
Consider the linear search algorithm belowRunning times could vary significantly:
• If String search near start of String[] target• method returns immediately
• But if towards end of list• return might be much later• providing very different run times for same algorithm
boolean linearSearch(String search, String[] target)
for (int i = 0; i < target.length; i += 1)if (target[i].contains(search))
return true;return false;
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 7/33
![Page 8: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/8.jpg)
Algorithm efficiencyChoices available
We have a choice to make:• Best case
• Considering all possible inputs• Linear search runs in constant time• Returns immediately
• Worst case• Search time proportional input size• Linear search is linear in size list
• AverageWe will choose worst case.
• We will seek an upper bound onnumber operations in algorithm.
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 8/33
![Page 9: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/9.jpg)
Counting operationsAgree some rules
Basic assumptions• Concept of random access machine (RAM)• An abstract computer• Measure size of input• Sequential execution operations• Operations take constant time
• Assignment (a = 10)• Comparison (a==10)• Arithmetic (a/10)• Memory access (b[i])
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 9/33
![Page 10: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/10.jpg)
Counting operationsSimple example
Approximate number operations or steps in countOperations:• 2001 + 2n + 2n2
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 10/33
![Page 11: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/11.jpg)
Count operationsNumber steps or operations
2001 + 2n + 2n2 (ignoring for overhead)• If n small output dominated by 2000 term• For large n 2000 term insignificant
• If n is 10 output is 2221• If n is 10,000 output exceeds 200 million (2.0002E+08)
static int countOperations(int n)
int ans = 0;for (int i = 0; i < 1000; i += 1)
ans += 1;for (int i = 0; i < n; i += 1)
ans +=1;for (int i = 0; i < n; i += 1)
for (int j = 0; j < n; j += 1)ans += 1;
return ans;
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 11/33
![Page 12: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/12.jpg)
Algorithm complexityComparing measurement formulae
In the case of 2001 + 2n + 2n2
we use n2
• We use the highest ordervariable
• And disregard any coefficient• Provides approximate upper
bound measurement• Importantly, it provides
representation of growth ofcomplexity as input becomesvery large.
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 12/33
![Page 13: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/13.jpg)
Algorithm complexityUse of asymptotic notation
Last example: Some constant times n2
provides upper bound on numberoperationsAsymptotic notation used to describegrowth of algorithm operations as inputsize approaches infinity.We now introduce Big O notation
• Running time of 2001 + 2n + 2n2
• O(n2) or• Big-O(n2)
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 13/33
![Page 14: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/14.jpg)
Algorithm complexityFurther asymptotic notation
Big Θ (Big Theta)• For large n :
• k · f (n) provides upperbound
• c · f (n) provide lowerbound
• k and c are some constants.
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 14/33
![Page 15: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/15.jpg)
Algorithm complexityFurther asymptotic notation
Big Ω (Big Omega)• For large n, c · f (n) provides
lower bound run time• c is some constant.
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 15/33
![Page 16: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/16.jpg)
Algorithmic complexityAlternative measurement approach
Tilde notation• Proposed by Robert
Sedegwick• Denoted by: ~f(n)• Differs from Big-O
• Coefficient of term withhighest exponent retained
• Consider: f(x) = 2000 + 2n+ 2n2
• ~2n2 is considered thecomplexity of thisalgorithm
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 16/33
![Page 17: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/17.jpg)
Big-O complexityCategories
• Constant O(1) : independent of input• Linear O(n): find smallest item unsorted array• Logarithmic O(log n) : binary search• Linearithmic O(n log n) : mergesort• Polynomial O(nc) , c > 0: selection sort quadratic (c == 2)• Exponential O(2n) : Binary exponential backoff (networking)• Factorial O(n!) : Brute force search travelling salesman
problem
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 17/33
![Page 18: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/18.jpg)
Big-O complexityCategories
• Constant O(1)• Logarithmic O(log n)• Linear O(n)• Linearithmic O(n log n)• Polynomial O(nc)• Exponential O(2n)• Factorial O(n!)
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 18/33
![Page 19: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/19.jpg)
Algorithm complexityAcceptable categories
Very large performance differences between categoriesDoes this really matter?
• For small problems, not really• For very large problems, definitely yes.
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 19/33
![Page 20: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/20.jpg)
Sort algorithmsSelection sort
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 20/33
![Page 21: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/21.jpg)
Sort algorithmsMerge sort
This implementation comprises two methods:• public method sort• private method merge
public sortloop
subdivide array into pairs blocksloop all pairs blocks
invoke merge on each pairendloop
endloop
private mergeSuccessively compare elementsin left and right blocks.Incrementally populate target sorted
array.
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 21/33
![Page 22: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/22.jpg)
Merge sortMethod sort
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 22/33
![Page 23: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/23.jpg)
Merge sortMethod sort
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 23/33
![Page 24: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/24.jpg)
Merge sortMethod sort
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 24/33
![Page 25: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/25.jpg)
Merge sortMethod merge
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 25/33
![Page 26: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/26.jpg)
Search algorithmsBinary search
Linear search complexity is O(n).A more efficient Binary search has complexity O(log n)
• Works with a sorted array• Array considered as upper and lower half• Check carried out: is item in upper or lower?• Check repeated in the half array containing item.• This search method repeated until item found if it exists.• Each iteration reduces the search space in two.• This explains the O(log n) order of complexity.
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 26/33
![Page 27: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/27.jpg)
Search algorithmsBinary search
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 27/33
![Page 28: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/28.jpg)
Search algorithmsBinary search
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 28/33
![Page 29: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/29.jpg)
Big O NotationTime classification of algorithms
• One method of categorizing algorithmic processing time
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 29/33
![Page 30: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/30.jpg)
Big O NotationSorting
Important to have regard to• Best• Average• Worst
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 30/33
![Page 31: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/31.jpg)
Summary
• Methods to measure complexity• Elapsed time• Big O• Big Θ• Big Ω• Tilde notation
• Example algorithms• Selection sort• Merge sort• Linear search• Binary search
• Relative performance of algorithms
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 31/33
![Page 32: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/32.jpg)
Referenced Material
1. Algorithms (Khan Academy)www.khanacademy.org/computing/computer-science/algorithms/asymptotic-notation/a/big-o-notation
[Accessed 2015-03-03]2. TimSort bug fixed with formal methods http://www.cwi.nl/news/2015/java-bug-fixed-formal-methods-cwi
[Accessed 2015-03-03]3. MOOC: edX | MITx: 6.00.1x Grimson. Introduction toComputer Science and Programming Using PythonMassachusetts Institute of Technologyhttps://www.edx.org
[Accessed 2015-03-05]
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 32/33
![Page 33: WaterfordInstituteofTechnology March1,2016 JohnFitzgerald › modules › jf-programming-2016 › topic07 … · Algorithmsandtheircomplexity Lecture12 WaterfordInstituteofTechnology](https://reader033.vdocument.in/reader033/viewer/2022060511/5f2930dc54c5807a3f7c974a/html5/thumbnails/33.jpg)
Referenced Material
4. MOOC: Algorithms I: Sedgewick & Wayne. PrincetonUniversity.https://www.coursera.org/course/algs4partI
[Accessed 2015-03-03]5. Robert Sedgewick : Algorithms for the masseshttp://osric.com/chris/accidental-developer/2012/04/robert-sedgewick-algorithms-for-the-masses/ [Accessed2015-03-05]
Waterford Institute of Technology, Algorithms and their complexity Lecture 12 33/33