computer science approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/comp610/02analysis.pdf ·...
TRANSCRIPT
![Page 1: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/1.jpg)
Computer Science Approach to problem solving
• If my boss / supervisor / teacher formulates a problem to be solved urgently, can I write a program to efficiently solve this problem ???
![Page 2: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/2.jpg)
29
Polynomial-Time
Brute force. For many non-trivial problems, there is a natural brute force search algorithm that checks every possible solution. ■ Typically takes 2N time or worse for inputs of size N. ■ Unacceptable in practice.
Desirable scaling property. When the input size doubles, the algorithm should only slow down by some constant factor C.
Def. An algorithm is poly-time if the above scaling property holds.
Property: poly-time is invariant over all (non-quantum) computer models.
There exists constants a > 0 and d > 0 such that on every input of size N, its running time is bounded by a•Nd steps.
choose C = 2d
N ! for stable matchingwith N men and N women
![Page 3: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/3.jpg)
30
Average/Worst-Case Analysis
Worst case running time. Obtain bound on largest possible running time of algorithm on any input of a given size N. ■ Generally captures efficiency in practice. ■ Draconian view, but hard to find effective alternative. ■ For probabilistic algorithms, we take the worst average running time.
Average case running time. Obtain bound on running time of algorithm on random input as a function of input size N. ■ Hard (or impossible) to accurately model real instances by random
distributions. ■ Algorithm tuned for a certain distribution may perform poorly on
other input distributions.
![Page 4: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/4.jpg)
31
Worst-Case Polynomial-Time
Def. An algorithm is efficient if its running time is polynomial.
Justification: It really works in practice! ■ Although 6.02 × 1023 × N20 is technically poly-time, it would be
useless in practice. ■ In practice, the poly-time algorithms that people develop almost
always have low constants and low exponents. ■ Breaking through the exponential barrier of brute force typically
exposes some crucial structure of the problem.
Exceptions. ■ Some poly-time algorithms do have high constants and/or exponents,
and are useless in practice. ■ Some exponential-time (or worse) algorithms are widely used
because the worst-case instances seem to be rare.simplex methodUnix grep
Primality testing
![Page 5: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/5.jpg)
32
Why It Matters
Big-O ComplexityO
pera
tion
s
Elements
O(1)
O(log n)
O(n)
O(n log n)
O(n2)
O(2n)
O(n!)
1000
900
800
700
600
500
400
300
200
100
00 10 20 30 40 50 60 70 80 90 100
![Page 6: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/6.jpg)
33
Why It Matters
Note: age of Universe ~ 1010 years…
![Page 7: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/7.jpg)
34
Chapter 2 Basics of Algorithm Analysis
Slides by Kevin Wayne. Copyright © 2005 Pearson-Addison Wesley. All rights reserved.
![Page 8: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/8.jpg)
2.2 Asymptotic Order of Growth
![Page 9: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/9.jpg)
36
Asymptotic Order of Growth
Let f:ℕ→ℝ+ be a function, we define
Upper bounds.
O(f) = { g:ℕ→ℝ+ | ∃c∈ℝ+, n0∈ℕ s.t. ∀n ≥ n0 [ g(n) ≤ c · f(n) ] }.
Lower bounds.
Ω(f) = { g:ℕ→ℝ+ | ∃c∈ℝ+, n0∈ℕ s.t. ∀n ≥ n0 [ g(n) ≥ c · f(n) ] }.
Tight bounds. Θ(f) = O(f) ∩ Ω(f).
Ex: T(n) = 32n2 + 17n + 32.
T(n) ∈ O(n2), O(n3), Ω(n2), Ω(n), and Θ(n2) .T(n) ∉ O(n), Ω(n3), Θ(n), or Θ(n3).
![Page 10: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/10.jpg)
37
Notation
Abuse of notation. T(n) = O(f(n)).
■ Not transitive: – f(n) = 5n3; g(n) = 3n2 – f(n) = O(n3) and g(n) = O(n3) but f(n) ≠ g(n).
■ Better notation: T(n) ∈ O(f(n)). ■ Acceptable notation: T(n) is O(f(n)). (if scared by ∈ !)
Meaningless statement. Any comparison-based sorting algorithm requires at least O(n log n) comparisons.
■ Statement doesn't "type-check". ■ Precisely, f(n)=1 ∈ O(n log n), therefore "at least one comparison". ■ Use Ω for lower bounds: "at least Ω(n log n) comparisons". ■ "requires at least cn log n comparisons for c>0 and all large enough n".
![Page 11: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/11.jpg)
38
Big-O Notation
Limit theorems.
Let f,g:ℕ→ℝ+ be functions, such that
lni➝m∞ f(n)/g(n) = c∈ℝ+,
then f ∈ Θ(g), g ∈ Θ(f), Θ(f) = Θ(g)
lni➝m∞ f(n)/g(n) = 0,
then f ∈ O(g), f ∉ Ω(g), O(f) ⊊ O(g), Ω(g) ⊊ Ω(f)
![Page 12: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/12.jpg)
39
Properties
Let f,g:ℕ→ℝ+ be functions Transitivity. ■ If f ∈ O(g) and g ∈ O(h) then f ∈ O(h) since O(f)⊂O(g)⊂O(h). ■ If f ∈ Ω(g) and g ∈ Ω(h) then f ∈ Ω(h) since Ω(f)⊂Ω(g)⊂Ω(h). ■ If f ∈ Θ(g) and g ∈ Θ(h) then f ∈ Θ(h) since Θ(f)⊂Θ(g)⊂Θ(h).
Additivity. ■ If f ∈ O(h) and g ∈ O(h) then f + g ∈ O(h)
since f(n) < cf h(n), g(n) < cg h(n) ⇒ f(n) + g(n) < (cf+cg) h(n). ■ If f ∈ Ω(h) and g ∈ Ω(h) then f + g ∈ Ω(h). ■ If f ∈ Θ(h) and g ∈ O(h) then f + g ∈ Θ(h).
Consequence: ■ f + g ∈ O( max{f,g} ) since f + g ≤ 2max{f,g}. ■ f + g ∈ Ω( max{f,g} ) since f + g ≥ max{f,g}. ■ f + g ∈ Θ( max{f,g} ) since max{f,g} ≤ f + g ≤ 2 max{f,g}.
![Page 13: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/13.jpg)
Consequence: ■ f + g ∈ O( max{f,g} ). ■ f + g ∈ Ω( max{f,g} ). ■ f + g ∈ Θ( max{f,g} ).
40
Properties
max ← a1
for i = 2 to n { if (a
i > max)
max ← ai
}
min ← (x1 - x
2)2 + (y
1 - y
2)2
for i = 1 to n { for j = i+1 to n { d ← (x
i - x
j)2 + (y
i - y
j)2
if (d < min) min ← d } }
foreach set Si {
foreach other set Sj {
foreach element p of Si {
determine whether p also belongs to Sj
} if (no element of S
i belongs to S
j)
report that Si and S
j are disjoint
} }
Θ(n)
Θ(n2)
Θ(n3)
Θ(n3)
![Page 14: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/14.jpg)
41
Asymptotic Bounds for Some Common Functions
Polynomials. a0 + a1n + … + adnd ∈ Θ(nd) if ad > 0.
Polynomial time. Running time ∈ O(nd) for some constant d independent of the input size n.
Logarithms. O(log a n) = O(log b n) for any constants a, b > 0.
Logarithms. For every x > 0, log n ∈ O(nx).
Exponentials. For every r > 1 and every d > 0, nd ∈ O(rn).
every exponential grows faster than every polynomial
can avoid specifying the base
log grows slower than every polynomial
![Page 15: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/15.jpg)
2.4 A Survey of Common Running Times
![Page 16: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/16.jpg)
43
Linear Time: O(n)
Linear time. Running time is proportional to input size.
Computing the maximum. Compute minimum of n numbers a1, …, an.
min ← a1 for i = 2 to n { if (ai < min) min ← ai }
![Page 17: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/17.jpg)
44
O(n log n) Time
O(n log n) time. Arises in divide-and-conquer algorithms.
Sorting. Mergesort and Heapsort are sorting algorithms that perform O(n log n) comparisons.
Closest Points on a line. Given n numbers x1, …, xn, what is the smallest distance xi-xj between any two points?
O(n log n) solution. Sort the n numbers. Scan the sorted list in order, identifying the minimum gap between two successive points.
also referred to as linearithmic time
![Page 18: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/18.jpg)
45
O(n log n) Time
O(n log n) time. Arises in divide-and-conquer algorithmsMedian Finding. Given n distinct numbers a1, …, an, find i such that
|{ j : aj<ai }| =⎣n-1 / 2⎦ and |{ j : aj>ai }| =⎡n-1 / 2⎤.
Straight forward approach. Θ(n log n) arithmetic operations (to sort first). Fundamental question. Can we improve upon this approach ? Remark. This algorithm is Ω(n log n) and it seems inevitable in general, but this is just an illusion: Θ(n) is actually possible and optimal…
22 31 44 7 12 19 20 35 3 40 27
3 7 12 19 20 22 27 31 35 40 44
9 4 5 6 7 1 11 2 8 10 3
45
![Page 19: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/19.jpg)
46
Quadratic Time: O(n2)
Quadratic time. Enumerate all pairs of elements.
Closest pair of points. Given a list of n points in the plane (x1, y1), …, (xn, yn), find the pair that is closest.
O(n2) solution. Try all pairs of points.
Remark. This algorithm is Ω(n2) and it seems inevitable in general, but this is just an illusion: Θ(n log n) is actually possible and optimal…
min ← (x1 - x2)2 + (y1 - y2)2 for i = 1 to n { for j = i+1 to n { d ← (xi - xj)2 + (yi - yj)2 if (d < min) min ← d } }
don't need to take square roots
see chapter 5
![Page 20: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/20.jpg)
First Repetition. Given N numbers a1, …, aN, find the smallest n s. t.
there exists 1≤i<n such that ai=an.
Most natural solutions will be Θ(n2).
An O(n log n) algorithm (where n is the location of the first repetition). can be obtained from Red-Black Trees or similar to MergeSort.
22 31 44 7 12 19 22 35 3 40 27
47
Quadratic Time: O(n2)
![Page 21: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/21.jpg)
48
Quadratic Time: O(n2)
Quadratic time. Solve O(n2) independent sub-puzzles each in constant-time.
nxnxn Rubik’s cube. Given a scrambled nxnxn cube, put it in solved configuration.
Remark. This algorithm is Ω(n2) and it seems inevitable in general, but this is just an illusion: Θ(n2/log n) is actually possible and optimal…
![Page 22: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/22.jpg)
49
Cubic Time: O(n3)
Cubic time. Enumerate all triples of elements.
Matrix multiplication. Given two nxn matrices of numbers A,B, what is their matrix product C ?
O(n3) solution. For each entry cij compute as below.
Remark. This algorithm is Ω(n3) and it seems inevitable in general, but this is just an illusion: O(n2.3728639) is actually possible…
![Page 23: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/23.jpg)
50
Polynomial Time: O(nk) Time
Independent set of size k. Given a graph, are there k nodes such that no two are joined by an edge?
O(nk) solution. Enumerate all subsets of k nodes.
■ Check whether S is an independent set = O(k2).
■ Number of k element subsets = ■ O(k2 nk / k!) = O(nk).
foreach subset S of k nodes { if (S is an independent set) report S } }
€
nk"
# $ %
& ' =
n (n−1) (n− 2)! (n− k +1)k (k −1) (k − 2)! (2) (1)
≤ nk
k!
poly-time for k=17,but not practical
k is a constant
![Page 24: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/24.jpg)
51
Exponential Time
Independent set. Given a graph, what is the maximum size of an independent set?
O(n2 2n) solution. Enumerate all subsets.
S* ← ∅ foreach subset S of nodes { if (S is an independent set and |S|>|S*|) update S* ← S } }
![Page 25: Computer Science Approach to problem solvingcrypto.cs.mcgill.ca/~crepeau/COMP610/02analysis.pdf · 2017. 1. 21. · 30 Average/Worst-Case Analysis Worst case running time. Obtain](https://reader035.vdocument.in/reader035/viewer/2022070216/611d2fd6ca4ed218b5171cac/html5/thumbnails/25.jpg)
52
Chapter 2 Basics of Algorithm Analysis
Slides by Kevin Wayne. Copyright © 2005 Pearson-Addison Wesley. All rights reserved.