instructor: shengyu zhang. first week part i: about the course part ii: about algorithms and...
TRANSCRIPT
![Page 1: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/1.jpg)
Instructor: Shengyu Zhang
![Page 2: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/2.jpg)
First week
Part I: About the course
Part II: About algorithms and complexity What are algorithms? Growth of functions What is the complexity of an algorithm / a problem
Part III: Review of probability Tail bounds
![Page 3: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/3.jpg)
Part I: About the course
![Page 4: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/4.jpg)
Info
Webpage: http://www.cse.cuhk.edu.hk/~syzhang/course/MScAlg15 Information (time and venue, TA, textbook, etc.) Lecture slides Homework Announcements
Flavor: More math than programming.
![Page 5: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/5.jpg)
Homework
Homework assignments (100%). No exam.
12 homework.
You only need to complete 10. If you do more than 10, the 10 with the highest
scores count.
![Page 6: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/6.jpg)
textbook
No textbook.
Lecture notes available before classes.
Some general references are listed in the course website as well.
![Page 7: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/7.jpg)
Part II: About algorithms and complexity
![Page 8: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/8.jpg)
A good example: driving directions Suppose we want to drive from CUHK to
Central. How to route? Let’s ask Google.
![Page 9: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/9.jpg)
What’s good here: Step by step. Each step is either turn left/right, or go straight for
… meters. An estimated time is also given.
An algorithm is a computational procedure that has step-by-step instructions.
It’ll be good if an estimated time is given.
![Page 10: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/10.jpg)
More on complexity
Why time matters? Time is money! Being late means 0 value
Weather forecast. Homework.
Running time: the number of elementary steps Assuming that each step only costs a small (or
fixed) amount of time.
![Page 11: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/11.jpg)
complexity
The worst-case time complexity of an algorithm A is the running time of A on the worst-case input instance.
The worst-case time complexity of a computational problem P is the worst-case complexity of the best algorithm A that solves the problem. the best algorithm that gives right answers on all inputs.
![Page 12: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/12.jpg)
Hardness of problems can vary a lot Multiplication:
1234 * 5678 = ? 7006652
2749274929483758 * 4827593028759302 = ? Can you finish it in 10 minutes?
Do you think you can handle multiplication easily?
![Page 13: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/13.jpg)
Complexity of integer multiplication In general, for -digit integers:
[Q] How fast is our algorithm? For each ()
we calculate , single-digit multiplications single-digit additions
We finally add the results (with proper shifts) single-digit additions.
Altogether: elementary operations single-digit additions/multiplications
Multiplication is not very hard even by hand, isn’t it?
--------------------------- * * * ∙∙∙ * * * * ∙∙∙ * ∙∙∙ ∙∙∙+ * * * ∙∙∙ * --------------------------------- * * * * * * ∙∙∙ ∙∙∙ *
![Page 14: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/14.jpg)
Inverse problem
The problem inverse to Integer Multiplication is Factoring.
35 = ? * ? 437? 8633? It’s getting harder and harder,
Much harder even with one more digit added! The best known algorithm: running time
![Page 15: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/15.jpg)
The bright side
Hard problems can be used for cryptography!
RSA [Rivest, Shamir, Adleman]: widely-used today, broken if one can factor quickly!
One-line message: Quantum computers can factor quickly!
![Page 16: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/16.jpg)
Messages
Message 1: We care about the speed of the increase, especially when the size is very large.
Many interesting instances in both theory and practice are of huge (and growing!) sizes.
![Page 17: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/17.jpg)
Message 2: We care about the big picture first.
Is the problem as easy as multiplication, or as hard as factoring?
![Page 18: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/18.jpg)
In this regard, we consider the so called asymptotic behavior,… Eventually, i.e. for large , is the function like , or ,
or ? with constant factors ignored at first
i.e. we care about the difference between and much more than that between and
Engineering reason: speedup of a constant factor (say of 10) is easily achieved in a couple of years
![Page 19: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/19.jpg)
Some examples
Which increases faster?
![Page 20: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/20.jpg)
Big-O and small-o
In general: : for some constant , , when is sufficiently
large. i.e. , s.t. , we have .
: for any constant c, , when is sufficiently large. i.e. , s.t. , we have .
![Page 21: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/21.jpg)
The other direction
for some constant and large . i.e. , s.t. , we have .
: for some constant and large . i.e. , s.t. , we have .
: and i.e. for two constants and and large .
![Page 22: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/22.jpg)
Intuition
![Page 23: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/23.jpg)
Examples
![Page 24: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/24.jpg)
Part III: Probability and tail bounds
![Page 25: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/25.jpg)
Finite sample space
Sample space : set the all possible outcomes of a random process. Suppose that is finite.
Events: subsets of . Probability function. , s.t.
, . .
For event , the probability of event happening is .
![Page 26: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/26.jpg)
Union of events
Consider two events and . .
In general, we have the following union bound:
![Page 27: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/27.jpg)
Independence of events
Two events and are independent if
Conditional probability: For two events and with , the probability of conditioned on is .
![Page 28: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/28.jpg)
Random variable
A random variable is a function .
Two random variables and are independent if
![Page 29: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/29.jpg)
Expectation
Expectation:
Linearity of expectation:
no matter whether ’s are independent or not.
![Page 30: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/30.jpg)
variance
The variance of is
The standard deviation of is
![Page 31: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/31.jpg)
31
Concentration and tail bounds In many analysis of randomized algorithms,
we need to study how concentrated a random variable is close to its mean . Many times .
Upper bounds of Pr[ deviates from a lot]
is called tail bounds.
![Page 32: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/32.jpg)
32
Markov’s Inequality: when you only know expectation [Thm] If , then
In other words, if , then
Proof. . Dropping some nonnegative terms always make it
smaller.
![Page 33: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/33.jpg)
33
Moments
Def. The th moment of a random variable is
: variance.
![Page 34: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/34.jpg)
34
Chebyshev’s Inequality: when you also know variance [Thm]
In other words, .
Proof.
// Markov on
// recall:
![Page 35: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/35.jpg)
35
Inequality by the th-moment (: even) [Thm] . Proof.
// is even// Markov on
![Page 36: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/36.jpg)
36
Chernoff’s Bound
[Thm] Suppose and let
.
Then
where
![Page 37: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/37.jpg)
37
Some basic applications
One-sided error: Suppose an algorithm for a decision problem has : no error : output with probability 1/2
We want to decrease this ½ to . How? Run the algorithm times. Output 0 iff all
executions answer 0.
![Page 38: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/38.jpg)
38
Two-sided error
Suppose a randomized algorithm has two-sided error : output with probability > 2/3 : output with probability > 2/3
How? Run the algorithm steps and take a majority
vote.
![Page 39: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/39.jpg)
39
Using Chernoff’s bound
Run the algorithm times, getting outputs. Suppose they are .
Let if : w.p. , thus . if : w.p. , so .
![Page 40: Instructor: Shengyu Zhang. First week Part I: About the course Part II: About algorithms and complexity What are algorithms? Growth of functions](https://reader036.vdocument.in/reader036/viewer/2022062517/56649f1d5503460f94c3371a/html5/thumbnails/40.jpg)
40
Recall Chernoff: If : .
, so . . Similar for . The error prob. decays exponentially with #
of trials!