mat-72006 aads, fall 2016 14-sep-16 116elomaa/teach/aads-2016-3.pdf · mat-72006 aads, fall 2016...
TRANSCRIPT
9/14/2016
1
• A more intricate example is the recurrence( ) = ( /3) + ( /3) + ( )
• represents the constant in the term• Adding the values across the levels of the
recursion tree yields value for every level• The longest simple path from the root to a
leaf is (2 3) (2 3) 1• (2 3) = 1 when = log /
• The height of the tree is log /
14-Sep-16MAT-72006 AADS, Fall 2016 116
14-Sep-16MAT-72006 AADS, Fall 2016 117
9/14/2016
2
• We expect the solution to the recurrence tobe number of levels × cost of each level:
( log ) = ( lg )• Not every level contributes a cost of• If this recursion tree were a complete binary
tree of height log , there would be2 = leaves
• The cost of each leaf is a constant• The total cost of all leaves would then be
( ) which, since log 2 > 1, is( lg )
14-Sep-16MAT-72006 AADS, Fall 2016 118
• This recursion tree is not a complete binary tree,however, and has fewer than leaves
• As we go down from the root, more and moreinternal nodes are absent
• Consequently, levels toward the bottomcontribute less than to the total cost
• we can use the substitution method to verify that( lg ) is an upper bound for the solution to
the recurrence
14-Sep-16MAT-72006 AADS, Fall 2016 119
9/14/2016
3
4.5 The master method
Theorem 4.1 (Master theorem)
Let 1 and > 1, and ( ) a function, ( )is de ned on nonneg. integers by recurrence
= ( ) + ,where we / means either or .
( ) has the following asymptotic bounds:
14-Sep-16MAT-72006 AADS, Fall 2016 120
1. If = ( ) for some constant > 0,then ( ) =
2. If ( ) = ,then ( ) = lg
3. If ( ) = for some constant > 0,and if ( ) ( ) for some constant < 1and all suf ciently large ,
then ( ) = ( )
14-Sep-16MAT-72006 AADS, Fall 2016 121
9/14/2016
4
• In each of the three cases, we compare thefunction ( ) with the function
• The larger of the two determines the solutionto the recurrence– In case 1, the function is the larger,
then the solution is ( ) =– In case 3, the function ( ) is the larger, then
the solution is ( ) = ( )– In case 2, the two functions are the same
size, we multiply by a logarithmic factor, andthe solution is ( ) = lg
14-Sep-16MAT-72006 AADS, Fall 2016 122
• Be aware of some technicalities:• In case 1, not only must ( ) be smaller than
, it must be polynomially smaller• ( ) must be asymptotically smaller than
by a factor of for some > 0• In case 3, not only must ( ) be larger than
, it also must be polynomially largerand satisfy the “regularity” condition that
( ) ( )14-Sep-16MAT-72006 AADS, Fall 2016 123
9/14/2016
5
Using the master method
( ) = ( 3) +
• We have = 9, = 3, ( ) = , and thus= ( )
• Since ( ) = ( ), where = 1, wecan apply case 1 of the master theorem andconclude that the solution is ( )
14-Sep-16MAT-72006 AADS, Fall 2016 124
( ) = (2 3) + 1
• Here = 1, = 3 2, ( ) = 1, and == = 1
• Case 2 applies, since =(1), and thus the solution to the recurrence
is (lg )
14-Sep-16MAT-72006 AADS, Fall 2016 125
9/14/2016
6
( ) = ( 4) + lg
• We have = 3, = 4, = lg , and= = ( . )
• Since ( ) = ( ), where 0.2,case 3 applies provided that the regularitycondition holds for ( )
• For suf ciently large , we have that( ) = 3( 4) lg( 4)
(3 4) lg = ( ) for = 3 4• By case 3, the solution is ( lg )
14-Sep-16MAT-72006 AADS, Fall 2016 126
• The master method does not apply to( ) = ( 2) + lg
even if it seems to have the proper form:= 2, = 2, = lg , and =
• One might think that case 3 applies, sinceis asymptotically larger than
• It is, however, not polynomially larger: ratio= lg is asymptotically less than
for any positive constant• Consequently, the recurrence falls into the
gap between case 2 and case 314-Sep-16MAT-72006 AADS, Fall 2016 127
9/14/2016
7
• The recurrence of the running time of thesimple divide-and-conquer algorithm formatrix multiplication
( ) = ( 2) ( )
• Now = 8, = 2, and and so= =
• Since is polynomially larger than– i.e., = for = 1case 1 applies, and ( ) =
14-Sep-16MAT-72006 AADS, Fall 2016 128
• The recurrence of the running time ofStrassen’s algorithm
( ) = ( 2) ( )
• Here we have = 7, = 2, andand so = =
• Recalling that 2.80 < lg 7 < 2.81, we see that= O( ) for = 0.8
• Again, case 1 applies, and we have thesolution ( ) =
14-Sep-16MAT-72006 AADS, Fall 2016 129
9/14/2016
8
5 Probabilistic Analysis andRandomized Algorithms
• We need to hire a new of ce assistant with thehelp of an employment agency that sends us acandidate each day
• We interview that person and then decide eitherto hire her or not
• We pay the agency a fee for each interview• To actually hire an applicant is more costly
– We must re the current of ce assistant and– pay a substantial hiring fee to the agency
14-Sep-16MAT-72006 AADS, Fall 2016 130
• We are committed to having, at all times, thebest possible person for the job
• After interviewing an applicant, if she is betterquali ed than the current assistant, we will
re the current of ce assistant and hire thenew applicant
• We are willing to pay the price of this strategy,but wish to estimate what that price will be
14-Sep-16MAT-72006 AADS, Fall 2016 131
5.1 The hiring problem
9/14/2016
9
HIRE-ASSISTANT( )
1. best 0 // candidate 0 is a least-quali ed dummy candidate
2. for 1 to3. interview candidate4. if candidate is better than best5. best6. hire candidate
14-Sep-16MAT-72006 AADS, Fall 2016 132
• Interviewing has a low cost, , whereas hiring isexpensive, costing
• Letting be the number of people hired, thetotal cost of the algorithm is ( + )
• No matter how many people we hire, we alwaysinterview candidates and thus always incur thecost
• Let us concentrate on analyzing , the hiringcost
• This quantity varies with each run of thealgorithm
14-Sep-16MAT-72006 AADS, Fall 2016 133
9/14/2016
10
Worst-case analysis
• In the worst case, we hire every candidate thatwe interview– candidates come in increasing order of quality– we hire times, for a total hiring cost of
• We have no idea about the order in which thecandidates arrive, nor do we have any controlover this order
• It is natural to ask what we expect to happen in atypical or average case
14-Sep-16MAT-72006 AADS, Fall 2016 134
Probabilistic analysis
• We can assume that the applicants come in arandom order
• Assume that we can compare any twocandidates and decide the better quali ed
there is a total order on the candidates• Rank each candidate with a unique number
1, … ,• Use ) to denote the rank of applicant
14-Sep-16MAT-72006 AADS, Fall 2016 135
9/14/2016
11
• A higher rank corresponds to a better quali edapplicant
• Ordered list 1 , 2 , … , ( ) is apermutation of the list 1,2, … ,
• “The applicants come in a random order”– “this list of ranks is equally likely to be any
one of the ! permutations of the numbers 1through ”
• Alternatively, the ranks form a uniform randompermutation; i.e., each of the possible !permutations appears with equal probability
14-Sep-16MAT-72006 AADS, Fall 2016 136
Randomized algorithms
• We call an algorithm randomized if its behavioris determined also by values produced by arandom-number generator RANDOM
• A call to RANDOM( , ) returns an integerbetween and , inclusive, with each suchinteger being equally likely
• E.g., RANDOM(0,1) produces 0 with probability1 2, and 1 with probability 1 2
14-Sep-16MAT-72006 AADS, Fall 2016 137
9/14/2016
12
• A call to RANDOM 3,7 returns either 3,4,5,6,7,each with probability 1 5
• Each integer returned by RANDOM isindependent of the integers returned on previouscalls
• In analyzing the running time of a randomizedalgorithm, we take the expectation of the runningtime over the distribution of values returned bythe RANDOM
• We refer to the running time of a randomizedalgorithm as an expected running time
14-Sep-16MAT-72006 AADS, Fall 2016 138
5.2 Indicator random variables
• Indicator random variables provide a convenientmethod for converting between probabilities andexpectations
• Suppose we are given a sample space and anevent
• Then the indicator random variable { }associated with is de ned as
= 1 if occurs0 if does not occur
14-Sep-16MAT-72006 AADS, Fall 2016 139
9/14/2016
13
• Let us determine the expected number of headsthat we obtain when ipping a fair coin
• Our sample space is = { , }, withPr = Pr = 1 2
• We de ne an indicator random variable ,associated with the coin coming up heads ( )
• This variable counts the number of headsobtained in this ip
= = 1 if H occurs0 if T occurs
14-Sep-16MAT-72006 AADS, Fall 2016 140
• Recall that E = Pr{ = }• The expected number of heads obtained in one
ip of the coin is simply the expected value ofour indicator variable
E = E= 1 Pr + 0 Pr= 1 (1 2) + 0 (1 2)= 1 2
• Thus the expected number of heads obtained byone ip of a fair coin is 1 2
14-Sep-16MAT-72006 AADS, Fall 2016 141
9/14/2016
14
• The expected value of an indicator randomvariable associated with an event is equal tothe probability that occurs
Lemma 5.1 Given a sample space and anevent in , let = { }. Then E = Pr{ }
Proof By the de nition of an indicator and thede nition of expected value, we have =
= 1 Pr + 0 Pr = Pr , wheredenotes , the complement of
14-Sep-16MAT-72006 AADS, Fall 2016 142
• Compute the number of heads in coin ips• Let be the indicator associated with the event
in which the th ip comes up heads:= {the th flip results in the event }
• Let denote the total number of heads in thecoin ips, so that
=
• To compute the expected number of heads, takethe expectation of both sides to obtain
E = E
14-Sep-16MAT-72006 AADS, Fall 2016 143
9/14/2016
15
• Linearity of expectationE + = E + E
makes the use of indicator random variables apowerful analytical technique– it applies even when there is dependence
among the variables• We now can easily compute the expected
number of heads:
E = E = E
1 2 = 2
14-Sep-16MAT-72006 AADS, Fall 2016 144
Analysis of the hiring problem
• Let be the random variable whose valuenumber of times we hire a new assistant
• let be the indicator random variableassociated with the event in which the thcandidate is hired
= candidate is hired
= 1 if candidate is hired0 if candidate is not hired
• = + + +14-Sep-16MAT-72006 AADS, Fall 2016 145
9/14/2016
16
• By Lemma 5.1E = Pr candidate is hired
and we must compute the probability that lines5–6 of HIRE-ASSISTANT are executed
• Candidate is hired (line 6) exactly when she isbetter than each of candidates 1, … , 1
• We assume that the candidates arrive in arandom order: the rst candidates haveappeared in a random order
• Any one of these rst candidates is equallylikely to be the best-quali ed so far
14-Sep-16MAT-72006 AADS, Fall 2016 146
• Candidate has a probability of 1/ of beingbetter quali ed than candidates 1, … , 1 andthus a probability of 1/ of being hired.
• By Lemma 5.1, we conclude that E = 1/• Now we can compute E :
E = E = E
= 1 = ln + (1)
by harmonic series
= 1 = ln + (1)
14-Sep-16MAT-72006 AADS, Fall 2016 147
9/14/2016
17
• We interview people, but actually hire onlyapprox. ln of them, on average
• The following lemma summarize this
Lemma 5.2 Assuming that candidates come inrandom order, algorithm HIRE-ASSISTANT has anaverage-case total hiring cost of ( ln )
Proof Follows immediately from our de nition ofthe hiring cost and the equation, which shows thatthe expected number of hires is approximately ln
14-Sep-16MAT-72006 AADS, Fall 2016 148
5.3 Randomized algorithms
• The algorithm above is deterministic• The number of times we hire a new of ce
assistant differs for different inputs• Given the rank list = 1,2,3,4,5,6,7,8,9,10 , a
new assistant is always hired 10 times• For the list of ranks = 10,9,8,7,6,5,4,3,2,1 , a
new assistant is hired only once• = 5,2,1,8,4,7,10,9,3,6 , a new assistant is
hired three times
14-Sep-16MAT-72006 AADS, Fall 2016 149
9/14/2016
18
• Consider an algorithm that rst permutes theinput before determining the best candidate
• Now we randomize in the algorithm, not in theinput distribution
• For input we cannot say how many times thebest is updated – it differs with each run
• Not even an enemy can produce a bad input,random permutation yields input order irrelevant
• The algorithm performs badly only if the random-number generator produces an “unlucky”permutation
14-Sep-16MAT-72006 AADS, Fall 2016 150
RANDOMIZED-HIRE-ASSISTANT( )
1. randomly permute the list of candidates2. best 0 // candidate 0 is a least-quali ed
// dummy candidate3. for 1 to4. interview candidate5. if candidate is better than candidate best6. best7. hire candidate
14-Sep-16MAT-72006 AADS, Fall 2016 151
9/14/2016
19
Randomly permuting arrays
• We assume that we are given an array which,w.l.o.g., contains the elements 1, 2, … ,
• Produce a random permutation of the array• Assign element [ ] a random priority [ ], and
sort the elements according to priorities• E.g., if our initial array is = 1,2,3,4 and we
choose random priorities = 36,3,62,19 , wewould produce array = 2,4,1,3
• We call this procedure PERMUTE-BY-SORTING
14-Sep-16MAT-72006 AADS, Fall 2016 152
PERMUTE-BY-SORTING( )
1. . length2. let [1. . ] be a new array3. for 1 to4. [ RANDOM(1, )5. sort , using as sort keys
• We use a range of 1 to for random numbersto make it likely that all the priorities in areunique
14-Sep-16MAT-72006 AADS, Fall 2016 153
9/14/2016
20
• It remains to prove that the procedure producesa uniform random permutation,– It is equally likely to produce every
permutation of the numbers 1 through
Lemma 5.4 PERMUTE-BY-SORTING produces auniform random permutation of the input,assuming that all priorities are distinct
Proof See the book.
14-Sep-16MAT-72006 AADS, Fall 2016 154
• It is better to permute the given array in place• RANDOMIZE-IN-PLACE does so in ( ) time• In its th iteration, it chooses the element [ ]
randomly from among elements through[ ]
• After the th iteration, is never altered
RANDOMIZE-IN-PLACE( )1 .2 for 1 to3 swap with [RANDOM( , )]
14-Sep-16MAT-72006 AADS, Fall 2016 155
9/14/2016
21
• A -permutation on a set of elements is asequence containing of the elements, withno repetitions
• The number of -permutations is ! !
loop invariant:• Just prior to the th iteration of the for loop of
lines 2–3, for each possible ( 1)-permutationof the elements, the subarray [1. . 1]contains this ( 1)-permutation with probability
+ 1 ! !
14-Sep-16MAT-72006 AADS, Fall 2016 156
Initialization: loop invariant trivially holds
Maintenance:• Consider a particular -permutation, and denote
the elements in it by , , … ,• This permutation consists of an ( 1)-
permutation , , … , followed by thevalue that the algorithm places in [ ]
• Let denote the event in which the rst ( 1)iterations have created the particular ( 1)-permutation , , … , in [1. . 1]
14-Sep-16MAT-72006 AADS, Fall 2016 157
9/14/2016
22
• By the loop invariant, Pr = + 1 ! !• Let be the event that th iteration puts in
position [ ]• The -permutation , , … , appears in
1. . precisely when both and occurPr = Pr Pr{ }
• Pr = 1 + 1 because in line 3 thealgorithm chooses randomly from the +1 values in positions . .
Pr =1
+ 1+ 1 !!
=!
!14-Sep-16MAT-72006 AADS, Fall 2016 158
Termination:• At termination, = + 1, and we have that the
subarray 1. . is a given -permutation withprobability
+ 1 + 1 ! ! = 0! ! = 1 !
• Thus, RANDOMIZE-IN-PLACE produces a uniformrandom permutation
14-Sep-16MAT-72006 AADS, Fall 2016 159
9/14/2016
23
5.4.1 The birthday paradox
• How many people must there be in a roombefore there is a 50% chance that two of themwere born on the same day of the year?
• The answer is surprisingly few• The paradox is that it is in fact far fewer
– than the number of days in a year, or– even half the number of days in a year
14-Sep-16MAT-72006 AADS, Fall 2016 160
An analysis using indicatorrandom variables
• We use indicator random variables to provide asimple but approximate analysis of the birthdayparadox
• For each pair ( , ) of the people in the room,de ne the indicator random variable , for
< , by= { and have the same birthday}
= 1 and have the same birthday0 otherwise
14-Sep-16MAT-72006 AADS, Fall 2016 161
9/14/2016
24
• Once birthday for is chosen, the probabilitythat is chosen to be the same day is 1 ,where = 365
E = Pr and have the same birthday = 1
• Let be a random variable counting the numberof pairs of individuals having the same birthday
=
14-Sep-16MAT-72006 AADS, Fall 2016 162
• Taking expectations of both sides and applyinglinearity of expectation, we obtain
E = E = [ ]
= 21
=( 1)
• When ( 1) 2 , the expected number ofpairs of people with the same birthday is at least1
14-Sep-16MAT-72006 AADS, Fall 2016 163
9/14/2016
25
• Thus, if we have at least + 1 individuals in aroom, we can expect at least two to have thesame birthday
• For = 365, if = 28, the expected number ofpairs with the same birthday is
(28 27) (2 365) 1.0356• With at least 28 people, we expect to nd at
least one matching pair of birthdays• Mars has = 687 days, need = 38 aliens• Analysis using only probabilities gives a different
exact number of people, but sameasymptotically:
14-Sep-16MAT-72006 AADS, Fall 2016 164