Lecture3RecurrenceRelationsandtheMasterTheorem!
Announcements!
• HW1isposted!
• DueFriday.
• Sections willbeTuesdays4:30-5:20,room380-381U!
• Theyareoptional.
• Butvaluable!
• SignupforPiazza!
• There’salinkonthecoursewebsite.
• CourseannouncementswillbepostedonPiazza.
Moreannouncements
Lasttime….
• Sorting:InsertionSort andMergeSort
• Analyzingcorrectnessofiterative+recursivealgs
• Via“loopinvariant”andinduction
• Analyzingrunningtimeofrecursivealgorithms
• Bywritingoutatreeandaddingupalltheworkdone.
• Howdowemeasuretheruntimeofanalgorithm?
• Worst-CaseAnalysis
• Big-OhNotation
Today
•RecurrenceRelations!• Howdowemeasuretheruntimearecursivealgorithm?
• LikeIntegerMultiplicationandMergeSort?
• TheMasterMethod• Ausefultheoremsowedon’thavetoanswerthis
questionfromscratcheachtime.
RunningtimeofMergeSort
• Let’scallthisrunningtimeT(n).• whentheinputhaslengthn.
• WeknowthatT(n)=O(nlog(n)).
• Butifwedidn’tknowthat…MERGESORT(A):
n=length(A)
if n≤ 1:
return A
L=MERGESORT(A[:n/2])
R=MERGESORT(A[n/2:])
returnMERGE(L,R)
𝑇 𝑛 ≤ 2 ⋅ 𝑇 𝑛2 + 11 ⋅ 𝑛
Fromlasttime
RecurrenceRelations
• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛 isarecurrencerelation.
• ItgivesusaformulaforT(n) intermsofT(lessthann)
• Thechallenge:
GivenarecurrencerelationforT(n),findaclosedformexpressionforT(n).
• Forexample,T(n)=O(nlog(n))
TechnicalitiesIBaseCases
• Formally,weshouldalwayshavebasecaseswithrecurrencerelations.
• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛with𝑇 1 = 1
isnotthesameas
• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛with𝑇 1 = 1000000000
• However,T(1)=O(1),sosometimeswe’lljustomitit.
WhydoesT(1)=O(1)?
Siggi theStudiousStork
Pluckythe
PedanticPenguin
Onyourpre-lectureexercise
• Youplayedaroundwiththeseexamples(whennisapowerof2):
1. 𝑇 𝑛 = 𝑇 *+ + 𝑛, 𝑇 1 = 1
2. 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 𝑛, 𝑇 1 = 1
3. 𝑇 𝑛 = 4 ⋅ 𝑇 *+ + 𝑛, 𝑇 1 = 1
• Whataretheclosedforms?
[onboard]
Oneapproachforallofthese
Sizen
n/2n/2
n/4
(Size1)
…
n/4n/4n/4
n/2tn/2tn/2tn/2tn/2tn/2t
…
• The“tree”approach
fromlasttime.
• Addupallthework
doneatallthesub-
problems.
Anotherapproach:
Recursivelyapplytherelationshipabunchuntilyouseeapattern.
• 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 11 ⋅ 𝑛
• 𝑇 𝑛 = 2 ⋅ 2 ⋅ 𝑇 *1 + 22⋅*
+ + 11 ⋅ 𝑛• 𝑇 𝑛 = 4 ⋅ 𝑇 *
1 + 22 ⋅ 𝑛
• 𝑇 𝑛 = 4 ⋅ 2 ⋅ 𝑇 *3 + 22⋅*
1 + 22 ⋅ 𝑛• 𝑇 𝑛 = 8 ⋅ 𝑇 *
3 + 33 ⋅ 𝑛
• Followingthepattern…
• 𝑻 𝒏 = 𝒏 ⋅ 𝑻 𝟏 + 𝟏𝟏 ⋅ 𝒍𝒐𝒈 𝒏 ⋅ 𝒏 = 𝑶 𝒏 ⋅ 𝐥𝐨𝐠 𝒏
Formally,thisshould
beaccompanied
withaproofthat
thepatternholds!
Morenexttime.
Thisslideskippedinclass,providedhereincase
thiswaymakesmoresensetoyou.
Moreexamples
• Needlesslyrecursiveintegermultiplication
• T(n)=4T(n/2)+O(n)
• T(n)=O(n2)
• Karatsubaintegermultiplication
• T(n)=3T(n/2)+O(n)
• T(n)=O(𝑛?@AB C ≈ n1.6)
• MergeSort
• T(n)=2T(n/2)+O(n)
• T(n)=O(nlog(n))
T(n)=timetosolveaproblemofsizen.
What’sthepattern?!?!?!?!
Thesetwoare
thesameasthe
onesonyour
pre-lecture
exercise.
Themastertheorem
• A formulathatsolvesrecurrenceswhenallofthesub-problemsarethesamesize.
• We’llseeanexampleWednesdaywhennotallproblemsarethesamesize.
• ”Generalized”treemethod.
JedimasterYoda
A useful
formula it is.
Know why it works
you should.
Themastertheorem
• Suppose𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .Then
Many symbols
those are….
Threeparameters:
a:numberofsubproblems
b:factorbywhichinputsizeshrinks
d:needtodond worktocreateallthe
subproblems andcombinetheirsolutions.
Wecanalsotaken/bto
meaneither*F or
*F and
thetheoremisstilltrue.
TechnicalitiesIIIntegerdivision
• Ifnisodd,Ican’tbreakitupintotwoproblemsofsizen/2.
• However(seeCLRS,Section4.6.2),onecanshowthattheMastertheoremworksfineifyoupretendthatwhatyouhaveis:
• Fromnowonwe’llmostlyignorefloorsandceilingsinrecurrencerelations.
𝑇 𝑛 = 𝑇 𝑛2 + 𝑇 𝑛
2 + 𝑂(𝑛)
𝑇 𝑛 = 2 ⋅ 𝑇 𝑛2 + 𝑂(𝑛)
Pluckythe
PedanticPenguin
Examples(detailsonboard)
• Needlesslyrecursiveintegermult.
• T(n)=4T(n/2)+O(n)
• T(n)=O(n2)
• Karatsubaintegermultiplication
• T(n)=3T(n/2)+O(n)
• T(n)=O(nlog_2(3) ≈ n1.6)
• MergeSort
• T(n)=2T(n/2)+O(n)
• T(n)=O(nlog(n))
• Thatotherone
• T(n)=T(n/2)+O(n)
• T(n)=O(n)
𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .
a=4
b=2
d=1
a=3
b=2
d=1
a=2
b=2
d=1
a>bd
a>bd
a=bd
✓✓✓
a=1
b=2
d=1
a<bd ✓
Proofofthemastertheorem
• We’lldothesamerecursiontreethingwedidforMergeSort,butbemorecareful.
• Supposethat 𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑐 ⋅ 𝑛H.
Pluckythe
PedanticPenguin
Hangon!ThehypothesisoftheMasterTheoremwas
thetheextraworkateachlevelwasO(nd).That’sNOT
thesameaswork<=cnd forsomeconstantc.
That’strue… we’llactuallyproveaweaker
statementthatusesthishypothesisinsteadof
thehypothesisthat𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .
It’sagoodexercisetomakethisproofwork
rigorouslywiththeO()notation.
Siggi theStudiousStork
Recursiontree
Sizen
n/bn/b
(Size1)
…
n/b2
n/btn/btn/btn/btn/btn/bt
…
Level
Amountof
workatthis
level
0
#
problems
1
2
t
logb(n)
1
a
a2
at
𝑎?@AL *
Sizeof
each
problem
n
n/b
n/b2
n/bt
1
…
n/b
n/b2
n/b2n/b2
n/b2
n/b2
n/b2
𝑇 𝑛 = 𝑎 ⋅ 𝑇 𝑛𝑏 + 𝑐 ⋅ 𝑛H
𝑐 ⋅ 𝑛𝑑
𝑎+𝑐 𝑛𝑏+
H
𝑎𝑐 𝑛𝑏H
𝑎O𝑐 𝑛𝑏O
H
𝑎?@AL * 𝑐
Recursiontree
Sizen
n/bn/b
(Size1)
…
n/b2
n/btn/btn/btn/btn/btn/bt
…
Level
Amountof
workatthis
level
0
#
problems
1
2
t
logb(n)
1
a
a2
at
𝑎?@AL *
Sizeof
each
problem
n
n/b
n/b2
n/bt
1
…
n/b
n/b2
n/b2n/b2
n/b2
n/b2
n/b2
𝑇 𝑛 = 𝑎 ⋅ 𝑇 𝑛𝑏 + 𝑐 ⋅ 𝑛H
𝑐 ⋅ 𝑛𝑑
𝑎+𝑐 𝑛𝑏+
H
𝑎𝑐 𝑛𝑏H
𝑎O𝑐 𝑛𝑏O
H
𝑎?@AL * 𝑐
Totalwork(derivationonboard)isatmost:
𝑐 ⋅ 𝑛H ⋅ P 𝑎𝑏H
OQRSL(*)
OTU
Nowlet’scheckallthecases(onboard)
Evenmoregenerally,forT(n)=aT(n/b)+f(n)…
[FromCLRS]
OllietheOver-AchievingOstrich
Figureouthowtoadapt
theproofwegavetoprove
thismoregeneralversion!
UnderstandingtheMasterTheorem
• Whatdothesethreecasesmean?
• Suppose𝑇 𝑛 = 𝑎 ⋅ 𝑇 *F + 𝑂 𝑛H .Then
Theeternalstruggle
Branchingcausesthenumber
ofproblemstoexplode!
Themostworkisatthe
bottomofthetree!
Theproblemslowerin
thetreearesmaller!
Themostworkisat
thetopofthetree!
Considerourthreewarm-ups
1. 𝑇 𝑛 = 𝑇 *+ + 𝑛
2. 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 𝑛
3. 𝑇 𝑛 = 4 ⋅ 𝑇 *+ + 𝑛
Firstexample:tallandskinnytree
Sizen
n/2
n/4
n/2t
1
1. 𝑇 𝑛 = 𝑇 *+ + 𝑛, 𝑎 < 𝑏H
• Theamountofworkdoneatthe
top(thebiggestproblem)swamps
theamountofworkdoneanywhere
else.
• T(n)=O(workattop)=O(n)
Mostworkatthe
topofthetree!
WINNER
Thirdexample:bushytree
Sizen
n/2
3. 𝑇 𝑛 = 4 ⋅ 𝑇 *+ + 𝑛, 𝑎 > 𝑏H
• ThereareaHUGEnumberofleaves,andthetotalworkis
dominatedbythetimetodoworkattheseleaves.
• T(n)=O(workatbottom)=O(4depthoftree )=O(n2)
n/2n/2n/2
1 11111 111
1 11111 111
1 11111 111
1 11111 111
1 11111 111
1 111 111
1 1111 111
WINNER
Mostworkat
thebottom
ofthetree!
Secondexample:justright
2. 𝑇 𝑛 = 2 ⋅ 𝑇 *+ + 𝑛, 𝑎 = 𝑏H
• Thebranchingjustbalances
outtheamountofwork.
Sizen
n/2
n/4
1
n/2
n/4n/4n/4
11111 111
• Thesameamountofwork
isdoneateverylevel.
• T(n)=(numberoflevels)*(workperlevel)
• =log(n)*O(n)=O(nlog(n))
1
TIE!
Recap
• The”MasterMethod” makesourliveseasier.
• Butit’sbasicallyjustcodifyingacalculationwecoulddofromscratchifwewantedto.
NextTime
• Whatifthesub-problemsaredifferentsizes?
• Andwhenmightthathappen?
BEFORE NextTime
• Pre-LectureExercise4!• WhichshouldbeeasierifyoudidPre-LectureExercise3…