lecture 4 - stanford...

Lecture4TheSubstitutionMethodandMedianandSelection

Announcements!

• HW1dueFriday.

• (AndHW2alsopostedFriday).

LastTime:TheMasterTheorem

• Suppose𝑇 𝑛 = 𝑎 ⋅ 𝑇&

'+ 𝑂 𝑛* .Then

A powerful

theorem it is…

JedimasterYoda

Threeparameters:

a:numberofsubproblems

b:factorbywhichinputsizeshrinks

d:needtodond worktocreateallthe

subproblems andcombinetheirsolutions.

Todaymorerecursion,beyondtheMasterTheorem.

• TheMasterTheoremonlyworkswhenallsub-problemsarethesamesize.

• That’snotalwaysthecase.

I can handle all the recurrence

relations that look like

𝑇 𝑛 = 𝑎 ⋅ 𝑇&

'+ 𝑂 𝑛* .

Before this theorem I was but

the learner. Now I am the

master.Only a master of evil*, Darth.

*Moreprecisely,onlyamasterofsame-size

sub-problems…stillprettyhandy,actually.

• Todaywe’llseeanexample

wheretheMasterTheorem

won’twork.

• We’llusesomethingcalled

thesubstitutionmethod

instead.

ThePlan

1. TheSubstitutionMethod

• Yougotasneakpeakonyourpre-lectureexercise

2. TheSELECT problem.

3. TheSELECT solution.

4. ReturnoftheSubstitutionMethod.

Anon-treemethod

• Here’sanotherwaytosolve:

• 𝑇 𝑛 = 2 ⋅ 𝑇&

,+ 𝑛

• 𝑇 0 = 0, 𝑇 1 = 1

1. Guesswhattheansweris.

2. Formallyprovethatthat’swhattheansweris.

Youdidthisforyourpre-lectureexercise!

Let’sgothroughitnowquicklytomakesure

weareallonthesamepage.

Formostofthislecture,

divisionisintegerdivision:&

,means

&

,.

Aswenotedlasttimewe’ll

beprettysloppyaboutthe

difference.

• 𝑇 𝑛 = 2 ⋅ 𝑇&

,+ 𝑛

• 𝑇 𝑛 = 2 ⋅ 2 ⋅ 𝑇&

6+

&

,+ 𝑛

• 𝑇 𝑛 = 4 ⋅ 𝑇&

6+ 2 ⋅ 𝑛

• 𝑇 𝑛 = 4 ⋅ 2 ⋅ 𝑇&

8+

&

6+ 2 ⋅ 𝑛

• 𝑇 𝑛 = 8 ⋅ 𝑇&

8+ 3 ⋅ 𝑛

• Followingthepattern…

• 𝑻 𝒏 = 𝒏 ⋅ 𝑻 𝟏 + 𝒍𝒐𝒈 𝒏 ⋅ 𝒏 = 𝒏(𝒍𝒐𝒈 𝒏 + 𝟏)

Sothatisourguess!

1. Guesswhattheansweris.

• Inductivehypothesis:

• 𝑻 𝒌 ≤ 𝒌(𝒍𝒐𝒈 𝒌 + 𝟏) forall1 ≤ k ≤ n

• Basecase:

• 𝑻 𝟏 = 𝟏 = 𝟏(𝒍𝒐𝒈 𝟏 + 𝟏)

• Inductivestep:

• 𝑻 𝒏 = 2 ⋅ 𝑇&

,+ 𝑛

≤ 2&

,log

&

,+ 1 + 𝑛

= 2&

,log 𝑛 − 1 + 1 + 𝑛

= 2&

,log 𝑛 + 𝑛

= 𝒏(𝐥𝐨𝐠 𝒏 + 𝟏)

• Conclusion:

• Byinduction,𝑻 𝒏 = 𝒏(𝒍𝒐𝒈 𝒏 + 𝟏) foralln>0.

2.Proveourguessisright

Whathappenedbetween

thesetwolines?

We’llgofastthrough

thesecomputations

becauseyoualldidit

onyourpre-lecture

exercise!

That’scalledthe

substitutionmethod

• Sofar,justseemslikeadifferentwayofdoingthesamething.

• Butconsiderthis!

𝑇 𝑛 = 3𝑛 + 𝑇𝑛

5+ 𝑇

𝑛

2

𝑇 𝑛 = 10𝑛when1 ≤ 𝑛 ≤ 10

Gross!Step1:guesswhattheansweris

• Let’strythesameunwindingthingtogetafeelforit.• [Onboard]

• Okay,thatgetsgrossfast.Wecanalsojusttryitout.• [IPython Notebook]

• Whatelsedoweknow?:

• 𝑇 𝑛 ≤ 3𝑛 + 𝑇&

P+ 𝑇

&

,

≤ 3𝑛 + 2 ⋅ 𝑇&

,

= 𝑂(𝑛 log 𝑛 )

• 𝑇 𝑛 ≥ 3𝑛

• SotherightanswerissomewherebetweenO(n)andO(nlog(n))…

𝑇 𝑛 = 3𝑛 + 𝑇𝑛

5+ 𝑇

𝑛

2𝑇 𝑛 = 10𝑛when1 ≤ 𝑛 ≤ 10

Let’sguessO(n)Step2:proveourguessisright

• InductiveHypothesis:𝑻 𝒌 ≤ 𝑪𝒌 forall1 ≤ 𝑘 < 𝑛.

• Basecase:𝑻 𝒌 ≤ 𝑪𝒌 forallk ≤ 10

• Inductivestep:

• 𝑇 𝑛 = 3𝑛 + 𝑇&

P+ 𝑇

&

,

≤ 3𝑛 + 𝑪&

P+ 𝑪

&

,

= 3𝑛 +𝑪

P𝑛 +

𝑪

,𝑛

≤ 𝑪𝑛 ??

• Conclusion:

• Thereissome𝑪 sothatforall𝑛 ≥ 1, 𝑻 𝒏 ≤ 𝑪𝒏

• Aka,T(n)=O(n).

𝑇 𝑛 = 3𝑛 + 𝑇𝑛

5+ 𝑇

𝑛

2

𝑇 𝑛 = 10𝑛when1 ≤ 𝑛 ≤ 10

Cissome

constantwe’ll

havetofillin

later!

Whateverwe

chooseCtobe,it

shouldhaveC≥10

Let’ssolveforCand

makethistrue!

C=10works.

(onboard)

Nowpretendlikeweknewitallalong.

• InductiveHypothesis:𝑻 𝒌 ≤ 𝟏𝟎𝒌 forallk<n.

• Basecase:𝑻 𝒌 ≤ 𝟏𝟎𝒌 forallk ≤ 10

• Inductivestep:

• 𝑇 𝑛 = 3𝑛 + 𝑇&

P+ 𝑇

&

,

• 𝑇 𝑛 ≤ 3𝑛 + 𝟏𝟎&

P+ 𝟏𝟎

&

,

• 𝑇 𝑛 ≤ 3𝑛 + 2𝑛 + 5𝑛 = 10𝑛.

• Conclusion:

• Forall𝑛 ≥ 1, 𝑻 𝒏 ≤ 𝟏𝟎𝒏,akaT(n)=O(n).

𝑇 𝑛 = 3𝑛 + 𝑇𝑛

5+ 𝑇

𝑛

2

𝑇 𝑛 = 10𝑛when1 ≤ 𝑛 ≤ 10

Theorem:𝑇 𝑛 = 𝑂 𝑛

Proof:

Whathavewelearned?

• Thesubstitutionmethodcanworkwhenthemastertheoremdoesn’t.• Forexamplewithdifferent-sizedsub-problems.

• Step1:generateaguess• Throwthekitchensinkatit.

• Step2:trytoprovethatyourguessiscorrect• Youmayhavetoleavesomeconstantsunspecifiedtilltheend– thenseewhattheyneedtobefortheprooftowork!!

• Step3:profit• Pretendyoudidn’tdoSteps1and2andwritedownaniceproof.

ThePlan






Theproblemwewillsolve

• SELECT(A,k):

• Returnthek’th smallestelementofA.

Aisanarrayofsizen,kisin{1,…,n}

7 4 3 8 1 5 9 14

• SELECT(A,1)=MIN(A)

• SELECT(A,n/2)=MEDIAN(A)

• SELECT(A,n)=MAX(A)

• SELECT(A,1)=1

• SELECT(A,2)=3

• SELECT(A,3)=4

• SELECT(A,8)=14

Beingsloppyabout

floorsandceilings!

NotethatthedefinitionofSelectis1-indexed…

We’regonna doitintimeO(n)

• Let’sstartwithMIN(A)akaSELECT(A,1).

• MIN(A):

• ret=∞

• For i=0,...,n-1:

• IfA[i]<ret:

• ret=A[i]

• Return ret

• TimeO(n).Yay!

ThisstuffisO(1)

ThislooprunsO(n)times

HowaboutSELECT(A,2)?

• SELECT2(A):

• ret=∞

• minSoFar=∞

• For i=0,..,n-1:

• If A[i]<retandA[i]<minSoFar:

• ret=minSoFar

• minSoFar =A[i]

• Else ifA[i]<retandA[i]>=minSoFar:

• ret=A[i]

• Return ret

(Theactualalgorithmhereis

notveryimportantbecause

thiswon’tendupbeinga

verygoodidea…)

StillO(n)SOFARSOGOOD.

SELECT(A,n/2)akaMEDIAN(A)?

• MEDIAN(A):

• ret=∞

• minSoFar=∞

• secondMinSoFar =∞

• thirdMinSoFar =∞

• fourthMinSoFar =∞

• ….

• Thisisnotagoodideaforlargek(liken/2orn).

• BasicallythisisjustgoingtoturnintosomethinglikeINSERTIONSORT…andthatwasO(n2).

Amuchbetterideaforlargek

• SELECT(A,k):

• A=MergeSort(A)

• return A[k-1]

• RunningtimeisO(nlog(n)).

• Sothat’sthebenchmark….

Canwedobetter?We’rehopingtogetO(n)

It’sk-1andnotksincemy

pseudocodeis0-indexedand

theproblemis1-indexed…

Showthatyoucan’t

dobetterthanO(n)!

(Orseelecturenotes).

ThePlan






Idea:divideandconquer!

9 8 3 6 1 4 2Saywewantto

findSELECT(A,k)

First,picka“pivot.”

We’llseehowtodo

thislater.

Howabout

thispivot?

Next,partitionthearrayinto

“biggerthan6”or“lessthan6”

9 8 3 6 1 4 2

L=arraywiththings

smallerthanA[pivot]

R=arraywiththings

largerthanA[pivot]

ThisPARTITIONsteptakes

timeO(n).(Noticethat

wedon’tsorteachhalf).

Idea:divideandconquer!

6Saywewantto

findSELECT(A,k)

First,picka“pivot.”

We’llseehowtodo

thislater.

Howabout

thispivot?

Next,partitionthearrayinto

“biggerthan6”or“lessthan6”

9 83

6

1 4 2

L=arraywiththings

smallerthanA[pivot]

R=arraywiththings

largerthanA[pivot]

ThisPARTITIONsteptakes

timeO(n).(Noticethat

wedon’tsorteachhalf).

Ideacontinued…

9 83

6

1 4 2pivot

L=arraywiththings

smallerthanA[pivot]

R=arraywiththings

largerthanA[pivot]

Saywewantto

findSELECT(A,k)

• Ifk=5=len(L)+1:

• WeshouldreturnA[pivot]

• Ifk<5:

• WeshouldreturnSELECT(L,k)

• Ifk>5:

• WeshouldreturnSELECT(R,k– 5)

Thissuggestsa

recursivealgorithm

(stillneedtofigureout

howtopickthepivot…)

Pseudocode• getPivot(A)returnssomepivotforus.

• How??We’llseelater…

• Partition(A,p) splitsupAintoL,A[p],R.

• SeeLecture4notebookforcode

• Select(A,k):

• If len(A)<=50:

• A=MergeSort(A)

• Return A[k-1]

• p=getPivot(A)

• L,pivotVal,R=Partition(A,p)

• if len(L)==k-1:

• returnpivotVal

• Elseiflen(L)>k-1:

• returnSelect(L,k)

• Elseiflen(L)<k-1:

• returnSelect(R,k– len(L)– 1)

BaseCase:Ifthelen(A)=O(1),

thenanysortingalgorithm

runsintimeO(1).

Case1:Wegotluckyandfound

exactlythek’th smallestvalue!

Case2:Thek’th smallestvalue

isinthefirstpartofthelist


isinthesecondpartofthelist

Let’smakesureitworks

• [IPython NotebookforLecture4]

Nowweshouldbeconvinced

• NomatterwhatprocedureweuseforgetPivot(A),Select(A,k)returnsacorrectanswer.

Siggi theStudiousStork

Formallyprovethe

correctnessof

Select!

Whatistherunningtime?

• 𝑇 𝑛 =

𝑇 𝐥𝐞𝐧 𝐋 + 𝑂 𝑛 𝐥𝐞𝐧 𝐋 > 𝑘 − 1

𝑇 𝐥𝐞𝐧 𝐑 + 𝑂 𝑛 𝐥𝐞𝐧 𝐋 < 𝑘 − 1

𝑂 𝑛 𝐥𝐞𝐧 𝐋 = 𝑘 − 1

• Whatarelen(L) andlen(R)?

• Thatdependsonhowwepickthepivot…

• Whatdowehopehappens?

• Whatdowehopedoesn’thappen?

Inanidealworld*…

• Wesplittheinputinhalf:

• len(L)=len(R)=(n-1)/2

• Let’susetheMasterTheorem!


'+ 𝑂 𝑛* .Then

Apply here, the Master

Theorem does NOT.

Making unsubstantiated

assumptions about

problem sizes, we are.

JedimasterYoda

• 𝑇 𝑛 ≤ 𝑇&

,+ 𝑂(𝑛)

• Soa=1,b=2,d=1

• 𝑇 𝑛 ≤ 𝑂 𝑛* = 𝑂 𝑛

*Okay,reallyidealwouldbethatwealwayspickthe

pivotsothatlen(L)=k-1.Butsaywedon’thave

controloverk,justoverhowwepickthepivot.

Buttheworldisnotideal.

• Supposewechooseapivotfirst,butthen abadguywhoknowswhatpivotswewillchoose getstocomeupwithA.

• [Discussiononboard]

Thedistinctionmatters!

SeeLecture4IPython notebookforcodethatgeneratedthispicture.

Thisoneisarandom

pivot,soitsplitsthe

arrayaboutinhalf.

Looksprettyfast!

ForthisoneIchosetheworst

possiblepivot.LookslikeO(n2).

Question

• Inpractice,thereisoftennobadguy.Inthatcase,justpickarandompivotanditworksreallywell!

• (Moreonthisnextweek)

Aside:

• Howdowepickagoodpivot?

• Randomly?

• Thatworkswellifthere’snobadguy.

• Butifthereisabadguywhogetstoseeourpivotchoices,that’sjustasbadastheworst-casepivot.

Butfortoday

• Let’sassumethere’sthisbadguy.

• We’llgetastrongerguarantee

• We’llgettoseeareallycleveralgorithm

• Andwe’llgetmorepracticewiththesubstitutionmethod.

ThePlan





a) Theoutlineofthealgorithm.

b) Howtopickthepivot.


Howshouldwepickthepivot?

• We’dliketoliveintheidealworld.

• Pickthepivottodividetheinputinhalf!

• Aka,pickthemedian!

• Aka,pickSelect(A,n/2)

Howshouldwepickthepivot?

• We’dliketoapproximate theidealworld.

• Pickthepivottodividetheinputabout inhalf!

• Maybethisiseasier!

Inanidealworld…

• Wesplittheinputnotquiteinhalf:

• 3n/10<len(L)<7n/10

• 3n/10<len(R)<7n/10

• Ifwecoulddothat,theMasterTheorem wouldsay:


'+ 𝑂 𝑛* .Then

• 𝑇 𝑛 ≤ 𝑇]&

^_+ 𝑂(𝑛)

• Soa=1,b=10/7,d=1

• 𝑇 𝑛 ≤ 𝑂 𝑛* = 𝑂 𝑛

okayApply here, the Master Theorem

STILL does NOT. (Since we don’t

know that we can do this – and if we

could how long would it take?).

JedimasterYoda

Butatleastit

givesusagoal!

Luckythe

lackadaisicallemur

Goal

• Pickthepivotsothat

9 83

6

1 4 2pivot

L=arraywiththings

smallerthanA[pivot]

R=arraywiththings

largerthanA[pivot]

𝟑𝒏

𝟏𝟎< 𝐥𝐞𝐧 𝑳 <

𝟕𝒏

𝟏𝟎

𝟑𝒏

𝟏𝟎< 𝐥𝐞𝐧 𝑹 <

𝟕𝒏

𝟏𝟎

Anotherdivide-and-conqueralg!• Wecan’tsolveSelect(A,n/2) (yet)

• ButwecandivideandconquerandsolveSelect(B,m/2)forsmallervaluesofm(wherelen(B)=m).

• Lemma*:Themedianofsub-mediansisclosetothemedian.

*wewillmakethisabitmoreprecise.

sub-mediansub-mediansub-mediansub-mediansub-median

medianof

sub-medians

medianofthe

wholething≈

Whatwe’lluseasthepivotIdealpivot

Howtopickthepivot

• CHOOSEPIVOT(A):

• SplitAintom=&

Pgroups,ofsize<=5each.

• For i=1,..,m:

• Findthemedianwithinthei’th group,callitpi

• p= SELECT([p1,p2,p3,…,pm ],m/2 )

• return p

5 9 1 3 41 8 9 3 15 12 2 1 5 20 15 13 2 4 6 12 1 15 22 3

ThistakestimeO(1),foreachgroup,sinceeachgroup

hassize5.Sothat’sO(m)totalintheforloop.8

4

5

6

12PivotisSELECT(,3)=6:8 4 5 6 12

5 9 1 3 41 8 9 3 15 12 2 1 5 20 15 13 2 46

12 1 15 22 3

5 91 3 41 8 93 15 122 1 5 20 15 132 4

6121 15 223

PARTITIONaroundthat6:

ThispartisL ThispartisR:it’salmostthesamesizeasL.

CLAIM:thisworksdividesthearrayapproximately inhalf

• Empirically(seeLecture4IPython Notebook):

CLAIM:thisworksdividesthearrayapproximately inhalf

• Formally,wewillprove(later):

Lemma:Ifwechoosethepivotslikethis,then

𝐿 ≤7𝑛

10+ 5

and

𝑅 ≤7𝑛

10+ 5

SanityCheck𝐿 ≤

]&

^_+ 5and 𝑅 ≤

]&

^_+ 5

That’sthiswindow

Actuallyinpractice

(onrandomly

chosenarrays)it

looksevenbetter!

Butthisisaworst-

castbound.

Howabouttherunningtime?

• SupposetheLemmaistrue.(Itis).

• 𝐿 ≤]&

^_+ 5and 𝑅 ≤

]&

^_+ 5

• Recurrencerelation:

𝑇 𝑛 ≤?

Pseudocode• getPivot(A)returnssomepivotforus.

• How??We’llseelater…

• Partition(A,p) splitsupAintoL,A[p],R.

• SeeLecture4notebookforcode

• Select(A,k):

• If len(A)<=50:

• A=MergeSort(A)

• Return A[k-1]

• p=getPivot(A)

• L,pivotVal,R=Partition(A,p)

• if len(L)==k-1:

• returnpivotVal

• Elseiflen(L)>k-1:

• returnSelect(L,k)

• Elseiflen(L)<k-1:

• returnSelect(R,k– len(L)– 1)

BaseCase:Ifthelen(A)=O(1),

thenanysortingalgorithm

runsintimeO(1).

Case1:Wegotluckyandfound

exactlythek’th smallestvalue!


isinthefirstpartofthelist


isinthesecondpartofthelist

Howabouttherunningtime?

• SupposetheLemmaistrue.(Itis).

• 𝐿 ≤]&

^_+ 5and 𝑅 ≤

]&

^_+ 5

• Recurrencerelation:

𝑇 𝑛 ≤ 𝑇&

P+ 𝑇

]&

^_+ 𝑂 𝑛

ThePlan








Thissoundslikeajobfor…

[Onboard]

𝑇 𝑛 ≤ 𝑇&

P+ 𝑇

]&

^_+ 𝑂 𝑛

Conclusion:𝑇 𝑛 = 𝑂 𝑛

Step1:generateaguess

Step2:trytoprovethatyourguessiscorrect

Step3:profit

Likewedidlasttime,treatthis

O(n)ascn forouranalysis.

(Forsimplicityinclass– tobe

rigorousweshouldusethe

formaldefinition!)

Inpractice?• Withmydumbimplementation,ourfancyversionofSelect isworse

thanMergeSort-based Select.L

• ButO(n)isbetterthanO(nlog(n))!Howcanthatbe?

• What’stheconstantinfrontoftheninourproof?20?30?

• Onnon-adversarial inputs,randompivotchoiceisMUCH better.

Optimizetheimplementationof

Select (withthefancypivot).

CanyoubeatMergeSort?

Siggi theStudiousStork

Whathavewelearned?PendingtheLemma

• ItispossibletosolveSELECTintimeO(n).

• Divideandconquer!

• Ifyouexpectthatabadguy*willbepickingthelist,chooseapivotcleverly.

• Moredivideandconquer!

• Ifyoudon’texpectthatabadguy*willbepickingthelist,inpracticeit’sbetterjusttopickarandompivot.

*Abadguywhoknowsyourpivotchoicesaheadoftime.

ThePlan








5. (Iftime)Proofofthatlemma.

Iftime,backtotheLemma

• Lemma:IfLandRareasinthealgorithmSELECTgivenabove,then

𝐿 ≤7𝑛

10+ 5

and

𝑅 ≤7𝑛

10+ 5

• Wewillseeaproofbypicture.

• SeeCLRSforproofbyproof.

Proofbypicture

1

8

9

3

15

5

18

4

6

35

2

10

7

12

11

3

13

70

4

2

6

7

17

22

Saytheseareourm=[n/5]sub-arraysofsizeatmost5.

5

m

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

Inourhead,let’ssortthem.

5

m

Thenfindmedians.

8 6 10 4

7

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

Thenlet’ssortthembythemedian

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

Themedianofthemediansis7.That’sourpivot!

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

HowmanyelementsareSMALLERthanthepivot?

Wewillshowthatlotsofelementsare

smallerthanthepivot,hencenottoo

manyarelargerthanthepivot.

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

Atleasttheseones:everythingaboveandtotheleft.

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

mHowmanyofthosearethere?

atleast3 ⋅j

,− 2

3 ⋅j

,− 1 ofthese,but

thenoneofthemcouldhave

beenthe“leftovers”group.

Proofbypicture

1

3

8

9

15

4

5

6

18

35

2

7

10

11

12

2

3

4

13

70

6

7

17

22

5

m

SohowmanyareLARGERthanthepivot?Atmost

𝑛 − 1 − 3𝑚

2− 2 ≤

7𝑛

10+ 5

Remember

𝑚 =𝑛

5

(derivation

onboard)

Thatwasonepartofthelemma

• Lemma:IfLandRareasinthealgorithmSELECTgivenabove,then

𝐿 ≤7𝑛

10+ 5

and

𝑅 ≤7𝑛

10+ 5

Theotherpartisexactlythesame.

Recap








ThePlan

Recap

• Thesubstitutionmethodisanotherwaytosolverecurrencerelations.

• Canworkwhenthemastertheoremdoesn’t!

• OneplaceweneededitwasforSELECT.

• WhichwecandointimeO(n)!

Nexttime

• RandomizedalgorithmsandQuickSort!

BEFORE nexttime

• Pre-LectureExercise5

• Rememberprobabilitytheory?

• Thepre-lectureexercisewilljogyourmemory.

lecture 4 - stanford...

Documents