UNDERSTANDING PROGRAM EFFICIENCY: 2 (download slides and .py files and follow along!)

6.0001 LECTURE 11

6.0001LECTURE11 1

TODAY § Classesofcomplexity

§ Examplescharacteris;cofeachclass

6.0001LECTURE11 2

WHY WE WANT TO UNDERSTAND EFFICIENCY OF PROGRAMS § howcanwereasonaboutanalgorithminordertopredicttheamountof;meitwillneedtosolveaproblemofapar;cularsize?

§ howcanwerelatechoicesinalgorithmdesigntothe;meefficiencyoftheresul;ngalgorithm?◦ aretherefundamentallimitsontheamountof;mewewillneedtosolveapar;cularproblem?

6.0001LECTURE11 3

ORDERS OF GROWTH: RECAP Goals:§ wanttoevaluateprogram’sefficiencywheninputisverybig§ wanttoexpressthegrowthofprogram’srun5measinputsizegrows§ wanttoputanupperboundongrowth–as;ghtaspossible§ donotneedtobeprecise:“orderof”not“exact”growth§ wewilllookatlargestfactorsinrun;me(whichsec;onoftheprogramwilltakethelongesttorun?)§ thus,generallywewant5ghtupperboundongrowth,asfunc5onofsizeofinput,inworstcase

6.0001LECTURE11 4

COMPLEXITY CLASSES: RECAP § § § § § co§ coin

O(1)denotesconstantrunning;meO(logn)denoteslogarithmicrunning;meO(n)denoteslinearrunning;meO(nlogn)denoteslog-linearrunning;meO(nc)denotespolynomialrunning;me(cisanstant)

O(cn)denotesexponen;alrunning;me(cisanstantbeingraisedtoapowerbasedonsizeofput)

6.0001LECTURE11 5

COMPLEXITY CLASSES ORDERED LOW TO HIGH

O(1) : constant O(log n) : logarithmic O(n) : linear O(n log n): loglinear O(nc) : polynomial O(cn) : exponen;al

6.0001LECTURE11 6

COMPLEXITY GROWTH CLASS n=10 =100 =1000 =1000000

O(1) 1 1 1 1

O(logn) 1 2 3 6

O(n) 10 100 1000 1000000

O(nlogn) 10 200 3000 6000000

O(n^2) 100 10000 1000000 1000000000000

O(2^n) 1024 12676506 1071508607186267320948425049060 Goodluck!!00228229 001810561404811705533607443750340149670 88370351051124936122493198378813205376 5695858127594672917553146825187

14528569231404359845775746985748039345677748242309854210746050623711418779541821530464749835819412673987675591655439460770629145711964776865421676604298316

52624386837205668069376

6.0001LECTURE11 7

CONSTANT COMPLEXITY § complexityindependentofinputs

§ veryfewinteres;ngalgorithmsinthisclass,butcanoYenhavepiecesthatfitthisclass

§ canhaveloopsorrecursivecalls,butONLYIFnumberofitera;onsorcallsindependentofsizeofinput

6.0001LECTURE11 8

LOGARITHMIC COMPLEXITY § complexitygrowsaslogofsizeofoneofitsinputs

§ example:◦ bisec;onsearch◦ binarysearchofalist

6.0001LECTURE11 9

BISECTION SEARCH § supposewewanttoknowifapar;cularelementispresentinalist

§ sawlast;methatwecouldjust“walkdown”thelist,checkingeachelement

§ complexitywaslinearinlengthofthelist

§ supposeweknowthatthelistisorderedfromsmallesttolargest◦ sawthatsequen;alsearchwass;lllinearincomplexity◦ canwedobecer?

6.0001LECTURE11 10

BISECTION SEARCH 1.  pickanindex,i,thatdivideslistinhalf2.  askifL[i] == e 3.  ifnot,askifL[i] islargerorsmallerthane 4.  L e

Anewversionofadivide-and-conqueralgorithm§  breakintosmallerversionofproblem(smallerlist),plus

somesimpleopera;ons§  answertosmallerversionisanswertooriginalproblem

dependingonanswer,searchleYorrighthalfof for

6.0001LECTURE11 11

BISECTION SEARCH COMPLEXITY ANALYSIS

§ finishlookingthroughlistwhen

1=n/2i

soi=logn

§ complexityof

recursionisO(logn)–wherenislen(L)

6.0001LECTURE11 12

…

…

BISECTION SEARCH IMPLEMENTATION 1 def bisect_search1(L, e):!

if L == []:!

return False!

elif len(L) == 1:!

return L[0] == e!

else:!

half = len(L)//2!

if L[half] > e:!

return bisect_search1( L[:half], e)!

else:!

return bisect_search1( L[half:], e)!

6.0001LECTURE11 13

COMPLEXITY OF FIRST ETHOD BISECTION SEARCH M

§ implementa5on1–bisect_search1•  O(logn)bisec;onsearchcalls

ll,sizeofrangetobesearchediscutinhalfsizen,inworstcasedowntorangeofsize1whenk=logn;onsearchcalltocopylistupeachcall,sodothisforeachlevelof

(nlogn)eful,notethatlengthoflisttobedoneachrecursivecallosttocopyisO(n)andthisdominatesthelogursivecalls

6.0001LECTURE11 14

•  Oneachrecursiveca•  Iforiginalrangeisofwhenn/(2^k)=1;or

• O(n)foreachbisec•  Thisisthecosttosetrecursion

• O(logn)*O(n)àO•  ifwearereallycarcopiedisalsohalve•  turnsoutthattotalcncostduetotherec

BISECTION SEARCH ALTERNATIVE

§ s;llreducesizeofproblembyfactoroftwooneachstep§ butjustkeeptrackoflowandhighpor;onoflisttobesearched§ avoidcopyingthelist§ complexityofrecursionisagainO(logn)–wherenislen(L)

6.0001LECTURE11 15

def bisect_search2(L, e):! def bisect_search_helper(L, e, low, high):! if high == low:! return L[low] == e! mid = (low + high)//2! if L[mid] == e:! return True! elif L[mid] > e:! if low == mid: #nothing left to search! return False! else:! return bisect_search_helper(L, e, low, mid - 1)! else:! return bisect_search_helper(L, e, mid + 1, high)! if len(L) == 0:! return False! else:! return bisect_search_helper(L, e, 0, len(L) - 1)!

6.0001LECTURE11 16

BISECTION SEARCH IMPLEMENTATION 2

COMPLEXITY OF SECOND BISECTION SEARCH METHOD § implementa5on2–bisect_search2anditshelper• O(logn)bisec;onsearchcalls•  Oneachrecursivecall,sizeofrangetobesearchediscutinhalf•  Iforiginalrangeisofsizen,inworstcasedowntorangeofsize1whenn/(2^k)=1;orwhenk=logn

• passlistandindicesasparameters•  listnevercopied,justre-passedasapointer•  thusO(1)workoneachrecursivecall• O(logn)*O(1)àO(logn)

6.0001LECTURE11 17

LOGARITHMIC COMPLEXITY def intToStr(i):! digits = '0123456789'! if i == 0:! return '0'! result = ''! while i > 0:! result = digits[i%10] + result! i = i//10! return result!

!

6.0001LECTURE11 18

LOGARITHMIC COMPLEXITY def intToStr(i):!  onlyhavetolookatloopas

nofunc;oncalls

 withinwhileloop,constantnumberofsteps

 howmany;mesthrough! loop?

◦  howmany;mescanonedivideiby10?

◦ O(log(i))

digits = '0123456789'! if i == 0:! return '0'! res = ''! while i > 0:! res = digits[i%10] + res i = i//10! return result!

6.0001LECTURE11 19

LINEAR COMPLEXITY § sawthislast;me◦  searchingalistinsequencetoseeifanelementispresent

◦  itera;veloops

6.0001LECTURE11 20

O() FOR ITERATIVE FACTORIAL erofitera;vecalls

):!

p,constantcosteach

§ complexitycandependonnumbdef fact_iter(n):!

prod = 1!

for i in range(1, n+1

prod *= i!

return prod!

§ overallO(n)–n;mesroundloo;me

6.0001LECTURE11 21

O() FOR RECURSIVE FACTORIAL def fact_recur(n):! """ assume n >= 0 """! if n <= 1: ! return 1! else: ! return n*fact_recur(n – 1)!

trunsabitslowerthancallsffunc;oncallsislinearpcalllimplementa;onsare

22

§ computesfactorialrecursively§ ifyou;meit,mayno;cethatiitera;veversionduetofunc;on§ s;llO(n)becausethenumberoinn,andconstantefforttosetu§ itera5veandrecursivefactoriathesameorderofgrowth

6.0001LECTURE11

LOG-LINEAR COMPLEITY

ismergesort

§ manyprac;calalgorithmsarelog-linear

§ verycommonlyusedlog-linearalgorithm

§ willreturntothisnextlecture

6.0001LECTURE11 23

POLYNOMIAL COMPLEXITY § mostcommonpolynomialalgorithmsarequadra;c,i.e.,complexitygrowswithsquareofsizeofinput

§ commonlyoccurswhenwehavenestedloopsorrecursivefunc;oncalls

§ sawthislast;me

6.0001LECTURE11 24

EXPONENTIAL COMPLEXITY § recursivefunc;onswheremorethanonerecursivecallforeachsizeofproblem◦ TowersofHanoi

§ manyimportantproblemsareinherentlyexponen;al◦ unfortunate,ascostcanbehigh◦ willleadustoconsiderapproximatesolu;onsasmayprovidereasonableanswermorequickly

6.0001LECTURE11 25

COMPLEXITY OF TOWERS OF HANOI § Lettndenote;metosolvetowerofsizen§ tn=2tn-1+1§ =2(2tn-2+1)+1§  =4tn-2+2+1§ =4(2t +1)+2+1 Geometricgrowth

n-3

§ =8tn-3+4+2+1 a=2n-1+…+2+1§ =2kt k-

n-k+2 1+…+4+2+1 2a=2n+2n-1+...+2a=2n-1§ =2n-1+2n-2+...+4+2+1

§ =2n–1§ soorderofgrowthisO(2n)

6.0001LECTURE11 26

EXPONENTIAL COMPLEXITY § givenasetofintegers(withnorepeats),wanttogeneratethecollec;onofallpossiblesubsets–calledthepowerset

§ {1,2,3,4}wouldgenerate◦  {},{1},{2},{3},{4},{1,2},{1,3},{1,4},{2,3},{2,4},{3,4},{1,2,3},{1,2,4},{1,3,4},{2,3,4},{1,2,3,4}

§ orderdoesn’tmacer◦  {},{1},{2},{1,2},{3},{1,3},{2,3},{1,2,3},{4},{1,4},{2,4},{1,2,4},{3,4},{1,3,4},{2,3,4},{1,2,3,4}

6.0001LECTURE11 27

POWER SET – CONCEPT § wewanttogeneratethepowersetofintegersfrom1ton§ assumewecangeneratepowersetofintegersfrom1ton-1§ thenallofthosesubsetsbelongtobiggerpowerset

allofthosesubsetswithnlongtothebiggerpowerset

(choosingnotincluden);andaddedtoeachofthemalsobe(choosingtoincluden)§ {},{1},{2},{1,2},{3},{1,3},{2,3},{1,4},{3,4},{1,3,4},{2,3,4},{1,2,3,4}

2,3},{4},{1,4},{2,4},{1,2,

§ nicerecursivedescrip;on!

6.0001LECTURE11 28

EXPONENTIAL COMPLEXITY def genSubsets(L):! res = []! if len(L) == 0:! return [[]] #list of empty list! smaller = genSubsets(L[:-1]) # all subsets without

last elementlast element! extra = L[-1:] # create a list of just new = []! for small in smaller:! new.append(small+extra) # for all smaller solutions, add one with last element! return smaller+new # combine those with last element and those without!

6.0001LECTURE11 29

!

EXPONENTIAL COMPLEXITY  assumingappendisconstant;me

 ;meincludes;metosolvesmallerproblem,plus;me

smaller = genSubsets(L[:-1])! neededtomakeacopyof extra = L[-1:]! allelementsinsmaller new = []! problem for small in smaller:! new.append(small+extra)! return smaller+new!

def genSubsets(L):! res = []! if len(L) == 0:! return [[]] !

6.0001LECTURE11 30

EXPONENTIAL COMPLEXITY def genSubsets(L):!  butimportanttothin res = []! aboutsizeofsmaller if len(L) == 0:!

k

return [[]] !  knowthatforasetofsize smaller = genSubsets(L[:-1])! kthereare2kcases extra = L[-1:]! new = []!  howcanwededuce for small in smaller:! overallcomplexity? new.append(small+extra)! return smaller+new!

6.0001LECTURE11 31

EXPONENTIAL COMPLEXITY § lettndenote;metosolveproblemofsizen

§ letsndenotesizeofsolu;onforproblemofsizen

§ tn=tn-1+sn-1+c(wherecissomeconstantnumberofopera;ons)

§ tn=tn-1+2n-1+c§ =tn-2+2n-2+c+2n-1+c

Thus§  =tn-k+2n-k+…+2n-1+kc compu;ng§ =t0+20+...+2n-1+nc powersetis§ =1+2n+nc O(2n)

6.0001LECTURE11 32

COMPLEXITY CLASSES § O(1)–codedoesnotdependonsizeofproblem

§ O(logn)–reduceprobleminhalfeach;methroughprocess

§ O(n)–simpleitera;veorrecursiveprograms

§ O(nlogn)–willseenext;me

§ O(nc)–nestedloopsorrecursivecalls§ O(cn)–mul;plerecursivecallsateachlevel

6.0001LECTURE11 33

SOME MORE EXAMPLES OF ANALYZING COMPLEXITY

6.0001LECTURE11 34

COMPLEXITY OF ITERATIVE FIBONACCI def fib_iter(n):! §  if n == 0:! Bestcase: return 0! O(1) elif n == 1:! return 1! § Worstcase: else:! O(1)+O(n)+O(1)èO(n) fib_i = 0! fib_ii = 1! for i in range(n-1):! tmp = fib_i! fib_i = fib_ii!

fib_ii = tmp + fib_ii!turn fib_ii !

re

6.0001LECTURE11 36

COMPLEXITY OF RECURSIVE FIBONACCI def fib_recur(n):! """ assumes n an int >= 0 """! if n == 0:! return 0! elif n == 1:! return 1! else:! return fib_recur(n-1) + fib_recur(n-2)!

§ Worstcase:O(2n)

6.0001LECTURE11 37

COMPLEXITY OF RECURSIVE FIBONACCI fib(5)

fib(4) fib(3)

fib(3) fib(2) fib(2) fib(1)

fib(2) fib(1)

§ actuallycandoabitbecerthan2nsincetreeofcasesthinsouttoright§ butcomplexityiss;llexponen;al

6.0001LECTURE11 38

BIG OH SUMMARY § compareefficiencyofalgorithms•  nota;onthatdescribesgrowth•  lowerorderofgrowthisbecer•  independentofmachineorspecificimplementa;on

§ useBigOh•  describeorderofgrowth•  asympto5cnota5on•  upperbound•  worstcaseanalysis

6.0001LECTURE11 40

COMPLEXITY OF COMMON PYTHON FUNCTIONS

§ Lists:n islen(L) § Dic;onaries:n islen(d)•  index O(1) § worstcase•  store O(1) •  index O(n)•  length O(1) •  store O(n)•  append O(1) •  length O(n)•  == O(n) •  delete O(n)•  remove O(n) •  itera;on O(n)•  copy O(n) §  averagecase•  reverse O(n) •  index O(1)•  itera;on O(n) •  store O(1)•  inlist O(n) •  delete O(1)

•  itera;on O(n)

6.0001LECTURE11 41

MIT OpenCourseWarehttps://ocw.mit.edu

6.0001 Introduction to Computer Science and Programming in PythonFall 2016

For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.