05.09.13  

1  

Advanced  Algorithmics  (6EAP)    

MTAT.03.238      Order  of  growth…  maths  

Jaak  Vilo  2013  fall  

1  Jaak  Vilo  

COURSES.cs.ut.ee

Program  execuHon  on  input  of  size  n  

•  How  many  steps/cycles  a  processor  would  need  to  do  

•  Faster  computer,  larger  input?    

•  But  bad  algorithm  on  fast  computer  will  be  outcompeted  by  good  algorithm  on  slow…  

•  How  to  relate  algorithm  execuHon  to  nr  of  steps  on  input  of  size  n?      f(n)            

•  e.g.  f(n)  =  n  +  n*(n-­‐1)/2  +  17  n  +  n*log(n)      

Big-­‐Oh  notaHon  classes  

Class   Informal   Intui5on   Analogy  

f(n)  ∈  ο  (  g(n)  )     f  is  dominated  by  g   Strictly  below   <  f(n)  ∈  O(  g(n)  )     Bounded  from  above   Upper  bound   ≤  f(n)  ∈  Θ(  g(n)  )     Bounded  from    

above  and  below  “equal  to”   =  

f(n)  ∈  Ω(  g(n)  )     Bounded  from  below   Lower  bound   ≥  f(n)  ∈  ω(  g(n)  )      

f  dominates  g   Strictly  above   >  

MathemaHcal  Background  JusHficaHon  

•  As  engineers,  you  will  not  be  paid  to  say:  Method  A  is  be#er  than  Method  B  

 or  Algorithm  A  is  faster  than  Algorithm  B  

•  Such  descripHons  are  said  to  be  qualita-ve  in  nature;  from  the  OED:        

qualita5ve,  a.  a  RelaHng  to,  connected  or  concerned  with,    quality  or  qualiHes.  Now  usually  in  implied  or  expressed  opposiHon  to  quanHtaHve.  

MathemaHcal  Background  JusHficaHon  

•  Business  decisions  cannot  be  based  on  qualitaHve  statements:  – Algorithm  A  could  be  be#er  than  Algorithm  B,  but  Algorithm  A  would  require  three  person  weeks  to  implement,  test,  and  integrate  while  Algorithm  B  has  already  been  implemented  and  has  been  used  for  the  past  year  

–  there  are  circumstances  where  it  may  beneficial  to  use  Algorithm  A,  but  not  based  on  the  word  be#er  

05.09.13  

2  

MathemaHcal  Background  JusHficaHon  

•  Thus,  we  will  look  at  a  quan-ta-ve  means  of  describing  data  structures  and  algorithms  

•  From  the  OED:  

   quan5ta5ve,  a.    RelaHng  to  or  concerned  with  quanHty  or  its  measurement;  that  assesses  or  expresses  quanHty.  In  later  use  freq.  contrasted  with  qualita-ve.    

 

Program  Hme  and  space  complexity  

•  Time:  count  nr  of  elementary  calculaHons/operaHons  during  program  execuHon  

•  Space:  count  amount  of  memory  (RAM,  disk,  tape,  flash  memory,  …)  – usually  we  do  not  differenHate  between  cache,  RAM,  …  

–  In  pracHce,  for  example,  random  access  on  tape  impossible  

•  Program  1.17    float  Sum(float  *a,  const  int  n)    {      float  s  =  0;      for  (int  i=0;  i<n;  i++)                s  +=  a[i];      return  s;    }  – The  instance  characterisHc  is  n.  – Since  a  is  actually  the  address  of  the  first  element  of  a[],  and  n  is  passed  by  value,  the  space  needed  by  Sum()  is  constant  (Ssum(n)=1).    

•  Program  1.18      float  RSum(float  *a,  const  int  n)    {      if  (n  <=  0)    return  0;      else  return  (RSum(a,  n-­‐1)  +  a[n-­‐1]);    }  

 – Each  call  requires  at  least  4  words  

•  The  values  of  n,  a,  return  value  and  return  address.  – The  depth  of  the  recursion  is  n+1.    

•  The  stack  space  needed  is  4(n+1).  

n=997

n=1000

n=999

n=998

Input  size  =  n  

•  usually  input  size  denoted  by  n  •  Time  complexity  funcHon  =  f(n)    

– array[1..n]  – e.g.  4*n  +  3  

 •  Graph:  n  verHces,  m  edges      f(n,m)    

05.09.13  

3  

05.09.13  

4  

•  So,  we  may  be  able  to  count  a  nr  of  operaHons  needed  

•  What  do  we  do  with  this  knowledge?    

n2

100 n log n

n2

100 n log n

n = 1000

05.09.13  

5  

0.00001*n2

100 n log n

n = 10,000,000

0.00001*n2

100 n log n

n = 2,000,000,000

Logscale y logscale x logscale y

•  plot  [1:10]  0.01*x*x  ,  5*log(x),  x*log(x)/3  

05.09.13  

6  

Algorithm  analysis  goal  

•  What  happens  in  the  “long  run”,  increasing  n  

•  Compare  f(n)  to  reference  g(n)              (or  f  and  g  )    

•  At  some  n0  >  0  ,      if  n  >  n0    ,  always  f(n)  <  g(n)  

05.09.13  

7  

Θ  ,  O,  and  Ω  

Figure 2.1 Graphic examples of the Θ , O, and Ω notations. In each part, the value of n0 shown is the minimum possible value; any greater value would also work.

Dominant  terms  only…  

•  EssenHally,  we  are  interested  in  the  largest  (dominant)  term  only…  

•  When  this  grows  large  enough,  it  will  “overshadow”  all  smaller  terms  

05.09.13  

8  

Theorem  1.2  

).()( Thes, .for ,)(

have we,1 ,let Therefore,

.1for ,

)(

:Proof).()( then ,...)( If

0

00

0

0

00

01

mm

m

ii

m

ii

m

m

i

mii

m

m

i

ii

m

i

ii

mmm

nOnfnncnnf

nac

nan

nan

nananf

nOnfanananf

=≥≤

==

≥≤

≤

≤=

=+++=

∑

∑

∑

∑∑

=

=

=

−

==

AsymptoHc  Analysis  •  Given  any  two  funcHons  f(n)  and  g(n),  we  will  restrict  

ourselves  to:  –  polynomials  with  posiHve  leading  coefficient  –  exponenHal  and  logarithmic  funcHons  

•  These  funcHons                                            as  •  We  will  consider  the  limit  of  the  raHo:  

)g()f(limnn

n ∞→

∞→n∞→

AsymptoHc  Analysis  •  If  the  two  funcHon  f(n)  and  g(n)  describe  the  run  Hmes  of  two  

algorithms,  and  

     that  is,  the  limit  is  a  constant,  then  we  can  always  run  the  slower  algorithm  on  a  faster  computer  to  get  similar  results  

∞<<∞→ )g(

)f(lim0nn

n

AsymptoHc  Analysis  •  To  formally  describe  equivalent  run  Hmes,  we  will  say  that    

f(n) = Θ(g(n))  if  

•  Note:    this  is  not  equality  –  it  would  have  been  be{er  if  it  said  f(n) ∈ Θ(g(n))  however,  someone  picked  =

∞<<∞→ )g(

)f(lim0nn

n

AsymptoHc  Analysis  •  We  are  also  interested  if  one  algorithm  runs  either  

asymptoHcally  slower  or  faster  than  another  

•  If  this  is  true,  we  will  say  that  f(n) = O(g(n))

∞<∞→ )g(

)f(limnn

n

AsymptoHc  Analysis  •  If  the  limit  is  zero,  i.e.,  

 then  we  will  say  that  f(n) = o(g(n)) t  •  This  is  the  case  if  f(n)  and  g(n)  are  polynomials  where  f  has  a  

lower  degree  

0)g()f(lim =

∞→ nn

n

05.09.13  

9  

AsymptoHc  Analysis  

•  To  summarize:  

0)g()f(lim >

∞→ nn

n

))(g()f( nn O=

))(g()f( nn Θ=

))(g()f( nn Ω=

∞<∞→ )g(

)f(limnn

n

∞<<∞→ )g(

)f(lim0nn

n

AsymptoHc  Analysis  •  We  have  one  final  case:  

∞=∞→ )g(

)f(limnn

n

))(g()f( nn o=

))(g()f( nn Θ=

))(g()f( nn ω=

0)g()f(lim =

∞→ nn

n

∞<<∞→ )g(

)f(lim0nn

n

AsymptoHc  Analysis  

•  Graphically,  we  can  summarize  these  as  follows:  

       We  say                if  

AsymptoHc  Analysis  

•  All  of   n2 100000 n2 – 4 n + 19 n2 + 1000000 323 n2 – 4 n ln(n) + 43 n + 10 42n2 + 32 n2 + 61 n ln2(n) + 7n + 14 ln3(n) + ln(n)  are  big-­‐Θ  of  each  other  

 •  E.g.,  42n2 + 32 = Θ( 323 n2 – 4 n ln(n) + 43 n + 10 )

AsymptoHc  Analysis  •  We  will  focus  on  these        Θ(1) constant        Θ(ln(n)) logarithmic        Θ(n) linear        Θ(n ln(n)) “n–log–n”        Θ(n2) quadraHc        Θ(n3) cubic        2n, en, 4n, ... exponenHal  

O(1)<O(logn)<O(n)<O(nlogn)<O(n2)<O(n3)<O(2n)<O(n!)

Growth  of  funcHons  

•  See  Chapter  “Growth  of  FuncHons”  (CLRS)  

•  h{p://net.pku.edu.cn/~course/cs101/resource/Intro2Algorithm/book6/chap02.htm      

05.09.13  

10  

Logarithms  

•  Algebra  cheat  sheet:  h{p://tutorial.math.lamar.edu/pdf/Algebra_Cheat_Sheet.pdf   •  h{p://en.wikipedia.org/wiki/Logarithm  

05.09.13  

11  

Change  of  base  a  →  b  

xb

x aa

b loglog1log =

)(loglog xx ab Θ=

Big-­‐Oh  notaHon  classes  

Class   Informal   Intui5on   Analogy  

f(n)  ∈  ο  (  g(n)  )     f  is  dominated  by  g   Strictly  below   <  f(n)  ∈  O(  g(n)  )     Bounded  from  above   Upper  bound   ≤  f(n)  ∈  Θ(  g(n)  )     Bounded  from    

above  and  below  “equal  to”   =  

f(n)  ∈  Ω(  g(n)  )     Bounded  from  below   Lower  bound   ≥  f(n)  ∈  ω(  g(n)  )      

f  dominates  g   Strictly  above   >  

Family  of  Bachmann–Landau  nota5ons  

!

Notation Name Intuition As , eventually... Definition

Big Omicron; Big O; Big Oh

f is bounded above by g(up to constant factor) asymptotically

for some k

or

(Note that, since the beginning of the 20th century, papers in number theory have been increasingly and widely using this notation in the weaker sense that f = o(g) is false)

Big Omega

f is bounded below by g(up to constant factor) asymptotically

for some positivek

Big Theta f is bounded both above and below bygasymptotically for some positive k1, k2

Small Omicron; Small O; Small Oh

f is dominated by gasymptotically for every !

Small Omega f dominates gasymptotically for every k

on the order of; "twiddles"

f is equal to gasymptotically

05.09.13  

12  

How  much  Hme  does  sorHng  take?  

•  Comparison-­‐based  sort:    A[i]  <=  A[j]    – Upper  bound  –  current  best-­‐known  algorithm  – Lower  bound  –  theoreHcal  “at  least”  esHmate    –  If  they  are  equal,  we  have  theoreHcally  opHmal  soluHon  

Lihtne  sorteerimine  

     for  i=2..n  

 for  j=i  ;    j>1  ;  j-­‐-­‐          if  A[j]  <  A[j-­‐1]      then    swap(  A[j],  A[j-­‐1]    )      else  next  i    

   

The  divide-­‐and-­‐conquer  design  paradigm  

1.  Divide  the  problem  (instance)  into  subproblems.  

2.  Conquer  the  subproblems  by  solving  them  recursively.  

3.  Combine  subproblem  solu>ons.  

Merge  sort  

 Merge-­‐Sort(A,p,r)    if  p<r    then    q  =  (p+r)/2      

   Merge-­‐Sort(  A,  p,  q  )        Merge-­‐Sort(  A,  q+1,r)      Merge(  A,  p,  q,  r  )  

 

It was invented by John von Neumann in 1945.

Example  

•  Applying  the  merge  sort  algorithm:  

Wikipedia  /  viz.  

Valu

e

Pos in array

05.09.13  

13  

Quicksort  

Wikipedia  /  “video”  

Kui  palju  võtab  sorteerimine  aega?  

•  Võrdlustel  põhinev:    A[i]  <=  A[j]    – Ülempiir  –  praegu  parim  teadaolev  algoritm  – Kas  saame  hinnata  alampiiri?    

•  Aga  kui  palju  kahendotsimine?    •  Aga  kuidas  lisada  elemente  tabelisse?  •  (kahend-­‐otsimis)  Puu  ?    

n2.3727

n2.376

05.09.13  

14  

Conclusions  

•  Algorithm  complexity  deals  with  the  behavior  in  the  long-­‐term  – worst  case      -­‐-­‐  typical  – average  case      -­‐-­‐  quite  hard  – best  case      -­‐-­‐  bogus,  “chea-ng”  

•  In  pracHce,  long-­‐term  someHmes  not  necessary  – E.g.  for  sorHng  20  elements,  you  don’t  need  fancy  algorithms…