recursion and dynamic programming - statistical geneticscsg.sph.umich.edu › abecasis › class ›...
TRANSCRIPT
![Page 1: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/1.jpg)
Recursion and Dynamic Programming
Biostatistics 615/815Lecture 4Lecture 4
![Page 2: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/2.jpg)
Last Lecture
Principles for analysis of algorithms• Empirical Analysis• Th i l A l i• Theoretical Analysis
C l i hi b iCommon relationships between inputs and running time
Described two simple search algorithms
![Page 3: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/3.jpg)
Recursive refers to …
A function that is part of its own definition
e.g.⎪
⎪⎨
⎧ >−⋅
=
0 N if)1()(
NFactorialNNFactorial
⎪⎩ = 0 N if1
A program that calls itself
![Page 4: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/4.jpg)
Key Applications of Recursion
Dynamic Programming• Related to Markov processes in Statistics
Divide-and-Conquer AlgorithmsDivide and Conquer Algorithms
Tree ProcessingTree Processing
![Page 5: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/5.jpg)
Recursive Function in C
int factorial (int N){if (N == 0)return 1;return 1;
else* f i l( 1)return N * factorial(N - 1);
}
![Page 6: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/6.jpg)
Key Features of Recursions
Simple solution for a few cases
Recursive definition for other values• Computation of large N depends on smaller NComputation of large N depends on smaller N
Can be naturally expressed in a functionCan be naturally expressed in a function that calls itself• Loops are sometimes an alternativeLoops are sometimes an alternative
![Page 7: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/7.jpg)
A Typical Recursion:A Typical Recursion:Euclid’s Algorithm
Algorithm for finding greatest common divisor of two integers a and b• If a divides b
• GCD(a,b) is a
• Otherwise, find the largest integer t such that• at + r = bat + r = b• GCD(a,b) = GCD(r,a)
![Page 8: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/8.jpg)
Euclid’s Algorithm in C
int gcd (int a, int b){if (a == 0)return b;return b;
d(b % )return gcd(b % a, a);}
![Page 9: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/9.jpg)
Evaluating GCD(4458, 2099)
gcd(2099, 4458)gcd(350, 2099)g
gcd(349, 350)gcd(1 349)gcd(1, 349)
gcd(0, 1)
![Page 10: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/10.jpg)
Divide-And-Conquer Algorithms
Common class of recursive functions
Common feature• Process input• Di id i t i ll ti• Divide input in smaller portions• Recursive call(s) process at least one portion
Recursion may sometimes occur before input is processed
![Page 11: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/11.jpg)
Recursive Binary Searchint search(int a[], int value, int start, int stop)( [], , , p)
{// Search failedif (start > stop)
return -1;;
// Find midpointint mid = (start + stop) / 2;
// Compare midpoint to valueif (value == a[mid]) return mid;
// Reduce input in half!!!pif (value < a[mid])
return search(a, value, start, mid – 1);else
return search(a, value, mid + 1, stop);p}
![Page 12: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/12.jpg)
Recursive Maximumint Maximum(int a[], int start, int stop)
{int left, right;
// Maximum of one element//if (start == stop)
return a[start];
// Process half of the data …l ft M i ( t t ( t t + t ) / 2)left = Maximum(a, start, (start + stop) / 2);right = Maximum(a, (start + stop) / 2 + 1, stop);
// Combine results across halves …if (left > right)
return left;else
return right;}
![Page 13: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/13.jpg)
An inefficient recursion
Consider the Fibonacci numbers…
⎧ = 0Nif0
⎪⎪⎪
⎨
⎧
=
=
= 1Nif1
0Nif0
)(NFibonacci
⎪⎪⎪
⎩
⎨
−+− )2()1(
)(
NFibonacciNFibonacci
![Page 14: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/14.jpg)
Fibonacci Numbersint Fibonacci(int i)
{// Simple cases firstif (i == 0)
return 0;
if (i == 1)return 1;
return Fibonacci(i – 1) + Fibonacci(i – 2);}
![Page 15: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/15.jpg)
Terribly Slow!
60
Time (seconds)Calculating Fibonacci Numbers Recursively
30
40
50
0
10
20
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
Time
0
Fibonacci Number
![Page 16: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/16.jpg)
What is going on? …
![Page 17: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/17.jpg)
Faster Alternatives
Certain quantities are recalculatedq• Far too many times!
Need to avoid recalculation…• Ideally, calculate each unique quantity once.Ideally, calculate each unique quantity once.
![Page 18: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/18.jpg)
Bottom-Up Dynamic Programming
Evaluate function starting with smallest possible argument value• St i th h ibl l d ll i• Stepping through possible values, gradually increase
argument value
Store all computed values in an array
As larger arguments evaluated, precomputed values for smaller arguments can be retrieved
![Page 19: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/19.jpg)
Fibonacci Numbers in Cint Fibonacci(int i)
{int fib[LARGE_NUMBER], j;
fib[0] = 0;fib[1] = 1;
for (j = 2; j <= i; j++)fib[j] = fib[j – 1] + fib[j – 2];
return fib[i];}
![Page 20: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/20.jpg)
Fibonacci With Dynamic Memoryint Fibonacci(int i)
{int * fib, j, result;
if (i < 2) return i;
fib = malloc(sizeof(int) * (i + 1));
i [0] 0 i [ ]fib[0] = 0; fib[1] = 1;for (j = 2; j <= i; j++)
fib[j] = fib[j – 1] + fib[j – 2];
l fib[i]result = fib[i];free(fib);
return result;}}
![Page 21: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/21.jpg)
Top-Down Dynamic Programming
Save each computed value as final paction of recursive function
Check if pre-computed value exists as the first actionthe first action
![Page 22: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/22.jpg)
Fibonacci Numbers//// Note: saveF should be a global array initialized all zeros
int Fibonacci(int i){//// Simple cases firstif (saveF[i] > 0)
return saveF[i];
if (i )if (i <= 1)return i;
// Recursion[i] ib i(i 1) ib i(i 2)saveF[i] = Fibonacci(i – 1) + Fibonacci(i – 2);
return saveF[i];}
![Page 23: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/23.jpg)
Much less recursion now…
![Page 24: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/24.jpg)
Dynamic ProgrammingDynamic ProgrammingTop-down vs. Bottom-up
In bottom-up programming, programmer has to do the thinking by selecting values to calculate and order of calculation
In top-down programming, recursive structure of original code is preserved, but g p ,unnecessary recalculation is avoided.
![Page 25: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/25.jpg)
Examples of Useful Settings for Examples of Useful Settings for Dynamic Programming
Calculating Binomial Coefficientsg
Evaluating Poisson-Binomial DistributionEvaluating Poisson-Binomial Distribution
![Page 26: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/26.jpg)
Binomial Coefficients
The number of subsets with k elements from a set of size N
⎟⎟⎞
⎜⎜⎛ −
+⎟⎟⎞
⎜⎜⎛ −
=⎟⎟⎞
⎜⎜⎛ 11 NNN
⎟⎟⎠
⎜⎜⎝ −⎟⎟
⎠⎜⎜⎝
⎟⎟⎠
⎜⎜⎝ 1kkk
1⎟⎞
⎜⎛
⎟⎞
⎜⎛ NN
10
=⎟⎟⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛N
![Page 27: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/27.jpg)
Bottom-Up Implementation in Cint Choose(int N, int k)
{int i, j, M[MAX_N][MAX_N];
for (i = 1; i <= N; i++){M[i][0] = M[i][i] = 1;
for (j = 1; j < i; j++)M[i][j] = M[i - 1][j - 1] + M[i - 1][j];
}
return M[N][k];}
![Page 28: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/28.jpg)
Top-Down Implementation (I)##define MAX_N 30
int choices[MAX_N][MAX_N];
void InitChoose(){int i j;int i, j;
for (i = 0; i < MAX_N; i++)for (j = 0; j < MAX N; j++) (j ; j _ ; j )
choices[i][j] = 0;}
![Page 29: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/29.jpg)
Top-Down Implementation (II)int Choose(int N, int k)
{// Check if this is an easy caseif (N == k || k == 0)
return 1;
// Or a previously examined caseif (choices[N][k] > 0)
i [ ][ ]return choices[N][k];
// If neither of the above helps, use recursionchoices[N][k] = Choose(N – 1, k - 1) + Choose(N – 1, k);
return choices[N][k];}
![Page 30: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/30.jpg)
Poisson-Binomial Distribution
X1, X2, …, Xn are Bernoulli random variables
Probability of success is pk for XkProbability of success is pk for Xk
∑ X has Poisson Binomial Distribution∑kXk has Poisson-Binomial Distribution
![Page 31: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/31.jpg)
Recursive Formulation
)1(1)0(
11
11
pPpP
=−=
)1( 11 pP
)1()(
)0()1()0( 1
jPpjP
PpP jjj −
−=
−=
)()1()1()(
)1()(
11
1
iPpiPpiP
jPpjP
jjjjj
jjj
−−
−
−+−=
=
![Page 32: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/32.jpg)
Summary
Recursive functions• Arise very often in statistics
Dynamic programming• Bottom-up Dynamic ProgrammingBottom-up Dynamic Programming• Top-down Dynamic Programming
Dynamic program is an essential tool for statistical programming
![Page 33: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/33.jpg)
Good Programming Practices
Today’s examples used global variables• Variables declared outside a function• A ibl th h t th• Accessible throughout the program
In general these should be used sparinglyIn general, these should be used sparingly
Two alternatives are:Two alternatives are:• Using static variables that are setup on the first call• Using C++ to group a function and its data
![Page 34: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/34.jpg)
Function with Built-In InitializationFunction with Built-In Initializationint Choose(int N, int k)
{static int valid = 0, choices[MAX N][MAX N], i, j;static int valid 0, choices[MAX_N][MAX_N], i, j;
// Check if we need to initialize dataif (valid == 0)
{f (i 0 i < MAX N i++)for (i = 0; i < MAX_N; i++)
for (j = 0; j < MAX_N; j++)choices[i][j] = 0;
valid = 1; }}
// Check if this is an easy caseif (N == k || k == 0) return 1;
// Or a previously examined case// Or a previously examined caseif (choices[N][k] > 0) return choices[N][k];
// If neither of the above helps, use recursionchoices[N][k] = Choose(N – 1, k - 1) + Choose(N – 1, k);
return choices[N][k];}
![Page 35: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/35.jpg)
C++ Declarations (for .h file)class Choose
{public:
// This function is called to initialize data//// Whenever a variable of type Choose is created Choose();
// This function is called to calculate N_choose_k i (i i )int Evaluate(int N, int k);
private:int choices[MAX_N][MAX_N];
}};
![Page 36: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/36.jpg)
C++ Code (for .cpp file)Choose::Choose()Choose::Choose()
{for (int i = 0; i < MAX_N; i++)
for (int j = 0; j < MAX_N; j++)choices[i][j] = 0;
}
int Choose::Evaluate(){// Check if this is an easy case// Check if this is an easy caseif (N == k || k == 0) return 1;
// Or a previously examined caseif (choices[N][k] > 0) return choices[N][k];
// If neither of the above helps, use recursionchoices[N][k] = Choose(N – 1, k - 1) + Choose(N – 1, k);
return choices[N][k];}
![Page 37: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/37.jpg)
Using C++ Code##include “Choose.h”
int main(){{Choose choices;
// Evaluate 20_choose_10h i E l t (20 10)choices.Evaluate(20, 10);
// Evaluate 30_choose_10choices.Evaluate(30, 10);
return 0;}
![Page 38: Recursion and Dynamic Programming - Statistical geneticscsg.sph.umich.edu › abecasis › class › 2008 › 615.04.pdf · 2008-09-23 · Key Applications of Recursion zDynamic Programming](https://reader036.vdocument.in/reader036/viewer/2022063000/5f1004357e708231d44706f5/html5/thumbnails/38.jpg)
Reading
Sedgewick, Chapters 5.1 – 5.3