dynamic programming- mirage group

8/6/2019 Dynamic Programming- Mirage Group

1/37

DYNAMIC PROGRAMMINGDYNAMIC PROGRAMMING

SUBMITTED BY:DR. DEBANJANA GHOSHDR. PRABHPREET KAUR DR. PRASHANTH KULALADR. SANDIP KR NEOGI

SHWETNISHA BOSE


2/37

DYNAMIC PROGRAMMING

Technique that divides the problem to be solved into a numberof sub problems and then solves each sub-problem in such away that the overall solution is optimal to the originalproblem.

Invented by American mathematician Richard Bellman in the1950s to solve optimization problems

Programming here means planning

Dynamic programming is a stage-wise search method suitablefor optimization problems whose solutions may be viewed asthe result of a sequence of decisions.


3/37

MAIN IDEA

set up a recurrence relating a solution to a largerinstance to solutions of some smaller instancessolve smaller instances oncerecord solutions in a table

extract solution to the initial instance from thattable


4/37

Example: Fibonacci numbers using recursion

Fibonacci numbers: F (n ) = F (n -1) + F (n -2) F (0) = 0 F (1) = 1

Computing the n th Fibonacci number recursively (top -d own ):

F (n )

F (n-

1) + F

(n-

2)

F (n- 2) + F (n- 3) F (n- 3) + F (n- 4)... Bad algorithm as

numerous repetitive

calculations are there


5/37

Example Fibonacci Numbersusing Dynamic programming

f (5)

f (4) f (3)

f (3) f (2) f (2) f (1)

f (2) f (1)

C alculation from bottom to top and Values are stored for later use

This reduces repetitive calculation


6/37

CHARACTERISTICS OFDYNAMIC PROGRAMMING

The problem can be divided into stages with a decisionrequired at each stage.In the capital budgeting problem the stages were the allocations toa single plant. The decision was how much to spend. In the shortest

path problem, they were defined by the structure of the graph. Thedecision was were to go next.Each stage has a number of states associated with it.The states for the capital budgeting problem corresponded to theamount spent at that point in time. The states for the shortest path

problem was the node reached.The decision at one stage transforms one state into a state inthe next stage.The decision of how much to spend gave a total amount spent for the next stage. The decision of where to go next defined where youarrived in the next stage.


7/37

Given the current state, the optimal decision for each of theremaining states does not depend on the previous states or

decisions .In the budgeting problem, it is not necessary to know how themoney was spent in previous stages, only how much was spent.In the path problem, it was not necessary to know how you got to

a node, only that you did.There exists a recursive relationship that identifies theoptimal decision for stage j , given that stage j +1 has alreadybeen solved.The final stage must be solvable by itself.

The last two properties are tied up in the recursiverelationships given above.


8/37

STEPS OF DYNAMIC PROGRAMMING

IDENTIFY DECISIONVARIABLES

SPECIFY OBJECTIVE FUNCTION

DECOMPOSE INTOSMALLER PROBLEMS ( STAGE )


9/37

IDENTIFYSTATE VARIABLES

STATE THE GENERALRECURSIVE RELATIONS. ALSO DECIDE FORW ARD/BACKWARD

CALCULATE REQ UIRED VALUES OFRETURN FUNCTION BY APPR OPRIATE

FORMULAE AT EACH STAGE

REPEAT AND DETERMINE THE

OVERALL OPTIMALPOLICY


10/37

THE

STAGECOACHPROBLEM


11/37

A MINIMUM PATH PROBLEM

Given a series of paths from point A to point BA and B are not directly connectedEach path has a value linked to it

Find the best route from A to B

The idea for this problem is that a salesman is traveling from

one town to another town, in the old west. His means of

travel is a stagecoach. Each leg of his trip cost a certainamount and he wants to find the minimum cost of his trip,given multiple paths.


12/37

A SAMPLE STAGECOACH

PROBLEM


13/37

W e begin by dividing the problem into stages


14/37

Suppose we are at node i, we want to find the lowestcost route from i to 10

Start at node 10, and work backwards through thenetwork.

Define variables such that:

cost of travel from node i to node j

node chosen for stage n = 1; 2; 3; 4state


15/37

START AT STAGE 1 (THE LAST STAGE),THEN

STATE DEC ISION10

BESTDEC ISION

BESTDISTAN C E

8 2+0 10 2

9 4+0 10 4


16/37

AT STAGE 2

STATE DEC ISION

8

DEC ISION

9

BESTDEC ISION

BESTDISTAN C E

5 1+2=3 4+4=8 8 3

6 6+2=8 3+4=7 9 7

7 3+2=5 3+4=7 8 5


17/37

APPLICATIONS

This problem can be used inC

omputer Networks

Plane travel


18/37

BELLMANS PRINCIPLE OF OPTIMALITY

An optimal policy has the property that whatever the initial state and initial decisionsare, the remaining decisions must constitutean optimal policy with regard to the stateresulting from the first decision


19/37

OPTIMALITY PRINCIPLE

Assume that optimal path is the shortest oneOP indicates that any portion of any optimal path isalso optimal

Set of optimal paths from all sources to a givendestination forms a tree that is routed at the

destination

A B C

D E F

2 3

2 1 2

4 1

Optimal path between A and Fis A-B-C-E-F

Then, the optimal path betweenB and F is B-C-E-F


20/37

PRINCIPLE OF OPTIMALITY

ac

Assertion : If abe is the optimal path from a to e, then be is

the optimal path from b to e.Proof: Suppose it is not. Then there is another path(note that existence is assumed here) bce whichis optimal from b to e, i.e.

J bce > J be but then J abe = J ab+ J be< J ab+ J bce= J abceThis contradicts the hypothesis that abe is theoptimal path from a to e.

JabJbe

eJbceb


21/37

PRINCIPLE OF OPTIMALITY

1

2

3

4

5

6

7

8

9

Bellmans Principle of Optimality : At any intermediate state x i in an optimal path from x 0 to x f , the policy from x ito goal x f must itself constitute optimal policy


22/37

BELLMANS PROGRAMMING

Transforms a complex optimization problem into a

sequence of simpler ones.

Usually begins at the end and works backwards

C an handle a wide range of problems

Relies on recursion, and on the principle of optimality

Developed by Richard Bellman


23/37

MARKOVIAN REQUIREMENT

An optimal policy starting in a given state depends

only on the state of the process at that stage and not

on the state at preceding stages. The path is of no

consequence, only the present stage and state

The state variables fully describe the state of thesystem at each stage and capture all past decisions


24/37

Optimal Capacity Expansion: W hat isthe least cost way of building plants?

24

C ost of $15 million in any year in which a plant is built.At most 3 plants a year can be built


25/37

Forwards recursion

Start with stage 1 and determine values for larger stages using recursion.In forward recursion f(n) is determined from f(k)where k < n.Determine the value of the other stages using thevalue of the first stage with the help of forwardrecursion.

25


26/37

Backwards Recursion

Boundary conditions: optimum values for states at thelast stage.In backward recursion value of f(k) is determinedfrom f(n) where k


27/37

Determining state and stage

Stage : year Y

State : number of plants

f(j, Y) = optimum cost of capacity expansion starting

with j plants at the end of year Y.

We want to compute f(0, 2002).

We will first compute what is f(j, 2007) for different j.

27


28/37

Determining state and stage

From f(j,2007) for each j we will compute f(j, 2006)

for each j.

After that we will complete f(j, 2005) for each j.

Then compute f(j, 2004) for each j using f(j,2005).

From f(j,2004) f(j,2003) for each j will be computed.

Lastly from f(j,2003) f(j,2002)can be calculated.

28


29/37

TRAVELING SALESMAN

Start 1

East

2

3

4

5

6

7

8

9

End 10

West


30/37

INSURANCE POLICY COST

T O

2 3 4 5 6 7 8 9 10

F 1 2 4 3 2 7 4 6 5 1 4 8 3

R 3 3 2 4 6 6 3 9 5

O 4 4 1 5 7 3 3

M


31/37

Minimize the cost for each run

1 2 6 9 10Total cost = 14

However,1 4 6 9 10

Total cost = 12 !


32/37

CONDITIONS

Those that can satisfy Bellmans principle of optimality:

The globally optimum solution includes no suboptimal local

decision.

or in Bellmans original words from 1957:

An optimal policy has the property that, regardless of the

decisions taken to enter a particular state, the remaining decisions

made for leaving that stage must constitute an optimal policy.


33/37

APPLICATIONS

In numerous multistage decision problems of optimum fitsuch as:

Error correction coding (Viterbi algorithm).Automatic labelling of speech segments (dynamic time warping).Video coding.Digital watermark detection.C ell library binding (as part of logic optimization).Flight trajectory planning.Genome sequencing (Smith-Waterman and Needleman-Wunschalgorithms).Knapsack problems.Stereo vision.


34/37

Dynamic Programming v/s LinearProgramming

Dynamic Programming is different from Linear Programmingon two counts.

1). There does not exist a standard mathematical formulation of Dynamic Programming problem. There is no algorithm likeSimplex method , that may be programmed to solve all the

problems. Instead, Dynamic Programming is a technique whichdissects difficult problems into a sequence of sub- problems,which are then evaluated by stages.

2). While Linear Programming is a method which gives singlestage, i.e. one time period solutions. But Dynamic Programmingahs the power to determine the optimal solution e.g. one year time horizon divided into 12 months time horizon. Thus , it is amultistage approach to solve a problem.


35/37

Solving linear programming by dynamic programming

Eg: Maximize Z= c 1x1+c2x2+ +c nxn

Su bject to the constraints

a ixi+a ix+ ..+ax=0 where j=1,2 n

Consider this LPP as m u ltistage problem with eachactivity j (j = 1,2, ..n) as individ u al stage so this is an

n- stage problem


36/37

SUMMARYThe Dynamic Programming technique is useful for making a

sequence of inter-related decisions where the objective is tooptimize the overall outcome of the entire sequence of decisionsover a period of time.

This technique decomposes the original problem into asequence of stages, which can then be handled more efficientlyfrom the computational point of view.

In contrast, most linear and nonlinear programming approaches

attempt to solve such problems by considering all the constraintssimultaneously.

The dynamic programming technique is used in various fieldssuch as inventory control, allocation, Bidding problems etc.


37/37

dynamic programming- mirage group

Documents