Upload File

reinforcement learning for supply chain optimization · reinforcement learning for supply chain...

  • Home
  • Documents
  • Reinforcement learning for supply chain optimization · Reinforcement learning for supply chain optimization European Workshop on Reinforcement Learning 14 (2018) October 2018, Lille,
1
L. Kemmer, H. von Kleist, D. de Rochebouët, N. Tziortziotis, J. Read LIX, École Polytechnique, France Reinforcement learning for supply chain optimization European Workshop on Reinforcement Learning 14 (2018) October 2018, Lille, France References [1] Warren B. Powell. Approximate dynamic programming : solving the curses of dimensionality. Wiley series in probability and statistics. Wiley-Interscience, Hoboken, NJ,2007. ISBN 978-0-470-17155-4. [2] Lars Norman Moritz. Target Value Criterion in Markov Decision Processes. PhD thesis, 2014. [3] Gavin A Rummery and Mahesan Niranjan. On-line q-learning using connectionist systems. Technical report, Cambridge University, 1994. [4] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229{256, 1992. [5] Horst Tempelmeier. Inventory management in supply networks : problems, models, solutions. Books on Demand, Norderstedt, 2. ed. edition, 2011. ISBN 978-3-8423-4677- 2. Results § Approximate SARSA [3] and REINFORCE [4] agents are compared to the baseline (", $) agent [5] that replenishes each warehouse j by an amount $ & if the current stock is below ' & and there is still stock left in the factory § Scenario 1: 1 warehouse and 1 factory with costs for production, storage (only warehouse), transportation and penalty costs § Scenario 2: 3 warehouses and 1 factory with costs for production, storage (only for warehouse no. 1), transportation and penalty costs § The (s,Q)-policy is outperformed by REINFORCE and approximate SARSA in scenario 1, and by REINFORCE in scenario 2 § RL agents learn to invest in stock and transportation, despite short-term negative rewards Approximate SARSA § Use a linear Q-function approximation $ ( ), * =, - . (), *) with a parameter-vector , and a feature map . (), *) to deal with exponentially growing state and action spaces § Reasonable knowledge about the structure of the MDP is assumed and used to construct over 15 different features to design . (), *) § In order to achieve reasonable knowledge about the dynamics of the demand process past demands are used to create a simple forecast of future demands REINFORCE § Discretize action space: 3 actions per location, total of 3 012 actions § Maximize policy function: 3 *=* (4) |) = 6 7 8 9 : ; < => ? < |@ ∑ BCD E; 6 7 8 9 : ; B => ? B |@ where F * & |) =G 1, if * & is allowed in state ), 0, otherwise. § Different options for feature map . ()): Linear, Quadratic and RBF Fig. 2: Scenario 1 (1 warehouse). Fig. 3: Scenario 2 (3 warehouses). Fig. 5: Stocks for the (", $)-Policy based agent. Fig. 4: Stocks for the REINFORCE agent using a quadratic feature map .. Motivation § Optimizing supply chains consisting of a factory and multiple warehouses is a problem faced by many companies § Main decision: production quantities at the factory and quantities shipped to the warehouses § Small firms can manage supply chains manually but big companies need more elaborate tools. This motivates our adaptation of reinforcement learning (RL) agents that we test in 2 scenarios against a heuristic baseline algorithm The Model § The problem is modeled by a Markov decision process [1,2] where t denotes the current period and j a specific warehouse § It is defined by a state-space s t that represents stock levels at the factory and warehouses, the demand d t at each warehouse, an action space a t that sets the production level and transportation quantities to the warehouses, a set of feasible actions (it is not possible to ship more that what is in stock), a transition function X(Y Z ,[ Z , \),a one-step reward function ] ^ and a discount factor _ § The demand ` &,^ = a b;c d sin de ^1d& 2d + a b;c d +g &,^ (with h g &,^ =0 =h g &,^ =1 =0,5) incorporates a seasonal trend § The one step-reward function ] ^ consists of revenue from sold products, production costs, storage costs, penalty costs for unsatisfied demands and transportation costs from the factory to the warehouses Problem Setting Fig. 1: Supply chain network with 5 warehouses and 1 factory

Upload: others

Post on 19-May-2020

19 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Reinforcement learning for supply chain optimization · Reinforcement learning for supply chain optimization European Workshop on Reinforcement Learning 14 (2018) October 2018, Lille,

L. Kemmer, H. von Kleist, D. de Rochebouët, N. Tziortziotis, J. ReadLIX, École Polytechnique, France

Reinforcement learning for supply chain optimization

European Workshop on Reinforcement Learning 14 (2018) October 2018, Lille, France

References[1] Warren B. Powell. Approximate dynamic programming : solving the curses of dimensionality. Wiley series in probability and statistics. Wiley-Interscience, Hoboken, NJ,2007. ISBN 978-0-470-17155-4.

[2] Lars Norman Moritz. Target Value Criterion in Markov Decision Processes. PhD thesis, 2014.

[3] Gavin A Rummery and Mahesan Niranjan. On-line q-learning using connectionist systems. Technical report, Cambridge University, 1994.

[4] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229{256, 1992.

[5] Horst Tempelmeier. Inventory management in supply networks : problems, models, solutions. Books on Demand, Norderstedt, 2. ed. edition, 2011. ISBN 978-3-8423-4677- 2.

Results

§ Approximate SARSA [3] and REINFORCE [4] agents are compared to the baseline (", $) agent [5] that replenishes each warehouse j by an amount $& if the

current stock is below '& and there is still stock left in the factory

§ Scenario 1: 1 warehouse and 1 factory with costs for production, storage (only warehouse), transportation and penalty costs

§ Scenario 2: 3 warehouses and 1 factory with costs for production, storage (only for warehouse no. 1), transportation and penalty costs

§ The (s,Q)-policy is outperformed by REINFORCE and approximate SARSA in scenario 1, and by REINFORCE in scenario 2

§ RL agents learn to invest in stock and transportation, despite short-term negative rewards

Approximate SARSA

§ Use a linear Q-function approximation $( ), * = ,-. (), *) with aparameter-vector , and a feature map . (), *) to deal withexponentially growing state and action spaces

§ Reasonable knowledge about the structure of the MDP is assumedand used to construct over 15 different features to design . (), *)

§ In order to achieve reasonable knowledge about the dynamics of thedemand process past demands are used to create a simple forecastof future demands

REINFORCE

§ Discretize action space: 3 actions per location, total of 3 012 actions

§ Maximize policy function: 3 * = *(4)|) =67 8 9:

; < => ? < |@

∑BCDE; 6

7 8 9:; B => ? B |@

where F * & |) = G1, if * & is allowed in state ),0, otherwise.

§ Different options for feature map . ()): Linear, Quadratic and RBF

Fig. 2: Scenario 1 (1 warehouse). Fig. 3: Scenario 2 (3 warehouses).

Fig. 5: Stocks for the (", $)-Policy based agent. Fig. 4: Stocks for the REINFORCE agent using a quadratic feature map ..

Motivation§ Optimizing supply chains consisting of a factory and multiple warehouses is a problem faced by many companies

§ Main decision: production quantities at the factory and quantities shipped to the warehouses

§ Small firms can manage supply chains manually but big companies need more elaborate tools. This motivates our adaptation of

reinforcement learning (RL) agents that we test in 2 scenarios against a heuristic baseline algorithm

The Model§ The problem is modeled by a Markov decision process [1,2] where t denotes the current period and j a specific warehouse

§ It is defined by a state-space st that represents stock levels at the factory and warehouses, the demand dt at each warehouse, an

action space at that sets the production level and transportation quantities to the warehouses, a set of feasible actions (it is not

possible to ship more that what is in stock), a transition function X(YZ, [Z, \), a one-step reward function ]̂ and a discount factor _§ The demand &̀,^ =

ab;c

dsin

de ^1d&

2d+

ab;c

d+ g&,^ (with h g&,^ = 0 = h g&,^ = 1 =0,5) incorporates a seasonal trend

§ The one step-reward function ]̂ consists of revenue from sold products, production costs, storage costs, penalty costs for

unsatisfied demands and transportation costs from the factory to the warehouses

Problem Setting

Fig. 1: Supply chain

network with 5

warehouses and 1

factory

A Reinforcement Learning Based Approach for Welding ...rwlab.sjtu.edu.cn › tiwm › papers › V1N2-2.pdf · A Reinforcement Learning Based Approach for Welding Sequence Optimization

Reinforcement learning based traffic optimization at an

Deep Reinforcement Learning via Policy Optimization · Deep Reinforcement Learning via Policy Optimization John Schulman July 3, ... (termination = good) ... ( xed time) I Walking

Reinforcement Learning for Optimization of COVID-19

Convergent Policy Optimization for Safe Reinforcement Learning · reinforcement learning problems with safety constraints (in supplementary material). 2 Background A Markov decision

Exploratory Combinatorial Optimization with Reinforcement

Deep Reinforcement Learning Through Policy Optimization, John Schulman, OpenAI

Reinforcement Learning - Multi-Agent Reinforcement

Reinforcement and Imitation Learning for Diverse ...ai.stanford.edu/~yukez/papers/rss2018zhu.pdf · such as trust region policy optimization (TRPO) and proximal policy optimization

Reinforcement Learning via an Optimization Lens · 2020. 1. 3. · Reinforcement Learning via an Optimization Lens Lihong Li August 7, 2019 Google Brain Simons Institute Workshop

Policy Improvement: Between Black-Box Optimization and Episodic Reinforcement Learning

Reinforcement Learning or Active Inference?karl/Reinforcement Learning or Active... · Reinforcement Learning or Active Inference? ... From the point of view of reinforcement learning

Deep Reinforcement Learning in System Optimization · arXiv:1908.01275v2 [cs.LG] 7 Aug 2019. Deep Reinforcement Learning in System Optimization Figure 1. Reinforcement learning. By

AuTO: Scaling Deep Reinforcement Learning for Datacenter ...AuTO: Scaling Deep Reinforcement Learning for Datacenter-Scale Automatic Tra˙ic Optimization Li Chen, Justinas Lingys,

Reinforcement Learning with Particle Swarm Optimization

Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

Reinforcement Learning Lecture Inverse Reinforcement Learningipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2017/07/09... · Reinforcement Learning Inverse Reinforcement

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Local Policy Optimization for Trajectory-Centric Reinforcement … · 2020. 6. 2. · Local Policy Optimization for Trajectory-Centric Reinforcement Learning Patrik Kolaric1, Devesh

Constrained Combinatorial Optimization with Reinforcement ...Constrained Combinatorial Optimization with Reinforcement Learning Ruben Solozabal Department of Communications Engineering

Multi-objective optimization perspectives on … optimization perspectives on reinforcement learning algorithms ... could be deterministic, stochastic ... and stochastic optimization

Generalization in Reinforcement Learning: Successful ...papers.nips.cc/paper/1109-generalization-in-reinforcement-learning... · Generalization in Reinforcement Learning: Successful

Causal Inference, Reinforcement Learning, and Continuous Optimization

Bayesian Reinforcement Learning - mlg.eng.cam.ac.ukmlg.eng.cam.ac.uk/rowan/files/BayesianReinforcementLearning.pdf · Introduction Bayesian Reinforcement Learning Bayesian Reinforcement

Distributed Meta Optimization of Reinforcement Learning Agents · Greg Heinrich, Iuri Frosio - GTC San Jose, March 2019 Distributed Meta Optimization of Reinforcement Learning Agents

Reinforcement Learning-Based Optimization for Drone

Multi-Objective Reinforcement Learning using Sets of Pareto … · 2020. 10. 19. · learning and multi-objective reinforcement learning. 2.1 Reinforcement Learning A reinforcement

Eick: Reinforcement Learning. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning

Multi-Agent Reinforcement Learning via Double Averaging ......Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization Hoi-To Wai The Chinese University of

Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

Reinforcement Learning Chapter 13 What is Reinforcement Learning? Q-Learning Examples 1

Building a Long-Range AI/ML Vision...Learning Generative Modeling Reinforcement Learning Neural Networks Gaussian Processes Deep Q Learning Bayesian Optimization •Howtoaddresshaving

Reservoir Operation Optimization by Reinforcement Learning · 166 Reservoir Operation Optimization by Reinforcement Learning In an optimization model of a real-world application of

Comparing Deep Reinforcement Learning Methods for ...und...Reinforcement Learning methods (Deep Q-Learning, Deep Deterministic Policy Gra-dient and Distributed Proximal Policy Optimization);