Continuous Deep Q-Learning with Model-based Acceleration

Shixiang Gu1 2 3 SG717@CAM.AC.UKTimothy Lillicrap4 COUNTZERO@GOOGLE.COMIlya Sutskever3 ILYASU@GOOGLE.COMSergey Levine3 SLEVINE@GOOGLE.COM1University of Cambridge 2Max Planck Institute for Intelligent Systems 3Google Brain 4Google DeepMind

AbstractModel-free reinforcement learning has been suc-cessfully applied to a range of challenging prob-lems, and has recently been extended to han-dle large neural network policies and value func-tions. However, the sample complexity of model-free algorithms, particularly when using high-dimensional function approximators, tends tolimit their applicability to physical systems. Inthis paper, we explore algorithms and repre-sentations to reduce the sample complexity ofdeep reinforcement learning for continuous con-trol tasks. We propose two complementary tech-niques for improving the efficiency of such algo-rithms. First, we derive a continuous variant ofthe Q-learning algorithm, which we call normal-ized adantage functions (NAF), as an alternativeto the more commonly used policy gradient andactor-critic methods. NAF representation allowsus to apply Q-learning with experience replay tocontinuous tasks, and substantially improves per-formance on a set of simulated robotic controltasks. To further improve the efficiency of ourapproach, we explore the use of learned modelsfor accelerating model-free reinforcement learn-ing. We show that iteratively refitted local lin-ear models are especially effective for this, anddemonstrate substantially faster learning on do-mains where such models are applicable.

1. IntroductionModel-free reinforcement learning (RL) has been success-fully applied to a range of challenging problems (Kober& Peters, 2012; Deisenroth et al., 2013), and has recentlybeen extended to handle large neural network policies and

value functions (Mnih et al., 2015; Lillicrap et al., 2016;Wang et al., 2015; Heess et al., 2015; Hausknecht & Stone,2015; Schulman et al., 2015). This makes it possible totrain policies for complex tasks with minimal feature andpolicy engineering, using the raw state representation di-rectly as input to the neural network. However, the samplecomplexity of model-free algorithms, particularly when us-ing very high-dimensional function approximators, tends tobe high (Schulman et al., 2015), which means that the ben-efit of reduced manual engineering and greater generalityis not felt in real-world domains where experience must becollected on real physical systems, such as robots and au-tonomous vehicles. In such domains, the methods of choicehave been efficient model-free algorithms that use moresuitable, task-specific representations (Peters et al., 2010;Deisenroth et al., 2013), as well as model-based algorithmsthat learn a model of the system with supervised learningand optimize a policy under this model (Deisenroth & Ras-mussen, 2011; Levine et al., 2015). Using task-specificrepresentations dramatically improves efficiency, but limitsthe range of tasks that can be learned and requires greaterdomain knowledge. Using model-based RL also improvesefficiency, but limits the policy to only be as good as thelearned model. For many real-world tasks, it may be easierto represent a good policy than to learn a good model. Forexample, a simple robotic grasping behavior might only re-quire closing the fingers at the right moment, while the cor-responding dynamics model requires learning the complex-ities of rigid and deformable bodies undergoing frictionalcontact. It is therefore desirable to bring the generality ofmodel-free deep reinforcement learning into real-world do-mains by reducing their sample complexity.

In this paper, we propose two complementary techniquesfor improving the efficiency of deep reinforcement learn-ing in continuous control domains: we derive a variant ofQ-learning that can be used in continuous domains, andwe propose a method for combining this continuous Q-learning algorithm with learned models so as to acceleratelearning while preserving the benefits of model-free RL.Model-free reinforcement learning in domains with contin-

arX

iv:1

603.

0074

8v1

[cs

.LG

] 2

Mar

201

6

Continuous Deep Q-Learning with Model-based Acceleration

uous actions is typically handled with policy search meth-ods (Peters & Schaal, 2006; Peters et al., 2010). Integrat-ing value function estimation into these techniques resultsin actor-critic algorithms (Hafner & Riedmiller, 2011; Lil-licrap et al., 2016; Schulman et al., 2015), which combinethe benefits of policy search and value function estimation,but at the cost of training two separate function approxi-mators. Our proposed Q-learning algorithm for continu-ous domains, which we call normalized advantage func-tions (NAF), avoids the need for a second actor or policyfunction, resulting in a simpler algorithm. The simpler op-timization objective and the choice of value function pa-rameterization result in an algorithm that is substantiallymore sample-efficient when used with large neural networkfunction approximators on a range of continuous controldomains.

Beyond deriving an improved model-free deep reinforce-ment learning algorithm, we also seek to incorporate ele-ments of model-based RL to accelerate learning, withoutgiving up the strengths of model-free methods. One ap-proach is for off-policy algorithms such as Q-learning to in-corporate off-policy experience produced by a model-basedplanner. However, while this solution is a natural one, ourempirical evaluation shows that it is ineffective at accelerat-ing learning. As we discuss in our evaluation, this is due inpart to the nature of value function estimation algorithms,which must experience both good and bad state transitionsto accurately model the value function landscape. We pro-pose an alternative approach to incorporating learned mod-els into our continuous-action Q-learning algorithm basedon imagination rollouts: on-policy samples generated un-der the learned model, analogous to the Dyna-Q method(Sutton, 1990). We show that this is extremely effectivewhen the learned dynamics model perfectly matches thetrue one, but degrades dramatically with imperfect learnedmodels. However, we demonstrate that iteratively fittinglocal linear models to the latest batch of on-policy or off-policy rollouts provides sufficient local accuracy to achievesubstantial improvement using short imagination rolloutsin the vicinity of the real-world samples.

Our paper provides three main contributions: first, we de-rive and evaluate a Q-function representation that allowsfor effective Q-learning in continuous domains. Second,we evaluate several naıve options for incorporating learnedmodels into model-free Q-learning, and we show that theyare minimally effective on our continuous control tasks.Third, we propose to combine locally linear models withlocal on-policy imagination rollouts to accelerate model-free continuous Q-learning, and show that this produces alarge improvement in sample complexity. We evaluate ourmethod on a series of simulated robotic tasks and compareto prior methods.

2. Related WorkDeep reinforcement learning has received considerable at-tention in recent years due to its potential to automate thedesign of representations in RL. Deep reinforcement learn-ing and related methods have been applied to learn policiesto play Atari games (Mnih et al., 2015; Schaul et al., 2015)and perform a wide variety of simulated and real-worldrobotic control tasks (Hafner & Riedmiller, 2011; Lillicrapet al., 2016; Levine & Koltun, 2013; de Bruin et al., 2015;Hafner & Riedmiller, 2011). While the majority of deepreinforcement learning methods in domains with discreteactions, such as Atari games, are based around value func-tion estimation and Q-learning (Mnih et al., 2015), con-tinuous domains typically require explicit representation ofthe policy, for example in the context of a policy gradientalgorithm (Schulman et al., 2015). If we wish to incorpo-rate the benefits of value function estimation into contin-uous deep reinforcement learning, we must typically usetwo networks: one to represent the policy, and one to rep-resent the value function (Schulman et al., 2015; Lillicrapet al., 2016). In this paper, we instead describe how thesimplicity and elegance of Q-learning can be ported intocontinuous domains, by learning a single network that out-puts both the value function and policy. Our Q-functionrepresentation is related to dueling networks (Wang et al.,2015), though our approach applies to continuous actiondomains. Our empirical evaluation demonstrates that ourcontinuous Q-learning algorithm achieves faster and moreeffective learning on a set of benchmark tasks comparedto continuous actor-critic methods, and we believe that thesimplicity of this approach will make it easier to adopt inpractice. Our Q-learning method is also related to the workof Rawlik et al. (2013), but the form of our Q-function up-date is more standard.

As in standard RL, model-based deep reinforcement learn-ing methods have generally been more efficient (Li &Todorov, 2004; Watter et al., 2015; Li & Todorov, 2004;Wahlstrom et al., 2015; Levine & Koltun, 2013), whilemodel-free algorithms tend to be more generally applica-ble but substantially slower (Schulman et al., 2015; Lilli-crap et al., 2016). Combining model-based and model-freelearning has been explored in several ways in the literature.The method closest to our imagination rollouts approachis Dyna-Q (Sutton, 1990), which uses simulated experi-ence in a learned model to supplement real-world on-policyrollouts. As we show in our evaluation, using Dyna-Qstyle methods to accelerate model-free RL is very effectivewhen the learned model perfectly matches the true model,but degrades rapidly as the model becomes worse. Wedemonstrate that using iteratively refitted local linear mod-els achieves substantially better results with imaginationrollouts than more complex neural network models. Wehypothesize that this is likely due to the fact that the more

Continuous Deep Q-Learning with Model-based Acceleration

expressive models themselves require substantially moredata, and that otherwise efficient algorithms like Dyna-Qare vulnerable to poor model approximations.

3. BackgroundIn reinforcement learning, the goal is to learn a policy tocontrol a system with states x ∈ X and actions u ∈ Uin environment E, so as to maximize the expected sum ofreturns according to a reward function r(x,u). The dy-namical system is defined by an initial state distributionp(x1) and a dynamics distribution p(xt+1|xt,ut). At eachtime step t ∈ [1, T ], the agent chooses an action ut ac-cording to its current policy π(ut|xt), and observes a re-ward r(xt,ut). The agent then experiences a transition to anew state sampled from the dynamics distribution, and wecan express the resulting state visitation frequency of thepolicy π as ρπ(xt). Define Rt =

∑Ti=t γ

(i−t)r(xi,ui),the goal is to maximize the expected sum of returns, givenby R = Eri≥1,xi≥1∼E,ui≥1∼π[R1], where γ is a discountfactor that prioritizes earlier rewards over later ones. Withγ < 1, we can also set T =∞, though we use a finite hori-zon for all of the tasks in our experiments. The expected re-turn R can be optimized using a variety of model-free andmodel-based algorithms. In this section, we review severalof these methods that we build on in our work.

Model-Free Reinforcement Learning. When the sys-tem dynamics p(xt+1|xt,ut) are not known, as is oftenthe case with physical systems such as robots, policy gra-dient methods (Peters & Schaal, 2006) and value functionor Q-function learning with function approximation (Sut-ton et al., 1999) are often preferred. Policy gradient meth-ods provide a simple, direct approach to RL, which cansucceed on high-dimensional problems, but potentially re-quires a large number of samples (Schulman et al., 2015;2016). Off-policy algorithms that use value or Q-functionapproximation can in principle achieve better data effi-ciency (Lillicrap et al., 2016). However, adapting suchmethods to continuous tasks typically requires optimizingtwo function approximators on different objectives. We in-stead build on standard Q-learning, which has a single ob-jective. We summarize Q-learning in this section. The Qfunction Qπ(xt,ut) corresponding to a policy π is definedas the expected return from xt after taking action ut andfollowing the policy π thereafter:

Qπ(xt,ut) = Eri≥t,xi>t∼E,ui>t∼π[Rt|xt,ut] (1)

Q-learning learns a greedy deterministic policyµ(xt) = argmaxuQ(xt,ut), which corresponds toπ(ut|xt) = δ(ut = µ(xt)). Let θQ parametrize theaction-value function, and β be an arbitrary explorationpolicy, the learning objective is to minimize the Bellman

error, where we fix the target yt:

L(θQ) = Ext∼ρβ ,ut∼β,rt∼E [(Q(xt,ut|θQ)− yt)2]yt = r(xt,ut) + γQ(xt+1,µ(xt+1))

(2)

For continuous action problems, Q-learning becomes diffi-cult, because it requires maximizing a complex, nonlinearfunction at each update. For this reason, continuous do-mains are often tackled using actor-critic methods (Konda& Tsitsiklis, 1999; Hafner & Riedmiller, 2011; Silver et al.,2014; Lillicrap et al., 2016), where a separate parame-terized “actor” policy π is learned in addition to the Q-function or value function “critic,” such as Deep Determin-istic Policy Gradient (DDPG) algorithm (Lillicrap et al.,2016).

In order to describe our method in the following sections, itwill be useful to also define the value function V π(xt,ut)and advantage function Aπ(xt,ut) of a given policy π:

V π(xt) = Eri≥t,xi>t∼E,ui≥t∼π[Rt|xt,ut]Aπ(xt,ut) = Qπ(xt,ut)− V π(xt).

(3)

Model-Based Reinforcement Learning. If we know thedynamics p(xt+1|xt,ut), or if we can approximate themwith some learned model p(xt+1|xt,ut), we can usemodel-based RL and optimal control. While a wide rangeof model-based RL and control methods have been pro-posed in the literature (Deisenroth et al., 2013; Kober &Peters, 2012), two are particularly relevant for this work:iterative LQG (iLQG) (Li & Todorov, 2004) and Dyna-Q (Sutton, 1990). The iLQG algorithm optimizes tra-jectories by iteratively constructing locally optimal lin-ear feedback controllers under a local linearization of thedynamics p(xt+1|xt,ut) = N (fxtxt + futut,Ft) and aquadratic expansion of the rewards r(xt,ut) (Tassa et al.,2012). Under linear dynamics and quadratic rewards, theaction-value function Q(xt,ut) and value function V (xt)are locally quadratic and can be computed by dynamicsprogramming. The optimal policy can be derived ana-lytically from the quadratic Q(xt,ut) and V (xt) func-tions, and corresponds to a linear feedback controllerg(xt) = ut + kt +Kt(xt − xt), where kt is an open-loop term, Kt is the closed-loop feedback matrix, and xtand ut are the states and actions of the nominal trajectory,which is the average trajectory of the controller. Employingthe maximum entropy objective (Levine & Koltun, 2013),we can also construct a linear-Gaussian controller, wherec is a scalar to adjust for arbitrary scaling of the rewardmagnitudes:

πiLQGt (ut|xt) = N (ut + kt +Kt(xt − xt),−cQ−1u,ut)

(4)

When the dynamics are not known, a particularly effectiveway to use iLQG is to combine it with learned time-varying

Continuous Deep Q-Learning with Model-based Acceleration

linear models p(xt+1|xt,ut). In this variant of the algo-rithm, trajectories are sampled from the controller in Equa-tion (4) and used to fit time-varying linear dynamics withlinear regression. These dynamics are then used with iLQGto obtain a new controller, typically using a KL-divergenceconstraint to enforce a trust region, so that the new con-troller doesn’t deviate too much from the region in whichthe samples were generated (Levine & Abbeel, 2014).

Besides enabling iLQG and other planning-based algo-rithms, a learned model of the dynamics can allow a model-free algorithm to generate synthetic experience by perform-ing rollouts in the learned model. A particularly relevantmethod of this type is Dyna-Q (Sutton, 1990), which per-forms real-world rollouts using the policy π, and then gen-erates synthetic rollouts using a model learned from thesesamples. The synthetic rollouts originate at states visitedby the real-world rollouts, and serve as supplementary datafor a variety of possible reinforcement learning algorithms.However, most prior Dyna-Q methods have focused on rel-atively small, discrete domains. In Section 5, we describehow our method can be extended into a variant of Dyna-Qto achieve substantially faster learning on a range of con-tinuous control tasks with complex neural network policies,and in Section 6, we empirically analyze the sensitivity ofthis method to imperfect learned dynamics models.

4. Continuous Q-Learning with NormalizedAdvantage Functions

We first propose a simple method to enable Q-learning incontinuous action spaces with deep neural networks, whichwe refer to as normalized advantage functions (NAF). Theidea behind normalized advantage functions is to representthe Q-function Q(xt,ut) in Q-learning in such a way thatits maximum, argmaxuQ(xt,ut), can be determined eas-ily and analytically during the Q-learning update. While anumber of representations are possible that allow for ana-lytic maximization, the one we use in our implementationis based on a neural network that separately outputs a valuefunction term V (x) and an advantage term A(x,u), whichis parameterized as a quadratic function of nonlinear fea-tures of the state:

Q(x,u|θQ) = A(x,u|θA) + V (x|θV )

A(x,u|θA) = −1

2(u− µ(x|θµ))TP (x|θP )(u− µ(x|θµ))

P (x|θP ) is a state-dependent, positive-definitesquare matrix, which is parametrized by P (x|θP ) =L(x|θP )L(x|θP )T , where L(x|θP ) is a lower-triangularmatrix whose entries come from a linear output layer ofa neural network, with the diagonal terms exponentiated.While this representation is more restrictive than a generalneural network function, since the Q-function is quadratic

in u, the action that maximizes the Q-function is alwaysgiven by µ(x|θµ). We use this representation with a deepQ-learning algorithm analogous to Mnih et al. (2015),using target networks and a replay buffers as describedby (Lillicrap et al., 2016). NAF, given by Algorithm 1, isconsiderably simpler than DDPG.

Algorithm 1 Continuous Q-Learning with NAF

Randomly initialize normalized Q network Q(x,u|θQ).Initialize target network Q′ with weight θQ

′← θQ.

Initialize replay buffer R← ∅.for episode=1,M do

Initialize a random processN for action explorationReceive initial observation state x1 ∼ p(x1)for t=1, T do

Select action ut = µ(xt|θµ) +NtExecute ut and observe rt and xt+1

Store transition (xt,ut, rt,xt+1) in Rfor iteration=1, I do

Sample a random minibatch of m transitions from R

Set yi = ri + γV ′(xi+1|θQ′)

Update θQ by minimizing the loss: L = 1N

∑i(yi −

Q(xi,ui|θQ))2

Update the target network: θQ′← τθQ + (1− τ)θQ

′

end forend for

end for

Decomposing Q into an advantage term A and a state-value term V was suggested by Baird III (1993); Harmon& Baird III (1996), and was recently explored by Wanget al. (2015) for discrete action problems. Normalizedaction-value functions have also been proposed by Raw-lik et al. (2013) in the context of an alternative temporaldifference learning algorithm. However, our method is thefirst to combine such representations with deep neural net-works into an algorithm that can be used to learn policiesfor a range of challenging continuous control tasks. Ingeneral, A does not need to be quadratic, and exploringother parametric forms such as multimodal distributions isan interesting avenue for future work. The appendix pro-vides details on adaptive exploration rule derivation withexperimental results, and a variational interpretation of Q-learning which gives an intuitive explanation of the behav-ior of NAF that conforms with empirical results.

5. Accelerating Learning with ImaginationRollouts

While NAF provides some advantages over actor-criticmodel-free RL methods in continuous domains, we canimprove their data efficiency substantially under some ad-ditional assumptions by exploiting learned models. Wewill show that incorporating a particular type of learnedmodel into Q-learning with NAFs significantly improves

Continuous Deep Q-Learning with Model-based Acceleration

sample efficiency, while still allowing the final policy to befinetuned with model-free learning to achieve good perfor-mance without the limitations of imperfect models.

5.1. Model-Guided Exploration

One natural approach to incorporating a learned model intoan off-policy algorithm such as Q-learning is to use thelearned model to generate good exploratory behaviors us-ing planning or trajectory optimization. To evalaute thisidea, we utilize the iLQG algorithm to generate good tra-jectories under the model, and then mix these trajectoriestogether with on-policy experience by appending them tothe replay buffer. Interestingly, we show in our evalua-tion that, even when planning under the true model, the im-provement obtained from this approach is often quite small,and varies significantly across domains and choices of ex-ploration noise. The intuition behind this result is that off-policy iLQG exploration is too different from the learnedpolicy, and Q-learning must consider alternatives in orderto ascertain the optimality of a given action. That is, it’s notenough to simply show the algorithm good actions, it mustalso experience bad actions to understand which actions arebetter and which are worse.

5.2. Imagination Rollouts

As discussed in the previous section, incorporating off-policy exploration from good, narrow distributions, suchas those induced by iLQG, often does not result in signif-icant improvement for Q-learning. These results suggestthat Q-learning, which learns a policy based on minimizingtemporal differences, inherently requires noisy on-policyactions to succeed. In real-world domains such as robotsand autonomous vehicles, this can be undesirable for tworeasons: first, it suggests that large amounts of on-policyexperience are required in addition to good off-policy sam-ples, and second, it implies that the policy must be allowedto make “its own mistakes” during training, which mightinvolve taking undesirable or dangerous actions that candamage real-world hardware.

One way to avoid these problems while still allowing fora large amount of on-policy exploration is to generate syn-thetic on-policy trajectories under a learned model. Addingthese synthetic samples, which we refer to as imagina-tion rollouts, to the replay buffer effectively augments theamount of experience available for Q-learning. The par-ticular approach we use is to perform rollouts in the realworld using a mixture of planned iLQG trajectories and on-policy trajectories, with various mixing coefficients evalu-ated in our experiments, and then generate additional syn-thetic on-policy rollouts using the learned model from eachstate visited along the real-world rollouts. This approachcan be viewed as a variant of the Dyna-Q algorithm (Sut-

ton, 1990). However, while Dyna-Q has primarily beenused with small and discrete systems, we show that usingiteratively refitted linear models allows us to extend the ap-proach to deep reinforcement learning on a range of con-tinuous control domains. In some scenarios, we can evengenerate all or most of the real rollouts using off-policyiLQG controllers, which is desirable in safety-critic do-mains where poorly trained policies might take dangerousactions. The algorithm is given as Algorithm 2, and is anextension on Algorithm 1 combining model-based RL.

Algorithm 2 Imagination Rollouts with Fitted Dynamicsand Optional iLQG Exploration

Randomly initialize normalized Q network Q(x,u|θQ).Initialize target network Q′ with weight θQ

′← θQ.

Initialize replay buffer R← ∅ and fictional buffer Rf ← ∅.Initialize additional buffers B ← ∅, Bold ← ∅ with size nT .Initialize fitted dynamics modelM← ∅.for episode = 1,M do

Initialize a random processN for action explorationReceive initial observation state x1

Select µ′(x, t) from {µ(x|θµ), πiLQGt (ut|xt)} with proba-bilities {p, 1− p}for t = 1, T do

Select action ut = µ′(xt, t) +NtExecute ut and observe rt and xt+1

Store transition (xt,ut, rt,xt+1, t) in R and Bif mod (episode · T + t,m) = 0 andM 6= ∅ then

Sample m (xi,ui, ri,xi+1, i) from BoldUseM to simulate l steps from each sampleStore all fictional transitions in Rf

end ifSample a random minibatch of m transitions I · l timesfrom Rf and I times from R, and update θQ, θQ

′as in

Algorithm 1 per minibatch.end forif Bf is full thenM← FitLocalLinearDynamics(Bf ) (see Section 5.3)πiLQG ← iLQG OneStep(Bf ,M) (see appendix)Bold ← Bf , Bf ← ∅

end ifend for

Imagination rollouts can suffer from severe bias when thelearned model is inaccurate. For example, we found it verydifficult to train nonlinear neural network models for thedynamics that would actually improve the efficiency of Q-learning when used for imagination rollouts. As discussedin the following section, we found that using iteratively re-fitted time-varying linear dynamics produced substantiallybetter results. In either case, we would still like to preservethe generality and optimality of model-free RL while deriv-ing the benefits of model-based learning. To that end, weobserve that most of the benefit of model-based learning isderived in the early stages of the learning process, when thepolicy induced by the neural network Q-function is poor.As the Q-function becomes more accurate, on-policy be-

Continuous Deep Q-Learning with Model-based Acceleration

havior tends to outperform model-based controllers. Wetherefore propose to switch off imagination rollouts after agiven number of iterations.1 In this framework, the imag-ination rollouts can be thought of as an inexpensive wayto pretrain the Q-function, such that fine-tuning using realworld experience can quickly converge to an optimal solu-tion.

5.3. Fitting the Dynamics Model

In order to obtain good imagination rollouts and improvethe efficiency of Q-learning, we needed to use an effec-tive and data-efficient model learning algorithm. Whileprior methods propose a variety of model classes, includingneural networks (Heess et al., 2015), Gaussian processes(Deisenroth & Rasmussen, 2011), and locally-weighted re-gression (Atkeson et al., 1997), we found that we could ob-tain good results by using iteratively refitted time-varyinglinear models, as proposed by Levine & Abbeel (2014). Inthis approach, instead of learning a good global model forall states and actions, we aim only to obtain a good localmodel around the latest set of samples. This approach re-quires a few additional assumptions: namely, it requires theinitial state to be either deterministic or low-variance Gaus-sian, and it requires the states and actions to all be continu-ous. To handle domains with more varied initial states, wecan use a mixture of Gaussian initial states with separatetime-varying linear models for each one. The model itselfis given by pt(xt+1|xt,ut) = N (Ft[xt;ut]+ft,Nt). Ev-ery n episodes, we refit the parameters Ft, ft, and Nt byfitting a Gaussian distribution at each time step to the vec-tors [xit;u

it;x

it+1], where i indicates the sample index, and

conditioning this Gaussian on [xt;ut] to obtain the param-eters of the linear-Gaussian dynamics at that step. We usen = 5 in our experiments. Although this approach intro-duces additional assumptions beyond the standard model-free RL setting, we show in our evaluation that it producesimpressive gains in sample efficiency on tasks where it canbe applied.

6. ExperimentsWe evaluated our approach on a set of simulated robotictasks using the MuJoCo simulator (Todorov et al., 2012).The tasks were based on the benchmarks described by Lil-licrap et al. (2016). Although we attempted to replicatethe tasks in previous work as closely as possible, discrep-ancies in the simulator parameters and the contact modelproduced results that deviate slightly from those reportedin prior work. In all experiments, the input to the policyconsisted of the state of the system, defined in terms of joint

1In future work, it would be interesting to select this iterationadaptively based on the expected relative performance of the Q-function policy and model-based planning.

angles and root link positions. Angles were often convertedto sine and cosine encoding.

For both our method and the prior DDPG (Lillicrap et al.,2016) algorithm in the comparisons, we used neural net-works with two layers of 200 rectified linear units (ReLU)to produce each of the output parameters – the Q-functionand policy in DDPG, and the value function V , the advan-tage matrix L, and the mean µ for NAF. Since Q-learningwas done with a replay buffer, we applied the Q-learningupdate 5 times per each step of experience to acceleratelearning (I = 5). To ensure a fair comparison, DDPG alsoupdates both the Q-function and policy parameters 5 timesper step.

6.1. Normalized Advantage Functions

In this section, we compare NAF and DDPG on 10 repre-sentative domains from Lillicrap et al. (2016), with threeadditional domains: a four-legged 3D ant, a six-joint 2Dswimmer, and a 2D peg (see the appendix for the de-scriptions of task domains). We found the most sensitivehyperparameters to be presence or absence of batch nor-malization, base learning rate for ADAM (Kingma & Ba,2014) ∈ {1e−4, 1e−3, 1e−2}, and exploration noise scale∈ {0.1, 0.3, 1.0}. We report the best performance for eachdomain. We were unable to achieve good results withthe method of Rawlik et al. (2013) on our domains, likelydue to the complexity of high-dimensional neural networkfunction approximators.

Figure 1b, Figure 1c, and additional figures in the appendixshow the performances on the three-joint reacher, peg in-sertion, and a gripper with mobile base. While the nu-merical gap in reacher may be small, qualitatively there isalso a very noticeable difference between NAF and DDPG.DDPG converges to a solution where the deterministic pol-icy causes the tip to fluctuate continuously around the tar-get, and does not reach it precisely. NAF, on the other hand,learns a smooth policy that makes the tip slow down andstabilize at the target. This difference is more noticeable inpeg insertion and moving gripper, as shown by the muchfaster convergence rate to the optimal solution. Precision isvery important in many real-world robotic tasks, and theseresult suggest that NAF may be preferred in such domains.

On locomotion tasks, the performance of the two meth-ods is relatively similar. On the six-joint swimmer taskand four-legged ant, NAF slightly outperforms DDPG interms of the convergence speed; however, DDPG is fasteron cheetah and finds a better policy on walker2d. The lossin performance of NAF can potentially be explained bydownside of the mode-seeking behavior as analyzed in theappendix, where it is hard to explore other modes once thequadratic advantage function finds a good one. Choosinga parametric form that is more expressive than a quadratic

Continuous Deep Q-Learning with Model-based Acceleration

(a) Example task domains. (b) NAF and DDPG on multi-target reacher. (c) NAF and DDPG on peg insertion.

Figure 1. (a) Task domains: top row from left (manipulation tasks: peg, gripper, mobile gripper), bottom row from left (locomotiontasks: cheetah, swimmer6, ant). (b,c) NAF vs DDPG results on three-joint reacher and peg insertion. On reacher, the DDPG policycontinuously fluctuates the tip around the target, while NAF stabilizes well at the target.

could be used to address this limitation in future work.

The results on all of the domains are summarized in Ta-ble 1. Overall, NAF outperformed DDPG on the major-ity of tasks, particularly manipulation tasks that requireprecision and suffer less from the lack of multimodal Q-functions. This makes this approach particularly promisingfor efficient learning of real-world robotic tasks.

Domains - DDPG episodes NAF episodesCartpole -2.1 -0.601 420 -0.604 190Reacher -2.3 -0.509 1370 -0.331 1260

Peg -11 -0.950 690 -0.438 130Gripper -29 1.03 2420 1.81 1920

GripperM -90 -20.2 1350 -12.4 730Canada2d -12 -4.64 1040 -4.21 900Cheetah -0.3 8.23 1590 7.91 2390

Swimmer6 -325 -174 220 -172 190Ant -4.8 -2.54 2450 -2.58 1350

Walker2d 0.3 2.96 850 1.85 1530

Table 1. Best test rewards of DDPG and NAF policies, and theepisodes it requires to reach within 5% of the best value. “-” de-notes scores by a random agent.

6.2. Evaluating Best-Case Model-Based Improvementwith True Models

In order to determine how best to incorporate model-basedcomponents to accelerate model-free Q-learning, we testedseveral approaches using the ground truth dynamics, tocontrol for challenges due to model fitting. We evaluatedboth of the methods discussed in Section 5: the use ofmodel-based planning to generate good off-policy rolloutsin the real world, and the use of the model to generate on-policy synthetic rollouts.

Figure 2a shows the effect of mixing off-policy iLQG expe-rience and imagination rollouts on the three-joint reacher.It is noticeable that mixing the good off-policy experiencedoes not significantly improve data-efficiency, while imagi-nation rollouts always improve data-efficiency or final per-formance significantly. In the context of Q-learning, this

result is not entirely surprising: Q learning must experi-ence both good and bad actions in order to determine whichactions are preferred, while the good model-based rolloutsare so far removed from the policy in the early stages oflearning that they provide little useful information. Fig-ure 2a also evaluates two different variants of the imag-ination rollouts approach, where the rollouts in the realworld are performed either using the learned policy, or us-ing model-based planning with iLQG. In the case of thistask, the iLQG rollouts achieve slightly better results, sincethe on-policy imagination rollouts sampled around theseoff-policy states provide Q-learning with additional infor-mation about alternative action not taken by the iLQG plan-ner. In general, we did not find that off-policy rollouts wereconsistently better than on-policy rollouts across all tasks,but they did consistently produce good results. Perform-ing off-policy rollouts with iLQG may be desirable in real-world domains, where a partially learned policy might takeundesirable or dangerous actions. Further details of theseexperiments are provided in the appendix.

6.3. Guided Imagination Rollouts with FittedDynamics

In this section, we evaluated the performance of imagina-tion rollouts with learned dynamics. As seen in Figure 2b,we found that fitting time-varying linear models follow-ing the imagination rollout algorithm is substantially betterthan fitting neural network dynamics models for the taskswe considered. There is a fundamental tension between ef-ficient learning and expressive models like neural nets. Wecannot hope to learn useful neural network models with asmall number of samples for complex tasks, which makes itdifficult to acquire a good model with fewer samples thanare necessary to acquire a good policy. While the modelis trained with supervised learning, which is typically moresample efficient, it often needs to represent a more complexfunction (e.g. rigid body physics). However, having suchexpressive models is more crucial as we move to improvemodel accuracy. Figure 2b presents results that compare

Continuous Deep Q-Learning with Model-based Acceleration

(a) NAF on single-target reacher. (b) NAF on single-target reacher. (c) NAF on single-target gripper.Figure 2. Results on NAF with iLQG-guided exploration and imagination rollouts (a) using true dynamics (b,c) using fitted dynamics.“ImR” denotes using the imagination rollout with l = 10 steps on the reacher and l = 5 steps on the gripper. “iLQG-x” indicates mixingx fraction of iLQG episodes. Fitted dynamics uses time-varying linear models with sample size n = 5, except “-NN” which fits a neuralnetwork to global dynamics.

fitted neural network models with the true dynamics whencombined with imagination rollouts. These results indicatethat the learned neural network models negate the benefitsof imagination rollouts on our domains.

To evaluate imagination rollouts with fitted time-varyinglinear dynamics, we chose single-target variants of two ofthe manipulation tasks: the reacher and the gripper task.The results are shown in Figure 2b and 2c. We foundthat imagination rollouts of length 5 to 10 were sufficientfor these tasks to achieve significant improvement over thefully model-free variant of NAF.

Adding imagination rollouts in these domains provided 2-5factors of improvement in data efficiency. In order to re-tain the benefit of model-free learning and allow the policyto continue improving once it exceeds the quality possi-ble under the learned model, we switch off the imaginationrollouts after 130 episodes (20,000 steps) on the gripperdomain. This produces a small transient drop in the perfor-mance of the policy, but the results quickly improve again.Switching off the imagination rollouts also ensures that Q-learning does not diverge after it reaches good values, aswere often observed in the gripper. This suggests that imag-ination rollouts, in contrast to off-policy exploration dis-cussed in the previous section, is an effective method forbootstrapping model-free deep RL.

It should be noted that, although time-varying linear mod-els combined with imagination rollouts provide a substan-tial boost in sample efficiency, this improvement is pro-vided at some cost in generality, since effective fitting oftime-varying linear models requires relatively small initialstate distributions. With more complex initial state distri-butions, we might cluster the trajectories and fit multiplemodels to account for different modes. Extending the bene-fits of time-varying linear models to less restrictive settingsis a promising direction and build on prior work (Levineet al., 2015; Fu et al., 2015). That said, our results showthat imagination rollouts are a very promising approach to

accelerating model-free learning when combined with theright kind of dynamics model.

7. DiscussionIn this paper, we explored several methods for improv-ing the sample efficiency of model-free deep reinforcementlearning. We first propose a method for applying standardQ-learning methods to high-dimensional, continuous do-mains, using the normalized advantage function (NAF) rep-resentation. This allows us to simplify the more standardactor-critic style algorithms, while preserving the benefitsof nonlinear value function approximation, and allows us toemploy a simple and effective adaptive exploration method.We show that, in comparison to recently proposed deepactor-critic algorithms, our method tends to learn faster andacquires more accurate policies. We further explore howmodel-free RL can be accelerated by incorporating learnedmodels, without sacrificing the optimality of the policy inthe face of imperfect model learning. We show that, al-though Q-learning can incorporate off-policy experience,learning primarily from off-policy exploration (via model-based planning) only rarely improves the overall sampleefficiency of the algorithm. We postulate that this causedby the need to observe both successful and unsuccessfulactions, in order to obtain an accurate estimate of the Q-function. We demonstrate that an alternative method basedon synthetic on-policy rollouts achieves substantially im-proved sample complexity, but only when the model learn-ing algorithm is chosen carefully. We demonstrate thattraining neural network models does not provide substan-tive improvement in our domains, but simple iteratively re-fitted time-varying linear models do provide substantial im-provement on domains where they can be applied.

Continuous Deep Q-Learning with Model-based Acceleration

AcknowledgementWe thank Nicholas Heess for helpful discussion and TomErez, Yuval Tassa, Vincent Vanhoucke, and the GoogleBrain and DeepMind teams for their support.

ReferencesAtkeson, Christopher G, Moore, Andrew W, and Schaal,

Stefan. Locally weighted learning for control. In Lazylearning, pp. 75–113. Springer, 1997.

Baird III, Leemon C. Advantage updating. Technical re-port, DTIC Document, 1993.

de Bruin, Tim, Kober, Jens, Tuyls, Karl, and Babuska,Robert. The importance of experience replay databasecomposition in deep reinforcement learning. Deep Rein-forcement Learning Workshop, NIPS, 2015.

Deisenroth, Marc and Rasmussen, Carl E. Pilco: A model-based and data-efficient approach to policy search. InInternational Conference on Machine Learning (ICML),pp. 465–472, 2011.

Deisenroth, Marc Peter, Neumann, Gerhard, Peters, Jan,et al. A survey on policy search for robotics. Foundationsand Trends in Robotics, 2(1-2):1–142, 2013.

Fu, Justin, Levine, Sergey, and Abbeel, Pieter. One-shotlearning of manipulation skills with online dynamicsadaptation and neural network priors. arXiv preprintarXiv:1509.06841, 2015.

Hafner, Roland and Riedmiller, Martin. Reinforcementlearning in feedback control. Machine learning, 84(1-2):137–169, 2011.

Harmon, Mance E and Baird III, Leemon C. Multi-playerresidual advantage learning with general function ap-proximation. Wright Laboratory, WL/AACF, Wright-Patterson Air Force Base, OH, pp. 45433–7308, 1996.

Hausknecht, Matthew and Stone, Peter. Deep reinforce-ment learning in parameterized action space. arXivpreprint arXiv:1511.04143, 2015.

Heess, Nicolas, Wayne, Gregory, Silver, David, Lillicrap,Tim, Erez, Tom, and Tassa, Yuval. Learning con-tinuous control policies by stochastic value gradients.In Advances in Neural Information Processing Systems(NIPS), pp. 2926–2934, 2015.

Kingma, Diederik and Ba, Jimmy. Adam: Amethod for stochastic optimization. arXiv preprintarXiv:1412.6980, 2014.

Kober, Jens and Peters, Jan. Reinforcement learning inrobotics: A survey. In Reinforcement Learning, pp. 579–610. Springer, 2012.

Konda, Vijay R and Tsitsiklis, John N. Actor-critic algo-rithms. In Advances in Neural Information ProcessingSystems (NIPS), volume 13, pp. 1008–1014, 1999.

Levine, Sergey and Abbeel, Pieter. Learning neural net-work policies with guided policy search under unknowndynamics. In Advances in Neural Information Process-ing Systems (NIPS), pp. 1071–1079, 2014.

Levine, Sergey and Koltun, Vladlen. Guided policysearch. In International Conference on Machine Learn-ing (ICML), pp. 1–9, 2013.

Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel,Pieter. End-to-end training of deep visuomotor policies.arXiv preprint arXiv:1504.00702, 2015.

Li, Weiwei and Todorov, Emanuel. Iterative lin-ear quadratic regulator design for nonlinear biologicalmovement systems. In ICINCO (1), pp. 222–229, 2004.

Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander,Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David,and Wierstra, Daan. Continuous control with deep rein-forcement learning. International Conference on Learn-ing Representations (ICLR), 2016.

Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David,Rusu, Andrei A, Veness, Joel, Bellemare, Marc G,Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K,Ostrovski, Georg, et al. Human-level control throughdeep reinforcement learning. Nature, 518(7540):529–533, 2015.

Peters, Jan and Schaal, Stefan. Policy gradient methodsfor robotics. In International Conference on IntelligentRobots and Systems (IROS), pp. 2219–2225. IEEE, 2006.

Peters, Jan, Mulling, Katharina, and Altun, Yasemin. Rel-ative entropy policy search. In AAAI. Atlanta, 2010.

Rawlik, Konrad, Toussaint, Marc, and Vijayakumar, Sethu.On stochastic optimal control and reinforcement learn-ing by approximate inference. Robotics, pp. 353, 2013.

Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Sil-ver, David. Prioritized experience replay. arXiv preprintarXiv:1511.05952, 2015.

Schulman, John, Levine, Sergey, Abbeel, Pieter, Jordan,Michael I., and Moritz, Philipp. Trust region policyoptimization. In International Conference on MachineLearning (ICML), pp. 1889–1897, 2015.

Continuous Deep Q-Learning with Model-based Acceleration

Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan,Michael, and Abbeel, Pieter. High-dimensional con-tinuous control using generalized advantage estimation.International Conference on Learning Representations(ICLR), 2016.

Silver, David, Lever, Guy, Heess, Nicolas, Degris, Thomas,Wierstra, Daan, and Riedmiller, Martin. Deterministicpolicy gradient algorithms. In International Conferenceon Machine Learning (ICML), 2014.

Sutton, Richard S. Integrated architectures for learning,planning, and reacting based on approximating dynamicprogramming. In International Conference on MachineLearning (ICML), pp. 216–224, 1990.

Sutton, Richard S, McAllester, David A, Singh, Satin-der P, Mansour, Yishay, et al. Policy gradient methodsfor reinforcement learning with function approximation.In Advances in Neural Information Processing Systems(NIPS), volume 99, pp. 1057–1063, 1999.

Tassa, Yuval, Erez, Tom, and Todorov, Emanuel. Synthe-sis and stabilization of complex behaviors through onlinetrajectory optimization. In International Conference onIntelligent Robots and Systems (IROS), pp. 4906–4913.IEEE, 2012.

Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco:A physics engine for model-based control. In Inter-national Conference on Intelligent Robots and Systems(IROS), pp. 5026–5033. IEEE, 2012.

Wahlstrom, Niklas, Schon, Thomas B, and Deisenroth,Marc Peter. From pixels to torques: Policy learn-ing with deep dynamical models. arXiv preprintarXiv:1502.02251, 2015.

Wang, Ziyu, de Freitas, Nando, and Lanctot, Marc. Duelingnetwork architectures for deep reinforcement learning.arXiv preprint arXiv:1511.06581, 2015.

Watter, Manuel, Springenberg, Jost, Boedecker, Joschka,and Riedmiller, Martin. Embed to control: A locally lin-ear latent dynamics model for control from raw images.In Advances in Neural Information Processing Systems(NIPS), pp. 2728–2736, 2015.

Continuous Deep Q-Learning with Model-based Acceleration

8. Appendix8.1. iLQG

The iLQG algorithm optimizes trajectories by iterativelyconstructing locally optimal linear feedback controllers un-der a local linearization of the dynamics p(xt+1|xt,ut) =N (fxtxt + futut,Ft) and a quadratic expansion of therewards r(xt,ut) (Tassa et al., 2012). Under linear dy-namics and quadratic rewards, the action-value functionQ(xt,ut) and value function V (xt) are locally quadraticand can be computed by dynamics programming.2

Qxu,xut = rxu,xut + fTxutVx,xt+1fxut

Qxut = rxut + fTxutVx,xt+1

Vx,xt = Qx,xt −QTu,xtQ−1u,utQu,xt

Vxt = Qxt −QTu,xtQ−1u,utQut

Qx,xT = Vx,xT = rx,xT

(5)

The time-varying linear feedback controller g(xt) = ut +kt + Kt(xt − xt) maximizes the locally quadratic Q,where kt = −Q−1u,utQut,Kt = −Q−1u,utQu,xt, and xt, utdenote states and actions of the current trajectory aroundwhich the partial derivatives are computed. Employing themaximum entropy objective (Levine & Koltun, 2013), wecan also construct a linear-Gaussian controller, where c isa scalar to adjust for arbitrary scaling of the reward magni-tudes,

πiLQGt (ut|xt) = N (ut + kt +Kt(xt − xt),−cQ−1u,ut)

(6)

When the dynamics are not known, a particularly effectiveway to use iLQG is to combine it with learned time-varyinglinear models p(xt+1|xt,ut). In this variant of the algo-rithm, trajectories are sampled from the controller in Equa-tion (6) and used to fit time-varying linear dynamics withlinear regression. These dynamics are then used with iLQGto obtain a new controller, typically using a KL-divergenceconstraint to enforce a trust region, so that the new con-troller doesn’t deviate too much from the region in whichthe samples were generated (Levine & Abbeel, 2014).

8.2. Locally-Invariant Exploration for NormalizedAdvantage Functions

Exploration is an essential component of reinforcementlearning algorithms. The simplest and most common typeof exploration involves randomizing the actions accord-ing to some distribution, either by taking random actionswith some probability (Mnih et al., 2015), or adding Gaus-sian noise in continuous action spaces (Schulman et al.,

2While standard iLQG notation denotes Q,V as discountedsum of costs, we denote them as sum of rewards to make themconsistent with the rest of the paper

2015). However, choosing the magnitude of the randomexploration noise can be difficult, particularly in high-dimensional domains where different action dimensions re-quire very different exploration scales. Furthermore, inde-pendent (spherical) Gaussian noise may be inappropriatefor tasks where the optimal behavior requires correlationbetween action dimensions, as for example in the case ofthe swimming snake described in our experiments, whichmust coordinate the motion of different body joints to pro-duce a synchronized undulating gait.

The NAF provides us with a simple and natural avenueto obtain an adaptive exploration strategy, analogously toBoltzmann exploration. The idea is to use the matrix inthe quadratic component of the advantage function as theprecision for a Gaussian action distribution. This naturallycauses the policy to become more deterministic along di-rections where the advantage function varies steeply, andmore random along directions where it is flat. The corre-sponding policy is given by

π(u|x) = expQ(x,u|θQ) /

∫expQ(x,u|θQ) du

= N (µ(x|θµ), cP (x|θP )−1).(7)

Previous work also noted that generating Gaussian explo-ration noise independently for each time step was not well-suited for many continuous control tasks, particularly simu-lated robotic tasks where the actions correspond to torquesor velocities (Lillicrap et al., 2016). The intuition is that,as the length of the time-step decreases, temporally inde-pendent Gaussian exploration will cancel out between timesteps. Instead, prior work proposed to sample noise froman Ornstein-Uhlenbech (OU) process to generate a tempo-rally correlated noise sequence (Lillicrap et al., 2016). Weadopt the same approach in our work, but sample the in-novations for the OU process from the Gaussian distribu-tion in Equation 7. Lastly, we note that the overall scaleof P (x|θP ) could vary significantly through the learning,and depends on the magnitude of the cost, which introducesan undesirable additional degree of freedom. We thereforeuse a heuristic adaptive-scaling trick to stabilize the noisemagnitudes.

Using the learned precision as the noise covariance for ex-ploration allowed for convergence to a better policy on the“canada2d” task, which requires using an arm to strike apuck toward a target, as shown in Figure 3, but did not makea significant difference on the other domains.

8.3. Q-Learning as Variational Inference

NAF, with our choice of parametrization, can only fit a lo-cally quadratic Q-function. To understand its implicationsand expected behaviors, one approach is to approximatelyview the Q-learning objective as minimizing the exclusive

Continuous Deep Q-Learning with Model-based Acceleration

Figure 3. NAF with exploration noise generated using the preci-sion term (NAF-P) slightly outperforms the best DDPG result.Precision term is not used until step 50,000.

Kullback-Leibler divergence (KL) between policy distri-butions induced by the fitted normalized Q-function Q andthe true Q-function Q. This can be derived by assuming(1) no bootstrapping and the exact target Q is provided perstep, (2) no additive exploration noise, i.e. fully on-policy,and (3) use KL loss instead of the least-square. Specifi-cally, let π and π be corresponding policies of Q and Qrespectively, an alternative form of Q-learning could be op-timizing the following objective:

Le(Q) = Ext∼ρπ [KL(π||π)] = Ext∼ρπ,ut∼π[Q−Q]

(8)

We can thus intuitively interpret NAF as doing variationalinference to fit a Gaussian to a distribution, and it hasmode-seeking behavior. Empirically such behavior en-ables NAF to learn smoother and more precise controllers,as most effectively illustrated by three-joint reacher andpeg insertion experiments, and substantial improvements interms of convergence speeds in many other representativedomains explored in the main paper.

8.4. Descriptions of Task Domains

Table 3 describes the task domains used in the experiments.

8.5. More Results on Normalized Advantage Functions

Figures 4a, 4b, and 4c provide additional results on thecomparison experiments between DDPG and NAF. Asshown in the main paper, NAF generally outperformsDDPG. In certain tasks that require precision, such as peginsertion, the difference is very noticeable. However, thereare also few cases where NAF underperforms DDPG. Themost consistent of such cases is cheetah. While both DDPGand NAF enable cheetah to run decent distances, it is oftenobserved that the cheetah movements learned in NAF are

little less natural than those from DDPG. We speculate suchbehaviors come from the mode-seeking behavior that is ex-plained in Section 8.3, and thus exploring other parametricforms of NAF, such as multi-modal variants, is a promisingavenue for future work.

8.6. More Results on Evaluating Best-CaseModel-Based Improvement with True Models

Domains - 0.5 ImR ImR,0.5 ImR,1Reacher -0.488 -0.449 -0.448 -0.426 -0.548episodes 740 670 450 430 90

Canada2d -6.23 -6.23 -5.89 -5.88 -12.0episodes 1970 1580 570 140 210Cheetah 7.00 7.10 7.36 7.29 6.43episodes 580 1080 590 740 390

Table 2. Best-case model-based acceleration with true dynamicsmodels. Best test rewards of NAF policies (first row), and theepisodes it required to reach 5% of the best value (second row).“0.5” and “1” correspond to the fraction of MPC episodes. “ImR”means using imagination rollout with rollout length l = 10 forreacher, canada2d, and l = 5 for cheetah.

In the main paper, iLQG with true dynamics is used to gen-erate guided exploration trajectories. While iLQG worksfor simple manipulation tasks with small number of initialstates, it does not work well for random target reacher orcomplex locomotion tasks such as cheetah. We thereforerun iLQG in model-predictive control (MPC) mode for theexperiments reported in Figures 5c, 5b, and 5a, and Table 2.It is important to note that for those experiments, the hyper-parameters were fixed (batch normalization is on, learningrate is 10−3, and exploration size is 0.3), and thus the re-sults differ slightly from the experiments in the previoussection.

In cheetah and other complex locomotion tasks, MPC pol-icy is usually sub-optimal, and thus poor performance ofmixing MPC experience in Figure 5b is expected. On theother hand, MPC policy works reasonably in hard manipu-lation tasks such as canada2d, and there is significant gainfrom mixing MPC experience as Figure 5c shows. How-ever, the most consistent gain comes from using imagina-tion rollouts in all three domains. In particular, Figure 5cshows that in canada2d, MPC experiences gives very goodtrajectories, i.e. those that hit balls in roughly the right di-rections, and doing rollouts can generate more of this usefulexperience, enabling canada2d to learn very quickly. Whilewith true dynamics having the imagination experience di-rectly means more experience and such result may be triv-ial, it is still important to see the benefits of rollouts whichonly explore up to l = 10 steps away from the real expe-rience, as reported here. This is an interesting result, sincethis means the dynamics model only needs to be accuratearound the data trajectories and this significantly lessensthe requirement on fitted models.

Continuous Deep Q-Learning with Model-based Acceleration

(a) NAF significantly outperformsDDPG on moving gripper.

(b) NAF converges faster thanDDPG on swimmer6.

(c) DDPG converges faster thanNAF on cheetah.

Figure 4. NAF vs DDPG on three domains.

(a) NAF on multi-target reacher. In-significant gain from mixing MPCexperience. Significant gain fromimagination rollouts.

(b) NAF on cheetah. Great speedsup with imagination rollouts, nogain from mixing MPC experi-ences.

(c) NAF on canada2d. Very signif-icant speed-ups from mixing MPCexperiences, both with or withoutthe rollouts.

Figure 5. NAF on multi-target reacher, cheetah, and canada2d, with model-based acceleration using true dynamics: “ImR” denotes usingthe imagination rollout, l = 10 steps. “MPC-x” indicates mixing x fraction of MPC episodes.

Domain Description Domain Description

Cartpole

The classic cart-pole swing-up task. Agentmust balance a pole attached to a cart byapplying forces to the cart alone. The polestarts each episode hanging upside-down.

ReacherAgent is required to move a 3-DOF arm fromrandom starting locations to random targetpositions.

PegAgent is required to insert the tip of a 3-DOFarm from locally-perturbed starting locationsto a fixed hole.

GripperAgent must use an arm with gripper appendageto grasp an object and manuver the object to afixed target.

GripperMAgent must use an arm with gripper attached toa moveable platform to grasp an object andmove it to a fixed target.

Canada2d

Agent is required to use an arm withhockey-stick like appendage to hit a ballinitialzed to a random start location to arandom target location.

CheetahAgent should move forward as quickly aspossible with a cheetah- like body that isconstrained to the plane.

Swimmer6Agent should swim in snake-like mannertoward the fixed target using six joints, startingfrom random poses.

AntThe four-legged ant should move toward thefixed target from a fixed starting position andposture.

Walker2d

Agent should move forward as quickly aspossible with a bipedal walker constrained tothe plane without falling down or pitching thetorso too far forward or backward.

Table 3. List of domains. All the domains except ant are 2D.