Episodic Learning with Control Lyapunov Functions forUncertain Robotic Systems*

Andrew J. Taylor1, Victor D. Dorobantu1, Hoang M. Le, Yisong Yue, Aaron D. Ames

Abstract— Many modern nonlinear control methods aim toendow systems with guaranteed properties, such as stabilityor safety, and have been successfully applied to the domainof robotics. However, model uncertainty remains a persistentchallenge, weakening theoretical guarantees and causing im-plementation failures on physical systems. This paper developsa machine learning framework centered around Control Lya-punov Functions (CLFs) to adapt to parametric uncertaintyand unmodeled dynamics in general robotic systems. Ourproposed method proceeds by iteratively updating estimatesof Lyapunov function derivatives and improving controllers,ultimately yielding a stabilizing quadratic program model-based controller. We validate our approach on a planar Segwaysimulation, demonstrating substantial performance improve-ments by iteratively refining on a base model-free controller.

I. INTRODUCTION

The use of Control Lyapunov Functions (CLFs) [4], [38]for nonlinear control of robotic systems is becoming in-creasingly popular [26], [17], [29], often utilizing quadraticprogram (QP) controllers [2], [1], [17]. While effective, onemajor challenge is the need for extensive tuning, whichis largely due to modeling deficiencies such as parametricerror and unmodeled dynamics (cf. [26]). While there hasbeen much research in developing robust control methodsthat maintain stability under uncertainty (e.g., via input-to-state stability [39]) or in adapting to limited forms ofuncertainty (e.g., adaptive control [23], [20]), relatively littlework has been done on systematically reducing uncertaintywhile maintaining stability for general function classes ofmodels.

We take a machine learning approach to address the abovelimitations. Learning-based approaches have already showngreat promise for controlling imperfectly modeled roboticplatforms [22], [35]. Successful learning-based approacheshave typically focused on learning model-based uncertainty[5], [8], [7], [37], or direct model-free controller design [25],[36], [14], [42], [24].

We are particularly interested in learning-based approachesthat guarantee Lyapunov stability [21]. From that perspective,the bulk of previous work has focused on using learning toconstruct a Lyapunov function [31], [12], [30], or to assessthe region of attraction for a Lyapunov function [9], [6].One limitation of previous work is that learning is conducted

1Both authors contributed equally.All authors are with the Department of Computing and

Mathematical Sciences, California Institute of Technology,Pasadena, CA 91125, USA ajtaylor@caltech.edu,vdoroban@caltech.edu, hmle@caltech.edu,yyue@caltech.edu, ames@caltech.edu

Fig. 1. CAD model & physical system, a modified Ninebot Segway.

over the full-dimensional state space, which can be datainefficient. We instead constructively prescribe a CLF, andfocus on learning only the necessary information to choosecontrol inputs that achieve the associated stability guarantees,which can be much lower-dimensional.

One challenge in developing learning-based methods forcontroller improvement is how best to collect training datathat accurately reflects the desired operating environmentand control goals. In particular, exhaustive data collectiontypically scales exponentially with dimensionality of the jointstate and control output space, and so should be avoided. Butfirst pre-collecting data upfront can lead to poor performanceas downstream control behavior may enter states that are notpresent in the pre-collected training data. We will leverageepisodic learning approaches such as Dataset Aggregation(DAgger) [33] to address these challenges in a data-efficientmanner, and lead to iteratively refined controllers.

In this paper we present a novel episodic learning approachthat utilizes CLFs to iteratively improve controller design andachieve Lyapunov stability. To the best of our knowledge,our approach is the first that combines CLFs with generalsupervised learning (e.g., including deep learning) in amathematically integrated way. Another distinctive aspect isthat our approach performs learning on the projection of statedynamics onto the CLF time derivative, which can be muchlower dimensional than learning the full state dynamics orthe region of attraction.

Our paper is organized as follows. Section II reviewsinput-output feedback linearization focused on constructingCLFs for unconstrained robotic systems. Section III dis-cusses model uncertainty of a general robotic system andestablishes assumptions on the structure of this uncertainty.

These assumptions allow us to prescribe a CLF for the truesystem, but leave open the question of how to model its timederivative. Section IV provides an episodic learning approachto iteratively improving a model of the time derivative of theCLF. We also present a variant of optimal CLF-based controlthat integrates the learned representation. Finally, Section Vprovides simulation results on a model of a modified Ninebotby Segway E+, seen in Fig. 1. We also provide a Pythonsoftware package (LyaPy) implementing our experiments andlearning framework.1

II. PRELIMINARIES ON CLFS

This section provides a brief review of input-output feed-back linearization, a control technique which can be used tosynthesize a CLF. The resulting CLF will be used to quantifythe impact of model uncertainty and specify the learningproblem outlined in Section III.

A. Input-Output Linearization

Input-Output (IO) Linearization is a nonlinear controlmethod that creates stable linear dynamics for a selectedset of outputs of a system [34]. Outputs encode informationsuch as the position of a floating-based robot or roboticarm end effector as a function of configuration in a waythat is useful for designing controllers. Additionally, IOLinearization provides a constructive method for generatingLyapunov functions, a central tool in certifying stability andsynthesizing controllers for nonlinear systems.

Consider an affine robotic control system with configura-tion space Q ⊆ Rn and an input space U ⊆ Rm. AssumeQ is path-connected and non-empty. The dynamics of thesystem are specified by:

D(q)q̈+C(q, q̇)q̇+G(q)︸ ︷︷ ︸H(q,q̇)

= Bu, (1)

with generalized coordinates q ∈ Q, coordinate rates q̇ ∈Rn, input u ∈ U , inertia matrix D : Q → Sn++, centrifugaland Coriolis terms C : Q×Rn → Rn×n, gravitational forcesG : Q → Rn, and static actuation matrix B ∈ Rn×m. HereSn++ denotes the set of n × n symmetric positive definitematrices. Define twice-differentiable outputs y : Q → Rk,with k ≤ m, and assume each output has relative degree 2on some domain R ⊆ Q (see [34] for details). Intuitively,the relative degree assumption implies that no configurationin R results in an inability to actuate the system. Considerthe time interval I = [t0, tf ] for initial and final timest0, tf and define twice-differentiable time-dependent desiredoutputs yd : I → Rk with r(t) =

[yd(t)

> ẏd(t)>]>.

The error between actual and desired outputs (referred to as

1https://github.com/vdorobantu/lyapy

virtual constraints [45]) yields the dynamic system:

d

dt

[y(q)− yd(t)

ẏ(q, q̇)− ẏd(t)

]=

f(q,q̇)︷ ︸︸ ︷[∂y∂q q̇

∂ẏ∂q q̇−

∂y∂qD(q)

−1H(q, q̇)

]

−[ẏd(t)ÿd(t)

]︸ ︷︷ ︸

ṙ(t)

+

[0k×m

∂y∂qD(q)

−1B

]︸ ︷︷ ︸

g(q)

u,

(2)

noting that ∂ẏ∂q̇ =∂y∂q . For all q ∈ R, g(q) is full rank by the

relative degree assumption. Define η : Q× Rn × I → R2k,f̃ : Q× Rn → Rk, and g̃ : Q → Rk×m as:

η(q, q̇, t) =

[y(q)− yd(t)

ẏ(q, q̇)− ẏd(t)

](3)

f̃(q, q̇) =∂ẏ

∂qq̇− ∂y

∂qD(q)−1H(q, q̇) (4)

g̃(q) =∂y

∂qD(q)−1B, (5)

and assume U = Rm. The input-output linearizing controllaw kIO : Q× Rn × I → U is specified by:

kIO(q, q̇, t) = g̃(q)†(−f̃(q, q̇) + ÿd(t) + ν(q, q̇, t)), (6)

with auxiliary input ν(q, q̇, t) ∈ Rk for all q ∈ Q,q̇ ∈ Rn, and t ∈ I, where † denotes the Moore-Penrosepseudoinverse. Eliminating nonlinear terms, this controllerused in (2) generates linear error dynamics of the form:

η̇(q, q̇, t) =

[0k×k Ik×k0k×k 0k×k

]︸ ︷︷ ︸

F

η(q, q̇, t) +

[0k×kIk×k

]︸ ︷︷ ︸

G

ν(q, q̇, t),

(7)where (F,G) are a controllable pair. Defining gain matrixK =

[Kp Kd

]where Kp, Kd ∈ Sk++, the auxiliary con-

trol input ν(q, q̇, t) = −Kη(q, q̇, t) induces error dynamics:

η̇(q, q̇, t) = Aclη(q, q̇, t), (8)

where Acl = F−GK is Hurwitz. This implies the desiredoutput trajectory yd is exponentially stable, allowing us toconstruct a Lyapunov function for the system using conversetheorems found in [21]. With Acl Hurwitz, for any Q ∈ S2k++,there exists a unique P ∈ S2k++ such that the ContinuousTime Lyapunov Equation (CTLE):

A>clP+PAcl = −Q, (9)

is satisfied. Let C = {η(q, q̇, t) : (q, q̇) ∈ R × Rn, t ∈ I}.Then V (η) = η>Pη, implicitly a function of q, q̇, and t,is a Lyapunov function certifying exponential stability of (8)on C satisfying:

λmin(P)‖η‖22 ≤ V (η) ≤ λmax(P)‖η‖22V̇ (η) ≤ −λmin(Q) ‖η‖22 , (10)

for all η ∈ C. Here λmin(·) and λmax(·) denote the min-imum and maximum eigenvalues of a symmetric matrix,

https://github.com/vdorobantu/lyapy

respectively. A similar Lyapunov function can be constructeddirectly from (7) using the Continuous Algebraic RiccatiEquation (CARE) [1], [21].

B. Control Lyapunov Functions

Lyapunov functions encode information about the dynam-ics in a low-dimensional representation suitable for learning.The preceding formulation of a Lyapunov function requiredthe choice of the specific control law given in (6) to analyzestability of a closed-loop system. More generally, ControlLyapunov Functions (CLFs) extend this idea to enable syn-thesis of optimal nonlinear controllers. Let C ⊆ R2k. Afunction V : R2k → R+ is a Control Lyapunov Function(CLF) for (1) on C certifying exponential stability if thereexist constants c1, c2, c3 > 0 such that:

c1‖η‖22 ≤ V (η) ≤ c2‖η‖22infu∈U

V̇ (η,u) ≤ −c3‖η‖22, (11)

for all η ∈ C. We see that the previously constructedLyapunov function satisfying (10) satisfies (11) by choosingthe control law specified in (6). In the absence of a specificcontrol law, we may write the CLF time derivative as:

V̇ (η,u) =∂V

∂ηη̇ =

∂V

∂η(f(q, q̇)− ṙ(t) + g(q)u). (12)

Dynamic information directly appears within the scalar func-tion V̇ . Also note that V̇ is affine in u, leading to a QP basedcontrol law kQP : Q× Rn × I → U given by:

kQP (q, q̇, t) = argminu∈U

1

2u>Mu+ s>u+ r

s.t. V̇ (η,u) ≤ −c3 ‖η‖22 , (13)

for M ∈ Sm+ , s ∈ Rm, and r ∈ R, provided U is apolyhedron. Here Sm+ denotes the set of m ×m symmetricpositive semi-definite matrices.

III. UNCERTAINTY MODELS & LEARNING

This section defines the class of model uncertainty weconsider in this work and investigates its impact on thecontrol system, and concludes with motivation for a data-driven approach to mitigate this impact.

A. Uncertainty Modeling Assumptions

As defined in Section II, we consider affine robotic controlsystems that evolve under dynamics described by (1). Inpractice, we do not know the dynamics of the system exactly,and instead develop our control systems using the estimatedmodel:

D̂(q)q̈+ Ĉ(q, q̇)q̇+ Ĝ(q)︸ ︷︷ ︸Ĥ(q,q̇)

= B̂u. (14)

We assume the estimated model (14) satisfies the relativedegree condition on the domain R, and thus may use themethod of input-output linearization to produce a Control

Lyapunov Function (CLF), V , for the system. Using the def-initions established in (2) in conjunction with the estimatedmodel, we see that true system evolves as:

η̇ = f̂(q, q̇)− ṙ(t) + ĝ(q)u+ (g(q)− ĝ(q)︸ ︷︷ ︸

A(q)

)u+ f(q, q̇)− f̂(q, q̇)︸ ︷︷ ︸b(q,q̇)

. (15)

We note the following features of modeling uncertainty inthis fashion:• Uncertainty is allowed to enter the system dynamics

via parametric error as well as through completelyunmodeled dynamics. In particular, the function H cancapture a wide variety of nonlinear behavior and onlyneeds to be Lipschitz continuous.

• This formulation explicitly allows uncertainty in howthe input is introduced into the dynamics via uncertaintyin the inertia matrix D and static actuation matrix B.This definition of uncertainty is also compatible with adynamic actuation matrix B : Q× Rn → Rn×m givenproper assumptions on the relative degree of the system.

Given this formulation of our uncertainty, we make thefollowing assumptions of the true dynamics:

Assumption 1. The true system is assumed to be determin-istic, time invariant, and affine in the control input.

Assumption 2. The CLF V , formulated for the estimatedmodel, is a CLF for the true system.

It is sufficient to assume that the true system have relativedegree 2 on the domain R to satisfy Assumption 2. Thisholds since the true values of f̃ and g̃, if known, enablechoosing control inputs as in (6) that respect the same linearerror dynamics (8). Given that V is a CLF for the truesystem, its time derivative under uncertainty is given by:

V̇ (η,u) =

̂̇V (η,u)︷ ︸︸ ︷∂V

∂η(f̂(q, q̇)− ṙ(t) + ĝ(q)u)

+∂V

∂ηA(q)︸ ︷︷ ︸

a(η,q)>

u+∂V

∂ηb(q, q̇)︸ ︷︷ ︸

b(η,q,q̇)

, (16)

for all η ∈ R2k and u ∈ U . While V is a CLF for the truesystem, it is no longer possible to determine if a specificcontrol value will satisfy the derivative condition in (11) dueto the unknown components a and b. Rather than form a newLyapunov function, we seek to better estimate the Lyapunovfunction derivative V̇ to enable control selection that satisfiesthe exponential stability requirement. This estimate should beaffine in the control input, enabling its use in the controllerdescribed in (13). Instead of learning the unknown dynamicsterms A and b, which scale with both the dimension ofthe configuration space and the number of inputs, we willlearn the terms a and b, which scale only with the numberof inputs. In the case of the planar Segway model wesimulate, we reduce the number of learned components from

4 to 2 (assuming kinematics are known). These learnedrepresentations need to accurately capture the uncertaintyover the domain in which the system is desired to evolveto ensure stability during operation.

B. Motivating a Data-Driven Learning Approach

The formulation from (15) and (16) defines a general classof dynamics uncertainty. It is natural to consider a data-driven method to estimate the unknown quantities a and bover the domain of the system. To motivate our learning-based framework, first consider a simple approach of learninga and b via supervised regression [19]: we operate the systemusing some given state-feedback controller to gather datapoints along the system’s evolution and learn a function thatapproximates a and b via supervised learning.

Concretely, let q0 ∈ Q be an initial configuration. Anexperiment is defined as the evolution of the system over afinite time interval from the initial condition (q0,0) usinga discrete-time implementation of the given controller. Aresulting discrete-time state history is obtained, which is thentransformed with Lyapunov function V and finally differen-tiated numerically to estimate V̇ throughout the experiment.This yields a data set comprised of input-output pairs:

D = {((qi, q̇i,ηi,ui), V̇i)}Ni=1 ⊆ (Q×Rn×R2k×U)×R.(17)

Consider a class Ha of nonlinear functions mapping fromR2k × Q to Rm and a class Hb of nonlinear functionsmapping from R2k × Q × Rn to R. For a given â ∈ Haand b̂ ∈ Hb, define ̂̇W as:̂̇W (η,q, q̇,u) = ̂̇V (η,u) + â(η,q)>u+ b̂(η,q, q̇), (18)and let H be the class of all such estimators mapping R2k×Q×Rn×U to R. Defining a loss function L : R×R→ R+,the supervised regression task is then to find a function inH via empirical risk minimization (ERM):

infâ∈Hab̂∈Hb

1

N

N∑i=1

L(̂̇W (ηi,qi, q̇i,ui), V̇i). (19)This experiment protocol can be executed either in simulationor directly on hardware. While being simple to implement,supervised learning critically assumes independently andidentically distributed (i.i.d) training data. Each experimentviolates this assumption, as the regression target of each datapoint is coupled with the input data of the next time step. Asa consequence, standard supervised learning with sequential,non-i.i.d data collection often leads to error cascades [24].

IV. INTEGRATING EPISODIC LEARNING & CLFS

In this section we present the main contribution of thiswork: an episodic learning algorithm that captures the un-certainty present in the Lyapunov function derivative in alearned model and utilizes it in a QP based controller.

Algorithm 1 Dataset Aggregation for Control LyapunovFunctions (DaCLyF)Require: Control Lyapunov Function V , derivative esti-

mate ̂̇V 0, model classes Ha and Hb, loss function L,set of initial configurations Q0, nominal state-feedbackcontroller k0, number of experiments T , sequence of trustcoefficients 0 ≤ w1 ≤ · · · ≤ wT ≤ 1

D = ∅ . Initialize data setfor k = 1, . . . , T do

(q0,0)← sample(Q0 × {0}) . Get initial conditionDk ← experiment((q0,0),kk−1) . Run experimentD ← D ∪Dk . Aggregate data setâ, b̂← ERM(Ha,Hb,L, D, ̂̇V 0) . Fit estimatorŝ̇V k ← ̂̇V 0 + â>u+ b̂ . Update derivative estimatorkk ← k0 +wk · augment(k0, ̂̇V k) . Update controller

end forreturn ̂̇V T ,uT

A. Episodic Learning Framework

Episodic learning refers to learning procedures that itera-tively alternates between executing an intermediate controller(also known as a roll-out in reinforcement learning [22]),collecting data from that roll-out, and designing a newcontroller using the newly collected data. Our approachintegrates learning a and b with improving the performanceand stability of the control policy u in such an iterativefashion. First, assume we are given a nominal state-feedbackcontroller k : Q×Rn×I → U , which may not stabilize thesystem. With an estimator ̂̇W ∈ H as defined in (18), wespecify an augmenting controller as:

k′(q, q̇, t) = argminu′∈Rm

J(u′)

s.t. ̂̇W (η,q, q̇,k(q, q̇, t) + u′) ≤ −c3 ‖η‖22k(q, q̇, t) + u′ ∈ U , (20)

where J : Rm → R is any positive semi-definite quadraticcost function. This augmenting control law effectively findsthe minimal addition u′ to the input determined by the nom-inal control law k such that the sum stabilizes the system;however, stability degrades with error in the estimator ̂̇W .

In an effort to reduce the remaining error, we use this newcontroller to obtain better estimates of a and b. One option,as seen in Section III-B, is to perform experiments and useconventional supervised regression to update â and b̂. Toovercome the limitations of conventional supervised learning,we leverage reduction techniques: a sequential predictionproblem is reduced to a sequence of supervised learningproblems over multiple episodes [15], [32]. In particular,in each episode, an experiment generates data using adifferent controller. The data set is aggregated and a newERM problem is solved after each episode. Our episodiclearning implementation is inspired by the Data Aggregationalgorithm (DAgger) [32], with some key differences:

t = 0 t = 1 t = 2 t = 3 t = 4 t = 5

Fig. 2. (Left) Model based QP controller fails to track trajectory. (Right) Improvement in angle tracking of system with augmented controller over nominalPD controller. (Bottom) Corresponding visualizations of state data. Note that Segway is tilted in the incorrect direction at the end of the QP controllersimulation, but is correctly aligned during the augmented controller simulation. Video of this animation is found at https://youtu.be/cB5MY_8vCrQ.

• DAgger is a model-free policy learning algorithm,which trains a policy directly in each episode usingoptimal computational oracles. Our algorithm definesa controller indirectly via a CLF to ensure stability.

• The ERM problem is underdetermined, i.e., differentapproximations (â, b̂) may achieve similar loss for agiven data set while failing to accurately model a andb. This potentially introduces error in estimating V̇for control inputs not reflected in the training data,and necessitates the use of exploratory control actionto constrain the estimators â and b̂. Such explorationcan be achieved by randomly perturbing the controllerused in an experiment at each time step. This needfor exploration is an analog to the notion of persistentexcitation from adaptive systems [28].

Algorithm 1 specifies a method of computing a sequenceof Lyapunov function derivative estimates and augmentingcontrollers. During each episode, the augmenting controllerassociated with the estimate of the Lyapunov function deriva-tive is scaled by a heuristically chosen factor reflecting trustin the estimate and added to the nominal controller for usein the subsequent experiment. The trust coefficients forma monotonically non-decreasing sequence on the interval[0, 1]. Importantly, this experiment need not take place insimulation; the same procedure may be executed directly onhardware. It may be infeasible to choose a specific configura-tion for an initial condition on a hardware platform; thereforewe specify a set of initial configurations Q0 ⊆ Q from whichan initial condition may be sampled, potentially randomly.At a high level, this episodic approach makes progress by

gathering more data in relevant regions of the state space,such as states close to a target trajectory. This extends thegeneralizability of the estimator in its use by subsequentcontrollers, and improves stability results as explored in [43].

B. Additional Controller Details

During augmentation, we specify the controller in (20) byselecting the minimum-norm cost function:

J(u′) =1

2‖k(q, q̇, t) + u′‖22 , (21)

for all u′ ∈ Rm, q ∈ Q, q̇ ∈ Rn, and t ∈ I. For practicalconsiderations we incorporate the following smoothing terminto the cost:

R(u′) = R ‖u′ − uprev‖2

2 ,

for all u′ ∈ Rm, where uprev ∈ Rm is the previouslycomputed augmenting controller and R > 0. This is doneto avoid chatter that may arise from the optimization basednature of the CLF-QP formulation [27].

Note that for this choice of Lyapunov function, the gra-dient ∂V∂η , and therefore a, approach 0 as η approaches0, which occurs close to the desired trajectory. While theestimated Lyapunov function derivative may be fit with lowabsolute error on the data set, the relative error may stillbe high for states near the desired trajectory. Such relativeerror causes the optimization problem in (20) to be poorlyconditioned near the desired trajectory. We therefore add aslack term δ ∈ R+ to the decision variables, which appearsin the inequality constraint [2]. The slack term is additionally

https://youtu.be/cB5MY_8vCrQ

(rad)

Fig. 3. Augmenting controllers consistently improve trajectory tracking across episodes. 10 instances of the algorithm are executed with the shaded regionformed from minimum and maximum angles for each time step within an episode. The corresponding average angle trajectories are also displayed.

incorporated into the cost as:

C(δ) =1

2C

∥∥∥∥∥(∂V

∂ηĝ(q)

)>+ â(η,q)

∥∥∥∥∥2

2

δ2, (22)

for all δ ∈ R+, where C > 0. As states approach thetrajectory, the coefficient of the quadratic term decreasesand enables relaxation of the exponential stability inequalityconstraint. In practice this leads to input-to-state stablebehavior, described in [40], around the trajectory.

The exploratory control during experiments is naively cho-sen as additive noise from a centered uniform distribution,with each coordinate drawn i.i.d. The variance is scaled bythe norm of the underlying controller to introduce explorationwhile maintaining a high signal-to-noise ratio.

V. APPLICATION ON SEGWAY PLATFORM

In this section we apply the episodic learning algorithmconstructed in Section IV to the Segway platform. In particu-lar, we consider a 4-dimensional planar Segway model basedon the simulation model in [18]. The system states consist ofhorizontal position and velocity, pitch angle, and pitch anglerate. Control is specified as a single voltage input supplied toboth motors. The parameters of the model (including mass,inertias, and motor parameters but excluding gravity) arerandomly modified by up to 10% of their nominal valuesand are fixed for the simulations.

We seek to track a pitch angle trajectory2 generated forthe estimated model. The nominal controller is a linearproportional-derivative (PD) controller on angle and anglerate error. 20 experiments are conducted with trust valuesvarying from 0.01 to 0.99 in a sigmoid fashion. The ex-ploratory control is drawn uniformly at random between−20% and 20% of the norm of the underlying controller foran episode for the first 10 episodes. The percentages decaylinearly to 0 in the remaining 10 episodes. The model classesselected are sets of two-layer neural networks with ReLUnonlinearities with hidden layer width of 2000 nodes3. Theinputs to both models are all states and the CLF gradient.

Failure of the controller (13) designed for the estimatedmodel to track the desired trajectory is seen in the left portionof Fig. 2. The baseline PD controller and the augmented

2Trajectory was generated using the GPOPS-II Optimal Control Software3Models were implemented in Keras

controller after 20 experiments can be seen in the rightportion Fig. 2. Corresponding visualizations of the Segwaystates are displayed at the bottom of Fig. 2. The augmentedcontroller exhibits a notable improvement over the model-based and PD controller in tracking the trajectory.

To verify the robustness of the learning algorithm, the20 experiment process was conducted 10 times. After eachexperiment the intermediate augmented controller was testedwithout exploratory perturbations. For the last three ex-periments and a test of the final augmented controller,the minimum, mean, and maximum angles across all 10instances are displayed for each time step in Fig. 3. Themean trajectory consistently improves in these later episodesas the trust factor increases. The variation increases butremains small, indicating that the learning problem is robustto randomness in the initialization of the neural networks,in the network training algorithm, and in the noise addedduring the experiments. The performance of the controllerin the earlier episodes displayed negligible variation fromthe baseline PD controller due to small trust factors.

VI. CONCLUSIONS & FUTURE WORK

We presented an episodic learning framework that directlyintegrates into an established method of nonlinear controlusing CLFs. Our method allows for the effects of bothparametric error and unmodeled dynamics to be learned fromexperimental data and incorporated into an QP controller.The success of this approach was demonstrated in simulationon a Segway, showing improvement upon a model estimatebased controller.

There are two main interesting directions for future work.First, a more thorough investigation of episodic learning al-gorithms can yield superior performance as well as learning-theoretic converge guarantees. Other episodic learning ap-proaches to consider include SEARN [13], AggreVaTeD[41], and MoBIL [10], amongst others. Second, our approachcan also be applied to learning with other forms of guar-antees, such as with Control Barrier Functions (CBFs) [3].Existing work on learning CBFs are restricted to learningwith Gaussian processes [44], [16], [11], and also learn overthe full state space rather than over the low-dimensionalprojection onto the CBF time derivative.

Acknowledgements. This work was supported in part byfunding and gifts from DARPA, Intel, PIMCO, and Google.

REFERENCES

[1] A. Ames, K. Galloway, K. Sreenath, and J. Grizzle. Rapidly exponen-tially stabilizing control lyapunov functions and hybrid zero dynamics.IEEE Transactions on Automatic Control, 59(4):876–891, 2014.

[2] A. Ames and M. Powell. Towards the unification of locomotionand manipulation through control lyapunov functions and quadraticprograms. In Control of Cyber-Physical Systems, pages 219–240.Springer, 2013.

[3] A. Ames, X. Xu, J. Grizzle, and P. Tabuada. Control barrierfunction based quadratic programs for safety critical systems. IEEETransactions on Automatic Control, 62(8):3861–3876, 2017.

[4] Z. Artstein. Stabilization with relaxed controls. Nonlinear Analysis:Theory, Methods & Applications, 7(11):1163–1173, 1983.

[5] T. Beckers, D. Kulić, and S. Hirche. Stable gaussian process basedtracking control of euler–lagrange systems. Automatica, 103:390–397,2019.

[6] F. Berkenkamp, R. Moriconi, A. Schoellig, and A. Krause. Safelearning of regions of attraction for uncertain, nonlinear systems withgaussian processes. In 55th Conference on Decision and Control(CDC), pages 4661–4666. IEEE, 2016.

[7] F. Berkenkamp and A. Schoellig. Safe and robust learning control withgaussian processes. In 2015 European Control Conference (ECC),pages 2496–2501. IEEE, 2015.

[8] F. Berkenkamp, A. Schoellig, and A. Krause. Safe controller op-timization for quadrotors with gaussian processes. In 2016 IEEEInternational Conference on Robotics and Automation (ICRA), pages491–496. IEEE, 2016.

[9] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safemodel-based reinforcement learning with stability guarantees. InI. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish-wanathan, and R. Garnett, editors, Advances in Neural InformationProcessing Systems 30, pages 908–918. Curran Associates, Inc., 2017.

[10] C. Cheng, X. Yan, E. Theodorou, and B. Boots. Accelerating imitationlearning with predictive models. In International Conference onArtificial Intelligence and Statistics (AISTATS), 2019.

[11] R. Cheng, G. Orosz, R.M. Murray, and J.W. Burdick. End-to-endsafe reinforcement learning through barrier functions for safety-criticalcontinuous control tasks. 2019.

[12] Y. Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh. Alyapunov-based approach to safe reinforcement learning. In S. Bengio,H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar-nett, editors, Advances in Neural Information Processing Systems 31,pages 8092–8101. Curran Associates, Inc., 2018.

[13] H. Daumé, J. Langford, and D. Marcu. Search-based structuredprediction. Machine learning, 75(3):297–325, 2009.

[14] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel.Benchmarking deep reinforcement learning for continuous control.In International Conference on Machine Learning, pages 1329–1338,2016.

[15] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch modereinforcement learning. Journal of Machine Learning Research,6(Apr):503–556, 2005.

[16] J. Fisac, A. Akametalu, M. Zeilinger, S. Kaynama, J. Gillula, andC. Tomlin. A general safety framework for learning-based control inuncertain robotic systems. IEEE Transactions on Automatic Control,2018.

[17] K. Galloway, K. Sreenath, A. Ames, and J. Grizzle. Torque saturationin bipedal robotic walking through control lyapunov function-basedquadratic programs. IEEE Access, 3:323–332, 2015.

[18] T. Gurriet, A. Singletary, J. Reher, L. Ciarletta, E. Feron, and A. Ames.Towards a framework for realizable safety critical control through ac-tive set invariance. In Proceedings of the 9th ACM/IEEE InternationalConference on Cyber-Physical Systems, pages 98–106. IEEE Press,2018.

[19] L. Györfi, M. Kohler, A. Krzyzak, and H. Walk. A distribution-freetheory of nonparametric regression. Springer Science & BusinessMedia, 2006.

[20] I. Karafyllis, M. Kontorinaki, and M. Krstic. Adaptive control byregulation-triggered batch least-squares estimation of non-observableparameters. arXiv preprint arXiv:1811.10833, 2018.

[21] H.K. Khalil. Nonlinear Systems - 3rd Edition. PH, Upper SaddleRiver, NJ, 2002.

[22] J. Kober, J.A. Bagnell, and J. Peters. Reinforcement learning inrobotics: A survey. The International Journal of Robotics Research,32(11):1238–1274, 2013.

[23] M. Krstić and P. Kokotović. Control lyapunov functions for adaptivenonlinear stabilization. Systems & Control Letters, 26(1):17–23, 1995.

[24] H. Le, A. Kang, Y. Yue, and P. Carr. Smooth imitation learning foronline sequence prediction. In Proceedings of the 33rd InternationalConference on International Conference on Machine Learning-Volume48, pages 680–688. JMLR. org, 2016.

[25] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver,and D. Wierstra. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971, 2015.

[26] W. Ma, S. Kolathaya, E. Ambrose, C. Hubicki, and A. Ames. Bipedalrobotic running with durus-2d: Bridging the gap between theory andexperiment. In Proceedings of the 20th International Conference onHybrid Systems: Computation and Control, pages 265–274. ACM,2017.

[27] B. Morris, M. Powell, and A. Ames. Sufficient conditions for thelipschitz continuity of qp-based multi-objective control of humanoidrobots. In 52nd IEEE Conference on Decision and Control, pages2920–2926. IEEE, 2013.

[28] K. Narendra and A. Annaswamy. Persistent excitation in adaptivesystems. International Journal of Control, 45(1):127–160, 1987.

[29] Q. Nguyen and K. Sreenath. Optimal robust control for bipedalrobots through control lyapunov function based quadratic programs.In Robotics: Science and Systems, 2015.

[30] H. Ravanbakhsh and S. Sankaranarayanan. Learning lyapunov (po-tential) functions from counterexamples and demonstrations. arXivpreprint arXiv:1705.09619, 2017.

[31] S. Richards, F. Berkenkamp, and A. Krause. The lyapunov neuralnetwork: Adaptive stability certification for safe learning of dynamicsystems. arXiv preprint arXiv:1808.00924, 2018.

[32] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learningand structured prediction to no-regret online learning. In Proceedingsof the fourteenth international conference on artificial intelligence andstatistics, pages 627–635, 2011.

[33] S. Ross, G. Gordon, and J.A Bagnell. No-regret reductions forimitation learning and structured prediction. CoRR, abs/1011.0686,2010.

[34] S. S. Sastry. Nonlinear Systems: Analysis, Stability and Control.Springer, NY, 1999.

[35] S. Schaal and C. Atkeson. Learning control in robotics. IEEE Robotics& Automation Magazine, 17(2):20–29, 2010.

[36] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Prox-imal policy optimization algorithms. arXiv preprint arXiv:1707.06347,2017.

[37] G. Shi, X. Shi, M. O’Connell, R. Yu, K. Azizzadenesheli, A. Anand-kumar, Y. Yue, and S.J. Chung. Neural lander: Stable drone landingcontrol using learned dynamics. In IEEE International Conference onRobotics and Automation (ICRA), 2019.

[38] E. Sontag. A ’universal’ contruction of Artstein’s theorem on nonlinearstabilization. Systems & Control Letters, 13:117–123, 1989.

[39] E. Sontag. Input to state stability: Basic concepts and results. InNonlinear and optimal control theory, pages 163–220. Springer, 2008.

[40] E. Sontag and Y. Wang. On characterizations of the input-to-statestability property. Systems & Control Letters, 24(5):351–359, 1995.

[41] W. Sun, A. Venkatraman, G. Gordon, B. Boots, and A. Bagnell.Deeply aggrevated: Differentiable imitation learning for sequentialprediction. In Proceedings of the 34th International Conference onMachine Learning-Volume 70, pages 3309–3318. JMLR. org, 2017.

[42] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bo-hez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion forquadruped robots. arXiv preprint arXiv:1804.10332, 2018.

[43] A. Taylor, V. Dorobantu, M. Krishnamoorthy, H. Le, Y. Yue, andA. Ames. A control lyapunov perspective on episodic learning viaprojection to state stability. 58th Conference on Decision & Control(CDC), 2019.

[44] L. Wang, E. Theodorou, and M. Egerstedt. Safe learning of quadrotordynamics using barrier certificates. In 2018 IEEE InternationalConference on Robotics and Automation (ICRA), pages 2460–2465.IEEE, 2018.

[45] E. R. Westervelt, J. W. Grizzle, C. Chevallereau, J. H. Choi, andB. Morris. Feedback Control of Dynamic Bipedal Robot Locomotion.Taylor & Francis/CRC Press, 2007.

INTRODUCTIONPRELIMINARIES ON CLFsInput-Output LinearizationControl Lyapunov Functions

UNCERTAINTY MODELS & LEARNINGUncertainty Modeling AssumptionsMotivating a Data-Driven Learning Approach

INTEGRATING EPISODIC LEARNING & CLFsEpisodic Learning FrameworkAdditional Controller Details

APPLICATION ON SEGWAY PLATFORMCONCLUSIONS & FUTURE WORKReferences