1

Aggregated Electric Vehicle Charging Control forPower Grid Ancillary Service Provision

Lily BuechlerDepartment of Mechanical Engineering, Stanford University

ebuech@stanford.edu

Abstract—The increasing adoption of electric vehicles (EVs)in residential applications will require coordinated chargingstrategies to avoid operational challenges in power distributionsystems. Coordinated charging can also be used to effectivelyprovide grid operators with ancillary services such as rampingand load following. This project developed a framework utilizingfitted Q-iteration, a batch reinforcement learning method, forcharging control of an aggregation of electric vehicles to providegrid services and to minimize the cost of power consumptionunder time-of-use (TOU) rates. The problem was formulated asa Markov Decision Process such that computational costs do notscale with the number of vehicles. A simulator was developedusing high resolution residential EV charging data that accountsfor stochastic variation in EV charging demand and daily avail-ability patterns. Results indicate that 20-40 days of simulation arerequired to achieve good performance. Performance is also highlydependent on the appropriate selection of a feasible referencetrajectory for ancillary service provision. Small modifications tothe reward function allowed for significant improvements in costsavings by shifting charging patterns.

Index Terms—Fitted Q-Iteration, MDPs, Electric Vehicles,Batch Reinforcement Learning, Demand Response

I. INTRODUCTION

Reliable power grid operation requires that power gen-eration and demand are always in balance. Under currentoperational procedures, power grid operators account for real-time discrepancies between supply and demand through theprovision of ancillary services. Ancillary services, or operatingreserves, are power generation capacity that can be dispatchedfor load shifting, ramping, frequency regulation or other gridservices [1]. Large centralized generators typically providethese services, by increasing or decreasing their power outputin response to signals from the grid. However, future inte-gration of distributed energy resources (e.g. solar, wind) willintroduce significantly more uncertainty into power systemoperations. Decentralized control strategies that coordinatedistributed energy resources to provide ancillary services mayhelp address these challenges [2].

Electric vehicles (EVs) are projected to make up 55% ofall new automobile sales worldwide by 2040 [3]. This couldsignificantly increase residential demand and pose significantoperational challenges for distribution system operation if noteffectively managed. However, EVs often have significant loadflexibility, such that charging can be shifted in time to optimizefor some control objective.

A. Objective

This project focuses on using batch reinforcement learningto control of an aggregation of EVs in residential applica-tions for cost minimization and ancillary service provision,specifically ramping and load following. Provision of theseservices requires that an aggregation of loads track a time-varying power trajectory with a response timescale on theorder of minutes or hours. EV charging control is formulatedas a Markov Decision Process (MDP) that accounts for thestochastic nature of EV charging demand. An objective ofthis project is to develop a formulation that is tractable for alarge aggregation of EVs with heterogeneous charging demandpatterns, such that the computation does not scale with thenumber of loads.

B. Relevant Literature

Various studies have utilized MDPs for reinforcement learn-ing or planning in demand response or load control appli-cations [4]–[12]. These papers focus on different load types,number of loads, state and action space formulations, load con-trol objectives (e.g. cost minimization, load shifting), rewardfunctions, and solution methods. Brief highlights of relevantstudies are included below, and a comprehensive review isprovided in [4].

For EVs, Vandael et al. utilize fitted Q-iteration to controla fleet of EVs to follow a day-ahead schedule using a singleMDP [5]. Walraven and Spaan developed a group-based multi-agent MDP for optimal charging of an aggregation of EVs tomatch stochastic wind power generation [6]. However, theirformulation does not account for uncertainty in EV chargingdemand.

Wen et al. utilize Q-learning with Boltzman exploration tosolve an MDP for optimal demand response control for bothdeferrable and thermostatically controlled residential loads ina single home [7]. They characterize the state space as afunction of the on/off status of each load and define the rewardfunction as weighted sum of the cost of power consumptionand disutility of delaying device operation. A similar analysisfor deferrable loads is provided in [8] with a slightly differentstate-space characterization and disutility function. Kara et al.utilize Q-learning to control an aggregation of thermostaticallycontrolled loads (TCLs) for ancillary service provision. Theycharacterize the state space as a probability mass function overthe individual temperatures of the loads [9]. In [10], Ruelenset al. utilize fitted Q-iteration with Boltzman exploration to

2

control a water heater, while using an autoencoder to reducethe dimensionality of the state space and estimating the Q-function using an ensemble of extremely randomized trees.They apply a similar approach in [11] for control of a waterheater and heat pump using ε-greedy exploration. In [12], theyapply fitted Q-iteration to an aggregation of heterogeneouswater heaters, defining the action as a clearing priority overthe bid functions of individual water heaters that are definedin terms of the water heater state of charge.

II. METHODS

A. Simulation

A simulator was developed in Python using 1 year ofhistorical residential EV charging data from 2016 with 1-minute resolution for 59 homes in Austin TX from the PecanStreet database [13]. Most of the charging stations in thedataset were level 2 chargers with approximate power ratingsof 3.3 kW. For simplicity, it was assumed that all EVs havea constant charging rate of 3 kW in the simulation. Thecharging start and end times were extracted from the data usingthresholding heuristics. From the duration of each chargingepisode, the initial state of charge was inferred by assumingeach EV is fully charged at the end of each session. Theseobserved charging patterns were then set as the chargingrequirements during each period of time each EV is connectedto the grid in the simulation.

The primary sources of uncertainty in this problem are thearrival and departure time of the EVs and the initial stateof charge of the EV batteries. In general, these variablesare highly stochastic, given variation in consumer occupancyand travel patterns. In the simulator, it was assumed thateach consumer arrives home in the evening and departs inthe morning. Each departure and arrival time was drawnfrom a normal distribution (N (µd, 1 hour) and N (µa, 1 hour)respectively), where the mean arrival and departure timesfor each consumer are also drawn from normal distributions(µd ∼ N (8:00, 1.5 hours) and µa ∼ N (18:00, 1.5 hours)respectively). Daytime charging availability was defined bysetting the EV arrival times equal to the EV charging starttimes observed in the data and assuming the duration of avail-ability is drawn from a normal distribution (N (3, 1)). Chargingdata for individual homes from the Pecan Street database wasduplicated and shifted in time to generate synthetic data forlarger aggregations of vehicles. Data was shifted by 1 weekto preserve daily, weekly and seasonal EV charging patterns.

B. Simulation Scenarios

Under current grid operational procedures, resources areonly occasionally called upon to provide ancillary services.However, in future power systems with high penetrations ofvariable renewable generation, reserve requirements may sig-nificantly increase to compensate the increased power supplyuncertainty. When ancillary services are not required, theobjective of an aggregator would be to minimize electricitycosts for consumers. In this study, we consider three scenarios:

(1) ancillary service provision, where the objective is to contin-uously track a trajectory, (2) cost minimization, and (3) ramp-following during 4 hours of the day with cost minimizationduring the remaining time.

C. Markov Decision Process Formulation

A MDP is defined by the tuple(S,A, P (st+1|st, at), R(st, at), where S is a finite setof states, A is a finite set of actions, P (st+1|st, at) is a statetransition function, and R(st, at) is the immediate reward.Defining the state-space in terms of the state of chargeand availability of each individual vehicle and the actionspace in terms of the charging rate of each vehicle wouldlead to extremely high dimensional state and action spacesrespectively for large numbers of EVs. To keep the statespace small, the state is defined in terms of metrics evaluatedover the entire aggregation of EVs. Current EV chargingstations can measure the state of charge of the EV battery, soit is assumed that such information could be sent to an EVaggregator. Therefore the state is observable, given effectiveand reliable communication infrastructure. It is assumed thateach consumer specifies a desired charging completion timeat the beginning of each charging session and the chargingstrategy must explicitly obey those constraints. Given thestate of charge of the EV battery, the battery capacity, andthe power rating of the charging station, the time required fora full charge can be estimated with high accuracy. Let T (i,r)

t

be the charging time required for a full charge and T (i,a)t be

the time remaining until the desired completion time at timet for EV i. We define the flexibility f (i)t of EV i at time t as

f(i)t =

T

(i,t)t −T (i,a)

t

T(i,a)t

if EV i is available

1 otherwise(1)

This metric gives a measure of the amount of charging that isstill required for each charging session relative to the amountof time available for charging. F (x)

t , x ∈ [0, 100] is defined asthe x percentile flexibility and Ft as the mean flexibility of theaggregation of EVs. ht ∈ [0, . . . , 23] is the hour of the day,T

(s)t =

∑li=1 T

(i,r)t is the total amount of time required to

fully charge all of the EVs currently plugged in, n(min)t is theminimum number of EVs that must charge to meet chargingconstraints, and n(max)t is the total number of EVs available tocharge at time t. Based on preliminary results from differentstate space representations, we define the state at time t as

st = [ht, n(max)t , n

(min)t , T

(s)t , Ft, F

(10)t , F

(5)t , F

(2)t ]T (2)

The action is equal to the number of EVs that are chargedduring each control timestep. The action spaceAt is dependenton the state and is lower bounded by n

(min)t and upper

bounded by n(max)t . As a heuristic, EVs are charged in a

priority order defined by their flexibility f (i)t .Reward functions were defined for each of the three scenar-

ios. Let the power consumption trajectory of EV i over a timehorizon H be given by P (i) = [P

(i)1 , . . . , P

(i)H ]T ∈ RH and

the desired power trajectory P (ref) = [P(ref)1 , . . . , P

(ref)H ]T ∈

3

Fig. 1. (a) Reference trajectory for scenario 1 and ramp for scenario 3. (b)Time-varying TOU electricity price [14]

RH . Then the immediate reward at time t for scenario 1 isgiven by the squared tracking error:

rt = −

((l∑i=1

P(i)t

)− P (ref)

t

)2

(3)

For simplicity, the same reference trajectory was used foreach day of the simulation as shown in 1. This trajectory wasdesigned such that the total required daily energy consumptionis equal to the mean daily EV charging demand over thesimulation horizon.

For scenario 2, we assume the aggregation of EVs aresubject to a time-varying pricing schedule. For this study, thesummer weekday rate B residential EV time-of-use (TOU)pricing schedule from PG&E [14] was used. The time varyingelectricity price over a time horizon H is ρ = [ρ1, . . . , ρH ]T ∈RH such that the reward function for scenario 2 is given by

rt = −ρtl∑i=1

P(i)t (4)

Adding another term to the reward function that penalizeswaiting until the end of the charging window to charge eachEV was found to improve performance, as discussed in SectionIII:

rt = −ρtl∑i=1

P(i)t + µ

l∑i=1

f(i)t P

(i)t (5)

The reward function for scenario 3 is given by

rt =

−λ((∑l

i=1 P(i)t

)− P (ref)

t

)2if ts ≤ t ≤ te

−ρt∑l

i=1 P(i)t + µ

∑li=1 f

(i)t P

(i)t otherwise

(6)

which includes a negative squared tracking error term duringa ramping event from ts to te and the reward function for costminimization at all other times. For the results in the followingsection, the start and end times of the ramping event were setequal to ts = 19 : 00 and te = 23 : 00 respectively.

Given the stochastic nature of EV charging demand and theinitial state of charge of the EV battery upon arrival, the tran-sition function P(s,a) is not generally known. Therefore, theMDP must be solved using reinforcement learning methods.

Several model-free temporal difference learning methodsincluding Q-learning, Sarsa, and Sarsa Lambda were inves-tigated in preliminary tests using different control timesteps,

discretization schemes, learning rates, exploration strategiesand discount factors. In all cases the performance of themethod was relatively poor even after hundreds of thousands ofiterations, especially in less-explored regions of the state spaceand for large changes in the state variables (many EVs arrivingat once). Relevant literature also discusses issues with thesemethods for large state spaces with respect to convergence andvolatile changes in exogenous variables [5], [12]. These studiesalso indicate that batch reinforcement learning methods tendto achieve much better performance.

D. Fitted Q-Iteration

Based on preliminary results and literature review, the MDPwas solved using a version of fitted Q-iteration adapted from[15]. In the original version of fitted Q-iteration, the state-valuefunction

Q(s, a) = R(s, a) + γ∑s′P (s′|s, a)max

a′Q(s′, a′) (7)

is approximated from batches of tuples (st, st+1, rt, at)collected during simulation or real-time operation using su-pervised learning methods. In this study, the action-spaceis state-dependent and we use a control timestep δc thatis different from the simulation timestep δs. Therefore, weestimate Q(s, a) from tuples of the form (st, st+δc , rt, at,At).This adaptation of fitted Q-iteration is outlined in Algorithm1.

The inputs to the algorithm include a default policy πo, thecontrol timestep δc, the simulation timestep δs, the discountfactor γ ∈ [0, 1], the initial state so, the number of days ofsimulation D, and parameters that define the exploration policyc1 and c2.

Lines 4-23 of Algorithm 1 require simulation of the systemfor a single day d using the current approximation of the state-value function Qd. The default policy is only used to initializethe algorithm during the first day of simulation (line 7). Duringall future days, the upper and lower bounds on the actionspace are computed at each control timestep (line 10) and thecurrent action is chosen using the approximation of the state-value function and an exploration method (lines 11-12). Thesystem is forward-propagated in simulation at each simulationtimestep (lines 8, 13 and 20) and a tuple is added to the trainingset at each control timestep (lines 15 and 24). In lines 25-33, the state-value function is re-estimated with the updatedtraining set using a regression algorithm. By using regression,the state-value function can be generalized across the state andaction space. Based on literature review and preliminary tests,random forests were used for regression.

An ε-greedy exploration policy was utilized, where ε is anexponentially decreasing function of the batch number d:

ε = c1 e−c2∗d c1, c2 ∈ R (8)

Instead of randomly selecting an action from the entire actionspace, actions were selected from a restricted action space[a∗t − ∆a, . . . , a

∗t + ∆a], ∆a ∈ Z, where a∗t is the action

obtained from maximizing the state-value function.

4

Algorithm 1 Fitted Q-iteration adapted from [15]Require: πo, δc, δs, γ, Kmax, s0, D, c1, c2

1: initialize F ← ∅, T ← ∅, t← 02: for d = 0, . . . , D do3: m← 04: for m = 0, . . . , 1440/δs do5: if mod(t, δc) = 0 then6: if d = 0 then7: at ← πo(st)8: st+1, rt ← Simulate(st, at)9: else

10: At ← GetV alidActions(st)11: a∗t ← argmaxa∈At Qd(st, a)12: at ← Exploration(a∗t ,At, c1, c2)13: st+1, rt ← Simulate(st, at)14: if m 6= 0 then15: T ← T ∪ (st−δc , st, rt−δc , at−δc ,At−δc)16: end if17: end if18: else19: at ← at−120: st+1, rt ← Simulate(st, at)21: end if22: t← t+ 123: end for24: F ← F ∪ T25: set Q(0)

d+1 equal to 0 for all s,a26: for k = 1, . . . ,Kmax do27: for l = 1, . . . , |F| do28: il = (slt, a

lt)

29: ol = rlt + γmaxa∈At Q(k)d+1(slt+1, a)

30: end for31: Q

(k+1)d+1 ← Regress(i, o; ∀l = 1, . . . , |F|)

32: end for33: Qd+1 ← Q

(k+1)d+1

34: end for35: return QD+1

III. RESULTS AND DISCUSSION

The EV simulator and implementation of the MDP solutionalgorithm were implemented in Python 3.6. The Sci-kit Learnmachine learning package was used for implementation of therandom forest regression [16]. Simulations were performedfor 118 EVs with γ = 0.95, δc = 20 min, δs = 1 min, µ =0.1, c1 = −7.5, c2 = 192.5 ,and D = 150 days. For therandom forest regression, 100 trees were used with a maximumdepth of 10. In Equation 6, λ was set equal to 0.01 whichplaces significantly larger weight on ramp-following than costminimization. ∆a was set equal to 3 for scenarios 1 and 3 andequal to 13 for scenario 2, which required more exploration.

The expected discounted return at iteration k of the state-value function approximation in state s is defined as Jπ

∗k

k (s) =

maxa∈AQ(k)d+1(s, a). The convergence error for the iterative

Fig. 2. Convergence error for iterative approximation of the state-valuefunction for (a) scenario 1 (ancillary service provision) and (b) scenario 2(cost minimization) for each batch.

state-value function approximation is∑|F|l=1

(Jk+1(sl)− Jk(sl)

)2∑|F|l=1 (Jk(sl))

2(9)

The convergence errors for the first 20 days (batches) ofsimulation for scenarios 1 and 2 are shown in Figure 2 asa function of the number of iterations. For scenario 1, thestate-value function converges in approximately 30 iterations.The asymptotic value of the error increases slightly as thenumber of batches increases. For scenario 2, the state-valuefunction converges in approximately 40 iterations. Therefore,Kmax was set to 40 iterations for all of the following results.

The 10 day running average of the immediate reward forscenario 1 is shown in Figure 3a for the first 140 days ofsimulation. As shown, the performance improves for approx-imately the first 20 batches of the algorithm. Results showthat performance of the algorithm after the first 20 batches ishighly dependent on the feasibility of the reference trajectory,resulting in the observed fluctuations. Daylight savings timestarts on day 72, which likely explains one of the negativefluctuations in performance since the hour is one componentof the state. An example aggregate load profile for a single dayis shown in Figure 3b. On this specific day, overnight chargingdemand and midday charging demand (12:00 - 17:00) wererespectively slightly lower and higher than what was requiredby the reference trajectory. Daily charging requirements andthe arrival and departure patterns can vary considerably fromone day to another, such that it may be impossible to find areference trajectory that can be always be accurately followed.One possible solution is to use timeseries forecasting to predictthe day-ahead charging profile and to have the grid operatorconstruct the reference trajectory based on this prediction. This

5

Fig. 3. Scenario 1 (ancillary service provision/trajectory tracking) results: (a)10 day running average of rt (b) An example aggregate daily charging profileobtained from fitted Q-iteration.

would require incorporating the reference trajectory into thestate-space definition such that different trajectories could befollowed on different days. Kara et al. address this issue differ-ently, by learning different state-value functions for differentreference trajectories [9]. However, in practice there are likelyto be a very large number of possible reference trajectoriesprovided by a grid operator, which might make this approachinfeasible.

For scenario 2 (cost minimization), using Equation 4 forthe reward function was not effective in achieving goodperformance. This is because the piecewise constant structureof TOU price schedules makes it difficult to propagate rewardthrough time and shift charging over large numbers of controltimesteps. As shown in Figure 4b, fitted Q-iteration tends toarrive at a policy where most EV charging occurs early inthe morning during low prices, just before the middle tierprice begins. However, if EVs are not completely charged bythis time due to higher than average demand, some chargingmust occur at higher prices. The optimal strategy would beto begin charging at the beginning of the low-price period,but the algorithm is slow to learn such a strategy since theinstantaneous reward is constant for many control timesteps.The reward function given in Equation 5 imposes a penaltyon charging EVs at the end of the charging window, whichallows the algorithm to converge on the optimal policy muchmore quickly, as shown in Figure 4. Similar to scenario 1, theperformance of the algorithm fluctuates over the simulationhorizon due to variation in charging demand, especially during

Fig. 4. Scenario 2 (cost minimization) results for reward functions given inequations 4 and 5: (a) 10 day running average energy cost ($/kWh) (b) Anexample aggregate daily charging profile obtained from fitted Q-iteration.

high-priced hours.For scenario 3, performance improves for approximately

the first 40 days as the algorithm gains experience, beforeapproaching a steady-state value, as shown in Figure 5. Thereis generally not enough flexibility in charging demand to shiftload from the middle of the day (12:00-16:00) to time periodswith the lowest prices. By placing high relative weight onramp-following in the reward function in Equation 6, the rampsignal can be followed with as high accuracy as the controltimestep discretization allows. Charging generally increasessignificantly after the ramp ends to charge the EVs connectedto the grid during the low price period to achieve optimal costminimization.

IV. CONCLUSIONS

This project utilized fitted Q-iteration for charging controlof an aggregation of EVs to provide ancillary services to thepower grid and minimize the cost of power consumption withTOU prices. For the three scenarios considered, approximately20-40 days of experience were required to obtain maximumperformance, depending on the control objective. For contin-uous trajectory tracking, it was shown that performance ishighly dependent on the feasibility of the chosen referencetrajectory, which may vary considerably depending on dailycharging demand. Since learning a different optimal policyfor each possible reference trajectory is not efficient norpractical, future work could analyze methods for incorporatingthe reference trajectory into the state space definition, such that

6

Fig. 5. Scenario 3 (cost minimization) results : (a) daily average reward (b)An example aggregate daily charging profile obtained from fitted Q-iteration.

an algorithm could learn a single optimal policy, parameterizedin terms of the signal. When following a ramp signal for onlya few hours of the day, the selection of a feasible trajectory tofollow is less critical because charging can be shifted beforeand after the ramping event. For cost minimization, addingan additional term to the reward function to incentivize earliercharging causes the algorithm to more quickly find the optimalpolicy.

Depending on future regulations and control architecturesfor distributed energy resources in distribution systems, controldecisions may need to be made by each EV, rather than by acentralized aggregator. Therefore, future work may considermulti-agent approaches for addressing this problem. Futurework may also investigate methods to better learn optimalpolicies for non-stationary and stochastic environments.

REFERENCES

[1] E. Ela, M. Milligan, and B. Kirby, “Operating reserves and variablegeneration,” National Renewable Energy Laboratory (NREL), Golden,CO., Tech. Rep., 2011.

[2] A. Radovanovic, A. Ramesh, R. Koningstein, D. Fork, W. Weber,S. Kim, J. Schmalzried, J. Sastry, M. Dikovsky, K. Bozhkov et al.,“Powernet for distributed energy resource networks,” in Power andEnergy Society General Meeting (PESGM), 2016. IEEE, 2016, pp.1–5.

[3] “Electric vehicle outlook 2018,” Bloomberg New Energy Finance, Tech.Rep., 2018.

[4] J. R. Vazquez-Canteli and Z. Nagy, “Reinforcement learning for demandresponse: A review of algorithms and modeling techniques,” AppliedEnergy, vol. 235, pp. 1072–1089, 2019.

[5] S. Vandael, B. Claessens, D. Ernst, T. Holvoet, and G. Deconinck,“Reinforcement learning of heuristic ev fleet charging in a day-aheadelectricity market,” IEEE Transactions on Smart Grid, vol. 6, no. 4, pp.1795–1805, 2015.

[6] E. Walraven and M. T. Spaan, “Planning under uncertainty for aggre-gated electric vehicle charging with renewable energy supply.” in ECAI,2016, pp. 904–912.

[7] Z. Wen, D. ONeill, and H. Maei, “Optimal demand response usingdevice-based reinforcement learning,” IEEE Transactions on Smart Grid,vol. 6, no. 5, pp. 2312–2324, 2015.

[8] D. O’Neill, M. Levorato, A. Goldsmith, and U. Mitra, “Residentialdemand response using reinforcement learning,” in Smart Grid Commu-nications (SmartGridComm), 2010 First IEEE International Conferenceon. IEEE, 2010, pp. 409–414.

[9] E. C. Kara, M. Berges, B. Krogh, and S. Kar, “Using smart devices forsystem-level management and control in the smart grid: A reinforce-ment learning framework,” in Smart Grid Communications (SmartGrid-Comm), 2012 IEEE Third International Conference on. IEEE, 2012,pp. 85–90.

[10] F. Ruelens, B. Claessens, S. Quaiyum, B. De Schutter, R. Babuska, andR. Belmans, “Reinforcement learning applied to an electric water heater:From theory to practice,” IEEE Transactions on Smart Grid, 2016.

[11] F. Ruelens, B. J. Claessens, S. Vandael, B. De Schutter, R. Babuska,and R. Belmans, “Residential demand response of thermostaticallycontrolled loads using batch reinforcement learning,” IEEE Transactionson Smart Grid, vol. 8, no. 5, pp. 2149–2159, 2017.

[12] F. Ruelens, B. J. Claessens, S. Vandael, S. Iacovella, P. Vingerhoets, andR. Belmans, “Demand response of a heterogeneous cluster of electricwater heaters using batch reinforcement learning,” in Power SystemsComputation Conference (PSCC), 2014. IEEE, 2014, pp. 1–7.

[13] Pecan Street Inc. (2017) Dataport from pecan street. [Online]. Available:https://dataport.cloud/

[14] Pacific Gas and Electric Company. (2018) Electric schedule ev,residential time-of-use service for plug-in electric vehicle customoers.[Online]. Available: https://www.pge.com/tariffs/assets/pdf/tariffbook/ELEC SCHEDS EV%20(Sch).pdf

[15] D. Ernst, P. Geurts, and L. Wehenkel, “Tree-based batch mode rein-forcement learning,” Journal of Machine Learning Research, vol. 6, no.Apr, pp. 503–556, 2005.

[16] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion,O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vander-plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch-esnay, “Scikit-learn: Machine learning in Python,” Journal of MachineLearning Research, vol. 12, pp. 2825–2830, 2011.