quantifying uncertainty in vehicle simulation studiesthb/publications/geller.pdffirst step in...

INTRODUCTIONComputational modeling continues to play an increasing

role in the automotive design, development, and evaluationprocess. As vehicle technologies advance at a quick rate,researchers and manufacturers are challenged with not onlykeeping up with the state of the art, but also predicting andallowing for future design implementations. Computer basedsimulation plays an important role in supporting advancementof vehicle technology by assisting in systems engineeringdesign processes. As the level of detail included in vehiclemodels increases, so does the accuracy of the results; butcommonly at the cost of increased computational or systemdevelopment time.

Many modeling tools have been used to simulate a widerange of vehicle types, technologies, and operationalcharacteristics. Different objectives of these simulations cansupport different levels of detail and therefore acceptableuncertainty in the results. It is a requirement of the simulationend-user to account for the uncertainty that exists within thesystems considered and to understand how uncertainty willcontribute to the conclusions of any particular study.Simulations are commonly designed to represent a specificfunctional characteristic of the vehicles well, but cancommonly be misconstrued to represent a wider range of

operations than originally intended or validated. As thesesimulation tools continue to see more use in the academic andindustrial automotive design world, they are subjected tomore rigorous considerations and applications. The demandfor high level details is pushed by an increase in systemsengineering design methods that rely heavily on long designexplorations through computational based models.

Uncertainty exists in all simulations. The magnitude ofthis uncertainty must be considered in comparison to thebreadth of the results. A number of steps can be taken toevaluate an appropriate method for defining the uncertaintyand associating it correctly to the simulation outcomes. Thefirst step in determining uncertainty in a simulation is toclassify the type of study being performed. From the type ofstudy performed, objective outputs should be defined. Thecombination of study and objective type guides the studytowards a set of simulation tools that have been specificallydesigned for that application (whether they exist or not). Thesecond step of determining uncertainty is to define thesystems under consideration and their respective data flows(inputs and outputs). The third step requires a detailedevaluation of the equations, assumptions, and parametersimplemented in the simulation. The final step requires avalidation of the system relative to the type of system andstudy originally defined. Only after each of these steps have

2012-01-0506Published 04/16/2012

Copyright © 2012 SAE Internationaldoi:10.4271/2012-01-0506saepcmech.saejournals.org

Quantifying Uncertainty in Vehicle Simulation StudiesBenjamin Geller and Thomas Bradley

Colorado State University

ABSTRACTThe design of vehicles, particularly hybrid and other advanced technology vehicles, is typically complex and benefits

from systems engineering processes. Vehicle modeling and simulation have become increasingly important system designtools to improve the accuracy, repeatability, and flexibility of the design process. In developing vehicle computationalmodels and simulation, there is an inevitable compromise between the level of detail and the development/computationalcost. The tradeoff is specific to the requirements of each vehicle design effort. The assumptions and detail limitations usedfor vehicle simulations lead to a varying degree of result uncertainty for each design effort. This paper provides a literaturereview to investigate the state of the art vehicle simulation methods, and quantifies the uncertainty associated withcomponents that are commonly allocated uncertainty. By understanding the inaccuracies and inconsistencies within thesestudies, improved simulation methods can be proposed. The consequences or accuracy of common assumptions aredetermined which will aid future simulation efforts as well as provide metrics for evaluating the appropriateness of pastefforts. The results of this paper will aid future simulation design efforts and can begin to define standards by whichsimulation design studies are conducted.

CITATION: Geller, B. and Bradley, T., "Quantifying Uncertainty in Vehicle Simulation Studies," SAE Int. J. Passeng. Cars- Mech. Syst. 5(1):2012, doi:10.4271/2012-01-0506.

____________________________________

381

Downloaded from SAE International by Colorado State Univ, Monday, November 14, 2016

http://dx.doi.org/10.4271/2012-01-0506

http://saepcmech.saejournals.org/

been completed by the simulation developer and approved bythe simulation user can the uncertainty of the vehiclesimulation be accurately quantified. Each of these steps willbe discussed in further detail in the following sections.

PURPOSE OF DESIGN STUDIESThe first step in evaluating uncertainty in vehicle

simulation studies is to determine the type of study beingperformed. The type of study can most easily be classifiedbased on its purpose. Within each of the study types, adifferent set of considerations must be applied to theuncertainty characteristics. Simulation studies can beclassified into three main types:

1). Technical rankings2). Representations of futures3). System development and explorationAs a subset of each of the three simulations types listed,

simulations studies can be performed based on optimizationtechniques, design of experiments (DoE) parametric methods,or fixed-point formulations [1]. Optimization techniques caninclude a variety of algorithms ranging from linearprogramming to stochastic algorithms [2, 3]. Optimizationscommonly define an objective and perform simulationiterations to approach the objective within a specified set ofsolution requirements. DoE parametric methods operate asdesign space examination approaches that provide a uniformevaluation of a specified range of parameters, inputs, orassumptions. Optimizations commonly differ from DoEstudies as the number of simulation iterations increases,wherein optimizations continuously focus their designexplorations and DoE studies remain consistently distributed.Fixed-point formulation studies rely on one or a few pre-determined design space points and usually include muchfewer simulation iterations. Fixed-point studies mostcommonly apply to simulation of a pre-specified system withno design exploration.

Technical ranking (TR) studies consist of simulationefforts aiming to evaluate vehicle options in relation to oneanother. One or many objective evaluation metrics such asfuel economy, system efficiency, total cost of ownership(TCO), or greenhouse gas emissions (GHG), have commonlybeen used in previous vehicle simulation studies. As witheach of the three study types, TR can be performed as anoptimization, DoE, or fixed-point study. Optimization TRstudies consist of multiple independent optimization ofsystems such that optimal designs in different categories canbe compared. DoE TR studies are performed similarly tooptimization TR studies but with a more generalized designspace consideration. Fixed-point TR studies intend tocompare specific vehicle components or designs such ascomparing a specific conventional gasoline vehicle with itsmatching hybrid model. The TR studies can be particularlysensitive to parameter value specifications but less sensitiveto model structure. Details of these sensitivities and sourcesof uncertainty will be discussed in later sections.

Representations of futures (RoF) studies intend to providepredictions of future technology. These studies can exist in avariety of subsets including economic feasibility, technologylimits, technology goals, policy fulfillment, andenvironmental and social interaction to name a few [4, 5].RoF studies rely heavily on time sensitive predictions that areproposed to represent a projected future scenario. This type ofstudy usually exists with an initial base simulation for thestate of the art (SoA) technology and extends to a future time.Validation of RoF study can only accurately be performed astime progresses, but are commonly tested based on historicalcases. RoF studies are asserted to be most sensitive touncertainty in the assumptions made about future scenariosand definition of the technology SoA.

System development and exploration (SDE) studies aimto investigate the function of the vehicle or its subsystems.SDE studies can include such factors as controllerdevelopment strategies, component design specifications,trade-off analysis, and scenario implementation. The scenarioimplementation discussed in the SDE section differs fromscenarios from the RoF section as SDE scenarios are basedon an available operational environment test case (i.e.different drive cycles) and RoF scenarios are based in a futurecondition. SDE studies can commonly be associated withHardware in the Loop (HIL) development and testing. Modelstructure, including levels or detail, and equationspecification, are much more sensitive sources of uncertaintyfor SDE studies than in the other two study types discussed.

SIMULATION TOOLSSimulation studies can be performed using a number of

commercially available and custom vehicle simulation tools.In most cases, each specific tool has been created with theintent of fulfilling a design study type need, but there arealternative options for combining multiple tools ordeveloping a custom tool to meet study specifications. Themethods used to develop different simulation tools differ inmany ways including numerical solvers, direction ofinformation flow, level or detail, organizational structure, andsimulated system type. A few of the available simulationtools available as well as their background formulations aredetailed in this section.

A multitude of simulation tools are currently available fordesign studies. One of the primary differences between thesetools is the direction of information flow, or causality.Simulations for vehicles can exist in three configurations:forward facing, backward facing, or non-causal (acausal). Allthree of these simulation types operate in a time progressivemanner, the direction of information flow refers to datawithin the model.

Forward facing simulations of vehicle systems refers to amodel where controls and operation of the subsystemsoperate in a time-progressive feedback manner. For example,a forward facing vehicle simulation of an electric vehicledriving on a dynamometer schedule would follow the

Geller et al / SAE Int. J. Passeng. Cars - Mech. Syst. / Volume 5, Issue 1(May 2012)382


information path shown in Figure 1. A dynamometer drivecycle velocity demand is fed to a system driver that providesa desired torque or throttle demand to the controller. Thecontroller evaluates system limits and transmits the driverdemand to the propulsion unit. The propulsion unit suppliestractive effort based on its limited operating conditions aswell as calculation of resource/energy use. The resultingvehicle velocity is fed back to the driver and deviations canbe accounted for in future commands. Forward facingsimulations are generally representative of physical vehiclecontrol systems, and are commonly used for controlsdevelopment and HIL testing.

Figure 1. Forward facing simulation flow diagram

Backward facing simulations of vehicles have a similargeneral structure as forward facing simulations, but withdifferent information flows. In backward facing simulations,shown in Figure 2, it is assumed that a propulsive unit meetsdrive commands, and energy use can be calculated from therequired tractive effort. Backward facing simulations aretypically less computationally expensive than forward facingsimulations due to a lack of information feedback andcomplex controls. It is more difficult for backward facingsimulations to calculate maximum vehicle performance, suchas maximum acceleration, because the simulations are notdesigned to predict operation of the components at theirlimits.

Figure 2. Backward facing simulation flow diagram

Non-causal simulations use a combination of forward andbackward facing causality information flow. In some systemsthis can be implemented through switching calculation type.For example switching information can occur when abackward facing simulation reaches an event in the vehiclesimulation where a required component operation isunavailable (e.g. a motor reaches its peak torque). In this casethe drive cycle may not be met and continued simulation willrequire additional controller functionality to get the vehicle

operation back on track - which occurs in a forward facingmanner until normal operation resumes. An alternative non-causal simulation tool may have a combination of forwardand backward facing calculations simultaneously. Forexample, controllers send commands in a causal informationdirection, but power flows through an epicyclical gear(planetary) in multiple directions.

Each of the simulation tools available differs incalculation methods and considerations. Table 1 provides acompiled list of some of the simulation tools, the developerof the simulations, and a little information about thesimulation methods. Tools should be selected depending onthe type of simulation being performed. Additional aspects ofeach of the simulation tools and how they relate to theuncertainty of the vehicle simulations will be described inlater sections.

TYPES OF UNCERTAINTYUncertainty has classically been defined in many different

ways depending on the systems that provide and measure theuncertainty [23]. The primary focus of this section is tounderstand the uncertainty that exists in vehicle simulationsstudies. Vehicle simulation studies most directly relate touncertainties in simulation and computation methods, dataacquisition from physical systems, and equation formulation.Secondary sources of uncertainty such as environmentalrandom conditions, human error, and future forecasting mustalso be included but interact with the system at a higher leveland can be ignored for some studies.

Uncertainty in vehicle simulations has been classified intothree groups [12]:

1). Type-1 uncertainty: Variability of input orparameters. Type-1 uncertainty is usually handled byproviding distribution functions of the defined inputs andparameters when available.

2). Type-2 uncertainty: Similar to Type-1 uncertaintywhere variability exists in the inputs and parameters butwithout a known distribution. Fuzzy logic and evidencetheory have been used for solutions.

3). Type-3 uncertainty: Uncertainty from an unknownprocess. This type of uncertainty is the most difficult to findsolutions for.

Uncertainty can also be classified as either aleatory orepistemic [23]. Aleatory uncertainty pertains to informationthat can be represented by a distribution; epistemic refers tocompletely unknown factors. Type-1 uncertainty can beclassified as aleatory, Type-3 uncertainty can be classified asepistemic, with enough testing it is likely that Type-2uncertainty can also be described using a distribution of data;classifying it in the aleatory uncertainty.

Measuring error and uncertainty should include anunderstanding of the accuracy and precision of the data sets,wherein accuracy represents the measured difference betweena predicted and measured value, precisions compares the

Geller et al / SAE Int. J. Passeng. Cars - Mech. Syst. / Volume 5, Issue 1(May 2012) 383


distribution of the predicted and measured values. Modelsand simulations can be accurate without being precise orvice-versa. The measures of accuracy and precision can aid inidentifying the sources of uncertainty. For example, aninaccurate but precise simulation may account for inputdistributions well, but use a parameter value that deviatesfrom the value that should be used.

Control and dynamic systems define uncertainty by adifference between models and reality [23]. Error is themeasure of the difference between some observed value andits prediction from a model or simulation. The uncertainty ofsimulations can be determined through combining the inputparameter distributions, validation error, numericalapproximations, and the other uncertainty types as presentedin the following sections of this paper.

SOURCES OF UNCERTAINTYUncertainty in vehicle simulation studies usually occurs

from multiple sources. It is the responsibility of researchers toidentify the primary sources of uncertainty in the simulationmethods they are using and ensure that the uncertainty isproperly accounted for in the simulation and results. A few ofthe identifiable sources of uncertainty include: systemdynamics, numerical methods, parameters, assumptions, andvalidation criteria [24]. These five sources of uncertaintyeach fit into different portions of a simulation study as shownin Figure 3.

Figure 3. Sources of uncertainty in vehicle simulation.

SYSTEM DYNAMICS UNCERTAINTYReal world systems are highly dynamic. As these systems

are modeled in a computational domain considerations mustbe made as to the frequency of solving different systemequations. Continuous time step and fixed time stepcomputational solvers have been used in vehicle simulationstudies and will be discussed in more detail. Before selectingwhich solver should be used, an understanding of the rate ofvariable change in dynamic systems must be considered.

In the real world, changes in systems occur at aninfinitesimally small time scale. A common way ofconsidering these systems is to make measurements of thesystems based on common unit measurements. For example,even though ambient temperature can be measured to asmany significant figures as the measurement device allows,for vehicle systems only two significant figures arecommonly used because the performance of most automotivesystems is insensitive to small changes in temperature. Thismeans that the computation of the ambient temperature in avehicle system simulation only needs to occur such thatchanges can be accounted for at the specified level of detail.

Table 1. Comparison of simulation tools



Table 2. dynamic time scales for fuel cell vehiclessystems and hybrid vehicle

One way of determining the level of detail considered insystem dynamics is to evaluate the compounded effect oncomputational solutions. Significant figure inclusion shouldbe determined such that effects can be measured in theoutputs and the magnitude of the uncertainty is less than thedynamic detail. For example, measuring the same ambienttemperature introduced above to five significant figures isunnecessary if the uncertainty occurs up to one significantfigure. Table 2 provides a few suggested dynamic time scalesfor system simulation from fuel cell [25] and hybrid electricvehicles [26]. Dynamic time scales should be determined foreach calculation made in a simulation. As an example, aDC/DC converter operating at 50 kHz should not be modeledto provide dynamic output at 1Hz. Some subsystem timescales are more immediately identifiable such as InternalCombustion Engine (ICE) torque slew rates and switchingfrequencies, where others such as electrochemical reactionrates and thermodynamic interactions can be complex toimplement without high levels of detail.

Figure 4. Comparison of simulation time step effect onpower requirements.

Uncertainty occurs in dynamic vehicle simulation studieswhen the time scale of the systems is not accounted for. If thesimulation calculation occurs at a slower rate than thedynamics of the system, then functional details can be lost.Loss of detail in the vehicle simulation leads to uncertainty.For example, if electric motor (EM) torque output is

simulated at 0.5 Hz and the motor is capable of performing at1 Hz, then important operational characteristics of the EMsystem may be absent from the simulation results. Figure 4shows a sample comparison of a typical compact vehiclesimulated on the FUDS 505 drive cycle at differentfrequencies (1Hz base frequency). Table 3 shows the resultsof the vehicle simulation from Figure 4. It can be seen thatremoving dynamics from the system greatly influencesenergy use of the simulation, but does not affect the totaldistance traveled as much. It should be noted that both theenergy and distance observed are calculated by cumulativeintegration of other values. The difference in dynamicinfluence is directly related to the rates at which each sub-value changes (power and velocity).

Table 3. Comparison of dynamic frequency effects onenergy use and distance traveled.

A simplified comparison of simulation uncertaintyincorporating system dynamics and simulation calculationcausality is shown in Figure 5. The uncertainty shown inFigure 5 is representative of validation error values found forsimulation tools in each category when evaluating theprediction of simulated vehicle MPG. It can be seen in Figure5 that as the complexity of the system dynamics and themodel increases, the simulation uncertainty decreases, but ata decaying rate. The decaying rate exemplifies thediminishing returns on accuracy for increased simulation toolcomplexity.

Figure 5. Comparison of relative uncertainty associatedwith simulation methods for MPG.

NUMERICAL METHODSUNCERTAINTY

The numerical methods used to define a vehiclesimulation can greatly affect the uncertainty of the solutions.



The specific numerical methods discussed in this sectionpertain to the numerical approximation of physical systems aswell as the simulation solvers used to perform thecomputations [27].

Numerical approximation of real systems must occur invehicle simulations. On a broad spectrum, any equation usedto describe a physical system is in some way anapproximation. Equations are defined for systems based onan inability to disprove, not on the ability to prove. Equationsranging from Kirkoff's Laws for electrical systems toaerodynamic drag are all approximations of real systems andhave some inherent uncertainty, albeit usually very small.Progressing beyond physical system equations, it is notuncommon for vehicle system models to incorporateadditional system numerical approximation such as quasi-static lookup tables and functional surface fits. When usingthese approximations the data supplied to formulating theinitial approximation is finite and unable to represent everypossible operational state of the system. For example, lookuptables are commonly used for efficiency approximation ofICE operation. Data points are supplied from test benches atfinite points along predetermined dimensions such as speedand load. Although the data points may be considered to havelow uncertainty based on the data acquisition method used,the operation of the ICE at conditions that lie between datapoints provide some level of uncertainty. It is usually the casethat a very limited number of data points are supplied to thesemaps that immediately reduces the accuracy of the simulationas data located between measured points is probabilistic.Increasing the density of the data points taken can improveaccuracy but can never absolutely match the operation of thephysical system even in steady state considerations. A studyperformed by Echter [28] compares test data with simulatedengine fuel maps for large diesels. Using only the enginesubcomponents of the model, and feeding the test data enginespeed and load directly into the simulation, fuel consumption(L/100km) errors averaging 2.7% up to 7.7% were found.These errors have been associated directly with missedsystem dynamics due to the numerical approximation of thesystem.

As was mentioned previously in this paper, simulationtools can use fixed or continuous time step equation solvers.Fixed time step solvers take predefined advancements insimulation time space and calculate solutions to the modeledequations at each progressive state. Variable time step solvershave the ability to dynamically calculate the necessary timestep required to complete a calculation based on the dynamic

response of the system. Systems and events that exhibit fastresponse, such as in hydraulic or electrochemical systems,can be calculated with appropriate computation whennecessary since the time step taken is continuously changingto either increase or decrease the time scale considered.Variable time step solution methods commonly requireincreased amounts of computation when compared to fixedtime step systems, although limits to the scale of step takencan be applied to reduce this [29].

An example of a widely used vehicle simulation tool isMatlab/Simulink. Simulink has a variety of built in solveroptions including fixed and continuous time steps. Within thecontinuous time step simulation solvers, tests have beenperformed and recommendations made as to which systemsthe solvers should be used in. A simple comparison of thesolvers available in Simulink is shown in Table 4 [26]. Manyof the solvers available for vehicle simulation havecalculation error tolerances that can be set by the user. Thesetolerances are used to determine calculation convergence ateach time step for variable time step simulations. Simulationsshould be considered to always have uncertainty greater thanthe calculation error tolerance because of the compoundingeffect of solver error and other simulation uncertaintysources.

To demonstrate the numerical uncertainty found usingdifferent numerical solvers, a Matlab/Simulink demosimulation was used. The demo simulation was developed torepresent a HEV powertrain. The EM was observed operatingover 100 seconds of a FUDS drive cycle for three differentsolvers. The energy use results of the simulations are shownin Table 5. Although the errors are fairly low, it should benoted that these are integrated values. The errors present inthe example simulations would continue to propagate aslonger dynamic simulations progress. The ode113 solvercalculated value is set as the base value because of its claimfor high accuracy.

Table 5. Energy use of an EM using different numericalsolvers.

Table 4. Available Matlab/Simulink solvers and description of use.



PARAMETER DEFINITIONUNCERTAINTY

Depending on the study, uncertainty can come from avariety of sources. Parameters used in a simulation have errorresulting from the measurement of the representative vehicle(i.e. mass or frontal area). Also, evaluation metric parameterssuch as component costs or upstream GHG emissions maycontain error. Identifying errors within simulation, aftersimulation, or in both situations is a necessary task toquantifying the total simulation study uncertainty [1, 30].Sources or parameter uncertainty can occur through themeasurement of the parameter and in the definition andimplementation of the parameter.

Parameter uncertainty that occurs on the input side of thesimulation is associated with the definition of parametervalues to be used in the simulation [31]. This source ofuncertainty can arise from an inability to accurately measurea desired parameter, such as a fluid heat capacitance, withoutallocating for a wide range of assumptions. These types ofparameter definitions are commonly prescribed at standardoperating conditions for the vehicle system, but must beidentified as sources of uncertainty, particularly if the systemencounters non-standard operating conditions. Additionally,there is a source of uncertainty when taking measurements ofdesired parameters in that the specific measurement may notapply correctly to future systems [30]. An example of this canbe presented though manufacturing inconsistencies of hybridvehicle systems. When automotive battery packs aremanufactured, individual cells are combined to form acompleted unit. Due to manufacturing methods and materialvariation, the exact power and energy capacity of each cellmay be slightly different. To minimize cell failure due toimbalances within the pack, each cell is sorted according toits performance and alike cells are combined to form a batterypack. This method attempts to minimize inconsistencybetween successive battery pack characteristics, but theinconsistency cannot be eliminated.

Implementation of parameters in vehicle simulationstudies is a source for uncertainty in addition to theformulation of the parameters through measurement.Improved methods of allocating uncertainty exist inparameter definition such as applying a distribution to a givenparameter. But, if the applied distribution is not used in thesimulation study the uncertainty of the results increases.Approximations are not uncommon in parameter definition,but should be used sparingly and impacts should bemeasured. One common source of approximation uncertaintyfor parameters is scaling functions [32]. Many vehiclesimulation tools allow for subsystem components to be scaledbased on a defined factor, for example EM power scaling.The amount of uncertainty propagated through the simulationis sensitive to the inclusion of important factors in the scalingapproximation. In the previous EM example whereperformance maps are used, if the motor power rating isscaled, correct peak torques, corner speed, efficiency, mass,

and inertia calculations should also be performed todetermine new operational characteristics. Although thesescaling factors can be helpful in approximating a range ofsystems, as was discussed in the Numerical MethodsUncertainty section, uncertainty increases as dataapproximations are used further away from the measuredvalues.

As an example of the effects of input parameteruncertainty, a midsized HEV was modeled using Autonomiesimulated over the HWFET drive cycle. With the model,three simulations were run using the default ICE power andscaled powers 5% greater and 5% less than the default value.The resulting changes in MPG, CO2 emissions, andelectricity use are shown in Figure 6. For changes in ICEpower of 5%, all of the observed simulation results changedless than 1.5%; each with different magnitudes dependinghow sensitive the calculation is to the input parameter underconsideration. Parameter uncertainty for model inputs can beaccounted for most easily by including parameterdistributions. Delorme [4] uses the input parameterdistributions in a RoF studies to compare possible future fuelsfor passenger vehicles, accounting for uncertainty in theprediction of future technology scenarios.

Figure 6. Autonomie input parameter (ICE Power)variation effects.

UNCERTAINTY ASSOCIATED WITHASSUMPTIONS ANDSIMPLIFICATIONS

A few of the sources of uncertainty in vehicle simulationsassociated with assumption include constraints, initial andboundary conditions, and stochastic environments. Toimprove simulation, boundary conditions can be applied tovehicle simulation subsystems and components to either limitoperation based on control strategy, or to enforce physicallimitations that have been observed during data acquisitionbut have not accurately been modeled. An example ofboundary conditions includes ICE fuel injection rates. Invehicle simulations without highly detailed ICE models fuelrate functions or quasi-static maps have been used [28]. Toaccount for situations where the engine may behave



differently than standard conditions allow, such as overspeeding or rotating backwards, constraints are applied tomitigate inaccuracy [30]. These constraints may not bephysically accurate of the system being modeled.

Another type of uncertainty associated with constraintsassumptions involves the design space. When DoE andoptimization design studies are performed there is apossibility that limits will be applied to the allowable range ofdesign variables [2]. Occasionally these assumed limitationscan have functional requirements, such as having an ICE witha negative power rating, but other times they may intend tolimit the scope of the design space exploration such as notconsidering an ICE greater than 400 kW. Uncertainty indesign space limitations of the second type can be identifiedparticularly in RoF and TR studies because possible desirabledesigns may be excluded from the study unintentionally.

Initial conditions and simulation environmentassumptions affect study uncertainty similar to parametricdefinitions, but differ in application due to increased amountsof randomness. One common example of this type ofuncertainty lies in vehicle-road interactions. Many vehiclesimulation design studies assume a uniform road surface withideal friction interactions. More advanced vehicle modelsattempt to simulate road slip conditions such as uniformpavement, gravel, or even ice but require highercomputational costs due to increased detail. Exclusion ofstochastic road environment conditions has been shown tocause certain amounts of uncertainty in many of the designobjectives such as fuel economy, controls system design, andsystem robustness. In reality minor imperfections causesystems such as traction control to function that can greatlychange vehicle operational characteristics. External dynamicssuch as cornering, which cause power distribution changes inthe differential, are usually neglected. Most simulations areassumed to occur in a straight line over drive schedules thatare not representative of realistic vehicle operation. Gopal etal shows with multiple simulation tools that curved vehiclepaths can reduce vehicle fuel economy (MPG) by ∼25% forthe same operating speed [13].

VALIDATION CRITERIAAn area of uncertainty that can be easily overlooked is the

evaluation of vehicle simulations with validation criteria.Whether validation of a simulation is considered at thesubsystem component of vehicle system level, uncertaintymust be considered on both the model side and the physicalsystem side [27]. The uncertainty that can be associated withdata acquisition from physical systems was already discussedbriefly throughout previous paragraphs. Validation criteriauncertainty sources are more concerned with the methods ofvalidation.

When performing simulation validation, it is not enoughto just compare simulation and test data graphically.Graphical comparisons of data sets may appear reassuring toan observer, but offer no mathematical basis for an accuratevalidation. Advancing slightly beyond simple graphical

comparison, linear fitting of simulated and test datacorrelations can offer a metric for measuring accuracy andprecision of simulation tools. Statistical t-tests and p-valueanalysis offers another metric for analysis. Rebba et al [33]suggests the use of Bayesian methods to ensure statisticalcomparisons between test and simulation data that isdefensible. Very few of the vehicle design simulation toolsand studies examined as background for this paper includedstatistically defensible validation methods beyond visualcomparison.

The method of simulation validation performed shouldalways be associated with the objective outputs of thesimulation study. For example, if a study is focused onevaluating fuel economy of different vehicle designs, then thesimulations should be validated through comparisons ofsimulated and real vehicle fuel economy. Brooker et al [16]show validation performed for NREL's Future AutomotiveSystems Technology Simulator (FASTSim) using a variety ofvehicle architectures, allowing for extensibility of applyingthe simulation tool to many vehicles. All except for one of thevehicle validated for FASTSim produced fuel consumptionerrors within 10% as shown in Figure 7. As another example,if a design study aims to conduct control systemdevelopment, then validation should be performed on thephysical system relative to changes in control strategy. For acontrols system validation, it is likely that time and eventspecific operation should be validated instead of only end-of-test accumulated values such as fuel economy. Mismatchingof simulation validation criteria with study objectives can beidentified as a major source of uncertainty in many studies.For example, the vehicle modeling and simulation toolsADVISOR and PSAT have been developed and validatedwith vehicle fuel economy studies [8, 20, 34]. Researchersand other vehicle simulators continue to perform studies onvehicle control system development with the simulation toolslisted above, without proper validation of the simulationsrepresenting their systems.

Figure 7. Validation of FASTSim for fuel economy(image courtesy Aaron Brooker, NREL).

A second validation criteria area of uncertainty lies in thecorrectness of the criteria. A prominent example is thesimulation of vehicles over limited drive cycles. Studies thatconduct vehicle simulation over drive cycles must be careful



to consider that the results are only representative of the driveconditions simulated. Thus, a study performed for vehiclesoperating on a city driving schedule should not claim resultsfor all operating conditions. Additionally, the validation ofsuch driving can be increasingly difficult because of humanerror interactions. Unless validation vehicle systems aretested solely with HIL and computer controls, dynamometeror real world driving with people should be considered asource of uncertainty when comparing simulation andphysical validation criteria.

A few simulation tools have been validated using othersimulations tools [20, 35, 36]. This approach can besuccessful if the validation criteria for the original simulationtool and the second simulation tool match, but can lead toincreased uncertainty if not. When simulation models arevalidated using other simulation model, there is an advantageof being able to compare transient simulation values directlyand observe more variables than may be available fromvehicle testing efforts. The problem with compoundingvalidation is that uncertainty can be misleading. For example,PAMVEC was validated using ADVISOR [20]. There was a20% error between ADVISOR's total energy use calculationsand PAMVEC's simulation of similar vehicles. ADVISOR isclaimed to have 10% uncertainty for total energy use,creating the potential for ∼30% total energy use uncertaintythrough error compounding in the two simulation tools. Toreduce this uncertainty PAMVEC was also validated usingvehicle test data, showing ∼10% error for fuel consumption(MPG equivalent) for a fuel cell HEV.

MEASUREMENT OF UNCERTAINTYUncertainty should be measured based on the objective

evaluation metric. Within the vehicle study type performed,objectives should have been defined at the beginning of thestudy process. The same metrics that are being used toquantify the outcome of the simulations should be used forvalidation and uncertainty quantification.

Researchers using vehicle simulation tools should beaware of the uncertainty that exists in the models theyincorporate into studies. A broad investigation to finddocumentation for validation and uncertainty in differentsimulations tools returns limited information. Table 6 lists afew of the validation error values for different simulationtools. The objective evaluation metric used to perform thevalidation for each tool is also listed. In each validation case,different assumptions are inherent such as the drive cycleused, environmental conditions, etcetera; the assumptions arenot included in Table 6 and should be investigated forspecific studies.

Table 6. Validation errors found for specific metrics ofdifferent simulation tools

The simulation tools listed in Table 6 have been used in avariety of vehicle design studies. A few of the studies thathave been documented in literature are listed in Table 7 alongwith the design objective evaluation metrics, simulation tools,and results margins. The results margins for these studiesrepresent the deviation found between options within thestudy. For some of the studies the option is choosing betweenvehicle technologies or fuels, for others it can be improvedcontrol methods over a baseline vehicle, etc. The resultsmargin is important in these studies because in order for theresearcher to present a valid conclusion, the differencebetween two design options must be greater than theuncertainty for the respective simulation tool. If the resultsmargin is not larger than the validation error (and thus theuncertainty) then there is a probability that the results of thestudy may compare differently. Figure 8 shows a graphicalrepresentation of how results margins and simulationuncertainty interact to determine validity of solutions. Theexistence of the non-conclusive region for the overlappinguncertainty within the results margin is undesirable. Fullydefensible solutions would not contain a non-conclusiveregion.

One issue that arises in some of the simulation validationis the fine tuning of systems to get the desired results (fueleconomy, acceleration, etc.). For example, controllerparameters may be optimized so that a vehicle model with asimple controller produces results closely matching test data,when in fact the actual vehicle control system is likely to bemuch more complex. Although the tuning of the system maywork well for a single case, it is not necessarily representativeof real vehicle operation. For example, Cao [44] presentsvalidations methods for PSAT using a PHEV convertedToyota Prius. The standard PSAT PHEV Prius model has 9%error for fuel consumption (L/100km) for CD operation, butwith fine tuning of control strategy the error is reduced. Thelimited nature of the test case control tuning may lead to



increased errors in other facets of the simulation such asdifferent drive cycles.

Compounding uncertainty exists as design processes buildupon one another. Uncertainty present in different portions ofthe simulation is combined together and is likely to culminatein amplification of result uncertainty. This effect increasinglypromotes proper understanding of the uncertainty included inthe simulations used to perform vehicle studies. A graphicrepresentation of the uncertainty propagation through asimulation study is provided in Figure 9. Propagation ofuncertainty in complex vehicle models can only bedetermined directly from the models being used. Somecombinations of uncertainty can lead to increases in resultuncertainty. By limiting the sources of uncertainty that areintroduced to a study, the overall results uncertainty can alsobe controlled.

DISCUSSIONBy considering the sources of uncertainty in vehicle

simulation studies we can understand more quantitatively thecapabilities and weaknesses of vehicle simulation studies thathave been performed. Investigation of previous vehicledesign studies shows a lack of uncertainty consideration.Scientifically valid studies should include an accurateaccount of all information sources so that the uncertainty canbe quantified. By including all of the uncertainty typespresented in this paper (dynamics, numerical methods,parameter definition, assumptions/simplifications, andvalidation) future vehicle simulation studies can be improved.It is the job of the simulations tool developer to fullydocument the uncertainty that exists within their system, andthe job of the researcher and simulation tool user to accountfor this uncertainty in conclusions that they develop.

One example of how the use of uncertainty considerationcan improve a simulation study (other than just providing

Table 7. Comparison of vehicle simulation studies and results margins.

Figure 8. Evaluation of vehicle design improvement results margins and uncertainty for a design metric (ex. MPG).

Figure 9. Representation of uncertainty propagation in vehicle simulation studies. Different portions of studies (blue squares)contribute different sources of uncertainty (white squares).



defensibility) is in simulation optimization. One of the majorfactors in performing an optimization is determination ofconvergence criteria. If the objective function metric haspredetermined uncertainty, then the progression of theoptimization should be considered converged when thedifference between current iteration and the optimal answerare within the uncertainty range. Fellini [6] uses ADVISORto optimize fuel economy, as an example, and should have setthe optimization convergence criteria to be 5% (to beconsistent with validation error and uncertainty). Accuratedetermination of the convergence criteria will affect theiterations necessary to complete the optimization and mayeven have changed the solution if the convergence criterionwas set too broad.

Based on the information compiled through thedevelopment of this paper, the sources of uncertainty invehicle simulation can be ranked according to their influenceon uncertainty (% error). Figure 10 shows each of the fivesources of uncertainty ranked from greatest to least influence.The ranking of the uncertainty sources is not definitive aseach source has a probability of being either high or lowdepending on how they are applied. For example, eventhrough Assumptions and Simplifications are ranked ascontributing relatively high uncertainty, a researcher coulddevelop highly detailed models that include few assumptions.The order of influence proposed incorporates finding basedon literature and investigations from available sources as anaverage uncertainty found in each of the uncertainty sources.

SUMMARYAs demand for vehicle simulation increases in both

academic and professional areas, so does the requirement foraccuracy within the simulations. To improve the accuracy ofthese vehicle simulations, researchers must account foruncertainty. Uncertainty can come from a wide range ofsources throughout the simulation study process beginning atthe determination of the type of study being performed andprogressing through the evaluation of the study results. Thispaper quantifies the different types of uncertainty that existswithin state of the art vehicle simulation studies and identifiesareas that are important for future studies to consider. Anextensive literature review has been performed and thecombined conclusions of numerous sources have beenintegrated with author viewpoints to develop and broadenunderstanding of uncertainty in vehicle simulation studies.

REFERENCES1. Faron, G., Pagerit, S., and Rousseau, A., “Evaluation of PHEV's Fuel

Efficiency and Cost Using Monte Carlo Analysis,” EVS24, 2009.2. Geller, B., “Increased Understanding of Hybrid Vehicle Design through

Modeling, Simulation, and Optimization,” M.S. Thesis, Department ofMechanical Engineering, Colorado State University, 2010.

3. Golbuff, S., “Optimization of a Plug-in Hybrid Electric Vehicle,” M.S.Thesis, Department of Mechanical Engineering, Georgia Institute ofTechnology, 2006.

4. Delorme, A., Pagerit, S., Sharer, P., and Rousseau, A., “Cost BenefitAnalysis of Advanced Powertrains from 2010 to 2045,” EVS24,Argonne National Laboratory, 2009.

5. Bradley, T.H., and Frank, A.A., “Design, Demonstration, andSustainability Impact Assessments for Plug-in Hybrid ElectricVehicles,” Renewable & Sustainable Energy Reviews, Elsevier, 2007.

6. Fellini, R., Michelena, N., Papalambros, P., and Sasena, M., “OptimalDesign of Hybrid Powertrain Systems,” University of Michigan, AnnArbor, Michigan, 1998.

7. Xu, J-W., and Zheng, L., “Simulation and Analysis of Series HybridElectric Vehicle Based on ADVISOR,” 2010 International Conferenceon Measuring Technology and Mechatronics Automation, 2010.

8. Senger, R.D., “Validation of ADVISOR as a Simulation Tool for aSeries Hybrid Electric Vehicle Using the Virginia Tech FutureCarLumina,” M.S. Thesis, Department of Mechanical Engineering, VirginiaPolytechnic Institute and State University, 1998.

9. Wipke, K.B., and Cuddy, M.R., “Using an Advanced Vehicle Simulator(ADVISOR) to Guide Hybrid Vehicle Propulsion SystemDevelopment,” National Renewable Energy Laboratory, 1996.

10. Kellermeyer, W.F., “Development and Validation of a Modular HybridElectric Vehicle Simulation Model,” M.S. Thesis, Department ofMechanical Engineering, West Virginia University, 1998.

11. Schmitt, K., Madsen, J., Anitecsu, M., and Negrut, D., “A Gaussian-Process Based Approach to Handling Uncertainty in Vehicle DynamicSimulations,” Proceedings of IMECE2008, IMECE2008-66664, 2008.

12. Datar, M.V., “Uncertainty Quantification in Ground Vehicle Dynamicsthrough High Fidelity Co-Simulation,” M.S. Thesis, Department ofMechanical Engineering, University of Wisconsin - Madison, 2008.

13. Gopal, R.V. and Rousseau, A., “System Analysis Using Multiple ExpertTools,” SAE Technical Paper 2011-01-0754, 2011, doi:10.4271/2011-01-0754.

14. Jones, S., “System Simulation Tools for Electrified Vehicles,” Joint EC/EPoSS/ERTRAC Expert Workshop in Berlin, 2011.

15. Kwon, J., Rousseau, A., and Sharer, P., “Analyzing the Uncertainty inthe Fuel Economy Prediction for the EPA MOVES BinningMethodology,” SAE Technical Paper 2007-01-0280, 2007, doi:10.4271/2007-01-0280.

16. Brooker, A., Thornton, M., and Rugh, J., “Technology ImprovementPathways to Cost-effective Vehicle Electrification,” SAE TechnicalPaper 2010-01-0824, 2010, doi:10.4271/2010-01-0824.

17. Wilkins, S., and Lamperth, M.U., “An Object Oriented Modeling Toolof Hybrid Powertrains for Vehicle Performance Simulation,” ImperialCollege of Science, Technology and Medicine, London, 2003.

18. Rahman, Z., Butler, K.L., and Ehsani, M., “Designing Parallel HybridElectric Vehicles Using V-ELPH 2.01,” Proceedings of the AmericanControl Conference, San Diego, California, 1999.

19. Aceves, S. and Smith, J., “A Hybrid Vehicle Evaluation Code and ItsApplication to Vehicle Design,” SAE Technical Paper 950491, 1995,doi: 10.4271/950491.

20. Simpson, A.G., “Parametric Modeling of Energy Consumption in RoadVehicles,” Ph.D. Thesis, School of Information Technology andElectrical Engineering, The University of Queensland, Australia, 2005.

21. Powell, B.K., and Pilutti, T.E., “A Range Extender Hybrid ElectricVehicle Dynamic Model,” Proceedings of the 33rd Conference onDecision and Control, 1994.

22. Gao, D.W., Mi, C., and Emadi, A., “Modeling and Simulation ofElectric and Hybrid Vehicles,” Invited Paper, IEEE, Vol. 95, No. 4,2007.

Figure 10. Ranking of influence for sources of uncertainty in vehicle simulations.



http://www.sae.org/technical/papers/2011-01-0754

http://dx.doi.org/10.4271/2011-01-0754


http://dx.doi.org/10.4271/2007-01-0280


http://dx.doi.org/10.4271/2010-01-0824

http://www.sae.org/technical/papers/950491

http://dx.doi.org/10.4271/950491

23. Thunnissen, D.P., “Uncertainty Classification for the Design andDevelopment of Complex Systems,” California Institute of Technology,2003.

24. Oberkampf, W.L., DeLand, S.M., Rutherford, B.M., Diegert, K.V., andAlvin, K.F., “Estimation of Total Uncertainty in Modeling andSimulation,” Sandia Report, Sandia National Laboratory,SAND2000-0824, 2000.

25. Pukrushpan, J.T., “Modeling and Control of Fuel Cell Systems and FuelProcessors,” Ph.D. Thesis, Department of Mechanical Engineering, TheUniversity of Michigan, 2003.

26. Koprubasi, K., “Modeling and Control of a Hybrid-Electric Vehicle forDrivability and Fuel Economy Improvements,” Ph.D. Thesis,Department of Mechanical Engineering, The Ohio State University,2008.

27. Oberkampf, W.L., Trucano, T.G., and Hirsch, C., “Verification,Validation, and Predictive Capability in Computational Engineering andPhysics,” Johns Hopkins University/Applied Physics Laboratory, 2002.

28. Echter, N., “Parallel Hydraulic Pressure Assist/Work Circuit Hybrids forAutomated Side Loader Refuse Vehicles,” M.S. Thesis, Department ofMechanical Engineering, Colorado State University, 2012.

29. Liu, L., Felgner, F., and Frey, G., “Comparison of 4 Numerical Solversfor Stiff and Hybrid Systems Simulation,” Chari of Automation,Saarland University, Saarbrucken, Germany, IEEE, 2010.

30. Cuddy, M., “A comparison of Modeled and Measured Energy Use inHybrid Electric Vehicles,” National Renewable Energy Laboratory,NREL/TP-473-7444, 1995.

31. Bruggemann, H., and Kiencke, U., “Uncertainty Theory in VehicleDynamic Simulation,” Proceedings of the American ControlConference, 2002.

32. Petersheim, M.D., and Brennan, S.N., “Scaling of Hybrid ElectricVehicle Powertrain Components for Hardware-in-the-Loop Simulation,”17th IEEE International Conference on Controls Applications, 2008.

33. Rebba, R., Huang, S., Liu, Y., and Mahadevan, S., “StatisticalValidation of Simulation Models,” Int. J. Materials and ProductTechnology, 2004.

34. Sharer, P., Rousseau, A., Nelson, P., and Pagerit, S., “VehicleSimulation Results for Plug-in HEV Battery Requirements,” ArgonneNational Laboratory, 2006.

35. He, X., and Hodgson, J.W., “Modeling and Simulation for HybridElectric Vehicles - Part 1: Modeling,” IEEE Transactions on IntelligentTransportation Systems, Vol. 3, No. 4, 2002.

36. Hofman, T., Steinbuch, M., and van Druten, R.M., “Modeling andSimulation of Hybrid Drivetrain Components,” Technische UniversiteitEindhoven, Netherlands, 2001.

37. Wang, B., Lou, Y., “AVL Cruise-based Modeling and Simulation ofEQ6110 Hybrid Electric Public Bus,” 2010 International Conference onComputer Application and System Modeling, 2010.

38. Demirdoven, N., and Deutch, J., “Hybrid Cars Now, Fuel Cell CarsLater,” Science 305, (974), 2004.

39. Wu, X., Cao, B., Li, X., Xu, J., and Ren, X., “Component SizingOptimization for Plug-in Hybrid Electric Vehicles,” Journal of AppliedEnergy, Elsevier, 2010.

40. Gao, W., “Performance Comparison of a Fuel Cell-Battery HybridPowertrain and a Fuel Cell-Ultracapacitor Hybrid Powertrain,” IEEETransactions on Vehicular Technology, Vol. 54, No. 3, 2005.

41. Sangtarash, F., Esfahanian, V., Nehzati, H., Haddadi, S. et al., “Effect ofDifferent Regenerative Braking Strategies on Braking Performance andFuel Economy in a Hybrid Electric Bus Employing CRUISE VehicleSimulation,” SAE Int. J. Fuels Lubr. 1(1):828-837, 2009, doi:10.4271/2008-01-1561.

42. Ye, X., Jin, Z., Liu, B., Chen, M., and Lu, Q., “Design and Applicationof Parallel Hybrid Vehicle Simulation Platform,” State Key Laboratoryof Automotive Safety and Energy, Tsinghua University, Bejing, China,2008.

43. Geller, B. and Bradley, T., “Objective Comparison of Hybrid Vehiclesthrough Simulation Optimization,” SAE Technical Paper 2011-01-0943,2011, doi:10.4271/2011-01-0943.

44. Cao, Q., Pagerit, S., Carlson, R., and Rousseau, A., “PHEV HymotionPrius Model Validation and Control Improvements,” Argonne NationalLaboratory, 2007.

CONTACT INFORMATIONBenjamin M. GellerColorado State UniversityDepartment of Mechanical EngineeringFort Collins, CO [email protected]

DEFINITIONS/ABBREVIATIONSCD

Charge depletingCS

Charge sustainingAER

All Electric RangePHEV

Plug-in Hybrid Electric VehicleTCO

Total Cost of OwnershipGHG

Green House GasHIL

Hardware In the LoopMPG

Miles Per GallonDoE

Design of ExperimentsTR

Technical RankingRoF

Representation of FuturesSoA

State of the ArtSDE

System Development and Exploration



http://dx.doi.org/10.4271/2008-01-1561


http://dx.doi.org/10.4271/2011-01-0943

mailto:[email protected]

quantifying uncertainty in vehicle simulation studiesthb/publications/geller.pdffirst step in...

Documents