model sensitivity analysis

Lecture 8

Pr inc ip les of Model ing for Cyber-Phys ica l Systems

Instructor: Madhur Behl

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 1

Model Sensitivity Analysis

How do I know my model is any good ?

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 2

Sensitivity Analysis

In general ! = #(%, ' ( , ) ( )

We want to attribute, the uncertainty in y to the uncertainty and errors in parameters +, and inputs u

Sensitivity = ,-./012 3-45 624 -06706 8 129:;4<-= 9 129:;4 >: 9 5>:;?4 79=9/464= -= >:706 subject to, all other things being the same.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 3

Sensitivity Analysis

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 4

Input-Output Sensitivity

∆"∆#

Parameter Sensitivity

∆"∆$

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 5

Sensitivity Analysis

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 6

Sensitivity Analysis

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 7

Sensitivity Analysis

Better models for better control..

High PoorBuilding Model Accuracy

Sensor Placement & Density

Most BuildingsRequired for advanced control (MPC)

High retrofitting cost

Small and medium sized commercial buildings (90% of the commercial building stock) do

not want to spend thousands of dollars on retrofitting.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 8

“Accuracy costs money,

how accurate do you want it ?”

Sensor Data Quality vs Building Model Accuracy?

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 9

Sensor Data Quality and Uncertainty

1) Due to Sensor Placement and Density

Image courtesy Bryan Eisenhower (IMA talk)

4) Measurement Noise

3) Due to Inference: E.g. Heat gains from Occupancymeasured with people counters

2) Due to Sensor Precision

vs

$500 $1

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 10

1/Ugw

1/Ugi

1/Ugo

Cgo

Cgi

Tgo

Tg

Tgi

Tz

Q.rad,e

1/Uei1/Uew1/Ueo

Ceo Cei

Ta

Teo Tei

Q.sol,e

Q.solt/2

1/Uci

1/Ucw

1/UcoCci

Cci

Ta

Q.sol,c

Q.rad,c

Ta

Q.conv + Q.

sens

1/Uii 1/Uiw 1/Uio

Cii Cio

Tci

Tco

1/Uwin Tii Tio

Q.solt/2

Q.rad,g

[ExternalWalls]

Ti

[Ceiling]

[Floor]

[InternalWalls]

[Windows]

Building Modeling: “RC-Networks”

Wall Dynamics

Measure all Inputs and DisturbancesAmbient temperature, convective heat gain, radiative heat gain, external

solar gain, ground temperature, cooling rate

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 11

Discrete-Time State Space Model:(parameterized by θ)

States (All node temperatures): x = [Teo, Tei, Tco, Tci, Tgo, Tgi, Tio, Tii, Tz]T

Inputs (Disturbances and Control): u = [Ta, Tg, Ti, Qsole, Qsolc, Qrade, Qradc, Qradg, Qsolt, Qconv, Qsens]T

Parameter Estimation: Least Squares Error

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 12

Building Modeling: “RC-Networks”

Accuracy of an Inverse Model

1)Model Structure

2)Parameter estimation algorithm

3)Uncertainty in the input-output data

Non-Linear regression

FIXED

FIXED

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 13

MeasurementData

Cooling RateConvective Heat Gain

Solar IrradianceRadiative Heat GainTransmitted Solar

Ambient TemperatureSurface Temperatures

Input Uncertainty Analysis

ParameterEstimation(Inverse Model

Training)

Baseline Model

Principles of Modeling for CPS – Fall 2018 14Madhur Behl madhur.behl@virginia.edu

MeasurementData

Cooling RateConvective Heat Gain

Solar IrradianceRadiative Heat GainTransmitted Solar

Ambient TemperatureSurface Temperatures

Artificial DataCooling Rate (i)

Artificial DataCooling Rate (N)

ParameterEstimation(Inverse Model

Training)

Baseline Model

Perturb each input(N perturbations)

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 15

Input Uncertainty Analysis

MeasurementData

Cooling RateConvective Heat Gain

Solar IrradianceRadiative Heat GainTransmitted Solar

Ambient TemperatureSurface Temperatures

Artificial DataCooling Rate (i)

Artificial DataCooling Rate (N)

ParameterEstimation(Inverse Model

Training)

Baseline Model

Perturb each input(N perturbations) Parameter

Estimation

(N models)(N different fits)

Model w.Cooling Rate(i)

ParameterEstimation

Model w.Cooling Rate(i)

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 16

Input Uncertainty Analysis

MeasurementData

Cooling RateConvective Heat Gain

Solar IrradianceRadiative Heat GainTransmitted Solar

Ambient TemperatureSurface Temperatures

Artificial DataCooling Rate (i)

Artificial DataCooling Rate (N)

ParameterEstimation(Inverse Model

Training)

Baseline Model

Perturb each input(N perturbations) Parameter

Estimation

(N models)(N different fits)

Model w.Cooling Rate(i)

ParameterEstimation

Model w.Cooling Rate(i)

ComparePrediction

Error

Common input for

evaluation

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 17

Input Uncertainty Analysis

Uncertainty analysis with TRNSYS building

• North Facing

• 4 external brick walls

• 4 large windows

• Concrete floor and ceiling

• Philadelphia-TMY2 weather

• 3.5kW HVAC system

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 18

• 12 RC parameters

• 7 inputs, 1 output

• Baseline Model: RMSE 0.187

°C, R2 0.971

• Introduce fixed

perturbations/bias in each

input: !"# = !" ± (' ∗ !")

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 19

Uncertainty analysis with TRNSYS building

Ambient temperature, convective heat gain, radiative heat gain, external

solar gain, ground temperature, sensible cooling load

Normalized

% RMSE

change in

model

accuracy

% Input Perturbation

Madhur Behl madhur.behl@virginia.edu 20

Uncertainty analysis with TRNSYS building

Principles of Modeling for CPS – Fall 2018

Model Accuracy Sensitivity Coefficient

(for input u)

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 21

Uncertainty analysis with TRNSYS building

Case study: Building 101

Building 101 is located in Philadelphia and it’s the US DoE’s Energy Efficient Buildings Hub Headquarter

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 22

Model Accuracy for Training data

RMSE: 0.062 °CR2: 0.983

Model Accuracy for Test Data

RMSE: 0.091 °CR2: 0.948

Baseline

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 23

Case study: Building 101

Input Uncertainty Analysis: Building 101Normalized %

RMSE change in model accuracy

% Input Perturbation

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 24

Model Accuracy Sensitivity Coefficient: Building 101

Porch Temperature

Cooling Rate

Zone Temperature

Implications ?Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 25

26

Sensor Placement and Quality of Data: Suite 210

4 Indoor Air Quality Sensors 1 portable Cart Zone Thermostat

27

Compare the “true” (mean) temperature with thermostat measurement

Thermostat = G(Mean Temperature)

Is there a bias in the Thermostat data due to its location ?

28

Sensor Placement and Quality of Data: Suite 210

Comparison

Residuals ~ 1% bias

Porch Temperature

Cooling Rate

Zone Temperature

29

~1% bias in Zone temperature (Tz) is significant enough

Model accuracy can vary by > 20%

Sensor Placement and Quality of Data: Suite 210

Sensor Placement and Bias

Is comparing two means a good idea ?

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 30

T

2134

31

32 C

Maybe not..Multiple subsets could be compared

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 31

A closer look at temperature dataTemperature sensor data is not normal (Gaussian)

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 32

Non-parametric statistical methodsUse Wilcoxon’s rank sum test and Bland-Altman plots to quantify bias and identify best sensor placements.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 33

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 34

Non-parametric statistical methods

Zone Temperature – Business as usual

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 35

Zone Temperature – Business as usual

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 36

Zone Temperature – Business as usual

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 37

Model Accuracy for Training data

RMSE: 0.062 °CR2: 0.983

Model Accuracy for Test Data

RMSE: 0.091 °CR2: 0.948

Baseline

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 38

Case study: Building 101

Model Accuracy for Training data

RMSE: 0.062 °CR2: 0.983

Model Accuracy for Test Data

RMSE: 0.091 °CR2: 0.948

Baseline

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 39

Case study: Building 101

How did you learn this model ?

AHU Functional Tests: Suite 210Functional tests were carried out in Suite 210 in June 2013.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 40

AHU Functional Tests: Suite 210Functional tests were carried out in Suite 210 in June 2013.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 41

Is this optimal ?

What is Experiment Design ?

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 42

Optimal Experiment Design

Find the optimal input signal trajectory which maximizes the

information about the model parameters subject to

operational constraints.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 43

Find the optimal input signal trajectory which maximizes the

information about the model parameters subject to

operational constraints.

Requires some optimality criteria

Specify information metric

Needs a suitable parameter estimation

methodCannot play with set-points at the cost of

comfort

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 44

Optimal Experiment Design

Maximum Likelihood & Fisher Information

Likelihood functions play a key role in statistical inference and parameterestimation.

The probability that we see the given data due to the model we have assumedfor the building/equipment.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 45

L

Model(α)α2α1 Inconsistent

parameter valuesα*

True parameter value

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 46

Maximum Likelihood & Fisher Information

L

α* α2α1

Model(α)

Consistent parameter values

True parameter value

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 47

Maximum Likelihood & Fisher Information

(1) We want an estimate which maximizes the likelihood function.

Maximum Likelihood Estimate

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 48

Maximum Likelihood & Fisher Information

(2) Some way to quantify the difference between likelihood functions i.e. how quickly does it fall of around the maximum

= 0 at maxima α*Fisher information

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 49

Maximum Likelihood & Fisher Information

Cramer-Rao boundLet yt denote the set of t measurements y(0), y(1), ……, y(t-1).

The likelihood function

For any unbiased estimator we have the following Cramer-Rao lowerbound:

data

model

Error covariance of α

Fisher information matrix (FIM)

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 50

For the RC ‘grey box’ building model

State space model

Likelihood function

Need Kalman filter equations to compute

the likelihood function.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 51

But where is the experiment design ?

First we compute the Fisher Information Matrix

Depends only on the inputs and disturbances into the system

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 52

Optimality criteria of the information matrix ◦ A-optimal design ó average variance

◦ D-optimality ó uncertainty ellipsoid

◦ E-optimality ó minimax

◦ Almost a complete alphabet…

Stone, DeGroot and Bernardo

Optimality criteria

min$ %&'() *(,)./

min$ 0)% *(,)./

min$ max()34 * , ./ )

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 53

Example

Cai, Jie, et al. "Optimizing zone temperature set-point excitation to minimize training data for data-driven dynamic building models." American Control Conference (ACC), 2016. IEEE, 2016.

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 54

Input:

Tamb, Tstorage, Qsol, Qconv, Qsol,trans,Qrad, Tzone

Output:

Qsen

Optimal and conventional temperature set-point profiles

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 55

Principles of Modeling for CPS – Fall 2018 Madhur Behl madhur.behl@virginia.edu 56

Performance comparison of models trained with conventional and optimal training data sets