construction de réseaux de neurones dynamiques et application à

1

Context Construction of a spatio-temporal surrogate model Application to transient thermal engineering

Construction de réseaux de neurones dynamiqueset application à la modélisation de phénomènes de

thermique transitoires

Jonathan Guerra1 3, Patricia Klotz1,Béatrice Laurent2 and Fabrice Gamboa2

13 Novembre 2014

1ONERA2Institut de Mathématiques de Toulouse3Epsilon Ingénierie

JSO ONERA Jonathan Guerra

1


Table of Contents

1 Context

2 Construction of a spatio-temporal surrogate modelMultilevel optimizationRobust model selectionSensitivity Analysis to reduce input dimension

3 Application to transient thermal engineering


1


Table of Contents

1 Context

2 Construction of a spatio-temporal surrogate model



2


Physical problem: Electronic equipment in the avionic bay

• Physical modeling of an avionic bay isn’t easy: numerousinteractions between the equipment, fluid dynamics, radiation...

• The equations to solve are Navier-Stokes ones coupled with heatequation.

→ Thermo-fluidic modeling to perform. Several tools exist:

� Commercial softwares (FLOTHERM, Fluent)� ONERA softwares (CEDRE coupling CHARME and ACACIA)� Physical reduced models such as nodal models


2




=⇒ Need of a surrogate model in thermal engineering:

1 Optimization (of the lifetime of the equipment for instance)2 Multilevel simulation of the avionic bay


2




=⇒ This complex modeling implies for the surrogate:

� Few learning trajectories� The input/output dimension can be important� Construction time must be reasonable� Long-term in time prediction


2


Table of Contents

1 Context




3


Surrogate model for transient thermal engineering

The phenomenon follows heat equation so :

∂T∂t

(x , t) = D∆T (x , t) + C

By discretizing temporally this equation tk = k∆t with k ∈ [0,Nt ]:

T (x , tk) = F[T (x , tk−1)

]An interesting mathematical formulation for the surrogate F is:(

T k1 , · · · ,T k

Np

)= F

(T k−11 , · · · ,T k−1

Np, exogenous variables at tk

)(1)

with Np the number of points of interest


4


Dynamical model for spatio-temporal prediction

• Principle: a recursive formulation is used

• It means that the outputs at time tk−1 become the inputs of the samemodel at time tk

• It can be written{yk = F

(yk−1, uk ,∆tk ; w

)y0 = y0

• The parameters w are theone minimizing:

Elearning =

Ny∑j=1

l∑sample

Nt∑k=1

∥∥∥yk,lj − yk,l

j

∥∥∥2

2


5


Definition of an artificial neural networkMathematically, an artificial neural network can be written:

{s}j∈J1,NsK = {F (e;w)}j∈J1,NsK =

{Nn∑n=1

w0,n,jφ

(Ne∑i=1

wi,n,0ei + w0,n,0

)+ w0,0,j

}j∈J1,NsK

Derivative of the output sj relatively to its weights:

∂sj∂w

=∂Fj (e ; w)

∂w


φ is the logistic function

6


Neural network for spatio-temporal predictionFor spatio-temporal prediction:{

yk = F(yk−1, uk ,∆tk ; w

)y0 = y0

Derivative of the output of the network relatively to its weights:

∂ykj

∂w=

∂Fj (uk , x,∆tk ; w)

∂w

∣∣∣∣∣x=yk−1

+

Ny∑m=1

∂Fj(uk , yk−1,∆tk ; w

)∂yk−1

m

∂yk−1m

∂w


6


Multilevel optimization

Table of Contents

1 Context




7



Multilevel optimization - Justification with 2 outputs and 4 neurons

Theoretically, the optimization of the weights should be performedon the following complete network:

yk1 = F1

(yk−11 , yk−1

2 ; w)

yk2 = F2

(yk−12 , yk−1

1 ; w)

y0j = y0j

minw

2∑

j=1

l∑sample

k∑time


j

∥∥∥2

2


7





yk1 = F1

(yk−11 , yk−1

2 ; w)

yk2 = F2

(yk−12 , yk−1

1 ; w)

y0j = y0j

minw

2∑

j=1

l∑sample

k∑time


j

∥∥∥2

2

But it requires to solve a high dimensional optimization problem:for instance with 6 outputs, 4 exogenous variables and 10 neurons,there are 176 weights to fit.


7





yk1 = F1

(yk−11 , yk−1

2 ; w)

yk2 = F2

(yk−12 , yk−1

1 ; w)

y0j = y0j

minw

2∑

j=1

l∑sample

k∑time


j

∥∥∥2

2

⇒ This great dimension implies a degraded solving of the weights.Solution: A multilevel framework has to be introduced to overcomethis problem : the optimization is decomposed output by output


8



Initialization : Optimization with measured inputs

∀j ∈ J1,Ny K, the weights w0j are optimized for each output yj . The

number of neurons is chosen at this step.Example with two outputs and two neurons per network: Back


9



Iteration it > 0 : Optimization with predicted inputs

• Only weights wj that are used to compute the output yj are beingoptimized at iteration it while the other outputs are computedthanks to the weights optimized at the last step it − 1. Back


10



Multilevel optimization - Recap of the process

1 Initialization of the process:Complexity selection andoptimization of the weights onone-output networks withmeasured inputs (except for theone corresponding to the outputcomputed by the network)

Detail

2 For it > 0: The network isinitialized with w it−1 optimizedat it − 1, and the weightsinvolved in the computation ofeach yj are separately optimized

Detail


10


Robust model selection

Table of Contents

1 Context




11



Model Selection by Cross Validation• Model selection gives an estimation of the generalization errorwhich is used to� Choose the complexity (number of neurons)� Choose the best model for each complexity

• Why Cross Validation?� It uses all the samples at disposal by resampling

• Principle :� Samples at disposal are splitted in 5 groups (also called folds)� One of the folds is only used to test (test samples) the network

built with the 4 other ones (learning samples)=⇒ 5 models are constructed

• The final model is obtained by averaging the outputs of those5 models :

Fj =15

5∑fold=1

F foldj


12



Cross Validation- How to split spatio-temporal examples?• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)

• Subtrajectories: problem of proximity between test folds andlearning folds


12



Cross Validation- How to split spatio-temporal examples?• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)→ Solution: stop considering entire trajectories, and split the

time steps into the folds (Subtrajectories)



12



Cross Validation- How to split spatio-temporal examples?

• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)→ Solution: stop considering entire trajectories, and split the




12



Cross Validation- How to split spatio-temporal examples?

• Whole trajectories: problem of robustness (because some foldcan poorly fill the entire input space)→ Solution: stop considering entire trajectories, and split the


• Subtrajectories: problem of proximity between test folds andlearning folds→ Solution: get rid of the redundant points of the learning

examples before dividing them into the folds Criterions


12


Sensitivity Analysis to reduce input dimension

Table of Contents

1 Context




13



Sensitivity Analysis - Need

• The number of input dimensions can be quite important. Thisimplies:⇒ An important number of weights to optimize⇒ A degraded solution when they are optimized

• Moreover, the aim is to build a parcimonious model (a point ofinterest is not equally influenced by the other ones!)

• Solution: sensitivity analysis is used on the surrogate model toquantify the impact of each input. Once the non-influentinputs are detected, a new network is built without thoseinputs.


14



Process

Steps

1 Once a first network is built, non-influent inputs aredetermined (thanks to sensitivity analysis)

2 A new network is built without the non-influent inputs of step1 (so with less dimensions)

3 The result after and before Sensitivity Analysis can becompared


15



Sensitivity Analysis - Choice of the method

• Tool: DGSM (Derivative-based Global Sensitivity Measures)

• Idea: Because it does not depend on time, the sensitivityanalysis will be applied directly on the artificial neural network

• Limit: strong correlations between the inputs of each models(the signification of those coefficient is not obvious in thiscase)


16



Sensitivity Analysis - Definition of DGSM

• Explanation: it is based on the fact that if the derivative of themodel output relatively to one of its inputs is important, itmeans that this input has a great influence on this output (atleast locally)

• It is defined as follows:To measure the influence of an input xj ∈ H on the outputy = F(x ; w), one has to compute:

νj = E

[(∂F(X ; w)

∂xj

)2]

=

∫HNx

(∂F(x ; w)

∂xj

)2

dµ(x)

• With the hypothesis that ∂F(X ;w)∂xj

is square-integrable, thosecoefficients exist


17



Sensitivity Analysis - Application to the network

The derivative of a neural network relatively to one of itsinputs has an analytical formulation:

∂Fk (x ; w)

∂xj=

Nn∑n=1

w0,n,kwj ,n,0φ′

(Ne∑i=1

wi ,n,0xi + w0,n,0

)

� k being the index of the output� xj being an exogenous variable or an output at the previous

time step

⇒ The mathematical definition of those coefficient gives a meaningto them


17


Table of Contents

1 Context

2 Construction of a spatio-temporal surrogate model



18


Presentation of the test case• To evaluate the presented methodology, a mathematical modelof an equipment is used (Thermal nodal network model):

• The neural network built has4 exogenous inputs and 6outputs which represent:

1 T upper wall2 T left wall3 T radiator4 T air5 T component6 T wall next to the

componentJSO ONERA Jonathan Guerra

19


Design of Experiment

• The learning samples is composed of 37 trajectories of 300seach.

• They are generated from the following set of inputs in thethermal network model: (T (t = 0),Tamb, I ,P,Flow rate).


19


Design of Experiment• The learning samples is composed of 37 trajectories of 300seach.


20


Test sample - Result of Multilevel optimization• The test sample is an in-flight profile which has the followingexogenous inputs:

• After optimization, it gives this result on the six outputs:


21


Multilevel Optimization result

• To proove the benefits of the methodology, multilevel splittingand construction without it are compared on this test case.Here are the results :

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Relative error

P[e

rro

r]<

Re

lative_e

rro

r

Without methodology

Initialization

2nd Iteration

3rd Iteration

• Without methodology, the result is obtained with 30 neurons.With the methodology, it is able to go up to 47


22


Sensitivity Analysis: Computation of the coefficients• The DGSM coefficients are computed on the outputs of thepreviously built network

This gives the “matrix of influences”:

Tamb →P →Q →I →

T1 →T2 →T3 →T4 →T5 →T6 →

1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6

• Remark: The air temperature needs all the other outputs to be computed,

but the other outputs don’t need it in input (the flow rate and ambienttemperature suffice in this case)


22



only the inputs j with νj > 0.005 are kept:

This gives the “matrix of influences”:


T1 →T2 →T3 →T4 →T5 →T6 →

1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6




22



only the inputs j with νj > 0.005 are kept:This gives the “matrix of influences”:


T1 →T2 →T3 →T4 →T5 →T6 →

1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6

• Remark: The air temperature needs all the other outputs to be computed,but the other outputs don’t need it in input (the flow rate and ambienttemperature suffice in this case)


22



only the inputs j with νj > 0.005 are kept:This gives the “matrix of influences”:


T1 →T2 →T3 →T4 →T5 →T6 →

1 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 1 1 1 11 1 0 1 1 10 1 0 1 0 01 0 1 1 1 10 0 0 1 0 01 0 1 1 1 10 1 1 1 1 1↓ ↓ ↓ ↓ ↓ ↓T1 T2 T3 T4 T5 T6




23


Sensitivity Analysis: Result

• A new network is built with less inputs for each output.

• The result before and after sensitivity analysis can now becompared:Error in ◦C T1 T2 T3 T4 T5 T6

RMSE before S-A 0.45 0.35 0.47 0.07 0.46 0.49RMSE after S-A 0.38 0.53 0.33 0.04 0.33 0.37

• By decreasing the size of the optimization problem (378weights instead of 695 or 245 if we get rid of T2 and T4), thesolution obtained is better


23


Sensitivity Analysis: Result• A new network is built with less inputs for each output.

• By decreasing the size of the optimization problem, thesolution obtained is better


24


Conclusions and perspectives

• This presentation has introduced an innovative construction ofspatio-temporal surrogate model based on :� A multilevel framework to optimize the weights� Cross validation for the model selection� Sensitivity analysis to reduce the input dimension

• The main perspective is the construction of a spatio-temporaldesign of experiment (in progress):� Mathematical form of the exogenous inputs (sinusoidal, crenel

functions, both?)� Choice of the best trajectories to compose it


25


THANK YOU

Questions?


26


Criterions to exclude some points1 First one: The points tk are kept if:

∣∣∣∣∂yj∂t (tk)

∣∣∣∣+

∣∣∣∣∂yj∂t (tk+1)

∣∣∣∣ > k1 if k 6= Nt

∣∣∣∣∂yj∂t (tNt )

∣∣∣∣ > k2 elsewhere

2 Second one: time between two time steps kept cannotexceeds

threshold = 5 ·∆t = 5 · tNt − t0Nt

3 Third one: if ti is time step kept and if tj is as j > i + 1 anddoes not verify the two first conditions, tj is kept if :∣∣∣∣∣

j∑k=i+1

∂yj

∂t(tk)

∣∣∣∣∣ > k3


Back

construction de réseaux de neurones dynamiques et application à

Documents