Algorithms for Dimension and Complexity Reduction Workshop, Providence, March 23-27, 2020

Learning block-oriented nonlinear models of dynamical systems from dataIon Victor Gosea1, Athanasios C. Antoulas1,2,3

1Max Planck Institute for Dynamics of Complex Technical Systems, Data-Driven System Reduction and Identification (DRI) group, Sandtorstraße 1, 39106 Magdeburg, Germany2Rice University Houston, Electrical and Computer Engineering Department, 6100 Main St., Houston, TX 77005, USA3Baylor College of Medicine, 1 Baylor Plaza, Houston, TX 77030, USA

Introduction, motivation and practical insight

Approaches

• Finite element/difference schemes usuallylead to large-scale high fidelity models andexpensive computations in memory/time.• Using data-driven methods, learn/reveal

reduced-order models to be used as surro-gates with cheap computation time/memory.

Type of data measurements

• Time domain: Easy to collect, commonlyused in many fields, e.g., nonlinear PDEs,turbulent flow control, gas/energy networks.• Frequency domain: Typically measured by

DNS (direct numerical simulations) or withVNAs, e.g., the S(scattering) parameters.

Linear and nonlinear systems; block-oriented models

• Modeling nonlinear systems is challenging due to many different possible nonlinear structures,i.e., based on Volterra series, Wiener theory, NARMAX models, neural networks, etc.• Different types of responses to an excitation, i.e., with subharmonics, bifurcation, chaos, etc.

Linear Systems{Ex(t) = Ax(t) + Bu(t),

y (t) = Cx(t).

where E, A ∈ Rn×n, B ∈ Rn×m, C ∈ Rp×n.

Time Domain

t0 2 4 6 8 10 12 14 16 18 20

u(t)

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2time domain input

t0 5 10 15 20 25 30 35 40 45 50

y(t)

-0.2

-0.1

0

0.1

0.2

0.3

output

input/output

Frequency Domain

frequency0 1 2 3 4 5 6 7 8 9 10

mag

nitu

de

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

input

frequency0 1 2 3 4 5 6 7 8 9 10

mag

nitu

de

0

0.005

0.01

0.015

0.02

0.025

0.03output

input/output

Nonlinear Systems{Ex(t) = Ax(t) + F(x(t), u(t)) + Bu(t),

y (t) = Cx(t) + G(x(t), u(t)),

where u(t) input and y (t) output.

Time Domain

t0 2 4 6 8 10 12 14 16 18 20

u(t)

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2time domain input

t0 5 10 15 20 25 30 35 40

y(t)

-1.5

-1

-0.5

0

0.5

1

1.5

output

input/output

Frequency Domain

frequency0 1 2 3 4 5 6 7 8 9 10

mag

nitu

de

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

input

frequency0 2 4 6 8 10 12 14 16 18

mag

nitu

de

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

output

input/output

• Block-oriented models form a powerful and intuitive tool to handle nonlinear systems.• Constructed from 2 blocks: a linear time-invariant (LTI) block and a static nonlinear block.

f (x) LTIu(t) f [u(t)] y (t) LTI g(x)u(t) y (t) g[y (t)]

Hammerstein Model Wiener Model

• Wiener models can approximate any nonlinear system with arbitrarily high accuracy [Boyd/Chua ’85])

• Many different block combinations are possible: series, parallel or feedback connections.

f (x) LTI g(x)u(t) f [u(t)] y (t) g(y (t))

Hammerstein-Wiener Model

• Expand the static nonlinearities, i.e. the smooth nonlinear functions f , g, into Maclaurin series

f [u(t))] =∞∑

k=1

dk f (u)duk |u=0

uk(t)k !

, g[y (t))] =∞∑

k=1

dkg(y )dyk |y=0

yk(t)k !

.

• Keep only the the first two terms, i.e. f [u(t))] ≈ α1u(t) + α2u2(t) and g[y (t))] ≈ β1y (t) + β2y2(t).

Wiener (Generalized) Model

ΣW :

{Ex(t) = Ax(t) + Bu(t),

y (t) = Cx(t) + K[x(t)⊗ x(t)

].

Hammerstein-Wiener (Generalized) Model

ΣHW :

{Ex(t) = Ax(t) + Bu(t) + Lu2(t),

y (t) = Cx(t) + K[x(t)⊗ x(t)

].

In order that the Wiener/Hammerstein structure is conserved, ∃a, b ∈ R, K = a(C⊗C) and L = bB.

Main contribution Given input-output data in the time domain (for purely oscillating control inputs),extract information from the spectrum and construct reduced-order models that explain the data.

The Loewner framework - linear systems

Aim: Construct reduced-order linear models directly from measurements - the Loewner framework.

• Frequency domain: linear [Mayo/Antoulas ’07]) & nonlinear [Antoulas/G./Ionita ’16]), [G. et al ’18, ’19]) .• Time domain: linear [Peherstorfer/Gugercin/Willcox ’17] & nonlinear [Peherstorfer/Gugercin ’20] .

1. Given measurements {(ωk , fk) : k = 1, ... , 2n} →, divide the data into two disjoint sets:

S = [ω1, ... ,ωn︸ ︷︷ ︸µ

,ωn+1, ... ,ω2n︸ ︷︷ ︸λ

], F = [f1(ω1), ... , fn(ωn)︸ ︷︷ ︸V

, fn+1(ωn+1), ... , f2n(ω2n)︸ ︷︷ ︸W

].

2. The associated Loewner & shifted-Loewner matrices L & Ls are introduced:

L(i ,j) =vi −wj

µi − λj, Ls(i ,j) =

µivi − λjwj

µi − λj, i , j = 1, ... , n.

• Exact amount of data - regular Loewner pencil:

H(s) = W(L− sLs)−1V =

wn+1

...w2n

∗

v1−w1µ1−λ1

· · · v1−wmµ1−λm

.... . .

...vp−w1µp−λ1

· · · vp−wmµp−λm

− s

µ1v1−w1λ1µ1−λ1

· · · µ1v1−wmλmµ1−λm

.... . .

...µpvp−w1λ1µp−λ1

· · · µpvp−wmλmµp−λm

−1

v1...

vn

= f (s).

• Redundant data - singular Loewner pencil:

Project using the singular vectors of the Loewner matrix, i.e., [Y, S, X] = svd(L):

{C, E, A, B}︸ ︷︷ ︸model

= {W,−L,−Ls,V}︸ ︷︷ ︸original

⇒︸︷︷︸SVD

{WX,−Y∗LX,−Y∗LsX, Y∗V}︸ ︷︷ ︸reduced

= {Cr , Er , Ar , Br}︸ ︷︷ ︸ROM

.

Wiener models: transfer functions and lifting techniques

LTI g(x)u(t) y (t) = g[x(t)]

u(t) =∑L

`=1 eiω`t x(t) =∑L

`=1 (iω`E− A)−1B︸ ︷︷ ︸G1(iω`)

eiω`t g(x) = Cx + K(x⊗ x).

• The observed output of the generalized Wiener model can be split as follows

y (t) = Cx(t) + K[x(t)⊗ x(t)

]= C

L∑`=1

G1(iω`)eiω`t + K[ L∑

j=1

G1(iωj)eiωj t]⊗[ L∑

h=1

G1(iωh)eiωht]

=L∑`=1

H1(iω`)ejω`t +L∑

j=1

L∑h=1

H2(iωj, iωh)ei(ωj+ωh)t .

• The input-output behavior is characterized by two transfer functions

H1(s1) = C(s1E− A)−1B, H2(s1, s2) = K[(s1E− A)−1B⊗ (s2E− A)−1B

].

Goal: Extend the classical Loewner framework by incorporating data as samples of both functionsH1(s1) and H2(s1, s2); measurements are acquired by simulating the system with oscillatory signals.

The input is u(t) = e2it + e3it ;The output is given byy (t) = H1(2i)e2it + H1(3i)e3it

+ H2(2i , 2i)e4it + H2(3i , 3i)e6it

+ 2H2(2i , 3i)e5it .0 1 2 3 4 5 6 7 8 9 10

Frequency

0

0.2

0.4

0.6

0.8

1

Magnitude

Spectrum of u(t) - INPUT

0 1 2 3 4 5 6 7 8 9 10Frequency

0

0.5

1

1.5

2

Magnitude

Spectrum of y(t) - OUTPUT

Remark: By introducing new state variables (lifting), rewrite the Wiener model as bilinear or QB systems.

x(t) =

[x(t)

x(t)⊗ x(t)

]∈ Rn2+n

ΣW :

{x(t) = Ax(t) + Bu(t),y (t) = Cx(t) + K

[x(t)⊗ x(t)

].

Bilinear

ΣB :

{x(t) = Ax + Nxu(t) + Bu(t),y (t) = Cx.

The system matrices of the lifted bilinear system ΣB can be written as follows [Boyd/Chua ’85]:

A =

[A 00 A⊗ In + In ⊗ A

], N =

[0n 0n,n2

B⊗ In + In ⊗ B 0n2

], B =

[B

0n2

], C =

[C K

].

x(t) =

[x(t)y (t)

]∈ Rn+1

ΣW :

{x(t) = Ax(t) + Bu(t),y (t) = Cx(t) + K

[x(t)⊗ x(t)

].

QB

ΣQB :

{˙x(t) = Ax + Qx(t)⊗ x(t) + Nxu(t) + Bu(t),y (t) = Cx.

The system matrices of the lifted quadratic-bilinear system ΣQB can be written as [Pulch/Narayan ’19]

A =

[A 0

CA 0

], N =

[0n 0n×1

2K(B⊗ In) 0

], Q =

[0n2+n,n2 0n2+n×1

2K(A⊗ In) 01×2n+1

]B =

[B

CB

], C = en+1.

The second (symmetric) transfer function of a quadratic-bilinear system is given by:

HQB2 (s1, s2) =

12

C[(s1 + s2)I− A]−1N[(s1I− A)−1B + (s2I− A)−1B

]︸ ︷︷ ︸

bilinear part

+ C[(s1 + s2)I− A]−1Q[(s1I− A)−1B⊗ (s2I− A)−1B

]︸ ︷︷ ︸

quadratic part

Goal: Approximate nonlinear dynamical systems (bilinear, QB etc.) with block-oriented models (H, W, HW).

Numerical Experiments

A small system with nonlinear rational output [Xiong/Jiang/Schutt-Ainé/Chew ’17]

Time-domain multiplexer

I3(t) = [V1(t)V2(t)− V3(t)]/ZF

The observed nonlinear output (Vi = V )

y (t) =4V (t) + 2V 2(t) + V 3(t)

8− V (t)

=12

V (t) +516

V 2(t) +21128

V 3(t) + ...

Fit a cubic Wiener model (with O(10−4)coeff. approx. error) and compare results

0 0.2 0.4 0.6 0.8 1-0.01

0

0.01

y(t)

The observed outputOriginalWiener

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1Time (t)

10-8

10-6

10-4Error

PDE from [Benner/Breiten ’15]

Consider the viscous Burgers’ equation

∂v (x , t)∂t

+ v (x , t)∂v (x , t)∂x

=∂

∂x

(ν

v (x , t)∂x

),

with boundary conditions ∀t ≥ 0.

v (x , 0) = 0, v (0, t) = u(t), v (1, t) = 0.

——————————————————–

First experiment n = 100, ν = 0.1100 interpolation points in (101, 102)i

10 20 30 40 50 60 70 80 90 100

10-15

10-10

10-5

100 Singular value decay of the Loewner matrix

Second experiment n = 50, ν = 0.0280 interpolation points in (10−3, 103)i .

10 20 30 40 50 60 70 80

10-15

10-10

10-5

100 Singular value decay of the Loewner matrix

—————————————————-QB system n = 100, Wiener r = 50Bilinear system of order nb = 10100

0 0.2 0.4 0.6 0.8 1 1.2Time (t)

-0.04

-0.02

0

0.02

0.04

0.06

y(t

)

The observed output

OriginalBilinearWiener

QB system n = 50, Wiener r = 24Bilinear system of order nb = 2550

0 0.5 1 1.5 2 2.5Time (t)

0

0.02

0.04

0.06

0.08

y(t

)

The observed output

OriginalBilinearWiener

Spatial discretization with a step size h =1/(n + 1) produces a QB system

Rewrite as:x(t) = Ax(t)+Qx(t)⊗x(t)+Nx(t)u(t)+Bu(t)——————————————————–The control input is a multi-tone signalu(t) =

∑4k=1 cos(5πkt)

0.2 0.4 0.6 0.8 1 1.2Time (t)

10-7

10-5

10-3

Approximation errorBilinearWiener

The control input is u(t) = e−t cos(4t).

0.5 1 1.5 2 2.5Time (t)

10-7

10-4

10-1 Approximation error

BilinearWiener