(c) 2004-2013, ProSensus, Inc.
Optimization of Processes & Products using
Historical Data
John MacGregor
Distinguished University Professor Emeritus, McMaster University
Chairman, ProSensus Inc.
FOCAPO/CPC 2017, Jan. 8-12, 2017
Mark-John Bruwer
CTO, ProSensus Inc. (now with Aspentech)
(c) 2004-2013, ProSensus, Inc.
ProSensus – changes have occurred
• ProSensus Software Assets recently Purchased by Aspentech
– ProMV: BIG data analytics focussed on process industries
• Analysis and diagnosis of historical process data (batch & continuous)
• Robust multivariate LV models for inferentials/soft sensors
• Building and simulating multivariate monitoring plots
• Optimization of processes using models built from historical data
– ProMVOnline : Monitoring/soft sensors for continuous processes
– ProBatchOnline: Monitoring/soft sensors for batch processes
– ProBatchControl: MPC of final quality for batch processes
• ProSensus today:
– Engineering consulting on the analysis and optimization of processes
– ProVision: Industrial vision systems for monitoring and control
– ProFormulate: Rapid development of new products
(c) 2004-2013, ProSensus, Inc.
Outline of this presentation
• No new optimization methods
• Focus is on:
– What can be done using historical data collected from processes and with R&D data
– Optimization using empirical models
– Industrial examples
1. Optimization of processes using models built from historical data
• Latent variable models
• Uniqueness and causality issues
• Optimization of batch and continuous processes from historical data
2. Optimization of products
• Development of new products
– Brings modeling, multivariate DOE and optimization into R&D
(c) 2004-2013, ProSensus, Inc.
Concept of latent variables
Measurements are available on K physical variables: matrix=X
But, the process is actually driven by small set of “A” (A « K) latent
variables t1, t2, … tK (T).
– Raw material variations
– Equipment variations
– Environmental (temp, humidity, etc.) variations
K columns
= X
(c) 2004-2013, ProSensus, Inc.
Projection of data on a lower dimensional latent variable space (T)
Latent variable spaceMeasured variables
TX
t1
t2
•SPE and T2 distances provide information on differences among observations
•Analysis, Monitoring, Control and Optimization done in latent variable space
Window into process
(c) 2004-2013, ProSensus, Inc.
Latent variable regression models (PLS)
Two data matrices: X and Y
Choice of X and Y variables depend mainly upon the objectives of the study.
Symmetric in X & Y
PLS provides models for both X and Y and relationship between X and Y (Maximizes Cov(X,Y)).
X = TPT + E Y = TCT + F
TX Y
(c) 2004-2013, ProSensus, Inc.
Optimization
• Using theoretical models
– Optimization based around fundamental models is widespread throughout the process industries
• Off-line optimization (eg in design)
• Real-time optimization
• Using empirical models
– What if no fundamental model exists or only partial theoretical information is available?
– Can we make use of the abundant historical data to optimize processes?
• For the active use of empirical models (eg. optimization and control), we need causal models.
– Causality for any active changes in the MV’s the model will predict the effect on the y’s
(c) 2004-2013, ProSensus, Inc.
Causality in Empirical Models
• In the passive use of empirical models no causality is required• Model use only requires that future data have the same correlation structure
– Eg. Calibration; soft sensors; process monitoring
– Historical data is great for these
• But for active use of models as in optimization and control, one needs causal models
– For empirical models to be causal in certain x-variables – we need to have independent variation (DOE’s) in those x’s.
• Such data is often not readily available in industry.
– But industry has massive amounts of undesigned historical data.
• Causal models for the effects of individual x’s on the y’s cannot generally be extracted from such historical data
• All classical regression models (eg MLR, ANN, …) provide no causal information from such data
– Due to extreme correlation among the variables, these regression approaches provide an infinite number of indistinguishable solutions.
(c) 2004-2013, ProSensus, Inc.
Causality in Latent Variable models
• LV models do provide unique and causal models
– Why? Because they simultaneously model both the X and Y spaces
• But they provide causality only in the low dimensional LV space
– i.e. if we move in LV space (t1=w1*x, t2=w2*x, …) we can predict the causal effects of these moves on Y (Ypred= TCT)
– Can use this fact together with the model of the X-space (X = TPT) to perform optimization and control in the LV spaces
– Now illustrate this idea of causality and optimization with some industrial examples of batch processes.
(c) 2004-2013, ProSensus, Inc.
Optimization using batch production data (herbicide)
– Initial chemistry and charges (Z)
– Trajectories on 10 process variables (X)
– 12 final quality measures (Y)
– >200,000 measurements
Z X Y
Initial Conditions Variable Trajectories
End Properties
variables
bat
ches
LV model captures all relevant variation in Z, X and Y in 2 LV’s
Want to optimize y’s using all of Z and X.
(c) 2004-2013, ProSensus, Inc.
Batch-Wise Unfolding of the data array
– The batch trajectory observations are extracted horizontally in a time-wise fashion
– Append both Quality (Y) and Initial condition data (Z)
– Each batch becomes a single row of data in the model
– PLS models built on these unfolded data matrices
Variable 1 Variable 2 Variable 3 Variable 4 Variable KBatch
Trajectory Data
Variables, K
N Batches
Time, T
Unfolded trajectory dataN Batches
Variables x Time, TxK
YZ
(c) 2004-2013, ProSensus, Inc.
Trajectory alignment
(c) 2004-2013, ProSensus, Inc.
Analysis of Z block
Score plotAnalysis reveals obvious clustering
among on- and off-spec batches.
The model also allows an understanding
of why off-spec batches occur.
The critical to quality variables were
process related, not chemistry related.
tz1
tz2
VIP’s for Z model
(c) 2004-2013, ProSensus, Inc.
Analysis of unfolded X trajectories
• Loading vector w*1
for X model(Note the time varying w*s – captures the nonlinear time-varying behavior of the batch)
Score plot for X
• Clustering of good/bad batches
•Good batches have high t1
•Loadings w1* show that for high t1:a. Low tank level and used time for
entire batch
b. High Press, Jtemp, Dtemp in first
half only
(c) 2004-2013, ProSensus, Inc.
Model Explore Tool in ProMV
• location on score plot
• save data from current pointer position
• view different data blocks
• X & Y values at the location on the score plot
• view different data blocks
• view the variables scaled
(c) 2004-2013, ProSensus, Inc.
Often have multiple blocks of process data
• Latent variable software (ProMV) easily handles such multiple data blocks– Which blocks affect Y most (BIP) ?
– Which variables in each block are most important to Y?
– Get monitoring (MSPC) models for each block and the overall evolving process.
– Optimization in any of the blocks to maximize yield and quality
Z2 X1 Y
Initial Conditions
Batch Variable Trajectories
End product
quality
Z1
bat
ches
X3X2 X4 X5 X6
Intermediate processing steps
(c) 2004-2013, ProSensus, Inc.
Ex. 2: Optimization of a Batch Polymerization
Joint study with Air Products and Chemicals.
(c) 2004-2013, ProSensus, Inc.
Data after alignment
(c) 2004-2013, ProSensus, Inc.
Batch polymerization (Air Products & Chemicals)
• 13 variables in YDesire a new product with the following final quality attributes (Y’s):
• Solution– Build batch PLS latent variable model on existing data (Z, X, Y)
– Perform an optimization in LV space to find optimal LV’s
– Use LV model of X-space to find the corresponding recipes and process trajectories
Maintain in normal ranges: Y1 Y2 Y3 Y4 Y5 Y6 Y8
Constraints: Y7 = Y7des
Y9 = Y9des
Y10< Y10const
Y11< Y11const
Y12 < Y12const
Y13 < Y13const
… and with the minimal possible batch time (*)
(c) 2004-2013, ProSensus, Inc.
Optimization was initially done as a 2 step
procedure (Sal Garcia)
• Step 1: Solve for with constraints on T2 and on y’s
• Step 2: Solve for xnew that yields subject to certain constraints on SPE and x’s.
T T
* * * *new newnew 2 new new new new new new
new
minW W PW PW x τ G x τ x x Λ x x ηx
x
ˆnew
ˆnew
2
,T
des xnew 1 des xnew 21
ˆmin τˆ ˆ( ) ( ) ρ
ˆ
.
ˆ
Axnew a
axnew as
s t
xnew
y Q τ G y Q ττ
BQ τ b
Aspen-ProMV does this in one step by projecting all the constraints into the LV
space and optimizing in the constrained LV space.
(c) 2004-2013, ProSensus, Inc.
All solutions satisfy the requirements on ydes
Case 1 to 5: weight on time-usage is gradually increased
Different solutions: change the penalty for total time
(c) 2004-2013, ProSensus, Inc.
ProFormulate: Modeling & Optimization in
R&D
Rapid development of new products
(c) 2004-2013, ProSensus, Inc.
Motivation
• Innovation is important: 93% of executives surveyed say their companies depend on innovation for long-term success1
• Current innovation strategies aren’t working: only 18% of executives believe their company derives a competitive advantage from its innovation strategy1
• The situation is getting worse: executives are less satisfied with innovation performance today than they were in 20091
• Product development at many big companies is characterized by:
– Almost trial and error approach
– Long development times
– Intuitive experience of individual researcher is often only way past information is carried forward (little information from other developers)
– No model-based optimization approaches used.
1Accenture Report: Why Low-Risk Innovation Is Costly
http://www.accenture.com/us-en/Pages/insight-low-risk-innovation-costly.aspx
(c) 2004-2013, ProSensus, Inc.
Ex. Development of a new biologic media for mAb
• This example had 65 raw materials variables from 13 categories.
• Trial history (in 2-D LV space) shows typical clusters of experiments.
(c) 2004-2013, ProSensus, Inc.
What do we mean by Product Development?
• We mean:– Developing new chemical/food products for new
applications
– Improvement / reformulation of existing products
– Make existing products using alternative materials at lower cost
– Consolidation of products using fewer raw materials
(c) 2004-2013, ProSensus, Inc.
Degrees of Freedom in Product Development
RawA
RawB
RawC
……
Chemical
Molecular
Appearance
Texture
Taste
RawA
RawC
40%
30%
30%RawP
RawZ
e.g.
Flowrates
temperature
mixing
Desired properties
To capture the synergisms among these, they must all be considered
simultaneously in the experimentation, modeling and optimization
Otherwise many experiments & long development times
Formulation
ratios
Selection of
raw materials
Process
conditions
Final product
properties
(c) 2004-2013, ProSensus, Inc.
Product Development: An Integrated Program
1. Set up and maintain good databases• Never lose past development data
• Take sufficient measurements
2. Use multivariate models to continually extract the information from all past and new data
• Provides a repository for all the company’s development knowledge
• Continually add to and update models
3. Design Experiments to augment existing knowledge• Don’t start over with each new project - augment knowledge as needed
• DOE’s in LV space
4. Reformulate products via optimization in LV space• To achieve products with desired quality attributes, at lowest cost
(c) 2004-2013, ProSensus, Inc.
Some ProSensus Product Development Projects
• Srixon golf ball cores – (Mitsubishi Chemicals/Sumitomo))
• Functional polymers for medical devices – (Mitsubishi Chemicals)
• High performance polymeric coatings
• Muffin batters – (Pepsico Foods)
• Cake compounds
(c) 2004-2013, ProSensus, Inc.
Mitsubishi Chemicals example
Rubbers
Polypropylene
Oil
Functional
Polymers
Select materials from each class, their
blend ratios and the process conditions
Critical!
Pro
cess c
onditio
ns*
Develop functional polymers for medical devices, golf balls, etc
(Compounding problem)
(c) 2004-2013, ProSensus, Inc.
Expt
No.
MaterialFinal
properties
Ru
bb
er
pro
pe
rty R
ub
be
rs
Rubberproperty
T
oilX
proY
Oil
pro
pe
rty
PP
pro
pe
rty
T
rubberX
T
PPX
Rubber
Oil
PP
DBrubber_X
Oils
Oilproperty
DBoil_X
DBPP _X
PPproperty
PP
Raw material property data
Formulations
Partial data from database of
raw material properties
rubberR
Raw material
property data
(subset)
oilR PPR
XDB
Data structure
Process
conditions
Z
(c) 2004-2013, ProSensus, Inc.
00
01
10,1
0)ˆ(
..
)()(
,
,
,
1
,
1
2
,2
1
2
1
3
1
,21
jnew
jnew
j
jnew
NN
j
jnew
A
a a
anew
new
Kk
k
newmixnewmixnew
DBnewnewmix
NN
j
j
NN
j
jjnewPLSnewmixdes
T
PLSnewmixdesr
r
r
rr
consts
T
xxSPE
rx
ts
wcrwBxyWBxyMinnew
X
Mixture
constraint
PLS model
constraint
Ideal
mixing rule
Binary variable
constraint
00
01
10,1
0)ˆ(
..
)()(
,
,
,
1
,
1
2
,2
1
2
1
3
1
,21
jnew
jnew
j
jnew
NN
j
jnew
A
a a
anew
new
Kk
k
newmixnewmixnew
DBnewnewmix
NN
j
j
NN
j
jjnewPLSnewmixdes
T
PLSnewmixdesr
r
r
rr
consts
T
xxSPE
rx
ts
wcrwBxyWBxyMinnew
X
Mixture
constraint
PLS model
constraint
Ideal
mixing rule
Binary variable
constraint
Estimation error
Total
material cost
The number
of materials
Optimized variables:
The mixture ratios of all
possible raw materials available
in the database XDB, and
process conditions (Z)
Optimization using the models
(c) 2004-2013, ProSensus, Inc.
DOE’s for Product Development
• Often industrial data bases are very rich but only in limited regions/clusters
– Limited choices of materials
– Limited formulation ratios
– Limited processing conditions
• Optimal DOE’s can be used to provide a small number of runs that can:
– Upgrade these databases for better development
– Optimize for a particular product
(c) 2004-2013, ProSensus, Inc.
Concept of DOE in latent variable spaces
• Points show existing products
• Note regions of latent variable space where are no data
• Optimal DOE’s to find those scores (t1, t2, ….) that would fill in these holes
Latent variable
space
1
235
67
89 10
1112
13
14
15
18
19
20
23
24
2526
27
29
31
3233
34
35
3738
39
42
43
44
t1
t2
(c) 2004-2013, ProSensus, Inc.
DOE in latent variable spaces (to improve DB)
• Experiments ( ) in score space
• From the DOE in the scores (t1, t2, …) use LV model to provide corresponding DOE in the raw materials, formulations and processing conditions: [Z, X, R]
• DOE in low dimensional score space provides a corresponding DOE in the high dimensional original variable space
1
235
67
89 10
1112
13
14
15
18
19
20
23
24
2526
27
29
31
3233
34
35
3738
39
42
43
44
t1
t2
Latent variable
space
(c) 2004-2013, ProSensus, Inc.
Summary
• Latent Variable models have many advantages with “BIG” historical data:
– Easily handle many variables and missing data
– Models are unique and interpretable
– Causality in the Latent Variable space
• Useful for optimization of processes from historical data
– Allows for more powerful use of BIG process data
– Combination with fundamental models
• Useful for rapid development of new products
– Brings Multivariate modeling, DOE and optimization into R&D