Pat LangleySchool of Computing and Informatics

Arizona State UniversityTempe, Arizona

Computational Discovery of Explanatory Process Models

Thanks to K. Arrigo, D. Billman, S. Borrett, M. Bravo, W. Bridewell, S. Dzeroski, and L. Todorovski for their contributions to this research, which is funded by a grant from the National Science Foundation.

Computational Models of Scientific Discovery

numeric laws in physics and chemistry (Langley et al., 1983)

qualitative conjectures in number theory (Colton et al., 2000)

structural models of molecules (Langley et al., 1987)

causal models of urea biochemistry (Kulkarni & Simon, 1988) reaction pathways in catalytic chemistry (Valdes-Perez, 1994)

The past three decades have seen multiple accounts of scientific discovery cast in terms of problem-space search, including:

This work has focused on discovering knowledge cast in the same formalisms as used by scientists themselves.

Ecosystem Dynamics in the Ross Sea

d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto

d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo

d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus

d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus

Formal accounts of ecosystem dynamics are often cast as sets of differential equations.

Here four equations describe the concentrations of phytoplankton, zooplankton, nitrogen, and detritus in the Ross Sea over time.

Such models can match observed variables with some accuracy.

A Deeper Account of Ross Sea Dynamics

d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto

d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo

d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus

d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus

As phytoplankton uptakes nitrogen, its concentration increases and the nitrogen decreases. This continues until the nitrogen is exhausted, which leads to a phytoplankton die off. This produces detritus, which gradually remineralizes to replenish nitrogen. Zooplankton grazes on phytoplankton, which slows the latter’s increase and also produces detritus.

Processes in Ross Sea Dynamics

d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto

d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo

d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus

d[nitro,t,1] = - 0.098 0.411 phyto + 0.005 detritus

As phytoplankton uptakes nitrogen, its concentration increases and the nitrogen decreases. This continues until the nitrogen is exhausted, which leads to a phytoplankton die off. This produces detritus, which gradually remineralizes to replenish nitrogen. Zooplankton grazes on phytoplankton, which slows the latter’s increase and also produces detritus.

Processes in Ross Sea Dynamics

d[phyto,t,1] = 0.307 phyto 0.495 zoo + 0.411 phyto

d[zoo,t,1] = 0.251 zoo + 0.615 0.495 zoo

d[detritus,t,1] = 0.307 phyto + 0.251 zoo + 0.385 0.495 zoo 0.005 detritus

d[nitro,t,1] = 0.098 0.411 phyto + 0.005 detritus

As phytoplankton uptakes nitrogen, its concentration increases and the nitrogen decreases. This continues until the nitrogen is exhausted, which leads to a phytoplankton die off. This produces detritus, which gradually remineralizes to replenish nitrogen. Zooplankton grazes on phytoplankton, which slows the latter’s increase and also produces detritus.

A Process Model for the Ross Sea

model Ross_Sea_Ecosystem

variables: phyto, zoo, nitro, detritusobservables: phyto, nitro

process phyto_loss equations: d[phyto,t,1] = 0.307 phyto

d[detritus,t,1] = 0.307 phyto

process zoo_loss equations: d[zoo,t,1] = 0.251 zoo

d[detritus,t,1] = 0.251 zoo

process zoo_phyto_grazing equations: d[zoo,t,1] = 0.615 0.495 zoo

d[detritus,t,1] = 0.385 0.495 zood[phyto,t,1] = 0.495 zoo

process nitro_uptake equations: d[phyto,t,1] = 0.411 phyto

d[nitro,t,1] = 0.098 0.411 phyto

process nitro_remineralization; equations: d[nitro,t,1] = 0.005 detritus

d[detritus,t,1 ] = 0.005 detritus

We can formalize these links by recasting the equations as a quantitative process model.

Such a model is equivalent to a standard differential equation model, but it makes explicit assumptions about processes that are involved.

Each process indicates that certain terms in equations must stand or fall together.

process exponential_growth variables: P {population} equations: d[P,t] = [0, 1,] P

process logistic_growth variables: P {population} equations: d[P,t] = [0, 1, ] P (1 P / [0, 1, ])

process constant_inflow variables: I {inorganic_nutrient} equations: d[I,t] = [0, 1, ]

process consumption variables: P1 {population}, P2 {population}, nutrient_P2 equations: d[P1,t] = [0, 1, ] P1 nutrient_P2, d[P2,t] = [0, 1, ] P1 nutrient_P2

process no_saturation variables: P {number}, nutrient_P {number} equations: nutrient_P = P

process saturation variables: P {number}, nutrient_P {number} equations: nutrient_P = P / (P + [0, 1, ])

Inductive Process Modeling

model AquaticEcosystem

variables: nitro, phyto, zoo, nutrient_nitro, nutrient_phytoobservables: nitro, phyto, zoo

process phyto_exponential_growth equations: d[phyto,t] = 0.1 phyto

process zoo_logistic_growth equations: d[zoo,t] = 0.1 zoo / (1 zoo / 1.5)

process phyto_nitro_consumption equations: d[nitro,t] = 1 phyto nutrient_nitro, d[phyto,t] = 1 phyto nutrient_nitro

process phyto_nitro_no_saturation equations: nutrient_nitro = nitro

process zoo_phyto_consumption equations: d[phyto,t] = 1 zoo nutrient_phyto, d[zoo,t] = 1 zoo nutrient_phyto

process zoo_phyto_saturation equations: nutrient_phyto = phyto / (phyto + 0.5)

HeuristicSearch

observations

generic processes

process model

phyto, nitro, zoo, nutrient_nitro, nutrient_phyto

variables

Generic Processes for Aquatic Ecosystems

generic process exponential_loss generic process remineralization variables: S{species}, D{detritus} variables: N{nutrient}, D{detritus} parameters: [0, 1] parameters: [0, 1] equations: d[S,t,1] = 1 S equations: d[N, t,1] = D

d[D,t,1] = S d[D, t,1] = 1 D

generic process grazing generic process constant_inflow variables: S1{species}, S2{species}, D{detritus} variables: N{nutrient} parameters: [0, 1], [0, 1] parameters: [0, 1] equations: d[S1,t,1] = S1 equations: d[N,t,1] =

d[D,t,1] = (1 ) S1d[S2,t,1] = 1 S1

generic process nutrient_uptake variables: S{species}, N{nutrient} parameters: [0, ], [0, 1], [0, 1] conditions:N > equations: d[S,t,1] = S

d[N,t,1] = 1 S

Our current library contains about 30 generic processes, including ones with alternative functional forms for loss and grazing processes.

These form the building blocks from which to compose models.

An Approach to Process Model Construction

1. Instantiate known generic processes with specific variables, subject to type specifications;

2. Combine these instantiated processes into candidate model structures, limited by size and connectivity;

3. For each model structure, carry out search through parameter space to find good coefficients;

4. Return the parameterized model with the best overall score (e.g., lowest squared error).

We have developed a number of “IPM” systems that construct process models from generic components in four stages:

We have reported variants on this approach in numerous papers (Todorovski et al., AAAI-2005; Bridewell et al., MLj, 2008).

Estimating Parameters in Process Models

1. Selects random initial values that fall within ranges specified in the generic processes;

2. Improves these parameters using the Levenberg-Marquardt method until it reaches a local optimum;

3. Generates new candidate values through random jumps along dimensions of the parameter vector and continue search;

4. If no improvement occurs after N jumps, it restarts the search from a new random initial point.

To estimate the parameters for each generic model structure, the IPM algorithm:

This multi-level method gives reasonable fits to time-series data from a number of domains, but it is computationally intensive.

Results on Training Data from Ross Sea

We provided IPM with 188 samples of phytoplnkton, nitrate, and ice measures taken from the Ross Sea.

From 2035 distinct model structures, it found accurate models that limited phyto growth by the nitrate and the light available.

Some high-ranking models incorporated zooplankton, whereas others did not.

Results on Test Data from Ross Sea

Generalization to a second year’s data benefited from treating initial zooplankton concentration as a free model parameter.

Another good-fitting model suggested that the nitrogen to carbon ratio varies as a function of available light.

Results on a Protist Ecosystem

We also ran the system on protist data from Villeaux (1979), using 54 samples of two variables (P. aurelia and P. nasutum).

In this run, IPM considered a space of 470 distinct model structures and reproduced basic trends.

Results on Rinkobing Fjord

Data from a Danish fjord included measurements on fjord height, sea level, water inflow, and wind direction and speed.

We used 1100 samples for training and 551 samples for testing over a space of 32 model structures.

Results on Biochemical Kinetics

We also ran IPM on 14 samples of six chemicals involved in glycolysis from a pulse response study.

Here the system considered some 172 model structures. The best model fit the data but reproduced only part of the known pathway.

Results on Battery Data from Space Station

Data from the Space Station batteries included current, voltage, and temperature, with resistance and state of charge unobserved.

We used 6000 samples for training and 2640 samples for testing over a space of 162 model structures.

specify a quantitative process model of the target system;

display and edit the model’s structure and details graphically;

simulate the model’s behavior over time and situations;

compare the model’s predicted behavior to observations;

invoke a revision module in response to detected anomalies.

Because few scientists want to be replaced, we are developing an interactive environment, PROMETHEUS, that lets users:

The environment offers computational assistance in forming and evaluating models but lets the user retain control.

Interfacing with Scientists

For more details about PROMETHEUS, see Bridewell et al. (K-CAP-2007).

Viewing a Process Model Graphically

Viewing a Process Model as Equations

Adding a Process Manually

Requesting Automatic Model Revision

Results of Automatic Model Revision

liked the ability to alter models at an abstract process level;

appreciated the ability to simulate models and plot variables;

believed the environment would be a good teaching tool;

had some interference from STELLA’s graphical notation;

felt system would be more useful if it supported PDE models;

indicated that some model structures were implausible.

An Initial User Study

We asked three oceanographers to use PROMETHEUS to revise a model of the Ross Sea ecosystem; this study revealed that they:

We are addressing the last two issues in our next version of the modeling environment.

Knowledge and Search in Science

Traditional treatments of problem solving hold that knowledge reduces search.

But adding generic processes leads to a combinatorial increase     in the number of model structures.

Yet scientists are not overwhelmed by the size of their model spaces and they reject many model structures as unacceptable.

This suggests two distinct forms of scientific knowledge –    components of models and constraints on their combination.

This distinction seldom occurs in the literature, but it appears central to understanding scientific explanation.

Inducing Process Models with Constraints

represent modular constraints on process combinations

use these constraints to eliminate unacceptable models

reduce search through the model space, which

leads to far more efficient model construction

produces little or no increase in generalization error improves the comprehensbility of generated models

We have extended our framework for process model induction to:

The resulting system (SC-IPM) offers a more complete account of explanatory scientific discovery.

Four Types of Process Constraints

two processes must always occur together producer(P) growth(P), loss(P)

exactly one of a set of processes must be in a model producer(P), grazer(G) lotka-volterra(P, G), ivlev(P, G), watts(P, G)

at most one of a set of processes can appear in a modelproducer(P), energy(E) photoinhibition(P, E)

processes must always appear in a model nutrient(N), detritus(D) nutrient-mixing(N), remineralization(N, D)

The SC-IPM system supports four distinct types of constraints:

We hypothesize that these constraint types are sufficient to guide process modeling in many scientific domains.

no constraintsconstraints

R2 = 0.92R2 = 0.97

vari

anc

e e

xpla

ine

dExperimental Benefits of Constraints

computational scientific discovery (e.g., Langley et al., 1983)

qualitative physics and simulation (e.g., Forbus, 1984)

constraint-satisfaction methods (Freuder & Macworth, 1994)

languages for scientific simulation (e.g., STELLA, MATLAB)

interactive tools for data analysis (e.g., Schneiderman, 2001)

Intellectual Influences

Our approach to computational discovery incorporates ideas from many traditions:

However, it combines these ideas in novel ways to assist in the construction of explanatory scientific models.

Directions for Future Research

produce additional results on other scientific data sets

develop more efficient methods for fitting model parameters

extend framework to handle partial differential equations

explore evaluation metrics like match to trajectory shape

use knowledge of subsystems to support large-scale modeling

improve usability of the PROMETHEUS modeling environment

Despite our progress to date, we need further work in order to:

Taken together, these will make inductive process modeling a more robust approach to scientific model construction.

Key Contributions

incorporates a formalism that is familiar to many scientists;

takes into account background knowledge about the domain;

produces meaningful results from moderate amounts of data;

generates models that explain rather than describe observations;

provides an interactive environment for model construction.

In summary, our work on computational model construction has produced an approach that:

Our IPM systems search differently from human scientists, but they rely on the same kinds of processes and constraints.

End of Presentation

Results with Inductive Process Modeling

acquatic ecosystems protist dynamics

hydrology biochemical kinetics

The Task of Inductive Process Modeling

We can use these ideas to reformulate the modeling problem:

Given: A set of variables of interest to the scientist;

Given: Observations of how these variables change over time;

Given: Background knowledge about plausible processes;

Find: A process model that explains these variations and that generalizes well to future observations.

Background knowledge takes the form of generic processes that provide the building blocks for models.

Contributions of the Research

incorporates a formalism that is familiar to many scientists;

takes into account background knowledge about the domain;

produces meaningful results from small amounts of data;

generates models that explain rather than describe observations;

provides an interactive environment for model construction.

In summary, our work on computational model construction has produced an approach that:

We need much more research in computational systems science that addresses these challenges.