Diagnostic Solvers for Linear Systems with Constraints

by

Rondall E. JonesSandia National Labs, Retired www.rejonesconsulting.com

rejones7@msn.com

Presented by

Kevin DowdingSandia National Labs

Equation ContextWe are concerned here with the general linear algebra problem:

Ax = b = b0 + e

(b0 is an unknown true data vector;

e is an unknown vector of errors 0 or more equations; least squares if over-determined; may be ill-conditioned and/or singular; any shape) Ex == f (0 or more equations; exact equality or else discarded) Gx >= h (0 or more equations; <= must be recast as >=)

Background• rejtrix.h C++ library (free on author’s web site)

capably solves ill-conditioned linear systems with constraints, including auto-regularization based on the Discrete Picard Condition.

• But complete apps were needed for some users. • We choose Excel-compatible *.csv files format. • Result was an easy-to-use suite of solvers for

various weightings, manual vs auto, etc.• But…• Users wanted more information about the problem.• So more diagnostic outputs were added.• These Solve* programs are free on author’s web

site.

The solve* download page: (partial)

Each solve* program uses about the same format for output.

– Output varies a little with the input data and the solution method chosen

– Some analyses are not possible in some circumstances– For example, some only make sense if the equations

are over-determined.– Also some may only make sense if the user provides

error estimates.

Sample *.csv Input File• # Will use several problems for illustrations.• # These lines starting with # are comments.• # Note use of commas in input file is optional.• # ‘=’ flag means least-squares equation; ‘!’ means ‘exact’.• # (‘<‘ and ‘>’ are for inequalities.)• # Last column is error estimates; use 0.0 if not available.• # ‘NONNEG’ shorthand adds all non-negativity constraints.• # We have 8 lines of data and 6 unknowns…• 8 6• = 1.325,1.265, 1.21, 0.545, 1.06, 1.385 = 112.35 1.5729 • = 1.0 2.4 8.8 0.0 36.0 0.408 = 605.68 8.47952 • = 1.5 1.75 7.0 0.0 6.0 8.0 = 325.0 4.55 • = 0.4 3.0 3.7 10.0 4.4 5.2 = 410.0 5.74 • = 1.0 1.725 0.915 0.4 1.02 3.185 = 117.35 1.6429 • = 5.35 8.7 7.7 18.9 6.0 10.0 = 938.1 13.1334• ! 1 1 1 1 1 1 = 100 0• NONNEGNote: This mildly ill-conditioned problem is from a human enzyme profile

prediction study done by Dr. Drew Jones, UAMS. All enzyme levels must be non-negative, and add to 100%.

The “advise.exe” program If you don’t know where to start, start with advise.exe.Here is the output of advise.exe for a sample problem:

We begin with the following usual, straightforward reports…

An Echo of Input Problem(some columns collapsed for this display)

Then the solution…

In Excel it takes only a couple of clicks to make a graph of the solution…

Next an analysis of the residual

Note summary of constraint behavior at end. These comments will vary depending on the situation.

Next Some Error Analyses…

First the results of a Monte Carlo error analysis…

To do this, we generate a number of random variations using the user’s error estimates, then discard the high and low 2.5% of each variable. We report the extremes of the remaining values.Note that the effect of ALL CONSTRAINTS is implicitly included.

An Analysis of Consistency in the Matrix and RHS Vector

• Standard least squares minimizes the norm of the residual vector.

• This residual vector can be viewed as the set of changes required to the RHS to make the system of equations consistent.

• Total least squares can be viewed as computing the smallest set of changes to both A and b that are required to make the system consistent.

• So we compute the TLS solution and display to the user the largest of the “inconsistencies” the TLS suggests.

An Example of Consistency Analysis

Note: this data is from a different problem than most of the above slides, for better illustration.

Note: The constraints (equality and inequality) do not play a role in this analysis.

A Different Analysis of the Matrix…(this applies only to overdetermined systems)

• An old way of estimating the relative variance of the solution elements (ignoring right-hand-side issues) was to look at the diagonal elements of (ATA)-1 .

• We can use the diagonal of (ATA)-1 or some function of it as a heuristic measure of model quality in a single number. (We use the square root of the average variance).

• Then, we can re-measure this heuristic with an equation deleted.

• If the heuristic doesn’t increase, the equation may be irrelevant.

• If the heuristic increases greatly, the equation is crucial.• We call the increase in the heuristic the ‘importance’ of the

equation.

Here is an Example of the “Importance” Analysis(Equation 6 may be almost useless)

Note: this data is from a different problem than most of the above slides.

Question: when might the “Importance” measure be negative?That is, when would the “variances” be much smaller when an equation is removed??

One Common Answer:

When the system of equations is square or under-determined, and the approximate rank is one less than the row count. Then the only near-zero singular value may disappear when a particular row, or any of several rows, is deleted.

Linear Dependence Analysis (Examination of the U Matrix can reveal clear dependencies)

Note: this data is from a different problem than most of the above slides, for better illustration.

Note: The constraints (equality and inequality) do not play a role in this analysis.

Note: The programs will report several such sets of dependences, if present. In this case, the two rows are evidently the same (or differ by simply a scale factor.)

Next are Some Correlations…(which may interest some users)

Correlation of Solution with Rows of Matrix…

Ideally the solution should be a linear combination of the rows of the matrix. It may be worthwhile to know how each row correlates to the solution. We depict this relation as an angle, using v · w = |v| |w| cos(angle between v and w).

Correlation of RHS with Columns of Matrix…

Equally, the right hand side data should be a linear combination (given by the solution) of the columns of the matrix.So we depict that relationship by angles between the columns and the RHS.

And there may be other such angle relationships depicted also.

Now We Give Information About the Unregularized and Unconstrained Solution by SVD (aka the Pseudo-Inverse Solution)

in the next few displays…

We begin with the Singular Values

Next is the Vector UT *b…

And then the Vector S+ UT b …which is also known as the Discrete Picard Condition Vector

Notice beginning of divergence toward the end of the vector.

And the (Unregularized) Pseudo-Inverse Solution x= V S+ UT b

Now, the final solution after all regularization and constraints can be multiplied by VT to show

the Modification to the Picard Condition Vector…

Compare this vector to the D.P.C.V. two slides back to see how the regularized and constrained solution behaves versus the pseudo-inverse solution. The latter elements behave better here.

Finally, Some Measures of Quality of the SVD

Orthonormality error refers to the elements of the matrix UTU-I, for example. This matrix should be zero, within round-off.