winding stairs: a sampling tool to compute sensitivity indices

Statistics and Computing (2000) 10, 187–196

Winding Stairs: A sampling toolto compute sensitivity indices

KAREN CHAN, ANDREA SALTELLI and STEFANO TARANTOLA

Institute for Systems, Informatics and Safety, European Commission Joint Research Centre, [email protected]

Accepted October 1999

Sensitivity analysis aims to ascertain how each model input factor influences the variation in themodel output. In performing global sensitivity analysis, we often encounter the problem of selectingthe required number of runs in order to estimate the first order and/or the total indices accuratelyat a reasonable computational cost. The Winding Stairs sampling scheme (Jansen M.J.W., RossingW.A.H., and Daamen R.A. 1994. In: Gasman J. and van Straten G. (Eds.), Predictability and NonlinearModelling in Natural Sciences and Economics. pp. 334–343.) is designed to provide an economic wayto compute these indices. The main advantage of it is the multiple use of model evaluations, hencereducing the total number of model evaluations by more than half. The scheme is used in three simu-lation studies to compare its performance with the classic Sobol’ LPτ . Results suggest that the JansenWinding Stairs method provides better estimates of the Total Sensitivity Indices at small sample sizes.

Keywords: first order indices, Sensitivity Analysis, Sobol’ LPτ sequences, Total Sensitivity Indices,Winding Stairs sampling scheme

1. Introduction

This paper describes a sampling scheme used to perform aMonte-Carlo based sensitivity analysis (SA) of model output.In general, SA is conducted by the following steps:

(i) defining a model which approximates economic, engineer-ing, environmental, physical or social phenomenon of var-ious levels of complexity, and its input factors and outputvariable(s);

(ii) assigning probability distributions to the input factors;(iii) generating input values via an appropriate random sampling

method and evaluating the output; and(iv) assessing the influence or relative importance of each input

factor on the output variable.

The focus of this article is on steps (iii) and (iv). First we willintroduce some standard notation.

We assume a model output Y is dependent on k input factors,say x1, x2, . . . , xk , namely

y = f (x1, x2, . . . , xk).

Difficulty can arise from within a modelling process when themodel factors vary about nominal values in some manner whichis unknown to us. In this situation we model uncertainties about

the input factors by treating them as random variables. Hence,we often face the problem of what values to use for the inputfactors when we investigate real world phenomena with numer-ical experiments using a mathematical model. In SA, severalvalues of x= (x1, x2, . . . , xk), say x1, x2, . . . , xn , are generatedas successive sets of inputs in order to obtain the desired andaccurate information concerning Y .

For some complex models, the computational cost can be ex-pensive in that a single set of input values may require severalminutes in order to yield one output. In practice, the samplesize, n, should be large enough to provide accurate informa-tion about the relationships between the input factors and themodel output, while it should be small enough to minimise thecomputational cost. Several techniques to generate sample in-put points, such as Crude Monte Carlo, Latin hypercube, Sobol’LPτ , have been proposed and compared (Homma and Saltelli1995, McKay, Beckman and Conover 1979).

The next section describes two sampling methods for select-ing (sampling) input values of the input factors. In Section 3,methods for assessing the influence or relative importance ofeach input factor on the output variable are given. Three sim-ulation studies and their results are described and presented inSection 4. Finally, we summarise our conclusions and discussionin Section 5.

0960-3174 C© 2000 Kluwer Academic Publishers

188 Chan, Saltelli and Tarantola

2. Sampling schemes

2.1. The Winding Stairs sampling method

In ordinary Monte Carlo (MC) sampling a new realisation ofthe model output y is obtained by drawing values for the inputsaccording to their joint probability distribution, and calculatingy for each multivariate realisation x. Such a sample contains noinformation about the role of the individual input factors.

The winding stairs method (Jansen, Rossing and Daamen1994) consists of calculating y after each drawing of a newvalue of an individual factor, xi for i = 1, 2, . . . , k. Each newvalue is drawn at random from the marginal distribution of Xi .A sequence of sample points is generated as follows:

we generate a value from each input factor spaceto obtain a sample point in the k dimensionalspace, e.g. {x11, x21, x31, . . . , xk1}. (Note that thefirst suffix denotes parameter number and thesecond suffix denotes sample number.); we thensample a value from the sample space of thesecond parameter to obtain a second sample point,namely {x11, x22, x31, . . . , xk1}. Similarly, a thirdsample point is obtained by sampling a value fromthe sample space of the third parameter giving{x11, x22, x32, . . . , xk1}, and so on. Hence, the kthsample point is {x11, x22, x32, . . . , xk2} and the nextsampling point is obtained by changing the value ofthe first parameter giving, {x12, x22, x32, . . . , xk2}.

We repeat these steps until we obtain the desired number ofsample points.

This means that new input values are sampled in a fixed cyclicorder. The output is evaluated after each sample input point isgenerated, yielding a sequence of output yl , for l = 1, 2, . . . , N ,where N is the total number of model evaluations. We arrangethe sequence of N outputs into k columns and r rows, wherer (= N/k) is the number of turns to repeat the cyclic order. Forexample, the following output matrix shows a set of outputs y1,y2, . . . , y15 for k= 3 and r = 4,

y1 y2 y3

y4 y5 y6

y7 y8 y9

y10 y11 y12

y13 y14 y15

=

f (x11, x21, x31) f (x11, x22, x31) f (x11, x22, x32)

f (x12, x22, x32) f (x12, x23, x32) f (x12, x23, x33)

f (x13, x23, x33) f (x13, x24, x33) f (x13, x24, x34)

f (x14, x24, x34) f (x14, x25, x34) f (x14, x25, x35)

f (x15, x25, x35) f (x15, x26, x35) f (x15, x26, x36)

.

The entries within each column are independent of each other(e.g. y1= f (x11, x21, x31) and y4= f (x12, x22, x32)), since thevalues of each input factor are all different. However, theconsecutive points within each row are not independent in the

Fig. 1. A plot of 63 Winding Stairs sample points and the trajectory offirst 26 points for three factors

sense that the points differ in a single input factor value (e.g.the output y3 and y4 are computed from f (x11, x22, x32) andf (x12, x22, x32), respectively). In total, we generate k × (r + 1)input points and the sequence of these input points forms a tra-jectory in the input factor space. Figure 1 shows an example of atrajectory of the first 26 sample points generated by the WS sam-pling scheme in a three factor case; the total number of pointsplotted is 63 (i.e. r = 20). Note that a pseudo-random numbergenerator is used to obtain each input factor value.

2.2. Sobol’ LPτ sampling scheme

Sequences of LPτ vectors represent a strategy to produce sam-ple points uniformly distributed in a unit cube, and are essen-tially quasi-random sequences which are defined as sequencesof points that have no intrinsic random properties. Generally,sequences of quasi-random vectors, Q1, Q2, . . . , Qn , shouldfulfill the following requirements (Sobol’ 1994):

(i) The uniformity of the distribution is optimal when thelength of the sequence tends to infinity.

(ii) The uniformity of vectors Q1, Q2, . . . , Qn should be ob-served for fairly small n.

(iii) The algorithm used for the computation of the vectorsshould be simple.

Quasi-random sequences are used in place of random pointsto guarantee convergence of estimation in the classical sense.The use of points of LPτ sequences usually results in better con-vergence when employed instead of random points in a MonteCarlo algorithm with finite constructive dimension,1 n. The LPτsequences were introduced by Sobol’ in 1966 and obey therequirements listed above.

The theory underlying the Sobol’s LPτ sequences is given inSobol’ (1967, 1976) and the algorithm to generate the sequenceshas been coded in FORTRAN77 and C (Bratley and Fox 1988,Sobol’ et al. 1992). These programs are widely available hencewe will not describe the algorithm here. For readers who wish toknow more about the theoretical development of LPτ sequences,

Winding Stairs: A sampling tool 189

Fig. 2. Filling-in points generated by LP τ sequences with (a) 512 points, and (c) 1024 points, and (d) 4096 points and (b) 512 points generatedby pseudo-random algorithm

a good summary description can be found in Bratley and Fox(1988).

With as little as 512 points, the uniformly-regular filling-infeature of the LPτ sequence can clearly be seen (Fig. 2(a)),compared with the purely random placement of pseudo-randompoints (Fig. 2(b)). The structure of the image of quasi-randompoints changes as the number of points increases but the propertyof regularity remains (Fig. 2(c) and (d)).

However, sequences that are intended for numerical compu-tations must satisfy the following additional property∫ 1

0· · ·∫ 1

0g(x1, x2, . . . , xk) dx1 · · · dxk = lim

n→∞1

n

n∑j=1

g(Q j ),

where g(x1, x2, . . . , xk) is an arbitrary integrable function.

The main reason that investigators favour the use of quasi-random points is the fast rate of convergence; for large n, theapproximate error can be of the order 1/n compared with or-der 1/

√n for standard Monte Carlo methods. Thus, without

changing the computation algorithm, but merely by replacingthe random numbers with coordinates of quasi-random points,we can improve our results considerably.

3. Sensitivity Analysis

The ultimate goal in performing SA is to investigate the relativeimportance of each input factor with respect to the model out-put variation. This could be done by measuring the main effectand/or total effect of each individual input factor on the model


output. There is a large literature on these topics, the most re-cent are Sobol’ (1993), Jansen, Rossing and Daamen (1994),Homma and Saltelli (1996), McKay (1997), Saltelli and Bolado(1998), Saltelli, Tarantola and Chan (1999), and Jansen (1999).In the next two sections, we will discuss how to measure themain effects and the total effects of the input factors.

3.1. First order Sensitivity Indices, SI

The main effects can be measured by the so-called partial vari-ance (Sobol’ 1993) or correlation ratios (McKay 1997), or TopMarginal Variance (TMV) (Jansen Rossing and Daamen 1994),which is defined to be the expected variance reduction due tofixing factor Xi while the remaining X∼i vary. Here, x∼i denotesa vector of input values x excluding the input value for factor Xi .

Here we will describe briefly the Sobol’ and Jansen’ methods.

3.1.1. The Sobol’ method

In order to describe the Sobol’ method (Sobol’ (1990 in Russianand 1993 in English)), let us define the input factor spaceÄk asa k-dimensional unit cube, i.e. the region

Äk = {x | 0 ≤ xi ≤ 1; i = 1, . . . , k}.

The function f (x) is decomposed into summands of increas-ing dimensionality, namely

f (x1, . . . , xk) = f0 +k∑

i=1

fi (xi )+∑

1≤i< j≤k

fi j (xi , x j )

+ · · · + f1,2,...,k(x1, . . . , xk). (1)

A more general representation of this decomposition using mul-tiple integrals can be summarized as follows. For (1) to hold f0

must be a constant, and the integrals of every summand over anyof its own variables must be zero, i.e.∫ 1

0fi1,...,is

(xi1 , . . . , xis

)dxik = 0, if 1 ≤ k ≤ s. (2)

A consequence of (1) and (2) is that all the summands in (1) areorthogonal, i.e. if (i1, . . . , is) 6= ( j1, . . . , jl), then∫

Äk

fi1,...,is f j1,..., jl dx = 0. (3)

Since at least one of the indices will not be repeated, the corre-sponding integral will vanish due to (2). Another consequenceis that

f0 =∫Äk

f (x) dx.

Sobol’ (1993) showed that decomposition (1) is unique andthat all the terms in (1) can be evaluated via multidimensional

integrals, namely

fi (xi ) = − f0 +∫ 1

0· · ·∫ 1

0f (x) dx∼i

fi j (xi , x j ) = − f0 − fi (xi )− f j (x j )+∫ 1

0· · ·∫ 1

0f (x) dx∼(i j)

with the convention that dx∼i , dx∼(i j) denote integration overall variables except xi , and xi and x j , respectively. Analogousformulae can be obtained for the higher order terms.

The total variance D of f (x) is defined to be

D =∫Äk

f 2(x) dx− f 20 (4)

while the second order moments of the terms in (1) are given by

Di1,...,is =∫ 1

0· · ·∫ 1

0f 2i1,...,is

(x1, . . . , xs) dxi1 , . . . , dxis

where 1≤ i1< · · · < is ≤ k and s= 1, . . . , k. Since we treat thepoint x as random and uniformly distributed inÄk , then f (x) andfi1,...,is (x1, . . . , xs) are also random, hence D and Di1,...,is are themeasures of their variances. The partial variance for factor Xi

is given by

Di =∫ [ ∫

· · ·∫

f (x) dx∼i

]2

dxi − f 20 . (5)

The sensitivity indices develop very naturally, that is by squar-ing and integrating (1) over Äk and using (3) we obtain

D =k∑

i=1

Di +∑

1≤i< j≤k

Di j + · · · + D1,2,...,k

Hence, the sensitivity measure Si1,...,is are given by

Si1,...,is =Di1,...,is

Dfor 1 ≤ i1 < · · · < is ≤ k (6)

where Si is called the first order sensitivity index for factor xi ,which measures the main effect of xi on the output (the fractionalcontribution of xi to the variance of f (x)), Si j , for i 6= j , is calledthe second order sensitivity index which measures the interactioneffect (the part of the variation in f (x) due to xi and x j whichcannot be explained by the sum of the individual effects of xi

and x j ), and so on. It can easily be seen that all the terms in (6),sum to 1, namely

k∑i=1

Si +∑

1≤i< j≤k

Si j + · · · + S1,2,...,k = 1.

3.1.2. The Jansen method

Consider the analysis of variance decomposition of f with twogroups of input factors, X ={U, V } say. Then, by equation (1),we have

f (x) = f0 + f1(u)+ f2(v)+ f12(u, v)


with ∫f1 du =

∫f2 dv =

∫f12 du =

∫f12 dv = 0.

Let

Du =∫

f 21 du, Dv =

∫f 22 dv, Duv =

∫f 212 du dv.

Then, the total variance of the output is given by

D = Du + Dv + Duv. (7)

Jansen, Rossing and Daamen (1994) proposed to measure themain effect of a group of factors u, by

Du = D − 1

2E[ f (U,V)− f (U,V′)]2, (8)

where D is the output variance. This squared difference mea-sure has been described in Saltenis and Dzemyda (1982). Jansen(1996) showed that

D − 1

2E[ f (U,V)− f (U,V′)]2 = Cov [ f (U,V), f (U,V′)]

(9)

We shall see later that, from the computational point of view, theright hand of equation (9) is equivalent to that of the Sobol’ firstorder indices. By partitioning X into Xi and X∼i , we can measurethe main effect of the individual factor, Xi . For example, the firstorder index of factor Xi , denoted by D J

i here, is given by

D Ji = D − 1

2E[ f (Xi ,X∼i )− f (Xi ,X′∼i )]

2. (10)

In the other words, the expected squared difference on the righthand side of equation (10) is computed by fixing factor Xi andvarying the remaining factors X∼i .

3.2. Total Sensitivity Indices, TSI

The Total Sensitivity Index (TSI) is a measure of the total in-fluence of an input factor on the model output variation, and isdefined as the sum of all the sensitivity indices (including all theinteraction effects) involving the factor of interest (Homma andSaltelli 1996).

From equation (7), the total effect of factor Xi is given by

Dtoti = Di + Di(∼i) = D − D∼i ,

where D∼i is the total fractional variance complementary tofactor Xi , and is given by

D∼i =∫

f (xi , x∼i ) f (x ′i , x∼i ) dxi dx ′i dx∼i − f 20 .

Hence, from equation (6), the TSI of factor Xi , denoted by STi ,is given by

TS i = 1− S∼i ,

where S∼i is the sum of all the indices which do not includefactor Xi , and is computed by

D∼i

D.

The Jansen’ method can also be applied to compute the TSI.It can be shown (see Appendix) that

1

2E[ f (Xi ,X∼i )− f (X ′i ,X∼i )]

2

= D − Cov [ f (Xi ,X∼i ), f (X ′i ,X∼i )]. (11)

The right hand side of equation (11) can be written in terms ofan MC integral, namely

Cov [ f (Xi ,X∼i ), f (X ′i ,X∼i )]

=∫

f (xi , x∼i ) f (x ′i , x∼i ) dxi dx ′i dx∼i − f 20 .

= D∼i . (12)

Hence, the total effect of factor Xi can be measured by

D Jtoti =

1

2E[ f (Xi ,X∼i )− f (X ′i ,X∼i )]

2. (13)

Again from the computational point of view, Jansen’s methodto compute TSI is equivalent to that of Sobol’. In summary, forboth indices Jansen’s method uses the squared differences of twosets of model outputs; while the Sobol’ method uses the product.

In the next section, we will discuss how these indices arecomputed.

3.3. Computational issues

The computation of the Sobol’ sensitivity indices is usuallyassociated with the LPτ quasi-random numbers, described inSection 2.2. The partial variance for factor Xi given in equa-tion (5) can be computed by performing Monte Carlo integrals.Two sampling data matrices M1 and M2 both of dimensionn× k for the input factors X are generated. The MC estimatesof quantity Di and Dtot

i are given by

Di = 1

n

n∑m=1

f(xim, x(M1)

∼im

)f(xim, x(M2)

∼im

)− ˆf 20 , (14)

and

Dtoti = D −

[1

n

n∑m=1

f(x (M1)

im , x∼im

)f(x (M2)

im , x∼im

)− ˆf 20

],

(15)

where n is the sample size. For example, Di is computed bymultiplying the output corresponding to x from matrix M1 withthe output computed from a different matrix M2 with the i thcolumn taken from matrix M1. The matrix M1 is usually calledthe data base matrix and M2 the resampling matrix. The samplevariance of the output, D given in equation (4), is estimated


using the base matrix, and is given by

D = 1

n

n∑m=1

f 2(xm)− ˆf 20 , (16)

where ˆf 0= 1n

∑nm=1 f (xm) is an estimate of the mean of the

model output.So far, Jansen’s method has been associated with the Wind-

ing Stairs sampling scheme. Recall that the outputs within eachcolumn of the WS output matrix are independent of each other,so D can be estimated by pooling all the k sample variances ofthe r independent outputs. Hence, the WS sample estimate of Dis given by

DWS = 1

k(r − 1)

k∑i=1

{r∑

m=1

y2(m, i)−[

1

r

r∑m=1

y(m, i)

]2},

(17)

where {y(m, i)} denotes the (m, i)th element of the output matrixY. The expected squared difference on the right hand side ofequation (10) is computed by fixing factor Xi and varying theremaining factors X∼i . Hence, it can be estimated by using the(r + 1)× k WS output matrix as described in Section 2.1, namely

DWSi = DWS − 1

2r

r∑l=i

[yk(l−1)+i − ykl+i−1

]2, (18)

where DWSi denotes the estimate of Di using the Jansen’s method

with the Winding Stairs sampling scheme. For example, withk= 3 and r = 4, to compute the SI of factor X1, we calculate thesum of the squared differences between the entries of the firstand the third column of the first four rows of the output matrix;however, for factor X2, the squared differences are computedusing the values in column 1 and column 2 with a row shifted.

The corresponding WS sample estimate of the SI is given by

S WSi = 1− DWS

i

DWSfor i = 1, 2, . . . , k.

Similarly, the sum of all the partial variances which are comple-mentary to factor Xi , i.e. D∼i is computed by varying factor Xi

and fixing the remaining factors X∼i . By using the WS outputmatrix and equation (13), the WS estimates of Dtot

i for factor Xi

is given by

DWStoti =

1

2r

r∑l=1

[ylk − ylk+1]2 if i = 1

1

2r

r∑l=1

[yk(l−1)+i−1 − yk(l−1)+i

]2if i 6= 1.

(19)

For example, with k= 3 and r = 4, the TSI of Factor X1 is esti-mated by computing the sum of the squared differences betweenthe entries of the first and the last column of the output matrixwith a row shifted; however, the TSI of factor X2 is estimated

by taking the squared differences between the entries of the firstand the second column of the output matrix. It is easy to seethat both the LPτ and the Winding Stairs sampling schemescan be used to compute the Sobol’ and Jansen’s sensitivityindices.

3.4. Sobol’-LPτ vs Jansen-WS

A drawback of the Sobol’ method is that a separate computationis needed for each factor. To compute both sets of first-order andtotal indices, the Sobol’ method requires N = n(2k+ 1) modelevaluations, while the number of model evaluations required forthe WS method is N = rk, where r denotes the ‘sample size’ forthe WS method.

The efficiency of the winding stairs method lies mainly inthe multiple use of model evaluations in that the first order andtotal sensitivity indices of each factor can be computed using asingle set of model evaluations, each model evaluation is usedapproximately four times. With the same sample sizes (i.e. r = n)and computing all the first order and total sensitivity indices,the WS sampling method reduces the total number of modelevaluations by more than a half.

In the next section we will present results of three simulationstudies to compare the Sobol’ indices with LPτ sampling schemeand Jansen’s sensitivity measures with Winding Stairs.

4. Simulation studies

In this section we describe three simulation studies using twoanalytical test functions to illustrate Jansen’s method with theWS sampling scheme (referred to hereafter as the Jansen-WSmethod or simply the WS method) to estimate the first orderand total sensitivity indices, and to compare the method withthe Sobol’-LPτ .

4.1. Example 1

This first example uses Legendre polynomials which have beenused to compare different SA techniques (McKay 1997, Saltelli,Tarantola and Chan 1999). The Legendre polynomial of order dhas two input factors, namely, x (a uniformly distributed randomvariable taking values over the range [−1, +1]) and d (a discreteuniformly distributed random variable, assumed to take valuesfrom 1 to 5).

The analytical values of the partial variances due to the factorsd and x (Dd and Dx ) are given in Table 3 of McKay (1997).These analytical values are given as conditional expectationsand variances, e.g.

Var [E(Y | x)] = Dx,

which is the first-order partial variance of Y due to factor xand since Var [Y ]=Var [E(Y | x)] + E(Var [Y | x]), we havethat E(Var [Y | x]) is the partial variance complementary to x ,namely, D∼x + Dx,∼x.


Fig. 3. Plot of Jansen-WS and Sobol’-LP τ estimates of (a) first order and (b) total sensitivity indices against number of model evaluations.Analytical values of the respective indices for factors x and d are Sx = 0.2 & Sd = 0.0, and TS x = 1.0 & TS d = 0.8.

The Jansen-WS and Sobol’-LPτ estimates of the first orderand total sensitivity indices are plotted against the number ofmodel evaluations (correspond to sample sizes of 32, 64, 128,256, 512, 1024 and 2048) in Fig. 3(a) and (b), respectively. Withthe same number of model evaluations at small sample sizes,the Jansen-WS provides better estimates for both first and totalindices than the Sobol’ method.

To examine robustness of the Jansen-WS method, the experi-ment is repeated 100 times, the average, minimum and maximumof the 100WS estimates of the first order and total indices areplotted against the number of model evaluations. For both in-dices, as the number of model evaluations increases the rangesdecrease and the averages quickly converge to the analyticalvalues.

4.2. Examples 2 and 3

These two examples are illustrated by the Sobol’ g-function,which has been used by several investigators (Saltelli and Sobol’1995, Archer, Saltelli and Sobol’ 1997, and Saltelli and Bolado1998) to compare different techniques for computing sensitivityindices.

The Sobol’ g-function is defined in the k-dimensional unitcube, as follows:

f =k∏

i=1

gi (xi )

where gi (xi ) is given by

gi (xi ) = |4xi − 2| + ai

1+ ai, for 0 ≤ xi ≤ 1 and ai ≥ 0.

The factor ai is chosen to specify the role of the correspondinginput factor xi . For ai = 0, the function, displayed in Fig. 4,

reduces to a function used by Davis and Rabinowitz (1984) asa test case for performing multidimensional integration. Notethat, for each gi (xi ),

∫ 10 gi (xi ) dxi = 1 for i = 1, 2, . . . , k, and

hence∫ 1

0 · · ·∫ 1

0 f dx1 · · · dxk = 1. Furthermore, it can be shownthat the range of the function gi (xi ) is given by

1− 1

1+ ai≤ gi (xi ) ≤ 1+ 1

1+ ai.

Hence, the values of the ai ’s determine the relative importanceof the xi ’s since the range of uncertainty of gi (xi ) depends ex-clusively on the value of ai . The g-functions for four ai valuesare plotted in Fig. 4. The corresponding relative important of the

Fig. 4. Plot of g-function vs the input factor x for ai = 0 (solid), ai = 1(dotted), ai = 9 (dashed) and ai = 99 (long dashed). The relative im-portance of the corresponding input factors are “Very important”, “Im-portant”, “Non-important”, and “Non-significant”, respectively


Table 1. Analytical values of the first order and total indices for theg-function with six and eight factors

Sensitivity Index Sensitivity IndexParameter(ai ) First order Total

Parameter(ai ) First order Total

0.2 0.1261 0.2904 0.0 0.7162 0.78710.8 0.0560 0.1441 1.0 0.1790 0.02420.0 0.1816 0.3862 4.5 0.0237 0.00340.1 0.1550 0.3336 9.9 0.0072 0.01051.6 0.0269 0.0726 99.0 0.0001 0.00010.4 0.0926 0.2245 99.0 0.0001 0.0001

99.0 0.0001 0.000199.0 0.0001 0.0001

input factor can clearly be seen. Note that gi is non-monotonicin all its input variables and its derivative changes sign withinthe interval of variation, hence it is undefined at its midpoint.

The exact (analytical) value of the sensitivity index for eachinput factor can be computed (Saltelli and Sobol’ 1995).In this example, the numbers of input factors chosen areeight and six. The respective sets of ai values are chosento be {0, 1, 4.5, 9, 99, 99, 99, 99} and {0.2, 0.8, 0.0, 0.1, 1.6,0.4}. The first set represents the input factors in order of de-scending importance, while the second set represents the factorsas more or less equally important. Saltelli and Sobol’ (1995)showed that the case where all factors are equally importantor unimportant is the most difficult case to deal with. Theanalytical values of the first order and of the total sensitivityindices for these two examples are given in Table 1. The ex-periment is performed repeatedly so that the accuracy and pre-cision of the two methods can be assessed and compared. Thenumber of replicates is 100. For the Sobol’ LPτ method, the

Fig. 5. Comparisons between the Jansen-WS and Sobol’-LP τ methods using TAE criterion at different numbers of model evaluations: (a) Firstorder indices and (b) Total indices; g-function with eight factors

replicates are obtained by permuting the columns of the datamatrix.

Comparisons are based on the Total Absolute Error criterion(which we abbreviate here as TAE), namely

TAE =k∑

l=1

|Il − I l |

where Il and I l are the analytical value of either the first-order orthe total sensitivity index and its estimate for the factor l, respec-tively. This absolute difference measure is particularly useful ifone wants to compare methods resulting from a wide range ofmodels, e.g., linear, non-linear or additive, etc. In order to makea fair comparison between the two methods, both sets of sen-sitivity indices are computed with the same number of modelevaluations.

The average and the standard deviation (expressed as aver-age ± standard deviation) of the TAEs of the first order andthe total sensitivity indices are plotted against the number ofmodel evaluations for the eight factor case in Fig. 5(a) and (b),respectively. As expected, for both methods and both indices,the averages and the ranges of TAE’s decrease as the number ofmodel evaluations increases, indicating that the estimates con-verge to the analytical values and their precision increases, asthe sample size increases. The Sobol’-LPτ method gives a bet-ter estimate of SI than the Jansen-WS method. However, for theTSI, the Sobol’-LPτ method performs worse than the Jansen-WSmethod at small sample size, and seems to perform marginallybetter than the Jansen-WS method when the sample size is large,but the differences are relatively small (see Fig 5(b)).

Results of the six factor case are plotted in Fig. 6(a) for thefirst order indices and in Fig. 6(b) for the total indices. As in theeight factor example, for both methods and both set of indices,the TAE’s decrease as the number of model evaluations increases


Fig. 6. Comparisons between the Jansen-WS and Sobol’-LP τ methods using TAE criterion at different numbers of model evaluations: (a) Firstorder indices and (b) Total indices; g-function with six factors

and the Sobol’-LPτ method gives a better estimate of SI than theJansen-WS method. However, for TSI the Jansen-WS methodperforms better than the Sobol’-LPτ method at all sample sizes,and is considerably better at much lower sample size (n= 32 and64). Recall that this example represents a difficult case as all theinput factors are more or less equally important. The Jansen-WSmethod outperforms the Sobol’-LPτ method in computing theTotal sensitivity indices at very small sample sizes.

5. Conclusion and discussion

In this paper, we have described the Winding Stairs samplingscheme and Jansen’s sensitivity measures. Jansen’s methods canbe shown to be closely related to those of Sobol’. From the com-putational viewpoint these methods are equivalent. However, itcan be shown (see Chan et al. (2000) for example) that

Var[D Jtot

i

] ≤ Var[Dtot

i

], (20)

while

Var[D J

i

] ≥ Var [Di ]. (21)

In the other words, Jansen’s method provides better estimates formeasuring the total effects while the Sobol’ method gives betterestimates for the main effects. Our numerical examples haveconfirmed this. The advantage of the WS sampling scheme is themultiple use of model evaluations, giving a saving of more thanhalf and hence, for large problems, reducing the computationalburden. However, equations (20) and (21) might also suggestthe superiority of the Jansen-WS method could be due to themethod of computing the sensitivity measures and not to the WSsampling scheme. What remains, however, is the investigationof the Jansen’s method with the LPτ sequences and the Sobol’method with the WS sampling scheme.

It should be noted that the winding Stairs sampling schemeis similar to that of the One-At-Time Morris method (Morris1991). The Morris method consists of generating r independenttrajectories, each of (k+ 1) successive points. The successivepoints are generated by changing a value of one factor at a timewith equal steps so that the resulting points are equally spaced.Hence, the Morris method involves dividing the input factorsspace into a k-dimensional p-level equally spaced grid. Thesample points are used to compute two sensitivity measures foreach factor: the main effect and a measure which is the sum ofall the curvatures and interactions involving that factor. Like theWS scheme, the Morris method makes multiple use of the modelevaluations, reducing the number of model evaluations by half.

Appendix

Define EX[ f (X)]= f0, Var X[ f (X)]= E[ f (X)− f0]2. Then

D − 1

2E[ f (Xi ,X∼i )− f (X ′i ,X∼i )]

2

=Cov [ f (Xi ,X∼i ), f (X ′i ,X∼i )]

Proof:

1

2E[ f (Xi ,X∼i )− f (X ′i ,X∼i )]

2

= 1

2E[ f (Xi ,X∼i )− f0 − f (X ′i ,X∼i )+ f0]2

= 1

2E[( f (Xi ,X∼i )− f0)2 − 2( f (Xi ,X∼i )− f0)

× ( f (X ′i ,X∼i )− f0)+ ( f (X ′i ,X∼i )− f0)2]

= 1

2[D − 2E[( f (Xi ,X∼i )− f0)( f (X ′i ,X∼i )− f0)+ D]


(since E[ f (Xi ,X∼i )− f0]2 = D)

= D − Cov [ f (Xi ,X∼i ), f (X ′i ,X∼i )] ¤

Acknowledgment

The authors wish to thank Prof. Sobol’, two referees and the as-sociate editor for their helpful comments and suggestions. Also,the authors would like to acknowledge the financial support pro-vided by Eurostat through the SUPCOM Lot 14 (Contract No.13592).

Note

1. The constructive dimension of a modelling algorithm is themaximum number of points used to compute one realisationof the random variable 3, for instance (Sobol’ 1994).

References

Abramowitz M. and Stegun I.A. 1970. Handbook of MathematicalFunctions. New York, Dover Publications.

Archer G., Saltelli A., and Sobol’ I.M. 1997. Sensitivity measures,ANOVA like techniques and the use of bootstrap. Journal ofStatistical Computation and Simulation 58: 99–120.

Bratley P. and Fox B.L. 1988. Algorithm 659 implementing Sobol’quasirandom sequence generator. ACM Transactions on Mathe-matical Sofware 14: 88–100.

Chan K., Saltelli A., Tarantola S., and Sobol’ I.M. 2000. Variance basedmethods. In: Saltelli A., Chan K., and Scott E.M. (Eds.), Mathe-matical and Statistical Methods for Sensitivity Analysis. London,John Wiley & Sons, Forthcoming.

Homma T. and Saltelli A. 1995. Sensitivity analysis of model out-put. Performance of the Sobol’ quasi-random sequence gener-ator for the integration of the modified Hora and Iman impor-tance measure. Journal Nuclear Science and Technology 32: 1164–1173.

Homma T. and Saltelli A. 1996. Importance measures in global sen-sitivity analysis of nonlinear models. Reliability Engineering andSystem Safety 52: 1–17.

Jansen M.J.W. 1996. Winding stairs sample analysis program WIND-INGS 2.0. Technical Report, Private communication.

Jansen M.J.W. 1999. Analysis of variance designs for model output.Computer Physics Communications 117: 35–43.

Jansen M.J.W., Rossing W.A.H., and Daamen R.A. 1994. Monte Carloestimation of uncertainty contributions from several independentmultivariate sources. In: Gasman J. and van Straten G. (Eds.),Predictability and Nonlinear Modelling in Natural Sciences andEconomics, Kluwer Academic Publishers, pp. 334–343.

McKay M.D. 1997. Non-parametric variance-based methods for assess-ing uncertainty importance. Reliability Engineering and SystemSafety 57: 267–279.

McKay M.D., Beckman R.J., and Conover W.J. 1979. A comparison ofthree methods for selecting values of input variables in the analysisof output from a computer code. Technometrics 21: 239–245.

Morris M.D. 1991. Factorial sampling plans for preliminary computa-tional experiments. Technometrics 33: 161–174.

Saltelli A. and Bolado R. 1998. An alternative way to compute FourierAmplitude Sensitivity Test (FAST). Computational Statistics andData Analysis 26: 445–460.

Saltelli A. and Sobol’ I.M. 1995. About the use of rank transformationin sensitivity analysis of model output. Reliability Engineeringand System Safety 50: 225–239.

Saltelli A., Tarantola S., and Chan K. 1999. A quantitative, model in-dependent method for global sensitivity analysis of model output.Technometrics 41: 39–56.

Saltenis V. and Dzemyda G. 1982. Structure analysis of extremal prob-lems using an approximation of characteristics. Optimal DecisionTheory 8: 124–138.

Sobol’ I.M. 1967. On the distribution of points in a cube and the approx-imate evaluation of integrals. USSR Computational Mathematicsand Mathematical Physics 7: 86–112.

Sobol’ I.M. 1976. Uniformly distributed sequences with an additionaluniform property. USSR Computational Mathematics and Math-ematical Physics 16: 236–242.

Sobol’ I.M. 1993. Sensitivity analysis for nonlinear mathematical mod-els. Mathematical Modeling & Computational Experiment. 1:407–414.

Sobol’ I.M. 1994. A Primer for the Monte Carlo Method. Boca Raton,CRC Press, USA.

Sobol’ I.M., Turchaninov V.I., Levitan Y.L., and Shukhman B.V. 1992.Quasirandom sequence generators. Preprint, Keldysh Institute ofApplied Mathematics, Russian Academy of Sciences.

winding stairs: a sampling tool to compute sensitivity indices

Documents