selecting control variates to estimate multiresponse simulation … · multiresponse simulation...

80 European Journal of Operational Research 71 (1993) 80-94 North-Holland

Theory and Methodology

Selecting control variates to estimate multiresponse simulation metamodels

Acacio M. de O. Porta Nova

Department of Mathematics, Superior Technical Institute, 1096 Lisbon Codex, Portugal

James R. Wilson

Department of Industrial Engineering, North Carolina State University, Raleigh, NC 27695, USA

Received November 1990; revised November 1991

Abstract: In this paper we discuss the optimal selection of control variates for simulation experiments in which the objective is estimation of a multiresponse metamodel - that is, a linear regression model for an output vector of simulation response variables expressed in terms of an input vector of design variables for the target system. We consider.the control-variate selection problem in the context of some specific covariance structures for the responses and the candidate controls that commonly occur in certain types of econometric and psychometric simulation studies. We conclude that in these situations, the optimal set of controls is frequently larger than would be obtained by certain conventional control-variate selection procedures; moreover as a function of the number of selected controls, the efficiency of the controlled metamodel point estimator is often relatively insensitive in the neighborhood of the optimal number of controls.

Keywords: Simulation; Control variates; Multivariate statistical models; Variance reduction techniques

I. Introduction

Simulation is probably the most widely used technique of operational research. The main reason for this phenomenon is that simulation makes it possible to build and run detailed models of complex, real-world systems. However, because large-scale simulation experiments are in general computationally intensive, numerous variance reduction techniques (VRTs) have been developed to improve the efficiency of these experiments. A comprehensive survey of VRTs for discrete system simulation is given in Nelson (1987) and Wilson (1984). Although several VRTs have the potential for effective use in discrete simulation experiments, only the simplest variants of the methods of common random numbers and antithetic variates seem to have achieved widespread acceptance by simulation practitioners.

For general application to simulation studies, we believe that the method of control variates possesses some distinct advantages relative to other variance reduction techniques. In contrast to VRTs based on

* This work was supported in part by National Science Foundation under Grant No. DMS 8717799. Correspondence to: Prof. A.M. Porta Nova, Depar tment of Mathematics, Superior Technical Institute, Avenida Rovisco Pals, 1096 Lisbon Codex, Portugal.

0377-2217/93/$06.00 © 1993 - Elsevier Science Publishers B.V. All rights reserved

A.M. Porta Nova, J.R. Wilson / Estimating multiresponse simulation metamodels 81

importance sampling, the method of control variates does not require manipulation of the sample path followed by the simulation; instead this method uses regression analysis to exploit any inherent linear correlation between the system responses of interest and other concomitant random variables with known means (controls) that are observed on each run. Although the method of control variates has the potential to yield large efficiency increases in a diversity of applications, this method often requires relatively little additional computing overhead; see Avramidis, Bauer and Wilson (1991) and Fishman (1988).

To evaluate the efficiency of standard linear control-variate procedures for estimating the mean of a single system response, Lavenberg, Moeller and Welch (1982) introduced two basic performance measures: (a) the minimum variance ratio, which quantifies the optimal variance reduction obtained in the (unrealistic) situation that the optimal control coefficients are known; and (b) the loss factor, which measures the variance inflation that occurs when the optimal control coefficients are unknown and are estimated by the method of least squares. Thus when the covariance structure of the responses and the controls is unknown so that the optimal control coefficients must be estimated, the product of the minimum variance ratio and the loss factor yields the net variance ratio, which is the ratio of the variances of the controlled and uncontrolled estimators of the mean response. Nozari, Arnold and Pegden (1984) extended the use of control variates to the estimation of a simulation metamodel for a single system response; in this situation the system response is represented by a multiple linear regression model in which the independent variables (regressors) are deterministic design variables relevant to the target system. In contrast to this line of development, Rubinstein and Marcus (1985) extended the linear control-variate procedures of Lavenberg, Moeller and Welch (1982) to estimate the mean of a vector of simulation responses. Porta Nova and Wilson (1989b) consolidated all of these results within a unified framework and extended the application of control variates to the estimation of a multiresponse simulation metamodel - that is, a multivariate linear model for a vector of selected system responses expressed in terms of a vector of relevant design variables.

Currently there is no definitive resolution of the question of how to select effective control variates from a large pool of candidates so as to account completely for the trade-off between the variance reduction due to inclusion of additional controls and the variance increase due to estimation of the corresponding control coefficients. In the context of estimating a univariate mean response, Lavenberg, Moeller and Welch (1982) considered two approaches to this problem: (a) using all availab[e control variates, and (b) applying only the three 'best' controls as chosen by forward stepwise regression. These authors found that in large queueing network simulations, approach (a) often yielded net variance increases, and substantially better results were obtained with approach (b). To estimate a multivariate mean response, Bauer and Wilson (1992) devised some control-variate selection criteria that are based on minimizing the mean square volume of the delivered confidence region and that appear to be effective when the responses and controls are jointly normal. Unfortunately, much more extensive Monte Carlo studies are required to support any general conclusions about the performance of any of these selection procedures. To gain some insight into the control-variate selection problem for estimation of a multivariate mean response in a specific class of simulation applications, Rubinstein and Marcus (1985) postulated a repeated-measures type covariance structure (see Section 3) for the responses and the controls, and they examined the sensitivity of the net variance ratio to the number of control variates selected. None of the above-cited work on control-variate selection has been extended to estimation of simulation metamodels.

In this paper we seek to characterize optimal solutions of the control-variate selection problem in the context of estimating multiresponse simulation metamodels. In the spirit of the approach taken by Rubinstein and Marcus (1985), we examine some specific covariance structures that occur often enough in practice to be interesting and yet are simple enough to admit a complete analytical treatment. This paper is organized as follows. In Section 2, we summarize the relevant results of Porta Nova and Wilson (1989b). In Section 3, we analyze simulation metamodels with a repeated-measures covariance structure of the form suggested by Rubinstein and Marcus. In Section 4, we examine metamodels with an autoregressive covariance structure that frequently arises in econometric simulations. Finally in Section

82

Random Input

Processes

A.M. Porta Nova, J.R. Wilson / Estimating multiresponse simulation metamodels

Design Variables

X n . . . Xi,,,

1 1 Un Simulation . I/,,I

: Model : Uik (ith Run) , y/~

1 1 6i l . . . Gq

Control Variables

Figure 1. Layout of simulation inputs and outputs

Response

Variables

5, we draw some conclusions and make recommendations for further work in this area. Although this paper is based on Porta Nova (1985), some of our results were also presented in Porta Nova and Wilson (1989a).

2. Controlled estimation of simulation metamodels

First we introduce the nomenclature of Porta Nova and Wilson (1989b) to describe control-variate schemes for estimating a multiresponse simulation metamodel. In a statistically designed simulation experiment, we perform an independent simulation run at each of n design points defined by a combination of settings for rn design variables, and we observe the following quantities on the i-th run (i = 1 . . . . . n): (a) the 1 x m deterministic vector X i . - [ X n . . . . . Xim] of design variables; (b) the 1 x p random vector Y/.= [Y/1 . . . . . Y/p] of system responses; and (c) the 1 x q random v e c t o r Ci .~- [Cil . . . . , Ciq ] of concomitant control variables with known means. Figure 1 depicts the layout of the inputs and outputs associated with the i-th simulation run; for completeness we include a set of, say, k random number streams {U/1 . . . . . U/k} constituting the stochastic input processes of the i-th simulation run. Thus, all of the relevant information about the simulation experiment can be summarized in the n × rn design matrix X = [ X ; . , . . . , X ' . ] ', the n x p response matrix Y=-[YI' . . . . . . Y'.]', and the n X q control matrix C = [C~. , . . . ,C ' . ] ' . Without loss of generality, we assume that each control vector C r has mean 0'q, a q-dimensional row vector of zeros. For technical reasons, we also assume that n > m + p + q, that X has full column rank, and that C has full column rank with probability one.

A multivariate linear metamodel for the simulation response on the i-th run is given by

r~=xio+<, (1) where 19 is an (m xp)-mat r ix of metamodel coefficients and ~ . is a (1 xp) -vec tor of random errors with mean 0p. The direct simulation estimator (that is, the Ordinary Least Squares estimator) of 19 is

= ( x ' x ) - ' x ' r (2)

(see Seber, 1984, Sections 8.1-8.2). If the error ~/. has a linear regression on the control Ci., then the metamodel for the overall simulation experiment (including the control variates) is

¥ i . = X i . 1 9 " b C i . A - F R i . f o r i = l . . . . . n , n > m + p + q , (3)

where ,a i s t h e (q xp)-matr ix of (unknown) control coefficients, and R i. is the (1 Xp)-vector of residuals. Under some mild technical conditions, it can be shown that the covariance matrix for the metamodel (3),

,~ = Cov(Y/., Ci. ) = Cov(Ci., Y,..) Cov(Ci.) = ~ c r "~c ] (4)


exists and is positive definite, where Xr is the p x p covariance matrix of Y, Xrc is the p × q covariance matrix between Y and C, "~cr -- "~rc, and "~c is the q x q covariance matrix of C. Conditions sufficient to ensure existence and positive definiteness of X are detailed in Porta Nova and Wilson (1989b).

If • denotes an arbitrary fixed (q xp)-matr ix of control coefficients, then the corresponding controlled estimator of the metamodel coefficient matrix @ is

~9( ,t,) = ( X ' X ) - ' X ' ( Y - Cq ' ) . (5)

The basis for (5) is that we seek to compensate for the unobservable error component L ~-- [~( . . . . . . ~ ' . ] ' of the response matrix Y by subtracting from Y an appropriate linear transformation C ~ of the known deviation C - E[C] = C. The controlled estimator (5) is unbiased with generalized variance given by the determinant

I Cov[vec o(q,)] ! = I.r,y - ¢ " . V c y - + ¢ " . V c ' t ' I m l X ' X I - ' , (6)

where the 'vec' operator transforms the (m xp)-matr ix O(q~) into the (mp x D-vector vec 0(q~) by stacking the columns of O(4~) respectively under one another to form a single column; see Muirhead (1982), pp. 73-76.

The generalized variance (6) is minimized by the optimal control coefficient matrix

A = "Y'c l~'cv (7)

so that the minimum variance ratio is

I Cov[vec ,Cov(vec O ) , O(A)] I [ ]m w(a)= m= f l ( 1 - p 2) , (8) j=l

where Ip is the p × p identity matrix, v - min{p, q}, the quantities pl z >p22 > . . . >p~ > 0 are the ordered eigenvalues of the matrix ,~ r l ,~ rc ,~c l~c r , and the quantities {Ps:J = 1, 2,...,~,} are the canonical correlations between Y/. and C/.. An alternative interpretation of the {pj} may help to clarify their significance. The quantity Pl is the maximum correlation that can be obtained between any linear combination V~ = alY/.= E/'= ~auY/t of the responses observed on the i-th run and any linear combination t q W~ = b~Ci. = Et=lbliCil of the controls observed on the same run. For j = 2 . . . . , v, the j-th canonical correlation pj can be interpreted as follows: we consider all linear combinations of the form Vj = ajY/. and Wj = bjC'. such that Vj and Wj are uncorrelated with {V1, W 1 . . . . . Vj_ 1, Wj_1}; and we take pj to be the maximum correlation between all such linear combinations Vj and Wj. In the case of a univariate response, we have v = 1 and so p~ is simply the multiple correlation coefficient between the single response Yi and the control vector Ci. observed on the same run. For a discussion of canonical correlations, see Arnold (1981), pp. 440-444.

One might object to (6) on the grounds that the user is actually interested in using ~.=Xu@(q~) to predict E[~.] = Xi .O rather than using @(q~) to estimate O; and thus q~ should be taken to minimize an appropriate function of the prediction errors { ~ i . - ~ . - X i . @ : i = 1, 2 . . . . . n}. Note, however, that the generalized variance of ~ . is given by

Icov( , )1 = I z . - . , z . - + I x,,] a result which closely parallels (6). Thus the generalized variance (9) of the matrix of prediction errors at each design point is minimized by the same control coefficient matrix (7) that yields the minimum variance ratio (8) for the estimation of 0 . This provides some evidence that a generalized variance ratio like (8) is indeed an appropriate performance measure to be used as a basis for selecting effective control variates. On the other hand, (8) corresponds to the maximum reduction in generalized variance that can be obtained in the (unrealistic) case where we know the joint covariance structure (4) of the responses and the controls. Of course, in general we do not know • so that At has to be estimated; and a loss of efficiency will occur in this situation. We conclude that in the controlled estimation of multiresponse

84 A.M. Porta Nova, J.R. Wilson / Estimating multiresponse simulation metamodels

simulation metamodels, an appropriate performance measure for selecting effective control variates should incorporate the minimum variance ratio (8) as well as the efficiency loss resulting from estimation of A.

With probability one, the generalized least squares estimators of A and O exist and are uniquely given by

and

3 = (C'PC)-~C'PY with e =I n - X ( X ' X ) - I x ' (10)

0(3 ) = ( X ' x ) - l x ' ( Y - C3), (11)

where I n is the identity matrix of order n; see Proposition 3 in the Appendix of Porta Nova and Wilson (1989b). If the response matrix Y and the control matrix C jointly possess a matrix normal distribution (see pp. 310-313 of Arnold, 1981), then the aggregate output matrix Z = [Y, C] consists of jointly normal elements with mutually independent rows (corresponding to independent runs) and with the covariance matrix ~: for each row. In the notation of the matrix normal distribution, the joint normality of the responses and the controls is represented as

Z= [ r, C] ~ Nn,p+q( [ XO, 0n×q], In, X). (12)

If (12) holds, then the controlled estimator O(,a) of the metamodel coefficient matrix has the generalized variance

] Cov[vec O( zi)]l = ]X'XI-PI~v ] m I lp -- ~¢y l~yc.~ C 1.~Cy I m (13) n - m - q - 1

(see Porta Nova and Wilson, 1989b, pp. 1323-1325)• The corresponding net variance ratio is (nml )]m [C°v[vec O(z i ) ] I ( 1 - p 2 , (14) " 0 ( 3 ) = ICov(vec O) I = n - m - q - 1 Lj.=I

with v and {pj} as defined previously• By comparing (14) with (8), we infer the loss factor

A(3)= ( n - m - 1 ) r a p ' n - m - q - 1 (15)

which quantifies the variance increase that occurs when we estimate the covariance structure ,~ in order to estimate the optimal control coefficient matrix A.

3. Results for the repeated-measures setup

In the spirit of Section 2 of gubinstein and Marcus (1985), we consider a situation in which the joint covariance structure (4) of the responses and the controls has the simplest possible nontrivial form:

1 y y 1

~ _ c r 2 Y Y Y Y Y Y

Y Y

• . . , ) /

• • • , ) /

" ' " 1

• • . , ~

• . . , ) /

3' 3' " '" Y "y y . . . y

Y Y • . . , ) / • ° •

y 1 "-"

Y Y - - .

= 2[xY Yc] ["Y'cr "Y'c " (16)


This setup is widely used in psychometrics and arises naturally in experiments in which several treatments are applied to the same subjects (individuals), frequently at different times; see Arnold (1981) or Morrison (1976). Although we usually assume that the observations for different subjects are independent, the same assumption cannot be made for different observations on the same subject. In the context of a simulation experiment, the covariance matrix (16) represents the situation in which all the simulation outputs on the i-th run (that is, Y~. and Ci.) have the same variance or 2 and all pairs of outputs have the same correlation 3'. Without loss of generality, we take cr 2 = 1 in the development that follows.

Straightforward calculations yield the minimum variance ratio (8) and the net variance ratio (14) for the repeated-measures covariance structure (16). To compute the canonical correlations between Y/. and C i., we must first compute the inverses of the associated covariance submatrices. The inverse of ~ c is the (q × q)-matrix

- 1 - ( q - 2 ) 3 ' 3' "'" 3' ]

"~cl = dq ' 3' - 1 - ( q - 2 ) 3 ' " " 3' , (17)

3' 3' . . . . l - ( q - Z ) y

where dq = replaces q; see Kendall, Stewart and Ord (1983), p. 320. It follows that

A =~;c12;cr 3'(7 -1 , = - 1 ) d q l q l , .

Moreover, in (8) we have

q3"2 h ~ - ~ y l ~ y c ~ c l ~ c c y = [(q _ 1)3' + 1] [ ( p - 1)3' + 1] lt ' l~ = al , l~, .

(q - 1)3 ̀2 - (q - 2)7 - 1. Similarly, ,~y1 is a (p ×p)-matrix analogous to (17) in which p

(18)

(19)

Since the matrix on the right-hand side of (19) clearly has rank one, there is only one positive canonical correlation between Y,.. and C;.. In general the canonical correlations between Y~. and C i. are the eigenvalues of A; that is, the roots of the characteristic polynomial

= a - ~ a .-" a a a - ~ : . . - a

Ia-~Xl . = ( - ~ ) P - l ( p a - ~ ) . (20)

a a ... a

Thus, the eigenvalues of A are pa (with multiplicity one) and zero (with multiplicity p - 1). This implies that the minimum variance ratio for the covariance structure (16) is

[ qy p y ]m, (21) r / ( A ) = 1-- ( q - - 1 ) y + l ( p - - ~ - y + l

and the net variance ratio is

( n _ m _ l )mp[ qy py ]m (22) r / ( / i ) = n--m--q--1 1-- ( q - - 1 ) y + l ( p - - 1 ) y + l

Setting both p and m equal to one in (22) yields the case that was treated analytically by Rubinstein and Marcus (1985).

Expression (22) enables us to determine numerically the optimal number of control variates q* for minimizing the variance ratio as a function of the number of runs (n), the number of output responses


Table 1 Optimal number of controls q* for the repeated-measures setup with m = 1

No. runs n No. responses p

p = l p = 3 p = 5

Correlation y

0.1 0.5 0.9 0.1 0.5 0.9 0.1 0.5 0.9

10 0 1 2 0 1 1 0 0 1 50 0 5 6 0 4 5 0 4 4

100 0 8 9 0 7 8 0 6 7 1000 22 30 31 21 29 30 20 28 29



p = l p = 3 p = 5

Correlation y

0.1 0.5 0.9 0.1 0.5 0.9 0.1 0.5 0.9

10 0 0, 1 1 0 1 1 0 0 1 50 0 5 6 0 4 5 0 3 4

100 0 8 9 0 7 8 0 6 7 1000 22 30 30 21 29 29 20 28 29

(p) , the number of design variables (m), and the correlation parameter (7). These results are summarized in Tables 1, 2, and 3 for the cases m = 1, m = 5, and m = 10, respectively. It should be noted that the entries in Table 1 for n = 50 and n = 100 differ substantially from the corresponding entries in Table 1 of Rubinstein and Marcus (1985); we believe that the latter entries are incorrect.

Tables 1 -3 reveal that the optimal number of control variates q* is an increasing function of n and 3' and a decreasing function of p and m. Perhaps one surprising aspect of these tables is the relatively large number of controls that can be applied effectively to the estimation of a metamodel when the number of runs is very large (n -- 1000). If the number of runs is in the usual range 50 _< n _< 100 and the common correlation is in the range 0.50 _< 3' < 0.90, then the optimal number of controls is in the range 3 _< q* < 9 for a wide range of the number of design variables (1 < m < 10); moreover, q* is relatively insensitive to the number of responses p. These results suggest that simple rules of thumb for selecting control variates based exclusively on a quick examination of the loss factor (15) may be too conservative when a complex simulation metamodel is to be estimated.



p = l p = 3 p = 5

Correlation y

0.1 0.5 0.9 0.1 0.5 0.9 0.1 0.5 0.9

10 . . . . . . . . . . 50 0 4 5 0 4 4 0 3 4

100 0 8 8 0 7 8 0 6 7 1000 22 29 30 21 29 29 20 28 28

A.M. Porta Nova, J.R. Wilson / Estimating multiresponse simulation metamodels

n = 100 / • n = 5 0 . / /

, . / . / " . . n = 3 0

ii . . . . . . . .

- 1 , 0 ' ~ 0 , 5 0 , 0 0 . 5 1,0

r,"

p" / n = 100 ¢- - / / , ~ = so

?- / h : - - - - " =

~

(a) p = l a n d m = l (b) p = 1 a n d m = 5

u

¢d

col ;>

,<l

il .

d-

o_~1° 0 '

o o

o 1 . 0

/ n = 100 ~1

! n = 3 0 o

~ m

d

0 ° 0 0 , 5 1,0

tr

1

n = 100 n = 5 0 n = 3 0

' 0 ' , 5 ' --'o,s ' - :0 .s ' d .o 1',o 3'

(c) p = 5 a n d m = l (d) p = 5 a n d m = 5

Figure 2. Net variance ratio (22) vs. the correlation y for the repeated-measures setup with q = 1 control

87

To gain add i t iona l insight into the con t ro l -var ia te se lec t ion p r o b l e m in the context of es t imat ing mul t i r e sponse s imula t ion me tamode l s , we g r a p h e d the var iance ra t io (22) as a funct ion of 3, and q s epa ra t e ly for given f ixed va lues of p , m, and n. F igure 2 conta ins the co r r e spond ing plots of the ne t va r iance ra t io as a funct ion of 3, when we take q = 1 control ; s imilar resul ts for q = 5 cont ro ls a re d i sp layed in F igu re 3. F igure 4 dep ic t s the ne t va r iance ra t io as a funct ion of q for 3, = 0.5; ana logous resul ts for 3' = 0.9 are d i sp layed in F igure 5. W e also plot the op t ima l n u m b e r of cont ro ls q* as a funct ion of 3' in F igu re 6.

F r o m these plots, we conc lude tha t d rama t i c var iance r educ t ions can be ob ta ined , even for m o d e r a t e values of 3'. O n the o t h e r hand, va r iance increases usual ly occur when we have only spur ious co r re la t ion


0

c::

;>

,<1

~r',"

d"

/ n = 100 ~ t _ / t n = 5 0

~ .

~-~ .

"0,5 ' 0 . 0 0 , 5 1 . 0 - 1 , 0 ' "0,5 ' "7

~ . t ! = 1 0 0 • 5 0

3 0

doO' o . 5 ' 1'.o

(a) p = 1 and m = 1 (b) p = l a n d r a = 5

~r ~ - v ~ -

- ~ . . 3 ¢ ~ n = 1 0 0 - n = 1 0 0

~_ n = 5 0

. ~

~-~ . ~" .

d-

s., \ i i °-1.o 20.5' o.o o.-~ 1.o -1.o' -o.5' o'.o ' d.5 1'.o

7 7

(c) p = S a n d m = l (d) p = S a n d m = 5

F i g u r e 3. N e t v a r i a n c e ra t io ( 2 2 ) vs . t h e c o r r e l a t i o n y for t h e r e p e a t e d - m e a s u r e s s e t u p w i t h q = 5 c o n t r o l s

between the controls and the responses (that is, when l yr is near to zero). Looking at Figures 4 and 5, we observe also that as y increases, the net variance ratio remains below the break-even line r/(,~) = 1 for a much wider range of values of q. This means that when the responses and the controls are strongly correlated, there is little penalty for failing to identify precisely the optimal set of control variates to use.

4. Results for the autoregressive setup

Whereas in some sense the repeated-measures setup of the last section provides the 'most favorable' situation for applying multiple control variates to the estimation of a multiresponse simulation meta-

A.M. Porta Noua, J.R. Wilson / Estimating multiresponse simulation metamodels 89

n = 3 0 n = 5 0

~. n = 5 0

~- / / ,, :,oo

~ n = loo

.=.; , , , = , ,

q q

(a) p = 1 andre= 1 (b ) p = 1 a n d m = 5

n = 1 0 0 8

i t.~.

- - i ~ .

.! ~.1 / / / ,~ ol,/ / ~ ~ '~-

, , o , 3 , o , , o q

(c) p = s and m = 1

n : ] 0 0

n = 30[ n = 5 0

i ,

lo ~o q

n = O O

(d) p = 5 a n d r e = 5

Figure 4. Net variance ratio (22) vs. the number of controls q for the repeated-measures setup with correlation 3' = 0.5

model, the autoregressive setup described in this section may be viewed as the 'least favorable' situation for using multiple controls. First-order autoregressive or (Markov) processes are widely used to analyze time series in econometrics (Johnston, 1972) and psychometrics (Morrison, 1976). Such processes arise naturally when considering cross-sectional data collected over time. For example, we might be interested in explaining the monthly consumption expenditure of n households (or other sampling units) over p months using (a) independent variables like disposable monthly income over that p-month time horizon, and (b) controls like the consumption expenditures in months prior to the p-month time horizon.


n = 3 0

n = 50

~ I00

' 4o ' / o ' ~ ' ,'o q

.

,<1

.

l r , , ~ -

¢_:,

5

,<1

%,

n = 5 0

n = 3 0 = ~ g

J n . .~ cc

' ~'o ' ~o ' ~ ' '~o 0

(a) p = 1 and rn= 1 (b) p = 1 and r n = 5

8 i..¢

n = 3 0 n = 5 0

i

5,

lOO

n = ( X ~

, , , , 40 io ~o 20

17

"S"

0) O

',~ d- >

( 4 i .

%,

n = 5 0

n = 30

J J 10 ' 2'0

q

n ---- I00

I n----O0 l L

' 3'0 ' ,20

(c) p = 5 . a n d m = 1 (d) p = 5 a n d m = 5

Figure 5. Net variance ratio (22) vs. the number of controls q for the repeated-measures setup with correlation y = 0.9

Thus if Y/~ denotes consumption expenditure of sampling unit i (i = 1 . . . . . n) in month j, then the next monthly expenditure evolves according to the autoregressive scheme

r/,j+ 1 - E(Yi,j+I) = "/[ Yij - E(Yij)] -[- Eij, (23)

where [ y[ < 1 and the process variance 0 ~2 - var(Y u) is assumed to be positive and finite. The residual eij is normal with mean zero and variance ( r 2 ( 1 - y2); moreover the residuals { e u : i = 1 . . . . . n ; j =

1 . . . . ,p} are mutually independent random variables. Without loss of generality we assume that E ( Y u) = 0 and o~ 2 = 1 in the following development.


For the autoregressive setup described above, it is easily seen that the l ag- /covar iance between the responses of interest is

Cov(Y/.j+ l, Y,j) = ,7 Itl for l = 1 . . . . . p - 1, (24)

and thus for the covariance matrix of the responses and the controls observed in the i-th sampling unit,

1 7 " " " 7 p-1

7 1 • • • 7 p-2

7 p - I 7 p-2 " " 1

7 p 7 p - l " " " 7

7P+l 7 p . . . ,}/2

7 p + q 1 7 p + q - 2 . . . 7 q

Y =

we take

7 p 7 p + I . . . 7 P + q -1

7 p - I 7 p • . . 7P+q- -2

7 ,}/2 . . . 7q

1 7 ' ' ' 7 "~'~- 1

7 1 " " " 7 q-2

7 q - 1 7 q - 2 "" • 1

X y •YC ]

= ~Ycr "Yc ]" (25)

As in the case of the repea ted-measures setup, it is s traightforward to calculate the min imum variance ratio (8) and the net variance ratio (14) for the autoregresssive covariance structure (25). In the latter case, the approach used to derive (17)yields the ( p ×p) -ma t r ix

1 - y 0 0 - 0 0 0

- y 1 d--72 - 7 0 ' ' ' 0 0 0

0 - 7 1+,}/2 - 7 " " 0 0 0 1

, ~ f f l _ 1 - - T 2

0 0 0 0 . . . . 7

0 0 0 0 . . . 0

A similar expression can be obta ined for X c 1. If follows that

A = ~Yr l~Yrc~Yc l~Ycr =

0] 0 . . . 0

• °

0 . . - 0

T p + I T p • . . 7 2

l + T 2 - y

- y 1

(26)

(27)

Provided that 3' ~ 0, the matrix A has rank one; and this implies (hat there is only one positive canonical correlat ion between Y~ and C i.. Solving the characterist ic equat ion

- ~ o . - . o

o - ~ . . . o I A - ~ l l = . = 0, (28)

T P +1 T p . . . T 2 L

we see that the eigenvalues of A a r e T 2 (with multiplicity one) and zero (with multiplicity p - 1). It follows that with the autoregressive setup, the min imum variance ratio and the net variance ratio are respectively given by

P

r / (A) = I - ] ( 1 - p z ) m = ( 1 - T 2 ) m (29) j = l

92

:l A.M. Porta Nova, J.R. Vvilson / Estimating rnuhiresponse simulation metamodels

o_1,0 '

o

n = 1000

n = i00

_ n = 5 0

- n = 30

V v l l l l ~

-0 ,5 0,0 0,5 1,0 7

(a) p = l a n d m = l

ol

O O '

n = 1000

w

n = 100

- - n = 5 0

- n = 3 0

2o.~' d . o ' d . n ' 4.0 ,y

(b) p = l a n d r n = 5

" ~ .

o

n = 1000

o

n = 1 0 0 0

I n = 100 ]

_ - - n = _ 5 0 tol - - - -

_ _ n = 3 0

l | l v i | l v ! J ! v 1.o -0.5 o% o'.5 4.0 o_1.o -0.5 o.o o.-~ '7 '7

(c) p = 5 a n d m = l (d) p = 5 a n d m = 5

n = 100

n = 5 0

- - - - n = 3 0

4.0

F i g u r e 6. O p t i m a l n u m b e r o f c o n t r o l s q * vs. the correlation ~, for the repeated-measures s e t u p

and

( n-m-1 n(z~) = n - m - q - 1 (1 - T2)m" (30)

Since the m i n i m u m variance ratio (29) does not improve w h e n we use more than one control variate, the only reasonable alternatives are to use one control or no controls. In this s i tuation it is advantageous to use a control variable w h e n

r / ( z i ) < l ¢~ 3, 2 > 1 - (31) - m 1


G"

. L >

K'-

g

=

$-

P d-

g g-

g-

g g

-1,0

n = 100 n = 5 0

<,el "-U

!

2

-0o5 0 ,0 0 ,5 I , 0

g

/ n = 5 0 ¢.

.

o l w -~ .0 --O.5 0 ,0 0 ,5

n = 100

4.o

(a) p = 1 and m = 1 (b) p = 1 and m = 5

%- + ,a

U

>

<<1

g

! o

<;-'1.o' -'o.~' 6 . 0 ' 0'.5' 1'.o

(c) p = S a n d m = 1

g

~- 1 / 1 + n = l o o

/ / # H --:so

II/

<<1 "U

- 1 , 0 - 0 ,5 0o0 0=5 1 ,0

(d) p = 5 a n d m = 5

Figure 7. Net variance ratio (30) vs. the correlation y for the autoregressive setup with q = 1 control

Figure 7 depicts the net variance ratio (30) as a funct ion of the correlat ion y for various values of m, n, and p when we take q = 1 control. F rom these plots we see that an efficiency increase will occur if m < 5, n > 30 and l y l > 0.35. On the o ther hand, if only spurious correlat ion is present, significant variance increases may be p roduced - especially if the number of simulation runs is less than 30.

5. Conclusions and recommendations

In this paper we have examined the control-variate selection problem in the context of estimating mult i response simulation metamodels . By considering some specific covariance structures for the


responses and the controls that represent a diversity of application contexts and that can be handled analytically or numerically, we believe that we have gained some new insights into the control-variate selection problem. Our results suggest that when a complex metamodel is to be estimated, excessively conservative conclusions may result from the use of control-variate selection procedures based on forward stepwise regression or on simple rules of thumb designed merely to limit the loss factor (15). Moreover, we have observed that as an overall measure of the efficiency of a controlled estimation procedure, the net variance ratio (14) is relatively insensitive in the vicinity of the optimal set of controls. These findings imply that an effective control-variate selection procedure should directly account for the net precision of the resulting point estimator of the metamodel coefficient matrix; and if this is achieved, then the performance of the selection procedure should be fairly robust against minor errors in selection of effective control variates.

Several lines of investigation should be pursued to follow up the work reported here. The formulation (14) for the net variance ratio depends crucially on the normality assumption (12). The effects of nonnormality on our main conclusions should be examined. Moreover, it is important to note that assumption (12) specifies a homogeneous covariance structure for the responses and the controls across the points of the experimental design; and the efffects of heterogeneity of the covariance structure should also be evaluated. Finally, the analysis presented here for the covariance structures (16) and (25) should be extended to incorporate a broader range of stochastic dependencies between the responses and the controls.

References

Arnold, S.F. (1981), The Theory of Linear Models and Multivariate Analysis, Wiley, New York. Avramidis, T., Bauer, K.W., and Wilson, J.R. (1991), "Simulation of stochastic activity networks using path control variates", Naval

Research Logistics 38, 183-201. Bauer, K.W., and Wilson, J.R. (1992), "Control-variate' selection criteria", Naval Research Logistics, 39, 307-311. Fishman, G.S. (1988), "Monte Carlo, control variates, and stochastic ordering", SIAM Journal on Scientific and Statistical

Computing 10, 187-204. Johnston, J. (1972), Econometric Methods, second edition, McGraw-Hill, New York. Kendall, M., Stuart, A., and Ord, J.K. (1983), The Advanced Theory of Statistics, Vol. 3: Design and Analysis, and Time Series, fourth

edition, Macmillan, New York. Lavenberg, S.S., MoeUer, T.L., and Welch, P.D. (1982), "Statistical results on control variables with application to queueing

network simulation", Operations Research 30, 182-202. Morrison, D.F. (1976), Multivariate Statistical Methods, second edition, McGraw-Hill, New York. Muirhead, R.J. (1982), Aspects of Multivariate Statistical Theory, Wiley, New York. Nelson, B.L. (1987), "A perspective on variance reduction in dynamic simulation experiments", Communications in Statistics B 16,

385-426. Nozari, A., Arnold, S.F., and Pegden, C.D. (1984), "Control variates for multipopulation simulation experiments", liE Transactions

16, 159-169. Porta Nova, A.M. (1985), "A generalized approach to variance reduction in discrete event simulation using control variables",

Ph.D. Thesis, Mechanical Engineering Department, University of Texas, Austin, TX. Porta Nova, A.M., and Wilson, J.R. (1989a), "Analysis of a multivariate econometric metamodel using control variates", in: D.

Murray-Smith, J. Stephenson and R.N. Zobel (eds.), Proceedings of the 1989 European Simulation Congress, Simulation Councils Inc. , San Diego, CA, 830-836.

Porta Nova, A.M., and Wilson, J.R. (1989b), "Estimation of multiresponse simulation metamodels using control variables", Management Science 35, 1316-1333.

Rubinstein, R.Y., and Marcus, R. (1985), "Efficiency of multivariate control variates in Monte Carlo simulation", Operations Research 33, 661-677.

Seber, G.A.F. (1984), Multivariate Observations, Wiley, New York. Venkatraman, S., and Wilson, J.R. (1986), "The efficiency of control variates in multiresponse simulation", Operations Research

Letters 5, 37-42. Wilson, J.R. (1984), "Variance Reduction Techniques for Digital Simulation", American Journal of Mathematical and Management

Sciences 4, 277-312.

selecting control variates to estimate multiresponse simulation … · multiresponse simulation...

Documents