o-a181 steps tonard am empirical evrluation of robust regression applied … · 2014. 9. 27. ·...

O-A181 320 STEPS TONARD AM EMPIRICAL EVRLUATION OF ROBUST 1/1REGRESSION APPLIED TO REACTION-TIME DATA(U)PENNSYLVANIA UNIV PHILADELPHIA S STERNBERG ET AL.

UNCLRSSIFIED NOV 66 90014-9S-K-0643 F/O 5/9 ML

I

F-2

EL.2

1111.25 IL64

Mliii 116

MICROCOPY RESOLUTION TEST CHARTNATIONAL BUR AU OF STANDARDS_- 1963-A

- -',l " w . .. . .i - ... .. .. " - w .'. .J W -- V- .- W .

OurC FILE COPYa

TOCUMENTATION PAGEA D A l8i 3 28:CC: - b. RESTRICTIVE MARKINGSI IA Tn3A ,ISTRIU I AVAILABILITY OF REPORT

Approve or c reease;2b. DECLASSIFICATION iOOWNGRA EDULE Distribution Unlimited

4. PERFORMING ORGANIZATION REPORT NUMBER(S) OW S. MONITORING ORGANIZATION REPORT NUMBER(S)

Technical Report #2$4. NAME OF PERFORMING ORGANIZATION 6b. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATIONUniversity of Pennsylvania J ( f&kabk) g nne T in Re

or incy re e185T42

B. ADDRESS (Gly, State, and ZIPCode) 7b. ADDRESS (City, State, and ZIP Code)3815 Walnut StreetPhiladelphia, Pennsylvania 19104-6196 Arlington, Virginia 22217-5000

Go. NAME OF FUNDING I SPONSORING Sb. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERORANIZATION Of AA91kable)O T N00014-85-K-0643

S. ADORESS (C111 State, and ZIP Code) 10. SOURCE OF FUNDING NUMBERSPROGRAM PROJECT TASK WORK UNITELEMENT NO. NO. NO. ACCESSION NO

61153N RR04204 RR04206-01It. TITLE (Include Securty Classficatin)Steps Toward anEmpirical Evaluation of Robust Regression Applied to Reaction-Time Data

12. PERSONAL AUTORS)Sternberg, Saul; Turock, David L.(AT&T Bell Labs); Knoll, Ronald L. (AT&T Bell Labs)1pa .Y P Of R EPO RT r 13~b.TIM E8 S p 0 T 87 ug 1,vEREYD 19 6 o em e

eChTMca 1 14. DATEOF REPORT (Year, Month, Day) I. PAGD COUNT

16. SUPPLEMENTARY NOTATION

Work Collaborative with AT&T Bell Laboratories

17. COSATI CODES 16. SUIJECT TERMS (Continue on rverse if necessary and identify by W~ock number)FIELD GROUP SUB4RI P Psychology, Perception, Visual, Statistics, Mathematics,

05 10 ReactionTtime, Memory .

19. ABSTRACT (Conow an revese if necemsy and ktidleil by or number)

Current statistlcal\ theory provides little useful guidance about how to reduce the sensitivity ofanalyses of rfteationime data to aberrant observations and violations of statistical assunptions,either because robust methods, which are very inviting, are insufficiently understood, or becauseaspecs of the data are not characterized fully enough to permit devising theoretically justifiableanalyses. In this report we suggest an empirical approach, in which one applies the same criteriato the problem of selecting a statistical method as those that one uses to select amongalternative experimental procedures. We define six such criteria, and then describe five testsbased on a et of about 36,000 observations in which we compare ordinary least-squares multipleregression as a way of characterizing the data with Huber's robust iteratively reweighted least-squares method. Results favor the robust method. K' : . c

20. DISTRIBUTiONIAVAILAOUTY OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATION

MUNCLASSIFIEOJNLIMITIO 3 SAME AS RPT. 03 OTIC USERS22a NAME OF RESPONSIBLE iNOIVIOUAL 1 22b. TE.EPHONE (Include Area Code) 22c. OFFICE SYMBOL

Dr. Harold Hawkins 202-696-4323 1 ONRI142PT00 FORM 1473. 4 MAaR 3 APR edition may be used until ehaested. SECURITY CLASSIFICATION OF THIS PAGE

All other editions are obsolete.

Steps Toward an Empirical Evaluation ofRobust Regression Applied to Reaction-Time Data

l

Saul SternbergUniversity of Pennsylvania ,0'

AT&T Bell Laboratories,

David L. TurockAT&T Bell Laboratories,

andAccesion For

Ronald L. Knoll NTIS CRA&IDTiC TAB [AT&T Bell Laboratories. U,,a 'oarco [

JLJSt:Icdto IB y ........... .............. ..................

"Disti ibution I

Avdil"JIbity CcJes

;i1t da i ,d orDist Spcial

,J /-

1. Supported in part by Contract N00014-85-K-0643between the Office of Navel Research and the Universityof Pennsylvania.

Iopoefotles In vole ofa peot is "St" to* av pMW eo of t"a

gait".tae 8405.. g..a.5

Appoevo few pu.ti6 alaaOes diltlOSeam usluLd.

p0a"Rsed La pan aw as tesemmel a5 Praoma"I ,eene pean,re. bew1061 saiesse. atwlaaam. offis tof "als 111*9 .. 5Coltraq~ olt O4il--1l Coiar~t Aetbeeltty lIdstitl pip e ]Cam es U 1 14-113 -II -Ol , e 4A-9

1 1151: 1U- I V)07

Abstract

Current statistical theory provides little useful guidance about how to reducethe sensitivity of analyses of reaction-time data to aberrant observations andviolations of statistical assumptions, either because robust methods, which arevery inviting, are insufficiently understood, or because aspects of the data arenot characterized fully enough to permit devising theoretically justifiableanalyses. In this report we suggest an empirical approach, in which one appliesthe same criteria to the problem of selecting a statistical method as those thatone uses to select among alternative experimental procedures. We define sixsuch criteria, and then describe five tests based on a set of about 36,000observations in which we compare ordinary least-squares multiple regression as away of characterizing the data with Huber's robust iteratively reweighted least-squares method. Results favor the robust method.

1. Introduction

During the past two decades there has been a vast growth in the use ofreaction-time methods in psychological research, largely because psychologistshave recognized that the analysis of reaction-time data can lead to powerfulinferences about the structure of mental processes. However, the problem ofcontamination of RT distributions by aberrant observations is one that has notbeen solved either experimentally or analytically, and as inferences become moresubtle, even small distortions in the data take on added importance.

To our knowledge the properties that have been established for all of theexisting statistical methods of outlier elimination depend on assumptions thatare either known to be false or are difficult to validate in most reaction-timedata. (We assume that subjective methods of data trimming, although used,should be regarded as temporary expedients only, and we ignore those fortunatesituations in which secondary concomitant observations are available to serve asa basis for data exclusion.) One commonly required property, for example, isthat the form of the distribution is the same for the observations in each cell ofan experimental design; that is, distributions associated with different cellsdiffer at most in scale and location parameters. A priori this is unlikely to betrue, given the factors whose levels typically vary from cell to cell and what weknow about their effects on RT, and empirically the typical sample sizes per cellneeded to validate it are difficult to achieve. Other unpalatable assumptionsthat are sometimes required include symmetry and sometimes normality ofeither the underlying distribution or the contamination distribution; both ofthese are likely to be false for reaction-time data, which are typically skewedtoward large values.

Methods that trim data or weight them differentially have often been shownto lead to parameter estimates that are more efficient (smaller mean squarederror) than others, but these estimates may be biased in small samples to anextent that varies in unknown ways across levels of experimental factors, oracross "conditions". Moreover, bias is especially troublesome relative toinefficiency when precise quantitative models are being tested, or whenquantitative aspects of the data, such as additivity of effects in factorial designsor linearity of the effects of quantitative factors, are of interest. Theimportance of additivity and linearity also argues against the use of nonlineartransformations of reaction times, which is sometimes proposed as a way toincrease the likelihood that the data satisfy assumptions such as thosementioned above and to reduce the effects of skewness and contamination.

2

lak~~ ~ ~ ~ ~ ~ ~ .9 -1111 91111,111111 1111111-,I

Robust regression & reaction time -2 - Sternberg, Turock, & Knoll

In their book on applied regression analysis, Draper and Smith (1981, p. 344)express a conservative and idealistic view about the use of robust regression:"We believe use of robust regression methods is inadvisable at the present time,unless rules for deciding which robust method to use in which circumstanceshave been formulated and proved effective. If the model (which includes theassumptions about the error distribution) is wrong, the appropriate action totake is to change the model and use maximum likelihood estimation on the newmodel, not to change the method of estimation."

The principal difficulty with the approach advocated by Draper and Smith isthat the effort needed in the context of each experimental problem to collectand analyze the data required to carry it out is considerable, and unlikely to beattempted by any but the most patient and courageous scientists.

The experimentalist who uses reaction-time measures, then, is in a quandary,concerned on the one hand that contamination and other assumption failuresmight influence conclusions based on conventional (ordinary least squares)methods (which are well-understood theoretically), but afraid on the otherhand that the techniques available to limit the effects of such contamination(which are less well understood theoretically) may introduce other difficulties.

2. An Empirical Approach

One alternative to choosing a method arbitrarily and also to engaging in thetheoretical analysis advocated by Draper and Smith is the empirical approachthat we consider in the present report. It involves choosing a data analysismethod in the same way as we choose an experimental procedure. Roughlyspeaking, we propose that a legitimate acceptance criterion for either an

2. Additivity of factor effects is important in relation to the idea that some mental processesare organized in stages that can be selectively influenced by experimental factors; linearityarises if a mental process includes a sequence of operations whose number reflects thenumber of task components to be accomplished, and whose mean duration is independent ofthis number. An example of additivity that pervades many reaction-time studies involvesthe individual differences in mean RT in experiments in which sensory, perceptual, ormnemonic factors are manipulated; & substantial part of this variation appears to beassociated with decision or motor processes that are not systematically influenced by thesefactors, processes that may occur after the completion of those operations that such factorsdo influence. Insofar as such effects when measured in units of physical time are indeedadditive, such that subject effects can be "removed" as in the analysis of variance, anynonlinear transformation would render them non-removable, and they would instead enterinto the effects of interest, biasing the estimated mean values of such effects, and makingthem appear more subject to individual differences than they "really" are.

pRobust regression & reaction time -3 - Sternberg, Turock, & Knoll

experimental procedure or an analysis method is whether the results it producestend to be orderly and to make sense.

Whether a particular description or characterization of a set of data "makessense" can probably be decided only years after the data have been collected;"making sense" here means being consistent with related findings and relevanttheory. Orderliness, however, seems to be a quality that we can judgereasonably well without using a distant vantage point. For this purpose, anymeasure of order applied to the data description produced by a particularmethod must, of course, be known not to have been forced on the data by themethod. Some examples should help to clarify our notion of order:

2.1 Invaiance of the estimated effects of one factor (Factor 2) over levels ofanother (Factor 1)

Suppose that separate analyses are conducted for each level of anexperimental factor (Factor 1). Then insofar as the estimated effect of a secondfactor (Factor 2) is closer to being identical in each of these analyses (i.e., theeffects of the two factors are additive), and there is no reason to expect aninteraction, that analysis method is to be preferred. The measure of similarity(or divergence) of the effects of Factor 2 across levels of Factor 1 shouldprobably include an adjustment for difference in mean effect size (e.g., themean effect of Factor 2 over levels of Factor 1) across methods. There are twosenses in which the estimated effects of Factor 2 might vary in similarity acrosslevels of Factor 1: they might differ systematically, or nonsyste matic ally. In bothcases the argument is based on a principle of parsimony, or the assumption that"nature" or "truth" is simple. Results indicating less nonsystematic variationare therefore closer to the "struth," as are results that indicate invariance ratherthan systematic effects.

2.2 Minimum variability over individual subjects

Suppose that a separate analysis is used for each subject's data, and we

derive a measure of the effect of some experimental factor on performance.Suppose further that the analysis imposes no constraint on the size of this

in the size of the effect is smaller. (As the measure of variability it may beappropriate to use a coefficient of variation - i.e., dispersion as a proportion

of mean effect size - to compensate for any differences in scale associated 'with different methods.) Note that this criterion can be regarded as an instanceof the first, where Factor 1 now represents a "random effect" associated withsubjects. The argument here depends on the idea that the measured variationreflects a combination of "true"~ variability across subjects plus "nontrue"variability that derives from measurement error or other sources of ''random''influences. Since the analysis places no constraint on the estimated size of the

Robust regression & reaction time -4-Sternberg, Turock, & Knoll

effect, and separate analyses are performed for separate subjects, a methodthat produces less variation in effect size must be less sensitive to these -athersources, and must thus give values closer to the "truth".

2.3 Minimum residual variability

More generally, any indication that a method produces results that are lesssensitive to random variation (relative to true variation) should cause us tofavor that method.

2.4 Linearity of the effects of quantitative factors

Given that the methods under consideration do not force linearity on theeffect of a quantitative factor, the method that produces the best linearity is tobe preferred. The argument is again based on a principle of parsimony, as inthe case of the first criterion, together with the idea that except for a null effectof a factor, the simplest effect is a linear one. Because the form of a functionalrelationship depends on the scales of measurement of both the factor and theresponse, this criterion can only be applied if there is some basis for choosing ameasurement scale in each case. Note that linearity can be regarded as anotherinstance of invariance, or additivity: the effect of a quantitative factor is linearinsofar as the effect of a fixed-size increment in the factor is invariant acrossdifferent initial levels.

2.5 Similarity of smnall-samnple and large-sample characterisations of data

Suppose that data are partitioned, by subject for example, or by level of atreatment factor, and we perform separate analyses of the data subsets as wellas analysis of their union. Then that method is better whose characterizations ofthe subsets are more similar to its characterization of the union. Because thisproperty will generally be associated with the characterizations of the subsetsbeing similar to each other, it can be regarded as another instance of the first.

2.6 Clarity of choice among models

We often attempt to use our data to select among alternative theories. Inthe present context a theory is tested by representing it as a regression model,fitting it to the data, and measuring goodness of fit. The theory thatcorresponds to the best-fitting model is then selected. A method is good insofaras this test leads to the same conclusion for different data sets.

In the sections that follow we discuss the background for tests using thefirst, third, fourth, and sixth of the above criteria that we have applied incmaio ofapriua outrgeso ehdwt riayl astsurs

comprisn o a prtiula roust egrssin mthodwit orinay lest qua-es


3. Source of the data used for these tests: Experiments on the transformation

of short-term visual memory assessed by retrieval-time measurementIThe data to which we applied the comparative analysis were generated in an

experiment in which we measured the time to name a specified digit in ahorizontal row of from 2 to 6 briefly-displayed digits. The target was specifiedby means of a "marker" stimulus that denoted its spatial location; In different

blocks of trials we presented the marker at one of six different times relative toIthe onset of the array display, ranging from 350 msec before to 650 msec afterthe onset, and denoted -350 msec, . . . , +650 msec. In some blocks of trials themarker was visual: a 50-msec display of a two vertical line-segments, one aboveand one below the target's location. In other trial blocks the marker was tactile:a 50-msec vibration applied to one of six fingers; subjects had learned thecorrespondence between six fingertips and the six possible positions that definedthe display area and that could be occupied by digits. Combination of the sixmarker delays and two marker modalities defined twelve conditions, in each ofwhich we could examine the effect of array size on mean reaction-time. Each ofsix subjects had 14 hours of practice, followed by 22 hours of testing; the resultwas a data set for each condition and each subject that contained about 500observations. Each subject provided us with twelve such data sets; theexperiment therefore produced 72 such data sets, which we treated separately inour regression analyses.

At the start of a trial, subjects fixated in the center of the six-positiondisplay area. To avoid confounding the principal experimental factor - thenumber of displayed elements (array -size) - with their separation, we placedelements in contiguou8 locations. To reduce (but not eliminate) theconfounding of array size with retinal eccentricity of the possible targetelements (and hence of the possible marker locations), we placed the arrays atall possible positions within the display area: An array of size 8 could be placed

in any of 7- s such positions.

In assessing the effect of array size (our principal goal) one has to consider aInumber of other factors that are also known to influence performance. Thereasons they must be considered explicitly are twofold: In some cases we selecttheir levels randomly because the experiment is too small to permit complete

balancing; in other cases an orthogonal design is impossible and there is some

degree of inherent confounding. Indeed, it is for these reasons that an explicitImultiple regression method is necessary, rather than a more standard analysissuitable for multiway tables. 3


The other main factors are as follows: First, there is the serial position of thetarget element within the array. Because the number of serial positions changesfrom one array size to another, and because we cannot specify a correspondencebetween positions in arrays that differ in size, the serial position effect isregarded as a separate effect for each array size. Second, there is the position ofthe target (and hence of the marker) within the display area: its absoluteposition. Our findings have forced our model of the absolute position effect tobe somewhat complicated: the effect seems to be systematically smaller fortargets that are the leftmost or rightmost elements of an array - end elements

-than for interior elements. This generates a complicatecflnteractionbetween serial position and absolute position, which is embedded in themultiple regression model that we fit to the data. The third main factor is theidentity of the target element (one of the ten digits, in the present experiment)that must be identified and named. Some target elements are associated withlonger reaction times than others, possibly because of identification-timedifferences, or differences in naming latency given the identity, or differences inmeasurement delay of our speech-onset detector possibly related to the initialsound of the spoken name.

In other experiments using the present paradigm the effect of array size onmean reaction-time has been closely approximated by a linear function, and thishas been the case for all the delays studied. The aspect of our findings that weregard of most importance is the change in the parameters of this linearfunction as the marker is delayed: When the marker shortly precedes or issimultaneous with the array, the slope of the linear function is very close tozero: i.e., there is essentially no effect of array size. As the marker is delayed,the slope grows systematically, reaching what appears to be an asymptote at adelay of about a second; most of the change has occurred within about twothirds of a second. Together with results from other paradigms that we haveused, these findings indicate to us the existence of a rapid and dramatictransformation of the internal representation of the visual display, such that theinitial representation manifests a property of direct access by spatial location,and that this property is eliminated as the transformation proceeds.

For a synopsis of earlier experiments using a visual marker, as well as thepresent experiment in which tactile and visual markers are compared, seeSternberg, Knoll, & Turock, 1q85. For a detailed account of findings from the

3. Note, however, that robust alternatives to the analysis of orthogonal experiments are also ofgreat interest, and could be assessed in ways similar to those exemplified in the presentreport.

Robust regression & reaction time -7- Sternberg, Turock, & Knoll

earlier experiments, as well as a summary of results from three other paradigms,also aimed at investigating the first second of visual memory, see Turock, 1985,and Sternberg, Knoll, & Turock, 1986.

One aim of our analyses, then, is to permit an assessment of linearity and,given linearity, to characterize the obtained linear functions by intercept andslope parameters.

4. The Regression Model

The regression model can be expressed as follows, where RT denotes meanreaction-time:

RT + a, + ,i + -y, + '.

The {a,}, which represent the array-size effect, are defined for s = 2, 3, 4, 5,and 6. They are constrained to sum to zero, and thus reflect 4 degrees offreedom (d]). The {,8,i}, which represent the serial-position effect, are definedfor each s, and for i = 1, 2, . . , s; for each s-value they are constrained to sumto zero, and thus reflect 15 df. The {-yp}, represent the two assumed absolute-position effects, one for end elements, defined for p - 1, 2, 3, 4, 5, and 6, andconstrained to sum to zero, and the other for interior elements, defined forp - 2', 3', 4', and 5', and constrained separately to sum to zero. The absolute-position effects therefore reflect 8 df. Finally, the {6d} represent the element-identity effect and are defined for d - 0, 1, . . . , 8, and 9, and constrained tosum to zero, thus reflecting 9 dl. The number of degrees of freedom in themodel being fitted to the data is therefore 39.

5. The two regression methods

We applied each regression method to each of the 72 data sets discussedabove. The vector of observations in each data set contained about 500 entries.while the parameter vector contained 39. Both regression methods are available Ias functions within the S language (Becker & Chambers, 1984). The firstmethod was ordinary least squares regression, called reg within S. The secondmethod was robust regression, or iteratively reweighted least squares, and calledrreg within S. We used the variant of robust regression that embodied theHuber weighting function with constant 1.345. For readers unfamiliar with thismethod we provide a brief description. (For details see Coleman, Holland,Kaden, Klema, & Peters, 1080.)

Suppose we have a set of parameter estimates; the first such set might beobtained by ordinary least-squares regression; later sets are obtained byiteration. Let rk be the set of residuals obtained by fitting the model with a

' A&I-t -k


particular set of parameter estimates to our data set (of approximately 500observations). Define u as a scale parameter given by the median absoluteresidual divided by 0.6745. Now define a weight, wi, as follows:

if I I < 1.34.5u, then Wk - 1;

if I rk I > 1.345u&, then Wk {.345 }/2

Thus, all observations contribute to the solution, and observations that differfrom the model prediction based on the last set of parameter estimates by nomore than 1.345 estimated scale units are given unit weight, whereasobservations that differ by more than that amount are increasinglydownweighted The iterated set of parameter estimates is now obtained byminimizing the sum of squared weighted residuals, where each residual rk isweighted by Wk. The iteration continues until a criterion of convergence is met.

6. First teat: Extent of invariance across probe delays in the effect of targetidentity

The values of the parameters {6d} represent the effect of target identity, anddo in fact differ reliably from digit to digit. To convey the magnitude of thiseffect, we determined the mean value of each of the ten parameter estimatesover the six delays for each subject and each modality, and determined therange of the resulting ten means. We then determined the mean of this rangeover the six subjects for each modality, with the results shown separately inTable 1 for ordinary and robust regression.


Table 1Range over Target Elements of Mean Target-Element Parameter 6d-(Mean taken over six delays.)

Modality: Visual TactileRegression: Ordinary Robust Ordinary Robust

Subject:1 59.1 62.7 69.4 69.92 47.4 41.2 85.7 54.23 65.7 59.7 78.8 80.94 37.7 34.5 79.0 74.65 63.5 67.0 74.4 75.86 53.6 56.1 55.1 55.0

Mean Range: 54.5 53.5 70.4 68.4

The range of target-element parameters over target elements is clearlyaffected to a negligible extent, if at all, by the choice of regression method. Thismeans that variability estimates can be compared across methods without beingadjusted for scale differences. Note, however, that there is a clear differencebetween ranges for the two modalities. We believe that this reflects greaterunreliability of the data, and therefore of estimates based on those data, fortactile than visual probes; evidence that favors this interpretation will bepresented below.

Now we proceed to the test of invariance of digit effects over probe delays.Such invariance is, of course, an example of a criterion of the first typediscussed in Section 2, and we need to consider why we might either expect ornot expect an interaction. During the time interval between the probe anddetection of the subject's response there seem to be at least four processes thatare plausible loci of target-identity effects. Recall that the identity of the targetspecifies the (correct) spoken response (its name). The first plausible locus is inthe process of deriving the identity of the target from its internalrepresentation. The second is in the process of "computing" and preparing thevocalization of the name from the derived identity. The third is initiating thevocalization. The fourth locus is external to the subject, but indubitably acontributor to the measured reaction time: the process in our hardware andsoftw are of detecting the onset of speech. Because speech-onset detectors areimperfect, responding with different speeds to different initial sounds, thisprocess shares the property of the other three of being a potential locus of atarget-identity effect.


Once the identity has been derived there appears to be no basis for anyinteraction between marker delay and target identity: we would expect anytarget identity effects in the second, third, and fourth of the processes justmentioned to be invariant across delays.

We now turn to the first of these hypothesized processes. Here there wouldseem to be a possible source of an interaction between marker delay and targetidentity, as well as of a marker-identity effect. The state of the representationfrom which the identity is derived presumably depends on marker delay.Suppose that R 1 , a "raw" visual representation is transformed during the delayinto R 2 , perhaps a more abstract representation and one from which it takesless time to derive the identity. The transformation thus facilitates the processof deriving the identity. It is possible that the magnitude of this facilitationwould be different for different targets; if so, the target-identity effect shouldchange with - that is, interact with - probe delay.

This argument does not lead to a strong expectation of an interactionbecause the existence of a target-identity effect does not require that any of itbe localized in this first process, which is the only plausible locus among thefour processes mentioned for an interaction. Furthermore, as argued in Section2, if such an interaction existed, there is no reason why it should besystematically obscured by the analysis since the regression method was appliedindependently to data from different marker delays. Hence if one methodproduces a closer approximation to invariance than another, it is to bepreferred.

To assess the extent of invariance over probe delays in the parameterestimates provided by each regression method we determined, for each subjectand modality, and for each of the ten target elements (digits), d, the varianceover the six delays of the estimate 6d. We then averaged these variances overtarget elements; results are displayed in Table 2. The range analysis (see Table1) indicates that there is no need to scale the variances differentially for the tworegression methods.


Table 2Variance of target-element parameters over six probe delays (msec 2 ).(Quantities shown are mean variances, where mean is takenover target elements.)


Subject:

1 314 154 248 1872 403 292 591 3443 330 278 751 4744 437 180 g90 3905 231 153 757 3126 432 248 727 279

Mean Variance: 358 217 677 331

Note first that in all of the twelve comparisons, of which there are two persubject, robust regression provides a reduction in variance, such that the meanvariances over subjects reveal a 39% reduction for visual markers, and a 51%reduction for tactile markers. The difference between regression methods isstatistically significant (p < .01) for both marker modalities. This findingargues strongly in favor of the robust method.

7. Second test: Extent of invariance across probe modalities in the effect oftarget identity

As discussed above, we can conceive of a possible basis for a systematiceffect of the delay of the probe on target-element parameters. On the otherhand, we can think of no reason to expect differences between target digits todepend on the modality of the probe. Indeed, the remarkable similarity in theeffect of delay on the slope of the function relating mean reaction-time to arraysize is consistent with the idea that once the critical location has beenascertained on the basis of information provided by the probe, the remainingprocesses, leading up to identifying and orally naming the element in thatcritical location, are the same, regardless of the modality by which the locationinformation was conveyed. It follows that we expect to find invariance of thetarget-element parameters over modalities, and that insofar as a method revealsgreater invariance, it is to be preferred.


To assess the extent of the invariance over modalities we proceeded asfollows: For each target element and each delay we determined the variance ofthe (two) target-element parameters over modalities, by calculating half of theirsquared difference. For each subject and each regression type we then averagedthis quantity over the six delays and ten target elements. The results aredisplayed in Table 3.

Table 3Variance of target-element parameters overtwo modalities (msec 2 ). (Quantities shownare mean variances, where mean is takenover target elements and delays.)

Regression: Ordinary Robust

Subject:1 247.8 122.82 540.1 364.73 720.1 436.54 930.9 343.75 417.6 166.96 573.6 224.7

Mean Variance: 571.7 276.6

In all six comparisons, robust regression provides a reduction in variance,with a mean ratio of 0.49 ± 0.05. Like the finding in the section above, thisadditional application of the first criterion discussed in Section 2 argues stronglyin favor of the robust method.

We have thus found that robust regression produces estimates of target-element parameters that are more invariant over both the delay and modality ofthe probe than does ordinary least squares regression. One possible "truth"that the robust method therefore brings us closer to is full invariance of theseparameters relative to delay and modality: in that case any apparent failure ofinvariance would reflect nothing more than the influence of sampling error andcontamination in the basic reaction-time data on the estimates of target-elementparameters. A hint in this direction is the approximate equality of the variancesdisplayed in Tables 2 and 3, within regression type.

To test the idea of full invariance we subjected the target-elementparameters derived from each of the two regressions to an analysis of variance.


with the factors Subject, Modality, Delay, and Target Element. To avoid effectson the analysis of the linear dependence among target-element parametersinduced by the constraint that requires them to sum to one, we analyzed onlythe first nine of the ten parameters.

Values of F-statistics from analysis of the parameters derived from ordinary(robust) regression are as follows: In the first three F-tests, we evaluated a maineffect by comparing it to its interaction with subjects. We obtained, for targetelement, F(8, 40) = 9.53 (10.81); for delay, F(5, 25) = 0.90 (1.33); and formodality, F(1, 5) = 1.18 (0.08). For the fourth F-test we compared the subjectby target-element interaction to the four-way interaction, which we take as aresidual mean square; the result is F(5, 200) - 3.54 (6.87). Finally, the residualmean square for target-element parameters from ordinary (robust) regression is543.4 (264.1).

These results generate two clear implications. First, the values of F for delayand modality confirm the conjecture of full invariance: the observed variation ofparameters over delay and modality is small enough relative to the residual tobe entirely due to random variation. This conclusion strengthens the inferencesabove that favor the method producing greater invariance. Second, the amountof this random variation is halved by moving from ordinary to robust regression,as measured by the residual mean square. 4 This result means that the thirdcriterion of Section 2 also favors the robust method.

8. Third test: Deviation from linearity of the array-size effect

As mentioned in Section 3, we have found the relation of R T to array size fordata averaged over groups of subjects to be remarkably well-ap proxi mated by alinear function at all marker delays, with a slope that is close to zero fornegative or zero delays and that increases rapidly with positive marker delays.(Data from several experiments that support this generalization can be found inSternberg, Knoll, & Turock, 1985.)

In applying the regression analyses under consideration to our data weNwished to assess the goodness of fit of such'a linear function, so we deliberatelydid not impose a linear constraint on the parameters {a.1} that represent theeffect of array size. We are thus able to compare the linearity of the estimates ofthese parameters derived by means of the two regression methods. Such a

4. This reduction in noise is equivalent to doubling the size of the experiment which, for datacollection alone, would have added about $7000 to the cost of the project.

M" mnr


comparison bears on two of the criteria discussed in Section 2: the fifth(methods are preferred for which small-sample and large-samplecharacterizations of data are similar), and the fourth (methods are preferredthat produce better linearity).

We performed the comparison as follows: Each method provided a parameterset {a8} for each of the 72 sets of data. We fitted a line to each set (by ordinaryleast squares regression), and extracted for each parameter set the residual meansquare after fitting the line. The square root of this quantity is the residualstandard error (RSE). For each subject and regression method we determinedthe mean of the resulting RSE's over the six delays. The resulting measures ofdeviation from linearity are shown in Table 4 for each subject and markermodality, and for each type of regression.

Table 4Measure of deviation from linearity of the array-size effect(RSE in msec), averaged over six probe delays.


Subject:1 7.33 9.83 12.17 10.832 10.33 10.50 21.00 16.173 11.83 9.83 21.83 19.174 12.50 10.83 22.67 13.175 9.17 7.17 16.67 14.336 7.50 9.67 22.17 11.67

Mean: 9.78 9.64 19.42 14.22

While there is virtually no difference between the means for visual markers,the mean RSE for tactile markers (considerably greater) is reduced for everysubject as we move from ordinary to robust regression, with a mean reduction of27%, and is brought closer to the RSE for visual markers. (Note, however, thateven the results from robust regression differ significantly between modalities,with a mean difference of 4.59 ± 1.35 msec, so that although the robust methodcomes closer to the desired invariance, it is not achieved.) An analysis ofvariance applied to the mean RSE's showed that regression type, modality, andtheir interaction were all significant (p < .05); the effect of delay was alsosignificant, but none of its interactions were.


Although the advantage of robust over ordinary regression in thiscomparison of array-size parameters is less dramatic than in the twocomparisions based on target-element parameters, it is nonetheless present,providing us with independent but consistent conclusions.

9. Fourth test: Extent of invariance of the array-size effect across probemodalities

The mean array-size effect is estimated by the slope of a line fitted to the{a.} parameters. Because the magnitude of this effect changes dramaticallywith delay, and because we have no principle to guide us in aligning the delaysfor probes in different modalities, it is hazardous to compare the effects of arraysize across modalities. Nonetheless, in the mean data it seems clear that byassuming that physically equal delays for the two probe modalities correspondpsychologically we achieve remarkably equal slopes. (See Figure 12 inSternberg, Knoll, & Turock, 1985.) Thus it seemed reasonable to expectinvariance of the array-size effect, averaged over the six delays, across probemodalities, and thus to ask which type of regression produced a closerapproximation to such invariance. Results of an analysis designed to provide ananswer to this question are displayed in Table 5.

Table 5Comparison of Array-Size Effect Across Modalities.Quantities shown are slopes, B (in maec/element), of lines fittedto the {a,} for visual (Bv) and tactile (BT) probes (in mnec/element)and their absolute differences, I Br-BT I.

Regression: Ordinary RobustMeasure: Bv BT IBv- BTI By BT IBv-BTI

Subject:1 19.4 11.3 8.1 16.4 11.3 5.12 19.0 20.2 1.0 17.4 18.3 0.93 21.1 22.8 1.7 19.4 21.8 2.24 22.9 13.7 9.2 19.5 15.7 3.85 19.4 21.3 1.9 16.9 18.2 1.36 26.1 22.8 3.3 22.6 19.4 3.2

Mean 21.3 18.7 4.20 18.7 17.4 2.75

I*1


For five of the six subjects the robust method produces greater invarianceacross modalities of the mean array-size effect over delays, by this measure. Thedifference (1.45 ± 0.93 meec/element) is not statistically significant, however, sothis test is inconclusive.

10. Fifth test: Clarity of model comparisons

To compare ordinary and robust regression with respect to the clarity ofcomparisons of goodness of fit of alternative models, we fitted three regressionmodels, competitive with the model described in Section 4 (which we denoteModel 1), to each of our 72 sets of data, using both regression methods. Theexact structure of the three models is not important for the present report, sowe provide only rough descriptions. The three competing models differ fromModel 1 in the structure of their serial-position and absolute-position effects.

Model 2. Two separate absolute-position effects are contained in Model 1,one for end elements, and the other for interior elements. In Model 2 nodistinction is made between these two types of element: The same absolute-position effect is assumed to apply to both. As a result there are fewer freeparameters.

Model S. For any array-size in Model I the assumed serial-position effect isfitted in such a way that the position of the array in the display area isirrelevant. Thus the leftmost element in an array of three elements that isplaced as far to the left in the six-element display area as possible is regarded ashaving the same serial position as the leftmost element in an array that isplaced as far to the right as possible. In Model 3, separate sets of parameters arefitted to arrays of different eccentricities and to centered arrays, and, moreover,serial position is assigned symmetrically for arrays in different positions, suchthat the leftmost element in an array on the left, for example, is regarded ashaving the same serial position as the rightmost element in an array on theright. This model has more free parameters than Model 1.

Model 4. In this Model we use the simple absolute-position structure ofModel 2 and the complex serial-position structure of Model 3.

For each of the 72 data sets and each regression type, each of the threecompeting models was compared to Model 1, using two methods, as follows: Inthe first method, we computed root mean square residuals (RMSR) for eachmodel and then determined the number of data sets favoriag the most favoredmodel in each pair. The second method was the same, with the RMSR replacedby the (more robust) median absolute residual (MAR). Results are displayed inTable 6, where the model whose number is shown in boldface is the favored one.


Table 6Number of 72 data sets favoring the favored model.

Statistic: RMSR MARRegression: Ordinary Robust Ordinary Robust

Models:1,2 72 67 47 481,3 61 50 51 511,4 49 46 48 46

In this table, higher numbers indicate clearer choices between models. Ofthe comparisons using the RMSR, all three favor ordinary over robustregression, but with a difference that is relatively small. Of the threecomparsons using the MAR, robust and ordinary regression do about equallywell. More microscopic analysis of the data, in which we considered thesecomparisons separately for different delays and modalities, added nothing to theimpression conveyed by Table 6: For these data and models the regressionmethods do not differ substantially in relation to the criterion of clarity ofmodel comparisons.

11. Summary and Conclusion

In this report we have advanced a notion about how alternative statisticalprocedures might be compared in the absence of adequate characterization ofthe data in terms of requirements of the procedures and/or adequateunderstanding of the procedures. Our proposal is that the criteria normallyused to select among alternative experimental procedures be applied in thissituation. These criteria are often only implicit, however, and may well dependon the field of study (and quite possibly the particular investigator); hence inSection 2 we defined and discussed six such criteria. We then described adomain of data with respect to which we have thought about this issue, andintroduced the multiple regression model that we have been applying to thesedata. We described the two methods of fitting the multiple-regression modelthat we have been using, one being ordinary least-squares regression. the otherHuber's "robust" iteratively reweighted least-squares method. Finally wereported our findings from five tests in which we applied the criteria definedearlier in a comparison of the two methods: In two of the tests they did notdiffer substantially, while three tests favored the robust method.


References

Becker, R. A. & Chambers, J. M. (1984) S: An interactive environment for dataanalysis and graphics Belmont, CA: Wadsworth

Coleman, D., Holland, P., Kaden, N., Klema, V., & Peters, S. C. (1980) Asystem of subroutines for iteratively re-weighted least-squares computationsAssociation for Computing Machinery Transactions on mathematicalsoftware, 6, 327-336.

Draper, N. R. & Smith, H. (1981) Applied regression analysis New York: Wiley.

Sternberg, S., Knoll, R. L., & Turock, D. L. (1985) Direct access by spatialposition in visual memory: 1. Synopsis of principal findings.Unpublished manuscript.

Sternberg, S., Knoll, R. L., & Turock, D. L. (1986) Direct access by spatialposition in visual memory: 2. Visual location probes.Unpublished manuscript.

Turock, D. L. (1985) A new technique for measuring transformations of visualmemory.Unpublished manuscript.

----------------------------

Dlaltbut~an LUst :Pnn1S. sp-ornril !IN 5453?

f bllp L.AOO&M ArUWZla cae ts iUniversity ON Niaseoos LOWY AM3 Co 80230-500 cetfor Neura scioeeNtaaeaPet.. 655 Met 04DI 4mgA. @.30

1olergets Coaet g foraderoab r ynA aWAi eSA 2*0U*0 Lft aCbhtlaua., Voldg& a Learning no5 centerARItM&AM 1424 Rabnoba X Uiversity of pletabrghSwooa AM,. 12 7325DA 30 ODsae streetFUSSasrob, PA 7521)ArM DR- Gall C**,aew

Life sstee of"rotee xU"*a*%6M wavfe..tv C40%. Jet"~ Cerweta jeasinSel1sor Its argo ae"* O~Netta 09 Masbomaltes. 504za NeRAlb-?& MaprtleaftSM9Se. 00 2033S 3"5 Wasuagt AV@" 91804104 nto o U ewtee 0s IPee...5moa Ml 02N, A11S lice.S so Cowelgete,

Cad M711 Or. Pa'a Carmateu

@aleaj, uuS 32513 V1SS.Umu92 PA 15212 Va"efeee51 aftleymuG " Latw.ftnouauwjua Anevf MR. s 3tmt . asteul ?500? facts. FRAM~

Catesw ewves Beta" fto"St *1. = 70235 ft as a k nftevdes, as6293 limerltys eebCoe.0Pradesis" as ?lesfaes lihiala tgn"Cgn

PC* Slte a Asam.ew ftbwies 00UeaMeS "tesal"Dubwaft o pnaeflef National vaiveS t- i,,,st 2334 As Lelies. in3 USIREWS9Wievelow 00 "u"loe I Tpl AEAwCelle" ftgb. no 2ao4 aa DOWles.9 Dr. 50911 . clesmm AVUMLaM

O&5 tfli, AU2 3S"eDm" ofP3ehl LOWY, AM, CO 60230Alemesiffa, VA 23W33 Celaba& so29225 or. JinI Defte

te 0 am@*so 09 " Ua"*"*eorOf. Alam Sedieler Dr'. caw&" 42180001 Ca" 1141WMed'oS1 sowe mmu leftis Pan no Sam Smm #reoApleePess1g usit a VAemm o ft" r Puuwltsgom, ft 3221 7-1000IS O O ne fdtveeiiap a# NNW.

99IM~ ft~ti elm0 Mo Sh" Dom

Cu"~s of suvesq aLAWf sepsmov " voyawlBONNeae a" "alum*si 25msUui em a~ 35.hA~ I3 S 333ls hagge. 0*352 pi A m sft aeineatm Weeo'e LUw Set

2). MomeMs so ~~e I ev SO 1 Pmo"ASS0 34MMA090 aa ~ . W6mmVA " &ssft @i so Lit Scienc362e

or. 868641 owe 0910 wIY e, 9fhaft"&e bi0amm"M -* mani savu afaf me fmSe so B 2

M &M* A 70"0 e embmsmoosss 124 0el1;;

@tIA. I. 213 ,56 Peams F Sfam t amlef Abtetuapa..maso a.gnm a. US, somee Wes xlimi

9-901000ot~ 80 aO le""0l010Cb@Nooi"*3522 3ab 610204itCb

so CNGO"SM&IS* af9

pa lrdtho m. V. notchetBrown Univerity 9931 Cousea~ Street

or. Stephan 0:oeerlCenter for Adaptive SY3tenl

IAtoUy 00014rtat VIia& VA 22180II Coago Stsue. in 244

MedilSabs 31052D *Jhe.ftdihSkte. Univoerityp r v d s* s G a i z j ob s 1k F r e ~ i k s e V s o , A @ 2215

Un1Yet3is7 ad Californaa CeabzidgO PA @2130 Unversitamof g*"1 3arol

SanSB Dieg 3.taticsDopareatlat 01LliIties C-Cog ft. * Eloela "lla~h aapol Bills me3 27514Z'4 Jolla@ CA 92093 University of wea C420ltaa

cepar"Wse of parowevyg Fre Satd saaetelD Bgaes as Usee Chapo& Bil1sWe 27914 scoo of ad .atio

Universuty of K&Sme Staahw4 UniverdityPershel.Wr Depeatas D.Os estOOS #aafied CA 9430 5LAW90009 XX 4604S casw ahS oo

tagmgafttOSi DVsee ft Mrooy a. 3*111ft. Rando yle 36gb g swoOf CtlamO~a 93119 ooftO*laDopat'Utf e oh La, J4111, CA941 4016 4 3 4 Ma e fthUniveresy of got" CataLIna * Affligton, VA 22207Columbine S60 29208 or. Gem L. 010"

Dr. wiUais Npisi DoesumeIfsWvaeAfty oil Viaemsai 10303.- DVOM 01O Orlando* VL 2261)V. J. 1ei 051-powumeg 3149. Ch 10630132 ff. joaos atiet .. Dr. toy leoapellNa01ie VS 537" or. 63Seel Wasn esiestifie ami saftneezial

ftieomi l"eiwiit Paoveouas am Seatios,662 Vality.*aOqiittm DOPeStENeso fpeyaoaa2w watiosal Selesee Irniation

46)3 Begby Ave are" 3. was" 1% es. De 305103be osas ND 26014 Ptiao$oU 3M a6ss"

Dr. X. *14gMs 20,411910 DV ft""~e Gogbes getters n !behavioral and

DOP&Namms of P6766914117 a Ueaagoms 20 Bas $teot, suits 240

Or. pat" uaha ago DV. 3tO A. Mt11ya14DOP&CSaMt" Of Pprbeley or. Ioose 6so" Deprtment of woureseleneeicauudviessillo 9mlwualty *1ERAO Uivsi~ty of CalfornaIshsatey park 800600 Mot i 1 089 763m V iegepittass, PA 15318 . La Jella. CA 92093

or.fu bo"fre PbS.

Agwy 24006CAitsbo ~NflS taosta. clarsgio-Relles UnivaestySee "lebsw w ISS6 8496 &no*~ so weV com~o lofos Departnent

"Iolsit* &3 M mmisfte 0 USIT6101 PletaMRS. PA 11213

Oro ausbal so. for or. Nupo Saw Do. JobS lelland2220*1 SOSSUu 110mss60160. fesat wu Ualuosotty SO Nlo43d9

2526 ftnb I& MI oST 9" 1 a u 22406040 13 at I0mohfagA lwm. TA Sam3A3 A~*on NZ 46109

caf. III1 or. am*t @ges Ow. Lud Nomura"

la i's o i job" e a"""t O MAlyIbIg ouniestym of IleloiOwl.. 4 346b Oftet 46) set esto "1 11101

0'v. ifto &. roa ouathimov w aisl6 CUeepeig. IL 61820GV ~iv lt, 4w Usaboes acoo""sons emeefte" or. 31 hum feftoo or. am% "ast20866410409 ow 140? IMAS,6161t 49 ZiMS Dopoomemt of Pgyolog

Depfet~ of tordoa iobegvon gm saty of VaghififteScusspalg. aI Sim2 loattle. VA to10s

1' ~ ' J ~ *~

Clv. of 8" czolua or. Scott Rolfe Jae 3 Mc0 ride4u n v . o S o t C a r o l & a u k ia s L a b o ra t o i e , s h o w l a c r p a t n

270 Ciume CT041 2o oxf teet * , ,Dr£1100 180i San06 8 Diego, CA 32101

Depatment of Puvehossa?Di.zat o arln David zieimiDi Jay bocleoland

cat wVill*, 21220 univesity of Msloha 0ptamut 09 Psyebeogy;"e.-. ~sO chef Joe&im Toch.COM-Collogo 09 Iai00 iIot C8aetoemaounibversityDivdo Pes-Cnr " 50bo 1222 1. IaIaeorla 3.1141 Pittabwzch. PA 15213Serice do $4to "a LAnne@ Ana Aa3.. US 40109lag aselevea Plaoi ;Di. James L. 36086gb63372 Zyes Cod"s 030 VRMmD. David Exami Costae for tus aesouielegy

Casegi~e-Melles university Of1 M Le a" Usserymgo. Noboru Jesusase Deprtmeaft of PSOb0l"Y university of Calfornia, Irvine

OPt s0 of 1.gyf Schooley Park Irvine* CA gam1mv"eety Of Bomb curess" Pittsob, PA 15213 i ~ eebaC@4="& SC 29260Of "H4"A

toDr. Sylves Kerablm Wvy Persol Dam CosterCOL. 3eis go Jarl univesity of misbigna S63 DIOge. CA 92152cioa""o X&txnet eee"ttitae ..

Am205 Veobemw Plaoe o. Jaes Nfelliheeasso An, U3 7823S5640 1 AM Arbos 33 4010 ASiotat ter M Reee&Me

to Developmet* a" studios

as0 jeam s eguam ftversity u~ 10 t7W blgo.D200Uetimmbme. 21310 1220 Ili11m 1.3. mal1

33 Eiklad as. Dr. ftsiago a. mit"Col. Doudal~o J41MoLIS do aa Ca3dgs 3602130 De"Imeat 09 Perebelegstatt'"ser 60 I Aamg do Tome to Gas VMAICemto de o tlm 50660. "a Dr. Dai a81 . Urasta Pilses Dalversity,3 Mimesis Oaf a Ges", a Vashlaytom 846age villag Poleetos, wa eaSes7007 ftria 95353 AV%. * 118 1

nwe Tooke U! 10012 Dr. Im moeavs. Munn% Jamt zoom vanCasoliemilm usivemelt Dr. Nor67 Lamgmn 323 Coyeft niti NeeDopmeomo ad PorboRww U"Slve lt so novak caella P416 1169 CA 94344sos" PW The bo. 11bkeloemol "Dos.ritasabr, M MIS 15212 081 fi OltD. Deam Saws

00chapes Si.U6 a 7514 Uotute tow Coweilve belenseor. 9=191 maksomom La J@1le a Calit2302

Gk i ril so2052 club Dr.a Hieil :ffow's Uojts 80A CAoe92093

10"a. mu 3100 SONG&U 615? M

Gamm~s wne& GMopgal Di. M bof & apO0" 001

0 0 A sia XoAlW 410o1o. sunwim 0. I 00 U.4 0

no~ G .. ag* so"~ . so Go1m w SUP lagtoS. 30 26372i

Di.1 IN60"05 of ASM& amti ad"&s" 71? ""

AO 809 SMiO 1. sw n sltymgb PAUMM W 2313

t~o. 630443 ad some@*e 30. 44awor. MOM go, PB"O Z11U16 a 950017 uf414t9Wao"Mus of owe~ew 00 pIft~vq&1wad "I -or. n Sa PA 23I

a 0144

91. 001j7 Of NIAM68es cal MaCtte$ io COau0014 Lt. Jose Puenteo rtanijj&Uaivouit7of uiaemt W.~ fti .O0 C/Saattia Tta34a. 6. 4

0318 SlUott 3011sPVOtI) t 26010 Masdidxismpgel, X3 55455 Arlington, VA 2332?-:300 IFA

31o3sal& A. "MaS atulaniel2611149

ativato for cegin*Uve Delento Offiee so loyal 29Aeeea of. 36*101ogtv ad amersis Liast"* office, Far sa33 De"ates *BP~fi19

La Jolls, ch 92093 1 e rmie.C 0 $1 ee Ftft Ocil o

DI~egw. t~al~ Lheetey. Dr. Judith Owsas now 146110 WT loads1116 ede Oilsa Au4"reatomatia0 19 * ee (232 1001r siensu" Aantft cc. Daoid monelhan

iLseetui epeo a btseael £laadxa.YA222 0a9us CaliforniaLa"eiatfty Me13 WCae 04) DeAse, Palson La :ol& CA igeE93a"e Diego* MU ., Code 12 - Traalo ssinaJll, A923

" povemt no Can nt& atelisv

A ro $"*amaU~o Lou& Hos.363E.owf Wiog seottM(ee all or.au P4yeOZe OausoiPce1

0ea 914900 CA 92112 00330s3"a tfo of-e Walnutg, 10o. FZI.

~lo apeO C~eP .O. am 710%migyms (God" 301) PeOSlad OR 119?7W~a Le&rNateies

aft Ste",e CILg 225 70 efew suree"£l~euy. Wag gemCside IP201 Itoam 200 " ie

a Nsee CA 92158ommwW 4114160 Tale lad.UWritVooCAA ooe" g0.so re kyoef

DR. "" 1. Ooeaze*. . amS I IA. Vole nttiOft i v w o w d B o o m " C & M S A 4 s . J a m g p o z o o rM eg o s o e C T 0 0 5 3 0

aols. of aUbmeiowal lae Babaa r- ba ee

ol va4 a 4g"O veA w , w 21P U@ of Sm"% ~ CA 92 50W aesea 215801141

C~e 11W111w ASadO S wsaag- ALeMSOGa VA 22333

Cede 114111 --. Ie 16004ur o go"euaieb3003.1 mIse eo ne la 000 Mtno hrvlmbvs"4"Afte 6 W wag atseffNW M06 3SOv movesMletlae

&%G1mfm & -M Loosslag 140 coveoe b&Yvesio so qittabetyb

cc""e 4 sent 3temm we 3930 01hr elaeCede 1148W a *49010 ra ISMeu, A1

mu.elp Mr. dus-U ae~

fo1owof is Sto. fer ftyadow am1 AmebaeOMSO so MISS Dm mwo to @las". Jae eeee ovieehes 17 s. v2iCode 114M We low. vs~ In0 dcf"a ff. mseyI @Nop 1b'ow Se oepa nclwMAMI=$~m Va 2806090 asposam so 9oyabw,

oo o ft gone S ju Joset&&*0~~~~~ IM,.~ at se

Goitse vs" sa.s 27-

no m vae.i owis ~S-01 *4 oom D. osoamdSlip 421 e ,jDept."4 NVA.Su. x"

Ham~wo.CA 001WCa$.id~o VA 02139 I

gr ikusi Tate'aokaLIC WTh.*i lnS ivio CZ%, 252 tnq.ResarchGea.fqtrs. Training SectiDon lt 353 [n .Reoos WoNilitYa Psychology 06 :De L 61501 Dr. SOcb-Sian; LenPT. 919 Jackson State University81-00101 ielsiakt 10, 11MAND Richard F. hompson 1325 J. R. L7MC. St:eet

Stanford Univerity Jackbon. ns 39217r * . Deparnt of Pyohology ougls Wetel

Bolt Beranek & Newman, raG. SDr.. 4201 -- Jordan Nallso Multon .4Je1t Code 12Cambridge, NA 0213 Stanford, CA 94305 Navy personel R&D Center

;. 1i©hard ** *o Dr. xMchael T. TuVVy San Diego. CA 92152

or. Richar . snow Hskins Laboratories abr htDepartment of Psychology 2elCin Stree Dr. larbars WhitS20ord A S4ree0 sl Beranek & Newsa". Inc.a" go rowRven, CT 04510 Io tl r n t • vu ,%a

Stamford CA 94304 10 moulton Street

Dr. Amos Iversky Cambridge. MA 02235"t.xr iathees ?. s~esleyr Stanford University Barry whit**

grm University Dept. of Psychology

Depr nt of Psyhology Stanford, CA 94305 University of North Carolina

proaidenae, of 0291 Departmnt of PhysiologyeDr. Jame8 Tveed4ale Medical School

J.,se J. st8 sevelsi TObsial Droetaor Chapel mll, C 27514

Reear" Associate Navy Perusonel D Center Crisophr Wckens

Carneie-ellas Univesity Sam Diego, CA 92152 Der. e to Vichens

Dept. Psych. lSadsley Park * Dapar~sea of Psycholo'y

PittioMrgh, PA 15212 Dr. V. a. I. UpVulrt Universit? of Illinotsunion Carbide Corporatios Chapsailg. IL 61620

D. Tad Steinke Nuclear DivisioeDept. Of Geography P. 0. Box I r. ober A. Visber

livereity of soah Carolina Oak aid", TX 37830 U.S. Army Institute for the

Colesb1n, IC 29306 .. ebavora). and Soeial sciences

o8rto.U. S. Marine Corpe 5001 Xisenbwer Avenue

Dr. Robert sternberg Code PZ-20 Alexandria, VA 22333

Department of Psychology Washinton# DC 20200 , wieoef

Yale Univereity Dr. Marti .

Sn 114, Tale Static D. William Uttal Navy Personnel a & a Center

Nev avesn, C? 0530 NOses Ravsi Lab Sam Diego, CA 92153o u 997 ;t . o u . olfe

or. fe l Ste nberg ai l a, 312 96734 ron el a colfe

niverelty of Pensylvaia Navy 0e"oasel MD Caster

Departent of Pyoelegy Dr. J. V. 11. Van fewles laa Digo, CA 92152315 1Wlnut Street A9d.See.Vfteuse hapelijk Onderooer/Dl D.

Philadelphia, PA 19104 Addtalltitge b Dr. Donald Woodwardvan Der urohulas 31 Mr. 374 Office of Naval Research

Dr. Paul J. Stieh 3500 = "s-4rve he, 1mmuZ uoM Code 11413Senior Statf Scientist o 00 orth "ne streetTraing Reearh ivia D. Wei Aglingtnu, v& 22217-5000

1100. eass&"& 0ueeaiees Or. e100ee Dale

Alesmatla, VA 22214 PrisMetes. W 0851 Navy rerse l M" coaeSan Die"o, CA 92192

Sdee Pilippe Stivelet Or. br TwanatieDiv. Psy .Cstre Beebeahe f elt 1aeS & eiras. Zae.ervie Ge S e dee 14m4s N 602MAa 2re6 AnMM

101 aeelevad Pie" CAido s NA 0@130 LOr An, Co 00230

69272 LYge Cedft 03, VIA 0Dr. 3. J. N. Vaeseube or. Jeese L. Tou

M. Brad 8Ues aued f aft. Se l. My. meow.£.F. Namer? L CogaitsivNavy Perem m , no Co n u4,. Geie"voSetes"appesOMMV preoa ese n41m Di090. ci 92153 3isMretlaas 1N M. X4 patiasba emese od oS

ol. Dr ?Gag&" Sel - ilbergo Dr * 1tave* Sornetl ex

ellag AM, OC 20332 gajreity ef CalieflS Of ie of Naval SeseaorbCse f te eueb . c"Me 1140af LAwUiag ad bN eh 000

. Guise? St.

IrieO, C~9 93711l AIligt VA 22217-5000

* a !~ *~.- ~ * P * ~

o-a181 steps tonard am empirical evrluation of robust regression applied … · 2014. 9. 27. ·...

Documents