wi1th / multidimensional'item response data(u) city ir … · mark d. reckase research report...
TRANSCRIPT
AD-A125 099 THE USE OF THE GENERAL RASCH MODEL WI1TH /MULTIDIMENSIONAL'ITEM RESPONSE DATA(U) AMERICAN COLLTESTING PROGRAM IOWA CITY IR R L MCKINLEY ET AL.
UNCLASSIFIED AUG 82 RR-82-i-ONR N88@14-8i-K-98i7 F/G 12/1 N
IMMEM on MINEIEhIhI1hhhhhhhhhE111EhhhhhhhEND
-~ ~ ~~2 -- -- -
1.0 ki IL .I~632 ItI E _
"1.25 .-
MICROCOPY RESOLUTION TEST CHART
- PATIOW WRMf OF STANDARDS-1963-A
. -. * ..
The Use of the General Rasch Modelwith Multidimensional Item Response Data
Robert L. McKinleyand
Mark D. Reckase
Research Report ONR 82-1August 1982
The American College Testing ProgramResident Programs Department
Iowa City, Iowa 52243 DT1C
Prepared under Contract No. N00014-81-KA817
__ with the Personnel and Training Research ProgramsC" Psychological Sciences Division
, ¢Office of Naval Research
Approved for public release; distribution unlimited.-.J- Reproduction In whole or In part is permitted for
any purpose of the United States Government.
83 03 02 016
"'-. - "'Y':'., ', ,'% '-:'. :. ."_ . -. : " :-". : . :-""T . , ." , : -."•"" - ": :"A ,: : -
SECURITY CLASSIFICATION OF THIS PAGE ('Nmen Data nIterod.
REPORT DOCUMENTATION PAGE READ ISTRUCIONS' BEFORE COMPLETING FORM1. REPORT NUMBER 2. GOVT ACCESSION NO. S. RECIPIENT'S CATALOG NUMBER
4. TITLE (end Subtitle) AS. TYPE OF REPORT & PERIOD COVERED
The Use of the General Rasch Model with Technical ReportMultidimensional Item Response Data
6. PERFORMING ORG. REPORT NUMBER
3 7. AUTHOR(q) 8. CONTRACT OR GRANT NUMMER(a)
Robert L. McKinley andMark D. Reckase N00014-81-K0817
9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT PROJECT. TASKAREA & WORK UNIT NUMBERSThe American College Testing Program P.E.: 61153N ProJ.: RR042-0
P.O. Box 168 T.A.: 042-04-01Iowa City, IA 52243 W.V.: NR150-474
11. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATEPersonnel and Training Research Programs August 1982Office of Naval Research 13. NUMBER OF PAGES
Arlington, VA 22217 2314. MONITORING AGENCY NAME & ADDRESS(i! dilferent frog Coltrolling Office) IS. SECURITY CLASS. (of this reponj
UnlasifiedIS. OECL ASSI FIC ATION/DOWNGRADING
SCHEDULE
6IS. DISTRIBUTION STATEMENT (of thl Report)
Approval for public release; distribution unlimited. Reproduction inwhole or in part is permitted for any purpose of the United StatesGovernment.
17. DISTRIBUTION STATEMENT (of the ebstract ented In Block 20 it different Imam Repot)-i
IS. SUPPLEMENTARY NOTES
IS. KEY WORDS (Continue on reverse aide if necessary a Identify b bock number)
Latent Trait ModelsGeneral Rasch ModelMultidimensional Models
20. ABSTRACT (Continue on revese aide if necesiy mad identilt by blak mnbee)
-- Several special cases of the general Rasch model, varying in complexity,were investigated to determine whether they could successfully modelrealistic multidimensional item response data. Also investigated waswhether the parameters of the model could be readily interpreted. Ofthe formulations of the model investigated, all but two were found tobe incapable of modeling realistic multidimensional item responsedata. One of the remaining formulations of the model was found tohave limited applications. The version of the model found to be most
DD I A 1473 EDITION Ol I NOV I5 1S OBSOLETES/N 0102. LF- 014- 6601 SECURITY CLASSIFICATION OF THIS PAGE (km, Date Bitoo0
........ ..... ....-.-.. .-.. .......--.-.... .
SE[CUWTY CL.ASSr4ICAT@W OF THIS PAGIC t D. ~A
- usefu1 i8 an extension of the two-parameter logistic model.
S/ 00- F01-60 ECRIYCL~~llATO O HI P6[(li~ at hxm
-,'.V. , , - ., ,.,/:" ,-', ". . , .-. *, , ,., , . .,/.-.". ,. . .. /'- . . --- , .- . o
: :j CONTENTS
~Page
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I
Multidimensional Latent Trait Models ........ ................. 1
Summary ....... ..... ............................. 4
Method ....... ....... .... ........................... 5
Design ....... ...... ................ . .......... 5Analyses. ........ ..... ............................. 5
Results ........ ...... ................................ 6
Vector Model ........ .... ........................... 6Product Term Model ........ .. ........................ 9Vector and Product Term Model ...... .................. .10Reduced Vector and Product Term Model . .... ............. .12Item Cluster Model ......... ........................ .15
Discussion ....... .... .............................. ... 19
Vector Model ........ .. ........................... .19Product Term Model ......... ....................... .20Vector and Product Term Model ...... .................. .20Reduced Vector and Product Term Model .... .............. .. 20Item Cluster Model ....... .. ........................ .20
Summary and Conclusions ......... ........................ .21
References ........ ..... .............................. 22
jNTiS GP..&I
2 DO ~ ?1~TAB
. t- .at i
IAvsta1 iyCe
Av 11 it -( 1r
* *A f
, K o , , _.:.; % .,,. ,_' . . . ., . ,, .... , .. , . . . .., . ., . . . .. .. ,..;,%, ,. .. .. : .,..,
The Use of the General Rasch Modelwith Multidimensional Item Response Data
Latent trait theory has become an increasingly popular area for researchand application in recent years. Areas of application of latent traittheory have included tailored testing (McKinley and Reckase, 1980), equating(Marco, 1977; Rentz and Bashaw, 1977), test scoring (Woodcock, 1974), andcriterion-referenced measurement (Hambleton, Swaminathan, Cook, Eignor, andGifford, 1978). While many of these applications have been successful,they are limited to areas in which the tests used measure predominantly onetrait. This limitation is a result of the fact that most latent traitmodels that have been proposed assume unidimensionality. Because of thisrequirement, in some situations latent trait models have not been successfullyapplied. For example, in achievement testing the goal is not to measure asingle trait, but to sample the content covered by instruction. Therefore,most latent trait models are inappropriate since tests designed for thispurpose generally cannot be considered to be unidimensional. Even when thegoal is to measure a single trait, if dichotomously scored items are usedno generally accepted method exists for forming unidimensional item sets,for determining the dimensionality of existing item sets, or for testingthe fit of the unidimensional model to the data.
An alternative to trying either to construct unidimensional item setsor to fit a unidimensional model to already existing item sets is to developa multidimensional latent trait model. Several such models have beenproposed (Bock and Aitkin, 1981; Mulaik, 1972; Rasch, 1961; Samejima, 1974;Sympson, 1978; Whitely, 1980), but little research has been done usingthese models. Some work has been completed on the estimation of the parametersof the Bock and Aitkin model (Bock and Aitkin, 1981), the multidimensionalRasch model (Reckase, 1972), and the Whitely model (Whitely, 1980), but noextensive research has been completed on the characteristics and propertiesof any of these models. The purpose of this paper is to present the resultsof research on the characteristics and properties of the multidimensionalRasch model. Before presenting these results, however, the multidimensionalmodels that have been proposed will be briefly discussed, as will theresearch that has investigated the characteristics of these models.
Multidimensional Latent Trait Models
Three of the multidimensional latent trait models that have beenproposed have been extensions to the multidimensional case of the unidimen-sional Rasch model (Rasch, 1960). The unidimensional Rasch model is givenby P( jexp(x 11 (8j + a1 )) (1)
P(x 1 I8leJ' 1) 1+ exp(ej + ai )
where e. is the ability parameter for Person j, (. is the item easiness3 1
parameter for Item i, and P(xij t Ojloi ) is the probability of response xij
(0 or 1) to Item i by Person j. The multidimensional model proposed byRasch (1961) is given by
1 exp[ ) (x) - - + 6jX X + p(x)I (2)
P(ibjS) ~is)_
'ii)-2-
where j, a., and P(xij I eji) are as defined above; j, , x, and p arescoring functions which are functions of x only; and y(_.,o.) is a normal-
-J -Iizing factor necessary to make the probabilities of the response alternativessum to 1.0. The scoring functions t, t, and X act as weights for theparameters, while the p term is used to adjust the scale for differentscoring procedures. Both the scoring functions and the p term depend onthe score obtained by a person on the item. In order to apply this modelto multidimensional data, 8, C-, t, and t are defined as column vectors,
t(x)'8 and t(x)a i are inner products of vectors, and X(x) is defined as a
matrix. The terms t and t now represent vectors of weights for the differentelements in the a- and 6-vectors. The [ matrix is a matrix of weights.Rasch never attempted to apply this model.
Reckase (1972) tried to apply the generalized Rasch model to real andsimulated item response data with limited success. In this study, the
0 multidimensional model fit multidimensional data no better than did theunidimensional Rasch model. However, Reckase did not include the 8. X(x)a.
term in the model, which may have resulted in the poor fit of the model tomultidimensional data. In addition, several methodological problems mayhave contributed to the poor results of the study. First, the sample sizeused to estimate the parameters of the model was relatively small for thenumber of parameters estimated. Second, in addition to estimating theparameters of the model, Reckase also estimated the values of the t andscoring functions. Finally, in order to estimate the parameter vectors,the dichotomously scored items used in the study were combined into clustersto form nominal response patterns. The most appropriate way to form theclusters was not known, which may have caused problems in the estimation ofthe parameters. Despite these difficulties, a least squares estimationprocedure was developed which did yield somewhat reasonable parameterestimates.
Hulaik (1972) also proposed a multidimensional model that is ageneralization of the Rasch model. The model proposed by Mulaik is givenby
klexp[( jk + ark)Xij]P(xi 18J'-i k- (3)
1 + Cexp(e jk + 'ik)
where 8.k is the ability parameter for Person j on Dimension k, and aik is
the difficulty parameter for Item i on Dimension k. Although lulaik neverapplied this model, he did suggest procedures for estimating the parametersof the model for three separate cases: when item responses are normallydistributed and have a common variance for all items and subjects; whenitem reponses follow a Poisson distribution; and when item responses aredichotomous.
-3-
Samejima (1974) proposed a multidimensional latent trait model that isa generalization of a different unidimensional model. The model proposedby Samejima is based on the two-parameter normal ogive model. This modelis given by
P(xle a j bl [a' (6j - bi ) 1 (4)
where O(x) is the normal distribution function, ai is a column vector of
item discrimination parameters, bi is a column vector of item difficulty
parameters, and e. is a column vector of ability parameters for Person j.-JUnfortunately, the basic derivation of this model used the continuousresponse version of the normal ogive model. Therefore, its use withdichotomous data requires that item scores be translated to the continuousscale. Since no procedure for translating item scores to the continuousscale is available, the model cannot at present be applied to dichotomousdata. Like Rasch and Mulaik, Samejima never applied this model, but onlysuggested how the parameters might be estimated.
Sympson (1978) proposed a multidimensional model based on the three-parameter logistic model. The Sympson model postulates that the probabilityof a correct response is determined by the product of the conditionalprobabilities of a correct response on each of the dimensions being measured.The three-parameter logistic model is given by
P(xij~l [0~lb~ i =-c + (1 - c i ) (5)~)1 + exp[Da (e - b )]
where 0. is the ability parameter for Person j, a. is the item discriminationJ1 1
parameter for Item i, b. is the item difficulty parameter for Item i, c. isi. i
a pseudo-guessing parameter for Item i, and D = 1.7. The three-parameterlogistic model is used to model the conditional probability for each dimension,although the ci parameter does not have a separate value for each dimension,
but rather is a scalar parameter related to the item as a whole. Themultidimensional model is given by
PM) b exp[aik(9 k - bik)xijlP(x- x 0,1 (6)
k-l I + exp[aik(0jk - bik)]
where the parameters are as defined above and m is the number of dimensions.Although Sympson has done some work on estimating the parameters of thismodel, no application of the model to multidimensional data has yet beenattempted.
The model proposed by Whitely (1980) is somewhat similar to Sympson'smodel. This model, called the multicomponent latent trait model, definesthe probability of a correct response to an item as the product of theprobabilities of performing successfully on each cognitive component of theitem. The Whitely model is given by
~-4-
m exp[(ejk - b ik)x i01P1 x, 16_ , i b f 7T , - 0 ,1 (7 )
ii ;j Zi k=1
I + exp[6jk - blk]
where all the parameters are as previously defined. It can be seen thatthis model is essentially another extension of the Rasch model. The modelfocuses on the different cognitive skills required to perform on an itemrather than the global dimensions hypothesized by Sympson. Estimationprocedures have been developed for the model and some applications havebeen made to real data. However, because of the emphasis placed on identi-fying the different cognitive skills required by an item, the applicationof this model is limited to data collected under very restricted experimentalconditions.
Bock and Aitkin (1981) have proposed a multidimensional two-parameter
normal ogive model for use with dichotomously scored response data. Thismodel is given by
P (X -± I !J,yf,aji - 0[(y, + Q -e ) / C) (8)
where Yi is the difficulty parameter for Item i, gi is a column vector of
discrimination parameters for Item i, ej is a column vector of ability
parameters for Person J, *(x) is the normal distribution function, and (Iis given by M
CT e (i E a. ak 2 ) (9)
k-l
Bock and Aitkin described a method for estimating the parameters ofthis model, and presented the results of the application of the model tothe data for the Law School Admissions Test (LSAT) presented in Bock andLieberman (1970). The results of the application of the model to the LSATindicated that a two-dimensional solution fit the data better than a one-dimensional solution. Fit was assessed using a likelihood ratio chi-squaretest.
Summary
Six different latent trait models have been proposed for use with multi-dimensional item response data. Of these six models, two are of littleinterest here. The Samejima model is uot designed for use with dichotomouslyscored item response data, and the Whitely model is appropriate only forspecial experimental conditions. Of the remaining four models, only theBock and Aitkin model and a special case of the Rasch model have beenapplied, and no attempt has been made to extensively investigate thecharacteristics and properties of any of the models. The purpose of thisresearch is to extensively investigate the characteristics of one of thosemodels, the generalized Rasch model.
-5-
Method
Design
The design of this research was to start with the most simple formulationof the multidimensional Rasch model, investigate its ability to describemultidimensional item response data, and if necessary to investigate increasinglymore complex versions of the model until good model/data fit was obtained.At each level of complexity the properties of the model were investigated,and the reasonableness and usefulness of the model were explored. This wasdone by generating test data to fit the particular form of the model beinginvestigated, and analyzing that data in an attempt to assess how well thecharacteristics of the data matched the characteristics of real data withmultidimensional characteristics.
The most general formulation of the model investigated in this researchis the model described by Rasch (1961), given by Equation 2. The simplerformulations of the model used in this research were obtained by eliminatingdifferent terms from the model statement by setting the appropriate scoringfunctions equal to zero for all item scores.
For each model statement that was obtained, simulated test data weregenerated to fit the model. Using the known parameters and model statement,predictions were made as to the dimensionality of the generated data andthe characteristics of the hypothetical items. Analyses were then performedon the simulation data in order to test the predictions. If it were foundthat a model statement could not be used to generate realistic data, interms of either dimensionality or item characteristics, then the model wasrejected, and a different model statement was investigated. This involvedaltering the terms of the general model (Equation 2) that were included andthose that were zeroed out. In some cases, all of the terms in a particularrejected model statement were retained, and one or more additional termsfrom the general model were added.
Analyses
The first analysis performed on the simulation data generated usingthe models was a factor analysis. Factor analysis, in this case, is notbeing used as a means of validating the models, but as a procedure fordetermining whether the data generated from the models have characteristicssimilar to those of real test data. All of the factor analyses performedin this research were performed using the principal components method onphi coefficients. When the obtained and expected factor structures of thedata did not match, follow-up analyses were performed in an attempt todetermine why the obtained factor structure was different from what wasexpected.
° Follow-up analyses included plotting the true item parameters againstthe factor loadings and against traditional item statistics such as proportion-correct difficulty values and point biserial discrimination values. Theseanalyses were performed using both the unrotated factor loading matrix andthe factor loading matrix rotated to the varimax criterion. The purposesof these analyses were three-fold. One purpose was to determine whether
e ' i . l',. i -. *% " , .-- ,. - N-C , '.?. '. .-..- .-... ,. ',
-6-
the obtained factor structure was a result of the model statement, thevalues used for the model parameters, or both. The second purpose was tofacilitate interpretation of the model parameters, and the third purposewas to determine whether the model yielded items with reasonable characteris-tics. In many cases it was necessary to generate additional data, usingdifferent values for the parameters of the model, in order to answer specificquestions about a particular model statement.
Using the results of all of the analyses performed for a particularmodel, a decision was made as to whether the model adequately generateddata similar to real test data. If a model statement were rejected, anattempt was made to determine from the results of the analyses what changesin the model would yield a more acceptable model.
Results
Vector Model
The first model that was investigated was a simple vector parameter
model. The j 1"X(x)ai and p(x) terms were eliminated, yielding the modelgiven by
P(xj_ = exp(q(x)'8. + (x) ) (10)Z Y(ej~) - -1 - -
- -
where all the terms are as defined for Equation 1, and 0, a., j and t are
vectors. For this model the scoring functions all took on values of onefor a correct response and zero for an incorrect response. This model wasselected first because it appeared to be a straightforward extension of theunidimensional Rasch model (Rasch, 1960) to the multidimensional case. Theexpectation was that data generated according to this model would have adimensionality that would vary with the number of elements in the parametervectors. For instance, when data were generated using two elements in boththe item and person parameter vectors, it was expected that the data wouldyield a two-factor solution when factor analyzed. This was not the case,however. Regardless of the number of elements in the parameter vectors,this model yielded one predominant factor. This was true regardless of theactual values of the parameters or the values that were used in the scoringfunctions.
Table 1 shows the first two eigenvalues from a typical principalcomponent solution for the vector model. As can be seen, there is a dominantfirst factor, with one minor factor. Table I also shows the unrotatedfactor liading matrix obtained for these particular data, as well as theproportion-correct difficulty and the inner product of the item parametervector and scoring function for each item (sum of the item parameters). Ascan be seen, there is little variation in the loadings on the first factor,while the minor factor is related to item difficulty. Factor II generallyhas positive loadings for easy items and negative loadings for hard items.Once it was ascertained that the vector model would not yield multi-factordata, it was not difficult to determine why. Equation 8 can be written as
P(xi 1 , Y(.e ,R) exP(Mj + i) (11)
" -7-
TABLE 1
IPrincipal Component Factor Loadings Based onPhi Coefficients with the Sums of the Item Parameters
and Observed Proportion Correct forTwo-Dimensional Vector Rasch Model
Item Sum of Item Observed FactorI FactorIlItem Parameters Difficulty
1 .89 .65 .57 .28
2 -.89 .33 .52 -.24
3 .43 .58 .56 .04
4 2.02 .80 .47 .29
5 .59 .60 .59 .27
6 -.91 .33 .52 -.14
7 -1.44 .24 .51 -.27
8 .47 .58 .56 .13
9 -1.05 .30 .51 -.24
10 -1.76 .20 .48 -.19
11 .98 .64 .58 .07
* 12 2.58 .88 .43 .35
13 -1.31 .25 .50 -.34
14 1.22 .71 .53 .29
15 -.64 .39 .55 -.09
16 .05 .48 .56 .06
17 -2.33 .15 .46 -.36
18 -.54 .39 .54 -.05
19 -.60 .38 .53 -.26
20 2.26 .82 .43 .39
Eigenvalues 5.42 1.18
7-8-
where a. = t(x)'. and Pi = (x)'Y" Equation 11 is the unidimensional
Rascb model, with inner products of vectors as parameters. Therefore,regardless of the values of the model parameters, as long as the innerproduct remains the same, the probability of a response is the same.Therefore, the dimensionality of the vectors is unimportant, only theproduct is critical. The model is still a unidimensional model. Thefactor analysis results typified by the solution shown in Table 1 serve asan empirical demonstration that the vector model is a unidimensional model.It can also be empirically demonstrated that the inner products of thescoring function and parameter vectors serve as parameters for the model.Figure 1 shows a plot of proportion-correct difficulty by the inner productof the scoring function and item parameter vectors, which for this case isjust the sum of the item paramaters. As can be seen, there is an almostperfect relationship between the inner products and the proportion-correctscores. When data were generated using the unidimensional Rasch model,with the inner products from the two-dimensional model as parameters,exactly the same plot was obtained.
Figure 1
Relationship Between the Proportion Correct and the Sumof the Item Parameters for the Two-Dimensional
Vector Model
0
2
0
It
0 °1
o5
-c3
% 1
o.m
01
.~
. 17
( SUM OF ITEM PRAaMETERSii Note: The numbers in the plot are the item numbers.
~-9-
Product Term Model
It was clear from the results just reported that using parametervectors in an otherwise unidimensional model did not make it a multidimensional
11 model. Therefore, the vector model was rejected as a multidimensionalmodel. The next model that was investigated contained only the 8.'X(x)a.
- - -term. This was the next model investigated because it involved more thansimple inner products of scoring and parameter vectors, but was simplerthan using both inner products and the 8jX'(x)o. term.
J
When 8. and u. are vectors, x(x) must be a matrix. The product e.a."-J -1 -J -'
represents a matrix of products of all possible pairs of the elements inthe B.- and 0.-vectors. For two-dimensional 8.- and 0.-vectors,
J 1 2 1
= (12)"'2 1° 2_2j
The X(x) matrix is a scoring matrix having an element for each elementof the 8.o. matrix. If the X(x) matrix for the matrix in Equation 12 were
x(x) = (13)0 1for a particular response x, then the numerator of the model statement forthat response would be exp(01o I + 020 2). The nonzero elements of the X(x)matrix indicate which elements of the O.o. matrix are included in the
exponent. It is clear from this that by selectively using zeros in the
_X(x) matrix, various products of . and o. elements can be selected.
Varying the values of the nonzero elements in X(x) assigns different weightsto different combinations. Thus, the product term model, given by
P(xij Ij,£1 ) a 1 exp[e 4 (x)ci] (14)J~~ ~ ~ _J -iY(1 5i
is a very rich model in terms of the number of alternative formulations ofthe exponent of the model that are available. Unfortunately, when datawere simulated using some of these alternatives, it was discovered thatthis model had an inconvenient property. Regardless of which formulationof this model was used, and regariless of what values were taken on by theitem parameters, the item proportion-correct difficulties were all approx-imately equal to each other. A closer examination of the product termmodel indicates why this occurred. Using the item parameters shown inTable 2, data were generated using
X(X 1) = (15)0. -
-10-
for a correct response, and
0)4(40) = (16)-0 0
for an incorrect response. This yields a model given by
P (X l ) exp(8 1 l + OJ2'i 2 (17),. P~xij = ll ) 1+ exp(Ojl',, + 'j2 aU)
where the 9jk and Ojk terms are elements in the 0- and o-vectors. From
Equation 17 it can be seen that the item parameters are similar to thediscrimination parameter in the unidimensional two-parameter logistic (2PL)model presented by Birnbaum (1968) since they are multipliers of the personparameters. In fact, if written as
P(xij = IO6jia) - 1 ) exP[o 1 (0jl + 0) + a 12 (a 2 + 0)] (18)
the model is essentially a two-dimensional two-parameter logistic modelwith both of the difficulty parameters equal to zero for all items. Becausethe data used for Table 2 were generated using a bivariate N(0,1) withp = 0 distribution of ability, difficulty parameters of zero yielded apredicted proportion-correct difficulty of .5.
A principal components analysis of phi coefficients yielded evidencethat the use of two item parameters resulted in a two-dimensional model.The first three eigenvalues obtained for the data generated using the itemparameter values in Table 2 were 4.0, 2.4, and .9. The role of the itemparameters as discrimination parameters in this model is indicated bycomparing the item parameters shown in Table 2 with the rotated factorloading matrix, also shown in Table 2. The correlations between the itemparameters and the factor loadings indicated that there was a strong linear
relationship between the item parameters and factor loadings (r = .98 for01 with Factor II, r = .99 for 02 with Factor I), supporting the conclusionthat 01 and 02 are acting as discrimination parameters.
Vector and Product Term Model
The vector model that was investigated first was essentially a uni-dimensional model that contained a difficulty parameter (the inner product.(x)'oi) as the only item parameter. The product term model is a multidimensional
model that contains discrimination parameters as the only item parameters.In order to obtain a multidimensional model which contained a difficultyparameter, the vector and product term models were combined. A combinationof these two models is given by
,4
4j
4 -' , , -- - . .- , . .- - . . . . ,. . ':; ,) ..' ,/ / . . ''''" . "' -,,.. ' .- ' ' .. ,-. .. ' ,
[C TABLE 2
Item Parameters, Proportion Correct Item Difficulty and Factor Loadingsfrom a Varimax Rotated Principal Components Solution on
Phi Coefficients for the Product Term Model
Item a1 02 p Factor I Factor II
1 0.150 1.150 .48 .52 .11
2 1.280 0.200 .50 .08 .56
A 3 0.260 1.350 .51 .57 .09
4 1.000 0.300 .52 .14 .52
5 0.250 1.050 .49 .47 .09
6 1.040 0.100 .51 .03 .57
7 0.110 1.150 .47 .52 -.01
8 0.200 1.200 .49 .57 .09
9 1.400 0.300 .50 .12 .58
10 0.300 1.200 .48 .54 .07
11 1..50 0.150 .51 .08 .59
12 0.400 1.200 .50 .53 .14
13 1.150 0.250 .50 .01 .52
14 0.150 1.300 .49 .61 .04
15 1.000 0.250 .51 .11 .49
16 0.100 1.400 .46 .61 .04
17 1.350 0.150 .50 -.01 .59
18 1.250 0.100 .52 -.03 .59
19 0.200 1.500 .48 .62 .03
20 1.150 0.500 .51 .25 .46
;:?~~~~......... ....... -.......... ..-.............-............ ............ .. .. ..... .
-12-
P (x ij 2i jg ) " xp W, (x'i + j '(x)W a l" (19)
As can be seen in Equation 19, the (x)'8. term was eliminated when the twomodels were combined. -3
Table 3 shows the item parameters used to generate data to fit thevector and product term model. These data were generated using two-dimensionalparameters. The scoring functions were also two-dimensional and werevectors of ones for a correct response and vectors of zeros for an incorrectresponse for all elements. Table 3 also shows the rotated factor loadingsobtained for the first two factors from the principal components analysisof phi coefficients obtained for that data. The first three eigenvaluesfrom the solution are 5.26, 2.28, and 1.07. Initial analyses indicatedthat this model could be used to model multidimensional data, and that itemdifficulties were not constant (see Table 3). However, these analyses alsoindicated that it was not realistic to use the same item parameters in boththe parameter vectors and the product term. The problem is indicated bythe magnitude of the correlation of the item proportion-correct difficultieswith the item point biserials. Because of the double role played by theitem parameters, the proportion-correct scores and point biserials had acorrelation of r = .94. That is not a very realistic situation. Therefore,this model was also rejected as a reasonable method for describing multi-dimensional item response data.
Reduced Vector and Product Term Model
Since the analyses of the vector and product term model indicated that
the same item parameters should not appear in both the parameter vectorsand the product term, the item parameter vector and the scoring functionswere altered so that parameters appeared in one or the other, but not both.In order to facilitate this, two additional elements were inserted into theitem parameter vector. For a correct response the first two elements werezeroed out of the product term, while the last two were elements werezeroed out of the vector term. This procedure results in the first twoitem parameters acting as difficulty parameters and the last two parametersacting as discrimination parameters. Although four item parameters wereused, only two dimensions were modelled in this case. For an incorrectresponse all of the parameters were zeroed out. All nonzero elements inthe scoring functions were set equal to one. The resulting model is givenby
P~x = l(ex 1J2) ( 1 O , exp[a 1l + a12 + aDa 1 + c14ej 4 ] (20)
where the 8 and u terms are elements of the corresponding vectors.
The first three eigenvalues obtained from the principal componentsanalysis for this model are 5.39, 1.30, and .99. Table 4 shows the itemparameters that were used to generate the data, as well as the factor
........................................,
-13-
TABLE 3
Item Parameters, Proportion Correct Item Difficulty,and Rotated Factor Loadings forthe Vector and Product Term Model
Item a p Factor I Factor I
1 .230 .190 .56 .63 .12
2 .880 2.180 .77 .73 -.10
3 .900 -1.920 .35 .08 .71
4 -.900 -.110 .32 .39 -.22
5 -.640 .830 .52 .58 -.35
6 -.540 -1.040 .16 .10 .31
7 1.730 -1.350 .54 .34 .65
8 .940 .630 .69 .69 .21
9 .030 -.110 .46 .56 .11
10 -1.610 -.570 .13 .01 -.37
11 -1.170 1.260 .74 .75 .05
12 -.550 -1.070 .17 .04 .22
13 -.420 -.480 .31 .32 -.01
14 .220 -.070 .53 .60 .15
15 -.020 -1.670 .21 -.04 .55
16 2.420 .370 .78 .61 .40
17 1.230 .400 .69 .68 .28
18 .250 .410 .58 .65 .08
19 .140 .760 .64 .67 -.10
20 -1.770 .550 .30 .27 -.60
.-
-14-
TABLE 4
Item Parameters and Rotated Factor Loadingsfor the Reduced Vector and Product Model
Item 2 aa3 a4 Factor I Factor II
1 .206 -.503 .373 .997 .51 .01
2 -.164 .888 1.205 1.832 .60 .32
3 .448 .261 .766 .876 .34 .36
4 .814 -.008 1.321 1.714 .55 .34
5 .111 -.908 1.344 1.216 .41 .42
6 -.947 .044 1.758 1.694 .46 .51
) 7 -.490 .111 .687 .738 .40 .24
8 .553 -.502 .347 1.454 .61 .07
- 9 -.344 .639 1.307 .127 -.06 .64
-. 10 -.257 .303 .851 .824 .26 .39
U U -.069 -.542 .472 .404 .22 .25
12 .779 .432 .392 .656 .30 .13
13 -.611 .571 .578 1.252 .59 .13
14 -.140 -1.032 .334 1.066 .60 -. 04
15 -.705 .081 .821 .480 .07 .44
16 -.386 -.164 1.912 .244 .03 .71
17 -.154 .044 1.193 .537 .16 .56
18 .474 .249 1.385 1.287 .49 .44
19 .438 -.210 1.320 1.110 .42 .45
20 .294 .190 1.634 1.492 .45 .49
-15-
,3 loadings obtained from a varimax rotation of the first two principal components.The item parameters that were used as multipliers (oY and 04) were allpositive in order to avoid having items with negative discriminations.
The results of the factor analysis of these data indicate that adominant first factor is present. However, there was a second componentpresent in the data which was strongly related to the item parameters(r = .87 for 03 and Factor II, r = .87 for 04 and Factor I). The itemparameters in the product term were related to the factor loadings, whilethe sum of the item parameters in the vector term were found to be relatedto the proportion correct difficulty. The correlation between the sum of
the parameters in the vector term and the proportion correct difficulty wasr = .98, indicating that the sum of the vector parameters act as difficultyparameters. There was not a significant correlation between the itemdifficulty and point biserial values (r = .12). The sum of 03 and 04 had acorrelation of r = .96 with the item point biserials.
The analyses of the model set out in Equation 20 indicate that it has* many desired characteristics. The rotated factor loadings are highly
related to the item parameters in the product term, the item difficulty ishighly correlated with the sum of the item parameter vector elements, andthere is no correlation between item difficulty and item discrimination.
One problem that does exist with the data that were generated is thatthe factor analysis results indicated that the data had only one predominantfactor. One possible reason for this is that so many of the items hadlarge values for both of the item parameters in the product term. In orderto test this, data were generated for the set of item parameters shown inTable 5. The eigenvalues from the principal components analysis for thesedata are 2.49, 2.28, 1.05, and 1.03. As can be seen, when using the itemparameters from Table 5 to generate data, there are two factors of approxi-mately equal magnitude present in the data.
Item Cluster Model
Although the reduced vector and product term model appears to adequatelymodel multidimensional data, the presence of the product term complicatesparameter estimation, since the separation of the item and person parametersis not possible through techniques of conditional estimation. Because ofthis, one more model that does not have a product term was investigated.This model is the item cluster model.
One of the reasons the item vector model, given by Equation 10, doesnot adequately model multidimensional data is that no information about thedifferent dimensions is preserved in the item score when the item is dichoto-mously scored. The elements for the different dimensions are sumed, andthe sums are treated as parameters. If it were possible to score thedimensions separately, then the vector model might be able to model multi-dimensional data. This requires, however, polychotomous item scoring.
.. ,.4. --:. .
5-~~~~ ~ ~ ~ ~ -CST r- 27 ..-'g 7-"- 70p. - -- -- "
-16-
TABLE 5
Item Parameters for the ReducedVector and Product Model
a i
item 0 2 03 4
1 .977 .258 1.000 .000
2 .359 .728 .000 1.000
3 -.322 .377 1.000 .000
* 4 -1.289 1.128 .000 1.000
5 -.613 .219 1.000 .000
6 1.299 .797 .000 1.000
7 .029 -.213 1.000 .000
8 -.360 -.862 .000 1.000
9 .769 -.487 1.000 .000
10 -1.447 2.092 .000 1.000
11 -1.252 -.243 1.000 .000
12 -.778 -1.426 .000 1.000
13 .668 -1.860 1.000 .000
14 2.102 -.025 .000 1.000
15 -.724 .968 1.000 .000
16 1.230 -.535 .000 1.000
17 .260 -1.216 1.000 .000
18 -1.092 -.432 .000 1.000
19 -.994 1.479 1.000 .000
20 -.206 -.525 .000 1.000
4 " " " ' 5*.. " ." ." - -, ' v "
' " " . . . . "/ -" • ' " " " - . ' " - ,- .- ," " -' i. - . .-- -
-17-
.4
Scoring an item on each dimension would require 2n response categories,where n is the number of dimensions. Unfortunately, most test data are notscored polychotomously.
An alternative to having polychotomous item scoring is to considermore than one item at a time. If two dichotomously scored items are clusteredtogether, and the cluster is treated as a single unit, then the cluster has22 or 4 response categories - (0,0), (0,1), (1,0), and (1,1). The modelgiven by Equation 10 can then be applied, with the exception that theoG-vector now represents a cluster rather than a single item, the scoringfunctions now take on values for 4 response categories instead of 2 and the
* response x is a vector with two elements. Further, the number of elementsin the o vector need not be the same as the number of items in the cluster,but rather should reflect the dimensionality of the cluster.
The procedure by which this model was investigated is as follows. For thetwo-dimensional case, item parameters were selected for 20 items. The
' items were paired so that Items 1 and 2 formed Cluster 1, Items 3 and 4
nformed Cluster 2, and so on until 10 clusters were formed. For each clusterthere were four response categories, which were scored as follows:
a) t(x) = t(x) = 181 for k equal to both items incorrect;
b) t(x) = t(x) = [?] for x equal to the first item incorrect, thesecond item correct;
c) (x) = !(x) = [ ] for x equal to the first item correct, the seconditem incorrect;
and d) t(x) = t(x) = [11 for x equal to both items correct.
For any one cluster two responses were generated, one for each dimension,using the parameters shown in Table 6. Table 6 also contains the unrotatedfactor loadings for the first two factors from a principal componentsanalysis of phi coefficients obtained for these data. The first foureigenvalues were 3.61, 3.06, 1.33, and 1.21.
As can be seen, for the factor analysis the simulation data weretreated as 20 items, rather than as 10 clusters. The eigenvalues listedabove indicate that there were two roughly equal components in the data.Table 6 shows that the first component was defined by the items that wereplaced first in the cluster, and the second component was defined by theitems that were in the second position in the cluster. Consistent with thescoring functions, there were two equal independent factors.
s In order to demonstrate that the factors need not be independent, thesame item parameters were used to generate data using the following scoringfunctions:
-18-
TABLE 6
Unrotated Factor Loadings on First Two Principal Componentsfor the Two-Dimensional Item Cluster Model
Independent Model Dependent Model
IteItem Factor I Factor II Factor I Factor IIParameters
1 .893 .56 .00 .40 .21
2 -.850 .02 .65 .33 -.36
3 -.892 .66 -.02 .38 .24
4 .690 .01 .66 .33 -.33
5 .430 .64 .00 .41 .20
6 3.200 .02 .19 .25 -.24
7 2.016 .36 -.04 .36 .16
. 8 -3.310 .07 .22 .10 -.35
9 .594 .61 -.05 .40 .33
10 .470 .00 .69 .26 -.39
11 -.913 .66 .01 .47 .30
12 1.220 .06 .55 .32 -.38
13 -1.437 .58 -.01 .37 .20
14 -1.260 .00 .58 .32 -.43
15 .467 .65 -.07 .45 .28
,, 16 .880 .04 .62 .28 -.36
17 -1.048 .66 -.04 .44 .28
18 -.970 -.02 .64 .28 -.37
19 -1.760 .56 .07 .43 .10
20 -2.140 .01 .42 .20 -.31
.J. .., JL. P. .7. P . r .. . ... , .. ... -.6. ,.-7.; -. ,--. '.r-, -- ,.-. .. , .......
-19-
a) t(x) = (x) = 181 for x equal to both items incorrect;
b) t(x) = j(x) = [ 1] for x equal to the first item incorrect,the second item'2orrect;
c) V(x) = t(x) = [>] for x equal to the first item correct,the second item'incorrect;
and d) ( (x) = [1] for x equal to both items correct.
The principal components analysis of phi coefficients for the datagenerated according to this model yielded six factors with eigenvaluesgreater than one [2.46, 1.83, 1.09, 1.08, 1.01, 1.00]. Table 6 shows theunrotated factor loadings. As can be seen, there are still two factorspresent in the data. However, the factors are no longer defined only bythe items in the corresponding position in the cluster. The first componentis a general factor, while the second component indicates the position ofthe item in the cluster. Clearly these two sets of items are not independentin this case.
Discussion
The use of simulation data to study the characteristics of a modelbefore selecting it for application is perhaps atypical of research onlatent trait models. Usually a model is adopted, estimation procedures arederived, and the model is applied without ever going through the processthis study has employed. In this study this approach has been taken fortwo main reasons. First, it was felt that when dealing with multidimensionallatent trait models much of the acquired wisdom concerning latent traitmodels might no longer apply. It was felt that considerable research wasnecessary in order to gain an understanding of how these models work andwhat the model parameters represent before they could be applied. Thisbelief has been borne out several times in this study by findings indicatingthat the models were not behaving in the anticipated manner.
A second reason for taking this approach was that it seemed impracticalto attempt to develop estimation procedures for some of these models.Specifically, the general model set out by Rasch has a very large number ofparameters. It seemed impractical to try to estimate all of them, and itwas hoped that research on the model could help simplify the estimationprocess by eliminating some terms from the model and by discovering restrictionson the range of values for the parameters. With these considerations inmind, the results of this study will now be discussed.
Vector Model
The simplest formulation of the general model that was investigatedwas the vector model. This model is simply the unidimensional Rasch model,but with vectors for parameters instead of scalars. This model was foundto be totally inadequate for modelling multidimensional data. When datawere generated according to this model, the resulting data were unidimen-sional, with item characteristics determined by the inner product of theitem parameter vectors and scoring functions. From this it follows thatthis model would fit multidimensional data no better than a unidimensionalmodel having parameters equal to the inner products from the vector model.
. .- t . .,
-20-
Product Term Model
Because of its slight similarity to Birnbaum's two-parameter logisticmodel, it was felt that the product term model would be better able tomodel multidimensional data. It was anticipated that the item parametersin the product term would behave as discrimination indices, and that is howthey did behave. Unfortunately, without the vector terms in the modelthere were no terms playing the role of difficulty parameters. The datagenerated for this model had items of constant difficulty. From this itwas concluded that this model would be useful only for modelling items ofconstant difficulty, and when items have varying difficulties this model isinappropriate.
Vector and Product Term Model
Based on the findings for the vector model and the product term model,it was hypothesized that a combination of the two models would be necessary
to model items that were both multidimensional and of varying difficulty.Analyses of the vector and product term model indicated that it would modelmultidimensional data, and that it would model items of varying difficulty.However, it was also found that, as long as the item parameter vectorelements appeared both in the vector terms and in the product term, theitem difficulties and disciminations would be highly correlated. Sincethis is rarely the case in real test data, it was concluded that this modelwould be useful only in a very limited number of circumstances.
*t Reduced Vector and Product Term Model
In order to overcome the deficiencies of the vector and product termmodel, it was clear that a given item parameter vector element shouldappear only in the vector term or the product term, but not both. It wasanticipated that similar problems might exist if the person parametervector elements occurred in both the vector term and the product term, sothe same restriction was placed on the person parameters as was placed onthe item parameters.
The resulting model appears to be quite successful at modelling realisticmultidimensional data. It is capable of modelling correlated as well asindependent dimensions, and the item parameters are readily interpretable.The only real problem there seems to be with this model is with the estimationof the parameters. Although there are fewer parameters to estimate than isthe case with the general model, there are still a fair number to estimate.Moreover, there are no observable sufficient statistics for the parametersin the product term. These problems do not make estimation of the modelparameters impossible, and probably not even impractical. However, they domake estimation more difficult.
Item Cluster Model
The item cluster model was proposed as an alternative to the vectormodel. This model does not involve a product term, but it still cansuccessfully model multidimensional data. However, it % oes involve clustering
,..
-
: : . . , . ', N , '-.''o ''"',. -,'." ". ""-. " .-.- " .. "' ..
-21-
items, which gives rise to a number of new problems. For instance, as yetit is unclear what the effect is of forming different combinations ofitems, or whether all items should be clustered with the same item.Preliminary investigations seem to indicate that the optimal clusteringprocedure is to cluster all items on a subtest with one item taken from adifferent subtest. Another alternative, which has not been explored, is to
%! apply the model only in situations where items are already clustered, suchas is the case with passage units. Clearly more research is needed on this
type of application of the item cluster model.
Summary and Conclusions
The purpose of this study was to investigate the application of thegeneral Rasch model to multidimensional data. Several formulations of themodel, varying in complexity, were investigated to determine whether theycould successfully model realistic multidimensional data. Also investigatedwas whether the parameters of the models could be readily interpreted. Themodels investigated included: a) the vector model; b) the product termmodel; c) the vector and product term model; d) the reduced vector andproduct term model; and, e) the item cluster model.
Of the models investigated, all but the reduced vector and productterm model and the item cluster model were rejected as being incapable ofmodelling realistic multidimensional data. The item cluster model appearsto be a useful model, but its applications may be limited in scope. Of themodels studied, the reduced vector and product term model was found to bethe most capable of modelling realistic multidimensional data. Althoughthe estimation of the parameters of the reduced vector and product term
* model may be more difficult than it would be for other models, this modelappears to be the model that is most worth pursuing.
.4
A..
4.
-22-
REFERENCES
,irnbaum, A. Some latent trait models and their use in inferring anexaminee's ability. In F.M. Lord and M.R. Novick, Statisticaltheories of mental test scores. Reading, MA: Addison-Wesley, 1968.
* . Bock, R.D., and Aitkin, M. Marginal maximum likelihood estimation of item
paramaters: An application of an EM algorithm. Psychometrika,1981, 46, 443-459.
Bock, R.D., and Lieberman, M. Fitting a response model for n dichotomoslyscored items. Psychometrika, 1970, 35, 179-197.
Hambleton, R.K., Swaminathan, H., Cook, L.L., Eignor, D.R., andGifford, J.A. Developments in latent trait theory, models, technicalissues, and applications. Review of Educational Research, 1978,48, 467-510.
Marco, G.L. Item characteristic curve solutions to three intractabletesting problems. Journal of Educational Measurement, 1977, 14,139-160.
McKinley, R.L., and Reckase, M.D. A successful application of latenttrait theory to tailored achievement testing (Research Report80-1). Columbia: University of Missouri, Department of EducationalPsychology, February 1980.
Mulaik, S.A. A mathematical investigation of some multidimensional Raschmodels for psychological tests. Paper presented at the annual meetingof the Psychometric Society, Princeton, NJ, March, 1972.
Rasch, G. On general laws and the meaning of measurement in psychology.Proceedings of the Fourth Berkeley Symposium on MathematicalStatistics and Probability, Berkeley: University of California Press,1961, 4, 321-334.
Rasch, G. Probabilistic models for some intelligence and attainment tests.Copenhagen: Danish Institute for Educational Research, 1960.
Reckase, M.D. Development and application of a multivariate logisticlatent trait model (Doctoral dissertation, Syracuse University,1972). Dissertation Abstracts International, 1973, 33. (UniversityMicrofilms No. 73-7762)
Rentz, R.R., and Bashaw, W.L. The National Reference Scale for Reading:An application of the Rasch model. Journal of Educational Measurement,1977, 14(2), 161-180.
Samejima, F. Normal ogive model for the continuous response level inthe multidimensional latent space. Psychometrika, 1974, 39,111-121.
'. a-t. .....- . ... -* .-. .. .- . .
-23-
Sympson, J.B. A model for testing with multidimensional items. InD.J. Weiss (Ed.), Proceedings of the 1977 Computerized AdaptiveTesting Conference. Minneapolis: University of Minnesota, 1978.
Whitely, S.E. Measuring aptitude processes with multicomponent latenttrait models (Technical Report No. NIE-80-5). Lawrence: Universityof Kansas, Department of Psychology, July 1980.
Woodcock, R.W. Woodcock Reading Mastery Tests. Circle Pines, Minnesota:American Guidance Service, 1974.
.,. ... . . .
. .... +. ... -. . -. -. -
Navy Navy
Dr. Ed Aiken 1 Dr. Jim HollanNavy Personnel R&D Center Code 304San Diego, CA 92152 Navy Personnel R & D Center
San Diego, CA' 921521 Meryl S. Baker
NPRDC 1 Dr. Norman J. KerrCode P309 Chief of Naval Technical TrainingSan Diego, CA 92152 Naval Air Station Memphis (75)
. Millington, TN 380541.Dr. Jack R. Borsting
' Provost & Academic Dean 1 Dr. William L. MaloyU.S. Naval Postgraduate School Principal Civilian Advisor forMonterey, CA 93940 Education and Training
Naval Training Command, Code OOADr. Robert Breaux Pensacola, FL 32508Code N-711NAVTRAEQUIPCEN 1 CAPT Richard L. Martin, USNOrlando, FL 32813 Prospective Commanding Officer
USS Carl Vinson (CVN-70)Chief of Naval Education and Training Newport News Shipbuilding and Drydock Co
Liason Office Newport News, VA 23607Air Force Human Resource LaboratoryFlying Training Division 1 Dr. James McBrideWILLIAMS AFB, AZ 85224 Navy Personnel R&D Center
San Diego, CA 92152CDR Mike CurranOffice of Naval Research 1 Dr William Montague800 N. Quincy St. Navy Personnel R&D CenterCode 270 San Diego, CA 92152Arlington, VA 22217
1 Mr. William NordbrockDR. PAT FEDERICO Instructional Program DevelopmentNAVY PERSONNEL R&D CENTER Bldg. 90SAN DIEGO, CA 92152 NET-PDCD
Great Lakes Naval Training Center,Mr. Paul Foley IL 60088Navy Personnel R&D CenterSan Diego, CA 92152 1 Ted M. I. Yellen
Technical Information Office, Code 2011 Dr. John Ford NAVY PERSONNEL R&D CENTER
Navy Personnel R&D Center SAN DIEGO, CA 92152San Diego, CA 92152
1 Library, Code P201LDr. Patrick R. Harrison Navy Personnel R&D CenterPsychology Course Director San Diego, CA 92152LEADERSHIP & LAW DEPT. (7b)DIV. OF PROFESSIONAL DEVELOPHMENT 1 Technical DirectorU.S. NAVAL ACADEMY Navy Personnel R&D CenterANNAPOLIS, MD 21402 San Diego, CA 92152
','
Navy Navy
6 Com. anding Officer 1 Dr. Worth Scanland, DirectorNaval Research Laboretory Research. Development, Test & FvaluationCode 2627 N-5Washington, DC 20390 Nava] Education and Training Command
NAS, Pensacola, FL 32508. Psychologist
ONR Branch Office 1 Dr. Robert G. SmithBldg 114, Section D Office of Chief of Naval Operations666 Summer Street OP-987HBoston, 14A 02210 Washington, DC 20350
1 Office of Naval Research 1 Dr. Alfred F. SmodeCode 437 Training Analysis & Evaluation Group800 N. Quincy SStreet (TAEG)Arlington, VA 22217 Dept. of the Navy
Orlando, FL 328131-Office of Naval Research
Code 441 1 Dr. Richard Sorensen800 N. Quincy Street Navy Personnel R&D CenterArlington, VA 22217 San Diego, CA 92152
5-Personnel & Training Research Programs 1 W. Gary Thomson(Code 458) Naval Ccean Systems Center
Office of Naval Research Code 7132Arlington, VA 22217 San Diego, CA 92152
Psychologist 1 Roger Weissinger-BaylonONR Branch Office Department of Administrative Sciences1030 East Green Street Naval Postgraduate SchoolPasadena, CA 91101 Monterey, CA 93940
1 Office of the Chief of Naval Operations 1 Dr. Ronald WeitzmanResearch Development & Studies Branch Code 54 WZ
(OP-115) Department of Administrative SciencesWashington, DC 20350 U. S. Naval Postgraduate School
Monterey, CA 93940LT Frank C. Petho, MSC, USN (Ph.D)Selection and Training Research Division 1 Dr. Robert WisherHuman Performance Sciences Dept. Code 309Naval Aerospace Medical Research Laborat Navy Personnel R&D CenterPensacola, FL 22508 San Diego, CA 92152
Dr. Bernard Rimland (03B) 1 Mr John H. WolfeNavy Personnel R&D Center Code P310San Diego, CA 92152 U. S. Navy Personnel Research and
Development CenterSan Diego, CA 92152
Army Army
1 Technical Director 1 Dr. Robert SasmorU. S. Army Research Institute for the U. S. Army Research Institute for the
Behavioral and Social Sciences Behavioral and Social Sciences5001 Eisenhower Avenue 5001 Eisenhower Avenue
. Alexandria, VA 22333 Alexandria, VA 22333
'.1,Mr. James Baker 1 CommandantSystems Manning Technical Area US Army Institute of AdministrationArmy Research Institute Attn: Dr. Sherrill
5001 Eisenhower Ave. FT Benjamin Harrison, IN 46256
Alexandria, VA 22333 1. Dr. Joseph Ward
1 Dr. Beatrice J. Farr U.S. Army Research Institute
U. S. Army Research Institute 5001 Eisenhower Avenue
5001 Eisenhower Avenue Alexandria, VA 22333Alexandria, VA 22333
1 Dr. Myron Fischl Air ForceU.S. Army Research Institute for the
Social and Behavioral Sciences.-. 5001 Eisenhower Avenue- w1 Air Force Human Resources Lab
Alexandria, VA 22333,., AFHRL/tPPD
. 1 Dr. Michael Kaplan Brooks AFB, TX 78235
U.S. ARMY RESEARCH INSTITUTE
5001 EISENHOWER AVENUE 1 U.S. Air Force Office of Scientific
ALEXANDRIA, VA 22333 ResearchLife Sciences Directorote, NL
" 1 Dr. Milton S. Katz Bolling Air Force Base
* Training Technical Area Washington, DC 20332
, U.S. Army Research Institute 1 Dr. Earl A. Alluisi5001 Eisenhower Avenue HQ, AFHRL (AFSC)Alexandria, VA 22333 Brooks AFB. TX 78235
1 Dr. Harold F. O'Neil, Jr. 1 Dr. Alfred R. FreglyAttn: PERI-OK AFOSR/NL, Bldg. 410]Army Research Institute Rolling AFB5001 Eisenhower Avenue Washng AFBAlexandria, VA 22333 Washington, DC 20332
1 DR. JAMES L. RANEY 1 Dr. Genevieve Haddad
U.S. ARMY RESEARCH INSTITUTE Program Manager
5001 EISENHOWER AVENUE Life Sciences Directorate
ALEXANDRIA, VA 22333 AFOSRBolling AFB, DC 2033:2
ir. Robert Ross 1 David R. Hunter" U.S. Army Research Institute for the AFHRL/11OAM
Social Pnd Behavioral Sciences Brooks AFB, TX 78235;' 5001 Eisenhower Avenue
Alexandria, VA 22333
*t*.. . .. .. .......
Air Force CoastGuard
1 Research and Measurment Division Chief, Psychological Reserch BranchResearch Branch, AFMPC/MPCYPR U. S. Coast Guard (G-P-1/2/TP42)Randolph AFB, TX 78148 Washington, DC 20593
1 Dr. Malcolm Ree Mr. Thomas A. Warm
AFHRL/MP U. S. Coast Guard InstituteBrooks AFB, TX 78235 P. 0. Substation 18BOklahoma City, OK 73169
2 3700 TCHTW/TTGH Stop 32Sheppard AFB, TX 76311
Other DoD1 Dr. Joe Ward, Jr.
AFHRL/MPMDBrooks AFB, TX 78235 12 Defense Technical Information Center
Cameron Station, Bldg 5*' Alexandria, VA 22?14
Attn: TCMarines
1 Dr. William GrahamTesting Directorate
1 H. William Greenup MEPCOM/MEPCT-PEducation Advisor (E031) Ft. Sheridan, IL 60037Education Center, MCDECQuantico, VA 22134 1 Director, Research and Data
OASD(MRA&L)1 Director, Office of Manpower Utilization 3B919, The Pentagon
HO, Marine Corps (MPU) Washington, DC 20301BCB, Bldg. 2009
SQuantico, VA 22134 1 Military Assistant for Training and
1 Headquarters, U. S. Marine Corps Personnel TechnologyOffice of the Under Secretary of Defense
Code MPI.20 for Research & EngineeringWashington, DC 20380 Room 3D129, The Pentagon
Washington, DC 203011 Special Assistant for Marine
Corps Matters 1 Dr. Wayne SellmanCode lOOM Office of the Assistant SecretaryOffice of Naval Research of Defense (MRA & L)800 N. Quincy St. 2B269 The PentagonArlington, Vh 22217 Washington, DC 20301
1 Major Michael L. Patrow, USMC 1 DARPASHeadquarters, Marine Corps 1400 Wilson Blvd.
(Code MPI-20) Arlington, VA 22209Washington, DC 20380
1 DR. A.L. SLAFKOSKY
* SCIENTIFIC ADVISOR (CODE RD-1)HQ, U.S. MARINE CORPSWASHINGTON, DC 20380
1
" -~~~~~~~~~~~~~~~~~~. . ...... .... ... .,•, .- .- .- .- .- ...............-.... ,
.A~~~ ~.. .. .. . .. . . . .. . . .
Civil Govt Non Govt
1 Dr. Paul G. Chapin 1 Dr. James AlginaLinguistics Program University of FloridaNational Science Foundation Gainesville, FL 32611Washington, DC 20550
S1 Dr. Erling P. Andersen1 Dr. Susan Chipman Department of Statistics
Learning and Development Studiestraede 6National Institute of Education 1455 Copenhagen1200 19th Street NW DENMARKWashington, DC 20208
1 Anderson, Thomas H., Ph.D.I Dr. John Mays Center for the Study of Reading
National Institute of Education 174 Children's Research Center1200 19th Street NW 51 Gerty DriveWashington, DC 20208 Champiagn, IL 61820
1 Dr. Arthur Melmed 1 Dr. John AnnettNational tntitute of Education Department of Psychology1200 19th Street NW University of WarwickWashington, DC 20208 Coventry CV4 7AL
ENGLANDDr. Andrew R. MolnarScience Education Dev. 1 1 psychological research unit
and Research Dept. of Defense (Army Office)National Science Foundation Campbell Park OfficesWashington, DC 20550 Canberra ACT 2600, Australia
Dr. H. Wallace Sinaiko 1 Dr. Alan BaddeleyProgram Director Medical Research CouncilManpower Research and Advisory Services Applied Psychology UnitSmithsonian Institution 15 Chaucer Road801 North Pitt Street Cambridge CB2 2EFAlexandria, VA 22314 ENGLAND
" Dr. Vern W. Urry 1 Ms. Carole A. BagleyPersonnel R&D Center Minnesota Educational ComputingOffice of Personnel Management Consortium1900 E Street NW 2354 Hidden Valley Lane
* Washington, DC 20415 Stillwater, MN 55082
1 Dr. Frank Withrow 1 Dr. Isaac BejarU. S. Office of Education Educational Testing Service400 Maryland Ave. SW Princeton, NJ 08450Washington, DC 20202
1 Capt. J. Jean BelangerDr. Joseph L. Young, Director Training Development DivisionMemory & Cognitive Processes Canadian Forces Training SystemNational Science Foundation CFTSHQ, CFB TrentonWashington, DC 20550 Astra, Ontario KOK 1B
Mon Govt Non Govt
1 Dr. Fenucha Birenbaum 1 Dr. Norman Cliff
School of Education Dept. of PsychologyTel Aviv University Univ. of So. CaliforniaTel Aviv, Ramat Aviv 69978 University ParkIsrael Los Angeles, CA 90107
Dr. Werner Pirke 1 Dr. William E. CoffmanDezWPs im Streitkraefteamt Director, !owa Testing ProgramsPostfach 20 50 03 334 Lindquist CenterD-5300 Bonn 2 University of IowaWEST GERMANY Iowa City, IA 52242
1 Dr. R. Darrel Bock 1 Dr. Meredith P. Crawford
Department of Education American Psychological AssociationUniversity of Chicago 1200 17th Street, N.W.Chicago, IL 60637 Washington, DC 20036
1 Liaison Scientists 1 Dr. Hans Crombag
Office of Naval Research, Education Research CenterBranch Office , London University of LeydenBox 39 FPO New York 09510 Boerhaavelaan 2
2334 EN LeydenI Dr. Robert Brennan The NETHERLANDS
American College Testing ProgramsP. 0. Box 168 1 Dr.,Fritz DrasgowIowa City, IA 52240 Yale School of Organization and Manageme
". Yale University1 DR. JOHN F. BROCK Box 1A
Honeywell Systems & Research Center New Haven, CT 06520(MN 17-2318)
2600 Ridgeway Parkway 1 Mike DurmeyerMinneapolis, MN 55413 Instructional Program Development
Building 90"1 DR. C. VICTOR BUNDERSON NET-PDCD
WICAT INC. Great Lakes NTC, IL 60098UNIVERSITY PLAZA, SUITE 101160 SO. STATE ST. 1 ERIC Facility-Acquisitions
OREM, UT 84057 4833 Rugby AvenueBethesda, MD 20014
1 Dr. John B. Carroll
Psychometric Lab 1 Dr. Benjamin A. Fairbank, Jr.Univ. of No. Carolina McFann-Gray & Associates, Tnc.Davie Hall 013A 5825 CallaghanChapel Hill, NC 27514 Suite 225
San Antonio, Texas 7822F
.4
.
• ...
Non Govt Non Govt
Dr. Leonard Feldt 1 Dr. Delwyn HarnischLindquist Center for Measurment University of IllinoisUniversity of Iowa 242b EducationIowa City, IA 52242 Urbana, IL 61801
Dr. Richard L. Ferguson 1 Dr. Chester HarrisThe American College Testing Program School of EducationP.O. Box 168 University of CalifornipIowa City. IA 52240 Santa Barbara, CA 93106
Dr. Victor Fields 1 Dr. Dustin H. HeustonDept. of Psychology Wicat, Inc.Montgomery College Box 986Rockville, MD 20850 Orem, UT 84057
Univ. Prof. Dr. Gerhard Fischer 1 Dr. Lloyd HumphreysLiebiggasse 5/3 Department of PsychologyA 1010 Vienna University of IllinoisAUSTRIA Champaign, IL 61820
Professor Donald Fitzgerald 1 Dr. Steven HunkaUniversity of New England Department of EducationArmidale, New South Wales 2351 University of AlbertaAUSTRALIA Edmonton, Alberta
CANADADR. ROBERT GLASERLRDC 1 Dr. Earl HuntUNIVERSITY OF PITTSBURGH Dept. of Psychology3939 O'HARA STREET University of WashingtonPITTSBURGH, PA 15213 Seattle, WA 98105
Dr. Daniel Gopher 1 Dr. Jack HunterIndustrial & Management Engineering 2122 Coolidge St.Technion-Israel Institute of Technology Lansing, MI 48906HaifaISRAEL 1 Dr. Huynh Huynh
College of EducationDr. Bert Green University of South CarolinaJohns Hopkins University Columbia, SC 29208Department of PsychologyCharles & 34th Street 1 Professor John A. KeatsBaltimore, MD 21218 University of Newcastle'-: AUSTRALIA 2308
Dr. Ron Hambleton
School of Education 1 Mr. Jeff KeletyUniversity of Massechusetts Department of Instructional TechnologyAmherst, MA 01002 University of Southern California
Los Angeles, CA 92007
Non Govt Non Govt
1 Dr. Stephen Kosslyn 1 Dr. Scott Maxwell
Harvard University Department of PsychologyDepartment of Psychology University of Houston33 Kirkland Street Houston. TX 77004Cambridge, MA 021?6 1. Dr. Samuel T. Mayo
1 Dr. Marcy Lansman Loyola University of Chicago
Department of Psychology. I 25 820 North Michigan Avenue
University of Washington Chicago, IL 60611Seattle, WA 98195
1 Professor Jason Millman1 Dr. Alan Lesgold Department of Education
Learning R&D Center Stone HallUniversity of Pittsburgh Cornell UniversityPittsburgh, PA 15260 Ithaca, NY 14853
1 Dr. Michael Levine 1 Dr. Allen Munro
Department of Educational Psychology Behavioral Technology Laboratories
210 Education Bldg. 1845 Elena Ave., Fourth Floor
University of Illinois Redondo Beach, CA 90277
Champaign, IL 61801 1 Dr. Melvin R. Novick
1 Dr. Charles Lewis 356 Lindquist Center for Measurment
Faculteit Sociale Wetenschappen University of IowaRijksuniversiteit Groningen Iowa City, IA 52242
Oude Boteringestraat 239712GC Groningen 1 Dr. Jesse Orlansky
Netherlands Institute for Defense Analyses400 Army Navy Drive
1 Dr. Robert Linn Arlington, VA 22202College of EducationUniversity of Illinois 1 Wayne M. Patience
Urbana, IL 61801 American Council on EducationGED Testing Service, Suite 20
1 Dr. Frederick M. Lord One Dupont Cirle, NW
Educational Testing Service Washington, DC 20036*Princeton, NJ 08540
1 Dr. James A. Paulson
1 Dr. James Lumsden Portland State University
Department of Psychology P.O. Box 751University of Western Australia Portland, OR 97207Nedlands W.A. 6009AUSTRALIA 1 MR. LUIGI PETRULLO
2431 N. EDGEWOOD STREET
1 Dr. Gary Marco ARLINGTON, VA 22207
Educational Testing ServicePrinceton, NJ 08450
t4
,%
Non Govt Non Govt
Dr. Steven E. Poltrock 1 Dr. J. RyanDepartment of Psychology Department of EducationUniversity of Denver University of South CarolinaDenver,CO 80208 Columbia, SC 2920q
DR. DIANE M. RAMSEY-KLEE I PROF. FUMIKO SAMEJIMAR-K RESEARCH & SYSTEM DESIGN DEPT. OF PSYCHOLOGY3947 RIDGEMONT DRIVE UNIVERSITY OF TENNESSEEMALIBU, CA 90265 KNOXVILLE, TN 37916
MINRAT M. L. RAUCH 1 Frank L. SchmidtPII 4 Department of PsychologyBUNDESMINISTERIUM DER VERTEIDIGUNG Bldg. GGPOSTFACH 1328 George Washington UniversityD-53 BONN 1, GERMANY Washington, DC 20052
1-Dr. Mark D. Reckase 1 Dr. Kazuo Shigemasu
Educational Psychology Dept. University of Tohoku
University of Mi3souri-Columbia Department of Educational Psychology4 Hill Hall Kawauchi, Sendai 980Columbia, MO 65211 JAPAN
1 Dr. Lauren Resnick 1 Dr. Edwin ShirkeyLRDC Department of PsychologyUniversity of Pittsburgh University of Central Florida3939 O'Hara Street Orlando, FL 32816Pittsburgh, PA 15213
1 Dr. Richard SnowMary Riley School of EducationLRDC Stanford UniversityUniversity of Pittsburgh Stanford, CA 943053939 O'Hara StreetPittsburgh, PA 15213 1 Dr. Kathryn T. Spoehr
Pscyhology DepartmentDr. Leonard L. Rosenbaum, Chairman Brown UniversityDepartment of Psychology Providence, RI 02912Montgomery CollegeRockville, MD 20850 1 DR. PATRICK SUPPES
INSTITUTE FOR MATHEMATICAL STUDIES INDr. Ernst Z. Rothkopf THE SOCIAL SCIENCESBell Laboratories STANFORD UNIVERSITY600 Mountain Avenue STANFORD, CA 94305Murray Hill, NJ 07974
1 Dr. Hariharan Swaminathan1 Dr. Lawrence Rudner Laboratory of Psychometric and
403 Elm Avenue Evaluation ResearchTakoma Park, MD 20012 School of Education
University of MassachusettsAmherst, MA 01003
.4
% w'. . ."• ."" -" -, .". " .. ''' .-.... , .. ,-.-.-. ..-.-.-... .- -.... ". ...- ,- ...
Non Govt
Dr. Brad SympsonPsychometric Research GroupEducationnl Testing ServicePrinceton, NJ 08541
*- 1 Dr. Kikumi T3tsuoka
Computer Based Education ResearchLaboratory
252 Engineering Research LaboratoryUniversity of IllinoisUrbana, IL 61801
Dr. David Thissen
Department of PsychologyUniversity of KansasLawrence, KS 66044
* 1. Dr. Douglas Towne
Univ. of So. CaliforniaBehavioral Technology Labs184I5 S. Elena Ave.
" Redondo Beach, CA 90277
Dr. Robert TautakawaDepartment of StatisticsUniversity of MissouriColumbia, MO 65201
Dr. David ValeAssessment Systems Corporation2395 University AvenueSuite 306St. Paul, MN 55114
* 1. Dr. Howard WainerDivision of Psychological StudiesEducational Testing ServicePrinceton, NJ 08540
DR. SUSAN E. WHITELYPSYCHOLOGY DEPARTMENTUNIVERSITY OF KANSASLAWRENCE, KANSAS 660441
* 1 Wolfgang WildgrubeStreitkraefteamtBox 20 50 03D-5300 Bonn 2
"S
:. ,:''G .' ,',T h' ' "., ' . .-. .'.- '.- "J ," ".$".-.. ,. .
. . . . . . ..,. ...._ .. .- . .*,, , , , -,..,', *, ,.-.. . . . . *: , . .. .. >* .,-;,.