validation of cognitive structures_ sem (struct eq model) approach

7/30/2019 Validation of Cognitive Structures_ SEM (Struct Eq Model) Approach

1/23

MULTIVARIATE BEHAVIORAL RESEARCH 1

Multivariate Behavioral Research, 38 (1), 1-23

Copyright 2003, Lawrence Erlbaum Associates, Inc.

Validation of Cognitive Structures:

A Structural Equation Modeling Approach

Dimiter M. DimitrovKent State University

and

Tenko Raykov

Fordham University

Determining sources of item difficulty and using them for selection or development of test

items is a bridging task of psychometrics and cognitive psychology. A key problem in this

task is the validation of hypothesized cognitive operations required for correct solution of

test items. In previous research, the problem has been addressed frequently via use of the

linear logistic test model for prediction of item difficulties. The validation procedure

discussed in this article is alternatively based on structural equation modeling of cognitive

subordination relationships between test items. The method is illustrated using scores of

ninth graders on an algebra test where the structural equation model fit supports the

cognitive model. Results obtained with the linear logistic test model for the algebra test are

also used for comparative purposes.

Determining sources of item difficulty and using them for analysis,

development, and selection of test items necessitates integration of cognitive

psychology and psychometric models (e.g., Embretson, 1995; Embretson &

Wetzel, 1987; Mislevy, 1993; Snow & Lohman, 1984). The cognitivestructure of a given test is defined by the set of cognitive operations and

processes, as well as their relationships, required for obtaining correct

answers on its items (e.g., Riley & Greeno, 1988; Gitomer & Rock, 1993).

Knowledge about cognitive and processing operations in a model of item

difficulty prediction allows test developers to (a) construct items with

difficulties known prior to test administration, (b) avoid piloting of individual

items in study groups and thus reduce costs, (c) match item difficulties to

ability levels of the examinees, and (d) develop teaching strategies that target

specific cognitive and processing characteristics. Previous studies have

integrated cognitive structures with (unidimensional and multidimentional)

item response theory (IRT) models allowing prediction of item difficulty from

cognitive and processing operations that are hypothesized to underlie item


2/23

D. Dimitrov and T. Raykov

2 MULTIVARIATE BEHAVIORAL RESEARCH

solving (e.g., Embretson, 1984, 1991, 1995, 2000; Fischer, 1973, 1983; Spada,

1977; Spada & Kluwe, 1980; Spada & McGaw, 1985).

The validation of hypothesized cognitive components with IRT models is

focused on the accuracy of item difficulty prediction. While such tests are

important for prediction purposes, they do not tap into cognitive relationshipsamong items that logically relate to the validity of a cognitive structure.

Therefore, the purpose of this article is to discuss a method that (a) furnishes

an overall validation of cognitive structures and (b) provides diagnostic

feedback about cognitive relationships among items. It should be noted that

this method and previously used IRT methods do not compete, since they

address the validation of cognitive structures from different perspectives.

To illustrate this, a brief description of the IRT linear logistic test model

(LLTM; Fischer, 1973, 1983) is provided below. The LLTM is suitable forsuch anillustration due toits relative simplicity and frequent use inprevious

IRT studies on cognitive test structures (e.g., Dimitrov & Henning, in press;

Embretson & Wetzel, 1987; Fischer, 1973; Medina-Diaz, 1993; Spada, 1977;

Spada & Kluwe, 1980; Spada & McGaw, 1985; Whitely & Schneider, 1981).

In the LLTM, the item difficulty parameter ifor each ofn items (i = 1, ..., n)

is presented as a linear combination of cognitive operations (Fischer, 1973, 1983):

(1) @ =i ik k k

pw c= +

=

,1

wherep n-1,k

represents the impact of the kth cognitive operation on the

difficulty of any item from the target set of items, wik

(weight ofk) adjusts the

impact ofkfor item i, and c is a normalization constant (k= 1, ...,p). The matrix

W= (wik)

is referred to as the weight matrix [i = 1, 2, ..., n, k= 1, ...,p; in the

remainder, the symbol (.) is used to denote matrix]. The LLTM is obtained

by replacing the item difficulty parameter, i, in the Rasch measurement

model (Rasch, 1960) with the linear combination on the right-hand side of

Equation 1:

(2) P(Xij

= 1|j,

k, w

ik) =

exp

exp

,

K =

K =

j ik k

k

p

j ik k

k

p

w c

w c

+F

HG

I

KJ

L

NMM

O

QPP

+ +FHG I

KJL

NMM

OQPP

=

=

1

1

1


3/23



whereXij

is a dichotomous observed variable taking the value of 1 if person

j correctly solves item i and 0 otherwise, and P(.|.) denotes the probability

of a person with ability j

to answer correctly the ith item with difficulty i

as obtained from (1) (i = 1, ..., n,j = 1, ...,N;Nbeing sample size). In an

IRT context, the term ability connotes a latent trait that underlies thepersons performance on a test (e.g., Hambleton & Swaminathan, 1985).

The ability score, , of a person relates to the probability for this person to

answer correctly any test item. Ability and item difficulty aremeasured on

the same scale. The units of this scale, called logits, represent natural

logarithms of odds for success on the test items. With the Rasch model,

log[P/(1-P)] = - , where P is the probability for correct response on an item

with difficulty for a person with ability . Thus, when a person and an item

share the same location on the logit scale ( = ), the person has a .5probability of answering the item correctly. Also, using that exp(1) = e = 2.72

(rounded to the nearest hundredth), a difference of one point on the logit

scalecorresponds to a factor of 2.72 in theodds for success on the test items.

The LLTM relates test items to cognitive operations inferred from a

theoretical model of knowledge structures and cognitive processes. For

example, Embretson (1995) proposed seven cognitive operations required for

correct solution of mathematical word problems by operationalizing three

types of knowledge (factual/linguistic, schematic, and strategic) defined in thetheoretical model of Mayer, Larkin, and Kadane (1984). It should be noted that

the LLTM has been found to be very sensitive to specification errors in the

weight matrix, W (Baker, 1993). This implies the necessity of careful and

sound validation of the cognitive model that underlies an LLTM application.

It is important to note that the LLTM is a Rasch model with the linear

restrictions in Equation 1 imposed on the Rasch item difficulty parameter, i.

Therefore, testing the fit of an LLTM requires two steps: (a) testing the fit

of the Rasch model (which includes testing for unidimensionality) and (b)

testing the linear restrictions in Equation 1 (Fischer, 1995, p. 147). While

there are numerous tests for the Rasch model (e.g., Glas & Verhelst, 1995;

Smith, Schumacker, & Bush, 1998; Van den Wollenberg, 1982; Wright &

Stone, 1979), testing the restrictions in Equations 1 is typically performed

with the likelihood ratio (LR) test for relative goodness-of-fit of the LLTM

and Rasch models (e.g., Fischer, 1995, p. 147). The pertinent test statistic

is asymptotically chi-square distributed with degrees of freedom equal to the

difference in the number of independent parameters of the two compared

models. There is also a simple graphical test for the relative goodness-of-fit of the LLTM and Rasch model (Fischer, 1995). The logic of this test is

that if the LLTM fits well, the points with coordinates that represent

estimates of item difficulty with the LLTM and the Rasch model,


4/23



respectively, should scatter around a 45line through the origin (see Figure

4). As Fischer (1995) noted, the LR test very often turns out significant in

empirical research, thus leading to rejection of the LLTM even when the

graphical goodness-of-fit test indicates a good match between Rasch and

LLTM item difficulties. Medina-Diaz (1993) applied the quadraticassignment technique for analysis of cognitive structures with the LLTM, but

her approach similarly failed to provide good agreement with this LR test,

and inorder to proceedrequired data on the response of each examinee on

each cognitive operation information that is difficult or impossible to obtain

in most testing situations.

While LLTM tests are important for accuracy in the prediction of Rasch

item difficulty from cognitive operations and processes, they do not tap into

cognitive relationships among items that are logically inferred from thecognitive structure. The idea underlyingthe methodproposed in this article

is to reduce the cognitive weight matrix, W, to a diagram of cognitive

subordination relationships between items, and then test the resulting model

for goodness-of-fit using structural equation modeling (SEM; Jreskog &

Srbom, 1996a, 1996b). Triangulations with logical fits in the diagram of

cognitive subordinations among items can provide additional heuristic

evidence in the validation process.

It is worthwhile stressing thatthe LLTM tests and the proposed SEMmethod do not compete, because they target different aspects of the

validation of cognitive structures. LLTM tests relate to the accuracy of

prediction of Rasch item difficulty from cognitive operations, while the SEM

method relates to the validation of cognitive relations among items for a

hypothesized cognitive structure. The proposed SEM method can be used

to justify, not replace, applications of LLTM or other models for prediction

of item difficulty such as multiple linear regression (e.g., Drum, Calfee, &

Cook, 1981) or artificial neural network models (Perkins, Gupta, &

Tammana, 1995). The information about cognitive relations among items

provided by the proposed SEM method can be used also for purposes other

than prediction of item difficulty (e.g., task analysis, content selection,

teaching strategies, and curriculum development).

A Structural Equation Modeling Based Procedure for Validation of

Cognitive Structures

Oriented Graph of Cognitive Structures

Given a set ofm cognitive operations required for solution ofn items

comprising a considered test, the weight matrix Wrepresents a two-way


5/23



table with elements wik

= 1 indicating that itemIirequires cognitive operation

k, or wik

= 0 otherwise; (i = 1, ..., n; k= 1 , ..., m). In this article, an itemIj

is referred to as subordinated to item Ii

if and only if the cognitive

operations required for the correct solution of item Ij

represent a proper

subset (in the set-theoretic sense) of the cognitive operations required for thecorrect solution of itemI

i. For purposes of symbolizing the subordination

relationships between items, the W matrix can be reduced to a matrix of item

subordinations, denoted S = (sij), with elements s

ij=1 indicating that itemI

j

is subordinated to itemIiand s

ij= 0 otherwise (i,j = 1, ..., n ). We note that

two arbitrarily chosen test items need not necessarily be in a relation of

subordination; if they are not, then sij

= 0 holds for their corresponding entry

in the matrix S.

One can now represent graphically the matrix S as an oriented graphwhere an arrow fromI

jto I

i(denotedI

j I

iin the graph) indicates that item

Ij

is subordinated to itemIi. Figure 1 presents a hypothetical weight matrix

W, its corresponding S matrix, and the oriented graph associated with them.

An examination of thematrixW shows, for example, that itemI1is subordinated

to itemI4

as the set of cognitive operations required by itemI1

(viz. operations

O1and O

3) is a proper subset of the cognitive operations required by itemI

4(viz.

operations O1, O

3, and O

4). Therefore, we have s

41= 1 in thematrix S and an

arrow from itemI1 to itemI4 in the graph diagram. In the matrix S, one canalso see that all entries in the column that corresponds to itemI2equal 0. This

is because itemI2

is not subordinated to any of the other items. Indeed, the

set of cognitive operations required by itemI2(viz. operations O

2and O

3) do

not represent a proper subset of the cognitive operations required by any

other item. Thus, si2

= 0 (i = 1, 2, 3, 4) and there are no arrows from item

I2to other items in the graph diagram. Similarly, one can check other entries

in the matrix S.

The transitivity of the subordination relationship holds when successive

arrows in the oriented graph connect more than two items. For example, the

three-item pathI3 I

1 I

4indicates that (a)I

3is subordinated toI

1,(b)

I1

is subordinated to I4, and by transitivity (c) I

3is subordinated to I

4.

Relationships of subordination by transitivity are not represented by arrows

in a diagram in order to simplify the graphical representation.

Relationship to Structural Equation Modeling

The presently proposed method of cognitive structure validation makesthe assumption that for each test item there exists a continuous latent

variable, denoted (appropriately sub-indexed if necessary), which

represents the ability required for correct solution of the item. Persons solve


6/23



an item correctly if they posses in sufficient degree this ability, that is, if their

ability relevant to the item exceeds or equals a certain minimal level (denoted

) required for its correct solution. Thus, formally, the assumption is

(3) Xij = 1 ifiji,0 if

ij<

i,

where ij

denotes thejth persons level of ability required for solving the ith

item, and iis a threshold parameter above/below which a correct/incorrect

response results for the ith item (i = 1, ..., n,j = 1, ...,N). We further assume

that the n-dimensional vector of latent variables = (1, ...,

n) is

multivariate normal.

W matrix S matrix

Cognitive Operation Item I1

I2

I3

I4

I5

Item O1

O2

O3

O4

I1

0 0 1 0 1

I2

0 0 1 0 0

I1

1 0 1 0 I3

0 0 0 0 0

I2

0 1 1 0 I4

1 0 1 0 1

I3

0 0 1 0 I5

0 0 0 0 0

I4

1 0 1 1

I5

1 0 0 0

Figure 1

Graph Diagram of Item Subordinations for a Hypothetical W Matrix

Graph Diagram


7/23



The assumptions in this subsection are identical to those made when

analyzing ordinal data using structural equation modeling (SEM; Jreskog &

Srbom, 1996b, ch. 7). Thus we are now in a position to evaluate the degree

to which a set of cognitive subordination relationships among given items is

plausible, by using the corresponding goodness of fit test available in SEM.That is, we can validate the cognitive structure represented by an oriented

graph by applying the popular SEM methodology. Specifically, in this

framework we consider the item subordination relationships to be the

building blocks of a structural equation model relating the items.

Accordingly, the model of interest is

(4) = B +

where B = (rs) is the n n matrix of relationships between the abilities

required for successful solution of the rth and sth items while is the vector

of corresponding zero-mean residuals, each assumed unrelated to those

variables that are predictors of its pertinent ability variable (r, s = 1, ..., n).

Hence, in modeling subjects behavior on say the rth item it is assumed that

up to an associated residual rthe required ability for its correct solution,

r,

is linearly related to those of corresponding other items (r= 1, ..., n; existence

of the inverse of the matrix In - B is also assumed, as in typical applicationsof SEM; e.g., Jreskog & Srbom, 1996b).

Figure 2 represents the oriented graph of item subordinations for the

weight matrix W from the example in the next section. The structural

equation model corresponding to Equation 4 is represented in Figure 3. Its

path-diagram follows widely adopted graphical convention of displaying

structural equation models. In this diagram, each item is represented by a

circle denoting the latent ability required for its correct solution. Other than

this inconsequential detail for the following developments, the model in

Figure 3 symbolically differs from the oriented graph in Figure 2 only in that

residuals are associated with latent variables receiving one-way arrows that

represent the cognitive subordination relationship of the item at its end to the

item at its beginning. With the proposed SEM approach, this model is fitted

to the n n tetrachoric correlation matrix of the observed dichotomous

variablesXithat each evaluate whether the ith item has been correctly solved

or not (i = 1, ..., n; cf. Equation 3).

Thus, with the SEM method used in this article, the validity of a cognitive

structure is tested by the following three-step procedure. First, reduce theweight matrix W to a matrix S of cognitive subordinations. Second, as

described in the previous section, represent graphically the cognitive

subordinations among items in the matrix S as an oriented graph. Third, using


8/23



the ordinal data SEM approach, fit the latent relationship model based on the

last graph (as its path diagram) to the tetrachoric correlation matrix of the

items (Jreskog & Srbom, 1996b, ch. 7). The goodness of fit test available

thereby represents a test of the plausibility of the originally hypothesized set

of cognitive subordination relationships among the studied items. As abyproduct of this approach, one can also estimate the thresholds

iof all items

(i = 1, ..., n), which allow comparison of their difficulty levels. Evaluation of

significance of elements of the matrix B in Equation 4 indicates whether,

under the model, one can dispense with the assumption of cognitive

subordination for certain pairs of items (viz. those for which the

corresponding regression coefficient is nonsignificant). Further, in case of

lack of model fit, modification indices may be consulted to examine if model-

data mismatch may be considerably reduced by introduction of substantivelymeaningful subordination relationships between items, which were initially

omitted from the model. (This use of modification indices would initiate an

exploratory phase of analysis; Jreskog & Srbom, 1996c.)

The discussed SEM approach is based on the tacit assumption of

subjects not guessing on any of the items. This is an important assumption

made also in many applications of item response models and is justified by

the observation that guessing would imply for pertinent item(s) more than one

trait underlying their successful solution. Further, the approach is best usedwhen the sample of subjects having taken all items of a test under

consideration is large (Jreskog & Srbom, 1996b). Moreover, for this SEM

approach, the assumption of an underlying multinormal latent ability vector

is relevant, as well as that of linear relationships among its components as

reflected in the underlying model Equation 4 that is tested in an application

of the discussed SEM method (Jreskog & Srbom, 1996a).

Triangulation with Logical Fits

Logical fits in the path diagram also lend support to the validity of a

cognitive structure. One possible logical fit, referred to here as path-wise

increase of item difficulty, is basedon the assumption that the difficulty of

items increases with the increase of processing tasks required for item

success. Under this assumption, the item difficulty is expected to increase

down the arrows (path-wise) for any path of the oriented graph. With the

example provided later in this article, the items are simple linear algebra

equations with their difficulty logically increasing with the increase ofproduction rules required for the correct solution of these equations (see

Appendix). For this example, there was an increase of Rasch item difficulty

down the arrows for each path of the oriented graph, thus providing


9/23



triangulation support to the validation based on the SEM goodness-of-fit test.

However, the assumption of path-wise increase of item difficulty may not be

appropriate with other cognitive structures. Depending on the W matrix,

cognitive subordinations among items may not translate into ordered difficulty

values. In such cases, other logical fits may better reflect the cognitiverelationships among items in the path diagram of the W matrix. One can even

decide to modify the working definition of cognitive subordinations among

items to accommodate the substantive context of the W matrix. The validation

of cognitive structures is, after all, a flexible logical process of collecting

evidence that may include, but is not limited to, an isolated statistical act.

Example

This example illustrates the proposed validation method for a cognitive

structure on an algebra test. The test consisted of 15 linear equations to be

solved (see Appendix) and was administered toN= 278 ninth-grade high

school students in northeastern Ohio. The students were required to show

work toward solution in order to avoid guessing on the test items. The

cognitive structure of this test was determined by 10 production rules

(PRs) required for the correct solutions of the 15 equations. An earlier

version of the test and the PRs were developed in a previous study (Dimitrov& Obiekwe, 1998) in an attempt to improve the system of production rules

for algebra test of linear equations proposed by Medina-Diaz (1993).

Table 1 presents the W matrix of 15 items and 10 production rules. Table

2 gives the S matrix of item subordinations determined from W. The path

diagram corresponding to the matrix S is given in Figure 2. As an illustration,

itemI2

is the algebra equation 5x- 3 = 7 + 4x. The production rules (see

Appendix) for the correct solution of this equation are: PR6 (balancing):

5x- 4x= 7 + 3, PR3 (collecting numbers): 5x- 4x= 10, and PR4 (collecting two

terms):x= 10. On the other hand, the correct solution of equation 2(x+ 3) =x

- 10" (itemI1) requires the same three production rules plus PR7 (removing

parenthesis with a positive coefficient). Thus, the PRs required by itemI2

represent a proper subset of the PRs required by item I1

(cf. Table 1). This

indicates that itemI2is cognitively subordinated to itemI

1:I

2 I

1(Figure 2).

The tetrachoric correlation matrix resulting from the data is presented in Table

3 (and obtained with PRELIS2; Jreskog & Srbom, 1996a).

We note that in the general case of application of the method of this article,

residuals are associated with each dependent latent variable unless theoreticalconsiderations allow one to hypothesize lacking such for a certain latent

variable(s). In the present case, in which an algebra test is used to illustrate the

method, no residual term is assumed to be associated with the ability


10/23



Table 1

Matrix W for Fifteen Algebra Items and Ten Production Rules

Production Rules

Item PR1 PR2 PR3 PR4 PR5 PR6 PR7 PR8 PR9 PR10

I1

0 0 1 1 0 1 1 0 0 0

I2

0 0 1 1 0 1 0 0 0 0

I3

1 1 0 0 0 1 1 0 0 0

I4

1 0 1 1 0 1 1 0 0 0

I5

1 0 0 0 0 0 0 0 0 1

I6

1 1 1 1 0 1 0 0 0 0

I7

1 0 1 1 0 1 1 0 1 0

I8

0 0 1 0 0 1 0 0 0 0

I9

0 0 1 1 0 0 0 0 0 0

I10

1 0 1 1 1 1 0 0 0 0

I11

1 0 1 1 0 1 0 0 0 1

I12

1 0 1 1 0 1 1 1 0 0

I13

1 0 1 1 1 1 1 1 0 0

I14

1 1 1 1 1 1 1 1 0 0

I15

1 0 1 1 1 1 1 0 0 0

Table 2

Matrix S for the W Matrix of the Algebra Test

Item I1

I2

I3

I4

I5

I6

I7

I8

I9

I10

I11

I12

I13

I14

I15

I1

0 1 0 0 0 0 0 1 1 0 0 0 0 0 0

I2 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0I3

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I4

1 1 0 0 0 0 0 1 1 0 0 0 0 0 0

I5

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I6

0 1 0 0 0 0 0 1 1 0 0 0 0 0 0

I7

1 1 0 0 0 0 0 1 1 0 0 0 0 0 0

I8

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I9

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I10

0 1 0 0 0 0 0 1 1 0 0 0 0 0 0

I11 0 1 0 0 1 0 0 1 1 0 0 0 0 0 0I12

1 1 0 0 0 0 0 1 1 1 0 0 0 0 0

I13

1 1 0 0 0 0 0 1 1 1 0 1 0 0 0

I14

1 1 1 1 0 0 0 1 1 1 0 1 1 0 1

I15

1 1 0 1 0 0 0 1 1 1 0 0 0 0 0


11/23



corresponding to itemI2. This is because the ability that underlies the correct

solution of itemI2is determined by the abilities required for solving itemsI

8and

I9

that are related to itemI2

in the considered model. Indeed (see Table 1), the

production rules required for the correct solution of itemI2(PR3, PR4, and PR6)

are fully represented by those required for the correct solutions of itemI8 (PR3and PR6) and itemI

9(PR3 and PR4). Similarly, no residual is assumed to be

associated with the latent variable corresponding to item I13

. Indeed, an

examination of the solution of itemI12

that is related to itemI13

in the model

suggests that the ability underlying correct solution of the latter is determined by

those required to solve correctly itemI12

. This is because itemsI12

andI13

require

the same PRs, with the exception that itemI13

requires slightly higher frequency

for two of these PRs (namely, collecting terms and removing parentheses).

SEM Test for Goodness-of-Fit

Figure 2 represents the path diagram of the initial structural equation model

resulting from the matrix S of item subordination relationships. This model was

Table 3

Tetrachoric Correlation Matrix of the Algebra Test Items (N= 278)

I1

I2

I3

I4

I5

I6

I7

I8

I9

I10

I11

I12

I13

I14

I15

I1

1.00

I2

0.78 1.00

I3

0.52 0.48 1.00

I4

0.54 0.53 0.63 1.00

I5

0.55 0.48 0.56 0.56 1.00

I6

0.51 0.45 0.77 0.63 0.71 1.00

I7

0.58 0.41 0.44 0.40 0.51 0.57 1.00

I8

0.54 0.58 0.31 0.57 0.33 0.60 0.49 1.00

I9 0.56 0.51 0.44 0.50 0.51 0.55 0.66 0.61 1.00I

100.56 0.53 0.56 0.64 0.50 0.61 0.53 0.58 0.60 1.00

I11

0.41 0.29 0.32 0.26 0.54 0.53 0.34 0.53 0.48 0.33 1.00

I12

0.63 0.66 0.68 0.65 0.48 0.64 0.53 0.72 0.68 0.71 0.44 1.00

I13

0.55 0.50 0.43 0.47 0.49 0.55 0.47 0.51 0.57 0.46 0.25 0.82 1.00

I14

0.29 0.34 0.56 0.56 0.45 0.61 0.51 0.33 0.44 0.64 0.21 0.81 0.34 1.00

I15

0.41 0.47 0.46 0.60 0.39 0.55 0.45 0.58 0.66 0.74 0.41 0.74 0.59 0.57 1.00

Note. N= sample size.


12/23



fitted to the tetrachoric correlation matrix of thedichotomously scored test

items that is presented in Table 3, and found to be associated with a chi-square

value 2 = 162.81 with degrees of freedom (df) = 84 and a root mean square

error of approximation (RMSEA) = .058 with 90%-confidence interval (.045;

.071). In this model, the relationship between itemsI14 andI15 turned out tobe nonsignificant:

14,15= -.28, standard error = .29, t-value = -.96. Inspecting

the cognitive processes required for correct solution of the (model-assumed)

related itemsI3,I

13,I

14, andI

15, we noticed that the operations necessary for

solving itemsI3

andI13

were in fact those required for correct solution of item

I14

(see Table 1). In the context of correct solution of itemsI3andI

13, therefore,

the cognitive operations underlying itemI15

are inconsequential for correct

solution of item I14

. This suggests that we can dispense with the earlier

assumed relationship between itemsI14 andI15. Indeed, fixing the path fromitemI

15toI

14in the model displayed in Figure 2 resulted in a nonsignificant

increase of the chi-square value up to 2 = 163.89, df= 85, RMSEA = .058

(.044; .071) (2 = 1.08,df= 1,p > .10). In addition, these model fit indices

are themselves acceptable (e.g., Jreskog & Srbom, 1996b), and hence the

last model version can be considered a tenable means of data description and

explanation. The parameter estimates and standard errors associated with this

final model are presented in its path-diagram provided in Figure 3. We

comment next on the elements of this solution by comparing path estimates totheir standard errors (the ratio of parameter estimate to standard error equals

the t-value; Jreskog & Srbom, 1996b).

First, moving from top to bottom in Figure 3 we note the relatively strong

relationship between each of items I8

and I9

with item I2. This is not

unexpected, given the similarity of these items in content and production

rules they require (see Appendix). At the same time, we note the

nonsignificant relationships between itemI11

and each of itemsI2andI

5. One

possible explanation is the lack of a linear relationship between the abilities

to solve each of itemsI2

andI5, with that of itemI

11. Indeed, an inspection

of the students test work on these items showed that those who did not solve

correctly itemI11

failed at the initial step, PR10 (removing denominators),

because of difficulty with finding the least common denominator. Therefore,

success of these students on the subsequent steps (production rules PR1,

PR3, PR4, and PR6) did not lead to correct solution of itemI11

. When solving

itemI2, however, the successful application of PR3, PR4, and PR6 led to a

correct solution because PR10 (removing denominators) was not required

by itemI2. Further, the weak relationship between itemsI11 and I5 can beexplained by the fact that most students who failed on the least common

denominator with itemI11

avoided this problem with itemI5by applying the

cross-multiplication rule.


13/23



Moving to the lower parts of the model solution in Figure 3, we notice that

itemI2is strongly related to itemsI

1,I

6, andI

10. This indicates that the ability

associated with itemI2plays a central role for solving each of itemsI

1,I

6, and

I10

. This is not unexpected because the production rules required by itemI2

represent the major part of the production rules required by itemsI1,I6, andI

10. Similarly, the ability associated with itemI

1plays a central role for solving

correctly itemsI4,I

7, andI

12 the paths leading from itemI

1into each of these

three items are significant. In this context, the ability of solving itemI10

is

only weakly related to that of solving itemI12

. This indicates that there is no

unique contribution of the ability underlying itemI10

to the correct solution of

itemI12

after controlling for the contribution of the ability underlying itemI1

to solving correctly itemI12

.

The relationships of itemsI4 and I10 to itemI15 are significant and thusindicate that, in the context of correctly solving itemI

4, the ability to correctly

solve itemI10

is also relevant (i.e., has unique contribution) for successful

solution of item I15

. An explanation of this finding is that removing

parentheses, which is required by itemI15

, can be avoided in the solution of

itemI4

by collecting the two terms within parentheses: 5(2x- 5x) ... (see

Appendix). Also, collecting more than two terms is required by itemsI10

and

I15

, but not necessarily by itemI4.

Focusing on the lowest part of the model depicted in Figure 3, it isstressed that there is a strong relationship between the ability underlying item

I12

to that associated with itemI13

. Thus, according to the model, the ability

to solve the latter item is essentially proportional to that for solving itemI12

.

This is not surprising given the fact that the same production rules are

required by both itemsI12

andI13

with the exception that collecting terms

and removing parentheses is more frequently required with itemI13

.

Last but not least, the relationship between items I3

and I14

is

nonsignificant while that between itemsI13

andI14

is significant. Thus, in the

context of the ability necessary for handling item I13

, there is no unique

(linear) contribution of itemI3to successful solution of itemI

14. This can be

explained by the fact that, out of the eight production rules required for

solving correctly itemI14

, seven are required also by itemI13

. There is only

one production rule required by item I14

that is required also by itemI3but not

by item I13

(PR2; see Table 1).

For triangulation purposes, if one assigns to each item in Figure 2 its

Rasch difficulty reported in Table 4, there is a path-wise increase of item

difficulty for all paths. This can be illustrated, for example, with the four-itempathI

8(-2.487) I

2(-1.632) I

1(-1.305) I

7(0.793), where the Rasch

difficulty (in logits) is given in parentheses. This logical fit is consistent with

the results from the SEM goodness-of-fit test for the validation of the weight


14/23



matrix W in Table 1. In this example, the path-wise increase of item difficulty

can be expected to occur due to the orderly structure of production rules

required for the solution of simple linear algebra equations (see Appendix).

However, as noted earlier in this article, this may not be true in the context

Figure 2

Graph Diagram of Item Subordinations for the W Matrix in Table 1


15/23



of other cognitive structures. Yet, depending on the W structure, one may

find other plausible heuristic triangulations of the SEM goodness-of-fit test

for the validation of the hypothesized cognitive structure.

Figure 3

Path Diagram (final model) for the SEM Analysis with the W Matrix in Table 1

* Statistically significant beta estimates (p < .01)


16/23



The SEM treatment of the oriented graph of cognitive subordinations

may provide useful validation feedback to subsequent applications of the W

matrix in various measurement and substantive studies. In the next section,

this is illustrated with an application of the LLTM for the test of simple linear

algebra equations (see Appendix). The LLTM, if it fits the data well,provides estimates of the

kparameters in Equation 1 for the prediction of

difficulty of simple linear algebra equations. The accuracy of this prediction

is the LLTM criterion for validity of the W matrix. Evidently, the LLTM

validation can complement, but cannot substitute, the validation of the same

W matrix provided by the SEM method. Moreover, the latter will be the only

validation perspective in this study should the LLTM fail to fit the data.

LLTM Application

The LLTM was conducted with the W matrix in Table 1 using the

computer program for structural Rasch modeling LPCM-WIN 1.0 (Fischer &

Ponocny-Seliger, 1998). As noted earlier, the LLTM can be used only if the

Rasch model fits the data. The test of model fit [2(14) = 22.83, p > .05]

indicates a good fit of the Rasch model. The estimates of LLTM parameters

are provided in Table 4, with kindicating the contribution of the kth production

rule to the difficulty of any item that requires this production rule (k= 1, ..., 10).Estimates of the Rasch difficulty and the predicted (LLTM) difficulty for the

algebra test items are also provided in Table 4. The graphical goodness-of-fit

test (Fischer, 1995) in Figure 4 indicates very good fit, with a Pearson

correlation of .975 between the LLTM and Rasch item difficulties. However,

the LR difference in chi-square test does not lend support to the fit of the

LLTM relative to the Rasch model [2(4) = 46.91, p < .01]. This is not

surprising given the reported strictness of this LR test in the literature (e.g.,

Fischer, 1995, p. 147). Thus, while the graphical test supports the validation

of the W matrix in terms of accurate prediction of Rasch item difficulty with

the LLTM, the strict LR test does not.

In this situation, the SEM validation of the W matrix in the previous

section can moderate in support of subsequent applications of Equation 1

for the LLTM prediction of item difficulty in a test of simple liner algebra

equations from 10 production rules (see Appendix).

Discussion and Conclusion

In previous research, validation of cognitive structures was addressed

primarily with LLTM applications for predicting item difficulties. As is well

known, LLTM goodness-of-fit tests focus on the similarity in fit between the


17/23



Table 4

LLTM Analysis with the W Matrix of the Algebra Test

Parameter Estimates Item Difficulty

PRs k(SE)a Item LLTM Rasch

PR1 -0.735 ( 0.151)** I1

-1.5659 -1.3050

PR2 2.401 ( 0.205)** I2

-1.1484 -1.6323

PR3 -1.536 ( 0.271)** I3

1.0500 1.1704

PR4 -0.773 ( 0.175)** I4

-0.4557 -0.2654

PR5 0.426 ( 0.178)* I5

0.3042 -0.1923

PR6 -0.950 ( 0.115)** I6

0.7432 0.7265

PR7 -0.314 ( 0.110)** I7 1.0813 0.7931PR8 -2.328 ( 0.145)** I

8-2.9167 -2.4871

PR9 0.388 ( 0.157)* I9

-1.5659 -1.6323

PR10 0.458 ( 0.138)** I10

-0.4245 -0.1440

I11

1.9139 2.3913

I12

-0.3993 -0.6997

I13

1.0828 0.9262

I14

1.8541 2.0282

I15

0.4469 0.3225

aSE= standard error of the LLTM parameter k

(k= 1,..., 10).

*p < .05. **p < .01.

Figure 4

Graphical Goodness-Fit-Test for the LLTM with the W Matrix in Table 1


18/23



LLTM and theRasch model. However, the associated LR difference in chi-

square test rather frequently turns out significant, thus leading to rejection of

the LLTM even when the graphical test for goodness-of-fit indicates a good

match between the estimates of the Rasch item difficulties and their LLTM

predicted values (e.g., Fischer, 1995). The quadratic assignment test for theLLTM (Medina-Diaz, 1993) requires data on the subjects responses on each

cognitive operation information that is difficult or impossible to obtain in most

testing situations. The LLTM tests are important for accuracy in the prediction

of Rasch item difficulties but they do not tap into cognitive relationships among

items for the validation of the hypothesized cognitive structure. In addition, the

LLTM works under relatively strong assumptions: (a) the test is

unidimensional, (b) the Rasch model fits the data well, and (c) the restrictions

in Equation 1 hold.The SEM approach discussed in this article provides an overall

goodness-of-fit test as well as valuable information about cognitive

subordinations among test items. An item is referred to here as cognitively

subordinated to another item if the cognitive operations required by the first

item represent a proper subset of the cognitive operations required by the

second item. With the proposed validation method, the W matrix of a

cognitive structure is reduced to a matrix Sof item subordinations. The

model corresponding to the path diagram associated with the matrix S is thensubmitted to SEM analysis for goodness-of-fit by fitting it to the tetrachoric

correlation matrix of dichotomously scored items.

It should be emphasized that the SEM validation model is not tied to

specific assumptions of the LLTM or other IRT models that focus on

prediction of item difficulty from cognitive operations. Lack of

unidimensionality for the test may preclude the use of the LLTM, but not

applicability of the SEM method proposed in this article. Conversely, lack

of fit ofthe SEM model has no direct implications for the Rasch model, but

may question the substantive validity of Equation 1 with the LLTM. Indeed,

just as accuracy of prediction with a multiple linear regression equation does

not exclude specification problems, accuracy of item prediction with

Equation 1 does not exclude possibility for misspecifications of cognitive

components.

The path diagram corresponding to the matrix S can also be used for

additional substantive and logical analyses of cognitive structures. With the

cognitive structure of some tests (e.g., orderly structured production rules in

solving algebra equations), it is sound to expect that the more processingoperations are required by the items, the more difficult the items become. Under

this assumption, the increase of item difficulty along the paths of the diagram

provides heuristic support to the validity of the cognitive structure. This was


19/23



illustrated in the previous section by associating the items of the path diagram in

Figure 2 with their Rasch difficulties. When the Rasch model does not fit the

data, one can associate the items in the path diagram with their difficulty

estimates produced by other IRT models that assume no guessing (e.g., the

two-parameter logistic model; see Embretson & Reise, 2000, p. 70). With somecognitive structures, however, cognitive subordinations between items do not

necessarily parallel the item difficulty estimates. In such cases, one may use

other logical fits in the path diagram to triangulate the validation provided by the

SEM goodness-of-fit test. As noted earlier, one can even modify the working

definition of cognitive subordinations among items to accommodate the

substantive context of the W matrix. Combining the SEM goodness-of-fit test

with logical fits for the path diagram is a dependable process of collecting

evidence about the validity of a hypothesized cognitive structure.It should be noted that the path diagram corresponding to the matrix S

displays paths of subordinated items generated from the matrix W, but does not

include partial overlaps of cognitive operations between items. Despite this

limitation, the SEM goodness-of-fit test and triangulation logical fits of the paths

have strong potential toprovide evidence for the validity of a cognitive structure.

As illustrated in the previous section with the LLTM for the algebra test, the strict

LR difference in chi-square test can be soundly counterbalanced by consistent

results of the LLTM graphical goodness-of-fit test, the SEM goodness-of-fittest, and logical fits for the path diagram. In addition, the joint analysis of

cognitive substance and SEM coefficients of paths in the oriented graph can

provide valuable feedback about linearity of relationships, redundancy, and

uniqueness of contribution for abilities that underlie the correct solution of items.

This was illustrated with the joint analysis of SEM path estimates and substantive

implications for abilities that underlie the correct solution of items in the algebra

test. Also, the information about cognitive relations among items provided by the

path diagram and its SEM analysis can be used, for example, in task analysis,

content selection, teaching strategies, and curriculum development.

When using the method of this article, we emphasize that repeated

modification of the initial structural equation model effectively represents

elements of an exploratory analysis along with the confirmatory ones that are

inherent in a typical utilization of SEM. In this type of application of the method,

it is important that researchers validate the final model on data from an

independent sample before more general conclusions are drawn. Similarly,

such a validation is recommended also when the model that was fitted turns out

to be associated with acceptable fit indices and presents substantivelyplausible means of description of the relationships between the analyzed items.

Even in the latter likely rare cases in empirical work, it is highly recommendable

to replicate the model in other studies aiming at cognitive validation of the same


20/23



test items. We also stress that the method of this article necessitates large

samples (Jreskog & Srbom, 1996b). This essential requirement is justified

on two related grounds. One, the analytic approach underlying the method is

the SEM methodology that represents itself a large-sample modeling

technique. Two, since the analyzed data is dichotomous (true/false solution oftest items), application of the weighted least squares method of model

estimation and testing is necessary, which requires as a first step the estimation

of a potentially rather large weight matrix (Jreskog & Srbom, 1996b).

Further, with other than large samples, the polychoric correlation matrix to

which the structural equation model is fitted may contain entries with

intolerably large standard errors, thus weakening the conclusions reached with

the SEM approach. Since we are not aware of explicit, specific, and widely

accepted guidelines as to determination of sample size in the dichotomous itemcase of relevance here, based on recent continuous variable simulation

research reviewed and presented by Boomsma and Hoogland (2001; see also

Hoogland, 1999; Hoogland & Boomsma, 1998) we would like to suggest that

a sample size of at least several hundred is needed even with a relatively small

number of items (up to dozen, say). In this regard, we would generally

discourage use of the method of this article with a sample size of less than 200

subjects even with a smaller number of items; with more than a dozen of items

say, we would view samples considerably higher than this as minimally needed,and perhaps higher than a thousand as required with larger models.

In conclusion, the SEM method discussed in this article combines confirmatory

and exploratory procedures in a flexible process of collecting statistical and

heuristic evidence about the validity of hypothesized cognitive structures.

References

Baker, F. B. (1993). Sensitivity of the linear logistic test model to misspecification of the

weight matrix. Applied Psychological Measurement, 17(3), 201-210.Boomsma, A. & Hoogland, J. J. (2001). The robustness of LISREL modeling revisited. In

R. Cudeck, S. H. C. duToit, & D. Srbom (Eds.), Structural equation modeling:

Presenceand future. A festschrift in honor of Karl Jreskog. Chicago: Scientific

Software International.

Dimitrov, D. M. & Henning, J. (in press). A linear logistic test model of reading

comprehension difficulty. In R. Hashway (Ed.), Annals of the Joint Meeting of the

Association for the Advancement of Educational Research and the National Academy

for Educational Research 2000. Lanham, Maryland: University Press of America.

Dimitrov, D. M. & Obiekwe, J. (1998, February). Validation of item difficulty components

for algebra problems. Paper presented at the meeting of the Eastern EducationalResearch Association, Tampa, FL.

Drum, P. A., Calfee, R. C., & Cook, L. K. (1981). The effects of surface structure variables

on performance in reading comprehension tests. Reading Research Quarterly, 16,

486-514.


21/23



Embretson, S. E. (1984). A general latent trait model for response processes.

Psychometrika, 49, 175-186.

Embretson, S. E. (1991). A multidimensional latent trait model for measuring learning and

change. Psychometrika, 56, 495-516.

Embretson, S. E. (1995). A measurement model for linking individual learning to process

and knowledge: application to mathematical reasoning. Journal of Educational

Measurement, 32, 277-294.

Embretson, S. E. (2000). Multidimensional measurement from dynamic tests: Abstract

reasoning under stress. Multivariate Behavioral Research, 35, 505-543.

Embretson, S. E. & Reise, S. P. (2000). Item response theory for psychologists . Mahwah,

NJ: Erlbaum.

Embretson, S. & Wetzel, C. D. (1987). Component latent trait models for paragraph

comprehension tests. Applied Psychological Measurement, 11, 175-193.

Fischer, G. H. (1973). The linear logistic test model as an instrument in educational

research. Acta Psychologica, 37, 359-374.

Fischer, G. H. (1983). Logistic latent trait models with linear constraints. Psychometrika,

48, 3-26.

Fischer, G. (1995). The linear logistic model. In G. H. Fischer & I. W. Molenaar (Eds.)

Rasch models. Foundations, recent developments, and applications (pp. 131-155).

New York: Springer-Verlag.

Fischer, G. & Ponochny-Seliger, E. (1998). Structural Rasch modeling: Handbook of the

usage of LPCM-WIN 1.0. Groningen, Netherlands: ProGAMMA.

Gitomer, D. H. & Rock, D. (1993). Addressing process variables in test analysis. In N.

Frederiksen, R. J. Mislevy, & I. Bejar (Eds.), Test theory for a new generation of tests

(pp. 243-268). Hillsdale, NJ: Erlbaum.

Glas, C. A. W. & Verhelst, N. D. (1995). Testing the Rasch model. In G. H. Fisher & I. W.

Molenaar (Eds.). Rasch models. Foundations, recent developments, and applications

(pp. 69-95). New York: Springer-Verlag.

Hambleton, R. K. & Swaminathan, H. (1985). Item response theory: Principles and

applications. Boston: Kluwer-Nijhoff.

Hoogland, J. J. (1999). The robustness of estimation methods for covariance structure

analysis. Unpublished doctoral dissertation. University of Groningen, The

Netherlands.

Hoogland, J. J. & Boomsma, A. (1998). Robustness studies in covariance structure

modeling: An overview and a meta-analysis. Sociological Methods and Research, 26,

329-367.

Jreskog, K. G. & Srbom, D. (1996a). PRELIS2: Users reference guide. Chicago, IL:

Scientific Software International.

Jreskog, K. G. & Srbom, D. (1996b). LISREL8: Users reference guide. Chicago, IL:

Scientific Software International.

Jreskog, K. G. & Srbom, D. (1996c). LISREL8: The SIMPLIS command language.

Chicago, IL: Scientific Software International.

Mayer, R., Larkin, J., & Kadane, P. (1984). A cognitive analysis of mathematical problem

solving. In R. Stenberg (Ed.),Advances in the psychology of human intelligence (Vol.

2, pp. 231-273). Hillsdale, NJ: Erlbaum.

Medina-Diaz, M. (1993). Analysis of cognitive structure using the linear logistic test model

and quadratic assignment. Applied Psychological Measurement, 17(2), 117-130.

Mislevy, R. J. (1993). Foundations of a new theory. In N. Frederiksen, R. J. Mislevy, &

I. Bejar (Eds.), Test theory for a new generation of tests (pp. 19-39). Hillsdale, NJ:

Erlbaum.


22/23



Perkins, K., Gupta, L., & Tammana, R. (1995). Predicting item difficulty in a reading

comprehension test with an artificial neural network. In A. Davies & J. Upshur (Eds.)

Language testing, Vol. 12, No. 1 (pp. 34-53). London: Edward Arnold.

Rasch, G. (1960). Probabilistic models for intelligence and attainment tests. Copenhagen:

Danmarks Paedagogiske Institut.

Riley, M. S. & Greeno, J. G. (1988). Developmental analysis of understanding language

about quantities and of solving problems. Cognition and Instruction, 5, 49-101.

Smith, R. M., Schumacker, R. E., & Bush, M. J. (1998). Using item mean squares to

evaluate fit to the Rasch model. Journal of Outcome Measurement, 2, 66-78.

Snow, R. E. & Lohman, D. F. (1984). Implications of cognitive psychology for educational

measurement. In R. Linn (Ed.), Educational measurement (3rd ed., pp. 263-331).

New York: Macmillan.

Spada, H. (1977). Logistic models of learning and thought. In H. Spada & W. F. Kempf

(Eds.), Structural models of thinking and learning (pp. 227-262).

Spada, H. & Kluwe, R. (1980). Two models of intellectual development and their reference

to the theory of Piage. In R. Kluwe & H. Spada (Eds.), Developmental model of

thinking (pp. 1-32). New York: Academic Press.

Spada, H. & McGaw, B. (1985). The assessment of learning effects with linear logistic test

models. In S. Embretson (Ed.), Test design: New directions in psychology and

psychometrics (pp. 169-193). New York: Academic Press.

Van den Wollenberg, A. L. (1982). Two new statistics for the Rasch model .

Psychometrika, 47, 123-139.

Whitely, S. E. & Schneider, L. (1981). Information structure for geometric analogies: A

test theory approach. Applied Psychological Measurement, 5, 38 3-397.

Wright, B. D. & Stone, M. H. (1979),Best test design . Chicago: MESA Press.

Accepted April, 2002.

Appendix

Test of Algebra Linear Equations

ItemI1: 2(x+ 3) = x- 10

ItemI2: 5x- 3 = 7 + 4x

ItemI3: 2(x- n) = 5ItemI

4: 5(2x- 5x) = 20x

ItemI5: 3x/7 = 10

Item I6: nx- a = 5a

ItemI7: 4[x+ 3(x- 2)] = 10

ItemI8: -7 +x= 10

ItemI9: 20 = 5x- 5x

ItemI10

: 2x+ 7 - 10x+ 3 = 12 - 2x

ItemI11: 4x/5 + 2 + 2x/3 - 10 = 0ItemI

12: -5(8 - 2x) = 2x- 2


23/23


ItemI13

: 5 - 2(x+ 3) =x+ 5(2x- 1) + 10

ItemI14

:x- 4(5x- 4) + 5x= 10 - n + 2n

ItemI15

: 6(x- 4) + 2x = 5x - 10

Production rules (PRs) required for the correct solution of the algebraequations:

PR1: Solve for variable with only numeric coefficients

Example: If 5x= 20 thenx= 20/5

PR2: Solve for variable with nonnumerical coefficients

Example: Ifnx= a + 5 thenx= (a + 5)/n

PR3: Collecting numbers

Example: If 2x= 5 - 10 then 2x= -5PR4: Collecting two terms

Example: If 2x- 5x= 10 then -3x= 10

PR5: Collecting more than two terms:

Example: If 2x- 5x+ 8x= 10 then 5x= 10

PR6: Balancing

Example: If 5 + 2x= 3 then 2x= 3 - 5

PR7: Removing parentheses with a positive coefficient

Example: If 5(2x+ 3) = 5 then 10x+ 15 = 5PR8: Removing parentheses with a negative coefficient

Example: If - 5(2x+ 3) = 5 then -10x- 15 = 5

PR9: Removing brackets:

Example: If 4[x+ 3(x- 2)] = 10 then 16x- 24 = 10

PR10: Removing denominators

Example: If 3x/7 = 1/2 then 6x= 7

Note. The examination of items and PRs shows that (a) PR2 subsumes PR1,

(b) PR4 subsumes PR3, (c) PR5 subsumes PR4, and (d) PR8 subsumes PR7.

For example, PR4 subsumes PR3 because collecting terms (PR4) includes

also collecting numbers (PR3) that represent the coefficients of the terms

involved in PR4. Therefore, when an item requires PR4, this item

automatically requires PR3 (see Table 1).

validation of cognitive structures_ sem (struct eq model) approach

Documents