Maximal Reliability of Unit-Weighted Composites

Peter M. BentlerUniversity of California, Los Angeles

SAMSI, NOV 05 In S. Y. Lee (Ed.) (in press). Handbook of Structural Equation Models.

“Reliability” = Internal Consistency of the composite X based on p components

1 1 2 2 p pX=w Y +w Y +...+w Y (X=w Y)

(all 1); , uncorrelatedY C U p C U

ccov( ) ( psd, rank<p; diag or pd)cY

1cxx

w w w ww w w w

(Bentler, 1968, eq. 12; Heise & Bohrnstedt, 1970, eq. 32;this is H-B's coefficient - special case is McDonald's 1999 )

c

is a not necessarily a measure of homogeneityor unidimensionality. A coefficient for uni-dimensional common scores requires, in addition, that rank( ) 1.What , defined below, has to say (or, no

xx

t say)about unidimensionality has been an issue widely discussed in the literature.

Since uniqueness = specificity + error, i.e.,. If both are p.d.,s e

*

*

is a lower bound to "true" reliability, where

1

xx xx

exx

w ww w

In this talk I will emphasize, the unit vector.

This excludes maximal reliability.w 1

Under mild assumptions consistent with the above, Novick & Lewis (1967) showed that

xx

'1

1 '

where, with diag( ) and a unit vector,

is the well-known alpha coefficient. This saysnothing about as a lower bound to a uni-dimensional component of a multidimensional

1 D 1pp 1 1

D 1

c.

cWhat models for are consistent with ?

Let be the average covariance, and defineij

=( ), where =diag( ). ThenijD I D

. It follows thatc ijD I

.xx

cThis choice does not guarantee that hasany interesting properties, e.g., psd, rank 1.

This is not the only condition for .xx

If , where are communalities, we only need

c H

H

D DD

c

( ) /

Interesting seem not to be known.Another well known case of equivalence for a rank 1 is given below.

H ij

H

1 D 1 p

D

cActually, I will argue that should be psd,which is not guaranteed for arbitrary .HD

This Hoyt-Guttman-Cronbach α is by far themost widely used measure of the internal con-sistency reliability of a composite X. In prac-tice, one substitutes the sample covariancematrix S and its diagonal DS to get

'ˆ 1 .1 '

This is not necessarily a lower bound topopulation internal consistency.

S1 D 1pp 1 S1

Some Recent References• Becker, G. (2000). Coefficient alpha: Some terminological ambiguities and

related misconceptions. Psychological Reports, 86, 365-372.• Bonett, D. G. (2003). Sample size requirement for testing and estimating

coefficient alpha. Journal of Educational and Behavioral Statistics, 27, 335-340. • Enders, C. K., & Bandalos, D. L. (1999). The effects of heterogeneous item

distributions on reliability. Applied Measurement in Education, 12, 133-150. • Green, S. B., & Hershberger, S. L. (2000). Correlated errors in true score models

and their effect on coefficient alpha. Structural Equation Modeling, 7, 251-270. • Hakstian, A. R., & Barchard, K. A. (2000). Toward more robust inferential

procedures for coefficient alpha under sampling of both subjects and conditions. Multivariate Behavioral Research, 35, 427-456.

• Kano, Y., & Azuma, Y. (2003). Use of SEM programs to precisely measure scale reliability. In H. Yanai, A. Okada, K. Shigemasu, Y. Kano, & J. J. Meulman (Eds.), New developments in psychometrics (pp. 141-148). Tokyo: Springer-Verlag.

• Komaroff, E. (1997). Effect of simultaneous violations of essential tau-equivalence and uncorrelated error on coefficient alpha. Applied Psychological Measurement, 21, 337-348.

• Miller, M. B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, 2, 255-273.

• Raykov, T. (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items. Applied Psychological Measurement, 22, 375-385.

• Raykov, T. (2001). Bias of coefficient α for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69-76.

• Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.

• Shevlin, M., Miles, J. N. V., Davies, M. N. O., & Walker, S. (2000). Coefficient alpha: A useful indicator of reliability? Personality & Individual Differences, 28, 229-237.

• Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.

• Yuan, K.-H., Guarnaccia, C. A., & Hayslip, B. J. (2003). A study of the distribution of sample coefficient alpha with the Hopkins Symptom Checklist: Bootstrap versus asymptotics. Educational & Psychological Measurement, 63, 5-23.

Some Advantages of • Widely taught and known• Simple to compute and explain• Available in most computer packages is a lower bound to population internal

consistency under reasonable conditions (not true of sample )

• Not reliant on researcher judgments (i.e., same data, same result for everyone)

Note: Independence of researcher judgment is not totally correct

• Different covariance matrices, or scalings of the variables, yield different

and hence different • Sometimes alpha is computed from the

correlation matrix, and not the covariance matrix, implying the sum X is a sum of standardized variables

Some Disadvantages of • α is not a measure of homogeneity or

unidimensionality• It may underestimate or overestimate uni- dimensional reliability• Since it will overestimate reliability if , the average covariance, is spuriously high;

or underestimate it if the average is spuriously low

2 /ijp 1 1 ij

This illustrates the advantages and disadvantages of alpha

• The plus: The average covariance is what it is, period. No judgment can change it. Hence, no judgment will change the reported .

• The minus: A researcher’s model of sources of variance does not influence the reported reliability (alpha) when, perhaps, it should.

• Some examples from Kano/Azuma (2003) illustrate this.

If for a one factor model and possibly correlated residuals ( not necessarily diagonal)

with (p 1), then c

Three examples from Kano & Azuma (2003) showing alpha as accurate, and as an overestimate. 1-factor reliability (rho= omega) ignoring correlated errors can overestimate.

Alpha as a unidimensional lower-bound: the comparison of in a 1-factor model If , and is px1, is diagonal, . Example: McDonald (1999)

showed

with equality when

11,

11

111( ) { ( )} 0

1 ip 1 1 Var

p

c1

11

11

11a 11b 11c

If ,there is no known general relation of to .In fact, (= ) is itself not defined. We canhave , , etc., depending on how the 1-factor model is computed (ML,LS,GLS,etc.)[Th

11

at is, how ( ) is fit to under thismisspecification. No sample data are involved.]This nonuniqueness of under misspecificationis a weakness of this model-based coefficient.

Thus,• α is a good indicator of association but it is not clear what interesting sets of

models it represents in the general case• A model based coefficient such as provides a clear partitioning of variance, but it also can be (1) misleading (e.g., Kano -Azuma), (2) not relevant

( )ij

11( )

(e.g., )

To give up α for a model-based coefficient, the model must be correct for Σ. That is, it should fit the data (say, sample covariance matrix S). I would conclude:

• If is computed, the researcher must also provide evidence of acceptability of the 1-factor model.

• If a modified estimator is computed, the researcher should provide

an argument for the variance partition.

11

11ˆ( with nondiagonal )

Should the correlated error be part of the residual covariance Ψ (on left), or part of the common variance Σc (on right)? Substantive reasoning should determine the variance partitioning. This is more than just model fit.

What other model-based coefficients could be used instead?

• Arbitrary latent variable model (Raykov & Shrout, 2002; EQS 6)

• Dimension-free lower bound (Bentler, 1972; bias correction Shapiro & ten Berge, 2000)

• Greatest lower bound (Woodhouse & Jackson, 1977; Bentler & Woodward, 1980 etc.; ten Berge, Snijders, & Zegers, 1981; bias correction Li & Bentler, 2001)

Arbitrary LV Model

c

model c

Suppose an arbitrary SEM (e.g., a LISREL model) contains additive errors, whether correlated or not. That is, = ( )= . It must be acceptable statistically (fit ). Then

( ) /( )is a

S1 1 1' 1

11 more meaningful coefficient than (= ).It is probably more meaningful than .

Dimension-free Lower Bound

+

*11 +

for some arbitrary, unknown number of factors

is diagonal, and

min subject to above.

This has the property that ( , )

c

c

cxx

xx

1 11 1

Greatest Lower Bound This is a constrained version of the

dimension-free coefficient. In addition to

glb

+

*glb

for some number of factors, is not only diagonal, but also psd. Thus

,

with equality when has no Heywoodvariables (no negative variances).

Also, .xx

+ glb

Every researcher will get the same values of and . They are based on a model that

does not depend on researcher choices.Since off-diag( ) [off-diag( ), as replaces

in practice] is exactly repS S

+ glb

roduced, all covariation is assumed to stem from common variance. However, and do not allow

nondiagonal .

Maximal Unidimensional Reliability

• The problem with α and the multi-dimensional coefficients seems to be that they do not represent unidimensional reliability

• Although not obvious, unidimensional reliability can be defined for multi-

dimensional latent variable models. That is the main new result in this talk.

T E

1,p

iX X 1

piT T 1

piE E

2

2 1T T E

Xxx

1 1 1 11 1 1 1

Repeating the Basic Setup

Xi = Ti + Ei,

X = T + E ,

2 221

2

but we compute something like

( )( ) 1d

pT i u

X11

1 11 1 11 1 1 1 1 1 1 1

0. ., truth is ui e

Can we have something like1-Factor Based Reliability when the latent variables are multidimensional?

Maximal Unit-weighted Reliability x (p x k) for some k

(“small k” <

or “large k”)

.5(2 1 8 1)p p

for some acceptable k-factor model

[ | ] , where is (px1) and is (px(k-1))

contains unrestricted free parameters.01 , that is, the k-1 columns of sum to zero.

contains free parameters subject to (k-1)(k-2)/2 restrictions (usual EFA identification conditions)

Reliability under this Parameterization

X 1 x 1 1

1[ | ] [ | ] [ | 0] [ | 0]

[ | 0]

pi

x

1 1 1 1 1

1X X d dX T E X is based on 1 factor! 2 2

2 2 2d

X

T X

X Xkk

2

2

( )( ) ( )

1 1 11 1 1 1 1

2

2 2

ˆˆ ˆ ˆ1ˆ ˆ ˆ ˆ

ˆX

X

Xkk

1 1 1 11 1 1 1

This is Maximal Unit-Weighted Reliability

Let and let t be a normal vector ( 1)t t

Then the factor loading vector t that maximizes 2( )1

is given by 1/ 2( )1 1 1 and the residual factors

where have zero column sums

( 0).1

Applications to Arbitrary Structural Models

Any structural model with additive errors: ( )

Linear structural model with additive errors:

( )

Let 1 1( ) ( )I B I B with 1 1/ 2( )I B

Greatest lower bound:

1 max min 1 ,

with ( ) and

glb1 1 1 11 1 1 1

psd

ˆ ˆ ˆS

EFA Example (It’s all in EQS)/TITLEMaximum Reliability EFA Model SetupNine Psychological Variables/SPECIFICATIONSVARIABLES= 9; CASES=101;MATRIX=COVARIANCE; METHOD=ML;/EQUATIONSV1=*F1+*F2+0F3+E1;V2=*F1+*F2+*F3+E2;V3=*F1+*F2+*F3+E3;V4=*F1+*F2+*F3+E4;V5=*F1+*F2+*F3+E5;V6=*F1+*F2+*F3+E6;V7=*F1+*F2+*F3+E7;V8=*F1+*F2+*F3+E8;V9=*F1+*F2+*F3+E9;

/VARIANCESF1 TO F3 = 1.0;E1 TO E9 = .5*;/CONSTRAINTS(V1,F2)+(V2,F2)+(V3,F2)+(V4,F2)+(V5,F2)+(V6,F2)+(V7,F2)+(V8,F2)+(V9,F2)=0;(V2,F3)+(V3,F3)+(V4,F3)+(V5,F3)+(V6,F3)+(V7,F3)+(V8,F3)+(V9,F3)=0;

/MATRIX1.00 .75 1.00 .78 .72 1.00 .44 .52 .47 1.00 .45 .53 .48 .82 1.00 .51 .58 .54 .82 .74 1.00 .21 .23 .28 .33 .37 .35 1.00 .30 .32 .37 .33 .36 .38 .45 1.00 .31 .30 .37 .31 .36 .38 .52 .67 1.00/END

for 9 Psychological Variables ˆ .886

Variable Number

1-FactorModel

3-FactorModel

1 .636 .727

2 .697 .738

3 .667 .754

4 .867 .789

5 .844 .767

6 .879 .803

7 .424 .492

8 .466 .597

9 .462 .635

Sum 5.942 6.302

227 190.6

ˆ .88011

212 1.6

ˆ .939kk