multidimensional reliability: a proposal with examples
TRANSCRIPT
![Page 1: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/1.jpg)
Multidimensional Reliability:
A Proposal with Examples
Joseph F. Lucke
Research Institute on Addictions
State University of New York at Bu�alo, NY
Modern Modeling Methods Conference
University of Connecticut
May 23, 2017
Version: 2017.6.26.705
![Page 2: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/2.jpg)
Motivation
� Interest in congeneric measurement models (linear latent
variable models with normal distribution theory).
� Interest in multidimensional (nonhomogeneous,heterogeneous)
models.
� Investigate multidimensional reliability.
� Numerous previous investigators, but main contributors are
(Heise & Bohrnstedt, 1970; McDonald, 1999; Bentler, 2007)
� Reliability will be expressed as McDonald's !.� Goal here is to �nd an expression for the reliabilities of
individual factors.
![Page 3: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/3.jpg)
Motivation
� Interest in congeneric measurement models (linear latent
variable models with normal distribution theory).
� Interest in multidimensional (nonhomogeneous,heterogeneous)
models.
� Investigate multidimensional reliability.
� Numerous previous investigators, but main contributors are
(Heise & Bohrnstedt, 1970; McDonald, 1999; Bentler, 2007)
� Reliability will be expressed as McDonald's !.� Goal here is to �nd an expression for the reliabilities of
individual factors.
![Page 4: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/4.jpg)
Motivation
� Interest in congeneric measurement models (linear latent
variable models with normal distribution theory).
� Interest in multidimensional (nonhomogeneous,heterogeneous)
models.
� Investigate multidimensional reliability.
� Numerous previous investigators, but main contributors are
(Heise & Bohrnstedt, 1970; McDonald, 1999; Bentler, 2007)
� Reliability will be expressed as McDonald's !.� Goal here is to �nd an expression for the reliabilities of
individual factors.
![Page 5: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/5.jpg)
Motivation
� Interest in congeneric measurement models (linear latent
variable models with normal distribution theory).
� Interest in multidimensional (nonhomogeneous,heterogeneous)
models.
� Investigate multidimensional reliability.
� Numerous previous investigators, but main contributors are
(Heise & Bohrnstedt, 1970; McDonald, 1999; Bentler, 2007)
� Reliability will be expressed as McDonald's !.� Goal here is to �nd an expression for the reliabilities of
individual factors.
![Page 6: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/6.jpg)
Motivation
� Interest in congeneric measurement models (linear latent
variable models with normal distribution theory).
� Interest in multidimensional (nonhomogeneous,heterogeneous)
models.
� Investigate multidimensional reliability.
� Numerous previous investigators, but main contributors are
(Heise & Bohrnstedt, 1970; McDonald, 1999; Bentler, 2007)
� Reliability will be expressed as McDonald's !.
� Goal here is to �nd an expression for the reliabilities of
individual factors.
![Page 7: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/7.jpg)
Motivation
� Interest in congeneric measurement models (linear latent
variable models with normal distribution theory).
� Interest in multidimensional (nonhomogeneous,heterogeneous)
models.
� Investigate multidimensional reliability.
� Numerous previous investigators, but main contributors are
(Heise & Bohrnstedt, 1970; McDonald, 1999; Bentler, 2007)
� Reliability will be expressed as McDonald's !.� Goal here is to �nd an expression for the reliabilities of
individual factors.
![Page 8: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/8.jpg)
Review
� The congeneric model for a p-item test:
yp×1
= �p×1
+ �p×m
�m×1
+ �p×1;
� ∼ N(0, m×m
); and � ∼ N(0, �p×p).
� Consider the unit-weighted sum-score Y = 1′y.� var(Y ) = 1′(��′ +�)1� Multidimensional reliability is given by McDonald's
! = 1′��′1var(Y )
� Interested in cases where m > 1.
![Page 9: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/9.jpg)
Source for Examples
� AED: Alcoholic Energy Drink Expectancies Scale (Miller et al.,
In press).
� p = 15 Likert items. Range: 1�6.
� N = 3064
� Maximum Likelihood Estimation:� lavaan (Rosseel, 2012) in R� Mplus (Muthén & Muthén, 2017).
� MLE:
!̂ = !(�̂, ̂, �̂).
![Page 10: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/10.jpg)
Two-Dimensional Scale
To �x ideas I begin with
� Two-factor measurement model
� 5 of 15 items cross-load on both factors
� Factors are correlated
![Page 11: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/11.jpg)
Two-Dimensional Model
![Page 12: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/12.jpg)
Two-dimensional Reliability
! =1′[
�1 11�′1 + �1 12�′2 + �2 21�
′1 + �2 22�
′2]
1var Y
= 0.935
De�ne subscale reliabilities as
!1 =1′[
�1 11�′1 + �1 12�′2]
1var(Y )
= 0.266
!2 =1′[
�2 22�′2 + �2 21�′1]
1var(Y )
= 0.670
So that
! = !1 + !2.
![Page 13: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/13.jpg)
Two-dimensional Reliability
! =1′[
�1 11�′1 + �1 12�′2 + �2 21�
′1 + �2 22�
′2
]
1
var Y= 0.935
De�ne subscale reliabilities as
!1 =1′[
�1 11�′1 + �1 12�′2
]
1
var(Y )= 0.266
!2 =1′[
�2 22�′2 + �2 21�′1]
1var(Y )
= 0.670
So that
! = !1 + !2.
![Page 14: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/14.jpg)
Two-dimensional Reliability
! =1′[
�1 11�′1 + �1 12�′2 + �2 21�′1 + �2 22�
′2
]
1
var Y= 0.935
De�ne subscale reliabilities as
!1 =1′[
�1 11�′1 + �1 12�′2]
1var(Y )
= 0.266
!2 =1′[
�2 22�′2 + �2 21�′1
]
1
var(Y )= 0.670
So that
! = !1 + !2.
![Page 15: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/15.jpg)
Multidimensional Scale
Now consider a 4-factor scale.
� 4-factor measurement model
� Items form a perfect cluster con�guration
� Factors are correlated
![Page 16: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/16.jpg)
Multidimensional Model
![Page 17: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/17.jpg)
Multidimensional Reliability
! = 1′��′1var(Y )
= 0.954
De�ne the (general) subscale reliability for factor k as
!k =1′�k ′k�
′1var(Y )
.
Then for this 4-dimensional model
!1 =1′�1 ′1�
′1var(Y )
= .307; and !2 =1′�2 ′2�
′1var(Y )
= .374;
!3 =1′�3 ′3�
′1var(Y )
= .209; and !4 =1′�4 ′4�
′1var(Y )
= .063.
![Page 18: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/18.jpg)
Detail on Subscale Reliability
The subscale reliability for factor k as can be further explicated as
!k =1′�k ′k�
′1var(Y )
=1′�k kk�′k1 +
∑
j≠k 1′�k kj�′j1
var(Y )= standard reliability + sum of all �cross-reliabilities� .
� If factor k is uncorrelated with the other factors, then !kreduces to standard reliability.
� Otherwise, !k incorporates all the correlations between the
loadings on factor k and the loadings on the remaining factors.
![Page 19: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/19.jpg)
Hierarchical or Bifactor Scale
Now let's consider the hierarchical or bifactor model.
� One general factor for all 15 items
� 4 correlated speci�c factors
� General factor is uncorrelated with speci�c factors
� Items form a perfect cluster con�guration on each speci�c
factor
This example also demonstrates that a the reliability of a subset of
factors, as opposed that for a single factor, can also be obtained.
![Page 20: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/20.jpg)
Bifactor Model
![Page 21: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/21.jpg)
Bifactor Reliability
The multidimensional reliability is
! = 1′��′1var(Y )
= .957
The subscale reliability for the general factor is
!1 =1′�1 ′1�
′1var(Y )
=1′�1 11�′11var(Y )
= .679.
The multidimensional subscale reliability for the speci�c factors is
!2∶5 =1′�2∶52∶5,2∶5�′2∶51
var(Y )= .278.
The speci�c subscale reliabilities may be obtained as in the
multidimensional case.
![Page 22: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/22.jpg)
Extended Congeneric Model
The extended congeneric measurement model is required for
higher-order models. The extended model is
y = � + �� + � with � = � + B� + � .
Thus, var(Y ) = 1′{
�(I − B)−1[(I − B)−1]′�′ +�}
1.
Bentler's extension to ! is
! =1′�(I − B)−1[(I − B)−1]′�′1
var(Y ).
The MLE estimate is now
!̂ = !(�̂, B̂, ̂, �̂).
![Page 23: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/23.jpg)
Second-order Measurement Model
Now consider a second-order measurement model.
� Items form a perfect cluster con�guration on each of 4 factors
� A second-order factor accounts for the correlation among the 4
�rst-order factors.
� �Direct� reliability among �rst-order factors
� �Indirect� reliability for second-order factor
![Page 24: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/24.jpg)
2nd Order Measurement Model
![Page 25: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/25.jpg)
2nd-order Multidimensional Reliability
! =1′�(I − B)−1(I − B)−1]′�′1
var(Y )= 0.954
Let �̃ = �(I − B)−1. De�ne subscale reliability for factor k as
!k =1′�̃k ′k�̃
′1var(Y )
.
Then
!1 = .028; !2 = .039; !3 = .008; !4 = .025;!5 = .854.
![Page 26: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/26.jpg)
2nd-order Multidimensional Reliability, cont'd
Because of the presence of (I − B)−1 in the expression, the
interpretation of the subscale reliabilities may be obscure. However,
in this case,
(I − B)−1 = I + B.
Thus
! =1′�(I + B)(I + B)′�′1
var(Y )= 0.954
De�ne the layer reliabilities as
Ω1 =1′��′1var(Y )
= !1 + !2 + !3 + !4 = .100;
Ω2 =1′�BB′�′1var(Y )
= !5 = .854.
![Page 27: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/27.jpg)
Third-order Measurement Model
Now consider more complicated third-order scale.
� Items form a perfect cluster con�guration on each of 4 factors
� Two �rst-order factors are accounted for by a second-order
factor.
� Other two �rst-order factors are accounted for by another
second-order factor.
� The two second-order factors are accounted for by a
third-order factor.
![Page 28: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/28.jpg)
3rd-Order Measurement Model
![Page 29: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/29.jpg)
3rd-order Multidimensional Reliability
! =1′�(I − B)−1[(I − B)−1]′�′1
var(Y )= 0.954
De�ne subscale reliabilities as before
!k =1′�̃k ′k�̃
′1var(Y )
.
Then
!1 = .028; !2 = .028; !3 = .008; !4 = .024;!5 = .008; !6 = .008;
!7 = .848
![Page 30: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/30.jpg)
3rd-order Multidimensional Reliability, cont'd
Again the presence of (I − B)−1 obscures the interpretation of the
subscale reliabilities. However, in this case,
(I − B)−1 = I + B + B2.
The layer reliabilities are
Ω1 =1′��′1var(Y )
= !1 + !2 + !3 + !4 = .089;
Ω2 =1′�BB′�′1var(Y )
= !5 + !6 = .017.
Ω3 =1′�B2B2′�′1
var(Y )= !7 = .848.
![Page 31: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/31.jpg)
Summary
For the extended congeneric measurement model, multidimensional
reliability for unit-weighted sum scores is McDonald-Bentler's !
! =1′�(I − B)−1
[
(I − B)−1]′ �′1
var(Y ).
Letting �̃ = �(I − B)−1, the subscale reliability for factor k(k = 1,… , m) is
!k =1′�̃k ′k�̃
′1var(Y )
.
The layer reliability for level r (r = 1,… , s and B0 = I) of ans-order recursive model is
Ωr =1′�Br−1
[
Br−1]′ �′1
var(Y ).
![Page 32: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/32.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 33: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/33.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 34: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/34.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 35: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/35.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 36: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/36.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 37: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/37.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 38: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/38.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 39: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/39.jpg)
Summary
� Natural de�nition
� Consistent method for decomposing multidimensional reliability
� Uses all the information in multidimensional reliability
� Applies to congeneric and extended congeneric models
� Subscale reliability is general
� Level reliability requires recursive higher-order model.
� Assumes all loadings are non-negative�Negative loadings can
yield negative subscale reliability!?!
� There is a parallel development for internal consistency, �.
![Page 40: Multidimensional Reliability: A Proposal with Examples](https://reader030.vdocument.in/reader030/viewer/2022012708/61a88ecfd9981235a163da67/html5/thumbnails/40.jpg)
References
Bentler, P. M. (2007). Covariance structure models for maximal reliability of unit-weighted composites.In S.-Y. Lee (Ed.), Handbook of latent variable and related models (Vol. 1). Amsterdam:Elsevier. doi: 10.1016/b978-0-444-52044-9.x5000-9
Heise, D. R., & Bohrnstedt, G. W. (1970). Validity, invalidity, and reliability. In E. F. Borgatta &G. W. Bohrnstedt (Eds.), Sociological methodology 1970 (Vol. 2, pp. 104�129). SanFrancisco, CA: Jossey-Bass. Retrieved from StableURL:http://links.jstor.org/sici?sici=0081-1750%281970%292%3C104%3AVIAR%3E2.0.CO%3B2-J
McDonald, R. P. (1999). Test theory: A uni�ed approach. Mahwah, NJ: Lawrence Erlbaum Associates.Muthén, L. K., & Muthén, B. O. (2017). Mplus user's guide: statistical analysis with latent variables
(Vol. 8). Los Angeles, CA: Muthén & Muthén.Rosseel, Y. (2012, May 24). lavaan: An R package for structural equation modeling. Journal of
Statistical Software, 48(2), 1�36. Retrieved from http://www.jstatsoft.org/v48/i02