asvab: e pluribus unum? martin j. ippel, ph.d. cognimetrics inc, san antonio,tx steven e. watson,...
TRANSCRIPT
![Page 1: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/1.jpg)
ASVAB: E Pluribus Unum?
Martin J. Ippel, Ph.D.CogniMetrics Inc,San Antonio,TX
Steven E. Watson, Ph.D.U.S. Navy Selection & Classification (CNO 132)
Washington, DC
1
![Page 2: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/2.jpg)
The ASVAB is the principal instrument for selection and classification in the U.S. Armed Forces.
Assumption: measurement invariance acrossfull range of scores.
Relevance: what is the “population of interest”of the ASVAB?
Recent studies cast doubt on this assumption.
2
![Page 3: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/3.jpg)
Two related phenomena suggest a changing factor structure along the dimension of general intelligence (g):
• The g factor gets smaller in high-g samples
• Cognition tests have smaller loadings on “g” in high-g samples
3
![Page 4: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/4.jpg)
Spearman (1927) noticed already a decrease in the positive manifold of cognition variables at higher g levels.
differentiation of intelligence
Spearman’s explanation:
4
![Page 5: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/5.jpg)
5
The present study adheres to an alternative explanation:
The phenomenon follows from the Pearson-Lawley selection rules.
an underlying selection processchanges the variance-covariancestructure and the mean structure
![Page 6: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/6.jpg)
6
One phenomenon:
Decrease in positive manifold of cognition variables in high-g samples
Two explanations:
differentiation of intelligence
selection effects
![Page 7: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/7.jpg)
Consequences of:
• differentiation:
• selection effects:
structure is changing
underlying structureinvariant
7
![Page 8: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/8.jpg)
8
Critical developments in psychometric theory:
• Meredith (1964) showed that both the covariance structure and mean structure change if samples are selected based on one or more latent variables (e.g., the g factor).
• Meredith (1965) developed procedures to derive the single best fitting (i.e., invariant) factor pattern derived from sets of factors obtained on populations differing on a latent variable.
• Jöreskog (1971) formalized this viewpoint as an extension of the common factor model for a parent population to multiple groups based on one or more latent variables in the model.
![Page 9: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/9.jpg)
(df) Measurement Invariance:
If we compare groups, or individuals of different groups, then the expected value of test scores of a person of a given level of ability should be independent of membership of these groups (Mellenbergh, 1989).
In formule:
f (Y | η, ν) = f (Y | η)
y1ij = τ1i + λ1i ηij + ε1ij
f depends on the measurement model of choice:
9
![Page 10: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/10.jpg)
10
y1ij = τ1i + λ1i ηij + ε1ij
change
invariant
![Page 11: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/11.jpg)
ratings cluster 1
ratings cluster 2
η11 η12
Unequal intercepts
11
![Page 12: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/12.jpg)
ratings cluster 1
ratings cluster 2
η12
η11
Unequal factor loadings
12
![Page 13: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/13.jpg)
13
parentpopulation(N = 48,222)
a-selectsample(n=1,000)
hi-gsample(n=600)
av-gsample(n=600)
lo-gsample(n=600)
StatisticalExperiment:
![Page 14: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/14.jpg)
14
parentpopulation(N = 48,222)
a-selectsample(n=1,000)
hi-gsample(n=600)
av-gsample(n=600)
lo-gsample(n=600)
StatisticalExperiment:
determine factorstructure and thensample
![Page 15: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/15.jpg)
eigenvalue database 1 2 3 4 5 61 3.93 3.971 3.876 2.475 3.888 3.811 4.208
2 1.328 1.324 1.314 1.241 1.337 1.386 1.212
3 1.039 1.008 1.061 1.164 1.077 1.039 0.974
4 0.7 0.701 0.713 0.922 0.702 0.689 0.683
N= 48222 1000 1000 1000 966 954 1015
samples
Eigenvalues from a-select samples drawn from the parent population of Air Force recruits
15
![Page 16: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/16.jpg)
General Science (GS): a 25 items knowledge test of physical and biological
sciences.
Arithmetic Reasoning (AR): a 30 items arithmetic word problem test.
Word Knowledge (WK): 35 items testing knowledge of words and synonyms.
Paragraph Comprehension (PC): 15 items testing the ability to extract meaning from short paragraphs.
Auto and Shop I nformation (AS): a 25 items knowledge test of automobiles, shop practices, tools and tool use.
Mathematical Knowledge (MK): a 25 items test of algebra, geometry, fractions, decimals, and exponents.
Mechanical Comprehension (MC): a 25 items test of mechanical and physical principles and ability to visualize how illustrated objects work.
Electronics I nformation (EI ): a 20 items test measuring knowledge about electronics, radio, and electrical principles.
Assembling Objects (AO): a 16 items spatial visualization test.
ASVAB tests and their measurement claims
16
![Page 17: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/17.jpg)
0
QUANTI-TATIVE
MK
0,
e2
1
1
AR
0,
e1
1
0
VERBAL
PC
0,
e4
WK
0,
e3
1
11
0
TECHNICALKNOWLEDGE
MC
0,
e7
GS
0,
e6
1
11
AS
0,
e8
1
EI
0,
e9
1
0,
GENERALINTELLI-GENCE
0,
e11
1
0,
e10
1
0,
e12
1
AO
0,
e5
1
1
Model 1: A hierarchical model of “g”17
![Page 18: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/18.jpg)
0, v
QUANTI-TATIVE
MK
0,
e2
1
1
AR
0,
v1
1
0, v
VERBAL
PC
0,
e4
WK
0,
e3
1
1
1
0, v
TECHNICALKNOWLEDGE
MC
0,
e7
GS
0,
e6
1
11
AS
0,
e8
1
EI
0,
e9
1
AO
0,
e5
1
0,
GENERALINTELLIGENCE
1
Model 2: A “g as first principal factor” model
18
![Page 19: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/19.jpg)
19
parentpopulation(N = 48,222)
a-selectsample(n=1,000)
hi-gsample(n=600)
av-gsample(n=600)
lo-gsample(n=600)
StatisticalExperiment:
determine factorstructure andsample
![Page 20: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/20.jpg)
20
sample N m s.d.skewnes
skurtosi
s
random 1000 41.03 3.7 0.04 -0.68
g-hi 600 44.76 1.73 0.12 -0.11
g-av 600 41.19 1.73 0.12 -0.11
g-lo 600 37.61 1.73 0.12 -0.11
Distributional properties of samples generated from the parent population based on a latent
variable "g"
![Page 21: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/21.jpg)
45.0
47.0
49.0
51.0
53.0
55.0
57.0
59.0
61.0
63.0
MK AR WK PC GS MC AS EI AO
ASVAB tests
mea
n t
est
sco
res
random
g-hi
g-av
g-lo
ASVAB tests mean scores in samples with different levels of "g"
21
![Page 22: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/22.jpg)
y = -0.4812x + 0.3567
R2 = 0.9388
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
-10.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0%
Reduction Variance
Ave
rag
e C
orr
elat
ion
22
Average correlation lower with lower variance
![Page 23: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/23.jpg)
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
MK AR WK PC GS MC AS EI AO
ASVAB tests
per
cen
tag
e va
rian
ce r
edu
ctio
n
g-lo
g-av
g-hi
23
The effects of selection based on the latent variable “g”on the variance of ASVAB tests
![Page 24: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/24.jpg)
df par X2 X2 diff. sign. CMI N/ DFRMSEAPCLOSE NFI CFI
109 53 3106.26 - 28.498 0.124 0 0.959 0.961106 56 709.27 2396.99 p < 0.001 6.691 0.056 0.004 0.991 0.99282 80 254.29 454.97 p < 0.001 3.101 0.034 1 0.997 0.99874 88 209.08 45.22 p < 0.0012.825 0.032 1 0.997 0.998
3. RESVAR free model4. Model 3 & some intercepts free
model
1. Full MI model2. RESVAR group-invariant model
Sequence of MCFA model fits and goodness of fit indices
24
![Page 25: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/25.jpg)
Non-standardized MCFA factor loadings.
parent parentestimate estimate s.e. sign. estimate estimate s.e. sign. estimate estimate s.e. sign. estimate estimate s.e. sign.
WK 0.543 0.501 0.115 0 1 1 0 0PC 0.675 0.484 0.044 0 0.461 0.484 0.044 0
MK 1.039 -0.317 0.162 0.05 1 1 0 0AR 1.338 0.453 0.11 0 0.243 0.39 0.063 0
GS 1.093 0.339 0.145 0.02 0.673 0.839 0.073 0 0.335 0.328 0.05 0MC 1.346 0.941 0.261 0 1 1 0 0AS 0.715 0.568 0.249 0.02 1.497 1.8 0.115 0EI 0.96 0.113 0.175 0.52 0.257 0.42 0.061 0 0.962 1.059 0.087 0
AO 1 1 0 0 -0.207 -0.259 0.092 0
mcfa mcfa mcfa mcfaG Verbal Quantitative Technical Knowledge
25
![Page 26: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/26.jpg)
Standardized MCFA factor loadings
aselect high average low aselect high average low aselect high average low aselect high average low aselect high average low
gverbal 0 0 0 0quant. 0 0 0 0 -0.251 -0.003 -0.103 -0.571
TK 0 0 0 0 0.18 0.158 0.157 0.21 -0.054 -0.43 -0.505 -0.573
WK 0.468 0.306 0.24 0.254 0.798 0.841 0.808 0.817 0.856 0.802 0.71 0.732PC 0.482 0.269 0.199 0.212 0.304 0.37 0.335 0.342 0.325 0.393 0.433 0.435
MK 0.656 -0.158 -0.117 -0.137 0.618 0.934 0.832 0.603 0.772 0.897 0.706 0.382AR 0.766 0.242 0.187 0.224 0.129 0.389 0.361 0.269 0.603 0.21 0.165 0.122
GS 0.633 0.154 0.126 0.133 0.361 0.523 0.526 0.529 0.179 0.185 0.22 0.2 0.586 0.373 0.378 0.382MC 0.673 0.382 0.301 0.325 0.463 0.506 0.577 0.539 0.667 0.402 0.423 0.396AS 0.379 0.197 0.146 0.172 0.733 0.779 0.835 0.85 0.681 0.646 0.718 0.751EI 0.528 0.046 0.035 0.039 0.131 0.232 0.217 0.232 0.49 0.532 0.587 0.569 0.559 0.393 0.433 0.435
AO 0.531 0.431 0.333 0.324 -0.102 -0.154 -0.145 -0.135 0.292 0.21 0.132 0.123
Table 6. Standardized MCFA factor loadings.
G Verbal Quantitative Technical Knowledge Communality
26
![Page 27: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/27.jpg)
27
y1ij = τ1i + λ1i ηij + ε1ij
change
should remain invariant
![Page 28: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/28.jpg)
28
y1ij = τ1i + λ1i ηij + ε1ij
change
not invariant
![Page 29: ASVAB: E Pluribus Unum? Martin J. Ippel, Ph.D. CogniMetrics Inc, San Antonio,TX Steven E. Watson, Ph.D. U.S. Navy Selection & Classification (CNO 132)](https://reader037.vdocument.in/reader037/viewer/2022110322/56649d145503460f949e8ccc/html5/thumbnails/29.jpg)
29
Discussion:
• ASVAB is measurement invariant in a limited sense: only factor loadings are invariant across different levels of “g”. (weak factorial invariance).
• ASVAB seems to be measuring too many factors with too few tests.
• more factors than eigenvalues larger than 1.• many tests have communalities < 0.60.• intercepts could not be constrained to be equal
(indicating: other factors influence test scores).