imta 2011 technical aptitude pres ii
TRANSCRIPT
Predicting Technical Aptitude: Relations between Predictor Variables,
Technical Aptitude and Technical Training Performance
(O.N.R. Contracts Nr. N00014-10-M-0087 & N00014-10-C-0505)
Ryan Glaze & Martin J. Ippel
CogniMetrics, Inc., San Antonio, TX
1
Paper presented at the 53rd Annual Conference of the International Military Testing Association, Bali (Indonesia).
October 31 – November 4, 2011
Preview
Analysis I
Overview of ASVAB and Technical Knowledge Tests
Introduction to Technical Aptitude
Proposed Model
Methods, Results, and Discussion
Analysis II
Incremental Validity Analysis Predicting Technical Aptitude with ASVAB Selection Composites and the ITAB
Methods, Results, and Discussion
2
ASVAB
US Armed Forces must select, classify, and train personnel to work in highly technical work environment
ASVAB and Technical Knowledge Subtests used for technical Navy Ratings
General Science (GS)
Mechanical Comprehension (MC)
Auto Shop (AS)
Electrical Information (EI)
3
Technical Knowledge Subtests
Technical Knowledge Subtests have several limitations:
Represent arbitrary and limited sample of domain
Measures Technical Knowledge but not Technical Skills
Provide modest predictive validity
4
Technical Aptitude
Technical Aptitude (Ippel & Glaze, 2011)
Technical Knowledge Aptitude (TKA) Aptitude to learn technical knowledge (concepts)
Derived from performance on eight common knowledge tests
Technical Skill Aptitude (TSA) Aptitude to learn technical skills
Derived from performance on seven common skill tests
5
Technical Aptitude
Technical Aptitude (TA)
Represents a construct with a short logical distance to criterion performance in Apprentice Technical Training (ATT) performance
Will be used to assess construct validity of TK subtests
6
7
GS
Post-Test
Analysis I
8
Post-Test
GS
Technical Aptitude
9
Technical Aptitude
AFQT
GS
Post-Test
10
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
11
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
12
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
13
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
14
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
15
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
Method
410 Navy Recruits participating in A.T.T. program
ASVAB
TK Subtests: GS, MC, AS, EI
AFQT Considered a measure of crystallized intelligence
A.T.T. post-training test scores
Eight common knowledge tests
Seven common skill tests
Dichotomously scored (Pass/Fail with 70 point cut score)
16
Method
Technical Knowledge Aptitude
IRT-based ability estimate derived from common eight knowledge tests
Technical Skill Aptitude
IRT-based ability estimate derived from common seven skill tests
17
Results: First Knowledge Test
18
RMSEA = 0.000 PClose = 0.886 WRMR = 0.040
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
Results: First Knowledge Test
19
.508*
.316* .189*
.226*
.084
.352*
Results: First Knowledge Test
20
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
.137 .130
.175 .530*
Results: First Knowledge Test
21
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
.217* .017 -.119 -.021
Results: All Knowledge Tests
22
MeanRMSEA = 0.000 MeanPClose = 0.868 MeanWRMR = 0.039
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
Results: All Knowledge Tests
The paths between Technical Knowledge Aptitude, Technical Knowledge Subtests, AFQT were nearly identical for all knowledge tests
Focus will be on:
AFQT → Post-test
TA → Post-test
23
Results: All Knowledge Tests
24
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
.530* .084
Results: All Knowledge Tests
AFQT was related to all Technical Knowledge Tests, but not post-training test scores
Technical Knowledge Aptitude was related to post-training test scores, but not Technical Knowledge test scores
Indirect effects of Technical Knowledge Aptitude on post-training test scores via TK subtests were not significant
25
Results: First Skill Test
26
RMSEA = 0.024 PClose = 0.477 WRMR = 0.187
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
Results: First Skill Test
27
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
.066
.373* .536* .179* .267*
Results: First Skill Test
28
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
.218*
.174* .032 .064
Results: First Skill Test
29
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
.076 .062 -.029 .062
Results: All Skill Tests
30
MeanRMSEA = 0.024 MeanPClose = 0.477 MeanWRMR = 0.187
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
Results: All Skill Tests
The paths between Technical Skill Aptitude, Technical Knowledge Subtests, AFQT were nearly identical for all skill tests
Focus will be on:
AFQT → Post-test
TA → Post-test
31
Results: All Skill Tests
32
Technical Aptitude
AFQT
GS MC AS EI
Post-Test
.218* .066
Results: All Skill Tests
AFQT was related to all Technical Knowledge Tests, but not post-training test scores
Technical Skill Aptitude was related to post-training test scores, but not Technical Knowledge test scores
Indirect effects of Technical Knowledge Aptitude on post-training test scores via TK subtests were not significant
33
Results: Technical Aptitude
Technical Knowledge Aptitude was only slightly related to Technical Skill Aptitude (r = .136)
Technical Aptitude was related to post-training test scores, but not Technical Knowledge test
34
Analysis II
Results of Analysis I suggest Technical Aptitude was related to post-training test scores, but Technical Knowledge was not
Analysis II seeks to identify predictors of Technical Aptitude
Current predictors of training performance consist of various ASVAB Selection Composites (ASC)
I.T. Aptitude Battery (ITAB) was designed to measure technical aptitude
35
Method
Two selection composites that the Navy currently uses to assign recruits to ratings
ASC01 consists of WK, PC, AR, and MC
ASC02 consists of MK, AR, GS, and EI
ITAB consists of two fully interactive tests
Hidden Target Test
Battery Test
36
Results
37
Variable 1. 2. 3. 4.
1. ITAB
2. ASC01 .33
3. ASC02 .31 .78
4. TSA .17 .16 .34
5. TKA .30 .44 .69 0.14
All Correlations significant at p < .01.
Results
38
Predictors Multiple R Incremental Validity
Selection
Composite Test ASC ITAB R2 F Sig ΔR2 %ΔR2 F sig
ASC01 TKA 0.383 0.174 0.22 57.66 p < .0001 0.027 13.99% 14.11 p < .001
ASC02 TKA 0.66 0.095 0.484 191.1 p < .0001 0.008 1.68% 6.504 p < .05
ASC01 TSA 0.117 0.132 0.041 8.703 p < .01 0.015 57.69% 6.574 p < .05
ASC02 TSA 0.318 0.071 0.12 27.81 p < .0001 0.005 4.35% 2.146 p > .05
Results
Selection composites (ASC01 and ASC02) significantly predicted Technical Knowledge Aptitude and Technical Skill Aptitude
Technical Knowledge Aptitude
ITAB provided incremental validity over selection composites for Technical Knowledge Aptitude
Technical Skill Aptitude
ITAB provided incremental validity over ASC01, but not ASC02, for Technical Skill Aptitude
39
Thank You
40