a quantitative model for unifying human factors with cognitive load theory
TRANSCRIPT
A Quantitative Model for Unifying Human Factors With Cognitive Load Theory
E M B RY- R I D D L E A E R O N AU T I C A L U N I V E R S I T Y
NATHAN A. SONNENFELD [email protected] R. KEEBLER, PH.D. [email protected]
Highlights• Cognitive Load Theory (CLT) is a useful foundation for applied HF
research in education and instructional design
MeasureMethodModel
However:1. CLT model (additive) not empirically supported2. CLT measures do not quantify theory constructs3. CLT methods lack ‘in vivo’ practicality
IntroductionFOUNDATIONS OF HF-CLT INTEGRATIONUSEFULNESS OF CLT FOR HF-ED
HF
HCI
UXID
ED-PSY
CLT
Why Human Factors in Education?Domain Total System Failure Failure Rate Source
Healthcare Lethal Adverse Events Rate (2010) 1.6% James (2013)
AviationU.S. Flight Cancellation Rate (2015) 1.98% U.S. DOT (2016)
Global Fatal Crash Death Rate (2015) <0.0000001% Smith (2016); The World Bank (2016)
Education
Average U.S. High School Dropout Rate 6.5% Velez (2014)
U.S. High School Students Attending ‘Dropout Factory’ (>40% Do Not Matriculate) 16.7% Balfanz & Legters
(2014)University ‘Dropout’ Rate
(Full Time Students @ 4-Year Institutions Without Bachelors in < 6 Year) 41.7%Weissman (2012)
Unemployed College Grads 7.2%
Underemployed College Grads 14.9% Davis et al. (2015)
• Obstacles facing reform & improvement of educational outcomes• Addressed by integration of HF principles & methods in systems of education1
1Keebler et al., 2010;
Cognitive Load Theory (CLT) - Overview
• CLT = Instructional Design (ID) Approach• Influential, adaptive, practical in modern education234
• Numerous ID principles developed based on consistent effects in research
• History of systems perspective insight for CLT• Foundations
• Cognitive Architectures & Learning Frameworks• Integrations
• Human-Computer Interaction (HCI) 5 & Human Factors (HF) 1
1Keebler et al., 2010; 2Jones et al., 2010; 3Kirschner et al., 2011; 4Paas et al., 2010; 5Hollender et al., 2010;
Cognitive Load Theory (CLT) - Overview
• Cognitive Load (CL)• Total mental load imposed on learner
cognitive system by ID• Essentially, task load + workload for ID
• Three Load Factors• Intrinsic, Extraneous, Germane
• Instructional strategies• Those which promote successful
WM/LTM processing enhance learning,• Thus educational outcomes
Current IssuesNECESSITY OF HF-CLT INTEGRATION
MeasureMethodModel
Cognitive Load Theory (CLT) – The (Additive) Model
I + E + G = CLLoad Factor Description
I IntrinsicCLT: Element Interactivity; interaction of system with user’s expertiseHF-CLT: Element interactivity; Inherent difficulty of KSA, based on quantity and type of information
E ExtraneousCLT: ‘Ineffective load’ – cognitive capacity not being used for expert schema formationHF-CLT: Quantified measure ID’s inhibition of effective expert schema formation
G GermaneCLT: ‘Effective load’ – cognitive capacity being used for expert schema formationHF-CLT: Quantified measure of ID’s ability to support effective expert schema formation
CL Cognitive LoadCLT & HF-CLT: Total cognitive load imposed on learner by ID system / cognitive capacity used by learning during instruction
Cognitive Load Theory (CLT) – The (Additive) Model
Paas et al. (2003) in reference to Xie & Salvendy (2000) EEG measurement of mental workload
Cognitive Load Theory (CLT) – The (Additive) Model
• Additive CLT Model Issues• Lack of divergent validity between Intrinsic & Germane loads
• Kalygula, 2011; Sweller, 2010• Lack of divergent validity between Extraneous & Germane loads
• Sweller, 2010• Germane load found to have independent negative correlation with CL
• Cierniak et al. 2009• Cognitive Load not empirically supported by additive model
• de Jong, 2010; Park, 2010; Schnotz & Kyrschner, 2007; Hollender et al., 2010
• Non-additive performance & transfer effects
Cognitive Load Theory (CLT) – The (Additive) Model:
SHOULD:Quantitatively describe the relationship between instructional design
factors and cognitive load(giving quantitative insight into relationship between design & educational outcomes)
DOES:Conceptually suggest a relationship between subjective perceptions of
instructional design and perceived cognitive load(giving conceptual insight into relationship between design & educational outcomes)
Cognitive Load(Ideal)
Attention Resources
Perception
Working Memory
Long-Term Memory
Response Selection
Response Execution
Environment (Feedback)
Cognitive Load (Current)
Cognitive Load(Objective)
Task Load Workload
Instructional Design
STSS
Adapted from Wickens (1992)
The Model - ‘Take-Away’:
“Cognitive Load is not supported by the additive model.”
Our community is ideal to support the iterative development of an accurate model, and use it to
optimize instructional design and educational system.
Cognitive Load Theory (CLT) – The Measures
WANTED:Valid, consistent, non-invasive measures of cognitive load, sensitive to
different changes in ID, standard for use across ID mediums.
Cognitive Load Theory (CLT) – The Measures
• There are no definitive guidelines for the measurement of cognitive load and its factors.1234
• Various measures for CL permeate through the literature, although their validity is heavily debated.245
• The issue of standardizing valid, consistent, and reliable measures for CLT research is considered the ‘holy grail’ of CLT research. 3
• Development of a valid and sensitive method measuring different load types (intrinsic, extraneous, germane) is needed to improve CLT. 245
2de Jong, 2010; 3Kirschner et al., 2011; 4Moreno, 2010; Park, 20105
Cognitive Load Theory (CLT) – The Measures
• Cognitive Load Measurement• Subjective Rating Scales
• Cognitive Load (Paas, 1992)• Mental Effort (Paas et al., 1994)• Instructional Difficulty (Kalygula et al., 1999)• NASA-TLX (Hart & Staveland, 1988)
• Secondary Task Performance (Dual Task Paradigm)• Rhythm Precision Method (Park & Brunken, 2015)
• Physiological Measurement (Brunken et al., 2003)• Pupillary Dilation / Fixation• Neuroimaging (EEG, MRI, fMRI, etc.) • Time-on-Task• Stress Measures
Cognitive Load Theory (CLT) – The Measures
• Load Factor Measurement• Cognitive Load Scale (Leppink et al., 2013)
• Support for three-factor model, but uses two-factor (I&E) model• Potential as comparative post-measurement tool
• Battery of subjective & objective measures (Wernaart, 2012)• No measures found for germane load; intrinsic + germane bundled
2de Jong, 2010; 3Kirschner et al., 2011; 4Moreno, 2010; 5Park, 2010
Cognitive LoadMeasurement
(Ideal)
Subjective Load Measurement
(Cognitive Load / Workload)
Cognitive Load Measurement
(Objective)
Task LoadMeasurement
Instructional Design
Measurement
Adapted from Wickens (1992)
Expertise Measurement
STSS
Attention Resources
Perception
Working Memory
Long-Term Memory
Response Selection
Response Execution
Environment (Feedback)
The Measures - ‘Take-Away’:
“HF will improve measures used in CLT research.”
Our community is ideal continue the iterative development of CLT measures, and develop best
practices for their use both the laboratory and the operational domain.
Cognitive Load Theory (CLT) – The Method• Current CLT methods not useful for use by practitioners
• CLT only currently offers:• Highly invasive in-vivo measurement• Indirect post-test measurement and analysis methods
• No method to quantify load factors to pre-estimate impact of ID on CL and performance, and directly compare IDs• Would be an effective tool for instructional designers and educators alike
The ‘Take-Away’:
Cognitive Load Theory needs to not just inform ID, but be used by practitioners to evaluate ID
Our community is ideal to support the development of tools for instructional designers and educators to
evaluate and improve instruction systems
- Bringing the benefits of CLT from the lab to the classroom -
New ContributionQUANTITATIVE HF-CLT MODEL & METHODS
MeasureMethodModel
Quantified HF-CLT Model
• Theoretical Basis• Independent & antagonistic relationship between Germane & Cog. Load
• Proposed interaction effects between G, E, & I intrinsic loads on C.L. during acquisition• With non-additive model, qualities of ID make quantifiable impact upon KSA acquisition
• HF-CLT Integration Model (Keebler et al., 2010)• Still retained additive interactions
• Correct identification of load factors (I, E, & G) as types of ‘task load’ factors• In operational domain, load factors thus identical for each student, with individual C.L.s
• Expertise in specific KSA as an independent modifier of ID impact on C.L.• Accounts for additive model breakdown in Expertise-Reversal Effect • Creates ‘Learning Curve’ –type interactions for ID impact upon KSA acquisition
2de Jong, 2010; 3Kirschner et al., 2011; 4Moreno, 2010; Park, 20105
Quantified HF-CLT Model
CL = ( I ) ( 1 + E ) ( 1 – G) X
• CL = Total Cognitive Load (% of cognitive capacity required for optimal performance);• I = Intrinsic Load (% difficulty of content of material);• E = Extraneous Load (% detriment in performance by instructional design);• G = Germane Factors (% benefit in performance by instructional design);• X = Expertise (% expertise of target KSA);• Values (I,E,G,X) greater than zero and less than one
Quantified HF-CLT Model
• Not intended to be the definitive – or only – quantitative CLT model
• Only quantitatively describes relationship of load factors to C.L.• Neither intended to be descriptive of processes addressed by Cognitive
Theory of Multimedia Learning6
• Nor the important motivational & affective factors addressed by Cognitive-Affective Theory of Learning with Media3,4
• Does not describe optimal C.L., as initial research needed• However, model could be used to assist C.L.-performance relationship research• Model may be compatible with Arousal Theory & Yerkes-Dodson Law5
• Does not address relations between C.L., mental load, mental effort• Previous conceptualizations of these relationships in CLT lack both clarity and validity2,4
6Mayer, 2005; 2de Jong, 2010; 3Kirschner et al., 2011; 4Moreno, 2010; Park, 20105
Quantified HF-CLT Model
• Limitations (continued)• Environmental, individual, task factors, and interactions between these
factors (Choi et al., 2014) – to include experiential factors – impact all aspects of learning at all junctions of KSA acquisition and transfer (Sonnenfeld & Meyers, 2014).
• This model, can be immediately evaluated for its potential to be applied for the comparison of instructional materials and the optimization of instructional design• Using HF & usability methods, and measures suggested here
Quantified HF-CLT Model
• Cognitive Load Curves (CLC)
CLCs for Average Difficulty Material (left) and High Difficulty Material (right)
Quantified HF-CLT Measures
• Usability testing methods fundamental to human-systems research• HCI/Usability in CLT (Hollender et al., 2010 )
• Factors (I, E, & G) can be quantified using HF/HCI usability methods• Intrinsic load can be given a score based on element interactivity and
heuristically-defined relative difficulty• Perhaps I = (B + S) + [(5 * GL * R) /2]• Bits & Schema (E.I) + Grade Level & Evaluated Difficulty
• Extraneous & Germane loads can be given a score derived from heuristic evaluations of material• Based on current CLT effects & findings
Quantified HF-CLT Method
• Iterative refining of these measures should inform the development of an automated ID analysis tool for future use by practitioners • By refining and creating validated, sensitive, standardized measures…• We may further quantify the relationship between qualities of ID and
educational outcomes for the optimization of learning
• Future research findings can then be used to further enhance individualized adaptive training education systems.
References (pg. 1/4)• Balfanz, R., & Legtlers, N. (2014). Locating the dropout crisis. Center for Social Organization of Schools at Johns
Hopkins University’s School of Education. Retrieved on September 18, 2016 from http://www.csos.jhu.edu/crespar/techReports/Report70.pdf
• Cierniak, G., Scheiter, K., Gerjets, P. (2009). Explaining the split-attention effect: Is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? Computers in Human Behavior, 25(2), 315-324.
• Davis A., Kimball, W., & Gould, E. (2017, May 17). The class of 2015. Economic Policy Institute. Retrieved on September 18, 2016 from http://www.epi.org/publication/the-class-of-2015/
• Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology,52, 139-183.\
• Hollender, N., Hofmann, C., Deneke, M., Schmitz, B. (2010). Integrating cognitive load theory and concepts of human–computer interaction. Computers in Human Behavior, 26(6), 1278-1288.
• James, J. T. (2013). A new, evidence-based estimate of patient harms associated with hospital care. Journal of patient safety, 9(3), 122-128
• Jones, S.J., Fong, C.J., Torres, L.G., Yoo, J.H., Decker, M.L., Robinson, D.H. (2010). Productivity in educational psychology journals from 2003 to 2008.Contemporary Educational Psychology, 35(1), 11-16.
References (pg. 2/4)• Kalyuga, S. (2011). Cognitive load theory: How many types of load does it really need?. Educational
Psychology Review, 23(1), 1-19.
• Keebler, J.R., Ososky, S., Jentsch, F., Fincannon, T. (2010, September). Gaining ground: Merging cognitive load theory with human factors principles. Proceedings of the human factors and ergonomics society annual meeting, 54(8), 667-671.
• Kirschner, P.A., Ayres, P., Chandler, P. (2011). Contemporary cognitive load theory research: The good the bad and the ugly. Computers in Human Behavior, 27(1), 99-105.
• Leppink, J., Paas, F., van der Vleuten, C.P.M., Van Gog, T., Van Merriënboer, J.J.G. (2013). Development of an instrument for measuring different types of cognitive load. Behavior Research Methods (Online), 45(4), 1058-1072
• Mayer, R. E. (Ed.). (2005). The Cambridge handbook of multimedia learning. Cambridge University Press.
• Moreno, R. (2010). Cognitive load theory: More food for thought. Instructional Science, 38(2), 135-141.
• Paas, F. G. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of educational psychology,84(4), 429.
References (pg. 3/4)• Paas, F.G., van Gog, T., Sweller, J. (2010). Cognitive load theory: New conceptualizations, specifications, and
integrated research perspectives. Educational Psychology Review, 22(2) 115-121.
• Paas, F.G., van Merrienboer, J.J. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of educational psychology, 86(1), 122-133.
• Park, B. (2010). Testing the additivity hypothesis of cognitive load theory. Unpublished dissertation, University of Saarlandes.
• Park, B., Brünken, R. (2015). The rhythm method: A new method for measuring cognitive load – An experimental dual-task study. Applied Cognitive Psychology, 29(2), 232-243.
• Schnotz, W., Kürschner, C. (2007). A reconsideration of cognitive load theory. Educational Psychology Review, 19(4), 469-508.
• Smith, O. (2016, January 6). 2015 was the safest year in aviation history. The Telegraph. Retrieved on September 18, 2016 from http://www.telegraph.co.uk/travel/news/2015-was-the-safest-year-in-aviation-history/.
• Sonnenfeld, N.A., Meyers, M. (2014). A conceptual model of transfer of training via virtual environments. Presented at the Human Factors Applied Psychology Conference, Daytona Beach, Florida.
References (pg. 4/4)• The World Bank Group. (2016). Air transport, passengers carried. Retrieved on September 18, 2016 from
http://data.worldbank.org/indicator/IS.AIR.PSGR?end=2015&name_desc=true&start=2015&view=chart
• U.S. Department of Transportation (DOT). (2016). On-time performance: Flight delays at a glance. Retrieved on September 18, 2016 from http://www.transtats.bts.gov/homedrillchart.asp.
• Velez, E.D. (2014, January). America’s college drop-out epidemic: Understanding the drop-out population. National Center for Analysis of Longitudinal Data in Education Research. http://www.caldercenter.org/sites/default/files/WP-109-Final.pdf
• Weissmann, J. (2014, November 19). America’s awful college dropout rates, in four charts. Slate. Retrieved on September 18, 2016 from http://www.slate.com/blogs/moneybox/2014/11/19/u_s_college_dropouts_rates_explained_in_4_charts.html
• Wernaart, G. (2012). Cognitive load measurement: Different instruments for different types of load? Unpublished master’s thesis, Utrecht Univeristy, Utrecht, Netherlands.
• Wickens, C.D. (1992). Engineering psychology and human performance, 2nd ed., Harper-Collins, New York.