frameworks, methods, and applications in diagnostic ......rohrschach, mmpi, and more diagnostic time...
TRANSCRIPT
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Frameworks, Methods, and Applications in Diagnostic Educational Testing
International Symposium on “Diagnostic Testing in Education”
André A. RuppEducational Testing Service (ETS)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Overview of Keynote
• Diagnostic educational testing
• Cross-walk of current landscape
- Cognitive models - Principled assessment design- Validation- Scoring and reporting
• Implications for R&D
2
Diagnostic Educational Testing
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 4
We clearly want this!
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Under appropriate (systemic) circumstances, diagnostic educational testing can:
- Supplement coarse-grained summaries about learners from summative assessments
- Provide fine-grained feedback that is actually useful (“interpretable”, “actionable”)
- Foster openness to new forms of learning and assessment as a result of needed diagnostic information
- Serve as a mechanism for professional development of teachers, administrators, and research teams
5
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 6
But it is not as easy as we had thought…
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Implementing a successful diagnostic educational testing effort at a system level is very challenging due to:
- Design and implementation complexities for individual diagnostic assessments
- Alignment complexities across all components in educational systems
- Financial and political constraints for individual companies, funders, and governing bodies
- Cultural resistances to aspects such as additional testing time for learners to gather “richer” diagnostic information or automated scoring of performances
7
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
So we have learned some lessons…
8
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Diagnostic testing ≠Diagnostic testing
9
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Foci: Attention disorders, learning disabilities, psychiatric problems
Cases: 13-year old child with combined performance issues in mathematics such asstruggles with fluent retrieval of math facts, with the ability to retain new knowledge,and with the calculation or representation of fractions and percentages
Diagnosis: After thorough testing, it may be determined that the child hasdyscalculia and an attention disorder
Testing batteries (about 20 subtests): WISC, Woodcock Johnson Test ofAchievement and Test of Oral Language, Tower of London, Rey-O, WRAML-2, IVA-2,TOWRE-2, OWLS-2, Boston Naming, Autosequencing, CTOPP-2, Nelson-Denney,Rohrschach, MMPI, and more
Diagnostic time (about 30 hours): intake interview (2 hours), batteries (8-12hours), analysis (4-6 hours), writing (10-12 hours), interpretative conference /feedback session for parents (3 hours)
Total cost: $ 4000 - $ 6000
Professional training: Doctorate in clinical psychology or neuro-psychology, postdoc, clinical testing experience
10
Neuropsychological (Diagnostic) Testing
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Be Careful to Manage Expectations!
11
Cross-walk of Current Landscape
(Fast Walk)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 13
+ Glossary
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Key Themes
• Cognitive, socio-cognitive, and non-cognitive factors • 21st-century / complex performance skills• Learning progressions• Principled assessment design • Validation argumentation• Diagnostic measurement • Learning analytics / process data• Automated scoring• Digital learning environments• Stealth assessment• Accessibility
14
Cognitive Models
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
New Constructs
16
Figure 1 Viral Internet image, circa 2014.
Shute & Wang (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Framework for Non-cognitive Skills
17
Kyllonen (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
New Constructs = New Assessments
18
Oranje et al., Gobert et al., Shute et al., Jackson et al. (2016)
Principled Assessment Design
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Three characteristics of principled assessment design:
• Construct-centered nature of approach• Engineering towards intended interpretation and uses• Explicit design decisions and rationales
Three criteria to evaluate the fitness of theories of learning and cognition to inform assessment design and development:
• Clarify the targets of inference• Identify relevant content features for items or tasks• Identify relevant features of learner performance
20
Ferrara et al. (2016)
Nichols et al. (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Several Ways to be Principled
21
• Evidence-centered design (ECD)• Cognitive design system (CDS)• Assessment engineering (AE)• Principled design for efficacy (PDE)• BEAR assessment system (BAS)
They are not methodologically or contextually interchangeable but share certain deep-structure principles and goals
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 22
Ferrara et al. (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 23
Gierl & Lai (2016)
Validation
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Construct Validity Framework
25
Embretson (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 26
Bachman (2005)
Validation Argumentation Framework
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Cognitive Model Validation(Theory)
27
Keehner et al. (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 28
Cognitive Model Validation(Practice)
Keehner et al. (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Nichols et al. (2016)
29
Validation is complex!
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 30
Especially for digitally-delivered tasks…
Figure 3 Screenshot of the Inq-ITS system with all phases of inquiry shown.
Gobert & Sao Pedro (2016)
Figure 2 Assessment report for teachers reported out by inquiry practice and sub-skills.
Scoring and Reporting
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Alignment with Design (I)
32
Shute & Wang (2016)
Figure 6 CM of creativity and sample indicators; [R] indicates reverse coded.
Figure 1 Viral Internet image, circa 2014.
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 33
Alignment with Design (II)
Shute & Wang (2016)
Figure 8 Bayes net fragment for creativity for “Swamp People” level.Figure 7 “Swamp People” level in Physics Playground (a) with typical ramp solution (b), less common springboard solution (c), and rare
lever solution (d) .
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Process Data Design and Analysis
Phase 1: Principled design of logfile structure and tasksPhase 2: Alignment of analytic goals with study designPhase 3: Feature generation and aggregationPhase 4: Initial descriptive and inferential analysesPhase 5: Formal measurement and learning analytics modelingPhase 6: Refinement of learning theories, platform design, and
research plans
All of this requires computational infrastructures and investments into professional development of staff!
34
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Principled Data Capture from Logfiles
35
Kerr et al. (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Modern Modeling Approaches
• Explanatory item response theory
• Diagnostic classification models (parametric)
• Diagnostic classification approaches (semi-parametric)
• Bayesian inference networks
• Models for repeated-measures data
• Data mining and learning analytics
• Related modeling families
Selected future directions:
- Integrating indicators from process and product data into parametric models with uncertainty estimation
- Advancing methodologies for dynamic predictions within digital environments
- Leveraging subject-matter expertise more smartly for model specification, criticism, and refinement to be generative
36
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Principled Criticism and Refinement
37
Harring & Hauser (2016)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Automated Scoring Technologies
• Extended answers / essays• Short content-focused answers• Mathematical responses (numerical, graphical)• Spoken responses• Multimodal responses (combinations of above)• Multimodal responses (novel constructs)• Activity sequences (including chats)• Complex performance products (semantic webs,
flowcharts, designs)
This requires feature development, model training, model evaluation, and score validation!
38
Bejar et al. (2016)
Implications for R&D
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 40
Bachman (2005)
Conceptual diagrams are helpful but….
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
…we need more “PAD Live / PAD Illustrated”
• More “knowledge transfer” from agencies, companies, and research institutions about process, rather than product
• More “worked exemplars” that describe the design decisions, rationales, and processes
• More flexibility in design, prototype, and refine cycles of innovation for assessments
• More generalizable capabilities and architectures that facilitate principled assessment design and scoring work
• More sharable artifacts that can be tagged, annotated, adapted, and adopted
• More principled leveraging of digital technology for learning and assessment
41
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
This symposium seems like a great step in that direction!
42
You’re Awesome!
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Therefore: my excitement about this symposium…
43
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728
Frameworks, Methods, and Applications in Diagnostic Educational Testing
International Symposium on “Diagnostic Testing in Education”
André A. RuppEducational Testing Service (ETS)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING are registered trademarks of Educational Testing Service (ETS). 34728 45
+ Glossary