the underpinnings of developing an instrument to measure ... · the underpinnings of developing an...
TRANSCRIPT
The Underpinnings of Developing an Instrument to Measure Systems Thinking Abilities in the Context of Earth System ScienceNICHOLAS A. SOLTIS AND KAREN S. MCNEAL
DEPA RT M EN T OF GEOSCI ENCES , AUBUR N UN I VERS I TY, 2050 BEA R D EAVES C OL I SEUM , AUBUR N , A L 36849
1
Thinking like a Geoscientist
Spatial Thinking
Temporal Thinking
Systems Thinking
2
Thinking like a Geoscientist
Spatial Thinking
Temporal Thinking
Systems Thinking
3
Thinking like a Geoscientist
Spatial Thinking
Temporal Thinking
Systems Thinking
4
Thinking like a Geoscientist
Spatial Thinking
Temporal Thinking
Systems Thinking
5
Challenges to Earth Systems ThinkingConceptualization of natural Earth environments as systems
Understanding the complex characteristics of systems◦ Multiple variables, above and below average data, not at
equilibrium
The application of conceptual models of complex Earth systems to support environmental problem solving
Systems thinking skills must be explicitly taught, How do we know if we are teaching them?
6Herbert, 2006
Concept Inventories
7
Methods
Write instrument based on table of
specifications
Internal and external
expert review of items
Think-aloud interviews with target audience
Pilot Study (250
individuals)Analysis
Full Study
(750 individuals)
8
Revise
Content Validity Construct Validity
Cultural Validity
Developing a Framework
How we teach
What the literature says
EST CI Framework
9
How we teach
Earth Systems Thinking Teaching
Systems Model
Elements
Systems Thinking Elements
Real World Application
Elements
Soltis et al., In Press 10
Developing a Framework
How we teach
What the literature says
EST CI Framework
Soltis et al.
11
What the literature says
12
Earth System Perspective
Emphasis on high-level interconnections between major Earth spheres
Earth System Thinking SkillsEmphasis on transformation of matter in Earth
cycles and specific system thinking concepts
Complexity Sciences
Emphasis on scientific study of complex systems
Authentic Complex Earth and Environmental Systems
Emphasis on knowledge of a specific Earth system or phenomenon
Scherer et al., 2017
What the literature says
13
Earth System PerspectiveEmphasis on high-level interconnections
between major Earth spheres
Scherer et al., 2017
What the literature says
14
Earth System Thinking Skills
Emphasis on transformation of matter in Earth cycles and specific system thinking concepts
Scherer et al., 2017
What the literature says
15
Complexity SciencesEmphasis on scientific study of complex
systems
Scherer et al., 2017
What the literature says
16
Authentic Complex Earth and Environmental
Systems Emphasis on knowledge of a specific Earth
system or phenomenon
Scherer et al., 2017
What the literature says
17
Earth System Perspective
Emphasis on high-level interconnections between major Earth spheres
Earth System Thinking SkillsEmphasis on transformation of matter in Earth
cycles and specific system thinking concepts
Complexity Sciences
Emphasis on scientific study of complex systems
Authentic Complex Earth and Environmental Systems
Emphasis on knowledge of a specific Earth system or phenomenon
Scherer et al., 2017
Developing a Framework
How we teach
What the literature says
EST CI Framework
Scherer et al.
Soltis et al.
18
Systems Thinking Elements
Systems ModelElements
Real World Application
Elements
Earth System Perspective
Earth System Thinking Skills
Authentic Complex Earth and Environmental Systems
Perspective
Earth System Perspective
Complexity Sciences
Developing a Framework
How we teach
What the literature says
EST CI Framework
Scherer et al.
Soltis et al.
20
An access point: Biogeochemical Cycles?
Undergraduates largely have incomplete conceptions of the carbon cycle and other biogeochemical cycles
21Soltis et al., in revision
CI Framework
22
Framework Sample Question Stem
Earth System Perspective Photosynthesis represents a flux of carbon dioxide from where to where?
Earth System Thinking Skills If the ocean absorbs more carbon than it releases, in this capacity it is best thought of as what?
Complexity Sciences Based on the diagram, which of the following would enhance the rate of soil erosion?
Authentic Complex Earth and Environmental Systems* * Too contextualized to be appropriate for general concept inventory
Participants
Expert Review◦ 9 experts in Earth system science, systems thinking, and/or instrument development
Think-Aloud Interviews◦ 6 undergraduate students
◦ Compensated with a $20 Amazon gift card
Pilot and Implementation ◦ Online crowd sourcing: MTURK
◦ MTURK samples are representatively similar to traditional research subject pools in terms of race, gender, age, and education
◦ 250 in initial pilot
◦ Full implementation of 750
◦ Compensated through MTURK at a fair market rate for their time
23
MTurk: A Crowdsourcing WebsiteVirtual labor marketplace where “requesters” create tasks for “workers” to complete for monetary compensation◦ Requester: employer/researcher
◦ Worker (Turker): employee/participant
24
Advantages and Disadvantages ADVANTAGES
Subject pool access◦ Access to wide range of participants
Subject pool diversity◦ Access to diverse range of
participants
Low overall cost◦ Low payment
◦ Quick data collection
DISADVANTAGES
Data quality◦ Data accuracy◦ Participant identity
Ethicality◦ Payment◦ Participant treatment◦ Length/effort
Sample bias
Buhrmester, 2017 25
Analysis
Both Classical Test Theory and Item Response Theory will be used
Item difficulty and item discrimination will be determined for each item and exploratory factor analysis will be employed (CTT)
Rasch analysis will be applied to each unidimensional scale determined (IRT)
Will be conducted for initial pilot with revisions made as needed
Repeated for full implementation-◦ Confirmatory factor analysis instead of exploratory factor analysis
◦ Iterative process, with revisions being made as needed based on analysis
26
Classical Test Theory
• Test is unit of analysis• Assumption that everyone
has a true score that is unknowable
• Observed score equals true score plus error
• Uses linear model
Item Response Theory
• The item is the unit of analysis
• Focuses on comparing and analyzing each item
• Uses non-linear model (logistic)
27
ConclusionsSystems thinking is integral
Research on systems thinking is moving forward
We need a quantitative instrument to assess students’ systems thinking skills
Concept Inventories are key in gathering quantitative data
A strict development process is required for reliability and validity
Tools like mTurk can be used to better gather large amounts of data
28
Acknowledgements Chris Schnittka, Stephanie Shepherd, Joni Lakin, Anne Gold, Leilani Arthurs, Juliette Rooney-Varga, Lisa Gilbert, Hannah Scherer for their insights in the development and revision of this instrument
Auburn University Department of Geosciences and Advisory Board for travel funds
29
ConclusionsSystems thinking is integral
Research on systems thinking is moving forward
We need a quantitative instrument to assess students’ systems thinking skills
Concept Inventories are key in gathering quantitative data
A strict development process is required for reliability and validity
Tools like mTurk can be used to better gather large amounts of data
30