serious play conference los angeles, ca – july 21, 2012 girlie c. delacruz and ayesha l. madni...
TRANSCRIPT
![Page 1: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/1.jpg)
Serious Play Conference
Los Angeles, CA – July 21, 2012
Girlie C. Delacruz and Ayesha L. Madni
Setting Up Learning Objectives and Measurement for Game Design
![Page 2: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/2.jpg)
Components of Assessment Architecture
Create assessment architecture (Your Example)
Assessment Validity
Overview
![Page 3: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/3.jpg)
What is so hard?
![Page 4: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/4.jpg)
What are some of your challenges?
![Page 5: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/5.jpg)
Passed the Game
![Page 6: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/6.jpg)
Gameplay
Log data
Domain
![Page 7: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/7.jpg)
Challenges We Have
• Translating objectives into assessment outcomes– Purpose of assessment information– Communication between designers and educators
• Game is developed—need to assess its effectiveness– Cannot change code, wraparounds
![Page 8: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/8.jpg)
How can we meet the challenge?
![Page 9: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/9.jpg)
Assessment requirements
Technology requirements
Instructional requirements
Front-end Efforts Support Effectiveness
![Page 10: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/10.jpg)
Model-Based Engineering Design
Communication
Collaboration
![Page 11: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/11.jpg)
Model-Based Engineering Design
z
![Page 12: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/12.jpg)
Part One
ASSESSMENT VALIDITY
![Page 13: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/13.jpg)
Assessment (noun) = Test
What Is Assessment?
![Page 14: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/14.jpg)
=
Assessment As A Verb
Process of drawing reasonable inferences about what a person
knows by evaluating what they say or do in a given situation.
ASSESSMENT
![Page 15: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/15.jpg)
Games As Formative Assessment
Formative Assessment:
Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967).
![Page 16: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/16.jpg)
Games As Formative Assessment
Games as Formative Assessment:
Use and interpretation of game performance information with intent to adapt learning, such as provide feedback.
![Page 17: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/17.jpg)
What is Validity?
![Page 18: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/18.jpg)
Assessment Validity as a Quality Judgment
Critical Analysis
Legal Judgment
Scientific Process
![Page 19: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/19.jpg)
=
Assessment Validity
Bringing evidence and analysis to evaluate the propositions of interpretive argument.
(Linn, 2010)
ASSESSMENT VALIDITY
![Page 20: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/20.jpg)
How Does This Relate to Design?
① Identification of the inferences to be made.• What do you want to be able to say?
② Specificity about the expected uses and users of the learning system.• Define boundaries of the training system
• Determine need for supplemental resources
③ Translate into game mechanics
④ Empirical analysis of judgment of performance within context of assumptions.
![Page 21: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/21.jpg)
What do you want to be able to say about the gameplayer(s)?
• Player mastered the concepts.• How do you know?• Because they did x, y, z (player history)• Because they can do a, b, c (future events)
![Page 22: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/22.jpg)
Identify Key Outcomes: Defining Success Metrics
• Quantitative Criteria (Generalizable)– % of successful levels/quests/actions– Progress into the game– Changes in performance
• Errors• Time spent on similar levels• Correct moves
• Qualitative Criteria (Game-specific)– Patterns of gameplay– Specific actions
![Page 23: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/23.jpg)
motion
pre1
speed direction duration
o1 o2 o3
pre2 pre3 pre4 pre5
o4 o5 o6 o7 o8
BACKGROUND LAYER• Prior knowledge• Game experience• Age, sex• Language
proficiency
CONSTRUCT LAYERConstruct, subordinate constructs, and inter-dependencies
INDICATOR LAYERBehavioral evidence of construct
EVENT LAYERPlayer behavior and game states
FUNCTION LAYERComputes indicator value
fn(e1, e2, e3, ...; s1, s2, s3, ...): Computes an indicator value given raw events and game states
Game events and states (e1, e2, e3, ...; s1, s2, s3, ...)
![Page 24: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/24.jpg)
General Approach
• Derive structure of measurement model from ontology structure
• Define “layers”– Background: Demographic and other variables that may
moderate learning and game performance– Construct: Structure of knowledge dependencies– Indicator: Input data (evidence) of construct– Function: Set of functions that operate over raw event
stream to compute indicator value– Event: Atomic in-game player behaviors and game states
• Assumptions– Chain of reasoning among the layers are accurate
![Page 25: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/25.jpg)
Part Two
ASSESSMENT ARCHITECTURE
![Page 26: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/26.jpg)
Components of Assessment Architecture
COGNITIVE DEMANDS
• defines targeted knowledge, skills, abilities, practices
• domain-independent descriptions of learning
DOMAIN REPRESENTATION• instantiating domain-specific related
information and practices• guides development• allows for external review
TASK SPECIFICATIONS
• defines what the students (tasks/scenarios, materials, actions)
• defines rules and constraints)
• defines scoring
![Page 27: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/27.jpg)
Cognitive Demands
What kind of thinking do you want capture?
• Adaptive, complex problem solving
• Conceptual, procedural, and systemic learning of content
• Transfer
• Situation awareness and risk assessment
• Decision making
• Self-regulation
• Teamwork
• Communication
![Page 28: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/28.jpg)
Domain Representation
• External representation(s) of domain-specific models
• Defines universe (or boundaries) of what is to be learned and tested
![Page 29: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/29.jpg)
Ontologies
Item specifications
Example: Math
Knowledge specifications
![Page 30: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/30.jpg)
Task Specifications
① Operational statement of content and behavior for task
• Content = stimulus/scenario (what will the users see?)
② Behavior = what student is expected to do/ response (what will the users do?)
• Content limits
③ Rules for generating the stimulus/scenario posed to the student
• Permits systematic generation of scenarios with similar attributes
• Response descriptions
④ Maps user interactions to cognitive requirements
![Page 31: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/31.jpg)
Force and Motion Pushes and pulls, can have different strengths and directions. Pushing and pulling on an object can change the speed or direction of its motion and can start or stop it. Each force acts on one particular object and has both strength and a direction. Energy The faster a given object is moving, the more energy it possesses
NGSS performance expectation
Plan and conduct an investigation to compare the effects of different strengths of pushes on the motion of an object (K-PS2-1).
Analyze data to determine if a design solution works as intended to change the speed or direction of an object with a push (K-PS2-2).
Content limits Effects: change in position; increased or decreased accelerationStrengths of pushes: Qualitative (small, medium, big), or quantitative Type of Motion: Rotational Constraints on planar objects: Must be something that can be pushed horizontally and attached to its fulcrum (e.g., the door to a house) Allowable variations on objects: Mass, height and width, location of object Constraints on fulcrum objects: Must be attached to the planar object; position of fulcrum object cannot be changed
Data: distance, slope, time, speedSpeed change: increase in accelerationDirection: Vertical movement Constraints on planar objects: Must be something flat (e.g., book, frame, ruler) that can be placed on another object and can be pushed in a downward movementAllowable variations on planar objects: Mass, height and width, location of object in the room, surface materialConstraints on fulcrum objects: The structural properties of the fulcrum should support some, but not all of the set of planar objects; position of fulcrum object can be changed
Targeted science and engineering practice(s)
Ask questions that can be investigated based on patterns such as cause and effect relationships.
Use observations to describe patterns and/or relationships in that natural and designed world(s) in order to answer scientific questions and solve problems.
Response description
Ask questions: Query the MARI about the properties of the objects (e.g., what is the distance between the hinge and where I pushed) based on observed outcomes (e.g., how hard it was to push the door, or how far the door moved).
Use observations: use snapshot images of activity in the HRLA with overlaid measurement data generated by the MARI to sort situations based on the physical features, behaviors, or functional roles in the design.
Task complexity Student only has 4 attempts to pass the ball to the girl and can only vary position and strength of push.
Easy: Student can vary the position and strength of the push, but must apply force by placing additional objects on the planar object and pushing downward with both hands (to connect the kinesthetic experience of applying the force with hands on experience of the object). Harder: Student can vary both the position and strength of the push and how the planar object is placed on the fulcrum (e.g., load is moved closer or further away from fulcrum)
Available resources Iconic and graphical representation of underlying physics laws will be on the screen, and will change based on student actions. Guided questions will ask students about distance, mass, force magnitude and direction, height, and slope based on observed outcomes.
![Page 32: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/32.jpg)
Components of Computational Model
![Page 33: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/33.jpg)
Components of Decision Model
Do nothing: move on, end taskGet more evidence or information: repeat same task, perform similar task, ask a question
Intervene (instructional remediation): give elaborated feedback, worked example or add scaffolding, more supporting information
Intervene (task modification): new task (reduced or increased difficulty), new task (qualitatively different)
Courses of Action
![Page 34: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/34.jpg)
Components of Decision Model
Confidence of diagnosis : How certain are we about hypothesized causal relation?
Consequence of misdiagnosis: What happens if we get it wrong? What are the implications of ignoring other possible states or causal relations?
Effectiveness of intervention: How effective is the intervention we will give after diagnosis?
Constraints: Do we have to efficiency concerns with respect to time or resource constraints?
Decision Factors
![Page 35: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/35.jpg)
Part Three
ASSESSMENT ARCHITECTURE(YOUR EXAMPLE)
![Page 36: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/36.jpg)
36
Person (prior knowledge and experience)
Task characteristics
Context (test, simulation, game)
Fixed Variables
+
+
Assumptions and Design Rationale
Assessment Architecture
![Page 37: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/37.jpg)
37
Person (prior knowledge and experience)
Task characteristics
Context (test, simulation, game)
Fixed Variables
+
+
Performance to be Assessed
Assessment Architecture
Observed Event(s)
What happened?(Raw data, scored
information?)
![Page 38: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/38.jpg)
38
Person (prior knowledge and experience)
Task characteristics
Context (test, simulation, game)
Fixed Variables
+
+
Assessment Architecture
Observed Event(s)
What happened?(Raw data, scored
information?)
Translation
What does this mean?Judgment of performance
![Page 39: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/39.jpg)
39
Person (prior knowledge and experience)
Task characteristics
Context (test, simulation, game)
Fixed Variables
+
+
Assessment Architecture
Observed Event(s)
What happened?(Raw data, scored
information?)
Translation
What does this mean?
Assessment Validation
Inferences
What are the potential causes of the observed events?
Lack of Knowledge?
Context?
Characteristics of the task?
Not sure?
![Page 40: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/40.jpg)
Potential Course of Actions
Repeat Same Trial
Get more evidence or information
Perform Similar Task
Ask a question
No intervention
Move On End Task Instructional Remediation
Intervene
Give Elaborated Feedback
Worked Example
More Information
Modify Task
New Task With
Reduced Difficulty
Add Scaffolding
![Page 41: Serious Play Conference Los Angeles, CA – July 21, 2012 Girlie C. Delacruz and Ayesha L. Madni Setting Up Learning Objectives and Measurement for Game](https://reader038.vdocument.in/reader038/viewer/2022110207/56649d825503460f94a6751b/html5/thumbnails/41.jpg)