design talk as a way to assess student learning of practices janet l. kolodner jennifer holbrook,...
TRANSCRIPT
Design Talk as a Way to Design Talk as a Way to Assess Student Learning of Assess Student Learning of
PracticesPractices
Janet L. KolodnerJanet L. Kolodner
Jennifer Holbrook, Jackie Gray, Jennifer Holbrook, Jackie Gray, Paul J. CampPaul J. Camp
College of ComputingCollege of Computing
Georgia Institute of TechnologyGeorgia Institute of Technology
Science PracticesScience Practices
Understanding a problem what might need to be Understanding a problem what might need to be investigatedinvestigated
Investigation with a purpose -- experimentation, Investigation with a purpose -- experimentation, modeling, learning from cases, ...modeling, learning from cases, ...
Informed decision making, reporting on and Informed decision making, reporting on and justifying conclusionsjustifying conclusions
Iteration towards understandingIteration towards understanding Explaining scientificallyExplaining scientifically Teamwork, collaboration across teams, giving creditTeamwork, collaboration across teams, giving credit
Design PracticesDesign Practices
Identifying criteria, constraints, problem Identifying criteria, constraints, problem specificationspecification
““Messing about” with materialsMessing about” with materials Investigation for the purpose of applicationInvestigation for the purpose of application Informed decision making, reporting on and Informed decision making, reporting on and
justifying design decisionsjustifying design decisions Tradeoff and optimizationTradeoff and optimization Iteration towards a good enough solutionIteration towards a good enough solution Explaining failures and refining solutionsExplaining failures and refining solutions Teamwork, collaboration across teams, giving creditTeamwork, collaboration across teams, giving credit
Learning by Design™Learning by Design™
A project-based inquiry approach to science A project-based inquiry approach to science education for middle schooleducation for middle school
Students learn science and technology Students learn science and technology concepts and practices in the context of concepts and practices in the context of attempting to achieve design challenges.attempting to achieve design challenges.
Highly collaborativeHighly collaborative A variety of “community rituals” are A variety of “community rituals” are
embedded in the approach to promote embedded in the approach to promote learning of science and design practices. learning of science and design practices.
Our UnitsOur Units
Physical SciencePhysical Science– Apollo 13Apollo 13 – introduction to practices of design and – introduction to practices of design and
sciencescience– Vehicles in MotionVehicles in Motion – motion and forces – motion and forces– Machines that HelpMachines that Help – simple machines and mechanical – simple machines and mechanical
advantageadvantage
Earth ScienceEarth Science– Digging InDigging In -- introduction to practices -- introduction to practices– Managing Erosion Managing Erosion –– erosion and accretion erosion and accretion– Tunneling through GeorgiaTunneling through Georgia – geology, rocks and – geology, rocks and
minerals, rock formations, underground waterminerals, rock formations, underground water
A Typical Design CycleA Typical Design Cycle Challenge is presentedChallenge is presented Students Students mess aboutmess about to generate questions for inquiry to generate questions for inquiry Investigation to address questions followed by Investigation to address questions followed by
presentation of results to classpresentation of results to class Design planningDesign planning Pin-up sessionPin-up session Construction & testingConstruction & testing Gallery walkGallery walk Additional investigation, demo, reading, discussion of Additional investigation, demo, reading, discussion of
contentcontent IterationIteration over last three steps to solution over last three steps to solution
Two LBD™ community ritualsTwo LBD™ community rituals
Gallery walks Gallery walks (explanation and presentation (explanation and presentation of results)of results)
Pin-up sessions Pin-up sessions (justification)(justification)
Ritualized public ways of participating in Ritualized public ways of participating in science and design practicesscience and design practices
Well-articulated expectationsWell-articulated expectations Repeatedly practiced and publicly discussedRepeatedly practiced and publicly discussed
Research GoalsResearch Goals
To what extent are students learning targeted To what extent are students learning targeted content, and what is responsible for content, and what is responsible for differences?differences?
To what extent are students learning targeted To what extent are students learning targeted practices, and what is responsible for practices, and what is responsible for differences?differences?
How do student conceptions and capabilities How do student conceptions and capabilities develop over time?develop over time?
What practices by teachers are most What practices by teachers are most conducive to promoting learning?conducive to promoting learning?
What’s needed to makequalitative/observational data trustworthy?
• Focu s infor medb y theoretica l perspective• Consistenc y acros s environment s an d a wa y of
managin g reliability• Carefu l planning• Analysi s informe d by expectation – s looki ng for
confirmin g and disconfirmin g evidence• We -ll controlled quantitativ e result s t o bac k up
observations• Collectio n o f qualitative dat a tha t ca n be analyzed
quantitatively using developmenta l coding rubrics• Severa l source s of dat a tha t ca n be triangulated
Assessment MethodologiesAssessment Methodologies
Content Test Pre/PostContent Test Pre/Post– multiple choice & some essaymultiple choice & some essay– what they know and development of their understandingwhat they know and development of their understanding
20 Questions20 Questions– self-assessment of capabilitiesself-assessment of capabilities
Videotaped Performance AssessmentsVideotaped Performance Assessments– design an experiment, run an experiment, analyze datadesign an experiment, run an experiment, analyze data
Videotaped Structured InterviewsVideotaped Structured Interviews– interactive design, experiment design, & performanceinteractive design, experiment design, & performance
Ethnography & Informal DataEthnography & Informal Data
Performance AssessmentsPerformance Assessments
Ask students to carry out practices in the context Ask students to carry out practices in the context of an authentic, but short performance task.of an authentic, but short performance task.
Have students work in teams so as to be able to Have students work in teams so as to be able to record their deliberations.record their deliberations.
Video deliberations.Video deliberations. Code written products and team interactions for Code written products and team interactions for
evidence of participation in practices.evidence of participation in practices. Repeat periodically throughout experimental Repeat periodically throughout experimental
period.period.
1Group Members_____________________________________________________
_____________________________________________________
Teacher ___________________________ Block/Period___________
Group Number ______ Date___________
Where the Rubber Meets the RoadGroup Problem Solving Activity
PART I: Design and Describe a Simple Experiment in 15 M inutes or Less
Race cars and large trucks have very different types of t ires. Different kinds of rubber requiredifferent amounts of force to overcome sliding friction. In this activity, your group will try tof igure out and describe a simple experiment that tests the amount of force needed to overcomesliding friction for different types of rubber and under different road conditions. You onl y have
15 minutes from right now to f inish. Everyone read the instructions together and begi n.
INSTRUCTIONS
(1) Examine the rubber block that you were given. I t has hard rubber on one side and soft
rubber on the other side.(2) Work together as a group to design and describe a simple experi ment that compares the
amounts of force needed to overcome slidi ng fr iction for both kinds of rubber.(3) Your experiment should compare sliding friction for the two types of rubber under different
conditi ons that might be found on a road surface.
(4) On the next page, write your experiment description:(a) Select a person in your group to write down the experiment description. This personshould use black pen, and should write neatly. However, all members of the group areresponsible for what is written.
(3) On the next page, write the experiment description.
(4) Do not write on the back of the sheet; raise your hand and request an extra sheet, and writethe names of your group and “page 2” on this page.
(5) Give enough detail so that someone reading what you write wi ll know:1. What you are trying to f ind out
1 Holbrook/LBD Page 1 4/7/01
PART I: Design and Describe a Simple Experiment in PART I: Design and Describe a Simple Experiment in 15 Minutes 15 Minutes Race cars and large trucks have very different types of Race cars and large trucks have very different types of tires. Different kinds of rubber require different tires. Different kinds of rubber require different amounts of force to overcome sliding friction. In this amounts of force to overcome sliding friction. In this activity, your group will try to figure out and describe a activity, your group will try to figure out and describe a simple experiment that tests the amount of force simple experiment that tests the amount of force needed to overcome sliding friction for different types needed to overcome sliding friction for different types of rubber and under different road conditions. You of rubber and under different road conditions. You have 15 minutes. Read the instructions together and have 15 minutes. Read the instructions together and begin.begin.
INSTRUCTIONSINSTRUCTIONSExamine the rubber block. It has hard rubber on one Examine the rubber block. It has hard rubber on one side and soft rubber on the other side. side and soft rubber on the other side. Work together to design and describe a simple Work together to design and describe a simple experiment that compares the amount of force needed experiment that compares the amount of force needed to overcome sliding friction for each kind of rubber.to overcome sliding friction for each kind of rubber.Your experiment should look at sliding friction for the Your experiment should look at sliding friction for the two types of rubber under several different conditions two types of rubber under several different conditions that might be found on a road surface. that might be found on a road surface. Describe your experiment on the next page.Describe your experiment on the next page.
Coding CategoriesCoding Categories
Negotiation during collaborationNegotiation during collaboration Distribution of the taskDistribution of the task Access to prior knowledgeAccess to prior knowledge Adequacy of prior knowledgeAdequacy of prior knowledge Science/design talkScience/design talk Science practice (experiment design)Science practice (experiment design) Self checksSelf checks
Scoring the Quality of Scoring the Quality of PracticesPractices
Characterizing the Characterizing the practicepractice– 1 - didn’t do it1 - didn’t do it– 2 – recognized need; 2 – recognized need;
minimal abilityminimal ability– 3 – novice level3 – novice level– 4 – intermediate 4 – intermediate
levellevel– 5 – expert level5 – expert level
Characterizing group Characterizing group behaviorbehavior– 1 – nobody did it1 – nobody did it– 2 – one person 2 – one person
recognized need and recognized need and triedtried
– 3 – half the group 3 – half the group triedtried
– 4 – done well half the 4 – done well half the timetime
– 5 – group fluency 5 – group fluency
Scoring the quality of Scoring the quality of practicespractices
A 5 -point likert scale for each coding A 5 -point likert scale for each coding category reflects a quantitative continuum.category reflects a quantitative continuum. – 1 = Not at all: no evidence of the quality to be 1 = Not at all: no evidence of the quality to be
ratedrated– 2 = Some evidence that at least one episode or 2 = Some evidence that at least one episode or
one student exhibits the quality ratedone student exhibits the quality rated– 3 = The quality is exhibited by half the group3 = The quality is exhibited by half the group– 4 = The quality is exhibited for more than half 4 = The quality is exhibited for more than half
the episodesthe episodes– 5 = The quality completely captures the nature 5 = The quality completely captures the nature
of the episodesof the episodes
Performance Assessment tasks: Coding for science practice
Jackie Gray, Paul Camp, Jennifer Holbrook, Barbara Fasse, and Janet Kolodner
Additional notes are fine and can be recorded on the coding sheet.Please note which event segment is being coded for each episode:
planning an experiment; problem set up; experimental manipulation; response
to written questions.In general, the 5 -point likert scale reflects the following quantitative
continuum. Details for each item are also included below.1 = Not at all: no evidence of the quality to be rated
2 = Some evidence that at least one episode or one student exhibits the
quality rated3 = The quality is exhibited half the time
4 = The quality is exhibited for more than half the episodes5 = The quality completely captures the nature of the episodes
Design an experiment segment:
Within an episode, the context of the group is characterized by:
Negotiations
Not at all at least one of the
members of the
group suggests acompromise about
some aspect of the
procedure
at least one of the
members of the
group suggests thatcompromise or
debate is needed
for at least half the
issues that require
it
at least two of the
members of the
group questionsseveral aspect of
the procedure and
the group makes
the needed change
Most decisions are
made about
procedure by theentire team
contributing and
decision making is
consensual
1 2 3 4 5
Prior knowledge is defined as students referring to some aspect of the curriculum unitthat relates to the current problem; referring to some aspect of a personal experience
that seems to relate to the current problem; referring to some aspect of the scienceconcept or method at issue that appears to come from previous exposure to the concept
or skill.
Students show evidence of using prior knowledge to solve the problem
Not at all at least one of the
members of the
group mentions aprior event or
concept that relates
to the problem
at least half of the
team mentions a
prior event orconcept that relates
to the problem
Several events and
concepts are
mentioned andapplied to the
problem
The group
routinely recalls
events or conceptsthat assist in their
collaborative
problem solving
1 2 3 4 5
Prior knowledge seems adequate
Not at all at least one of the
mentions of priorknowledge is
followed up on and
is useful
At least half the
mentions of priorknowledge are
appropriate to the
problem
More than one
member of thegroup mentions or
follows up on
events or concepts
that are useful
Every mention of
prior knowledge isdirectly applicable
to the problem
1 2 3 4 5
Students use science practice to decide on method/procedures
Not at all at least one of the
members of the
group suggest a
method to test at
least one variable
at least one of the
members suggest a
method and
indicates an
understanding of
fair testing
at least one of the
members suggest a
method and
indicates an
understanding of
fair testing andcontrolling for
variables
Most of the team
agrees that the
method used will
fairly test the
important variables
and their decisionswould actually be
a reasonable
experiment
1 2 3 4 5
The episodes are characterized by group self-checks on procedures
Not at all at least one of the
members of the
group questions
some aspect of theprocedure
at least one of the
members of the
group questions
some aspect of theprocedure and the
makes the needed
change
at least one of the
members of the
group questions
several aspect ofthe procedure and
the group makes
the needed change
More than one of
the members of the
group questions
several aspect ofthe procedure and
the group makes
the needed change
1 2 3 4 5
Table 1:Means and standard deviations for categories from performance assessment coding for
LBD students (typical and honors) and Comparison students (typical and honors)
Coding
category
Means (SD)
LBD Typical
Means (SD)
Comparison
Typical
Means (SD)
LBD Honors
Means (SD)
Honors
Comparison
Negotiations 2.50 (1.00) 1.50 (.58) 4.50 (.58) *** 2.67 (.58)
DistributedEffort/tasks
3.25 (.50) * 2.25 (.50) 4.00 (1.15) 3.00 (1.00)
Prior
knowledge
2.25 (.50) `1.75 (.50) 3.75 (1.50) 3.0 (.00)
Prior
Knowledgeadequate
2.75 (.96) 1.50 (.58) 3.50 (1.00) 2.67 (1.15)
Science terms
used
2.50 (1.29) 1.75 (.50) 3.50 (1.00) 2.67 (1.15)
Science
practice skills
2.75 (.96) 2.25 (.50) 4.75 (.50) *** 2.67 (.71)
Self-checks 3.00 (.82) ** 1.50 (.58) 4.25 (.50) *** 2.33 (.58)
Significance levels: * = p < .03; ** = p < .02; *** = p < .01The means are based on the likert scale: 1 - 5
Putting it all togetherPutting it all together Code videos of performance tasks so as to document Code videos of performance tasks so as to document
performance capabilities of students over time.performance capabilities of students over time. Use documentation of teacher practices to identify Use documentation of teacher practices to identify
high, medium, and low-fidelity LBD classes and the high, medium, and low-fidelity LBD classes and the practices of particular teachers.practices of particular teachers.
Analyze across high-, medium-, and low-fidelity LBD Analyze across high-, medium-, and low-fidelity LBD classrooms and between LBD and non-LBD classrooms and between LBD and non-LBD classrooms to see differences.classrooms to see differences.
Use observational data to try to explain those Use observational data to try to explain those differences in terms of the ways the classroom differences in terms of the ways the classroom practices and culture and teacher facilitation differed.practices and culture and teacher facilitation differed.
Making Observations Consistent: Making Observations Consistent: Observational Prompt ToolObservational Prompt Tool
Sections focus onSections focus on Individual, small-group, whole-class Individual, small-group, whole-class
activities -- what to look for in eachactivities -- what to look for in each LBD rituals – gallery walks, messing LBD rituals – gallery walks, messing
about, wbabout, wb what to look for in student and teacher what to look for in student and teacher
behavior when certain goals are active –behavior when certain goals are active –generating questions for inquiry, focusing generating questions for inquiry, focusing investigation, debugging, construction, …investigation, debugging, construction, …
OPT Sample PromptsOPT Sample Prompts
QuestioningQuestioning– What are teacher questions What are teacher questions
about?about?– What are student questions What are student questions
about?about?– What question types are being What question types are being
used?used?– Purpose of teacher questions?Purpose of teacher questions?
– How does teacher deal with off-How does teacher deal with off-topic questions?topic questions?
– (each question has a menu of (each question has a menu of types and a set of examples types and a set of examples associated)associated)
Gallery WalksGallery Walks – Who initiates the session?Who initiates the session?
– Who displays the artifact?Who displays the artifact?
– Who asks questions?Who asks questions?
– Who gives feedback?Who gives feedback?
– In what ways is feedback In what ways is feedback constructive?constructive?
– What comparisons are made What comparisons are made between groups or to previous between groups or to previous work of the presenting group?work of the presenting group?
– To what extent are students To what extent are students being explicit about design being explicit about design decisions, expectations, decisions, expectations, explanations?explanations?
LBD’s FoundationsLBD’s Foundations
Case-based reasoning’s model of learning from Case-based reasoning’s model of learning from experience (Kolodner, Schank, Hammond, …)experience (Kolodner, Schank, Hammond, …)
Problem-Based Learning’s model of the classroom Problem-Based Learning’s model of the classroom (Barrows, …)(Barrows, …)
Communities of Learners (Brown, Campione), Communities of Learners (Brown, Campione), Constructionism (Papert, Harel, Kafai, …), Constructionism (Papert, Harel, Kafai, …), Cognitive Apprenticeship (Collins, Brown, …), Cognitive Apprenticeship (Collins, Brown, …), architecture studio, Decision-Based Design architecture studio, Decision-Based Design (Mistree, …), transfer, development, ...(Mistree, …), transfer, development, ...
A project-based inquiry approach with more.A project-based inquiry approach with more.