assessment what are the questions? what evidence will we accept? diane ebert-may department of plant...
TRANSCRIPT
Assessment
What are the questions?What evidence will we
accept?
Diane Ebert-MayDepartment of Plant Biology
Michigan State University
[email protected]://first2.org
The trouble with our times is that the future is not what it
used to be. -Paul Valery, The Art of Poetry
Question 1
Students learn science best by doing science.
Please respond on a scale of 1-5: 1=strongly agree; 2=agree; 3=neutral; 4= disagree;
5=strongly disagree
Question 2
Science should be taught as it is practiced.
Please respond on a scale of 1-5: 1=strongly agree; 2=agree; 3=neutral; 4= disagree;
5=strongly disagree
Question 3
How important is it to use multiple kinds of data to assess student learning?
Please respond on a scale if 0-100 in increments of 10:
How important is it to use multiple forms of data to assess student learning?
%
Relative Importance n=127
Question 4
How often do you use data to make instructional decisions?
Please respond on a scale of 0 - 100 in increments of 10:
Question 5
Large (+200) introductory science/engr lectures are active learning environments.
Please respond on a scale of 1-5: 1=strongly agree; 2=agree; 3=neutral; 4= disagree;
5=strongly disagree
Question 8
True or False?
Assessing student learning in science parallels what scientists do as researchers.
1.Description:
-What is happening?
2.Cause:
-Does ‘x’ (teaching strategy) affect ‘y’ (understanding)?
3.Process or mechanism:
-Why or how does ‘x’ cause ‘y’?
Parallel: ask questions
We collect data to find out what our students know.
Data helps us understand student thinking about concepts and content.
We use data to guide decisions about course/curriculum/innovative instruction
Parallel: collect data
Quantitative data - statistical analysis
Qualitative data
break into manageable units and define coding categories
search for patterns, quantify
interpret and synthesize
Valid and repeatable measures
Parallel: analyze data
What is assessment?
Data collection with the purpose of answering questions about…
students’ understanding
students’ attitudes
students’ skills
instructional design and implementation
curricular reform (at multiple grainsizes)
Why do assessment?
Improve student learning and development.
Provides students and facultysubstantive feedback about student understanding.
Challenge to use disciplinary research strategies to assess learning.
Multiple Choice … … Concept Maps … … Essay … … Interview
high Ease of Assessment low
low Potential for Assessment of Learning high
Theoretical Framework• Ausubel 1968; meaningful learning• Novak 1998; visual representations• King and Kitchner 1994; reflective judgment• National Research Council 1999; theoretical frameworks for assessment
Assessment Gradient
“Few faculty members have any awareness of the expanding knowledge about learning from psychology and cognitive science. Almost no one in the academy has mastered or used this knowledge base. One of my colleagues observed that if doctors used science the way college teachers do, they would still be trying to heal with leeches."
J.J. Duderstadt (2001), president emeritus of the University of Michigan, in "A University for the 21st Century."
Does active, inquiry-based instructional design influence students’ understanding of evolution and natural selection?
Pre-Posttest Analysis
■ Changes in a population occur through a gradual change in individual members of a population.
■ New traits in species are developed in response to need.
■ All members of a population are genetically equivalent, variation and fitness are not considered.
■ Traits acquired during an individual’s lifetime will be inherited by offspring.
Alternative Conceptions: Natural Selection
(AAAS 1999)
Explain the changes that occurred in the tree and animal. Use your current understanding of evolution by natural selection.
Misconception: individuals evolve new traitsMisconception: individuals evolve new traits
% o
f S
tud
ents
n=80; p<.01
Misconception: evolution is driven by needMisconception: evolution is driven by need
% o
f S
tud
ents
n=80; p<.01
In guppy populations, what are the primary changes that occur gradually over time?
In guppy populations, what are the primary changes that occur gradually over time?
a.The traits of each individual guppy within a population gradually change.
b.The proportions of guppies having different traits within a population change.
c.Successful behaviors learned by certain guppies are passed on to offspring.
d.Mutations occur to meet the needs of the guppies as the environment changes.
Anderson et al 2002
Animal/Tree Posttest: Gain in student understanding of fitnessAnimal/Tree Posttest: Gain in student understanding of fitness
% o
f S
tud
ents
n=80; p<.01
How do assessment questions help us determine students’ prior understanding and progressive thinking about the carbon cycle.
Question
Instructional Design
• Two class meetings on carbon cycle (160 minutes)
• Active, inquiry-based learning– Cooperative groups– Questions, group processing, large lecture
sections, small discussion sections, multi-week laboratory investigation
– Homework problems including web-based modules
• Different faculty for each course– One graduate/8-10 undergraduate TAs per
course
Experimental DesignTwo introductory courses for majors:
Bio 1 - organismal/population biology (faculty A)Bio 2 - cell and molecular biology (faculty B)
Three cohorts:Cohort 1 Bio 1 (n=141)Cohort 2 Bio1/Bio2 (n=63)
Cohort 3 Other/Bio2 (n=40)
Assessment DesignMultiple iterations/versions of the carbon cycle problem Pretest, midterm, final with additional formative assessments during classAdministered during instructionSemester 1 - pretest, midterm, final exam Semester 2 - final exam
Grandma Johnson Problem Hypothetical scenario: Grandma Johnson
had very sentimental feelings toward Johnson Canyon, Utah, where she and her late husband had honeymooned long ago. Her feelings toward this spot were such that upon her death she requested to be buried under a creosote bush overlooking the canyon. Trace the path of a carbon atom from Grandma Johnson’s remains to where it could become part of a coyote. NOTE: the coyote will not dig up Grandma Johnson and consume any of her remains.
Analysis of Responses
Used same scoring rubric (coding scheme) for all three problems - calibrated by adding additional criteria when necessary, rescoring:
Examined two major concepts: Concept 1: Decomposers respire CO2
Concept 2: Plants uptake of CO2
Explanations categorized into two groups:Organisms (trophic levels)Processes (metabolic)
Code Organisms Code Processes and pathways 1 Decomposers IA Cellular Respiration IB Release CO2 2 IIA Pathway of Carbon
Primary producers IIA _1: through Air IIA _2 : through Root IIA _3 : no mention about pathway IIB Make Glucose IIC Photosynthesis 3 Herbivore III Respiration
(glycolysis, Kreb cycle) 4 Carnivore IV Respiration
(glycolysis, Kreb cycle)
Coding Scheme
Corr
ect
Stu
den
t R
esp
on
ses
(%)
Cellular Respiration by Decomposers
Bio1/Bio2 Other/Bio2
Friedmans, p<0.01
Pathway of Carbon in Photosynthesis
Bio1/Bio2
Corr
ect
Stu
dent
Resp
on
ses
(%)
Other/Bio2
Friedmans, p<0.05
IRD Team at MSUJanet Batzli - Plant Biology [U of Wisconsin]Doug Luckie - PhysiologyScott Harrison - Microbiology (grad student)Tammy Long - Plant BiologyJim Smith - ZoologyDeb Linton - Plant Biology (postdoc)Heejun Lim - Chemistry EducationDuncan Sibley - Geology*National Science Foundation