a process for analyzing coherence in science instructional ... · pdf filea process for...
Post on 16-Mar-2018
215 Views
Preview:
TRANSCRIPT
A Process for Analyzing Coherence in Science Instructional Materials Presenter: April Gardner
Collaborators: Joseph A. Taylor, BSCS; Rodger W. Bybee, BSCS; Liu Enshan, Beijing Normal University
2 May 2010 AERA Annual Meeting Denver, CO
Project Background
• Asia Society convened meetings of American and Chinese math and science education leaders in mid-2000s
• This group encouraged collaborative work to examine and compare curriculum standards and materials in the two countries
• In 2006, BSCS received a grant from the Office of Science Education at NIH to collaborate with Beijing Normal University biology educators to compare U.S. and Chinese biology textbooks
Initial project goals
Compare U.S. and Chinese secondary biology programs for: • Foundational concepts • Role of scientific inquiry • Role of educational technology
Refining the goals
Goals were refined and expanded based on our recognition that: • both the U.S. and China had developed and adopted national
science curriculum standards in the recent past (1996 for U.S. and 2003 for China),
• both countries had popular, pre-standards programs in use as well as less-popular, post-standards programs in use, and
• it would be interesting to compare not only distinctions between American and Chinese instructional materials, but also distinctions between pre- and post-standards curriculum materials.
An emphasis on curricular coherence
As we continued this work, we further recognized that: • international comparison studies identified curricular coherence as a
key distinction between high- and low-performing countries, • focus and rigor have been described as two components of overall
coherence (Schmidt et al., 2005; Schmidt & Prawat, 2006), and • focus and rigor are key features of science curriculum standards in
both the U.S. and China.
Thus, our emphasis changed to • analysis of the coherence (focus and rigor) of one pre-standards
and one post-standards biology program from each country
A more interesting story
When we completed this work, we recognized that we had developed and tested a method for analyzing the coherence of instructional materials, and
that’s the more interesting story I’ll relate today . . .
Four programs reviewed
Process for evaluating coherence: Background
• Adaptation of part of the Analyzing Instructional Materials (AIM) process (Powell et al., 2002)
• Uses criteria based on research about how students learn science • These criteria address both focus and rigor
Process for evaluating coherence: Analyzing focus
1. Construct and score a conceptual flow graphic (CFG) • Write the overarching concept
of chapter and major concept(s) of each section
• Depict strength of connections among these using arrows of varying thicknesses
• Score by giving each connecting link a score based on its strength: 2 for strong, 1 for moderate, and 0 for weak or no link
• CFG score = sum of all links, divided by sum if all links are strong
• Transform % score to a score of 5, 3, or 1
Example:
Percent Score
33% weakly focused
1
33-66% moderately focused
3
>66% strongly focused
5
Process for evaluating coherence: Analyzing focus, continued
2. Search instructional materials for evidence of focus based on sequencing, context, and key concepts.
3. Evaluate evidence for strengths & weaknesses, and score as 5, 3, or 1 (strongly, moderately, or weakly address criterion, respectively).
Criterion Abbreviated Description
Sequencing -organized to promote student understanding -links facts and concepts explicitly -builds & extends previously developed concepts -connects concepts strongly to overarching conceptual framework
Context -presented in engaging context related to real world experiences -facilitates assimilation/reorganization of knowledge that builds on students’ prior conceptions and experiences
Key Concepts -structured using global, unifying themes of biology -emphasizes connections across discipline areas
Process for evaluating coherence: Analyzing rigor
4. Search instructional materials for evidence of rigor based on depth of treatment, engagement of prior knowledge, student metacognition, and abilities and understandings of scientific inquiry.
5. Evaluate evidence for strengths & weaknesses, and score as 5, 3, or 1 as for the focus criterion.
Criterion Abbreviated Description Depth of treatment
-treat concepts at developmentally appropriate level -require students to apply & demonstrate understanding in >1 way
Engaging prior knowledge
-help students make current understanding of concept explicit -challenge/confront current thinking about concept
Metacognition -include strategies to help students assess their own learning -include strategies to help students reflect on what & how they learned
Abilities of sci. inquiry
-require students to design & conduct scientific investigations -require students to formulate & revise explanations based on data
Understandings of sci. inquiry
-help students understand that scientists conduct investigations for a variety of reasons -help students understand that scientists use a variety of tools, technology, & methods to enhance their investigations
Process for evaluating coherence: Scoring
6. Focus, rigor, and overall coherence scores determined as indicated in the table:
Component Criteria Possible Range
Low Moderate High
Focus CFG Sequencing
Context Key Concepts
4 – 20 4 – 8 9 – 15 16 – 20
Rigor Depth of treatment Prior knowledge Metacognition
Abilities of sci. inquiry Understandings of
sci. inquiry
5 – 25 5 – 11 12 – 18 19 – 25
Overall Coherence
Focus & Rigor Subscores
9 – 45 9 – 20 21 – 32 33 – 45
Research question
• Can the process we devised for assessing coherence distinguish instructional materials developed before the adoption of national science standards from those developed after the adoption of the standards?
Hypothesis: Science instructional materials developed after the adoption of national standards will exhibit greater levels of coherence than those developed before the adoption of national standards.
Procedure followed
1. Researchers were trained in the use of the modified AIM process using an unrelated program.
2. Project director selected the genetics and ecology chapter(s) to be analyzed from each of the four programs.
3. Three researchers independently reviewed the chapter(s), identified the overarching chapter concept and section concepts, and collected evidence to support their scores for the focus and rigor criteria.
4. The research team met to compare and reach consensus on each overarching chapter concept and the major section concepts.
5. The team then collaboratively constructed a CFG for each chapter. 6. The team discussed and revised scores on the focus and rigor
criteria to reach consensus on the scores.
Sample CFG: Program B 50% of possible strong links
Moderately focused (score = 3)
Sample CFG: Program D 89% of possible strong links Highly focused (score = 5)
Example of evaluating rigor for “understandings of scientific inquiry”
• Weak rigor (score = 1) – The chapter described few or no scientists and their work. There is no
opportunity for students to connect how the evidence of scientists’ investigations leads them to their conclusions.
• Strong rigor (score = 5) – The chapter described how the understanding of inheritance progressed
over time, through the ensuing work of scientists. For example, following a detailed description of Mendel’s experiments, results, and elucidation of the principles of inheritance, the text goes on to explain how some results of genetic crosses conducted by other scientists did not fit Mendel’s patterns. The text presents this puzzle as a rationale for the subsequent work of T.H. Morgan who related the inheritance of genes to newer understandings about chromosomes and their movement. Morgan’s work then led to further studies about gene linkage and mapping.
Findings
Program/type Country Average Focus Score/Level
Average Rigor Score/Level
Overall Coherence
A/pre-standards China 12/Moderate 7/Low 19/Low
B/pre-standards USA 14/Moderate 11/Low 25/Moderate
C/post-standards China 14/Moderate 11/Low 25/Moderate
D/post-standards USA 20/High 23/High 43/High
• The process distinguished ecology chapters from the programs developed before the adoption of national science standards from the U.S. program developed after the adoption of standards, based on their levels of coherence.
• The process did not distinguish the U.S. ecology chapters developed before national standards from the Chinese ecology chapters developed after the national standards were adopted.
• Programs developed after the adoption of national science standards exhibited greater coherence than those developed before national standards, for both China and the U.S.
Ecology chapters:
Findings Program/type Country Average Focus
Score/Level Average Rigor
Score/Level Overall
Coherence A/pre-standards China 10/Moderate 9/Low 19/Low
B/pre-standards USA 4/Low 9/Low 13/Low
C/post-standards China 18/High 17/Moderate 35/High
D/post-standards USA 17/High 21/High 38/High
• The process distinguished genetics chapters from programs developed before and after the adoption of national science standards, based on their levels of coherence.
• Programs developed after the adoption of national science standards exhibited greater coherence than those developed before national standards, for both China and the U.S.
Genetics chapters:
Conclusions
• This adaptation of the AIM process is a promising method for evaluating the coherence of science instructional materials.
• This method can be used in research studies, curriculum development, and professional development. For example, it could be used: – to study the relationship between the coherence of instructional
materials and specific student learning outcomes, – by curriculum developers to strengthen the coherence of
materials they are developing, and – in curriculum-based professional development to identify areas
in programs where coherence should be strengthened by the teacher.
References cited on slides
Powell, J. C., Short, J. B. and Landes, N. M. (2002) Curriculum reform, professional development, and powerful learning. In R. W. Bybee (ed.), Learning Science and the Science of Learning (Arlington, VA: NSTA Press), 121 -- 136.
Schmidt, W. H. and Prawat, R. S. (2006) Curriculum coherence and national control of education: issue or non-issue? Journal of Curriculum Studies, 38 (6), 641 -- 658.
Schmidt, W. H., Wang, H. C. and McKnight, C. C. (2005) Curriculum coherence: an examination of US mathematics and science content standards from an international perspective. Journal of Curriculum Studies, 37 (5), 525 -- 559.
5415 Mark Dabling Blvd. Colorado Springs, CO 80918 info@bscs.org 719.531.5550
To download this presentation, visit www.bscs.org/sessions
top related