DEVELOPMENT OF THE 3D-‐LOP§ The Three-‐Dimensional Learning Observation Protocol (3D-‐LOP) will
characterize the three dimensions in teaching (in development) and what teaching activities are occurring in a class meeting (complete) (Matz, 2015).
§ Teaching activities: clicker questions, tasks, interactions, lecture, administration, and miscellaneous.
§ Preliminary analyses show that courses incorporating 3D questions on exams tend to be active, but active courses do not necessarily incorporate 3D questions on exams (Fig. 3).
DISCUSSION§ Coding exams with the 3D-‐LAP reveals a shift in how some faculty are
approaching assessments in gateway science courses.§ Different departmental cultures have contributed to, and help explain
why, some courses exhibit change and others do not: § Changes in introductory cell and molecular biology were
supported by a substantial, permanent increase in TA/LA funding.§ Changes in general chemistry were supported by the
department-‐wide adoption of the research-‐based CLUE curriculum (Cooper, 2013).
§ A dynamic system of interactions, going beyond resources provided for this project, have contributed to these outcomes, including faculty hires, related externally-‐funded projects, and demonstrated institutional commitment to STEM teaching and learning.
CONCLUSIONSWe have shown that assessments in some gateway courses increasingly reflect scientific practices and core ideas as the result of a system of transformation efforts at Michigan State University. Additional ongoing work includes studying persistence rates and the relationships between three-‐dimensional assessment, instruction, and student learning.
ACKNOWLEDGEMENTS: We thank the Association of American Universities, the Deans of the College of Natural Science, and all MSU faculty members who participated in disciplinary discussions and provided materials for this study.REFERENCES: (1) Cooper, M., & Klymkowsky, M. (2013). Journal of Chemical Education, 90(9), 1116–1122. (2) Cooper, M. M., et al. (2015). Science, 350(6258), 281-‐282. (3) Gwet, K. L. (2008). British Journal of Mathematical and Statistical Psychology, 61(1), 29-‐48. (4) Laverty, J. T., et al. (2016). PloSone, 11(9), e0162333. (5) Matz, R. L., et al. Paper presented at NARST, April 11-‐14, Chicago, IL. (6) National Research Council. (2012). Washington, DC: The National Academies.AFFILIATIONS: 1Hub for Innovation in Learning and Technology, 2Physics and Astronomy, 3Chemistry, 4STEM Transformation Institute, 5Plant Biology, 6Microbiology and Molecular Genetics, 7Biochemistry and Molecular Biology; aMichigan State University, bFlorida International University, cGrand Valley State University, dKansas State University
DATA COLLECTION & ANALYSIS – ASSESSMENTS§ We collected four years of midterm and final exams from all major
introductory biology, chemistry, and physics lecture courses (Table 1).§ In total, we analyzed 4,023 questions from 134 unique exams, fully
representing all 185 course sections of the eight relevant courses.
§ Disciplinary coding groups achieved acceptable inter-‐rater reliability as measured by Gwet’s AC1 (Gwet, 2008) and percent agreement.
RESULTS – ASSESSMENTSRQ: How has the fraction of assessment items that reflect scientific practices, crosscutting concepts, and core ideas changed over time?
Figure 2. Fraction of exam points that reflect the three dimensions over time in A) Bio I, B) Bio II, C) Chem I & II, and D) Phy-‐Algebra & -‐Calculus I and II. Points are scaled by the number of students who took each exam.
RQ: How have student grades and D-‐grade, F-‐grade, and withdrawal (DFW) rates changed in these courses of interest over time?
THREE-‐DIMENSIONAL LEARNINGA 2012 National Research Council report introduced the idea of three-‐dimensional learning as a vision for science and engineering education.§ Scientific practices: What scientists do, e.g. analyze and interpret data§ Crosscutting concepts: Concepts with common applications across
fields, e.g. structure and function§ Core ideas: Explanatory and generative disciplinary ideas, e.g.
evolution drives the diversity of life
GATEWAY COURSE TRANSFORMATIONThe goal of our transformation project is to develop and enact a shared vision for undergraduate gateway biology, chemistry, and physics courses by engaging faculty in disciplinary discussions about the three dimensions, among other activities (Cooper, 2015). We are characterizing changes in instruction (video observations) and assessments (exams).
Figure 1. Schematic of integrating the three dimensions.
DEVELOPMENT OF THE 3D-‐LAP§ The Three-‐Dimensional Learning Assessment Protocol (3D-‐LAP)
characterizes the potential for assessment items to elicit evidence of three-‐dimensional learning (Laverty, 2016).
§ Developed through iterations of literature review, discussion with disciplinary experts, and review of existing assessment items.
§ Criteria were developed for each scientific practice, crosscutting concept, and core idea, e.g., for analyzing and interpreting data:1. Question gives a scientific question, claim, or hypothesis to be
investigated.2. Question gives a representation of the data (e.g., table, graph, list of
observations) provided to answer the question or test the claim or hypothesis.
3. Question gives an analysis of the data or asks student to analyze the data.
4. Question asks student to interpret the results or assess the validity of the conclusions in the context of the question, claim, or hypothesis.
Characterizing Scientific Practices, Crosscutting Concepts, and Core Ideas in Science Assessments
Rebecca L. Matz1,a, Marcos D. Caballero2,a, Justin H. Carmel3,4,b, Diane Ebert-‐May5,a, Cori L. Fata-‐Hartley6,a, Debbie G. Herrington3,c, James T. Laverty2,d, Lynmarie A. Posey3,a, Ryan L. Stowe3,a, Jon R. Stoltzfus7,a, Sonia M. Underwood3,4,b, Melanie M. Cooper3,a
Table 1. Numbers of exams and questions coded with the 3D-‐LAP by discipline.
Discipline ExamsQuestions
Individual questions Clusters TotalBiology 68 2237 266 2503
Chemistry 32 718 93 811Physics 34 479 230 709Total 134 3434 589 4023
Table 2. Student outcomes: Average grades and DFW rates in introductory courses.
Approx. annual enrollment
Avg. Grade DFWYear 0 Year 3 Year 0 Year 3
Biology I (cell/mol) 2,400 2.3 2.6 31% 21%Biology II (org/pop) 1,000 3.0 3.0 11% 13%Chemistry 5,000 2.3 2.9 35% 17%Physics 4,500 3.0 3.0 8% 9%
