assessment in general education: a case study in scientific and quantitative reasoning b.j. miller...
TRANSCRIPT
![Page 1: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/1.jpg)
Assessment in General Education: A Case Study in Scientific and
Quantitative Reasoning
B.J. Miller & Donna L. SundreCenter for Assessment and Research Studies
James Madison University
http://www.jmu.edu/assessment/
![Page 2: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/2.jpg)
The First Habit
#1 Be proactive
![Page 3: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/3.jpg)
A Proactive Assessment Model
• Provides for program improvement
• Fulfills accountability requirements
![Page 4: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/4.jpg)
Preview
• JMU assessment model Background Five stages
• Assessment in Science & Math Methods Results Discussion
![Page 5: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/5.jpg)
James Madison University
• Public• Harrisonburg, VA• Enrollment
15,000 undergraduate 700 graduate
• Center for Assessment and Research Studies (CARS)
![Page 6: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/6.jpg)
General Education at JMUCluster Credits
1: Skills for the 21st Century 9
2: Arts and Humanities 9
3: The Natural World 10
4: Social and Cultural Processes 7
5: Individuals in the Human Community 6
Critical Thinking, Written and Oral Communication, Information Literacy
Culture, Philosophy, Fine Arts and Literature
Problem-solving Skills in Science and Mathematics
Global and American: History, Government, Economics, Anthropology
Wellness, Psychology, and Sociology
![Page 7: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/7.jpg)
Example from Cluster ThreeMATH 103. The Nature of Mathematics MATH 107. Fundamentals of Mathematics I MATH 205. Introductory Calculus I MATH 220. Elementary Statistics MATH 231. Calculus with Functions I MATH 235. Calculus I
GSCI 101. Physics, Chemistry, & the Human Experience
GSCI 102. Environment Earth GSCI 103. Discovering Life
GSCI 104. Scientific Perspectives
Choose one
Choose one
![Page 8: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/8.jpg)
Stages of the Assessment Process
EstablishingObjectives
Selecting/Designing
Instruments
CollectingData
AnalyzingData
UsingResults Continuous Cycle
![Page 9: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/9.jpg)
Definition of Assessment
“…the systematic basis for making inferences about student learning and development” (Erwin, 1991, p. 28).
![Page 10: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/10.jpg)
Establishing Objectives
• Expected/intended student outcomes
• Good objectives: Student-oriented Observable Measurable Reasonable
• Drive the assessment process
![Page 11: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/11.jpg)
Examples from Cluster Three
Use graphical, symbolic, and numerical methods to analyze, organize, and interpret natural phenomenon.
Discriminate between association and causation, and identify the types of evidence used to establish causation.
Formulate hypotheses, identify relevant variables, and design experiments to test hypotheses.
![Page 12: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/12.jpg)
Selecting / Designing Instruments
• Selecting an instrument Other universities Resources such as MMY
and TIP
• Considerations Does it match my objectives? How much will it cost? How do I administer it? Are the scores reliable and valid?
![Page 13: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/13.jpg)
Selecting / Designing Instruments
• Designing an instrument Table of specifications Item pool Pilot test Item analysis Item revision Pilot test again Reliability and validity
Learning Objective
# and % of
Items
.
.
.
Use graphical, symbolic, and numerical methods to analyze, organize, and interpret natural phenomenon.
1821%
Discriminate between association and causation, and identify the types of evidence used to establish causation.
911%
Formulate hypotheses, identify relevant variables, and design experiments to test hypotheses.
1214%
.
.
.
Total test 85 items
![Page 14: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/14.jpg)
Collecting Data• Who?• When?• How?• Why?
Assessment Days at JMU
• All students
• August and February
• P-&-P, Computer-based
![Page 15: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/15.jpg)
Assessment Day Data Collection Plan
COHORT 1
COHORT 2
COHORT 3
Repeated MeasuresRepeated Measures
Students in each cohort Students in each cohort are tested are tested twicetwice on the on the same instrument – once same instrument – once as as entering freshmenentering freshmen and again in the second and again in the second semester of thesemester of the sophomoresophomore year. year.
Spring 2003
Fall 2003Spring 2004
Fall 2004Spring 2005
Fall 2005Spring 2006
Fall 2002
![Page 16: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/16.jpg)
Analyzing Data
• Research questions Are there group differences? Do scores change over time? Are there expected relationships? Are students meeting expectations?
![Page 17: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/17.jpg)
Maintaining Data
• Organize and archive• Trends emerge• Informs future assessment
![Page 18: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/18.jpg)
Using Results
• Accountability
Allocating resources
Enhancing reputation
Compliance
![Page 19: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/19.jpg)
Using Results
• Program improvement
Curriculum
Teaching
Sequence of course offerings
![Page 20: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/20.jpg)
Using Results
• Benefits Faculty Students Institution
![Page 21: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/21.jpg)
Assessment in Cluster Three
![Page 22: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/22.jpg)
Measures
• NAW-5 Created by Cluster Three faculty 50-item multiple-choice test 12 of 17 learning objectives Reliability = .75
![Page 23: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/23.jpg)
Measures
• SOS Examinee motivation 10-item rating scale Two dimensions
• Effort• Importance
![Page 24: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/24.jpg)
Participants
• August, 2001 746 freshmen
• February 2003 316 sophomores
![Page 25: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/25.jpg)
Research Questions and Data Analysis
• Do scores change over time?
• Does number of courses impact scores?
• Does motivation impact scores?
• Paired samples t-test
• ANOVA
• Multiple regression
![Page 26: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/26.jpg)
Results: Question 1
• Do NAW-5 scores change over time? MeanDiff = 2.6, t(315) = 3.96, p < .001, d = .23
NAW-5 Pre -test and Post-test M ean Scores and 95% CI
50
55
60
65
Fall '01 Spr '03
NA
W-5
Me
an
Sc
ore
![Page 27: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/27.jpg)
Results: Question 2• Does the number of courses completed
impact NAW-5 scores? F(4, 315) = 1.145, p = .335
NAW-5 Mean Score and 95% CI by Number of Cluster Three Courses Completed
40
45
50
55
60
65
None One Two Three Four or more
Number of Cluster Three Courses Completed
NA
W-5
Mea
n S
core
![Page 28: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/28.jpg)
Results: Question 3
• Does examinee motivation have an impact on NAW-5 scores? Pre-test scores explain 22%
Motivation scores significant improvement in predictive utilityR2
change = .11, Fchange(2, 312) = 25.70, p < .001
Squared semi-partialsPre-test = .24 Effort = .07 Importance = .01
![Page 29: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/29.jpg)
Discussion
• Scores increased from FR to SO year• Number of courses had no impact• More effort, higher NAW-5 scores
• Using results Program improvement Accountability
![Page 30: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/30.jpg)
Using Results for Program Improvement
• Cluster Three objectives• NAW-8
• Better alignment of curriculum with learning objectives
• Test with full and balanced coverage, better reliability
![Page 31: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/31.jpg)
Using Results for Accountability
• Scientific and Quantitative Reasoning
• Assessment of core competency• Marketable instrument
![Page 32: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/32.jpg)
Conclusion
• Proactive assessment works for JMU1. Establish objectives2. Design instruments3. Collect data4. Analyze data5. Use results
• Assessment in Cluster Three
1
2
34
5
![Page 33: Assessment in General Education: A Case Study in Scientific and Quantitative Reasoning B.J. Miller & Donna L. Sundre Center for Assessment and Research](https://reader034.vdocument.in/reader034/viewer/2022051417/56649ebd5503460f94bc5ec8/html5/thumbnails/33.jpg)
Final Thoughts
• Stimulating• Meaningful• Challenging• FUN!
If you’re not having fun,
you’re doing it wrong!