sizing up your students, part 3: designing effective evaluations marti atkinson and bill doherty...
TRANSCRIPT
![Page 1: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/1.jpg)
Sizing up Your Students, Part 3:
Designing Effective Evaluations
Marti Atkinson and Bill DohertyApril 10, 2008
For audio call Toll Free 1-888-886-3951
and use PIN/code 240228
Parts of this presentation were adapted from materials created by:Noelle C. Griffin PhD, Assistant Director, UCLA Center for the Study of Evaluation
Original material can be found at: http://www.lmu.edu/Page18044.aspx
![Page 2: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/2.jpg)
Housekeeping
• Maximize your CCC Confer window.• Mute your phone (*6) if you have visitors or noise in your office.• Please note phone audio may be in presenter-only mode.• Ask questions over the phone when the presenter prompts. • Ask questions throughout presentation via the chat window.• Turn on or off Closed Captioning by clicking on the icon.• Save the presentation or chat by clicking on the icon.
![Page 3: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/3.jpg)
AgendaBackground and Terms
Planning your Evaluation Strategy - Defining your Goals, Objectives & Outcomes
- Choosing your Measurements
- Online Quizzes and Exams - Performance based assessments
Analyzing data – rubrics
![Page 4: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/4.jpg)
Background and TermsEnormous body of literature on this subject
Download list of resources at
http://nlight.com/ONE/evalsSurveys_resources.doc
![Page 5: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/5.jpg)
Background and Terms
Some Basics
Assessments versus evaluations
Reliability - repeated measurements are same or very similar
Validity – you are measuring what you say you are
![Page 6: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/6.jpg)
Background and Terms
Kinds of Measures
Quantitative: generates numeric data
Qualitative: generates descriptive data
![Page 7: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/7.jpg)
Background and Terms
Collection Methods
Survey, Quizzes, and Exams
Interviews
Observations
Work Products
![Page 8: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/8.jpg)
Background and Terms
Authentic – Performance & Behavioral
Assignments
Projects – individual and group
Behavioral observations– General Participation– Discussion contributions
Portfolio
![Page 9: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/9.jpg)
Background and Terms
Triangulation of Data
![Page 10: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/10.jpg)
Planning Your Evaluation Strategy
Define your course goals, learning objectives and outcomes
Determine what kind data and measurement tools will best indicate how well your Learning Objectives & Outcomes are being met?
![Page 11: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/11.jpg)
Planning your Evaluation Strategy Goals, Objectives, Outcomes
Goals - Represent the overall mission
Objectives - Broader perspective on what generally students should know, value or do
Outcomes - What students should demonstrate/represent relative to objective
![Page 12: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/12.jpg)
Why bother with Goals, Objectives, and Outcomes?
They provide the basis for: – Course content– Evaluating students– Communications
• Students• Other faculty and administration• Outside world
– Curriculum planning
![Page 13: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/13.jpg)
ExampleGoal: The course will prepare students to become a
rocket scientist
Objective: Students will be able to understand the working components of a rocket and methods for a successful launch
Outcome: Students will be able to build and launch a rocket.
![Page 14: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/14.jpg)
Learning Objectives and Outcomes Bloom’s Taxonomy of Learning
Knowledge of terminology (define, identify, describe)
Comprehension (discuss, explain, summarize)
Application (apply, utilize, interpret)
Analysis (analyze, compare, contrast)
Synthesis (integrate, organize, construct)
Evaluation (appraise, defend, judge)
![Page 15: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/15.jpg)
Measurement Tool – Surveys, Quizzes & Exams
Types of Items
Quantitative (numbers) often closed ended (limited responses)
• True/False• Multiple choice• Ratings• Single answer
Qualitative (narrative & content) often open ended items
• Short answer
• Essay
![Page 16: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/16.jpg)
Measurement Tool - Quizzes & Exams
Examples of Closed Ended Items
What is your gender?• Male – Female
What is your age ? (10-25) (26-40) (41-55) (55+)
Which college do you attend? EVC Butte Foothill De Anza
Who is your favorite artist of the 20th Century?• Picasso• Joan Miro• Marc Chagall• Salvador Dali
![Page 17: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/17.jpg)
Measurement Tool - Quizzes & Exams
Examples of Open Ended Items
Describe the stylistic elements of Art Deco.
Why did Van Gogh cut off his ear?
What is art?
What is your life story until this point?
![Page 18: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/18.jpg)
Measurement Tool - Quizzes & Exams
Watch for Bias in Constructing Items
How long ago did you stop using street drugs?• Just this week
• A month ago
• A year ago
• Over a year ago
![Page 19: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/19.jpg)
Measurement Tool Performance Assessments
Portfolios
Capstone assignments
Performances/projects
Internships/field activities
Students evaluating students
![Page 20: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/20.jpg)
Measurement Tool – Performance Assessments Potential Strengths
Direct evidence of student learning
Students motivated
Realistic context
Can be integrated into the grading process
Data collection can be relatively unobtrusive to students
![Page 21: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/21.jpg)
Measurement Tool – Performance Assessments Potential Weaknesses
Additional development/coordination time
Faculty trust/buy in critical
Faculty time for scoring
Reliability/validity uncertain
Lack of norms/comparative data
![Page 22: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/22.jpg)
Measurement Tool – Performance Assessments Portfolios
What is it? A collection of multiple targeted pieces of student work that provide evidence of a learner’s achievements along a continuum of learning.
Can be formative or summative (i.e., developmental vs. showcase)
Can be “online” or “old school”
![Page 23: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/23.jpg)
Measurement Tool – Performance Assessments Portfolio Pitfalls
“Kitchen sink” approach
Lack of pre-planning
Retrospective re-creation
Request vs. requirement (i.e., built into program)
Under-estimating faculty time involved
![Page 24: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/24.jpg)
Measurement Tool – Performance Assessments Capstone Assignment/Project
What is it? A culminating project or assignment that provides evidence of achievement of learning goals; generally integrative in nature.
Does not only have to be last year; can be spaced in terms of developmental milestones in the program providing longitudinal/comparative data
![Page 25: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/25.jpg)
Measurement Tool – Performance Assessments Capstone Considerations
Does assignment capture the breadth of program learning expectations?
Does assignment link back to what is actually being taught in the courses?
![Page 26: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/26.jpg)
Measurement Tool – Performance Assessments Data Collection Process
How will assignments be collected/saved for analysis?
Integrate into regular classroom process, otherwise compliance difficult
Will collection be online or hard copies?
Where will data be stored?
How will anonymity be maintained?
![Page 27: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/27.jpg)
Measurement Tool – Performance Assessments Scoring Criteria
Rubrics: A set of criteria that identify expected dimensions of achievement regarding specific assignments, as well as levels of achievement along those dimensions
Checklists
Other approaches
![Page 28: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/28.jpg)
Measurement Tool – Performance Assessments Strategies for Developing Rubrics
Professional org/ discipline standards
Review representative student work (failing though exemplary) and analyze what criteria led to those grades
Student feedback
Analysis of grading criteria already used by faculty to grade similar assignments
Ask faculty to grade sample work and provide criteria they used, looking for commonalities
Pilot/practice use
![Page 29: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/29.jpg)
Measurement Tool – Performance Assessments Students
Students evaluating themselves
Students evaluating other students
– Need to train students in critiquing guidelines http://nlight.com/ONE/critiquingGuidelines.doc
– Use of rating scales useful
Evaluating Team Projects
![Page 30: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/30.jpg)
References“Assessing for Learning” (2004). Peggy L. Maki, Stylus
Press
“Assessment Clear & Simple: A Practical Guide for Institutions, Departments, and General Education” (2004). Barbara E. Walvoord, Jossey-Bass.
“A Practical Guide to Alternative Assessment” (1992). Joan L. Herman, Pamela Aschbacher & Lynn Winters, ASCD.
“Scoring Rubrics in the Classroom” (2001). Judith Arter & Jay McTighe, Corwin Press.
![Page 31: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/31.jpg)
Q & A
Ask now, or via email if you prefer:
Marti Atkinson, [email protected]
Bill Doherty, [email protected]
![Page 33: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/33.jpg)
Upcoming/Archived Seminars
For upcoming desktop seminars and links to recently archived seminars, check the @ONE Web site at:
http://www.cccone.org/seminars/index.php
![Page 34: Sizing up Your Students, Part 3: Designing Effective Evaluations Marti Atkinson and Bill Doherty April 10, 2008 For audio call Toll Free 1 - 888-886-3951](https://reader033.vdocument.in/reader033/viewer/2022051515/5514e73955034693478b5a72/html5/thumbnails/34.jpg)
Thanks for Attending
For information on @ONE’s trainings go to:
http://www.cccone.org/index.php