Download - Developing and Applying Tools
-
8/13/2019 Developing and Applying Tools
1/43
Developing and Applying Tools:
Rubrics and Exam Questions
n a c ens, .
Dawn Zimmaro, Ph.D.
John Kucsera, Ph.D.
The University of Texas at Austin
-
8/13/2019 Developing and Applying Tools
2/43
Agenda
1. Overview of SACS
2. Mapping Assessments to Program Outcomes
3. Writing Critical Thinking Exam Questions4. Creating Rubrics
-
8/13/2019 Developing and Applying Tools
3/43
Academic units
Administrative units
-
8/13/2019 Developing and Applying Tools
4/43
Agenda
1. Overview of SACS
2. Mapping Assessments to Program Outcomes
3. Writing Critical Thinking Exam Questions4. Creating Rubrics
-
8/13/2019 Developing and Applying Tools
5/43
Mapping Assessments to Program Outcomes
At what level should you collect data to measure your
outcomes?
Program
-
8/13/2019 Developing and Applying Tools
6/43
Issues to consider
What existing course assignments, exams, projects, etc.
can you use to measure your learning outcomes?
What program level assessments can you identify or
What is unique about your program that is measurable?
-
8/13/2019 Developing and Applying Tools
7/43
Classroom level
Exam
Assignment
Presentation Paper
Project
-
8/13/2019 Developing and Applying Tools
8/43
Program level
Exit exam
Thesis/dissertation
Capstone course paper or project Portfolio
Licensure/professional exam
Exit survey
-
8/13/2019 Developing and Applying Tools
9/43
English Department B.A.
Example learning outcome:
Students will be able to construct arguments and
communicate them effectively in writing.
-
8/13/2019 Developing and Applying Tools
10/43
Classroom level assessment
Read description of E 370W Major Authors (Emerson)
-
8/13/2019 Developing and Applying Tools
11/43
Assignment
Writing assignment that asks students to develop an
argument about an issue
Emersons efforts to examine the authority and effects ofreli ious and educational institutions hel frame
discussions about literature and education for subsequent
generations.
Given the above statement develop a central thesis with
supporting arguments about how Emersons works havehelped to frame discussions about literature and
education.
-
8/13/2019 Developing and Applying Tools
12/43
Program level assessment
A collection of individual students writing assignments
on constructing arguments taken from several English
courses
Over the course of the whole degree program
-
8/13/2019 Developing and Applying Tools
13/43
Second learning outcome
Example learning outcome:
Students will be able to demonstrate knowledge of the
history and the structure of the English language.
-
8/13/2019 Developing and Applying Tools
14/43
Classroom level assessment
Classroom exam
Topics:
History of English: Old to Early Modern English
Topics should all relate to the learning objective
-
8/13/2019 Developing and Applying Tools
15/43
Program level assessment
Exit exam
Questions taken from all the courses that relate to the
learning outcome
-
8/13/2019 Developing and Applying Tools
16/43
Sample assessment map
See handout
-
8/13/2019 Developing and Applying Tools
17/43
Discussion
Have you tried this approach? What worked? What
didnt?
If you havent tried this, why not?
-
8/13/2019 Developing and Applying Tools
18/43
Agenda
1. Overview of SACS
2. Mapping Assessments to Program Outcomes
3. Writing Critical Thinking Exam Questions4. Creating Rubrics
-
8/13/2019 Developing and Applying Tools
19/43
Writing Crit ical Thinking Exam Questions
Purpose of session
Learning Outcomes
Critical thinking exam questions Objective and Subjective exam questions
Application activity
Final thoughts
-
8/13/2019 Developing and Applying Tools
20/43
Purpose
Objective:
To compose critical thinking questions for program or
classroom level exams
Aligning outcomes with methods
-
8/13/2019 Developing and Applying Tools
21/43
Learning Outcomes
Program to classroom level (alignment)
General statements to specific outcomes
Person centered ear, spec c, an measura e
Classroom learning outcomes should contain
Procedure/method (assignments that will be provided)
Behavior (what the students will do; usually the action verb)
Criteria (how well a learner performs to be judged adequate and thequalitative description of the criteria)
-
8/13/2019 Developing and Applying Tools
22/43
PhD Program Example
Program Educational Objective: Sixty percent of graduates from the College of Education will attain positions as
college faculty.
(Program) Learning Outcome 1:
Will prepare students as future researchers Graduatin students will demonstrate fundamental knowled e of uantitative,
qualitative, and mixed research designs.
Procedure 1.1 (Classroom learning outcome): Course EDP 382 will focus on the fundamental knowledge necessary to
conduct quantitative research.
Given an exam administered in EDP 382, at least 80% of students willdemonstrate a fundamental knowledge of quantitative research designs by
achieving a score at or above an 80 out of 100. The exam will test the extent towhich students can compute common statistics, identify assumptions ofresearch designs, evaluate.
-
8/13/2019 Developing and Applying Tools
23/43
Writing Crit ical Thinking
Exam Questions
For critical thinking:
Focus on higher levels of Blooms taxonomy
-
8/13/2019 Developing and Applying Tools
24/43
Blooms taxonomy for the cognitive domain
Remember
recall or memorize information
Understand
translate from one form to another (e.g., restate in own words)
Apply
Analyze
examine a concept and break it down into parts
Evaluate
make judgments using standards of appraisal Create
put information together in an unique or novel way
-
8/13/2019 Developing and Applying Tools
25/43
Writing Crit ical Thinking
Exam Questions
For critical thinking:
Focus on higher levels of Blooms taxonomy
Behavior component of learning outcome ,
of students will demonstrate a fundamental knowledge
Action verbs (handout)
-
8/13/2019 Developing and Applying Tools
26/43
Writing Crit ical Thinking
Exam Questions
For critical thinking:
Focus on higher levels of Blooms taxonomy
Behavior component of learning outcome __ ,
students will demonstrate a fundamental knowledge
Action verbs (handout)
Outcome to exam questions (and instruction)
Can use the action verb
Ex: Compute the standard deviation of.
Test blueprint (handout)
-
8/13/2019 Developing and Applying Tools
27/43
Objective and Subjective
Exam Questions
Objective questions
Select a correct response from several alternatives
Subjective questions
Present an original answer reng s wea nesses o eac
Assumptions (True/ False/ or Undetermined)
1. Essay exams require more thorough student preparation andstudy time than objective exams.
2. Essay and objective exams can be used to measure the samecontent or ability.
Examples of objective, critical thinking exam questions (packet)
-
8/13/2019 Developing and Applying Tools
28/43
Application Activity
In groups or pairs and with using the Emerson content
from Dawns previous presentation,
1. Construct one objective and one subjective examuestion.
2. Explain why you believe your questions are assessing
critical thinking.
-
8/13/2019 Developing and Applying Tools
29/43
Application Activity
Objective examples:
(Analysis) Emerson might be characterized as any of thefollowing EXCEPT:
(A) a transcendentalist
(B) an American Renaissance writer
(C) a public speaker
(D) a political prophet
(E) a literary critic
(Evaluate) Determine whether the following statement is true orfalse:
Melville and Hawthorne were both disillusionedtranscendentalists.
-
8/13/2019 Developing and Applying Tools
30/43
-
8/13/2019 Developing and Applying Tools
31/43
Agenda
1. Overview of SACS
2. Mapping Assessments to Program Outcomes
3. Writing Critical Thinking Exam Questions4. Creating Rubrics
-
8/13/2019 Developing and Applying Tools
32/43
1.What is a rubric?
2.When should you use a rubric?
3.What are the elements of a rubric?
Creating Rubrics
4.How do you create a rubric?
-
8/13/2019 Developing and Applying Tools
33/43
What is a rubric?
-A scoring guideline that measures student
achievement systematically:
specifies instructor expectations
lists performance criteria
describes levels of quality
-
8/13/2019 Developing and Applying Tools
34/43
When should you use a rubric?
On assignments that require subjective grading
Places structure on a subjective process
Essa s
Presentations
Multi-media
-
8/13/2019 Developing and Applying Tools
35/43
What are the elements of a rubric?
What the
student needs
to do
Descriptors and criteriathat identify performance
expectation for each
point on the scale
I.
Example:
Include a compelling, descriptive, and persuasive
conclusion that summarizes arguments set forth inpaper
-
8/13/2019 Developing and Applying Tools
36/43
Standards that identify
the range of quality orperformance levels
How well the
student does itII.
Example:--no or poor conclusion or summary of argument
--some summary of points made, but nothing beyond
summary; no broad conclusions/lessons
--
a conclusion going beyond summary of what was written inthe body of the essay
-
8/13/2019 Developing and Applying Tools
37/43
A scale of points
awarded to specificelements of student
performance
III.
What score the
student willreceive
5 = outstanding
3 = acceptable
1 = poor
-
8/13/2019 Developing and Applying Tools
38/43
Critical Thinking Rubric
A. Clearly and precisely
analyzes key information,
questions, and problems
B. Uses inference to reason
carefully from clearly stated
premises to important
A. Is unable to analyze
information, questions, and
problems or does so
superficially
B. Is unable to or infrequently
uses inference to reason
1 [2 3 4] 5
implications and
consequences
C. Uses deductive and
inductive reasoning and
problem-solving skills
consistently and with ease
from clearly stated premises
or recognize implications and
consequences
C. Is unable to or infrequently
uses deductive and
inductive reasoning andproblem-solving skills
-
8/13/2019 Developing and Applying Tools
39/43
Analytic Rubrics
Holistic Rubrics
-
8/13/2019 Developing and Applying Tools
40/43
How do you create a rubric?
Develop assignment Identify learning objectives
Decide which criteria to evaluate
Decide what kind of rubric to use Define performance indicators
Identify standards of performance
Decide on grading or scoring scale Review rubric to ensure it measures what you intend
-
8/13/2019 Developing and Applying Tools
41/43
Determine key criteria
If the assignment
objectives are
then consider these criteria
Creative products Wow factor, novelty
Technical quality
Adherence to conventions of the field
Development of ideas
Clarity
Oral presentation Voice projection
Body language
Organization
-
8/13/2019 Developing and Applying Tools
42/43
Identify Indicators
Concrete sign or evidence of a criterion beingmetnot meant to be exhaustive
Example:
Learning objective: Students will be able to write in
Criterion: Student clearly and precisely analyzes
key information, questions, and problems
Indicator:
Identifies main tension in story and relates itto sub-plots
-
8/13/2019 Developing and Applying Tools
43/43