the test assessment questionnaire: a tool for student self-assessment after the midterm exam...
TRANSCRIPT
The Test Assessment Questionnaire:A Tool for Student Self-Assessment
After the Midterm Exam
Katherine M. SauerMetropolitan State University of Denver
William MertensUniversity of Colorado at Boulder
Economics Teaching Conference November 8th – 9th 2012 Orlando, FL
Students are Overconfident
Walstad (2001) - suggests concepts from behavioral economics (e.g. overconfidence) could be used to explain student behavior
Falchikov and Boud (1989) - students have grade expectations that are higher than the typical distribution for the course
Grimes (2002) - has principles students predict exam scores and finds overconfidence in their understanding of material
Nowell and Alston (2007) - instructor grading practices can influence the degree of overconfidence
What happens when students are
overconfident?
Students may start to blame the instructor for the disconnect between (perceived) effort and desired grade.
Students may label the instructor as “unfair” or the course simply “too hard”.
Grimes, Millea, and Woodruff (2004) - degree to which students accept personal responsibility for performance affects their evaluation of teaching effectiveness and course satisfaction
Carrell and West (2010) - in course evaluations, students reward professors who increase achievement in the contemporaneous course, not those who facilitated deep learning for subsequent courses
Millea and Grimes (2002) - instructors need not “water-down” courses in order to receive favorable course evaluations… they can positively influence evaluations by addressing negative student attitudes about forthcoming coursework
How can I help my students
become more self-aware in their learning?
Get them to focus on the type of mistake they are making instead of the number of mistakes they are making.
Get them thinking about the link between their study activities and their exam performance.
The Test Assessment Questionnaire
This instrument guides students through an analysis of their midterm exam mistakes.
• questions about exam preparation activities
• studying activities for the course in general
• formulate a study strategy for the rest of the course
The TAQ starts with a series of multiple choice questions regarding course preparation activities.
How often did you miss our class?a. neverb. 1 or 2 timesc. more than 2 times
For how many chapters did you complete the Reading Outline?
a. all b. most of themc. a few of themd. none
When did you complete the Reading Outlines?a. before we covered the topic in classb. after we covered the topic in class
For how many chapters did you complete the online Homework?
a. allb. I missed 1 or 2c. I missed 3 or more
How often did you come to office hours or email the professor for clarification on the material?
a. neverb. 1 or 2 timesc. 3 or more times
When did you start working through the exam information / review sheet?
a. as soon as it was availableb. 2 weeks before the exam c. 1 week before the examd. the night before the exame. I didn’t work through it.
For how many chapters did you work some of the Optional Problem Set questions?
a. all chapters and all problemsb. most chapters and problemsc. a few chapters and problemsd. none
How did you prepare for this exam? How many hours did you spend preparing for the exam?
How does your exam score compare with your homework average? Why do you think that is the case?
Go through the questions you answered incorrectly on your exam.
For each question, choose the reason you feel you got the answer wrong.
Write the number of each question in the table next to the reason.
Do you notice any patterns with the type of question you missed? Explain.
What letter grade do you hope to earn in this course? To achieve your goal, what is your strategy for studying
between now and the next exam?
Logistics
TAQ worksheet is given to students when the exam is handed back
due one week later and counts as a homework grade
in the class period when it is due, debrief with
study strategies for each type of common mistake (10 minutes)
- students can come to office hours for additional guidance
Study Strategies for Common Mistakes
Didn’t know a definition- make flashcards (as we go along, not at the end!)
Couldn’t apply a definition I knew- think of examples /illustrations for each conceptand add to flashcards
Didn’t read the question/answers carefully- “mark up” exam as you go- every 3 to 5 questions, physically set down pencil,close eyes, take deep breath
Study Strategies for Common Mistakes Knew the answer but couldn’t come up with it during the exam
- check out the test-taking strategies and test-anxiety workshops on campus
Didn’t know how to set the problem up- for each type of problem we work, go back and write out in words the general steps we took to
solve the problem
Used the wrong formula- copy formulas onto one piece of paper- next to each, describe in words when you woulduse the formula- copy an example problem next to each formula
Study Strategies for Common Mistakes Debated between two answers and choose the wrong one
- good news! able to eliminate 2 answers- probably know material at 80-85% level- missing some detail in that last 15-20% - spend a little more time and effort on the details
Just didn’t know the material- students usually know why
Other- come to office hours and we’ll come up with somespecific strategies for you
Study Strategies for Common Mistakes
Benefits of Using the
TAQ
1) it guides students toward more self-awareness in their studies
2) it can be used as a part of assessment and assurance of learning efforts
3) may improve students’ final exam scores over the midterm
Student Self-Awareness
On the course evaluations: Please comment on the experience of completing the Test Assessment and/or meeting with me to discuss it.
- It made me improve my studying skills.
-Good, it gave me one-on-one time with you to figure out how I could improve.
-Helpful in learning how I make mistakes and how to fix them better.
- It helped me to personally analyze my test taking and study habits. Very helpful! I have never had a teacher who cared enough to take the time to help me improve in this aspect.
-Your tips helped me in other classes and got me running again and I don’t feel as stressed. My mom thanks you.
There was no negative feedback on this question.
Assessment & Assurance of Learning
More and more time is devoted to these activities.
We assume that students are doing their part in the learning process.
To what extent are they?
Routinely, I find students who earn a D or F on an exam reveal that they simply didn’t study.
- provides context to “does not meet expectations”
Improve Students’ Final Exam Scores
Experiment:
Fall 2011 pilot – University of Colorado
All Students in Principles of Microeconomics 10 course sections, 1713 students
62 recitation sections
Design:
Course instructors do not modify their class time or teaching activities in any way. Each TA that teaches two or more recitation sections has a control and experiment section assigned to them.
The experiment will take place during the recitation class in which the midterm is handed back.
Control Sections: TAs spend the class time going over the exam as they normally would and will offer some general study strategies.
Experiment Sections: Students will be given the correct answers and will spend the class time going over the exam by themselves while completing the TAQ.
After the TAQs are completed, the TA discusses study strategies that complement each of the common mistake categories.
All students complete Duttweiler’s (1984) Internal Control Index.
- many such instruments
- argue this one is most reliable and valid one for adults
- normed on college students
- 28 questions, Likert-type scale
Half of the items are worded so that high internally oriented respondents are expected to answer at the "usually" end of the scale and the other half at the "rarely" end. The "rarely" response is scored as 5 points on items 1,2,8,11,14,17,19,22,23,24,26,27. This produces a possible range of scores from 28 to 140 with higher scores reflecting higher internal locus of control.
For college students: MEAN 99.3- 120.8
Data:- ICI scores- scores for homework, recitation, exams- gender, race, GPA, etc from Office of Institutional Research
Two course instructors opted out of the experiment.n = 1337
27 treatment recitations, n = 66727 control recitations, n = 670
But not all sections were full.
We received 522 first-round ICI surveys from 28 different recitations.
We received 488 second-round ICI surveys also from 28 different recitations (but not exactly the same recitations as those for the first-round surveys).
We received 234 TAQs from 13 different recitations.
Data from One Course Section (n = 188)
Treatment (n = 94) 27 students completed the ICI twice and the TAQ.
12 students completed one ICI and the TAQ. Control (n=94)35 students completed the ICI twice and the TAQ.
30 students completed one ICI and the TAQ.
Fall 2012 – Rerun the Experiment
2 sections of Principles of Microeconomics
Course instructors are actively supportive
TAQ and ICI will have weight in course grade
Questions? Comments?
Keep in touch:
@yogiconomist
Katie Sauer Hart
303-556-3037
Katherine Sauer, Ph.D.