The Test Assessment Questionnaire

Katherine M. Sauer William MertensMetropolitan State College of Denver University of Colorado at Boulder ksauer5@mscd.edu mertens@colorado.edu

March 29, 2012 MBAA – ABE Annual Conference Chicago, IL

Students are Overconfident

Walstad (2001) - suggests concepts from behavioral economics (e.g. overconfidence) could be used to explain student behavior

Falchikov and Boud (1989) - students have grade expectations that are higher than the typical distribution for the course

Grimes (2002) - has principles students predict exam scores and finds overconfidence in their understanding of material

Nowell and Alston (2007) - instructor grading practices can influence the degree of overconfidence

What Happens When Students are Overconfident?

Students may start to blame the instructor for the disconnect between (perceived) effort and desired grade.

Students may label the instructor as “unfair” or the course simply “too hard”.

Grimes, Millea, and Woodruff (2004) - degree to which students accept personal responsibility for performance affects their evaluation of teaching effectiveness and course satisfaction

Carrell and West (2010) - in course evaluations, students reward professors who increase achievement in the contemporaneous course, not those who facilitated deep learning for subsequent courses

Millea and Grimes (2002) - instructors need not “water-down” courses in order to receive favorable course evaluations… they can positively influence evaluations by addressing negative student attitudes about forthcoming coursework

How can I help my students become more self-aware in their learning?

Get them to focus on the type of mistake they are making instead of the number of mistakes they are making.

Get them thinking about the link between their study activities and their exam performance.

The Test Assessment Questionnaire

This instrument guides students through an analysis of their midterm exam mistakes.

• questions about exam preparation activities

• studying activities for the course in general

• formulate a study strategy for the rest of the course

The Version I’m Currently Using

Instructors can tailor the first set of questions to their course.

Logistics

- TAQ worksheet is given to students when the exam is handed back

- due one week later and counts as a homework grade

- in the class period when it is due, debrief with study strategies for each type of common mistake (10 minutes)

- students can come to office hours for additional guidance

Three Benefits

1) it guides students toward more self-awareness in their studies

2) it can be used as a part of assessment and assurance of learning efforts

3) may improve students’ final exam scores over the midterm

1) self-awareness in studies

On the course evaluations: Please comment on the experience of completing the Test Assessment and/or meeting with me to discuss it.

- It made me improve my studying skills.

-Good, it gave me one-on-one time with you to figure out how I could improve.

-Helpful in learning how I make mistakes and how to fix them better.

- It helped me to personally analyze my test taking and study habits. Very helpful! I have never had a teacher who cared enough to take the time to help me improve in this aspect.

-Your tips helped me in other classes and got me running again and I don’t feel as stressed. My mom thanks you.

There was no negative feedback on this question.

2) Assessment & Assurance of Learning

More and more time is devoted to these activities.

We assume that students are doing their part in the learning process.

To what extent are they?

Routinely, I find students who earn a D or F on an exam reveal that they simply didn’t study.

- provides context to “does not meet expectations”

3) improve students’ final exam scores

Experiment:

Fall 2011 – University of Colorado

All Students in Principles of Microeconomics 10 course sections, 1713 students

62 recitation sections

Design:

Course instructors do not modify their class time or teaching activities in any way.  Each TA that teaches two or more recitation sections has a control and experiment section assigned to them.

The experiment will take place during the recitation class in which the midterm is handed back.

Control Sections: TAs spend the class time going over the exam as they normally would and will offer some general study strategies.

Experiment Sections: Students will be given the correct answers and will spend the class time going over the exam by themselves while completing the TAQ.

After the TAQs are completed, the TA discusses study strategies that complement each of the common mistake categories.

All students complete Duttweiler’s (1984) Internal Control Index.

- many such instruments

- argue this one is most reliable and valid one for adults

- normed on college students

- 28 questions, Likert-type scale

Half of the items are worded so that high internally oriented respondents are expected to answer at the "usually" end of the scale and the other half at the "rarely" end. The "rarely" response is scored as 5 points on items 1,2,8,11,14,17,19,22,23,24,26,27.  This produces a possible range of scores from 28 to 140 with higher scores reflecting higher internal locus of control.

For college students: MEAN: 99.3- 120.8

Data:

Two course instructors opted out of the experiment.n = 1337

27 treatment recitations, n = 66727 control recitations, n = 670

But not all sections were full.

We received 522 first-round ICI surveys from 28 different recitations.

We received 488 second-round ICI surveys also from 28 different recitations (but not exactly the same recitations as those for the first-round surveys).

We received 234 TAQs from 13 different recitations.

Treatment groups We received data that included both surveys and the TAQ for six recitations involving a maximum of 88 students having all 3 sheets (if they match).

We received data that included one survey and a TAQ from seven recitations involving a maximum of 104 students having both sheets (if they match). Control groupsWe received data that included both surveys for ten recitations involving a maximum of 175 students having both sheets (if they match).

The student’s TAQs and ICIs are being scanned and the data entered into a spreadsheet.

Control data is being obtained from Institutional Research.

Comments? Questions?

Katherine M. SauerMetropolitan State College of Denver

ksauer5@mscd.edu

March 29, 2012 MBAA – ABE Annual Conference Chicago, IL