This article was downloaded by: [University of Cambridge]On: 14 October 2014, At: 03:23Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Assessment & Evaluation in HigherEducationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/caeh20

Conditions affecting the usefulnessof pre- and post-tests for assessmentpurposesElise Boyas a , Lois D. Bryan b & Tanya Lee ba Katz Graduate School of Business/College of BusinessAdministration , University of Pittsburgh , Pittsburgh , USAb Department of Accounting and Taxation , Robert MorrisUniversity , Pittsburgh , USAPublished online: 18 Mar 2011.

To cite this article: Elise Boyas , Lois D. Bryan & Tanya Lee (2012) Conditions affecting theusefulness of pre- and post-tests for assessment purposes, Assessment & Evaluation in HigherEducation, 37:4, 427-437, DOI: 10.1080/02602938.2010.538665

To link to this article: http://dx.doi.org/10.1080/02602938.2010.538665

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (theContent) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

Assessment & Evaluation in Higher Education2011, 111, iFirst Article

ISSN 0260-2938 print/ISSN 1469-297X online 2012 Taylor & Francis

http://www.tandfonline.com

Conditions affecting the usefulness of pre- and post-tests for assessment purposes

Elise Boyasa, Lois D. Bryanb and Tanya Leeb*

aKatz Graduate School of Business/College of Business Administration, University of Pittsburgh, Pittsburgh, USA; bDepartment of Accounting and Taxation, Robert Morris University, Pittsburgh, USATaylor and FrancisCAEH_A_538665.sgm10.1080/02602938.2010.538665Assessment & Evaluation in Higher Education0260-2938 (print)/1469-297X (online)Original Article2010Taylor & Francis0000000002010TanyaLeeleeta@rmu.edu

Interest in measuring and evaluating student learning in higher education isgrowing. There are many tools available to assess student learning. However, theuse of such tools may be more or less appropriate under various conditions. Thisstudy provides some evidence related to the appropriate use of pre/post-tests. Thequestion of whether graded tests elicit a higher level of performance (betterrepresentation of actual learning gains) than ungraded post-tests is examined. Weexamine whether the difficulty level of the questions asked (knowledge/comprehension vs. analysis/application) affects this difference. We test whetherthe students level in the degree programme affects this difference. Resultsindicate that post-tests may not demonstrate the full level of student mastery oflearning objectives and that both the difficulty level of the questions asked and thelevel of students in their degree programme affect the difference between gradedand ungraded assessments. Some of these differences may be due to causes otherthan grades on the assessments. Students may have benefited from the post-test, asa review of the material, or from additional studying between the post-test and thefinal examination. Results also indicate that pre-tests can be useful in identifyingappropriate changes in course materials over time.

Keywords: assessment; pre-test; post-test; AACSB

Introduction

Colleges and universities are being called upon to be accountable for student learning.The interest in measuring student learning outcomes began in the USA in public K-12education with a call for more rigorous standards (Department of Education 1983) andwas operationalised in The No Child Left Behind Act of 2001. This interest has nowmoved to higher education. Stakeholders want institutions of higher education todemonstrate that students have achieved competence in general intellectual skills anddiscipline-specific skills.

Stakeholders in this process include students and their parents, employers,institutions of higher learning, and accrediting bodies. For a school of business, boththe Association of Collegiate Business Schools and Programs (ACBSP) and theAssociation to Advance Collegiate Schools of Business (AACSB) believe that learn-ing objectives for students should be linked to institutional learning objectives. Theseaccrediting bodies want student mastery of these objectives to be measured andreported and this information to be used for improvement of curriculum and course

*Corresponding author. Email: leeta@rmu.edu

Vol. 37, No. 4, June 2012, 427437

http://dx.doi.org/10.1080/02602938.2010.538665

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

2 E. Boyas et al.

content (ACBSP 2006; AACSB 2008). Neither accrediting body provides specificrequirements for how to accomplish this. The choice of learning objectives andassessment instruments is left to the discretion of each institutions faculty andadministrators.

One assessment tool often used to measure student learning is a pre/post-test.Students are given a test at the beginning of a course to determine their level ofmastery of course learning objectives. At the end of the course, the same assessmentis administered. The expectation is that student performance on the assessment willimprove, presumably as a result of completing course-related requirements.

Despite the popularity of this assessment tool, criticisms of the use of pre/post-tests exist (Suskie 2004, 110). Therefore, it is useful to consider when a pre/post-testmay be an appropriate choice. Identification of factors that should be considered whenan instructor chooses an assessment instrument could be helpful. It may be thatstudents in core courses (required but not in their major subject area) will be less moti-vated to perform well on an ungraded pre/post-test than will students in courses intheir major. Similarly, some course learning objectives may be more or less suited topre/post-testing. It may be difficult to measure learning for some of the higher orderskills using this mechanism since answering questions at this level requires somethingbeyond the recall needed to answer more basic knowledge and comprehensionquestions. These are empirical questions addressed in this paper.

Literature review

Student test performance on entry to and exit from university courses has been usedto measure student learning. Measuring the learning of students in this manner, avalue-added approach, has intuitive appeal but has also been criticised. One criticismis that there is the possibility that students actually learned the skills they needed forsuccess on the post-test outside the classroom (Suskie 2004, 110).

Another criticism of the use of the pre/post-test to measure student learning is thatstudents will necessarily perform poorly on the pre-test since they know nothing aboutthe material at the beginning of the course; the lower the pre-test score the higher thepotential learning gains (Warren 1984; Pascarella and Wolniak 2004). Post-tests undersuch conditions will always show improvement over the pre-test. Additionally,instructors may teach to the post-test in order to improve results, potentially neglect-ing desirable skills not tested (Ewell 2002).

A final criticism of the use of pre/post-tests to measure learning is the presence ofresponse-shift bias, which occurs because students frames of reference during thepre-test are not the same as during the post-test (Mann 1997; Drennan and Hyde2008). A response-shift bias can occur when students are self-evaluating their skillsprospectively without a real understanding of the skills they are evaluating. Thus, thepre-test results may reflect an overestimation of their skills while the post-test resultsreflect a more informed judgement of these skills, resulting in an understatement ofthe value added by the course (Drennan and Hyde 2008). Statistical analysis of pre/post-test results presumes that the frame of reference of the student has not beenaltered by the course content, hardly a realistic assumption (Mann 1997).

Despite these criticisms, attention has refocused on the use of value-added measuresof student learning to evaluate the effectiveness of course content (Kimmell, Marquette,and Olsen 1998; Banta and Pike 2007). The American Association of State Colleges andUniversities states (AASCU), It is time for states and their colleges and universities, in

428

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

Assessment & Evaluation in Higher Education 3

conjunction with the regional accrediting agencies, to lead the development of a consen-sus model for assessing the value added from undergraduate student learning (2006, 1).The AASCU points out that general intellectual skills required for students to be success-ful across career disciplines should be assessed (2006).

Recently, educators have concluded that the use of a pre/post-test can yieldexceptionally compelling information on what students have learned during a courseor program (Suskie 2004, 110). Suskie goes on to point out that even with its short-falls, value-added assessment may still be a good choice when it is important to docu-ment significant gains in student learning at the course or discipline level. Thisopinion is shared by the AACSB which clearly identifies the use of pre/post-tests as aviable measure of student learning:

Evaluation of the effectiveness of instruction begins with an examination of learninggoals. It goes on to include such things as student reactions, peer observation, expertobservation, and periodic assessment of the impact of instruction on later performance.To ensure quality, the schools faculty members measure overall student achievement byuse of such techniques as pre-testing and post-testing, assessment in subsequent course-work, surveys of employers, etc. (AACSB 2008, 54)

The AACSB expects business schools to use direct methods to assess student learning(Calderon 2005) and has documented best practices in assessment which include theuse of pre/post-tests (AACSB 2008). Meyers and Nulty (2009) emphasise the impor-tant role that assessment has in achieving desired student learning outcomes andsuggest that assessment should have a central role in the design of curriculum.

In accounting, the Accounting Education Change Commission (1990a, 1990b),Albrecht and Sack (2000) and the Core Competency Framework developed by theAmerican Institute of Certified Public Accountants (1999) call for accountinggraduates to be active lifelong learners with skill sets that include both competency indiscipline-specific accounting knowledge and inter-disciplinary skills such as criticalreasoning and problem solving. This combination of the desired characteristics foraccounting graduates and the need to actively assess student learning gains has moti-vated many accounting programmes to define specific student learning objectives forcourses and programmes and to develop assessment tools for these objectives.Calderon, Green, and Harkness (2004) reported that over half of the accountingdepartment chairs responding to a survey believed that formalised assessment ofstudent learning has improved student success in their programmes.

Clearly, assessing student learning has become an important focus in highereducation, and the use of pre/post-tests has increased. This study attempts to identifyconditions that could affect the usefulness of pre/post-tests to assess changes in thestudent mastery of particular learning objectives over the course of a semester.

Hypotheses

Pre/post assessments involve the administration of tests which are of importance tothe institution but of limited personal importance to the individual student. A signifi-cant body of research exists addressing the question of whether students are equallymotivated to succeed on low-stakes testing and high-stakes testing (Harlen and Crick2003). A useful way of representing student test-taking behaviour has been defined byexpectancy-value models of achievement motivation. Expectancy is the studentsbelief that he or she can successfully complete the assessment, and value embodies the

429

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

4 E. Boyas et al.

beliefs a student holds as to exactly why he or she should try to complete the tasksuccessfully (Wise and DeMars 2005). When the outcome of the test has great valueto the student, and the assessment tool is well-designed, demonstrated levels of profi-ciency are a good proxy for actual levels of proficiency (Wise and DeMars 2005).However, some research has shown that students do not perform to their actual levelsof proficiency on low-stakes tests (Wise and DeMars 2005; Cole, Bergin, andWhittaker 2008). We expect the outcome of ungraded pre/post-tests to have limitedvalue to students, leading to the following hypothesis:

H1: The use of a graded assessment instrument (questions embedded in a course finalexam) will elicit a higher level of performance than will the use of an ungraded post-test.

One factor that could affect the degree of differences in performance between agraded assessment and an ungraded post-test is the difficulty level of the learningobjectives being assessed. Blooms Taxonomy (Bloom 1956) is widely used to cate-gorise the cognitive difficulty level of course material. Applying Blooms Taxonomyto accounting, basic knowledge and comprehension learning objectives requiresstudents to be able to define terms or basic relationships. Application and analysislearning objectives require students to be able to combine or manipulate financialinformation during problem solving. Student learning objectives for courses incolleges and universities generally reflect a combination of basic and higher levelskills. Basic knowledge questions can be answered by recalling information. Ques-tions related to higher level skills require students to both recall the related knowledgeand use the knowledge to perform application or analysis tasks, leading to thefollowing hypothesis:

H2: Application and analysis questions will result in larger differences in performancebetween graded (course final exam) and ungraded (post-test) assessment instrumentsthan will knowledge and comprehension questions.

As students progress through a degree programme, more discipline-specificknowledge is acquired, and students become more focused on their discipline andmore motivated to succeed. This may lead to differences in the size of the differencesin performance between ungraded post-tests and graded assessments and the follow-ing hypothesis:

H3: Higher level students (taking a required course in their major) will demonstrate asmaller level of difference between the post-test and the final exam than lower levelstudents (taking a required course, probably not in their major).

Research design

This research used pre/post-tests in several undergraduate classes in accounting inorder to evaluate the usefulness of this technique to assess student learning outcomes.This study used a total of 120 subjects, all of whom were undergraduates attending aschool of business located in the mid-Atlantic region of the USA. Of these subjects,90 were enrolled in three sections of an introductory accounting course required in theundergraduate business programme; 60 were taught by one instructor and 30 byanother (see Table 1). All sections were treated as much alike as possible to minimisedifferences due to students being in different sections. Both instructors were full-timefaculty members experienced in teaching this course.

430

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

Assessment & Evaluation in Higher Education 5

In order to minimise differences in delivery, the instructors of the introductoryaccounting course standardised course content, quizzes, exams, homework, electroniconline support material, and in-class examples. The testing procedures were also thesame across classes and sections, with all students receiving a pre-test and a post-testin addition to the regular course examinations. Thus, except for personal differencesin teaching style and class dynamics, the students in the three sections of the introduc-tory accounting course received the same course content. Most of these students weresophomores, and most (74%) were not accounting majors. The 90 introductorystudents are referred to as lower level students.

The 30 remaining subjects were students enrolled in a required individual taxationcourse, taught by a full-time instructor who had taught this course many times in thepast. All 30 students were accounting majors who had made significant progress in thecurriculum, typically at least of junior standing. In addition, many of these studentshad been successful in obtaining internship opportunities in either taxation or account-ing prior to or simultaneous with their enrolment in this course; approximately onethird were honours students in the cooperative education programme, a programmewhere students work for outside firms as part of their education. In this paper, thesestudents are referred to as higher level students.

All subjects in both groups were asked to try their hardest on the pre/post-testseven though the results would not impact their grades. The pre-test was not reviewedwith the students after it was administered. The post-test was reviewed with thestudents after it was administered.

Two learning objectives in both the introductory accounting course and the taxcourse require students to: (1) analyse and interpret financial reports, and (2) compre-hend accounting terminology, concepts and methods. The pre/post-tests and the ques-tions on the final exams used for this study were designed to assess student masteryof these learning objectives within the context of each course. The analyses wereperformed separately for the students in the introductory course (lower level students)and for those in the discipline-specific course (higher level students).

Lower level students

The pre/post-tests consisted of 15 questions based on the year-end financial statementsof a large publicly traded firm. The pre-test and the post-test were identical. Thequestions were of varying degrees of difficulty, categorised into two levels. Simplequestions which required students to find an amount or an account were considered tobe knowledge/comprehension (K/C) questions in accordance with BloomsTaxonomy (Bloom 1956). An example of a K/C question is What is the firms netincome for this fiscal year? More complex questions were considered to be applica-tion/analysis (A/A) questions using terminology consistent with Blooms Taxonomy.These questions required students to use the information in the financial statements to

Table 1. Sample demographics.

Gender Lower level students Higher level students Number of subjects

Male 60 18 78Female 30 12 42Total 90 30 120

431

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

6 E. Boyas et al.

determine or calculate a response. An example of an A/A question is When the firmissued its common stock, did it sell for more than its par value? This question requiredan understanding of the interrelationships between accounts. Of the 15 questions onthe pre/post-tests, eight were classified as K/C questions and seven as A/A questions.

The pre-test was administered on the first day of class. During the semester, actualcorporate financial statements were only consulted occasionally. However, during thelast week of class, the students were given a set of financial statements for a differentfirm and a list of questions similar to those on the pre/post-tests as an in-class review.In order to give students an incentive to put forth some effort and master the material,they were told that one of the problems on the final exam would require them toanswer similar questions for a different set of financial statements. The questions onthe final exam related to the financial statements of two different publicly tradedcompanies not previously reviewed in class or on the pre/post-tests. The data analysispresented in this paper was limited to 11 questions common to the pre/post-tests andthe final exam.

The post-test was administered on the last day of class. Again, the students wereasked to try their hardest on the post-test and were told that it was in their best interestto view the post-test as a dry run for one of the final exam problems. After the post-test was collected, the answers to the questions were reviewed with the class.

Higher level students

The students enrolled in the taxation course (concentrating on individual income taxconcepts and required for accounting majors) took a pre-test during the first week ofclass. This pre-test asked students to answer 15 questions based on informationprovided in a fictitious completed individual income tax return. Of these questions,seven were classified as K/C questions and eight as A/A questions. An example of aK/C question is What amount did the taxpayer report as AGI (adjusted gross income)for the tax year 2005? An example of an A/A question is Did the taxpayers own theirown home and how do you know this? During the last week of the course, the post-test (which was identical to the pre-test) was administered. The answers to thequestions on the post-test were reviewed with students. The final exam contained aproblem that was identical to the pre/post-tests.

Analyses and results

Analyses were first run separately for the lower level students and for the higher levelstudents. Each analysis was performed using both the appropriate non-parametric testand the related parametric test, since the data were not normally distributed. Paramet-ric results were essentially the same as the non-parametric results. Both sets of resultsare provided in the tables, but the text principally discusses the parametric results.

The first hypothesis is that the use of a graded assessment instrument (course finalexam) will elicit a higher level of performance than will the use of an ungraded post-test. To analyse this, the questions that were common across the post-test and the finalexam were identified, resulting in 11 common questions for the lower level studentsand 15 for the higher level students. Scores on the post-tests were compared to scoresfor the corresponding questions on the final exam. The ungraded post-test and thegraded final exam were administered within one week of each other. Results areshown in Table 2.

432

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

Assessment & Evaluation in Higher Education 7

For the lower level students, the average score on the final exam was significantlyhigher than on the post-test, with an average of 68% correct on the post-test and 80%correct on the final exam (p < 0.0001); for the higher level students, an average of91% correct on the post-test and 95% correct on the final exam (p < 0.03). Thus H1was supported, indicating that an ungraded post-test may not reflect all the actualcourse learning gains for all students.

The second hypothesis states that more difficult questions will result in largerdifferences in performance gains between graded (course final exam) and ungraded(post-test) assessment instruments. More difficult questions are those that test thestudents ability to conduct analysis and apply their knowledge, categorised as A/A(application and analysis) questions in this research. To analyse this hypothesis,performance on the K/C (knowledge and comprehension) questions was analysedseparately from performance on the A/A questions for both subject groups. The resultsare summarised in Table 3.

The hypothesis is supported for the lower level students. This group had a signif-icant increase for the A/A questions but not for the K/C questions from the post-testto the final examination. It was not supported for the higher level students. The higherlevel students achieved very high scores on the post-test (the average is 91%); thus,there is little left to gain between the post-test and the final exam since these studentsare already achieving near the ceiling on the post-test. Support for this hypothesis forthe lower level students provides evidence that post-tests may be a more accurate indi-cator of student learning in an introductory class for simpler K/C questions than formore difficult A/A questions.

The third hypothesis states that students taking a required course in their major(higher level students) are expected to demonstrate a smaller difference between the

Table 2. Test of Hypothesis 1 comparison of student performance on the final exam vs. thepost-test.

Lower or higher level students

Number of subjects

Average % correct post-test

Average % correct final exam

Statistical testa p value

Lower level n = 90 68 80 t-testWSR

8 E. Boyas et al.

post-test and the final exam than students taking a required course probably not intheir major. The expectation is that higher level students enrolled in a course in theirmajor area will be more motivated, generally, throughout the course when comparedwith the lower level students. They will, as a result, experience lower achievementgains between the post-test and the final exam than the lower level students. Theresults of the comparison of the two groups using both parametric and non-parametrictests (one-sided) supported the hypothesis and are presented in Table 4.

Both the KruskalWallis and Wilcoxon two-sample tests are significant. Thus,while pre-test and post-test score differences can be used to indicate student learning,these differences may be less indicative of actual student learning in lower level (non-major) classes with lower level students than in upper level classes with higher levelstudents.

It is also interesting to note (see Table 5) that there are significant differences acrosspre-test and post-test results for both groups despite the limitations of the use of pre/post-tests. Lower level students in the introductory course scored significantly higheron the post-test than on the pre-test (an average of 49% correct on the pre-test and 68%correct on the post-test, p < 0.0001). Higher level students in the required taxationcourse also scored significantly higher on the post-test than on the pre-test (an averageof 83% correct on the pre-test and 91% correct on the post-test, p < 0.006). Given thedesign of this study, we cannot make statistically valid inferences as to the cause of theincrease in scores from pre-test to post-test. While the delivery of curriculum could becausal, the increases may also be at least partly a result of an increase in studentsdesires to do what their instructors asked of them over the course of the semester.

Table 4. Test of Hypothesis 3 comparison of student performance on the final exam vs. thepost-test across student groups.

Lower or higher level students

Number of subjects

Average % correct post-

test

Average % correct final

exam

Difference final versus post-test(%)

Test for differences across lower and

higher level studentsa

Lower level n = 90 68 80 12 0.002 (KW)0.002 (t-test)

Higher level n = 30 91 95 4

Notes: at-test = Wilcoxon and KW = KruskalWallis two-sample tests.

Table 5. Comparison of student performance on the post-test vs. the pre-test across studentgroups.

Lower or higher level students

Number of subjects

Average % correct pre-

test

Average % correct post-

testStatistical

testa p value

p value for test for differences across lower and higher level studentsb

Lower level N=90 49 68 t-testWSR

Assessment & Evaluation in Higher Education 9

Limitations and conclusions

There is currently very little guidance to help instructors design useful and reliablepre/post-tests for a course. The results of this study provide some evidence thatstudents do take ungraded pre/post-tests somewhat seriously but also indicate thatcertain conditions can affect the usefulness of these tests in assessment.

Graded assessments did demonstrate a higher level of learning than did ungradedpost-tests, supporting H1. There are several possible causes. One is that the post-testsserved as a form of review and increased student learning just prior to the final exam-ination. Another is that students rigorously studied the material covered by the post-test in the week before the final examination. A third is that graded examinations elicita higher level of motivation and effort than ungraded assessments. Thus, if an instruc-tor wishes to measure the actual end of course learning gains, it appears that the useof embedded questions in an examination might be a better choice than a post-test.

The study also indicates that the difference between graded and ungraded assess-ment results is significant for lower level students when K/C questions and A/Aquestions are separately considered. There were significantly higher gains between thepost-test and the final examination for A/A questions than for K/C questions, support-ing H2. Thus, irrespective of the reason for the difference between post-test andexamination results, it differentially affects K/C and A/A questions. Additionally agreater proportion of the learning exhibited on the final exam was captured by thepost-test for higher level students than for lower level students, supporting H3.

When only higher level students are considered, the difference in gains betweenthe two types of questions (K/C and A/A) is not significant. Given the level ofknowledge these students brought into the course, the pre/post-test for the higherlevel students may not have been difficult enough to allow for significant differ-ences between the post-test results and the final exam results. The higher levelstudents scored sufficiently well on the pre-test to make it difficult to determine thechange in student mastery of learning objectives that resulted from the delivery ofthe curriculum.

The use of a pre-test can allow instructors to determine entry level knowledge,which may well change over time, and to adjust curriculum over time to better meetstudent needs. This supports the periodic use of pre-tests to assess the adequacy ofcurriculum. Pre-tests may also indicate the need to add some form of levelling exer-cise early in the semester to equalise entry skills across students. For instance, ALEKSfor Financial Accounting (www.aleks.com) is a programme that can be used to assessthe accounting knowledge of students entering the Intermediate Accounting sequence.It identifies and remediates, through guided exercises, any gaps in learning studentsmay have, allowing all to begin the class with the same background knowledge. Othersuch levelling programmes may exist for other courses.

The design of the study has some limitations since it does not use a control group,relying instead on a comparison of one group to itself after the group is subjected tothe delivery of specific curriculum. A limitation resulting from this design is that it isnot possible to reliably isolate the changes that occurred as a result of delivery ofspecific curriculum from changes resulting from other possible causes, as discussedpreviously in this paper.

This study has several limitations related to the student body and the courses exam-ined. First, it involves only a small number of students at one university. Students atother institutions may have behaved differently. Second, there were students who did

435

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

10 E. Boyas et al.

not complete the course and whose pre-test results were, therefore, not included in thestudy. Third, lower level students participated in a review after the post-test but beforethe final exam. This may have increased the differences between the ungraded post-test and the graded final exam.

Additionally, the results could have been affected by the nature of the courses andcourse content. While lower level students are not very likely to have encounteredaccounting terminology outside the classroom, the higher level students are likely tohave had some exposure to the subject matter covered in an individual tax course.They are likely to have been filing their own tax returns since most have some workexperience and may have been exposed to some tax topics in other courses. The higherlevel students may also, on average, have taken the entire exercise, right from thebeginning, more seriously than the lower level students since non-performing studentswill have already abandoned the degree programme.

In conclusion, this study provides evidence that while pre/post-tests can be usefulin assessing student mastery of learning objectives for both lower and higher levelstudents, some conditions may affect this usefulness. Pre/post-test differences mayunderstate student mastery of learning objectives especially for students in lower level(non-major) classes and for more complex/higher order learning objectives.

Notes on contributorsElise Boyas is a clinical assistant professor of accounting at the Katz Graduate School of Busi-ness, University of Pittsburgh in Pittsburgh, Pennsylvania. She earned her PhD from Rutgersand has taught graduate and undergraduate accounting courses at a variety of institutionsincluding Robert Morris University, the University of Pittsburgh, and Rutgers. Prior to enteringacademia, she was an accounting practitioner. Her research interests centre on pedagogicalissues in the delivery of undergraduate and graduate accounting curriculum.

Lois D. Bryan earned her doctorate in information systems and communication from RobertMorris University and is a licensed certified public accountant. She is a professor of accountingat Robert Morris University in Moon Township, Pennsylvania. She teaches taxation, manage-rial accounting and financial accounting and conducts research in the areas of assessment,Sarbanes-Oxley compliance and taxation.

Tanya Lee earned her PhD at Arizona State University. She has taught at several universitiesand is an associate professor of accounting at Robert Morris University in Moon Township,Pennsylvania. She has published articles in a number of journals in both managerial accountingand accounting information systems. Her current research interests relate to managerialaccounting, accounting information systems and assessment.

ReferencesAACSB. 2008. Eligibility procedures and accreditation standards for business accreditation.

http://www.aacsb.edu/accreditation/process/documents/AACSB_STANDARDS_Revised_Jan08.pdf (accessed May 8, 2009).

AASCU. 2006. Value-added assessment: Accountabilitys new frontier. Perspectives, Spring.ACBSP. 2006. Best practices in outcomes assessment. http://www.acbsp.org/down-

load.php?sid=175 (accessed May 8, 2009).Accounting Education Change Commission (AECC). 1990a. Issues statement number one:

AECC urges priority for teaching in higher education. Sarasota, FL: American AccountingAssociation.

AECC. 1990b. Position statement number one: Objectives of education for accountants.Sarasota, FL: American Accounting Association.

436

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4

Assessment & Evaluation in Higher Education 11

Albrecht, W.S., and R.J. Sack. 2000. Accounting education: Charting the course through aperilous future. Sarasota, FL: American Accounting Association.

American Institute of Certified Public Accountants. 1999. Core competency framework forentry into the accounting profession. http://www.aicpa.org/edu/corecomp.htm (accessedSeptember 14, 2008).

Banta, T.W., and G.R. Pike. 2007. Revisiting the blind alley of value added. AssessmentUpdate 19, no. 1: 12, 1416.

Bloom, B.S. 1956. Taxonomy of educational objectives, handbook I: The cognitive domain.New York: David McKay.

Calderon, T.G. 2005. Assessment in the accounting discipline: Review and reflection. InAssessment of student learning in business schools: Best practices each step of the way,ed. K. Martell and T.G. Calderon, vol. 1, 187206. Tallahassee, FL: Association forInstitutional Research.

Calderon, T.G., B.P. Green, and M. Harkness. 2004. Best practices in accounting programassessment. Sarasota, FL: American Accounting Association.

Cole, J.S., D.A. Bergin, and T.A. Whittaker. 2008. Predicting student achievement for lowstakes tests with effort and task values. Contemporary Educational Psychology 33, no. 4:60924.

Department of Education. 1983. A nation at risk: The imperative for educational reform. http://www.ed.gov/pubs/NatAtRisk/recomm.html (accessed September 15, 2008).

Drennan, J., and A. Hyde. 2008. Controlling response shift bias: The use of the retrospectivepre-test design in the evaluation of a masters programme. Assessment & Evaluation inHigher Education 33, no. 6: 699709.

Ewell, P.T. 2002. An emerging scholarship: A brief history of assessment. In Building ascholarship of assessment, ed. T.W. Banta and associates, 325. San Francisco, CA: JohnWiley.

Harlen, W., and R.D. Crick. 2003. Testing and motivation for learning. Assessment inEducation 10, no. 2: 169207.

Kimmell, S.L., R.P. Marquette, and D.H. Olsen. 1998. Outcomes assessment programs:Historical perspective and state of the art. Issues in Accounting Education 13, no. 4: 85168.

Mann, S. 1997. Implications of the response-shift bias for management. Journal of Manage-ment Development 16, no. 5: 32836.

Meyers, N.M., and D.D. Nulty. 2009. How to use (five) curriculum design principles to alignauthentic learning environments, assessment, students approaches to thinking and learn-ing outcomes. Assessment & Evaluation in Higher Education 34, no. 5: 56577.

Pascarella, E.T., and G.C. Wolniak. 2004. Change or not to change Is there a question?Journal of College Student Development 45, no. 3: 3535.

Suskie, L. 2004. Assessing student learning: A common sense guide. Bolton, MA: AnkerPublishing Company.

Warren, J. 1984. The blind alley of value added. American Association for Higher EducationBulletin 37, no. 1: 103

Wise, S.L., and C.E. DeMars. 2005. Low examinee effort in low-stakes assessment: Problemsand potential solutions. Educational Assessment 10, no. 1: 117.

437

Dow

nloa

ded

by [U

nivers

ity of

Cam

bridg

e] at

03:23

14 O

ctobe

r 201

4