Wanted! Evidence based guidelines for

unseen invigilated examinations

Ieva StupansUniversity of South Australia

Three interrelated objectives for quality in student assessment in

higher education1. Assessment that guides and encourages effective

approaches to learning

2. Assessment that validly and reliably measures expected learning outcomes, in particular the higher-order learning that characterises higher education; and

3. Assessment and grading that defines and protects academic standards

EDUCATIONAL EFFECTIVENESS

(McInnis, 2004)

Wanted! Evidence based guidelines for unseen invigilated examinations

• EDUCATIONAL EFFECTIVENESS

• RESOURCE EFFICIENCY

Wanted! Evidence based guidelines for unseen invigilated examinations

• Analysis of marks on 1,712 modules at Oxford Polytechnic. Modules with 100% coursework had an average mark 3.5% higher than modules with 100% examinations, and there were three times as many failed students on modules where there were only examinations. (Gibbs & Lucas, 1997)

• Four universities - coursework marks higher by one third of a degree classification in English and History and higher by two thirds of a degree classification in Biology, Business Studies, Computer Studies and Law. (Bridges et al. 2002)

Wanted! Evidence based guidelines for unseen invigilated examinations

• Audit from an Australian university indicated that approximately 50% of first year subjects had final examinations within their assessment profile, within sciences this was approximately 80%. (Taylor, 2006)

• With respect to certification or accreditation of learning by an external body such as a health profession accreditation body, for example, FIP recommends ‘a final examination should lead to the granting of a diploma or degree’ . (FIP, 2000)

Wanted! Evidence based guidelines for unseen invigilated examinations

Assessment

for learning = coursework type assessments

of learning = invigilated examinations

Assessment

‘traditional’= learning at the lowest levels of Bloom’s cognitive domain

‘alternative’ = positioned to measure learners’ higher-level thinking skills of synthesis, analysis, and evaluation

Wanted! Evidence based guidelines for unseen invigilated examinations

Assessment of learning is not mutually exclusive to “alternative” assessment

but

is frequently associated with question spotting, cramming and short-term knowledge retention

therefore

the concept of structuring questions which allow students to demonstrate higher-level thinking skills is one which needs to be developed.

Returning to the unseen invigilated exam!

• “Blueprinting”

Assessment and grading that defines and protects academic standards

• problem solving MCQ

Assessment that validly and reliably measures expected learning outcomes, in particular higher-order learning

• “cheat sheets” and open book

Assessment that guides and encourages effective approaches to learning

Higher order learning

“Blueprinting” e.g. statsLevel of skill required

Standards Definitions Compre-hension

Application Analysis Problem solving

Total

Analyse data using mean, median etc

2 MCQ 2 MCQ 8 MCQ 1 written response

12 MCQ+written

Predict from data

6 MCQ 2 MCQ 1 written response

8 MCQ+written

“Blueprinting”

• Tension between efficiency and effectiveness (reliability)

• What is the optimal number of items, balance of items- MCQ, short answer and extended answer?

• Should components be weighted?• Assessment of different learning

outcomes but these are interrelated.

Problem solving MCQ

MCQ in Pharmacology• Questions which require application, analysis,

synthesis and evaluation can be written• Writing of such questions needs to be

intentional ( importance of “blueprinting”) • For NESB students to achieve similar results,

attention needs to be paid as to whether questions are assessing application of pharmacology knowledge or application of pharmacology knowledge and language proficiency.

Cheat sheets and open book

• Students have access to “essential reference books” in examination

• Does this impact on “quality” of answers particularly for weak students.

• Is there a way to optimise student learning using open book and “cheat sheets”

Cheat sheets and open book

Questions• Strategies such that there is motivation for

students to organize all of the materials in their minds prior to the exam

• Strategies so that exam time is not the time for students to work through the texts for the first time

Wanted! Evidence based guidelines for unseen invigilated examinations

Learning logs/diaries

Peer assessment

Annotated bibliographies Projects/case studies

Educational Effectiveness

Resource efficiency

Hornby, 2003

Unseen Examination