Online invigilated exams (staff guide)

Institute for Teaching and Learning Innovation (ITaLI)

11 May 2020

Online invigilated exams (staff guide) 2

Contents

Introduction ....................................................................................................................................................... 3

How does online invigilation work? ..................................................................................................................... 3

Before the exam ................................................................................................................................................ 4

Is my exam suitable for online invigilation? ........................................................................................................ 4

What type of exam can I create in Learn.UQ? ................................................................................................... 4

What do I have to do to prepare for an online exam? ........................................................................................ 4

Exam design ....................................................................................................................................................... 4 Writing effective Multiple Choice Questions (MCQs) ......................................................................................... 5 Writing the question stem ................................................................................................................................... 5 Writing the answer options ................................................................................................................................. 5 Update my Electronic Course Profile (ECP) ....................................................................................................... 6 Suggested ECP text ........................................................................................................................................... 6 Provision of sample exam paper ........................................................................................................................ 6 View online invigilation from a student’s perspective ......................................................................................... 7 Prepare an online deferred exam ....................................................................................................................... 7 What about supplementary exams? ................................................................................................................... 7

How do I help prepare students? ........................................................................................................................ 7

What about students that have technical problems? .......................................................................................... 7

What technical or other support is available? ..................................................................................................... 8

What are the processes for scheduling an online invigilated exam? ................................................................. 8

How are exams scheduled for students in different time zones? ....................................................................... 8

How are students with approved exam adjustments catered for? ..................................................................... 9

What are my responsibilities on exam day? ................................................................................................ 10

Student queries on exam content during the exam .......................................................................................... 10

ProctorU student assistance ............................................................................................................................. 10

Post exam ........................................................................................................................................................ 11

How do I know if an ‘incident’ has occurred? ................................................................................................... 11

What follow up is required if there has been an incident? ................................................................................ 11

Online invigilated exams (staff guide) 3

Introduction

The University of Queensland (UQ) is introducing online invigilation of examinations as a key capability to

mitigate against the COVID-19 impact on students and staff.

This is a rapid deployment. Many aspects of how we support online invigilation are emerging and need to be

responsive to wider developments beyond our control.

This document provides UQ staff with an end-to-end overview of the capabilities of the online invigilation

system (ProctorU) and how it will align with UQ’s existing policies, teaching practices and administrative and

academic processes.

Additional technical documentation will be made available soon, and one on one assistance with learning

design is available from ITaLI (book a teaching consultation).

How does online invigilation work?

Live invigilators guide test-takers through the setup of an account on ProctorU and check the exam

launch through Blackboard to the assessment item. This process involves multi-factor identity

verification and environment security scans where unpermitted materials can be removed before the

exam.

Human invigilators monitor test-takers during the full exam, flag suspicious events and intervene when

necessary. Artificial intelligence (AI) acts as a second set of eyes for the proctor, constantly monitoring

behaviours and alerting the proctor to suspicious activity such as lighting changes, unusual noises and

looking off-screen consistently. The whole exam taking process is recorded and can be reviewed in the

event of suspicious activity.

For closed book exams, ProctorU shuts down communication software and access to any

unauthorised online resources.

For open-book exams, online resources can be whitelisted. A scan of the room is also

performed for the remote invigilator to check what materials the student will be looking at.

Online invigilated exams (staff guide) 4

Before the exam

Is my exam suitable for online invigilation?

ProctorU only offers online invigilation services rather than providing an end-to-end eAssessment system.

You can create your exam using the Blackboard Test tool or produce a Word format exam paper that

students complete and upload to either Turnitin or the Blackboard Assignment tool. If your exam requires

functionality that is not possible using either of these two methods, such as a student handwriting

calculations or drawing, then your exam will not be suitable for online invigilation.

For the Faculty of Medicine courses that are already using ExamSoft, the option to create the exam using

ProctorU will be available.

What type of exam can I create in Learn.UQ? Using the Blackboard Test tool you can create exams that include standard question types such as multiple

choice, multiple answer, fill-in-the-blanks, and short answer. You can also create a document upload exam

including questions that require other documents such as spreadsheets to be completed and uploaded to

either TurnitIn or the Blackboard Assignment tool. Word format exam papers can include question types

such as short answer, essays, other extended writing genres and completing data tables.

What do I have to do to prepare for an online exam?

Exam design

Your exam tasks should be designed to enable students to demonstrate the learning verbs listed in your

course learning outcomes. What do your learning outcomes say students should be able to demonstrate?

(e.g. define, compare apply, analyse, predict , evaluate, justify). A useful way to check that your

questions/tasks do this is to ask, “How do I make that learning verb visible?” This will give you assurance

that you are setting the right kind of task (or mix of tasks) for your assessment.

Action verb Question or task options

define

A Multiple Choice Question (MCQ): where students have to select the correct definition.

A 'fill in the blanks' question: where students have to select the correct alternative from a drop-down list to complete the definition.

A short answer question: where students have to write out the definition.

compare An automatically marked matching question.

A ‘compare and contrast’ type essay question.

apply A series of multiple choice or short answer questions in relation to a short (60-100 words) scenario or other stimulus material (e.g. an image or a quote).

A short answer problem-based question.

analyse Ask an MCQ or short answer question in relation to a data set or other kind of stimulus (this will test interpretation as well as analysis).

Online invigilated exams (staff guide) 5

predict Follow up the analysis question and ask students to predict or extrapolate what might happen next (don’t randomise questions in Blackboard if you are going to do this).

Provide an incomplete diagram/graph/data set and ask an MCQ where students have to select which option would accurately complete the diagram/graph/data set requiring them to identify trends/patterns, perform calculations, or extrapolate.

Follow this up with a short answer question asking students to explain their answer. This makes the thinking behind the selection more visible.

evaluate An evaluative essay question.

Ask an MCQ that directly asks students to judge something (an assertion or a choice or a decision). Avoid phrasing the distractors as simple binary options e.g. the assertion is ‘wrong’ or ‘right’ or the ‘valid’ or ‘invalid’

Provide a short scenario (60–100 words). Ask an MCQ with a question like Which one of the following actions is the most appropriate in the circumstances? Students have to exercise judgement to select the right option.

justify Follow the evaluative multiple choice question with a short answer question asking students to justify why they selected that answer and not the others. If you are going to use this two question approach to assess students’ higher order thinking skills, do not randomise the question order (you can select to only randomise the answer options).

Writing effective Multiple Choice Questions (MCQs)

Writing effective MCQs and tests takes considerable time and care. This is particularly important when

preparing tests for high-stakes assessment. If you are new to writing MCQs, the following may be helpful:-

Write 1–2 questions each week from what was taught and discussed. It is much easier to do it

progressively than to try to put together a series of questions towards exam time.

As above, set a mix of questions to cover all the learning outcomes in your course. Be particularly

aware of this if there are multiple academics that contribute questions to an exam and make sure

they are not all recall based.

Ask colleagues to peer review the questions and refine if necessary.

Writing the question stem

For simple questions, use clear straightforward language, avoiding negatives (e.g. it is FALSE that).

Avoid unnecessary information or words, particularly if you are providing a short scenario. If using

stimulus material (data set, quote, image etc.), always check the capability of Blackboard first.

Put as much of the question in the stem as possible, rather than duplicating words in each of the

options.

Writing the answer options

Write the correct answer first to ensure you have a clear and defendable correct answer.

Online invigilated exams (staff guide) 6

MCQs should contain four or five answer options (i.e. correct answer plus at least 3 or 4 distractors).

Fewer options increases the statistical probability of students accurately guessing the correct

answer, thereby reducing the reliability of the exam.

Distractors must be incorrect, but should be plausible. To make them more plausible, you can use

words students should be familiar with, or include common misconceptions/mistakes.

Avoid providing ‘cues’. If you use a significant word in the correct answer, make sure you use it in

the distractors as well so as not to immediately exclude or direct attention to particular options.

Check combinations - make sure that multiple use of the same words across distractors does not

give the answer away.

Answer options should be about the same length. Too much detail or different grammatical structure

can give the answer away.

Avoid using “all of the above”. Eliminating one distractor immediately eliminates this option as well.

For exam or assessment task design support, please email italiS12020@uq.edu.au with your course code

and name and brief details of what you would like assistance with. Please put “ProctorU” or “online

invigilated exam” in the subject line to ensure your email is directed to the correct location.

Alternatively, book a time for an individual consultation.

Update my Electronic Course Profile (ECP)

If your application to use ProctorU is accepted, please update your ECP task description by 17 April 2020 to

advise students that their end of semester exam will be completed as an “online invigilated exam”.

You will also need to complete:

an examinations form to request a centrally supported exam

an approval request for change to assessment form.

Suggested ECP text

The final exam will be an online invigilated exam. An online invigilated exam is one that takes

place online, at a scheduled time, using ProctorU invigilation services. You will need a webcam,

speakers (usually built-in to your computer), microphone (often this is built-in to the webcam), a

reliable internet connection and uncluttered physical location where you will be able to take your

exam without interruption. If you do not have a laptop with webcam, speakers and microphone or

access to a PC with webcam, speakers and microphone you may be able to borrow a laptop from

the Library (web.library.uq.edu.au/library-services/services-students/covid-19) or arrange to take

your exam in one of the 24/7 Library spaces. ProctorU has an onboarding process to test your

equipment and to help you prepare for your online invigilated exam. More detail about this type of

exam, including the onboarding process, will be provided in myUQ.

Provision of sample exam paper

Exams created using the Blackboard Test tool can be produced in PDF format – see this guide. You will

need to upload a PDF copy of the exam to the Exams Portal for later release to the Library (refer to key

dates).

Online invigilated exams (staff guide) 7

View online invigilation from a student’s perspective

Staff who are selected to use the ProctorU platform for the Semester 1 2020 final examination period

will be invited to participate in a short mock exam to get a sense of the experience from a student’s

perspective.

To become more familiar with the student’s perspective, read the following ProctorU student guides and

resources:

ProctorU Live resource centre

ProctorU automated equipment check

Prepare an online deferred exam

In addition to the original end of semester online exam, you must prepare an online deferred exam. Deferred

exams must have the same conditions as the original online exam, including format, duration, weighting, etc.

What about supplementary exams?

As outlined in section 3.8 of PPL 3.10.02 Assessment Procedures, supplementary assessment may not

require an exam, but could be another form of assessment. Regardless of the form of assessment,

preparation must be made to deliver the assessment online (regardless of whether or not to be invigilated

online). ProctorU will be available for supplementary assessment if required.

How do I help prepare students? Students need to undertake a mandatory training quiz two weeks prior to their scheduled exam to ensure

they are familiar with the ProctorU online invigilation process. This quiz will ensure that students know how to

insert their exam password and ensure that their systems meet online exam requirements.

A Semester 1 2020 Online Exam Guide will be provided to students in courses with online invigilation, and

made available to staff so they can also provide this in their UQ.Learn (Blackboard) sites. Students will also

be able to seek support from the Library AskUs team.

Below is some sample text you can put in the ‘Announcements’ section of your Learn.UQ course site.

[insert course code and name] is going to be conducted as an online invigilated exam. Students who

have suitable devices will complete a digital exam from an appropriate location. Support will be

provided to determine student device suitability. Students must complete the ProctorU onboarding

process to become familiar with the processes and interface at least two weeks before their online

invigilated exam. More details will be provided in the course Learn.UQ site as the semester

progresses. Students will be expected to check the exam timetable to see when the online invigilated

exam will be scheduled. The minimum requirements to sit an online exam include a webcam,

speakers (usually built in to your computer), microphone (often this is built into the webcam), a

reliable internet connection and a suitable private space where you won’t be interrupted. The Library

has recently released a guide for students that includes a laptop borrowing program:

web.library.uq.edu.au/library-services/services-students/covid-19

What about students that have technical problems?

It is very important to manage your own (and students’) expectations and prepare to undertake the online

exam. Some students will attempt to set up a ProctorU account and determine that they cannot take an

Online invigilated exams (staff guide) 8

online invigilated exam. Similarly, some students may attempt to take the exam and encounter technical

difficulties.

In these cases, students will receive a notification from ProctorU which they can use as acceptable

supporting evidence for approval of a request to defer the exam. Students should endeavour to ensure they

are then in a position to undertake the online deferred exam.

In exceptional circumstances, if a student is unable to complete the online deferred exam, the student should

contact Student Services for advice. An alternate form of assessment may be considered.

What technical or other support is available?

Significant training and support resources are available from ProctorU:

ProctorU Help centre (for test-takers, instructors and administrators)

ProctorU Live resource centre

ProctorU automated equipment check

ProctorU ‘How it works’ guide (PDF, 2MB)

The Library has recently released a guide for students that includes a laptop borrowing program.

If students use the ProctorU automated equipment check and find out they are having issues, the ProctorU

technical support team can help them troubleshoot. They are also available to fix students’ technical

issues in real-time during their exam.

Familiarity sessions will be scheduled for course coordinators running online invigilated exams in late April.

Invitations will be emailed to all participating course coordinators.

What are the processes for scheduling an online invigilated exam? Online invigilated exams must be centrally scheduled. Even though they can be done remotely, they cannot

be completed at a time of your choosing. The process for scheduling an online invigilated exam will be the

same process as current central exam requests. Examinations section will be responsible for configuring

Blackboard courses with the scheduled exam date and time, and providing these details to ProctorU.

The maximum number of students who can start an online exam within the same hour timeslot is 500. All

online invigilated exams will be scheduled over a 24-hour window. When students complete the onboarding

process with ProctorU, they will select an available exam commencement time that best suits them.

How are exams scheduled for students in different time zones?

Students must ensure their student email address and residential address are up to date in mySI-net.

Arrangements can be made for students in different time zones to complete the exam online as near as

possible to the scheduled local exam time. They will receive their individual exam details in a personal

timetable emailed to them following the finalisation of the exam timetable.

The Examinations Section will be responsible for scheduling exams for students in different time zones, and

configuring these individual exceptions in Blackboard, provided that the student details are correct in mySI-

net.

Online invigilated exams (staff guide) 9

How are students with approved exam adjustments catered for? Students with exam adjustments approved through Diversity, Disability and Inclusion are required to notify

the Examinations Section of any upcoming exams for which they require their accommodations for as soon

as possible. Most exam accommodations such as extra time and rest breaks can be implemented for

proctored sessions, however there may be other adjustments where student and DDI Adviser will work with

the course coordinator to find a suitable alternative.

The Examinations Section will be responsible for scheduling exams for students with alternative exam

arrangements, and configuring these individual exceptions in Blackboard.

Online invigilated exams (staff guide) 10

What are my responsibilities on exam day?

Student queries on exam content during the exam

Academic staff are responsible for the quality of the online exam. Given students may not all undertake

the online exam at the same time, or in the same time zone, and that some questions may be

randomised, responding to student queries and/or relaying corrections to exam content during the exam

will not be feasible.

Provision must be made in the online exam for a free text field for students to insert any specific queries

or concerns they had regarding the content of the exam.

Reference to this should be included in the instructions to students. For example,

Please specify, in the text box below, any assumptions you have made in the completion of the examination. You may include a statement that represents a query you may have made with respect to a particular question, should you have been able to ‘raise your hand’ in an examination room.

Academics should review these comments when marking, and consider any appropriate action which may

be required. For example, in the event of similar comments from a number of students, consider the re-

weighting of the marks for the question.

ProctorU student assistance If any student experiences difficulties during exam day, they can get assistance from ProctorU and report technical issues. Live chat and web forms are also available for students.

Online invigilated exams (staff guide) 11

Post exam

How do I know if an ‘incident’ has occurred?

Examinations will be informed when unusual testing behaviour is recorded. Incident reporting on

questionable sessions is provided within 24 hours, including end-to-end recordings with time-stamps. Exam

administrators can also choose to receive real-time incident notifications. Consistent with the standard

procedure for on-campus exams, if Examinations finds sufficient cause, they will refer the unusual

behaviour to an Academic Integrity Officer to be investigated under UQ’s Student Integrity and Misconduct

Policy.

What follow up is required if there has been an incident?

See UQ’s Student Integrity and Misconduct policy for the appropriate follow-up steps.

Online invigilated exams (staff guide) 12

Contact details

Institute for Teaching and Learning Innovation (ITaLI) E italiS12020@uq.edu.au W itali.uq.edu.au

CRICOS Provider Number 00025B