does the medium have an effect on assessment?
DESCRIPTION
Does the Medium have an Effect on Assessment? . George McGuire Martin Youngson Douglas McMillan with Dave Fiddes and Athol Korabinski. Modes of exam delivery. Paper - current exam delivery Computer - growing trend For Mathematics, attractions include - PowerPoint PPT PresentationTRANSCRIPT
Does the Medium have an Effect on
Assessment?
George McGuireMartin YoungsonDouglas McMillan
with Dave Fiddes and Athol Korabinski
Paper - current exam delivery
Computer - growing trend For Mathematics, attractions include automatic (instantaneous) marking logging of results
Are exam results affected by mode of delivery?
Modes of exam delivery
Rewording effect - to fit the assessment package
Medium effect - questions read from the screen input of answers by typing
‘on’ the screen
What is the significance of each of these effects for exam marks?
Pilot study to investigate this question.
Changing the mode of delivery
The questions in the pilot study
Typical of the short questions in paper 1 of the Scottish Higher
exam
Require few steps to reach the answer but still involve a reasonable amount of mathematics.
Paper version of a test (P Format)
Example of a paper question
1. If f (x) = 2x - 3 and g (x) = 2x2 - 3
find an expression for g (f (x)). Write
your answer in the form ax2 + bx + c.
Computer version (ICT format)
Example of a computer question
1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c. What are a, b and c?
1. What is a?
2. What is b?
3. What is c?
CUE
Determination of the medium effect
Comparison between paper and computer versions would have
to take account of both rewording and medium effects
Exact paper copy of the computer version (RT format) eliminates any rewording effect
ICT versus RT for medium effect only
Reverse translation version (RT format)
Example of an RT question
1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c. What are a, b and c?
1. What is a?
2. What is b?
3. What is c?
Determination of the rewording effect
No medium effect between P format
and RT format (both paper)
Working influences marking in a
paper test
Modification of the RT test, RTW
format, allows space for working
Question in (RTW format)
Rough Working1. If f (x) = 2x - 3 and g (x) = 2x2 - 3 then g (f (x)) = ax2 + bx + c.
What are a, b and c?
1. What is a?
2. What is b?
3. What is c?
Pilot study 90 minutes in April 2001
Two schools with Higher Mathematics pupils
The Tests:
1 P format - Three 30 minute tests with SQA
questions
2 ICT format - Three computer tests using CUE
3 RT format - Three reverse translations of the
computer tests
Each candidate sat the 3 tests, one in each
format
Preparation for the pilot study
Pupils familiar with paper exams but generally unfamiliar with computer exams
A visit to each school before the day of the pilot study
Explain what would take place on the day of the pilot
Practice test of some simple questions to give pupils some familiarity with CUE
CUE
Setting up the groups for the test
SQA split the pupils into 3 groups based on gender mathematical ability
Mathematical ability measured by: standard grade results higher prelim results
Running the tests Each candidate took each test -
one in each format
Group sat tests in different orders
At any time no group was sitting
the same test as another group
a test in the same format as
another group
Time was not an issue
Setting up the pairings for the analysis
62 matched pairs in total
59 matched for gender and ability
3 matched for ability not gender
From different groups, matched pairs were set up:
Marking of the tests Paper tests marked by SQA
ICT tests marked automatically by computer
RT tests marked by me, looking only at final answer
RTW tests marked by me, taking account of rough working
Average mark over all tests: 39%
Individual marks range from 0% to 100%
Analysis of marks
2 statistical analyses: P - RTW and ITC -
RTC
Both done on 62 differences in marks
from matched pairs
Null hypothesis is true underlying mean
is zero
One sample t-test performed
62 is large enough to validate analysis
The results for the rewording effect
Well outside the 95% confidence
interval
Strong evidence of a difference
Was this a marker effect?
Marker effect eliminated by P papers
remarked by me
Still strong evidence of a difference
The results for the medium effect
Well inside the 95% confidence interval
No evidence of a difference
The gender effects
Analysis repeated for male and
female pairs
Same results as before
Conclusions on the rewording effect
Differences may have been due to
Rewording
Reformatting
The way the answers were set out
Pupils less inhibited doing rough
working
More detailed study needed to clarify
Conclusions from the medium effect
Candidates had only limited prior
experience of computer tests
There is no evidence of a
difference due to the medium
effect
Thank you
V
Does the Medium have an Effect on Assessment?
Directions for further study
Partial credit using steps
Steps may give strategy
May be able to test areas not
currently tested
CUE