how effective is feedback in cad
TRANSCRIPT
This article was downloaded by: [Universiti Teknologi Malaysia]On: 01 September 2013, At: 21:55Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Learning, Media and TechnologyPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/cjem20
How effective is feedback inComputer‐Aided Assessments?Mundeep Gill a & Martin Greenhow ba Learning and Teaching Development Unit, Brunel University,Middlesex, UKb Department of Mathematical Sciences, Brunel University,Middlesex, UKPublished online: 17 Sep 2008.
To cite this article: Mundeep Gill & Martin Greenhow (2008) How effective is feedbackin Computer‐Aided Assessments?, Learning, Media and Technology, 33:3, 207-220, DOI:10.1080/17439880802324145
To link to this article: http://dx.doi.org/10.1080/17439880802324145
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.
This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions
Learning, Media and TechnologyVol. 33, No. 3, September 2008, 207–220
ISSN 1743-9884 print/ISSN 1743-9892 online© 2008 Taylor & FrancisDOI: 10.1080/17439880802324145http://www.informaworld.com
How effective is feedback in Computer-Aided Assessments?
Mundeep Gilla* and Martin Greenhowb
aLearning and Teaching Development Unit, Brunel University, Middlesex, UK; bDepartment of Mathematical Sciences, Brunel University, Middlesex, UKTaylor and Francis LtdCJEM_A_332581.sgm(Received 30 November 2007; final version received 2 May 2008)10.1080/17439880802324145Learning, Media and Technology1743-9884 (print)/1743-9892 (online)Original Article2008Taylor & [email protected]
Computer-Aided Assessments (CAAs) have been used increasingly at Brunel Universityfor over 10 years to test students’ mathematical abilities. Recently, we have focussed onproviding very rich feedback to the students; given the work involved in designing andcoding such feedback, it is important to study the impact of the interaction between studentsand feedback. To make feedback more focussed, examination scripts have been analysedto identify common student errors and misconceptions. These have then been used to codedistracters in multiple-choice and responsive numerical input-type questions. Sincerandom parameters are used in all questions developed, distracters have to be coded asalgebraic or algorithmic mal-rules. This paper reports on the methodology used to identifystudents’ errors and misconceptions and how the evidence collected was used to code thedistracters. The paper also provides hard evidence that real learning has taken place whilestudents have interacted with the CAAs. Statistical analyses of exam performance overeight years indicate that students are able to improve performance in subsequent formativeand summative assessments provided that they have truly engaged with the CAA,especially by spending time studying the feedback provided.
Keywords: feedback; computer-aided assessments; formative assessments; mal-rules;evaluation
Introduction
Computer-Aided Assessments (CAAs) have been widely used at Brunel University over thelast 10 years for both summative and formative purposes. The need to provide feedback tostudents has become more pressing with the ever-increasing numbers of increasinglydiverse students entering Higher Education (HE) which makes the process of providingstudents with useful formative assessment much more difficult. Resource constraintshave also led to infrequent assignments being integrated into a course (Gibbs and Simpson2004; Yorke 2003). The combination of these two factors has resulted in a reduction in thequantity, quality and timeliness of feedback (Gibbs and Simpson 2004; Brown, Bull, andPendlebury 1997). Timeliness of feedback is a key factor for any formative assessmentmechanism and should be provided whilst students are still engaged with the assessmenttask. Hence, the demand for CAA has increased at Brunel University, both within andbeyond the mathematics department. The advantages of using CAA to design and deliverassessments have been widely discussed in the literature and an extensive overview isprovided by Bull and McKenna (2004). The feedback that is provided via CAA needs tobe in a form such that students are able to learn from mistakes and correct errors or
*Corresponding author. Email: [email protected]
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
208 M. Gill and M. Greenhow
misconceptions (Brown and Glasner 1999; Gibbs and Simpson 2004; Ramsden 2005). Self-evidently, it is important to alert students to their mistakes so that they can improve in areasof weaknesses and avoid repeating the same mistakes. At present, lack of feedback is arecurring student concern, as measured, for example, in the National Student Satisfactionsurvey (NSS 2007).
Formative assessment is not only about providing students with feedback, but it is alsoseen as a mechanism to drive and support students in their learning (Sadler 1989). Engagingwith formative assessments enables a process whereby students can recognise, evaluate andreact to their learning, which may involve them reflecting on the task or receiving feedbackon their learning (Bell and Cowie 2001).
Students therefore have a central role in formative assessments and must be activeparticipants, using the feedback provided to modify their concepts or cognitive map of thetopic, or to close skill gaps (Taras 2002). They must therefore identify their strengths andweaknesses before they can make progress with their learning (Harlen and James 1997).These benefits will only occur if they actually engage with the intended activity. Whilstsuch engagement is often driven by marks, at least initially, students will need feedback thatconnects with their existing knowledge and encourages them, thus promoting learning. Ifthe quality of the feedback provided is of a high standard and the overall assessment task isrelated to current work and students’ abilities (i.e. topics covered in the course at the rightlevel), then students should be able to see and capitalise on the benefits of engaging withfeedback. This paper provides evidence that this is precisely what happens and quantifiesthe efficacy of the CAA on students’ subsequent exams marks, which provide a measure ofstudent learning.
Although the advantages of CAA have been widely discussed in the literature, the prom-ised productivity benefits have been slow to appear (Conole and Warburton 2005). Manycase studies have reported the effectiveness of CAA, e.g. Pitcher, Goldfinch, and Beevers(2002), Wood and Burrow (2002) and Pattinson (2004). In each case study, the aim of test-ing the CAA is different, for example, Pitcher, Goldfinch, and Beevers (2002) reported onthe effectiveness of incorporating assessments into the computer-aided learning packageMathwise and one of the issues investigated was how useful students found the assessments.In a similar way, Wood and Burrow (2002) used questionnaires to obtain students viewson the assessment system TRIADS. Pattinson (2004) goes further than just collectingqualitative data, by incorporating statistical analysis in the evaluation of the CAA she usedin her module. Pattinson (2004) compared results students achieved in the online end-of-module examination against the number of times students accessed the CAA for formativepurposes before the (summative) examination. The results indicated that there was a highlysignificant positive correlation between the number of times a student accessed the CAAand the examination mark of the summative assessment. Although the results are positive,Pattinson also reported that a proportion of students (38%) did not make use of the CAA forformative purposes.
Providing directive feedback
In formative assessment, students’ work is marked not only to relay and justify the gradestudents have achieved, but also to aid students with additional comments (i.e. feedback) toimprove their knowledge, understanding and their problem-solving strategies (Brown, Bull,and Pendlebury 1997). For the objective questions of CAA, the first two are more-or-lessautomatic and we here concentrate on the third item. Our questions not only tell studentstheir mark, but, when they have answered incorrectly, we test their input against that resulting
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
Learning, Media and Technology 209
from a coded mal-rule so that students are also told not just that they are wrong, but quiteprecisely where they went wrong. A fully worked solution is also provided which studentscan then use to identify and compare their solution strategy with the correct one. In beingtold of the exact error made and being provided with a fully worked solution, it was conjec-tured that students would be able to use this feedback to improve upon their underlyingskill set, particularly with respect to the tested skill(s), and would then quickly identify theadvantages of engaging with the CAA. Consequently, students will invest time in doing thetests, expecting some return on this investment, such as useful feedback. Therefore, it isimportant to spend time and attention on providing students with detailed and constructivefeedback.
To make distracters more reliable and realistic, and hence the feedback more targeted,students’ examination scripts were analysed to identify the types of mistakes that studentsactually make and common misconceptions that they have in the area of mechanics. Intotal 341 end-of-module examination scripts were analysed across two levels (level zero andone) and across two departments in different institutes (the Physics Department at theUniversity of Reading and the department of Mathematical Sciences at Brunel University).Mistakes students made were categorised into error types and a taxonomy of error types wasproduced before and during the analysis (see Table 1). The classification has been kept asgeneral as possible so that the same taxonomy can be used for other areas of mathematics,and possibly in other areas of science and technology, although it may be necessary to addfurther classes.
In addition to identifying the most common types of mistakes, the exam script analysiswas used to code distracters in the CAA questions. Since all CAA questions developedat Brunel University make use of random parameters, the distracters have to be codedalgorithmically as mal-rules (Gill and Greenhow 2005). For this reason, before the commonerror types that students make could be coded, each one had to be checked for reliability.This means that identified errors have to be checked so that when coded as mal-rules (i.e.
Table 1. Taxonomy of errors and their classification.
Error type Classification
Assumption Students assume certain things that are not true, for example, in projectile questions, that vertical velocity is equal to initial velocity
Calculation Method correct but calculation errors are madeCopying errors Copying values incorrectlyDefinition Not knowing the definition of terms given in question text, e.g. magnitudeFormulas Incorrectly stating/recalling formulasIncorrect values
usedUsing incorrect values in method, for example, when substituting values into
formulasKnowledge Knowledge students are lacking that would enable them to answer questionsMethodology Students attempt to use an incorrect method to answer a questionModelling Unable to model a particular situation/arrangement, i.e. unable to identify all
forces or the correct forces acting on the particleProcedural The method student attempts to use is correct but he/she can only do initial/certain
stages of the method. They stop halfway through when they do not know the stages that follow or when they are unable to interpret initial results
Reading Reading the question text incorrectly and confusing the value of variablesTrigonometry
errorsBasic definitions of cosine, sine and tan incorrect. This is most apparent in
questions where students are required to resolve forces
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
210 M. Gill and M. Greenhow
when random parameters are used) each one returns a unique realistic numerical value (orexpression) and none returns the same value as the correct answer or any other distracter.
Once checked for reliability, these identified errors are coded as distracters in multiple-choice and responsive numerical input type questions. If students make an error thatcorresponds to a coded distracter, they are told of the likely error they may have made andhence shown in the feedback where they went wrong. Simply telling students whether theiranswer is correct or incorrect and being provided with a worked solution is generally notenough (Mevarech 1983). Students need to be told where they went wrong or be given someguidance on how to improve so that they can develop their knowledge and competency inthe skill that is being tested. Therefore, it is vital to have realistic and reliable distracters inthe questions so that the feedback could be tailored to those ends.
Figure 1a illustrates a question that was coded using evidence obtained from the analysisof students’ examination scripts. All distracters were coded algebraically from the mal-rulesshown in Table 2, where the numbering convention comes from Gill (2007).Figure 1a. Question coded as a responsive numerical input type, where the distracters are based on evidence collected from the analysis of students’ examination scripts.Figure 1b. Screen shot showing feedback students receive if they make an error that corresponds to a coded distracter.Figure 2. Graph showing the results obtained on analysing students’ examination scripts.If students make an error that does not correspond to a coded distracter, then they aresimply supplied with the fully worked solution and given some general feedback. The fully
Table 2. Evidence-based (EB) expressions that are returned for each error, when coded as a mal-rule (F1, F2 and F3 correspond to forces, LOD, LAB and LBC correspond to lengths, see Figure 1).
Option/mal-rule Q Classification of error
Correct option —
EB55(b) Modelling
EB60, EB67(b) Methodology
EB62(a) Methodology
EB62(b) Methodology
EB62(c) Methodology
EB62(d) Methodology
EB67(c) Methodology
EB67(d) Methodology
EB62(e) Methodology
Note: The mal-rules are categorised according to the taxonomy of Table 1.
F L F L F L
L L1 2 3OD AB AB
AB BC
+ ++( )
F L
L L3 AB
AB BC+( )F L F L F L
L L1 2 3OD AB AB
AB BC
– –
+( )F L F L F L
L L1 2 3OD AB AB
AB BC
– –
+( )F L F L F L
F F2 3 1AB AB OD
2 3
++( )
–
F L L F F
L L L
1 2OD AB 3
AB BC OD
+ +( )− +( )×[ ]
F L F L
L L1 3OD AB
AB BC
++( )
− − −F L F L L L
L2 3AB AB BC OD
BC
F L F F L
L L3 1 2AB OD
AB BC
− −( )+( )
F L F L F L
L L2 3 1AB AB OD
AB BC
+ −+( )
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
Learning, Media and Technology 211
Figure 1a. Question coded as a responsive numerical input type, where the distracters are based onevidence collected from the analysis of students’ examination scripts.
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
212 M. Gill and M. Greenhow
worked solution is provided with the expectation that students will be able to use this toself-identify their error(s). Initially, it was a thought that students might be ‘put-off’ bysuch lengthy feedback and simply ignore it. Therefore, it was important to identify whetheror not students actually engage with it and if they do, can the effect of this engagement beidentified?
Evaluation of questions
Evidence at school level shows that formative assessment can lead to significant gains(Black and Wiliam 1998; Bell and Cowie 2001). Here we seek to identify whether CAA isa valuable tool for providing feedback to HE students.
Figure 1b. Screen shot showing feedback students receive if they make an error that corresponds toa coded distracter.
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
Learning, Media and Technology 213
Methodology
To test the developed CAA questions, and more importantly, to identify the effectiveness ofproviding students with detailed feedback, computer lab sessions were incorporated into anundergraduate level one mechanics module for two academic years, 2004/5 and 2005/6. Thesessions were an hour long with one of the authors present at every session to help studentsif they had any problems with either the technology or with the subject matter. The labsessions were optional and marks students’ achieved from the CAA did not count towardstheir final module grade. In each session, a new assessment was made available to students,although access to previous assessments was maintained. This was done so that there was astructured approach to the assessments, linking in with topics that were being covered in thelectures at the same time. Each assessment consisted of, on average, five questions so thatstudents would have enough time to complete at least one assessment per lab session.Students could reattempt an assessment as many times as they wanted to, whether withinthe dedicated lab sessions or in their own time.
Answer files were saved for all attempts students made at the assessments and ques-tionnaires were distributed in the last session to collect feedback from the students. A logof student activities recorded how students were using the assessments, i.e. were theyattempting a question or simply moving on to the next one, and how they were interactingwith the questions, e.g. working independently or discussing the questions with theirpeers.
Initial research into the effectiveness of the feedback linked in with the findings fromFormative Assessment in Science Teaching (FAST) project (FAST 2007). The main focusof that project was to investigate, using a questionnaire, how students’ perceptions of theirassessment experience could be enhanced by the feedback they receive from their assess-ments. Results obtained from the lab sessions during the year 2004/5 were used to form ourcase study for this wide-ranging FAST project.
Whilst recording such perceptions is a worthwhile activity, it is certainly not synony-mous with the determination of whether real and lasting learning has actually taken place(here assumed to be measured by subsequent exam performance). We report on two of theevaluation methods that were used to identify whether students actually engaged with theCAA questions and to identify whether students could use this feedback to develop andenhance skills required for the subject area.
Indicators
To identify the level of engagement between the students and the feedback, a number ofindicators were used. An indicator can be any characteristic that is prominently used withinthe questions and feedback. It was of interest to identify whether students had engagedwith the feedback at such a level that they were able to use properly the indicators in theirwritten work. For mechanics, four pervasive indicators were used: diagrams, presentationof solution, units and vectors. Diagrams were used within question text and feedback toexplain the situation/solution more fully. All worked solutions were presented in a similarstep-by-step manner, stating any formulas used and assumptions being made. Units wereused throughout and vectors were displayed correctly (as bold). In order to identifywhether students had engaged with the CAA questions, and more importantly the feed-back, the use of these indicators in students’ examination scripts was investigated. Apartfrom the obvious logistical advantages, examinations usually record students’ activity atthe peak of their knowledge and engagement and consequently, this is when making use of
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
214 M. Gill and M. Greenhow
these four indicators is most revealing. Scripts were analysed from previous academicyears (1999/2000–2003/4), when students did not have access to such a resource, and fromthe two years when lab sessions were incorporated into the module (2004/5–2005/6). Datawas also available for the year after the lab sessions took place (2006/7), where a newlecturer had taken over the module and lab sessions were not integrated into the course,although students were made aware of the resource. Since the uptake of the CAA in theacademic year 2006/7 was not significant, it can be assumed that not much interaction tookplace between students and the CAA.
Figure 2 shows the results obtained on analysing students’ examination scripts.Figure 2 shows that the usage of diagrams increased significantly in students’ written
work in the two years when lab sessions were incorporated into the module. On average,86% of students made use of additional diagrams in their final module examination. Onengaging with the CAA, students clearly recognised the advantages of drawing gooddiagrams to understand and answer the question better. The positive effect the CAA had onstudents in terms of making use of diagrams was further highlighted by the (somewhatnegative) result from the academic year 2006/7 where there was a decrease of 48% in thenumber of students making use of additional diagrams in their written work.
The structured way in which students presented their solutions in the end-of-moduleexamination was consistent over the two years when lab sessions took place. Again thisconsistency may have been due to the engagement between students and the CAA. In theacademic year 2006/7, there was a decrease of 7% in the number of students presenting theirsolution in a step-by-step manner.
The percentage of students stating vectors correctly also increased during the two yearswhen the lab sessions were introduced to the module. Students had engaged with the CAAat such a level they observed that vectors were indicated as bold and hence emulated this intheir written work. The influence the CAA had on students in terms of indicating vectorscorrectly was again highlighted by the lack of students who did so in the academic year
Figure 2. Graph showing the results obtained on analysing students’ examination scripts.
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
Learning, Media and Technology 215
2006/7, where only 22% of students indicated vectors correctly, a decrease of 31% from theprevious year.
Although the feedback had a positive effect on students in terms of three of the fourindicators, the CAA played a detrimental role in terms of students stating units in theirwritten work. Since no question style was coded such that students were required to inputthe units to their corresponding calculated answer, the number of students stating units intheir written work decreased in the two years when lab sessions were incorporated into themodule. Students seemed to have disregarded the units and clearly did not recognise theimportance of units as part of the final answer. This result shows the need for a questiontype where students are required to input the units as well as their calculated numericalvalues, thereby forcing students to be more aware of them. It is interesting to note that thefeedback provided via the Mechanics CAA encouraged students to use diagrams, presentsolutions clearly and state vectors correctly. Without actually being asked to use diagrams(or the other two indicators), students were able to learn from the feedback that diagramswere helpful, presenting solutions in a structured way is practical and vectors need to beindicated correctly. Possibly, the feedback did not focus on units as much as the other threeindicators.
Data collected from the academic year 2006/7 (the year that lab sessions were not inte-grated into the module but students were made aware of the resource) illustrates theimportance of incorporating resources such as CAA into modules (i.e. timetabled slots)rather than just making them available as additional resources. The lack of studentsmaking use of the Mechanics CAA in this particular academic year illustrates that studentsdo not ‘go out their way’ to make use of extra resources. In academic years 2004/5 and2005/6, students had dedicated timetabled slots to access the CAA resource and spendtime interacting with the material. This was the only difference between the two sets ofcohorts since marks achieved from the CAA were not allocated towards students moduleassessment, i.e. students still attended lab sessions and engaged with the Mechanics CAAknowing full well that no marks would count towards their final grade.
Statistical tests
To identify whether students had engaged with the feedback at such a level that they wereable to demonstrate the skills and knowledge required for the mechanics exams a few weeksafter the CAA lab sessions, statistical tests were conducted using exam marks from the1999/2000 to 2005/6 academic years. The set of data from 1999/2000 to 2003/4 provides acontrol since students did not have access to any CAA resource in any topics of mechanics,in contrast to 2004/5–2005/6 when they did.
Figure 3 illustrates the data that was used and the different tests that were computed.Figure 3. Grid showing the different sets of data available and the tests to be computed in order to make the relevant comparisons. Tests 1a and 2a identify any influencing factors that may exist for level of difficulty in topics and academic ability of students, respectively.The mechanics module comprises eight topics, but students were only given access toCAA material that tested four, namely beams, static equilibrium, structures and vectors.Consequently, marks students achieved for these four topics can be compared to the marksawarded in the non-CAA topics, namely centre of mass, Hooke’s Law, kinematics andprojectiles.
A factor that may influence the results is that there may be differing levels of difficultybetween CAA and non-CAA topics. To quantify this, the marks students achieved in thesame questions for all eight topics from previous academic years were used i.e. fromacademic years 1999/2000 to 2003/4. A two-tailed Student’s t-test was used to compare themarks students achieved for the two sets of topics in the academic years when no CAA wasprovided (1999/2000–2003/4). The group statistics for the two groups of topics were:
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
216 M. Gill and M. Greenhow
N Mean mark (from Standard deviation a total of 25)
CAA topics 321 14.14 8.447Non-CAA topics 434 13.33 8.624
The result obtained from the two-tailed unpaired Student’s t-test was:
This result indicates that there is insufficient evidence to suggest that the marks achievedbetween the two sets of topics differ. This shows that there is no material difference in thelevel of difficulty between the two sets of topics and therefore the influence of topic choiceis not significant.
The marks students achieved in academic years 2004/5 and 2005/6, for the questionstesting skills from the four CAA topics, were combined, as were the marks studentsachieved in the non-CAA topics. These two averages were compared to identify whether,overall, students performed better in the topics for which they had access to CAA incomparison to those questions for which no CAA material was given. The group statisticsfor the two sets of topics in the academic years when CAA was available were:
N Mean mark (from Standard deviationa total of 25)
CAA topics 106 15.33 8.216Non-CAA topics 116 13.65 8.492
The result obtained from the one-tailed Student’s t-test was:
t p( ) . , .753 1 292 0 197= =
t p( ) . , .220 1 499 0 0675= =
Figure 3. Grid showing the different sets of data available and the tests to be computed in order tomake the relevant comparisons. Tests 1a and 2a identify any influencing factors that may exist forlevel of difficulty in topics and academic ability of students, respectively.
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
Learning, Media and Technology 217
This result indicates that there is insufficient evidence to suggest that students performedbetter in the topics for which they had access to CAA in comparison to the non-tested topics,at the p-value of 0.05. However, there is evidence at the p-level of 0.1 to support this claim.However, accepting at this p-value means that there is a possibility (i.e. a 1 in 10 chance)that there is no real difference between the two groups’ performance. Overall, this result mayimply that students may not have engaged enough with the CAA to develop and enhanceskills to the extent that a significant difference could be seen in students’ exam performance.
Since questions set in the final examination tested the same skills over a number ofyears, marks were also compared across academic years. The questions testing the differentskills across the academic years did not change significantly and therefore similar questionswere compared. A factor that may influence the results is that different student cohorts maydiffer in academic ability. To identify whether there was a difference in academic abilitybetween those students from previous academic years and those from the two years thatattended lab sessions, marks students obtained from questions that tested skills from thenon-CAA topics are analysed. The group statistics for the two cohorts of students for non-CAA topics were as follows:
Cohort N Mean mark (from Standard deviationa total of 25)
2004–2006 116 13.65 8.4921999–2004 434 13.33 8.624
The result obtained from the two-tailed unpaired Student’s t-test was:
This result implies that there is no evidence to suggest that there is a difference, academi-cally, between the two cohorts of students. Therefore, any improvement in marks found inthe comparison of data across academic cohorts is not due to the fact that students differ inacademic ability.
Marks students achieved for the questions that tested skills from the four CAA topicswere compared across the two sets of year groups (1999/2000–2003/4 and 2004/5–2005/6).Each topic was compared separately so that a clearer inference could be made on whetheror not students engaged successfully with the feedback. The one-tailed Student’s t-test wascomputed for each of the four comparisons, results of which are shown in Table 3.
From these tests, it was found that there was statistically significant evidence to suggestthat students performed better for the two topics static equilibrium and structures in theacademic years 2004/5 and 2005/6, in comparison to students from previous academicyears. However, there was no evidence to suggest that students performed better inacademic years 2004/5 and 2005/6 for beams and vectors. This implies that students wereable to engage with the feedback that was provided for the questions testing skills in staticequilibrium and structures and were able to develop their skills for these two topics.However, this was not true for the topics beams and vectors. The level of feedback providedwas comparable for all questions, as was the organisation of the feedback and so the qualityof the feedback did not alter in any material way between the different topics.
Observations made during the lab sessions, which were recorded in a log at the end ofeach session, revealed that students spent a great deal of time engaging with the feedbackfrom the questions testing skills on structures, which may explain why students were able
t p( ) . , .548 0 355 0 722= =
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
218 M. Gill and M. Greenhow
to use the CAA to develop and enhance skills for this topic rather than for beams or vectors.This was also indicated in the answer files which revealed that only 51% of students wereable to complete the assessment on structures in the hour given, in comparison to 86% ofstudents being able to complete the assessments on static equilibrium in the same time.Moreover, 145 attempts were made at questions on static equilibrium in comparison to only14 attempts made at questions on beams.
The reviewers of this paper both asked why the CAA marks are not substantially higherthan those from previous years. For the two student-engaged topics, the mean mark forstatic equilibrium increased from 44% to 61% and for structures from 43% to 57%. Theseincreases are not small but we remark that this study was embedded in an already successfuland well-run course, so improving the marks yet further is likely to be more difficult thanimproving marks on a course with known problems. Secondly, we believe that evidence-based data may show increases that are generally more modest than those you might expectbased solely on students’ perceptions of how much they are learning. However, this willrequire further study.
Results collected from answer files from the academic year 2006/7 have been usedpurely for subjective inferences. Although students had full access to the CAA, lab sessionswere not incorporated into the module like the previous two years, this meant that studentsdecided their own formative assessment needs and accessed the CAA only when and if theydecided to. From the 21 students registered on the module, only 9 students accessed theCAA, 19% of whom made attempts at four or more assessments (from a total of seven thatwere available). In total, 55 attempts were made at the different assessments, from which75% were made during the month of January 2007, the exam month. This implies thatstudents either waited until the last possible moment before accessing the assessments,when they were perhaps desperate for help or they used it as a revision tool.
On comparing the average result achieved by students who accessed the CAA againstthe class average for the end-of-module examination, it was found that these studentsachieved 10% higher in the examination. This may simply mean that keen students willmake use of a range of resources, including CAA, and plan for their own formative assess-ment needs, whereas less keen students need more support or motivation to do so. By having
Table 3. Results obtained from the one-tailed Student’s t-test and the group statistics for comparisonof marks for the topics for which students in academic years 2004/5–2005/6 had access to CAA, incomparison to students from academic years 1999/2000–2003/4 who did not have such access.
Result from one-tailed Student’s t-test
Topic Cohort NMean mark
Standard deviation Result
p-value obtained Inference
Beams 2004–2006 22 11.64 7.865t(67) = 0.495 p = 0.311 Not significant1999–2004 47 12.66 8.066
Static equilibrium 2004–2006 22 15.27 10.119t(115) = 2.031 p = 0.0225 p < 0.05
1999–2004 95 11.05 8.456Structures 2004–2006 30 14.37 7.01
t(81) = 2.034 p = 0.0225 p < 0.051999–2004 53 10.83 7.924
Vectors 2004–2006 32 6.75 2.553t(156) = 0.522 p = 0.301 Not significant1999–2004 126 6.49 2.481
Note: Questions on Beams, Static Equilibrium and Structures were marked out of 25 and questions on Vectors were marked out of 9.
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
Learning, Media and Technology 219
weekly lab sessions, all students are encouraged to make continuous use of the formativeassessment via CAA and will be able to make use of such resources.
Conclusion
One of the key aims for providing students with CAA for the mechanics module was toidentify whether students were able to engage with the assessment package and with theextensive feedback that was provided. The level of this engagement is important so that avalue judgement can be made on the overall worth of incorporating such a resource intoa module. The term worth here implies a consideration of costs and benefits. From a ques-tion setter’s point of view, costs include the time and energy required to code and developsuch rich feedback and from a students’ side, the amount of time and effort they needto invest.
The evaluation of the CAA has shown that, in the right setting, students do engage withformative assessment activities, even when no marks are allocated. We think that this is dueto both the quality of the CAA and the fact that lab sessions were scheduled in students’timetables, thereby providing a structured and supportive environment.
On successful engagement with the feedback, students are able to map aspects of thefeedback to their written work, such as use of diagrams, presentation of solutions and nota-tion of vectors correctly, i.e. they have developed organisation and presentation skills. Thestatistical analysis indicates that students were also able to develop their mathematicalskills for two of the four CAA topics. The reasons why students developed skills in just twoof the four topics seem to be due to differences in the level of engagement between thestudents and questions and, more simply, the number of attempts students made at thequestions for each topic.
The collected evidence indicates that CAA is an effective tool to provide formativefeedback to students, provided much development work is carried out to ensure thatdistracters are reliable, and which, in turn, focuses the feedback on likely student errors.Students engage with such high-quality feedback and benefits appear to go further thansimply short-term recall.
Much work has been done at Brunel University to provide extensive and useful feedbackto students via CAA in mathematical areas. Consequently, students use the feedback as alearning resource. We believe the same will apply throughout science and technologysubjects. Similarly, the methods used to evaluate CAA as a formative feedback packageshould be applicable to other subject areas.
Notes on contributorsMundeep Gill currently works in the Learning and Teaching Development Unit at Brunel Universityas a Learning Adviser, supporting students with Maths, Stats and Numeracy. Her principal researchinterests lie in computer-aided assessment and the effect it has on students’ performances, emphasis-ing successful engagement with the assessment.
Martin Greenhow is a Senior Lecturer in mathematics with interests in mathematics education,computer-aided assessment and study skills.
ReferencesBell, B., and B. Cowie. 2001. The characteristics of formative assessment in science education.
Formative Assessment in Science Education 85, no. 5: 536–53.
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013
220 M. Gill and M. Greenhow
Black, P., and D. Wiliam. 1998. Assessment and classroom learning. Assessment in Education:Principles, Policy & Practice 5, no. 1: 7–68.
Brown, G., J. Bull, and M. Pendlebury. 1997. Assessing student learning in higher education.London: Routledge.
Brown, S., and A. Glasner, eds. 1999. Assessment matters in higher education: Choosing and usingdiverse approaches. Buckingham: Open University Press.
Bull, J., and C. McKenna. 2004. Blueprint for computer-assisted assessment. London: Routledge-Falmer.
Conole, G., and B. Warburton. 2005. A review of computer-aided assessment. ALT-J, Research inLearning Technology 13, no. 1: 17–31.
FAST. 2007. Formative assessment in science teaching. http://www.open.ac.uk/fast/ (accessedNovember 2007).
Gibbs, G., and C. Simpson. 2004. Conditions under which assessment supports students’ learning.Learning and Teaching in Higher Education 1: 3–31.
Gill, M. 2007. Development and evaluation of CAA as a formative assessment tool for university-level mechanics. Unpublished PhD thesis, Brunel University.
Gill, M., and M. Greenhow. 2005. Learning and teaching via online maths tests. Proceedings of theScience Learning and Teaching Conference, June 27–28, Warwick University, Warwick.
Harlen, W., and M. James. 1997. Assessment and learning: Differences and relationships betweenformative and summative assessment. Assessment in Education: Principles, Policy & Practice 4,no. 3: 365–79.
Mevarech, M. 1983. A deep structure model of students’ statistical misconceptions. EducationalStudies in Mathematics 14, no. 4: 415–29.
NSS. 2007. The National Student Survey. http://www.thestudentsurvey.com/ (accessed November2007).
Pattinson, S. 2004. The use of CAA for formative and summative assessments – Student views andoutcomes. Proceedings of the 8th International CAA Conference, July 6–7, in LoughboroughUniversity, Loughborough.
Pitcher, N., J. Goldfinch, and C. Beevers. 2002. Aspects of computer-based assessment in mathe-matics. Active Learning in Higher Education 3, no. 2: 159–76.
Ramsden, P. 2005. Learning to teach in higher education. 2nd ed. London: RoutledgeFalmer.Sadler, D.R. 1989. Formative assessment and the design of instructional systems. Instructional
Science 18: 119–44.Taras, M. 2002. Using assessment for learning and learning from assessment. Assessment & Evaluation
in Higher Education 27, no. 6: 501–10.Wood, J., and M. Burrow. 2002. Formative assessment in engineering using ‘TRIADS’ software.
Proceedings of the 6th International CAA Conference, July 9–10, in Loughborough University,Loughborough.
Yorke, M. 2003. Formative assessment in higher education: Moves towards theory and the enhance-ment of pedagogic practice. Higher Education 45, no. 4: 477–501.
Dow
nloa
ded
by [
Uni
vers
iti T
ekno
logi
Mal
aysi
a] a
t 21:
55 0
1 Se
ptem
ber
2013