lecturers’ use of computer-aided assessment
DESCRIPTION
Lecturers’ use of computer-aided assessment. How history and contradictions help shape lecturers’ practice. Introduction. Background to CAA at the target institution Background to this project Activity theory as the theoretical framework Methodology Our findings at this stage - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/1.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Lecturers’ use of computer-aided assessment
How history and contradictions help shape lecturers’ practice
![Page 2: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/2.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Introduction
• Background to CAA at the target institution
• Background to this project
• Activity theory as the theoretical framework
• Methodology
• Our findings at this stage
• Conclusions and future work
![Page 3: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/3.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Background to CAA at the target institution
![Page 4: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/4.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
The HELM project
Helping Engineers Learn Mathematics
• Started in 2002 by a consortium of universities
• Includes workbooks and a CAA system
• Question bank with thousands of maths questions
Concurrently
• A separate bank of questions was developed for maths students
Harrison et al. (2007), HELM (2006).
![Page 5: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/5.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Example CAA question
![Page 6: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/6.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Example CAA solution
![Page 7: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/7.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Ten years on
• How do lecturers and students use CAA now?
• What sort of student improvement do lecturers and students desire from CAA?
• Does CAA practice affect whether these outcomes are achieved?
![Page 8: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/8.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Background to this project
![Page 9: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/9.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Previous work
Student focus groups
• How students use CAA
• The nature of the feedback given
• The definition of student improvement
• The role of the lecturer
Broughton, Hernandez-Martinez and Robinson (2011, 2012a, 2012b).
![Page 10: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/10.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Activity theory as the theoretical framework
![Page 11: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/11.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Aspects of interest
Students’ practices as influenced by their aims
Students’ identities and its impact on the way they act
Formative assessment with respect to desired results or “improvement”
The role of community and the development of tools
Lecturers’ practices and the influences on it
Dealing with contradictions between practice and idealsEngeström (2000); Roth and Lee (2006).
![Page 12: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/12.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Lecturer activity
SUBJECT Lecturer
OBJECT Assessing students
TOOLS CAA, coursework, exams
COMMUNITY School of MathematicsWider academic community
DIVISION OF LABOUR e-Learning staff create and upload tests
RULES University regulationsPlagiarism rulesDepartmental codes
![Page 13: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/13.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
… or perhaps…
SUBJECT Lecturer
OBJECT Helping students improve
TOOLS CAA, coursework, tutorials, support centres
COMMUNITY School of MathematicsWider academic community
DIVISION OF LABOUR e-Learning staff create and upload testsSupport centre staff direct students to sources of help
RULES University regulationsPlagiarism rulesDepartmental codes
![Page 14: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/14.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
… or even…
SUBJECT Lecturer
OBJECT Saving time (and money)
TOOLS CAA
COMMUNITY School of MathematicsWider academic community
DIVISION OF LABOUR e-Learning staff create and upload testsPostgraduate markers mark coursework
RULES University regulationsPlagiarism rulesDepartmental codes
![Page 15: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/15.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Lecturers’ CAA practice
For this part of the project, we ask:
1. Why are lecturers using CAA?
2. How do lecturers implement CAA in their maths modules?
3. What are lecturers’ perceptions of issues arising?
4. How do lecturers deal with these issues?
![Page 16: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/16.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Methodology
![Page 17: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/17.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Getting answers to questions
• Approached 13 lecturers of mathematics-based modules to complete a questionnaire
• Received nine questionnaires
• Six took part in a follow-up interview
![Page 18: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/18.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Findings
![Page 19: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/19.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Why are lecturers using CAA?
What are the aims of the activity?
Save time by continuing inherited practice
Support student learning by encouraging them to practise
Assess students more frequently by offering immediate feedback
Conform to department influence by following advice from colleagues
![Page 20: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/20.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Why are lecturers using CAA?
What are the aims of the activity?
Save time by continuing inherited practice
Support student learning by encouraging them to practise
Assess students more frequently by offering immediate feedback
Conform to department influence by following advice from colleagues
![Page 21: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/21.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Why are lecturers using CAA?
What are the aims of the activity?
Save time by continuing inherited practice
Support student learning by encouraging them to practise
Assess students more frequently by offering immediate feedback
Conform to department influence by following advice from colleagues
![Page 22: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/22.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
What actions have lecturers made towards their aims?
How do lecturers use CAA in their modules?
CAA coursework testPaper-based
coursework test No test
Invigilated Non-invigilated Invigilated
Access to practice test granted more than one week before a test
Access to practice tests granted after the test
0 2 1 1
Access to practice test granted up to one week before a test
Access to practice tests granted after the test
0 1 0 0
Access to practice tests not granted after the test
3 1 0 0
![Page 23: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/23.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
What issues have lecturers faced?
What contradictions have lecturers experienced while using CAA?
The questions are formulaic and not challenging enough.
The questions do not test conceptual knowledge and understanding.
The feedback is not beneficial to every student.
Lecturers do not know how students approach the tests.
![Page 24: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/24.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
What issues have lecturers faced?
What contradictions have lecturers experienced while using CAA?
The questions are formulaic and not challenging enough.
The questions do not test conceptual knowledge and understanding.
The feedback is not beneficial to every student.
Lecturers do not know how students approach the tests.
![Page 25: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/25.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
What issues have lecturers faced?
What contradictions have lecturers experienced while using CAA?
The questions are formulaic and not challenging enough.
The questions do not test conceptual knowledge and understanding.
The feedback is not beneficial to every student.
Lecturers do not know how students approach the tests.
![Page 26: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/26.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
What issues have lecturers faced?
What contradictions have lecturers experienced while using CAA?
The questions are formulaic and not challenging enough.
The questions do not test conceptual knowledge and understanding.
The feedback is not beneficial to every student.
Lecturers do not know how students approach the tests.
![Page 27: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/27.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
How have lecturers responded to these issues?
How has the activity changed?
Authoring new CAA questions that challenge conceptual knowledge
Replacing the online test with a paper version
Introducing other assessments and restructuring the assessment profile
Reducing the weighting of CAA towards the overall module mark
![Page 28: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/28.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
How have lecturers responded to these issues?
How has the activity changed?
Authoring new CAA questions that challenge conceptual knowledge
Replacing the online test with a paper version
Introducing other assessments and restructuring the assessment profile
Reducing the weighting of CAA towards the overall module mark
![Page 29: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/29.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
What changes do lecturers anticipate making?
How might the activity change in future?
If lecturers are happy with the current setup, we might not expect further change.
If they are unhappy, they might tweak CAA practice further.
They might also seek a different CAA system.
![Page 30: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/30.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Conclusions and future work
![Page 31: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/31.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
In ten years of CAA practice
• Lecturers practices have become more diverse and continue to change.
• Lecturers inherit CAA practice and are keen to continue using CAA.
• They retain CAA as a tool for assessing students, but look to other assessment tools to address contradictions.
• There is a growing desire for a new system that addresses these contradictions.
![Page 32: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/32.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Implications
• The lecturers are likely to continue to use CAA for the foreseeable future.
• They will not rely heavily on CAA for summative assessment and grading.
• They do not believe that CAA is the most appropriate assessment tool for assessing conceptual knowledge.
• There are concerns that students do not use CAA in the manner expected or desired.
![Page 33: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/33.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Future work
Student study
• Questionnaires and individual interviews with students that use CAA
• Aims of students
• Student activity
Lecturer study
• Questionnaires and individual interviews with lecturers that do not use CAA
• Compare aims and activity to lecturers that use CAA
![Page 34: Lecturers’ use of computer-aided assessment](https://reader036.vdocument.in/reader036/viewer/2022062517/56813a02550346895da1cb85/html5/thumbnails/34.jpg)
Stephen BroughtonPaul Hernandez-MartinezCarol L. Robinson
Further information
Short links
• References sbpr.es/rm12a
• This presentation sbpr.es/rm12b