toward an automated student feedback system for text based assignments - peter vitartas - la trobe...
TRANSCRIPT
![Page 1: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/1.jpg)
Dr Peter Vitartas, Prof Damminda Alahakoon, Dr Sarah Midford, Ms Nilupulee Nathawitharana, Dr Kok-Leong Ong, Prof Gillian Sullivan-Mort
Toward an automated student feedback system for text-based assignments
![Page 2: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/2.jpg)
2
Agenda
• Background
• Automated writing evaluation tools
• Next generation rubrics project
• Results
• Discussion
![Page 3: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/3.jpg)
3
Background
![Page 4: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/4.jpg)
4
Automated writing evaluation
e-rater®
Revision Assistant
![Page 5: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/5.jpg)
5
Types of error comments provided by Criterion ®
![Page 6: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/6.jpg)
6
Automated writing evaluation
e-rater®
Revision Assistant
![Page 7: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/7.jpg)
7
Pearson’s Intelligent Essay Assessor
![Page 8: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/8.jpg)
8
IEA’s feature list used in Write-to-learn
![Page 9: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/9.jpg)
9
Automated writing evaluation
e-rater®
Revision Assistant
![Page 10: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/10.jpg)
10
Vantage Learning
• My access provides:
• writing aids,
• word processing capabilities and
• teacher analytics
• IntelliMetric analyses “400 semantic-, syntactic-, and discourse-level features to form a composite sense of meaning”
![Page 11: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/11.jpg)
11
Automated writing evaluation
e-rater®
Revision Assistant
![Page 12: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/12.jpg)
12
LightSide and Revision Assistant
• LightSide is used as the turnitin scoring engine
• Revision Assistant – gives ‘signal checks’ as formative feedback– Needs to be trained
![Page 13: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/13.jpg)
13
Automated writing evaluation
e-rater®
Revision Assistant
![Page 14: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/14.jpg)
WriteLab
14
• Provides suggestions about:– Clarity– Logic– Cohesion– Grammar
![Page 15: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/15.jpg)
15
Next Generation Rubrics
Push towards larger subject cohorts:– 30% of La Trobe subjects = 100+ students– 41 subjects = 500+ students– BA Interdisciplinary Cores: ‘Rethinking Our Humanity’ (HUS1FAS) and ‘Ideas that Shook the World’ (HUS1TEN) = 1000+ students
Consequences:– Increased pastoral care, assessment feedback and marking– Less time to interact with individual students– Slower turn around on assessment feedback– Difficult to monitor gaps in knowledge at a cohort level– Difficult to assess critical thinking and written skills frequently and meaningfully– Lower student satisfaction
Good pedagogical practice demands quick turn-around times on student assessment feedback, but staff resources are limited and ever-increasing student diversity requires more pastoral care
![Page 16: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/16.jpg)
16
Our solution
Online feedback tool for text based assessments
Students get: immediate feedback on their written work designed to highlight areas that could be improved before submission for marking
Staff get: a dashboard summarising student performance, knowledge gaps and patterns at a cohort level
Benefits—
• Students receive more guidance on their written assessments than staff could otherwise provide
• Student feel less self conscious about the quality of their work because online submission is anonymous
• Staff can look at submission data to see where student skills and knowledge gaps lie and target their teaching more effectively to student needs
• Staff can use data to better design their assessments
![Page 17: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/17.jpg)
17
Student interface
![Page 18: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/18.jpg)
18
Student interface
![Page 19: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/19.jpg)
19
Staff interface
![Page 20: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/20.jpg)
20
Staff interface
![Page 21: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/21.jpg)
21
A framework for evaluating assignments
Assignment statistics
Readability
Research
Critical thinking
Discipline theory
![Page 22: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/22.jpg)
22
Pilot study subjects
Undergraduate Subject:
HUS1TEN – Ideas that Shook the World
1st year BA Interdisciplinary Core Subject
Assessment: Essay
Postgraduate Subject:
MKT5MMA – Marketing Management
MA level Business Subject
Assessment: Report
![Page 23: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/23.jpg)
23
Assignment statistics
Calculates average and individual:– word count– paragraph count– page count– spelling and grammar error count
Assignment statistics
![Page 24: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/24.jpg)
24
Readability
• Calculates Flesch reading ease and Flesch-Kincaid grade level to measure the readability
• Investigates the average readability of the assignments across marks categories
Marks Category Average Flesch Reading Ease
0-49 44.07000046
50-59 41.39999989
60-69 38.35882308
70-79 33.25000004
80-89 29.19999997
90-100 19.5
HUS1TEN Readability statistics
Readability
![Page 25: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/25.jpg)
25
Research
• Counts correctly formatted in-text references (citations) written following La Trobe Harvard style
• Counts all in-text references (citations) disregarding format• Counts number of distinct authors in in-text references (citations)
• Counts number of references in reference list
Research
![Page 26: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/26.jpg)
26
Critical thinking
• Extracts frequencies for both stemmed/non stemmed words using Porter’s stemming algorithm (Porter, 1980)
• Captures phrases with frequencies using TerMine web tool (Frantzi, Ananiadou, & Mima, 2000)
• Identifies Critical Thinking words and phrases and calculates the occurrence of relevant words and phrases across marks categories
Critical thinking
![Page 27: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/27.jpg)
27
Discipline theory
• Extracts frequencies for both stemmed/non stemmed words using Porter’s stemming algorithm (Porter, 1980)
• Captures phrases with frequencies using TerMine web tool (Frantzi, Ananiadou, & Mima, 2000)
Discipline theory
![Page 28: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/28.jpg)
28
Most frequent stemmed words – HUS1TEN
Word Frequency
relationship 1788
love 1558
peopl 648
idea 638
societi 590
half 564
marriag 513
find 507
individu 503
sexual 409
Essay Question:The idea that in love you are finding “your other half” has a long history. But is the idea that we are all one half of a whole still relevant in the current climate of changing gender norms, open relationships and soaring divorce rates? Please discuss the extent to which we are simply one half of a whole with reference to one or more of the following:• Gender ideals• Polyamory• Platonic love• Individualization• The ‘pure relationship’ (Giddens)
![Page 29: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/29.jpg)
29
Most frequent phrases
Phrase Frequency
pure relationship 197
romantic love 125
soul mate 76
many people 46
confluent love 39
platonic love 35
monogamous relationship 22
current climate 21
gay marriage 21
polyamorous relationship 20
Essay Question:The idea that in love you are finding “your other half” has a long history. But is the idea that we are all one half of a whole still relevant in the current climate of changing gender norms, open relationships and soaring divorce rates? Please discuss the extent to which we are simply one half of a whole with reference to one or more of the following:• Gender ideals• Polyamory• Platonic love• Individualization• The ‘pure relationship’ (Giddens)
![Page 30: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/30.jpg)
30
Human marked Grade 0-19(F/<50%)
20-23(D/50-59 %)
24-27(C/60-69%)
28-31(B/70-79%)
32-40(A/80+%)
All Assignments
# of Assignments 3 12 18 20 7 60Critical Thinking Average # Critical Thinking terms
16.7 17.5 15.7 17.7 18.7 16.3
Top 3 critical thinking terms (by frequency)
AnalyseWorldFact
WorldAnalyseData
AnalyseProblemWorld
AnalyseDataWorld
DataAnalyseProblem
AnalyseDataWorld
Citations Av Citations 9.7 17.7 12 13.2 30.7 15.2Av Distinct Authors 5.3 8.25 7.2 6.8 13.6 7.9Word Statistics Av Word Count 2151 2603 2697 2758 3284 2739Av Paragraph Count 45 66 66 68 110 70Av Spelling Error 8.3 26.8 27.5 26.5 35 26.9Av Grammar Error 1.7 6.8 10.0 7.7 4.9 7.6Readability Flesch Reading Ease 35.3 39.3 39.3 38.7 33.4 37.8Flesch-Kincaid Grade Level 14.5 12.8 12.7 12.9 14.1 13.0
![Page 31: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/31.jpg)
31
Initial findings
• Greater understanding of student performance in a large class– Distribution of mistakes– Terms and concepts not being considered– Use of references– Extent of critical thinking terms being used in assignments
• Greater understanding of tutor performance– Variability in marking weightings– Distribution of grades– Consistency in marking assignments
![Page 32: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/32.jpg)
32
Continuing work
• Refinement of dimensions for the assessment classifier– Testing of alternative word extraction methods (LDA, semantic analysis and NLP)– Development of ontologies for discipline areas
• Improved staff user interface being built– User-friendly settings for assignment parameters– Easy to use and relevant reporting outputs
• Improved student interface being built– User-friendly assignment submission– Assignment feedback that is easy to read, understand and put into action
• Further testing of assignments
![Page 33: Toward an automated student feedback system for text based assignments - Peter Vitartas - La Trobe University](https://reader035.vdocument.in/reader035/viewer/2022070523/58edc9671a28abea2b8b461f/html5/thumbnails/33.jpg)
33
Thank you!
Dr Peter [email protected]
Acknowledgement: La Trobe Learning and Teaching Digital Learning Strategy Innovative Research GrantSpecial Thanks: James Heath, Aleksandra Michalewicz and Tanvir Ahmed