Download - D Whitelock LAK presentation open_essayistfv
![Page 1: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/1.jpg)
OpenEssayist: A supply and
demand learning analytics tool
for drafting academic essays
Institute of Educational
TechnologyDenise Whitelock, Alison Twiner, John
Richardson, Debora Field, Stephen Pulman
![Page 2: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/2.jpg)
Providing meaningful automatic feedback
• Can we provide Advice
for Action to assist
essay writing?
• How can we capture
progress with a
visualisation?
![Page 3: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/3.jpg)
SAFeSEA: Supportive Automated Feedback for
Short Essay Answers
An automated tool
supporting
online writing and
assessment
of essays providing
accurate targeted
feedback
SAFeSEA
Professor Denise Whitelock
Professor John Richardson
Professor Stephen Pulman
![Page 4: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/4.jpg)
About SAFeSEA
• No tutor support for drafts of first
assignment
• Reduce dropout rate with automatic
feedback?
• Effect of summaristion
• What are the beneficial factors?
• Correlate measures of learner activity
and essay improvement
• http://www.open.ac.uk/iet/main/resea
rch-innovation/research-
projects/supportive-automated-
feedback-short-essay-answers
![Page 5: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/5.jpg)
Talk Back
• Checking
understanding by ‘talk
back’
• Summaries in
OpenEssayist
• Key words = key
ideas
• http://www.open.ac.uk
/researchprojects/safe
sea
![Page 6: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/6.jpg)
Key word extraction What we mean by 'key word'
• Not author-assigned key words
• Words that "describe the contents of a document" (Collins ED)
Process: Before graph making comes NLP pre-processing(Python-base NLP toolkit)
• Filter out unwanted parts of speech
o cardinals, modals, adverbs, symbols,...
• Filter out stop words (meaning-poor words)
o Large corpus analysis shows function words the most frequent
o Stop words prone to becoming key words owing to strong relationship between word
frequency and word 'key-ness'
Essay
V1 top
10 key
words
![Page 7: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/7.jpg)
Key word graphs
All remaining meaning-rich words of the essay become
nodes in a graph (network)
Adjacent words represented by links (edges) between
nodes
An algorithm traverses the graph to derive the key
words
• A key word is one that co-occurs (within a window of
N words) with lots of words that co-occur with lots of
words that co-occur...
No external trained model or reference source involved
Different from word frequency and from collocation
count
• Algorithm captures a word's connectedness to the
entire text
![Page 8: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/8.jpg)
Key sentence extraction Roughly similar process to key words: Same 33 x 1500-word essays
Some NLP text pre-processing
Each sentence becomes a node in a graph
Every sentence is compared to every other sentence to derive a
similarity score for each pair Currently using Cosine Similarity (vectors based on co-occurrence)
Links (edges) connect pairs of nodes that have a similarity score > 0 Similarity score = Strength of connection (edge weight)
TextRank algorithm (Mihalcea & Tarau 2004) calculates 'global
importance' score for each sentence Uses graph structure (what links to what) and edge weights
Like PageRank but with edge weights in the mix
Result is list of all essay's sentences returned in order of global
importance
![Page 9: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/9.jpg)
ESSAY
GRAPHICS
ANALYSIS
![Page 10: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/10.jpg)
OpenEssayist: What it tells you
The system’s focus is to present summaries of your own work in different ways, to encourage you to reflect constructively on what you have written.
In other words Open Essayist tells you from its analysis what are the most important or key points in your essay. You can then think about whether that was what you intended to emphasis in your essay. If not then you can make the appropriate changes.
A very important aspect of the OpenEssayist system is that it will not tell you what to write, or how to rewrite sections of your essay, or even what is correct or incorrect in your essay.
![Page 11: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/11.jpg)
![Page 12: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/12.jpg)
![Page 13: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/13.jpg)
![Page 14: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/14.jpg)
![Page 15: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/15.jpg)
Sample key phrases dispersion plot
![Page 16: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/16.jpg)
Most Used Features
Total access
Key words 115
Keyword cloud 103
Key sentences 99
Keyword dispersion plot 96
![Page 17: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/17.jpg)
Grades and use of OpenEssayist with H817
• Used by MAODE students
• Positive correlations
1. Grades for Essay 1 and number of drafts (r=+0.41)
2. Number of site visits and number of drafts (r=+0.65)
3. Number of visits and grade for Essay 2 was significant one
tailed test (r=+0.5)
4. Mean grade for overall module for students in cohort who used
OpenEssayist (64.2) and students in previous cohort (53.7)
(p=0.4)
![Page 18: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/18.jpg)
Student comments
The summary is an extension of
the key words and phrases and
will make me think about
whether this is really what I
wanted to say in the essay.
I can see the benefit because it is
talking
about the structure. It will help you
understand where you need to work
in, the different sections, what you
are missing maybe you need to fill
in a bit more or not.
![Page 19: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/19.jpg)
New feature Rainbow Diagrams
Pretend essay: 10 identical paragraphs
![Page 20: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/20.jpg)
Pretend essay: 50 identical sentences
![Page 21: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/21.jpg)
Stanford University Boothe Prize essay
![Page 22: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/22.jpg)
OU essay awarded high grade
![Page 23: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/23.jpg)
OU essay awarded low grade
![Page 24: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/24.jpg)
Good vs. bad?
Good:
densely connected
Red nodes (conclusion) central
Close links between violet (intro) and red notes
Bad:
Not densely connected
Red nodes (conclusion) not central
Few links between violet (intro) and red nodes
![Page 25: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/25.jpg)
Rainbow diagrams related to mark awarded
Multivariate analysis of variance on marks awarded to 45
students
Submitted two essays
Rainbow diagrams produced from these essays and rated as
high, medium or low attainment
Covariate showed a significant relationship with the marks
F(1, 43) = 5.92, p = .01 using a directional test
Essays rated as high would be expected to receive 8.56
percentage points more than essays rated as medium
17.2 percentage points higher than essays rated from rainbow
diagrams as low
![Page 26: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/26.jpg)
How about feedback first?
Hints before writing?
R.C.T.
2 essays
F(1,41) = 3.23 p = 0.04 for
hints
http://www.eden-
online.org/publications/best
-research-papers.html
![Page 27: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/27.jpg)
Creating teaching and learning dialogues:
towards guided learning supported by
technology
Learning to judge
Providing reassurance
Providing a variety of
signposted routes to
achieve learning goals
![Page 28: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/28.jpg)
ReferencesWhitelock, D., Twiner, A., Richardson, J.T.E., Field, D. & Pulman, S. (2015). OpenEssayist: A supply and demand
learning analytics tool for drafting academic essays. The 5th International Learning Analytics and Knowledge (LAK)
Conference, Poughkeepsie, New York, USA. 16-20 March 2015. ISBN 978-1-4503-3417-4
Whitelock, D., Twiner, A., Richardson, J.T.E., Field, D. & Pulman, S. (2014). OpenEssayist: Real-life testing of an
automated feedback system for draft essay writing. Paper presented at the #design4learning conference, 27 November
2014 at The Open University http://design4learning.org.uk/
Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. & Whitelock, D. (2014) Using student
experience as a model for designing an automatic feedback system for short essays. International Journal of e-
Assessment, 4(1), article no. 68
Field, D, Richardson, J.T.E., Pulman, S., Van Labeke, N. & Whitelock, D. (2014) An exploration of the features of
graded student essays using domain- independent natural language techniques. International Journal of e-Assessment,
4(1), article no. 69
Whitelock, D., Twiner, A.; Richardson, J.; Field, D.; Pulman, S. (2014). Feedback on Academic Essay Writing Through
Pre-Emptive Hints – Moving Towards ‘Advice for Action’ . Challenges for Research into Open & Distance Learning:
Doing Things Better – Doing Better Things Proceedings of the European Distance and E-Learning Network 2014
Research Workshop Oxford, 27-28 October, 2014. ISBN 978-615-5511-00-4
![Page 29: D Whitelock LAK presentation open_essayistfv](https://reader034.vdocument.in/reader034/viewer/2022042817/55a61a811a28abc6098b46d4/html5/thumbnails/29.jpg)
in service of The Open University