peer feedback on writing: a sotl work in progress
TRANSCRIPT
![Page 1: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/1.jpg)
Tracking a Dose-Response Curve in Peer Feedback on
WritingA Work in Progress
Christina HendricksCo-Investigator: Jeremy Biesanz
University of British Columbia-Vancouver
SoTL Symposium, November 2015
Slides available here: http://is.gd/PFBwriting2015Slides licensed CC-BY 4.0
![Page 2: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/2.jpg)
Literature on peer feedbackReceiving peer feedback improves writing (Paulus, 1999; Cho & Schunn, 2007; Cho & MacArthur, 2010; Crossman & Kite, 2012)
Giving peer feedback improves writing(Cho & Cho, 2011; Li, Liu & Steckelberg, 2010)
![Page 3: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/3.jpg)
GAPS: Most studies look at revisions to a single essay, not changes across different essays
Draft 1
Draft 2
Draft 3
Essay 1 Essay 2 Essay 3 Essay 4 Essay …n
PFB
PFB
PFB
PFB PFB PFB
Few studies look at “dose-response curve”
![Page 4: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/4.jpg)
Pilot study research questions1. How do students use peer comments given and
received for improving different essays rather than drafts of the same essay?
2. Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?
3. Does the quality of peer comments improve over time?
![Page 5: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/5.jpg)
• Interdisciplinary, full year course for first-years
• 18 credits (6 each in 1st year English, History, Philosophy)
• Students write 10-12 essays (1500-2000 words)
• Peer feedback tutorials every week (4 stdnts)
http://artsone.arts.ubc.ca
Toni Morrison, Wikimedia Commons, licensed CC BY-SA 2.0
Osamu Tezuka, public domain on Wikimedia Commons
Jane Austen, public domain on Wikimedia Commons
Friedrich Nietzsche, public domain, Wikimedia Commons
![Page 6: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/6.jpg)
Data for pilot study 2013-2014
• 10 essays by 13 participants (130 essays)
• Comments by students in tutorial group (4 in group) on all essays (n=1219)
• Comments by instructor on all essays (n=3331)
• All essays and comments coded according to a common rubric
![Page 7: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/7.jpg)
Coding RubricCategories(plus subcategories, for 11 options)
• Strength of argument• Organization• Insight• Style & Mechanics
Numerical value
1: Significant problem2: Moderate problem3: Positive comment/praise
E.g., STREV 2: could use more textual evidence to support your claims
![Page 8: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/8.jpg)
Progress so farAll student and instructor comments coded60 of 130 essays coded -- (10 essays each by 6 of the 13 participants)
![Page 9: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/9.jpg)
Inter-coder reliability
Fleiss’ Kappa Intra-class correlation
Student comments (n=141)
All categories: 0.61 (moderate)Most used categories: 0.8 (excellent)
0.96 (excellent)
Essays (n=60) 0.68 (adequate)
3 coders:• Daniel Munro & Kosta Prodanovich
(undergrads, former Arts One)• Jessica Stewart (author, editor)
![Page 10: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/10.jpg)
LOOKING AT TRENDS IN COMMENTS OVER TIME
![Page 11: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/11.jpg)
INSTRUCTORComments
-.28**
Strength
Style
Organiz.
Insight-.04*
Number of 2 comments in each categ.
![Page 12: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/12.jpg)
STUDENT comments
Strength
Style
Organiz.
Insight
-.16**
Number of 2 comments in each categ.
![Page 13: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/13.jpg)
INSTRUCTORComments .31***
Strength
Style
Organiz.
Insight
.08**
.19**
.11**
Number of 3 comments in each category
![Page 14: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/14.jpg)
STUDENTComments
Strength
Style
Organiz.
Insight
Number of 3 comments in each categ.
![Page 15: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/15.jpg)
Student/instructor agreement on comments
• Average numerical ratings for comments across all categories agree strongly between student and instructor (.48****)
• However, this agreement
goes down across essays (-.04**)o This is because student ratings increase
over time at only half the rate that instructor ratings do
*p < .05, **p< .01, ***p< .001, ****p < .0001
![Page 16: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/16.jpg)
HOW DOES ESSAY QUALITY CHANGE OVER TIME?From this slide onwards, we are looking only at 60 essays coded so far, out of the set of 130 (essays from 6 of the 13 participants)
![Page 17: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/17.jpg)
Essay quality
improves linearly
in 60 essays
Mean essay quality rating 4.12 out of 7, SD = .62
![Page 18: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/18.jpg)
MORE COMPLEX ANALYSES
![Page 19: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/19.jpg)
Cross-lagged panel design with auto-regressive structure
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
![Page 20: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/20.jpg)
Path A: Student comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant effects:• Ratings of 2 in Insight (-.53*)• Ratings of 3 in Strength (.25*)
![Page 21: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/21.jpg)
Path C: instructor commentsEssay Quality
Time 1Essay Quality
Time 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant effects • Ratings of “2” in Strength (-.31**)• Ratings of “3” in Strength (.51***) and
Style/Mechanics (.34**)
![Page 22: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/22.jpg)
Path C: student commentsEssay Quality
Time 1Essay Quality
Time 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant effects• Ratings of “2” in Style (.28*) and Insight
(.15*)• Ratings of “3” in Strength (.30*)
![Page 23: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/23.jpg)
Path D: Student comments
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
Significant effects:• Ratings of 2 in Organization (.16*)• Ratings of 3 in Style (.16*)
![Page 24: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/24.jpg)
Research question 2How do students use peer comments given and received for improving different essays rather than drafts of the same essay?
o Not enough evidence yet to say much about path D
o Haven’t yet looked at differences in comments given vs. received
![Page 25: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/25.jpg)
Research question 2Are students more likely to use peer comments given and received for improving their writing after more than one or two peer feedback sessions? How many sessions are optimal?
o No evidence yet that there is any change over time in path D
![Page 26: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/26.jpg)
Research question 3Does the quality of peer comments improve over time?o No evidence yet that there is any change
over time in path A
Essay QualityTime 1
Essay QualityTime 2
CommentsTime 1
CommentsTime 2
B
A
C
D
E… N
… N
![Page 27: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/27.jpg)
Some conclusions so far• Pilot study: is this sort of study
feasible for larger sample?o Yes, but probably more so if instructors
code essay quality rather than coders
• Facilitating easy collection of student & instructor comments is difficult
![Page 28: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/28.jpg)
References• Cho, K., & MacArthur, C. (2010). Student revision with peer and
expert reviewing, Learning and Instruction. 20, 328-338.• Cho, Y. H., & Cho, K. (2011). Peer reviewers learn from giving
comments. Instructional Science, 39, 629-643. • Cho, K. & Schunn, C. D. (2007). Scaffolded writing and rewriting
in the discipline: A web-based reciprocal peer review system. Computers & Education, 48, 409–426
• Crossman, J. M., & Kite, S. L. (2012). Facilitating improved writing among students through directed peer review, Active Learning in Higher Education, 13, 219-229.
• Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536.
• Paulus, T. M. (1999). The effect of peer and teacher feedback on student writing. Journal of Second Language Writing, 8, 265-289.
![Page 29: Peer Feedback on Writing: A SoTL work in progress](https://reader037.vdocument.in/reader037/viewer/2022103010/587376931a28ab3c1a8b66cb/html5/thumbnails/29.jpg)
Thank you! Christina Hendricks
University of British Columbia-Vancouver
Website: http://blogs.ubc.ca/christinahendricksBlog: http://blogs.ubc.ca/chendricksTwitter: @clhendricksbc
Slides available: http://is.gd/PFBwriting2015
Slides licensed CC-BY 4.0