The Role of Peer Review in Supporting the Sustainability of Technology-Enhanced

Learning Environments

Pantelis M. Papadopoulos, United Nations UniversityAntonio Cerone, United Nations University

2 / 24

Overview

Sustainability in Educational Technology

Research domain and argument

The peer review process

Collaboration script

Three studies on peer reviewThe role of coercionFree-selectionIndirect feedback

3 / 24

Sustainability in Educational Technology

Multiple definitions of sustainability in ETOrganizational, financial, technical, institutional, etc

We focus on pedagogy, using sustainability to refer toLife cycle of a technology-enhanced learning environment (TELE)Ability of a TELE to sustain an active group of students

Even with institutional support and financial viability, a TELE may fail, if it does not capture the interest of the students

Link between sustainability and student engagement

4 / 24

Research domain and argument

DomainPeer review method in computer supported collaborating learning (CSLC)

Student engagement can be affected byIndividual factors: learning profile, intrinsic motives, etcLearning activity characteristics: peer interaction, scaffolding method, etc

Our argumentPeer review method can (a) enhance student engagement, (b) support the creation of sustainable a learning community, and consequently support the sustainability of the TELE used by the community

Literature already reports multiple benefits from peer review in many levels

5 / 24

The peer review method

Four major phasesProducing initial work

Students work individually/collaboratively to produce the initial draft

Assigning reviewersThe teacher/system assigns peer work to reviewers

Review submissionReview comments are sent back to the authors

Revision and final version The author revises the initial draft according to reviews and creates the final version

6 / 24

Collaboration script

Didactical script guiding students into meaningful learning interactionsEven in scripted collaboration, there is a distance between the prescribed task and actual implemented activity

External script: the activity as prescribed by the teacher

Internal script: the mental representation of the script that the group builds from teacher's prescription

Actual script: the actual task and interactions that learners engage into

First Study

Peer review and script coercion

8 / 24

First Study: Method

Two groups of juniors majoring in Informatics studied in a TELE20 students in Low Coercion condition22 students in High Coercion conditionWe randomly assigned students into same-sex dyads

Collaboration scriptAll students worked individually to produce the initial draftStudents in dyads reviewed each other's work following a set of guidelinesStudents had to discuss, reach a consensus, and form a final common answer

Low Coercion group: submitting reviews in the TELE was optionalHigh Coercion group: submitting reviews in the TELE was mandatory

9 / 24

First Study: Test results

Five phases: pre-test, individual study, collaboration, post-test, interviewPre-test: t-test results showed that the two groups were comparable regarding prior domain knowledge (p>0.05)Post-test: ANCOVA results showed that the High Coercion group significantly outperformed the Low Coercion group (p<0.05)

High Coercion Low Coercion

M SD n M SD NPre-test 2.24 (0.71) 22 2.13 (0.59) 20Post-test 7.42 (1.30) 22 6.36 (0.83) 20

10 / 24

First Study: Collaboration patterns

Analysis of collaboration patterns based on: (a) student statements in the interviews, (b) comparison of individual and collaborative answers in the TELE, and (c) the TELE log files

Collaboration patterns: Ideal, moderate, weak

Only 1 out of 10 Low Coercion dyads submitted written reviews in the TELEThe others said that they shared review comments during discussionEven if review comments were shared, they would be unstructured and scattered in the discussion

Low Coercion group: 2 dyads in the "ideal", 5 dyads in the "moderate", and 3 in the "weak"High Coercion group: 5 dyads in the "ideal", and 6 dyads in the "moderate"

11 / 24

First Study: Conclusions

Script coercion can be used by the teacher as a mean to engage students into the activity

When left to decide, almost all the students in the Low Coercion group opted-out of the review process as described in the guidelines

Students in the High Coercion group demonstrated better collaboration patterns and post-test performance

Second Study

The free-selection protocol

13 / 24

Second Study: Method

Two groups of sophomores majoring in Informatics and Telecommunications Engineering studied in a TELE

20 students in Assigned-Pairs (pre-defined dyads by the teacher)22 students in Free-Selection

Collaboration scriptAll students worked individually to produce the initial draftStudents reviewed each other's drafts through a double-blinded processStudents had to review the initial submissions taking into account review comments

14 / 24

Second Study: Review conditions

Assigned-PairLearners in an author-reviewer dyad are assigned exclusively to each otherWe randomly assigned students into dyads

Free-SelectionLearners have access to all the submission made by the rest of the class

More points of view accessible

They can read and review as many submission they like

min. at least one review shift to dyad format

15 / 24

Second Study: Test results

Five phases: pre-test, study, review & revise, post-test, interviewPre-test: t-test results showed that the two groups were comparable regarding prior domain knowledge (p>0.05)Post-test: ANCOVA results showed that the Assigned-Pair group significantly outperformed the Free-Selection group (p<0.05)

Assigned-Pair Free-Selection

M SD n M SD NPre-test 2.69 (1.07) 20 2.59 (0.83) 17Post-test 7.71 (0.95) 20 8.43 (0.81) 17

16 / 24

Second Study: Student attitudes

Students in the Free-Selection group studied on average 8 and reviewed 2 out of the 16 drafts submitted by their peers

Two students did not receive reviews and we asked selected students to provide the missing reviews

Interviews: Two main trends in selecting drafts by FS studentsFind good answers to be able to give nice commentsFind weak answers to be able to provide more meaningful reviews

Interviews: FS student submit more reviews than asked because:It was a good exercise for them to clarify their thoughtsAfter a while it was easy to do and it would increase the possibility of everyone receiving at least one review ( community culture)

17 / 24

Second Study: Conclusions

FS students…Took advantage of the multiple points of view offered in their peers' drafts

Engaged deeper in the activity voluntarily far exceeding the minimum effort requirements

Explicitly reported a community culture in submitting more reviews

Had a more positive opinion about the activity than the AP students, despite the fact that the latter had to work less

Third Study

What if there is only indirect feedback?

19 / 24

Third Study: Method

Based on the findings of the Second StudyIndirect feedback: Students get new insights by reading others' draftsCan the lack of review comments be address by indirect feedback and self-review?

Two groups of sophomores majoring in Informatics and Telecommunications Engineering studied in a TELE

Free-selection protocol20 students in Self-Review (their drafts were excluded from review and they had to fill in a self-review form before revising their drafts)18 students in Peer-Review (they all received at least one review)

Collaboration script: same as in Second Study

20 / 24

Third Study: Test results

Five phases: pre-test, study, review & revise, post-test, interviewThe two groups were comparable both in pre-test and post-test (p>0.05)

Self Review Peer Review

M SD n M SD NPre-test 2. 20 (1.09) 20 2.04 (1.06) 18Post-test 8.19 (1.37) 20 8.13 (1.40) 18

21 / 24

Third Study: Student attitudes

Same attitudes as in Second Study

Students demonstrated strategies that far exceeded the minimum effort requirement

Interviews: Same two trends in selecting draftsFind good answers to be able to give nice commentsFind weak answers to be able to provide more meaningful reviews

22 / 24

Third Study: Conclusions

The study tried to provide evidence that reading other's draft can be equally beneficial to getting peer feedback

We supported non-reviewed students with a self-review process and results showed that they were comparable to typical Free-Selection students with peer reviews

Since receiving peer review is not the only way to receive feedback, it is easier for a student to participate in a learning community that applies the Free-Selection protocol

Even if the student does not get comments from peers, the previously submitted answers can provide valuable feedbackSelf-review can help the student perform comparative analysis and identify weak points in his own work

23 / 24

General Remarks

Peer review process can be an effective tool for structuring the interaction between students

When used right, the degree of coercion in scripted collaboration can result to deeper engagement and consequently to better learning outcomes

Free-selection protocol allows for more freedom, while it maintains a minimum level of engagement

FS protocol has resulted in increased student effort in both studies it was used

The community culture of working more to benefit others also appeared in the two studies with FS protocol

The indirect feedback in FS makes the protocol even appropriate for learning communities

24 / 24

Thank you!

http://to.ly/fyqO