PeerPigeon A Web Application to Support Generalized Peer Review

David Millard

Learning Societies LabUniversity of Southampton

Overview

What is Peer Review? Models of Peer Review A Generalised Model of Peer Review

Peer Review Cycles Peer Review Transforms

Dealing with Complexity PeerPigeon Conclusions and Future Work

Introduction

What is Peer Review? Also called Peer Assessment An important tool for educators

Reflective Learning Key Social and Professional Activity Helps to Manage Load!

Many different Types Almost infinite applications… …and complexity!

Models of Peer Review

Simple

Round Robin

Group Activity

Models of Peer Review

Group Review

Committee Review

Multiplicity (!)

Peer Review Cycles

Can View Any PR Process as combination of Generate, Submit, Distribute Cycles

The complexity of Peer Review is then accounted for in three ways:

The cycle can be started in any one of its three states.

The cycles can be interleaved, and can occur in parallel as well as in sequence.

Each stage within the process may involve 1...n participants (authors/ tutors/ reviewers),producing 1...m resources (artifacts/ reviews/ marks)

Peer Review Cycles

For example the Simple Case…

…can be described as two iterations of the cycle. The author Generates an Artifact which is then

Submitted and Distributed to the reviewer. The reviewer then Generates a review which they then

Submit and is Distributed back to the artifact’s author

Peer Review Transforms

Peer Review Cycles are good at capturing the stages but do not describe the most complex part How artifacts are distributed, and what is their

relationship is to the next stage of generated artifacts.

In PeerPigeon we do this through Peer Review Transforms

For example in the Simple Case this transform is straightforward: Every authored artifact is transformed by one reviewer

into a review.

Peer Review Transforms

For example: These are the transforms for a typical

academic conference…

Complexity

This is not easily generalisable! What does appropriate mean?

In fact these Peer Review Patterns are irreducibly complex

In PeerPigeon we use a Domain Specific Language (DSL) to get around this

PeerPigeon DSL (Ruby)

review doroles :students, :tutors

cycle :paper do |c|c.description 'Write Paper'c.deadline '0 days,'2 days,'3 daysc.formats :file

c.distribution :students do |student, students, group|c.transform nil, student, "paper_#{student}"

endend

cycle :review do |c|c.description 'Write Review'c.deadline '2 days,'4 days,'5 daysc.formats :textarea, :file

c.distribution :students do |student, students, group|c.transform "paper_#{students.wrap(student, +1)}", student,

"review_#{students.wrap(student,+1)}"end

end

…

c.transform :paper_2, 1, :review_2c.transform :paper_3, 2, :review_3c.transform :paper_1, 3, :review_1

c.transform nil, 1, :paper_1c.transform nil, 2, :paper_2c.transform nil, 3, :paper_3

We don’t ask users to write these scripts!

Instead PeerPigeon is designed to be preinstalled with a number of reusable scripts

More scripts can be added by administrators later on

PeerPigeon: A Web Application

Typical Flow of Activity

1. A Teacher creates a Peer Review Activity based on an example Peer Review Pattern (such as Round Robin)

2. Each Role in the Activity (author, reviewer, committee, etc.) has its own URL which can be advertised

3. The Teacher specifies an activity start date from which cycle start and end times are automatically generated

4. Participants enroll for specific roles using the advertised URLs

5. At the allotted time the Peer Review Starts, for each cycle:1. Participants with roles in this cycle are given an input artifact2. Participants are prompted to take action (e.g. “Please review this

paper and submit by April 29th”)3. Participants upload their output artifact to PeerPigeon (these

become input artifacts in the next cycle)

PeerPigeon: Building a Review 1

PeerPigeon: Building a Review 2

Conclusions and Future Work

Peer Review is an important part of e-learning

There are many different types

Not easily generalisable, but we have some helpful concepts: Peer Review Cycles Peer Review Transforms Peer Review Patterns

PeerPigeon is a Web Application for Peer Review Uses Domain Specific Language to support generalized Peer Review

Patterns

PeerPigeon is Open Source About to run as beta platform in University of Southampton Not a finished product (beta-level) Eager to find other people who have an interest in taking it forward