David MillardKaren FillHugh DavisLester GilbertGary WillsLearning Societies Lab, University of Southampton, UK

Towards a Canonical View

of Peer Assessment

PeerPigeon

PeerPigeon is an out-of-the-box system that runs any type of peer review process by supporting a generalised case of Peer Review.

Overview:

• Why Peer Assessment / Review?• How do we provide for the generalised case?• Introduction to PeerPigeon

Peer Assessment• Peer Assessment has many advantages:

– Giving a sense of ownership of the assessment process, improving motivation

– Encouraging students to take responsibility for their own learning– Treating assessment as part of learning, so that mistakes are

opportunities rather than failures– Practicing the transferable skills needed for life-long learning,

especially evaluation skills – Using external evaluation to provide a model for internal self-

assessment of own learning – Encouraging deep rather than surface learning.

Bostock S., “Student peer assessment”, Higher Education Academy Article, 16 Mar 2001

Case Studies

Simple - The simplest form of peer review is where authors and reviewers are paired together

Round Robin - Where participants are grouped, and each participant reviews the work of every other other participant in their group.

Group Activity - Where a group of authors work together to produce an artefact, and then that artefact is reviewed by a third party.

Group Review - Where a group of authors work together to produce an artefact, and then individually review the efforts of their group.

authors reviewers

participants

authors reviewer

group workers

Case Studies

Committee Review - Where a group of reviewers act together and look at several different artefacts in order to produce one review. In the research community we are familiar with this as the conference committee stage of peer review

Multiplicity - Where multiple authors create multiple artefacts which are then independently reviewed by multiple reviewers. For example, where students give a paper and presentation and are assessed by their classmates on both

paper reviewers

committee

authors

reviewers

Common Review Cycle

All these cases can be thought of as being built of common review cycles

1. The cycle can be started in any one of its three states. For example, to begin an activity the student may be asked to Generate an artefact, to Submit an existing artefact, or the tutor may provide it, in which case the first task is to Distribute it.

2. The cycles can be interleaved, and occurring in parallel as well as in sequence.

3. Each stage within the process may involve 1...n participants (authors/tutors/reviewers), producing 1...m resources (artefacts/reviews/marks).

Generate

Submit

Distribute

Example of Multiplicity:

• n students, m tutors• each student delivers a

presentation and answers questions (i.e. two artefacts)

• students and tutors review/mark the presentations

• only tutors review/mark the answers

Cycle CreatorsAuthors/Reviewers

ResourcesArtefacts/Reviews

ReceiversReviewers/Authors

1 1 1 n+1

2 n m 2

3 1 m n+1

4 n+1 n+1 1

5 1 1 1

6 1 1 1

Example of Multiplicity:

• n students, m tutors• each student delivers a

presentation and answers questions (i.e. two artefacts)

• students and tutors review/mark the presentations

• only tutors review/mark the answers

Use Case for PeerPigeon

• What is PeerPigeon?

Create

Adjust

Run

The complexity of the system is encapsulated in a Peer Review Plan with three elements:

• A Peer Review Pattern (an ordered description of the cycles of peer review and the roles of the participants in each cycle).

• A number of actual Participants (possibly arranged into Groups) that populate the roles in the plan.

• A Schedule of upcoming dates and times, that ties the pattern to a real timescale.

Peer Review Patterns

• We need to formalise the cycles (make them machine readable)– Generation occurs outside of the system– Submission is simple

• We expect X to submit Y– Distribution is complex

• Where does Y go next!

• We represent Distribution using transforms: – a participant takes an existing artefact and produces a new

artefact based on it – E.g. a reviewer takes a paper and transforms it into a review.

Peer Review Patterns

• Major challenge with specifying the transforms– Irreducible complexity– An infinite number of ways to choose the allocation algorithm

• We have used a Domain Specific Language (DSL)– a programming language for a specific set of tasks. – a clean syntax that can describe the specific steps/information– an agile alternative to XML

• (no need for schema/dtds, a data file, or a parser)

• Ruby is an ideal language for DSLs– Scripting language– Dynamic programming features– Lots of support / active community

PeerPigeon DSL Example

• Another simple review pattern:– in a group each person produces a paper and reviews the paper of

someone else• A simple algorithm:

– pass each paper along to the next person

• In our Ruby DSL:

cycle :review do |c| c.description 'Write Review' c.deadline '2 days from now','4 days from now' c.transform :paper_1, 2, :review_1 c.transform :paper_2, 3, :review_2c.transform :paper_3, 1, :review_3

end

c.distribution :reviewers, lambda { |group,person| c.transform

group.wrap('paper', person, -1),

person,

group.wrap('review', person, +1)

}

PeerPigeon Prototype

Conclusions and Future Work• We have shown that Peer Review can be generalised

– The building blocks are Peer Review Cycles– Generate -> Submit -> Distribute

• The Distribution can be encoded as a sequence of transforms– Paper 1 is transformed by Person 2 into Review 1

• The Distribution algorithm is irreducibly complex– But can be handled by a Domain Specific Language (DSL)

• We are currently prototyping the PeerPigeon system– Implements the generalised use case– Written in Ruby on Rails (Ruby DSL)

• An alpha version is available from our website– Release version (late beta) is planned for end of August– www.peerpigeon.ecs.soton.ac.uk