project e-scape

Richard Kimbell TERU: Goldsmiths University of LondonKarim Derrick TAG developments

evidence-based web-portfolios - to empower learning

- for reliable assessment

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.QuickTime™ and a

decompressorare needed to see this picture.

in association with …. QuickTime™ and a decompressor

are needed to see this picture.

to support and capture live performance • performance assessment• on task• in real time

science

geography

design

e-scape system enables us

to assess performance• for GCSE / Diploma • linked to Awarding Body systems• with very high reliability (0.95)

to display the performance in web-portfolios • easily distributed • URL / log in / password

capturing performance evidence

sketching(and collaborating)

photos of the state of play

audio reflections

video presentation

data and notes

mind-maps

continuum of control

• tight control of tasks• strict timings• controlled response modes• standardised portfolio

formal examinations

extended coursework projects

• learner autonomy of tasks• flexible timings• learner choice of tools• customised portfolio

client/user can choose the degree of

control

the portfolio as a ‘big-picture’ of the activity

… that can be dipped into for the detail

the assessment challenge

QuickTime™ and a decompressor

are needed to see this picture.

teachers / examiners review portfolios with ‘RED PEN’

criteria displayed with the work

teachers notes on

student portfolio

different markers have separate ‘layers’

teacher speaks directly to the student

and evidenceIs noted

but there is another (quite different) possibility

L.Thurstone (1927) • law of comparative judgement• judges’ personal standards cancel out

A.Pollitt (2004) • comparative pairs methodology• for reliability studies between exam boards

e-scape (2007 & 2009)• the 1st and 2nd complete data sets • the ‘pairs engine’ • for front line assessment

eye testing

teachers and assessment

teachers are good at • relative judgements• ranking

teachers are not good at • absolute judgements• scoring

because ‘criteria’ become personalised

because their judging standard cancels out

• teachers discuss criteria

• teachers use them as ‘signposts’ to look into the portfolios

• teachers identify strengths and weaknesses of portfolios

…but then they make a relative holistic judgement

portfolio A

portfolio B

the ‘pairs engine’ manages the assessment process

• judges in Ireland, Israel and across the UK

• shared viewing of exemplar work

• on-line marker training

the whole process is run on-line in the ‘pairs engine’

the pairs enginedynamically generates the rank

what emerges is the collective professional consensus of an expert group of judges

2009 sample

350 pupils

multiple judges (28)

multiple comparisons (17+)reliability coefficient = 0.95

awarding gradesis a separate process

}

“ Verbal and Mathematical Reasoning tests .. were developed for 11+ and similar purposes from about 1926 to 1976. These were designed to achieve very high reliability, by minimising inter-marker variability and maximising the homogeneity of the items that made them up. KR20 statistics for those were always between 0.94 and 0.96.With e-scape, a level of internal consistency comparable to those reasoning tests has been achieved without reducing the test to a series of objective items. “

Alastair Pollitt: Dec 2008

identifying problems

the engine identifies portfolios where judges disagree

and individual judge profiles show their consensuality or

‘misfit’

• the first 10 judgements do most of the work• the second 10 fix the scale

• thereafter the gain in accuracy per extra judgement is small

reducing standard error

judging times for 10 judges

00:00

02:24

04:48

07:12

09:36

12:00

14:24

16:48

19:12

21:36

00:00

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13thgroups of 10 pairs

min

s / s

ecs

judge Ajudge Bjudge Cjudge Djudge Ejudge Fjudge Gjudge Hjudge Jjudge K

mean time for judging

00:00

01:12

02:24

03:36

04:48

06:00

07:12

08:24

09:36

10:48

1 2 3 4 5 6 7 8 9 10 11 12 13groups of 10 pairs

min

s / s

ecs

how long do judgements take?

median: • for all 28 judges• 130 judgements each

= 4 mins 6 secs

= total of 8 hrs 45 mins

Judges’ reflections …..The variety of media used within the portfolio seems to give everyone a fair chance to display their design thinking regardless of their ability. It was fairly easy to see that there was a range of performance evident in the portfolios.

Portfolios displayed in this way have a huge advantage in that the big picture can be seen immediately. It’s very easy to get a feel for the project ... The ability to dip in and out of the different sections enabled me to reinforce my holistic mental picture of the project.

It’s brilliant not having to carry around boxes full of portfolios/folders and practical work.

Surprisingly pupils’ attitudes were also often evident.

much, much faster .. and less scary

the learner voice emerges

• would encourage learners’ understanding of capability• would encourage learning through critique • could support e.g. PLTs evidence

current implementations

Edexcel Examination Board: UK14-19 Diploma in ‘Creative & Media’

West Australia Curriculum Councilschool leaving certificate examinations(engineering / applied IT / phy ed / LOTE )

Bath Spa University & AstraZeneca Science Teaching Trustscience/technology for age 11-13

Scotland SQA / LTSformative assessment in transition years (ege 11-13)

UK government agencywriting assessments at age 11

new applications for e-scape & ‘pairs’

Limerick University: Irelandengineering design project (mobiles) + assessment

E-scape references and citations by authors outside the e-scape research team Benson C 2008 International Handbook of Research and Development in Technology Education. sensepublishers.com

https://www.sensepublishers.com/files/9789087908799PR.pdf#page=42 http://scholar.google.co.uk/scholar?q=related:I99KZ4RoaxEJ:scholar.google.com/&hl=en&as_sdt=2000 http://74.125.155.132/scholar?q=cache:I99KZ4RoaxEJ:scholar.google.com/+TERU+e-scape&hl=en&as_sdt=2000

Bramley, T Black,B 2008 Third International Rasch Conference. Cambridgeassessment.org.uk

http://www.cambridgeassessment.org.uk/ca/digitalAssets/171143_TB_BB_rank_order_Perth08.pdf

Clewes, P. 2009 ŅThe Great E-scapeÓ Design & Techno logy Association journal ŌDATA PracticeÕ (No01.2009 Jan 2009) Also on TERU web pages Š

http://www.gold.ac.uk/media/The Great E-SCAPE.pdf Davies D 2009 ŅDigital Portfolio Assessment of Secondary Students' Scientific Enquiry Skills: the e-scape projectÓ BERA 2009 Conference Futurelab publications: e-scape projects are cited in all the following reports from Futurelab

Assessment and Social Justice Review: Jan 2009 Š Gardner J, Holmes B, Leitch R http://www. futurelab.org.uk/resources/publications-reports-articles/literature-reviews/Literature-Review1186 Personalisation and digital technologies: Nov 2005 Š Green H, Facer K, Rudd T, Dillon P, Humphreys P http://www. futurelab.org.uk/resources/publications-reports-articles/opening-education-reports/Opening-Education-Report201 Developing and accrediting personal skills and competencies: Mar 2007 Facer K, Pykett J http://www. futurelab.org.uk/resources/publications-reports-articles/other-research-reports/Other-Research-Report1094 e-assessment: An Update on Research, Policy & Practise: Sept 2007 Š Ripley M http://www. futurelab.org.uk/resources/publications-reports-articles/literature-reviews/Literature-Review204 Creativity Techno logy and Learning Update: Sept 2007 Š Loveless A http://www. futurelab.org.uk/resources/publications-reports-articles/literature-reviews/Literature-Review382

Geographical Association: e-scape in geography: http://www.geography.org.uk/projects/e-scape/#top e-scape geography school trials and reports http://www.geography.org.uk/projects/e-scape/trialsandreport/#top e-scape assessments in geogra phy using comparative pairs http://www.geography.org.uk/projects/e-scape/comparativepairs/#top Thinking geographically http://www.geography.org.uk/projects/e-scape/thinkinggeographically/#top e-assessment and e-scape in geography http://www.geography.org.uk/projects/e-scape/e-assessment/#top e-scape: Controlled Assessment http://www.geography.org.uk/projects/e-scape/controlledassessment/#top digital fieldwork in geograph y http://www.geography.org.uk/projects/e-scape/digitalfieldwork/#top

Times Education Supplement (TES online)

Feb 2007 - New reportsÕ green light for PDAÕs in classroom: - John M http://213.232.94.135/merlinjohnonline/news.php?extend.118 Nov 2005 - Digital Challenge Š Ripley M http://www. tes.co.uk/article.aspx?storycode=2153804 Oct 2009 Š Innovative design is the talk of the classroom Š Seith E http://www. tes.co.uk/article.aspx?storycode=6026498 Nov 2005 Š The Great E-scape Š Kenny J http://www. tes.co.uk/article.aspx?storycode=2153805 May 2005 Š The future of education is in hand Š Brown-Martin G http://www. tes.co.uk/article.aspx?storycode=2097459

Malmborg L Peterson, B Pettersson M 2007 ŅAugmenting pen and paper to support creative collaboration in design educationÓ Perth

http://scholar.google.co.uk/scholar?q=related:8bOBTs_v2VMJ:scholar.google.com/&hl=en&as_sdt=2000 or http://dspace.mah.se/handle/2043/5359

Martin, F & Beibrach, T 2009 ŅThe e-scape route to assessmentÓ Teaching Geography autumn 2009 Norman, E Mitchell, A Zanker, N Paterson, A 2009 Developing the research infrastructure for technology education in England (and beyond) PATT-15, the 20th Anniversary Conference

http://www. tnw.tudelft.nl/live/pagina.jsp?id=60763a45-a56b-4742-9ba1-5273b884a1b6&lang=en&binary=/doc/Final.pdf

Qualifications and Curriculum Authority (QCA) 2006 ŅNew ways of assessing creativity in design and technologyÓ QCA/05/1513 Nov 2006 Ripley, M 2008 Unlocking assessment: understanding for reflection and application., Google scholar

http://scholar.google.co.uk/scholar?q=related:cRXA9xUqdVUJ:scholar.google.com/&hl=en&as_sdt=2000

Thomas, K 2008. ŅRewarding risk: how e-scape changes learningÓ July 2008 Futurelab

http://www. futurelab.org.uk/resources/publications-reports-articles/web-articles/Web-Article1063

e-scape references and citations

by authors outside the e-scape team

project e-scapee-solutions for creative assessment in portfolio environments

Contact me at: r.kimbell@gold.ac.uk

www.gold.ac.uk/teru

Goldsmiths University of London