archive.alt.ac.ukarchive.alt.ac.uk/.../programmes/elearningcapital/portishe… · web...

PortisHEad – Project Final Report – 1.2 – 2008-12-22

PortisHEad Project Final Report

ProjectProject Acronym PortisHEad Project IDProject Title ePortfolios In Successful HE AdmissionsStart Date 1 Mar 2007 End Date 31 Oct 2008

Lead Institution University of Wolverhampton

Project Director Shane Sutherland

Project Manager & contact details

Alan PaullAPS Ltd, 58 Norton Wood, Forest Green, Nailsworth, STROUD GL6 0HG01453 835009 [email protected]

Partner Institutions Pebble Learning LtdUCASAPS Ltd

Project Web URL http://www.portisheadproject.org/Programme Name e-LearningProgramme Manager

Sarah Davies / Paul Bailey

DocumentDocument Title Project Final ReportAuthor(s) & project role

Alan Paull, Project Manager; Shane Sutherland, Project Executive

Date 22 December 2008

Filename Portishead_Project_Report_1.2.doc

Access √ Project and JISC internal General dissemination

Document HistoryVersion Date Comments

1.0 2008-11-13 Draft1.1 2008-12-01 Draft: first full release1.2 2008-12-22 Draft: Final comments from project partners included;

release to JISC

of 40


PortisHEad:ePortfolios In Successful Higher Education Admissions

Final Report

of 40


AcknowledgementsThe Project Board consisted of:

Paul Travill (Chair of the PortisHEad Consortium)Shane Sutherland (Executive & Technical Director)Jill Johnson (UCAS) / later Geoff Ramshaw (UCAS)Colin Dalziel (Senior Supplier, Pebble Learning Ltd)Alan Paull (Project Manager, APS Ltd)We are very grateful to the assistance of Professor Alison Halstead, who helped with the tender and was initially Co-Chair of the Consortium before her new appointment as Pro-Vice-Chancellor Learning and Teaching Innovation at Aston University.

The Project Team consisted of:Paul Alder (UCAS)Rachel Challen (Pebble Learning Ltd)Andrew Everson (Pebble Learning Ltd)Colin Dalziel (Pebble Learning Ltd)Keith Holmer (UCAS)Paul Kent (Kingsmead Technology College)Toni Lavender (Pebble Learning Ltd)Dave Morgan (UCAS)Alan Paull (APS Ltd)Charlie Paull (APS Ltd)Richard Ralph (University of Wolverhampton)Geoff Ramshaw (UCAS)Linette Sobers-Jordon (UCAS)Shane Sutherland (Pebble Learning Ltd)

The Project Team would like to thank all those who played a part in the PortisHEad Project, including the students and staff and Kingsmead Technology College. In addition to those mentioned above, our thanks go to:

Sandra Winfield (University of Nottingham) for evaluation work and to Sarah Davies, our JISC Programme Manager for her support throughout.

of 40


Table of ContentsExecutive Summary...........................................................................................................6Part 1: Background and introduction...............................................................................7

JISC Capital Programme and e-Learning Programme context.........................................7PortisHEad Overview.......................................................................................................7PortisHEad Partners........................................................................................................8Links with other JISC Projects..........................................................................................8

Part 2: Aims and objectives.............................................................................................9Planned aims and objectives...........................................................................................9Revised aims and objectives.........................................................................................10

Part 3: Methods and activities.......................................................................................12Working with…..............................................................................................................12Process modelling.........................................................................................................14Software development..................................................................................................14Dissemination activities................................................................................................15

Part 4: Outputs..............................................................................................................17Description of revised business processes....................................................................17Learners’ presentation portfolios..................................................................................22Admissions tutors.........................................................................................................22Demonstrators..............................................................................................................23Interfacing with UCAS admissions systems...................................................................24Service Usage Model.....................................................................................................25

Part 5: Outcomes and conclusions.................................................................................26ePortfolios in UCAS-based admissions..........................................................................26Use of XCRI...................................................................................................................26Lessons learned............................................................................................................26Benefits and future developments................................................................................27

Glossary...........................................................................................................................29Appendix A: Demonstrators and technical outputs.........................................................31

Diagrams......................................................................................................................31Appendix B: EVALUATION REPORT: PortisHEad (ePortfolios In Successful HE Admissions)........................................................................................................................................ 32

Background...................................................................................................................32Project description........................................................................................................32Stakeholder identification and engagement.................................................................33Evaluation design and methods....................................................................................35Discussion and recommendations................................................................................39

Table of Figures

of 40


Figure 1: Presentation Introducing the revised PortisHEad Processes..............................17Figure 2: Application System Use Cases..........................................................................17Figure 3: Proposed PortisHEad Application Processes......................................................20Figure 4: PortisHEad Application Processes Continuation to Admissions and Induction. . .21

of 40


Executive SummaryThe PortisHEad Project implemented the integration of an ePortfolio system in the UCAS-based Higher Education admissions process, as an example of the ‘thin ePortfolio model’ recommended by the ePortfolio for Lifelong Learning (eP4LL) Reference Model Project. The next logical step in this development was to implement a practical application of the model using an existing ePortfolio system, with a view to learning lessons from a state-of-the-art implementation. The PortisHEad Project developed additional functionality in the PebblePad system to permit learners to choose courses to apply to, to create new applications against course advertising material, to edit the applications and to submit them to UCAS. We reviewed and amended existing business processes, so that the learner’s ePortfolio could be placed at the centre of the process.The project addressed the Schwartz report1 recommendations:

“to produce a more integrated service for applicants and specifically to facilitate … Transfer of information from applicants; … Structuring the personal statement and reference, in particular through the insertion of course-specific prompts; … Providing feedback to applicants …”

The project team was a partnership between the University to Wolverhampton (lead institution), Pebble Learning Ltd, UCAS and APS Ltd. It also involved staff and students from Kingsmead Technology College.The project initially attempted to use a case study approach with real applicants. However, owing to organisational and technical difficulties it did not prove possible to transfer ePortfolio data into live UCAS systems and beyond, and the project scaled back its scope to cover processes and supporting software development up to the point of connection with the UCAS application system. A small number of ePortfolio-based Personal Statements was completed. Process models for the whole revised process were drawn up.The project was able to produce two demonstrators: PortisHEad ePortfolio-based Personal Statement Interface (PePPSI) and PortisHEad Interoperable Form Fill (PIFF). PePPSI demonstrates how an applicant can use an ePortfolio system to create and send to UCAS a significant proportion of a UCAS application, including a short textual Personal Statement mapped against the requirements for a course advertised through an Entry Profile, and a web-based Personal Statement that the selector could use to gain a more holistic view of the applicant. PIFF is a generic form-fill toolkit that can be deployed alongside any web form to allow the learner to populate it with personal data from an HR-XML enabled repository.The project was able to demonstrate a step forward, providing practical evidence and experience of working with ePortfolio tools in HE admissions and illustrating the types of problems that need to be overcome for full scale implementation. It was also able to show the usefulness of the emerging XCRI-CAP standard for the transformation and mapping of UCAS Entry Profile and other courses data. Although interoperability with the UCAS admissions system was not achieved, the project shows how this could be done once technical interfacing and organisational difficulties have been overcome. The PIFF toolkit is a tangible benefit in terms of a software deliverable with a potentially wide usage, and the project partners, in particular the University and Pebble Learning, are actively seeking real uses of this toolkit.

1 ‘Fair admissions to higher education: recommendations for good practice’ Sep 04

of 40


Part 1: Background and introduction

JISC Capital Programme and e-Learning Programme context

1. PortisHEad is one of the Joint Information Systems Committee (JISC) projects funded under the JISC Capital Programme that ran from March 2007 to March 2009. The overall purpose of the Capital Programme was to help to accelerate the pace of change in e-learning, e-infrastructure, repositories and preservation, users and innovation, network infrastructure and digitisation.

2. JISC’s e-Learning Programme aims to identify how e-learning can benefit learners, practitioners and educational institutions, and advise on its implementation. Its goals are to achieve improved quality of e-learning in the UK through: Practitioners with confidence and skills in managing and facilitating e-learning in

different contexts and with different pedagogical approaches; A technical infrastructure that supports flexibility, diversity and extendibility Easy access to high quality, flexible learning materials; Effective and responsive e-learning policies, systems and structures in place at

local, regional and national level3. The e-learning activity within the Capital Programme complements and builds upon

the Programme’s core activities, with a focus on helping institutions to meet the diverse needs of learners throughout their lives. PortisHEad came under the e-Administration strand of activities and built on earlier work in the ePortfolio domain, principally the MLEs for Lifelong Learning Programme, and more particularly the Specifying an ePortfolio project, the Distributed e-learning programme, the Framework and tools strand of the e-Learning programme, and work on the e-Framework for Education and Research, in particular the ePortfolio for Lifelong Learning (eP4LL) and XCRI.

4. Outcomes and benefits of these projects were expected to be An understanding of the systems and business processes required by higher

education institutions to deliver technology-assisted admissions processes which are efficient and convenient for institutions and fair and supportive for applicants;

An understanding of how services within the e-Framework can support these processes;

Prototype systems for aspects of the admissions process and small-scale user trials of these

Higher education institutions are able to take significant steps towards improved, technology-assisted admissions processes;

Where institutions have taken these steps, applicants are better supported through the admissions process and/or receive appropriate feedback on their application

PortisHEad Overview

5. The PortisHEad Project addressed two primary areas identified in the call for eLearning projects: (i) structured personal profiles, course entry profiles and pre-assessment, and (ii) ePortfolio based admissions.

6. Using a case study approach the PortisHEad project proposed to implement an extension of the ePortfolio Reference Model to enable groups of students from a local school and college to use their ePortfolios to research, prepare and submit Higher Education applications via UCAS, to receive information, advice and guidance (IAG) about their applications from their school or college, to receive feedback from HE admissions staff and to form the starting point for enrolment and

of 40


induction into an HE institution. Using the ‘thin ePortfolio model’, based on Web Services and a Service Oriented Approach, the project aimed to put the learner, via the ePortfolio, at the centre of the HE admissions process.

PortisHEad Partners

University of Wolverhampton7. The University of Wolverhampton has established itself as a leader in ePortfolio

development as is evidenced by the rapid uptake, in both HE and FE, of the University’s ePortfolio product PebblePad.

8. From the University’s perspective the PortisHEad project aimed to design an HE admissions tool set and integrate it into PebblePad to add value for students seeking entry to HE.

Pebble Learning Ltd9. Pebble Learning Ltd is a spin-off company from the University, retaining a very

close relationship with it through the development of PebblePad, the University’s ePortfolio system. PebblePad gives the company a national reach across schools, colleges and universities, which enable developments supported by the product to be implemented widely.

UCAS10. UCAS is the UK’s central admissions organisation for Higher Education. It

administers applications for undergraduate full time courses and operates a number of other smaller scale services. During the early part of the project, JISC and UCAS signed a Memorandum of Understanding to govern their relationship. UCAS was initially committed to implementing the technical integration work necessary for the implementation of the project’s outputs within the undergraduate admissions system.

Links with other JISC Projects

11. PortisHEad is one of several JISC-funded projects within the admissions domain, including the Admissions Domain Map (ADoM) and DELIA Projects led by the University of Nottingham and the eAPEL Project led by the University of Derby. PortisHEad, ADoM and DELIA project staff had regular meetings and communications to share ideas and to co-ordinate outputs. There is significant synergy between the demonstrators produced by PortisHEad and DELIA and output models are included in the ADoM reference website.

of 40


Part 2: Aims and objectives

Planned aims and objectives

12. Working closely with a vendor (Pebble Learning) and with real students and practitioners, the project planned to provide practical knowledge and new open source services that would be a starting point for future implementations as well as providing capacity now. This aim was largely met.

13. The project expected to engage UCAS in technical development closely with its wider partners and stakeholders and to inform further activities under the Managing Information Across Partners (MIAP) initiative, by showing how UCAS information systems can be linked electronically with ICT systems in schools, colleges and other organisations supporting applicants. It did not prove possible to engage UCAS in practical technical development activity using live systems, but process models and demonstrators were developed up to the point of interface with UCAS.

14. The project addressed the Schwartz report2 recommendations ‘to produce a more integrated service for applicants and specifically to facilitate … Transfer of information from applicants; … Structuring the personal statement and reference, in particular through the insertion of course-specific prompts’.

15. The anticipated impact of the project was to enable a close coupling between the IAG and admissions processes on the one hand, and on the other, the applicants’ experience of learning through the IAG events and personal reflections that occur during the preparation, submission and assessment of their applications and onward through enrolment and induction into university life. The project sought to demonstrate the efficacy of fully electronic applications, as well as admissions administration within Higher Education Institutions (HEIs), including faster processing and better integration of applications and admissions processes than current systems.

16. The project planned to implement the integration of the PebblePad ePortfolio system throughout the applications and admissions process as an example of the ‘thin ePortfolio model’ recommended by the ePortfolio Reference Model Project. We intended to review and amend existing business processes, so that the learner’s ePortfolio could be placed at the centre of admissions.

17. The project placed a major emphasis on communications with the e-Learning Programme and with other projects within it. We co-ordinated with other projects undertaking work with UCAS, particularly the Demonstrating Enhanced Learner Information in Admissions (DELIA) project.

18. In particular we aimed to: Produce structured entry profiles for relevant University of Wolverhampton

courses; Revise current IAG and application management practices in a local school and

college, and within the University of Wolverhampton for the selected student groups;

Enable electronic data transfer from the students’ ePortfolio to the University via UCAS, linking into current UCAS centralised procedures;

Provide feedback to applicants, to include assistance to those who are not successful;

Improve induction mechanisms via further data transfer to populate the university ePortfolios of successful students.

2 ‘Fair admissions to higher education: recommendations for good practice’ Sep 04

of 40


19. The project was careful not to disadvantage the students involved and not to jeopardise live UCAS systems.

20. Project coverage was expected to include: IAG during the research and preparation stages of the application process, as

well as on receipt of feedback from the University and induction into the new study programme.

The learner’s research and preparation of the application, particularly personal statements, including peer review and formative assessment by IAG and other staff.

Registration at UCAS. Submission of the application to the University via UCAS. Feedback as a result of summative assessment of the application by the

University. Induction into the University, including further IAG via the University’s

PebblePad system.

Revised aims and objectives

21. During the early months of the project, engagement with partners proved more difficult than expected.

22. UCAS was not able to deliver the expected technical developments and linkages with the live UCAS applications system, except in an email-based off-line ‘pilot’ mode, which did not provide responses to applicants, or onward transmission of links to ePortfolio resources to admissions tutors. Despite proposed changes to the project delivery time scales, including a request for a further iteration of the application submission stage during September and October 2008 for learners entering university in October 2009, the project was not able to gain UCAS’ agreement to transfer ePortfolio data into the centralised application system using real or dummy application data. Therefore the project’s aims and objectives were revised, and the use of a case study approach was of necessity minimised.

23. The project retained the planned activities and approach up to the point at which application data was to have been passed to the UCAS system. At that point, the project was able to demonstrate how data would have been transferred into the admissions system, but work with learners and admissions tutors on the usage of ePortfolio information within admissions decision-making was limited to a consultation workshop, because we were not able to make live information about real students available to admissions staff.

24. The project aimed to continue to demonstrate the potential use of ePortfolios in applications and admissions through a more limited piece of software than originally planned with respect to UCAS linkages. However, our focus for the later stages of the project changed, so that we considered a more generic approach to the use of ePortfolio data in applications than purely for transfer into the UCAS admissions system. This proved to be a fruitful exercise and enabled us to produce a toolkit that we expect to have wide applicability.

25. The project made progress towards the integration of ePortfolio systems within the admissions process as an example of the ‘thin ePortfolio model’ recommended by the ePortfolio Reference Model Project. We reviewed existing admissions processes in relation to UCAS applications and gained insights into the current place of ePortfolios within that domain.

26. Owing to a major re-organisation of staff within the administration of the University, it did not prove possible for the University to commit resources to the production of Entry Profiles for use in the project. This was not unexpected, and was to have been part of the University’s contribution to the project rather than funded by JISC; even nearly two years after the start of PortisHEad it is quite exceptional for any university to manage its Entry Profiles as part of its own direct course advertising

of 40


systems; instead the vast majority of institutions use the UCAS Web-link service, which stores the data at UCAS.

27. Our fall-back plan was to obtain the data from UCAS, with the University’s permission, and to import this into PebblePad using the emerging eXchanging Course Related Information (XCRI) standard.

28. Our revised specific aims for the project were to: Produce a framework for transferring structured entry profiles for relevant

courses into an ePortfolio; Investigate the support needed and the learners’ perspective on ePortfolio

usage in a local school; Show how electronic data transfer from the students’ ePortfolio to the University

via UCAS would function, up to the point of linking into current UCAS centralised procedures.

29. Project coverage included: Technical design and delivery of a prototype tool for the use of Entry Profiles

within the research and preparation stages of the application process using an ePortfolio;

The learner’s research and preparation of the application, particularly personal statements, including peer review and formative assessment by IAG and other staff;

Links to the UCAS registration system; Demonstration of submission of the application to the University via UCAS; Development of a toolkit for generic form completion, showing how an

ePortfolio repository could be used to populate a UCAS or other application automatically.

of 40


Part 3: Methods and activities

Working with…

Learners in school and college30. We had intended to work with up to a dozen students within a cohort of primarily

Art & Design students from City College, Wolverhampton, together with a smaller number of students from Kingsmead Technology College. After initial meetings with both establishments, the team found that none of the identified cohort of Art & Design students at the College intended to apply to the University, so we concentrated on the students from Kingsmead Technology College.

31. Our primary method of supporting the learners and the school staff involved in HE admissions was to provide training in PebblePad for staff and students, as well as individual support in the creation of presentation ePortfolios by a member of the Pebble Learning team. This activity was in addition to the regular IAG support from the school and took place over several weeks from September to November 2007.

32. Support to staff consisted of project briefings and training in the use of PebblePad in general and the HE admissions tool in particular.

33. Support to the learners included face-to-face training sessions in PebblePad usage, plus extensive email, face-to-face and telephone support, both in the PebblePad functionality and the development of the ‘rich personal statements’ (presentation ePortfolios).

34. Close ‘hands-on’ support was very productive in triggering enthusiasm for the creation of new material in the applicants’ ePortfolios. Once the applicants had been introduced to the new process and became familiar with the software functionality – particularly that it was easy to use – they were keen to produce their presentation ePortfolios and proud of their work.

35. Participants were encouraged to use their ePortfolios to reflect on their course choices, as well as to produce presentation ePortfolios, named ‘webfolios’ in PebblePad terminology, to be used as ‘rich Personal Statements’ alongside their normal UCAS applications.

Admissions tutors36. Our work with admissions tutors was limited, because we were unable to

implement a case study approach. We did, however, consult with admissions tutors in Applied Science at the University on their views of the use of ePortfolios for admissions purposes, during the application process and for induction. In addition a workshop was conducted in which views of admissions tutors were gathered.

37. The University of Wolverhampton is primarily a recruiting institution; it seeks to attract as many students as possible and for most of its courses, applicants with more than the minimum entry requirements will receive an offer of a place. This is true of Applied Sciences courses, and this situation coloured the response of admissions tutors.

UCAS38. Working with UCAS proved to be time-consuming and less productive than

expected. In formal terms negotiation of the Consortium Agreement was protracted, owing to UCAS’ understandable concern that they did not sign up to produce outputs that they could not deliver. However, it became clear that engagement of individuals, particularly from technical areas within the organisation, was not a straightforward matter, despite UCAS’ involvement in the

of 40


tendering process and its formal letter of support. Internal structures within UCAS seemed to restrict or at least to delay its ability to engage with externally managed projects, and it took several months to engage managers in UCAS who could decide on technical matters and therefore had the capability to deploy resources to develop required software components. UCAS signed the consortium agreement in March 2008, by which time it had been determined that there was little prospect of enhancement of UCAS’ components in line with the original project objectives.

39. A change request was initiated formally in early January 2008 suggesting a further iteration of the pilot work for October 2009 entry, and with a time scale extending the project to January 2009. This involved UCAS technical work, which was considered by the UCAS Programme Board. However, the necessary business analysis could not be carried out within the UCAS Programme on a timescale compatible with the project schedule, and the Portishead Project Board therefore decided to pursue an alternative that did not require UCAS technical involvement. UCAS was, however, able to provide Entry Profile structural information and (with the permission of the University) relevant courses data for the University of Wolverhampton.

40. An internal group in UCAS led by a UCAS project manager was set up during the summer of 2007 to work on the delivery of linkages between Pebble Pad and UCAS systems. A number of critical concerns were raised by this group. Time scale – an initial time scale for completion of applicant ePortfolios of early

in September was suggested, so that any potential impact during busier times of the application cycle could be avoided. As the applicants were not able to complete their ePortfolios by this date, which was at variance with the advertised PortisHEad project schedule, it was agreed that data could be supplied by the end of November, well before the closing date for applications (15 January 2008).

Data structures – The UCAS data model is highly structured, while Pebblepad data is structured differently and has many areas of free text. Significant sections of the UCAS application record, for example qualifications already taken or to be taken could not be transferred from Pebblepad into the UCAS system, either because it did not exist in Pebblepad, or was structured so differently that automatic transformations could not be carried out. A minimum useful data set was identified between Pebble Learning and UCAS, and we acknowledged that applicants would have to complete their applications through UCAS Apply in the normal way.

Potential impact on live systems and the development cycle – The project team and UCAS technical staff were properly concerned that automatic delivery of data into live UCAS systems might jeopardise operational systems and might compromise the students’ applications. The difficulty here was that the normal development cycle for UCAS technical changes is two years in advance of launch; for example any new developments for 2008 entry systems would have been agreed upon in 2005, designed, developed, tested and implemented before the spring of 2007 for launch in the summer or autumn of 2007. As the PortisHEad project was not linked to this normal development cycle (the project was launched in March 2007), its technical requirements cut across other UCAS initiatives and pressures. For the avoidance of this problem, we proposed in the project planning to deliver data into a separate non-live system for automatic and manual validation prior to transfer into the live databases. UCAS staff completed and tested these systems, and responsibility for operations was passed to the support team in the first week of September. We had arranged to carry out manual follow-up with applicants to acknowledge the receipt of data at UCAS, which would have been marginally satisfactory, but not a significant test of interoperability. Test data was passed through this system, but no live data was available from the schools (see below).

41. While these concerns severely limited UCAS’ involvement in the project, a security issue resulted in a further more critical implementation problem for all applications

of 40


via UCAS services. Following a security audit UCAS decided to remove URLs from the Personal Statements of all applications. This action effectively ended student involvement in the project, because the project’s solution to provision of the rich Personal Statement was via a URL link from the existing application into the applicant’s presentation ePortfolio (see paragraph 72 below). It is interesting that UCAS took this action as it prevents any student from including web links in their Personal Statement, precluding art and design students from referencing online folios, IT students referencing websites or programs, and so on.

Entry Profiles data42. Entry Profiles data for the University’s courses was obtained from UCAS in a series

of pipe-delimited flat files. The fields were mapped to the XCRI-CAP 1.0 XML schema, transformed into a valid XML file and loaded into the PebblePad system. For the demonstrator, it was not necessary to create an updateable web service, particularly since the UCAS data structures were subject to revision, and this was a ‘once off’ process. However, it provided a successful test of the XCRI-CAP information model, albeit on a small scale.

Process modelling

43. The existing UCAS application process is a well known one, and new diagrammatic representations of it were not produced. However, the project did obtain useful information from UCAS about data required within the process, and this information was helpful in producing a model of the revised processes that were to include ePortfolios.

44. We used UML 2.0 diagrams to model the revised processes. The main diagram types used were Use Cases, Activity Diagrams and Class Diagrams. There were three primary purposes of the diagrams: To aid our own understanding of the processes and revisions that we were

proposing; To provide a means of showing people outside the project what the revised

processes were; To enable the business analyst and modeller to communicate user, system and

data requirements to the developers.45. While the first and last purposes were largely satisfied by the technique, we found

that for communication with other people outside the project, particularly those with no experience of UML diagrams and little or no admissions knowledge, a much simplified PowerPoint presentation showing the information flows and activities of applicants, personal tutors, referees and selectors was preferable. The most successful presentation was produced relatively early in the project, with a timed voice-over.

46. The process modelling was intensive activity over a period of several weeks, with a lot of iteration and review by Pebble Learning and APS staff. We would have preferred to engage UCAS and others in this activity, but a lack of familiarity with UML diagrams was a barrier.

Software development

47. There were two types of software produced during the project: Proprietary work within the PebblePad product to extend its human-computer

interface functionality to include the links to the UCAS system and other relevant items;

Open source technical interfaces and toolkits for general application to any ePortfolio or website product.

of 40


PebblePad48. Development was carried out within PebblePad using HTML representations of the

standard flash-based input forms. This provided more agile development which is suited to an innovative project. The backend developments were built upon the .net architecture and utilised HR-XML1.0 for the web services.

49. It proved necessary to dovetail the PortisHEad development with the development and testing of significant new releases of PebblePad, specifically PebblePad version 2, which was launched in September 2007. Although requiring a small and non-critical adjustment in timing for PortisHEad, it proved relatively easy to co-ordinate the two operations, and it took less time to incorporate the changes to the system into version 2 than it would have taken to place them in version 1 and then upgrade them and test them within version 2.

50. Pebble Learning’s experience with their own development of the easy-to-use user interface for PebblePad stood the project in good stead for the development of the PortisHEad pieces. The PebblePad interface metaphor was highly relevant to UCAS applications; based on creating and editing shared or private assets, and publishing them through gateways, with much functionality driven by step-by-step wizards.

51. Many of the facilities already in existence within PebblePad directly supported the processes of IAG and application. For example, creation of presentation ePortfolios, communication via secure channels with peers and personal tutors for the purposes of formal and informal comment, capture of basic personal details.

Open source items and toolkits52. All server scripts for the PIFF toolkit are processed using AJAX technology, which

enables the client JavaScript to gather information from external sources. These scripts have been developed using C# and can easily be modified for other languages.

53. Entry Profile information was imported via a flat file XCRI feed of the University’s data obtained from UCAS. It is important to note that to date, very few if any universities hold Entry Profile data themselves; the data is collected by UCAS via its Web-link data collection system and is displayed on the UCAS website. Similarly the courses information needed for the search facility in PebblePad was obtained via UCAS. The latter data was limited to learning provider titles, course titles and codes, because it was not the intention to replicate a comprehensive course search facility within PebblePad.

54. The main source for the overall design of the service supporting UCAS applications in PebblePad was the demonstrator in the earlier JISC-funded project ‘ePortfolios for Lifelong Learning’. The design implemented much of this demonstrator in a fully functioning fashion. The PebblePad demonstrator for this was named PortisHEad ePortfolio-based Personal Statement Interface (PePPSI).

55. With 5 months of the project left to run, and when it had become clear that it was not possible to integrate the PortisHEad tools with UCAS systems, APS and Pebble Learning agreed to develop an alternative means of supporting the generic aims of the project, choosing a method which firmly situated the control of learner-owned data with the ePortfolio user. PortisHEad Interoperable Form Fill (PIFF) allows students to import their personal data safely from a HR-XML web-service enabled ePortfolio system or data source allowing them to fill in web forms on the internet automatically (PIFF, 2008). This toolkit was designed to be entirely open source.

Dissemination activities

56. The project played a full part in JISC and CETIS discussions, through attendance at workshops, conferences and exhibitions. The demonstrators have formed part of Pebble Learning’s contribution to the promotion of ePortfolios in general and PebblePad in particular.

of 40


57. Project website: http://www.portisheadproject.org/.58. The project participated in a successful workshop session at the UCAS Admissions

Conference in March 2008, at which senior university staff were briefed. It was also represented on the JISC exhibition stand.

59. The project submitted a paper for publication at the ascilite 2008 conference in Melbourne, which was accepted and will be presented in December 2008.

60. Project team members intend to participate in UCAS’ Data Conference in February 2009.

61. The experiences gained via PortisHEad will continue to be used during other JISC activities with which members of the project team are engaged. In particular we have found that the synergies with other eAdministration projects have been considerable, and we are seeking to take forward joining up of the PortisHEad demonstrators with the ADoM and DELIA projects, which have a slightly longer life span than PortisHEad.

of 40

http://www.portisheadproject.org/


Part 4: Outputs

Description of revised business processes

62. A simple explanation of the revised business processes can be found by accessing our PowerPoint presentation introducing the project:

FIGURE 1: PRESENTATION INTRODUCING THE REVISED PORTISHEAD PROCESSES

63. Despite the fact that the project was not able to implement fully the links with the UCAS application system, nevertheless a significant amount of process modelling was completed, to show how this might be done. UML diagrams showing these processes, including use case, activity and class diagrams, are included in the project outputs, and overall application system use cases are shown in Figure 2.

FIGURE 2: APPLICATION SYSTEM USE CASES

of 40


64. The model has been implemented up to and including the point at which the data would be sent to UCAS. From this point onwards the model envisages a continuing service oriented approach to communication between the systems with as much automatic processing as is possible, yet still compatible with the learner remaining in control and the UCAS system rules being followed. These later parts of the processes - everything after ‘Send file’ in the ‘Submit UCAS Application’ UML Activity Diagram and all of ‘WebfolioPS Review’ UML Activity Diagram – have not been implemented.

65. Figure 3 shows how the proposed revised application processes would work. The central swim lane illustrates a strand of processes that involve the applicant and his or her adviser up to the point of sending the application to the universities via UCAS. Above are more detailed activities that the actors carry out. Beneath are some services that are delivered through the PortisHEad demonstrators to support the processes and activities.

66. Some activities are directly supported by functionality in the ePortfolio system, for example editing personal contact details and gathering together relevant data about the learner. Standard communications via email can be handled via the ePortfolio system, but in addition PebblePad provides the capability to share items within groups or individuals in the PebblePad community, so the learner can receive IAG from an adviser to assist in the process of reflection.

67. Others are gaps in current ePortfolio functionality; gaps that we show can be plugged by services in the PePPSI demonstrator. The capability to enable a learner to search for courses, compare courses and create an application while remaining within the familiar online environment provided by the ePortfolio system is a powerful one, integrating the following strands of information and function: Course advertising information, particularly Entry Profile information describing

the types of qualities students will need to succeed on the course; Course discovery functions – typically search facilities; Components from the learner’s existing ePortfolio; The ability to create and refine a standard UCAS style Personal Statement, while

reviewing the relevant Entry Profile; Coupled with this, the ability to create a different Personal Statement for each

HEI to which an application is being made; Creation and refinement of a secure rich Personal Statements (webfolios) that

selectors can access if more information about the learner is required; Finally the application can be sent electronically to UCAS.

68. The UCAS application process is a complex one, and the PortisHEad Project has not addressed all aspects of the process, which would have necessitated a duplication of all its functionality. Specific areas that have not been replicated include the registration process, through which the learner acquires a UCAS personal identifier and login permissions, post-application communications, currently handled via the UCAS Track service, and feedback to unsuccessful applicants. This final point is being addressed by the Effective Feedback to Improve Fair Admissions (EFIFA) demonstrator project.

69. Some complex parts of the application information, in particular the details of all qualifications achieved and to be taken, have not been included in the PortisHEad demonstrators. While the latter appears technically possible, data about qualifications is not yet available in a widely used standardised format, making interoperability very difficult. Areas covered by the systems to transfer data into the UCAS application included: Contact details, such as personal ID, name, address, email address and

telephone number; Choices of institution and course, including course titles and UCAS codes; Employment Personal Statement

of 40


Education (partial data only, using reference data from UCAS in respect of schools)

70. During the PortisHEad work, it became clear that some applicants might wish to deliver basic personal details to UCAS or other application systems, without necessarily adopting the whole ePortfolio approach. The PIFF demonstrator shows how a simple service could be deployed, for example on the UCAS Apply or other direct application website, to enable the transfer of data from an HR-XML enabled repository to a web form. Most importantly the process is under the learner’s control, minimising security and authorisation difficulties.

71. While not included in the PortisHEad demonstrators, the project did address the continuation of the admissions process from application to admissions to induction, as described in Figure 4.

72. In this scenario the selector can gain access to the applicant’s rich Personal Statement via a URL in the standard UCAS Personal Statement. For security purposes, this URL leads to a ‘landing page’ on the ePortfolio system that permits the selector to request access to the relevant one of those multiple rich Personal Statements that the applicant may have created, the correct one identified by an embedded code in the URL, filtering only the URLs for the selector’s institution. An authentication service checks that the email is from a valid HE domain, then sends the specific URL for that applicant’s rich Personal Statement relating to that specific course. An important part of this proposed solution is that the selector need only consult as much of the rich Personal Statement as desired, it is an optional process.

73. Finally the PortisHEad processes envisaged the transfer of material from the school or college ePortfolio, based on the rich Personal Statement in the first instance, to the accepting University, as the basis for induction, learning support and the start of personal development planning in HE. These facilities were contemplated but not delivered, owing to the technical halt at the application submission stage.

of 40


FIGURE 3: PROPOSED PORTISHEAD APPLICATION PROCESSES

of 40


FIGURE 4: PORTISHEAD APPLICATION PROCESSES CONTINUATION TO ADMISSIONS AND INDUCTION

of 40


Learners’ presentation portfolios

74. Four students agreed to participate in the activity. All four were committed to applying to Higher Education and were, at least initially, intending to include the University of Wolverhampton as one of their choices. It should be noted here that they were not applying to heavily over-subscribed courses; students applying for selecting courses might have different views from those included in the PortisHEad Project.

75. Three of the students were enthusiastic about using PebblePad in this context, and their comments during the process and in our post-application survey showed that they valued the self-analytical aspects of the experience and welcomed the opportunity to present themselves more comprehensively than in a conventional UCAS application.

76. Two students completed their presentation portfolios, while the other two only partially completed them. The reason for the limited success of this part of the activity was that the school had brought forward its internal deadlines for completion of UCAS applications to October, leaving insufficient time for our students to finish their presentations. The project team was not aware of this internal change until it impacted on our activity.

77. A member of staff from Pebble Learning gave guidance to the students about how to structure their presentation portfolios. This was resource intensive, because the college was not using PebblePad in their normal teaching and learning activities.

78. Notwithstanding the termination of the case study approach, the foreshortened experience of the students was encouraging:

The webfolios that have been created are shaping up to be fantastic and the students themselves are very excited about the prospect of using them… Although not quite finished, you can clearly see how valuable they have found this process and all have added the URL link into their personal statement… Two of the courses applied for, social care and law, are always popular [and the students] are really pleased they have another way of showing themselves to the admissions tutor, which may help in the selection process.

Student Mentor79. While none of our applicants found that creation of their presentation portfolios

changed their choice of course or university, they found the process useful as an aid to their thinking about future career choices and helpful to clarify why they had made their selections.

“I had already chosen the course before creating the webfolio but it made me think more about why I was doing the course and made me think of which career paths I could take in the future.”“…It didn't change my mind about the course that I wanted to do or where I wanted to study it, however I did really enjoy doing it because it felt as though it gave a more honest picture of me rather than the set word limit of the personal statement on UCAS.”

Comments from applicants

of 40


Admissions tutors

80. For recruiting courses, often operating in circumstances in which all qualified applicants would be offered a place, the usefulness of ePortfolios was seen as minimal, because extra information about the learner was not required for selection purposes. These admissions tutors felt that use of an ePortfolio in the application process was an unnecessary burden on applicants and themselves, because the primary evidence for selection for the vast majority would be the academic qualifications listed on the UCAS application form.

81. However, they were clear that an ePortfolio provided a more rounded view of the individual, so that there was an opportunity for dialogue between tutors, other advisory staff and the learner. This would be likely to provide highly relevant information for better targeting of learner support of all kinds, leading to better retention and an easier transition to HE for the learner. The ePortfolio could also be the foundation of student induction and the basis of the development of the student’s personal development plan. For these purposes transferability of information from the school or college system or via UCAS would be key.

82. On the other hand three concerns expressed were that ePortfolios in holistic admissions might leave the selectors open to legal challenge over what could be considered to be fundamentally subjective assessment of ePortfolios in selection decisions, there was a potential question mark over the reliability of the information, and over potential costs in terms of staff time to digest presentation portfolios and to support effective ePortfolio usage. In addition some selectors mentioned that where the section of their Entry Profiles related to personal qualities, it might be aspirational, rather than definitive, and it was not necessarily the intention to force all applicants to demonstrate all the qualities outlined in it.

Demonstrators

83. The project has produced demonstrators showing the main new processes from the applicants’ viewpoint: the PortisHEad ePortfolio-based Personal Statement Interface (PePPSI), and PortisHEad Interoperable Form Fill (PIFF). Figure 3 shows how the demonstrators fit with the application process.

PortisHEad ePortfolio-based Personal Statement Interface (PePPSI)http://www.pebbleweb.co.uk/portishead/84. PePPSI demonstrates how an applicant can create, edit and send to UCAS both the

short textual Personal Statement currently required in the conventional UCAS application and a rich Personal Statement that will sit alongside it. The short Personal Statement is created with easy reference inside PebblePad to Entry Profiles that Higher Education Institutions supply for their courses, so that the learner can relate his or her text to the academic and other entry requirements as well as the qualities desired by the HEI for its students. The rich Personal Statement provides additional information about the learner, and controlled by the learner, including text, graphics and other media, together with evidence of achievement, activities and reflections, in the form of authenticated artefacts or links to them. The UCAS application would simply contain a URL link to the rich Personal Statement, so that the extent of usage of the new information source is left in the hands of the selector.

85. The PePPSI demonstrator tool allows ePortfolio users to: Access the UCAS application services and register themselves; Conduct course searches from within their ePortfolio environment; Utilize course information to write Personal Statements against learner Entry

Profiles; Submit their Personal Statement using a web service to their application on the

UCAS system;

of 40

http://www.pebbleweb.co.uk/portishead/


Finally, and perhaps most significantly, the tool allows users to publish specific ‘presentational’ or ‘application’ ePortfolios to any of their 5 named institutions.

86. As the current Personal Statement on the UCAS application is common to each of the institutions applied to, the ability to create and publish unique ePortfolios to each institution is seen as an important contributor to enhancing the relevant information about the learner that is available to selectors, thereby supporting holistic assessment of the individual learner.3

PortisHEad Interoperable Form Fill (PIFF)http://www.pebblepad.co.uk/piff/87. With 5 months of the project left to run APS and Pebble Learning, supported by the

Project Board, agreed to develop an alternative means of supporting the generic aims of the project, choosing a method which firmly situates the control of learner-owned data with the ePortfolio user. PortisHEad Interoperable Form Fill (PIFF) allows students to import their personal data safely from a HR-XML web-service enabled ePortfolio system or data source allowing them to fill in web forms on the internet automatically.

88. Webmasters of sites containing educational or employment related forms can very easily implement this tool kit on their web-based forms. It is simple to use and secure. None of the student’s personal information is stored anywhere and the information can only be obtained by the student after they log on to their ePortfolio system using their user name and password.

89. The system has been designed to be compatible with every major browser and has been tested with Internet Explorer 7, Firefox 3, Opera 9 and Safari 3.1.1. The system is not dependent on any libraries and makes use of Javascript and Ajax to perform its activities. The system is relatively light on resources.

90. In simple terms the web master includes a button on the web page which allows the student to log on to his or her ePortfolio account; single version data (for example a single surname) automatically populates the mapped field on the form. Where data has multiple values (for example address or email) the toolkit allows the user to choose the ones they wish to use for this current application. As part of the demonstration toolkit examples have been created which allow users to use their ePortfolio data to populate forms for: A popular online job-search company; An undergraduate programme for overseas students; A post-graduate programme; An application for a health-service card; The core fields of the UCAS registration form.

The PIFF toolkit is available at http://www.pebblepad.co.uk/piff/. The toolkit contains a non-technical introduction, technical details, including the data fields involved and relevant code, as well as a range of apposite forms as examples.

Interfacing with UCAS admissions systems

91. For reasons stated earlier it was not possible to complete the PortisHEad interfaces with UCAS admissions systems. The UML diagrams give an overview of how the processes were originally intended to have worked.

Entry Profile and other courses data92. We had intended to produce a simple web service to obtain Entry Profile and other

courses data from UCAS, on the basis of earlier work with the EP4LL Project, which had sample WSDL files and service definitions. Unfortunately this did not prove possible, owing to expected revisions to UCAS course information management

3 See Schwartz, 4.4

of 40

http://www.pebblepad.co.uk/piff/



arrangements. Instead the data was obtained as a single output of courses data that would enable an applicant to pick any UCAS course (by institution, course code and course title), and to refer to Entry Profiles for the University of Wolverhampton.

Application data93. The nature of the interfaces was discussed in detail with UCAS technical staff.

Owing to the complexity of the information requirements of the UCAS application processes, the necessity of maintaining the integrity of the system and the quality of data, and the need for sign off from the applicant and the adviser in the school or college, the original intention to complete the whole application from within the ePortfolio system was scaled back. Instead the ePortfolio system would output a series of files for import into an intermediate UCAS data store for verification and approval prior to final transfer into the UCAS Apply system. Once the data transfer was completed, a process which was expected to take a couple of working days at most, the applicant would have access to the data through his or her normal UCAS Apply login. The applicant would then be able to complete the remaining portions of the application in the conventional manner.

94. UCAS application data is partitioned into sections – Personal Details, Education, Additional Information, Choices, Employment, and the Personal Statement – which made it simpler to handle partial applications. The main sections addressed were Personal Details, Choices, Employment and the Personal Statement, whereas other data was required to be in a detailed proprietary format that did not lend itself to easy transfer. A test system not linked to the live UCAS application system was developed by UCAS and test data was loaded into it. Although this did not provide an interoperable connection between Pebblepad and UCAS systems with appropriate automatic acknowledgements to users to prove the technology, there is no reason to doubt that data items could be transferred from an ePortfolio directly into UCAS systems in future, after the development of appropriate import mechanisms into them.

95. Each output file used the UCAS personal identifier to uniquely identify the applicant. In future implementations this could be replaced very simply with the MIAP Unique Learner Number, which is currently being implemented in UCAS systems.

Service Usage Model

[Current SUM “UK HE Admissions Structured Personal Statement” to be developed with service expressions rather than service genres – this section not yet completed.]

of 40


Part 5: Outcomes and conclusions96. The outputs of the PortisHEad Project have been different from the planned ones.

On the other hand the project was able to develop both an admissions demonstrator (PePPSI) and a generic form-fill tool (PIFF).

97. The necessary abandonment of the planned case study approach meant that testing of the human factors in the use of the proposed new processes and tools was not possible, and only relatively weak anecdotal evidence was gathered for this. However, the project gained insights into the difficulties of working across institutions with very different cultures and work practices, and despite many difficulties two very useful demonstrators were produced, one of which forms a toolkit with potentially wide implementation usage.

98. The project has taken a step forward from the eP4LL Project’s outputs, providing practical evidence for the utility of ePortfolio tools to support admissions, while illustrating a range of problems for adoption.

ePortfolios in UCAS-based admissions

99. Successful use of ePortfolios in admissions is likely to be based on four conditions: Embedded use of ePortfolios in the school or college teaching and learning

culture, so that recording of reflections and relevant evidence-based material reduces the extra time required by the learner, and minimises the support from IAG staff.

Sufficient staff resources in the school or college to assist students in the creation of good quality presentation portfolios, bearing in mind that for many students HE application may be their first experience of this activity. Support would be two-fold: use of the ePortfolio’s functionality and creation of an appropriate quality of portfolio against the requirements of the courses applied for.

The integration of information systems in the feeder school or college, so that the students’ personal data, including qualifications taken and to be taken, can be included in the data supplied via UCAS.

The integration of information flows between the school and college and UCAS, so that the application process from the learners’ point of view remains as technically straightforward as the current UCAS Apply process.

100. In the absence of these conditions, take-up of ePortfolios in admissions is likely to be severely restricted, because applicants would be inadequately advised by professional staff, and the application process would be too cumbersome.

Use of XCRI

101. It proved possible to use XCRI-CAP v1.0 to transform and map UCAS Entry Profile and other courses data to XCRI-CAP v1.0 for the albeit limited purposes of the project. This outcome was passed on to the XCRI Support Team and other interested parties, and has been useful in the further development of the XCRI-CAP.

Lessons learned

102. ePortfolio services and systems allow users to personalise their ePortfolios and make extensive use of relatively unstructured data. In addition the lack of standards for large areas of personal data, including educational data such as qualifications, means that different systems have different structures and vocabularies for describing their data. In contrast UCAS has very heavily structured data. These factors make interoperability between these disparate systems problematic and require technical mapping and transformation processes to be planned and implemented with a great attention to the detail.

of 40


103. It proved difficult to engage UCAS staff in project work despite a Memorandum of Understanding between JISC and UCAS. While some individual managers and other members of staff were at times willing to take an active role, UCAS decisions and activities were not timely in respect of the needs of the project, nor did these decisions and activities reflect its formal role as a project partner supporting and actively part of the project. The result was a drastic curtailment of the initial scope of the project to the absolute minimum.

104. At the time of tendering and the start of the project, it was not clear to the team, including UCAS members, that UCAS had or would adopt a policy that would not allow development projects to run on live production systems. This proved to be a major stumbling block in a case study based project.

105. It is important to gain the sign up of managers in organisations who have the ability directly to deploy the staff and other resources likely to be used in the project. For engagement to be effective, this requires: Routine contact with relevant personnel, not management by exception; with

the latter, the accepted practice for PRINCE2, the project may never hear of important changes to personnel or other circumstances.

Either a good understanding by the project planners and key managers in the project partner organisations of what is expected and when; or a close involvement of the key managers in the development of project plans.

Long time scales for project planning and implementation. Agreed escalation routes, so that blocks can be cleared.

106. Technical problems pale into insignificance compared to political and organisational ones. It is difficult to over-estimate the time required to overcome this type of problem.

107. More active risk management and greater attention to detailed activities to reduce the project’s exposure to risk would have been preferable.

108. The project adopted a formal Change Request procedure for major changes, which was used three times to great effect. This process helped to clarify what was practical in the changed circumstances.

109. The project used a school and college that were well known to the University of Wolverhampton. However, neither of them were existing users of PebblePad, and, although a positive side of the project was its extension to new audiences, it would have been preferable to engage with a culture with PebblePad already embedded. An alternative approach that might have been more successful would have been to invite participation from schools and colleges already using PebblePad. This would have reduced the requirement for basis PebblePad training alongside the specifics of the project extensions.

110. We found it very useful to share our experiences with our sister eAdmissions projects, especially ADoM, DELIA and the XCRI projects. We have been able to join up the demonstrators to some extent and have produced a joint overview of an admissions scenario from which our own model of ePortfolio based admissions has been drawn. Figures 3 and 4 show the PortisHEad aspects of this model.

Benefits and future developments

111. The main tangible benefit to come from PortisHEad has been the PIFF toolkit, which will, we believe, have potentially wide usage, permitting quick and easy form filling from HR-XML repositories. The project partners, in particular the University and Pebble Learning, are actively seeking real uses of this toolkit, for example for its job application form system. Pebble Learning staff are promoting the toolkit as part of their marketing activity. Other PebblePad users could consider using it for their own purposes, for example for evaluation forms, and the toolkit contains examples of potential usage that we strongly recommend to JISC for further promotion.

of 40


112. The PePPSI demonstrator shows the potential of ePortfolio usage in the UCAS admissions domain and has identified technical barriers. Implementation in this particular area depends on UCAS. However, it would be possible for the methods demonstrated in PePPSI to be adapted for use in other application systems, and the service expressions given in the Service Usage Model (to be completed) will be an aid to this potential future work.

113. Pebble Learning has an ongoing strand of work to develop a simplified version of PIFF as a browser extension for Firefox. While not a part of the original specifications of the project, Pebblebar is an extremely powerful example to demonstrate the capabilities of the PortisHEad project. Pebblebar is a simple Firefox extension which can help candidates filling personal information in web forms over the internet using information from their existing ePortfolio accounts. It works on the same principle as PIFF and makes use of the same web service that PIFF uses to authenticate and identify candidates. However Pebblebar further extends the concept of PIFF by simplifying the process. The user installs the extension and then updates her login details within the extension by entering her username and password in the options menu of the extension. The toolbar automatically authenticates the user and fetches her information from her ePortfolio system. The user can simply right click anywhere in the form and select "import from PebblePad" option and the extension does the rest. However, the extension does have a few limitations when compared with PIFF. Pebblebar will only fill in textboxes. PIFF is capable of matching and filling in data in TextArea and Select fields as well. PIFF also allows the user to select the information she wants to fill in where there are multiple data items for the same field. Pebblebar will fill in the first item in the system in case of multiple data items. In order to make your forms "PebbleBar friendly" make sure that the name and id tags for all your form elements are the same.

of 40


GlossaryTerm Description

ADoM JISC funded Admissions Domain Map Project led by The University of Nottingham

CETIS Centre for Educational Technology and Interoperability Standards

DELIA JISC funded Demonstrating Enhanced Learner Information for Admissions Project led by The University of Nottingham

eAPEL JISC funded Electronic Assessment of Prior Experiential Learning Project led by the University of Derby

EFIFA JISC funded Effective Feedback to Improve Fair Admissions Project led by UCAS

Entry Profile A structured collection of information about a course, usually published on the UCAS website (http://www.ucas.com)

eP4LL JISC-funded ePortfolio for Lifelong Learning (eP4LL) Reference Model Project

FE Further education

HE Higher education

HEI Higher education institution

HR-XML A library of XML schemas to support human resource management

IAG Information, advice and guidance

JISC Joint Information Systems Committee

MIAP Managing Information Across Partners; government initiative to streamline the collection, handling and sharing of information on learning and achievement for education and training organisations. See: http://www.miap.gov.uk/.

Pebblepad ePortfolio software system produced by Pebble Learning Ltd

PePPSI PortisHEad ePortfolio-based Personal Statement Interface demonstrator. See: http://www.pebbleweb.co.uk/portishead/

PIFF PortisHEad Interoperable Form Fill demonstrator and toolkit. See: http://www.pebblepad.co.uk/piff/

PortisHEad ePortfolios In Successful HE Admissions

Personal statement Description of the learner’s own qualities and experience in support of an application. In the UCAS system it is currently (2008 entry) limited to 4,000 characters.

PRINCE2 An industry standard project management methodology

of 40



http://www.miap.gov.uk/

http://www.ucas.com/


Rich Personal Statement

A Personal Statement (see above) using a web-accessible multi-media artefact (see also Webfolio)

Schwartz report ‘Fair admissions to higher education: recommendations for good practice’ report, September 2004

SPA Supporting Professionalism in Admissions project

SUM Service Usage Model. Used within the e-Framework, SUMs show the interoperability requirements that support the functional process requirements and how various technical components can be used to support those interoperability requirements.

Track A UCAS service allowing an applicant to see the progress of their application online

Webfolio Term used in PebblePad to describe a presentation portfolio created as one or more HTML (web) pages.

UCAS Organisation that manages undergraduate (and some other) higher education applications on behalf of most HEIs

ULN Unique Learner Number. Implemented via a MIAP service, the ULN identifies a specific learner.

UML The Unified Modeling Language

URL Uniform Resource Locator. Identifies a page on the internet

WSDL Web Service Definition Language

XCRI-CAP eXchanging Course Related Information – Course Advertising Profile; see http://www.xcri.org/

of 40

http://www.xcri.org/


Appendix A: Demonstrators and technical outputsAll demonstrators and technical outputs are available on the web, as follows:

PortisHEad ePortfolio-based Personal Statement Interface (PePPSI)http://www.pebbleweb.co.uk/portishead/

PortisHEad Interoperable Form Fill (PIFF)http://www.pebblepad.co.uk/piff/

Service Usage ModelTo be completed.

Diagrams

A range of overview UML process diagrams and descriptions are available to support this report:

PDFs of PortisHEad’s proposed new system, detailed in overview use case, activity and class diagramsHTML files describing the use cases shown in the Use Case diagrams

of 40




Appendix B: EVALUATION REPORT: PortisHEad (ePortfolios In Successful HE Admissions)

Background

PortisHEad ran from March 2007 to October 2008. The project was funded by JISC under the eLearning Capital Programme under the eLearning and eAdministration themes. The programme activity was for technical underpinning of developments to support learning and teaching. The request from JISC was for Demonstrator projects implementing the reference models, working with key sector partners such as UCAS.The project was to build on the work of the ePortfolio for Lifelong Learning (eP4LL) ePortfolio reference model project, developing a practical working model of that project’s demonstrator for use of an ePortfolio to support an application to UK undergraduate higher education through UCAS. PortisHEad undertook to implement a practical working model using an established ePortfolio system (PebblePad) to demonstrate a real-life pilot version of the web services developed by eP4LL , including use of structured Entry Profiles, with year 13 students applying to the University of Wolverhampton through the UCAS Apply system.JISC funded two other projects under this programme in the area of eAdmissions: ADoM (Admissions Domain Map) aimed to produce a reference model specifically for UK undergraduate eAdmissions to HE, and DELIA (Demonstrating Enhanced Learner Information for Admissions) aimed to conduct a wider investigation into institutions’ use and potential use of information not currently available to them through UCAS Apply, including the use of ePortfolios and structured Entry Profiles. It was therefore anticipated from an early stage that all three projects would inform and feed into each other, with PortisHEad’s practical case study approach as a possible test bed for new technology and ideas. Outcomes and benefits of these projects were expected to be An understanding of the systems and business processes required by higher

education institutions to deliver technology-assisted admissions processes which are efficient and convenient for institutions and fair and supportive for applicants;

An understanding of how services within the e-Framework can support these processes;

Prototype systems for aspects of the admissions process and small-scale user trials of these

Higher education institutions are able to take significant steps towards improved, technology-assisted admissions processes;

Where institutions have taken these steps, applicants are better supported through the admissions process and/or receive appropriate feedback on their application

The project also aimed to support the fair admissions agenda by addressing the recommendations of the Schwartz report [1] to produce ‘a more integrated service for applicants and specifically to facilitate… Transfer of information from applicants… Structuring the personal statement and reference, in particular through the insertion of course-specific prompts… Providing feedback to applicants’. It also built on the results of three scoping studies commissioned by JISC: mechanisms for assessing the fairness and effectiveness of selection processes in admissions to Higher Education, good practice in the provision of feedback to applicants and good practice in supporting learners through application to Higher Education.

Project description

The project aimed to implement the integration of the PebblePad ePortfolio system throughout the admissions process as an example of the ‘thin ePortfolio model’

of 40


recommended by the eP4LL project. It would review and amend existing business processes so that the learner’s ePortfolio could be placed at the centre. The objectives were to: Produce structured entry profiles for relevant University of Wolverhampton courses; Revise current IAG and application management practices in a local school and

college, and within the University of Wolverhampton for the selected student groups; Enable electronic data transfer from the students’ e-Portfolio to the University via

UCAS, linking into current UCAS centralised procedures; Provide feedback to applicants, to include assistance to those who are not successful; Improve induction mechanisms via further data transfer to populate the university e-

Portfolios of successful students.However as circumstances meant that these could not be achieved in the timescale of the project, in the final phase of the project these were revised to: Produce a framework for transferring structured entry profiles for relevant courses

into an ePortfolio; Investigate the support needed and the learners’ perspective on ePortfolio usage in a

local school; Show how electronic data transfer from the students’ e-Portfolio to the University via

UCAS would function, up to the point of linking into current UCAS centralised procedures.

Stakeholder identification and engagementBrief summary of stakeholder analysis, more detail regarding interest in project and outcomesThe stakeholder analysis below is based on that from the original evaluation plan for the project, with updates and comments in italics.

Stakeholder Interest/stakeUniversity of Wolverhampton senior management

Successful outcome of project important for its own sake and for development of HEI expertiseAlthough the project achieved buy-in from senior management, which was beneficial in the initial phases, staff changes meant that the effect of this diminished as the project progressed. One of the lessons learned is the importance of obtaining both bottom-up and top-down buy in from institutional staff.

University of Wolverhampton admissions staff

Potential for enhanced eAdmissions in future for greater efficiency and effectiveness of processAlong with the ADoM and DELIA projects, this project has found that admissions staff exist in a pressured environment with inflexible deadlines, which tends to limit their openness to visionary and experimental practice. They have also been subjected to a significant number of new initiatives and expectations in recent times. So while the University staff have an overall interest in this outcome, their main priority lies with their immediate day-to-day business. They were clear that the ePortfolio could provide a rounded view of the individual and saw its potential for supporting induction, but expressed some concerns over reliability, cost, subjectivity and time needed to process.

University of Wolverhampton academic staff

Better selection leading to more committed studentsWolverhampton is essentially a recruiting institution. However both this project and the DELIA project have highlighted the potential of ePortfolio to support fair

of 40


admissions practice for Widening Participation and borderline candidates

University of Wolverhampton registry staff

Enhanced expertise in eAdmissions technologies

Pebble Learning Enhanced PebblePad productInvolvement in the project has enabled staff development opportunities, with staff gaining knowledge and experience in use of standards (notably HR-XML), and the two demonstrators produced have influenced thinking and planning for further development and use of the product in the sector

College/school IAG staff Integrated IAG and applications is easier for studentsThis appears to be the understanding of the staff involved in the piloting. However this area requires further exploration and joint evaluation with the outcomes of the SRM and learner experience of eLearning projects.

Applicants Easier to apply; better integration with existing ePortfolioThe learners who completed their webfolios were proud of them and gave anecdotal evidence that they had found the process helpful. The demonstrators require further refinement and then piloting with a further group of real students to explore this further: as above, this needs to be integrated with feedback from other projects working in the area of admissions. It would be worthwhile to explore this area with 14-19 admissions, which is being explored by a number of other projects.

UCAS Potential for enhanced eAdmissions in future for greater efficiency and effectiveness of process;Enhanced expertise in service-oriented approachTechnology transferable to support any ePortfolio productInvolvement in the project from UCAS was limited and they were unable to deliver the expected technical developments and linkages to their live systems which would have made the project a genuine testbed for ePortfolio-based applications, which was disappointing. However useful insights into UCAS internal and business processes were gained, the benefits of which are being realised by other projects working with UCAS, especially DELIA and EFIFA.

APS Enhancement to experience in business process modelling, service-oriented approach and requirements engineering technologiesAPS has developed significant expertise in this area, which other projects have been able to benefit from.

HE admissions community Potential for enhanced eAdmissions in future for greater efficiency and effectiveness of process;Better selection leading to more committed studentsEnables deeper and wider eAdmissions technologiesThe lessons from this project need to be aggregated with those learned from other projects working in admissions, including eAPEL and projects working in 14-19 admissions, which may include those not immediately funded by JISC. The technological lessons learned from the PortisHEad demonstrators, other eAdmissions projects, XCRI and others have potential to support the wider data sharing agenda, including MIAP and transition from education to employment.

of 40


The project plan identified critical success factors as including: Successful software development at Pebble Learning and the University of

Wolverhampton in line with identified requirements Liaison with UCAS, specifically for technical development Good quality communications, both within the project and to the wider communityIn the event, these were accurately identified. While the development of software by Pebble Learning remains one of the major successful technical outputs of the project, issues with development by the University of Wolverhampton and liaison with UCAS were the two major contributing factors that necessitated the rethinking of the project at an advanced stage.

Evaluation design and methods

Data collection and processing proceduresThe evaluation of the project has necessarily qualitative. It was hoped that it might be possible to address the efficiency question in terms of monetary costs, but this was not possible as the process was not completed. Reporting is therefore textual rather than statistical. Similarly, evaluation has been summative rather than formative.The evaluator was given access to all project documentation and outputs, including the two demonstrators. Further information was gathered through informal discussions with the project team, especially the Project Manager and Director. It was originally intended to interview students and IAG staff on their engagement with the process, but as only three students completed webfolios, it was felt to be sufficient to examine feedback from them and from IAG staff to the immediate project team in the context of other evidence.

Initial evaluation questionsThe evaluation plan drawn up in the project planning stage focused very much on the process, its effectiveness and its efficiency. The main areas highlighted in the original plan were: Practicality of processes studied Effectiveness, efficiency and fairness, compared with current practice Potential usefulness of outputs for enhancing admissions processes in future, from

the point of view of various actors Positioning within the context of eAdministration and eAdmissions developments.Original questions were expected to be: Is it practical to use ePortfolios in UCAS admissions in terms of the flow of

information? Is the process effective: will it enhance Admissions? Is the process efficient? Is the process fair to applicants? Do the applicants find the application webfolios useful? Do admissions tutors find the application webfolios more useful than not having

them? Have the project outputs met the stated quality criteria? How useful is the project material to other current projects in eAdmissions?As the process itself was not fully demonstrated, this approach has been modified to look more at the actual outcomes and processes of the project and their effects and value. The questions addressed draw on the framework drawn up for the ADoM and DELIA projects by Delyth Chambers.

of 40


Project outcomes Have the project objectives been met? If not, why?

It was not possible to meet the original project objectives, demonstrating an end-to-end process, because UCAS was not able to deliver the expected technical developments and linkages within their live system. While the team had anticipated a certain level of difficulty in interfacing with UCAS systems, at the outset this was not expected to have the level of impact that actually arose. Further issues arose through unpredictable circumstances – students at one post-16 institution did not want to apply to Wolverhampton, which illustrated the wisdom of choosing at the outset to work with more than one institution – and through unexpected changes in local procedures and timings, carried out by the institution in order to support their students, whose interests had to remain paramount above the work of the project.

However it is to the credit of the project team that despite changes to the original plan and in the process they had to follow, no students were adversely affected in their application to HE and feedback suggests that in fact they benefited from participating.

Have the objectives changed over time? If so, why and in response to what? The impracticability of pursuing a case study approach inevitably meant that the original objectives had to be modified. The project team were able to draw on their own experience and knowledge in the field to turn project around and achieve a useful outcome, when it would have been acceptable to just terminate the project once it was clear that the original passage of data through UCAS systems was not possible. There is still space for the original objectives to be tested, possibly in a modified form, at some future date. The project has shown that existing processes and practices are not yet ready to carry out a full test of ePortfolio-based admissions through UCAS: however with the growth of largely and fully electronic application and admissions processes in other sectors, the set of process diagrams and the two demonstrators could prove useful in, for example, postgraduate admissions or the 14-19 sector. The use of HR-XML in the PEPSI demonstrator makes it particularly suitable for further development in work involving transitions between education and employment.

How effective were communications between the project team? The planned intention was for the whole team to use resources and facilities within PebblePad to ‘practise what they preached’. The project website is a PebblePad webfolio and includes the project manager’s blog. However this has been of limited success; it has not been advisable to document some politically sensitive issues in a public medium, and input to the website is based on work by the project manager and project director, together with links to technical outputs. There have also been issues over ease of export to other blogs. Choice of tools is influenced by how easy they are to publish from. There is scope for further exploration of the original idea, linking a project site to webfolios from a wider selection of team members blogging both personal and technical progress, which could form the basis for an alternative style of project report.

However it is clear that communication between the immediate project team has been good, facilitated by their shared work in other areas. Communication with the wider team, including staff at Wolverhampton and at UCAS, has been less successful. This has demonstrated an issue common to many projects: while it is important to get support and buy-in from senior management in institutions, this needs to be percolated down within the organisation, with both top-down and bottom-up agreement and commitment. The project also suffered from changes in senior management at Wolverhampton, highlighting the issues that can arise when a project champion moves on.

What outputs have there been and how well have they been received?

of 40


The project sees its two major outputs as the two demonstrators, PEPSSI and PIFF, which show how learner data can be gathered and transferred, and have the potential for further development by other projects. There is some immediate progress in this area: the DELIA project is actively pursuing a way of joining these up with its own demonstrator work; PebblePad will look at how they can be developed further and integrated into the existing product, and a number of other possible uses are being explored through the project team’s involvement in other work.

The technical documentation for the PIFF demonstrator explains how institutions can quickly and easily integrate the tool into their own sites, and the range of examples extends beyond educational admissions (it includes an application for a European Health Insurance Card). By developing a relationship with one or two key production sites, this could easily be extended into a mainstream development and shared with the wider HR-XML community.

The technical modelling and diagramming developed for the project is also potentially useful to other work in admissions: it is recommended that these be input to the new Innovation Knowledgebase to facilitate sharing with other projects. However it has been interesting to note that for communication with non-technical practitioners, the project has found the most useful instrument to be an animated Powerpoint presentation produced early in the project.

What technological developments have been achieved, and to what extent does their function, quality and supporting information make them suitable for uptake by the sector? How modular, extendable, flexible, scalable and standards-compliant are any technical innovations? The PEPSSI demonstrator is the next stage of development from the eP4LL demonstrator model, using an actual ePortfolio system as the medium through which the learner is able to access a variety of resources including course search, matching personal statements to Entry Profiles and submission of their Personal Statement via a web service. It shows how to demonstrate both a short and a rich personal statement. There is still further work to be done to integrate this fully into a working system, and into UCAS, but it allows users and practitioners to develop their understanding of how rich personal statements can be developed and used. The demonstrator has contributed to understanding and use of the emerging XCRI standard, and its work with XCRI and Entry Profile data will be utilised further by the DELIA project.

The PIFF demonstrator was generated in the later stages of the project, and demonstrates matching of ePortfolio data to a variety of external application forms. This could be developed further to demonstrate use of ePortfolio in a number of application situations, including for employment or for part-time study from within employment. There is still a need to address IPR and data protection issues, which may be possible using the outcomes of the TAS³ project work on sticky policies and secure data sharing. As this demonstrator engages with HR-XML, opportunities should be sought to raise awareness of its potential within the community working in use of the standard within education and across education and employment.

At the moment the published documentation for the demonstrators is minimal and they require a login provided by the project. Publication of documentation, possibly including a dummy system with a generic login or of a Flash movie of the process, may help to facilitate further engagement.

There is a lot of scope for further development based on the process models for the revised application process: this has been noted by the ADoM project for inclusion in its admissions knowledgebase, but inclusion in the JISC IB may also

of 40


allow others to benefit from the thinking behind the modelling. Other projects may also benefit from feedback on use of UML with non-technical staff and the greater effectiveness of using an animated Powerpoint presentation with this target group.

What are the drivers and brakes for successful implementation/use of technology?

In common with a number of other eLearning projects, PortisHEad has found that sign up by senior managers in partner institutions is not enough to guarantee co-operation. There is a need for a pincer movement: looking for both top-down and bottom-up commitment. Senior managers may see the political benefits of involvement, but the engagement of technical staff and, importantly, the middle management responsible for their regular deployment, is crucial to achievement of any degree of progress.

A further brake on implementation came from attempting to supplement an existing process. The UCAS form was found to be too complex to replicate in the ePortfolio, which meant that the design had to be modified so that the ePortfolio sent information to Apply. However the dialogue within the form proved too complex even for this approach (for example it can be very long; the fields seen by users, especially at registration stage, depend on the information that has been entered). The final compromise solution was to access the actual registration from a link within PebblePad. While this limits the usefulness within the UCAS system, however, it suggests that the resulting implementations could be more easily adapted to work with other application processes.

As with most use of technology, a deciding factor for success is the engagement of the users themselves and those working to support them. Learners who have been using, and encouraged and supported to use, ePortfolios or ePortfolio-type systems to generate material and evidence over a sustained period are more likely to see the value of, and even have the expectation of, making use of this in an application process.

To what extent has our development work led to improved or more efficient practice in learning, teaching and administration? What are the issues around integrating student-and practitioner-owned technologies with institutional systems? Many of the immediate questions around use of ePortfolios for admissions are ‘just technology’; the bigger issues are cultural, and lie with changing practice. The feedback from individual students who engaged with the project is encouraging and suggests that they found the process of using an ePortfolio to prepare for application to HE useful. The fact that most support offered to learners and IAG staff was in the form of training in use of the PebblePad system suggests that they were able to engage readily with the idea of using ePortfolio data to support the applications. Close support was productive in triggering students to generate and populate ePortfolios: however it is not clear how scalable this could be with larger numbers of students. The small size of the sample who completed webfolios means that it is not possible to identify whether the project was able to demonstrate support for fairness in the admissions process.

For learners to use ePortfolios successfully in admissions they need to be embedded in the school or college culture, otherwise there is a risk that the focus shifts to the process of evidence gathering and the likelihood of generating good quality presentations is lowered; similarly, IAG staff need to be familiar and comfortable with the concept, value and potential of ePortfolios in order to offer the best quality advice to learners in their use.

of 40


What was the impact of innovations on learners? On teachers? On the institution? Students who actually created webfolios were found to be those most likely to do it anyway, and therefore those who needed to do it least. There is also the issue of coaching for personal statement writing: does use of an ePortfolio to gather evidence over time diminish the need for/likelihood of coaching?

It was also found that despite promises and commitment made by institutions, there were practical issues with taking time out of a packed curriculum to work on webfolios. It would be useful to explore use of ePortfolio-based applications with an institution or course which already has an established tradition of ePortfolio use, in order to isolate issues that are specifically related to the process and do not involve introduction of a change in culture or local practice.

How are the findings of this work being disseminated? How successful has this/is this likely to be?Despite the issues with UCAS technical systems, the project outcomes have been and are being disseminated to the admissions community through UCAS-sponsored events, including the annual Admissions Conference and the Data Conference to be held in Spring 2009. The usual CETIS SIG channels have been utilised, and the project has been able to take advantage of other events such as the CRA residential, where a highly successful workshop engaged with a wider range of staff than those working in admissions, and the ascilite 2008 event in Australia.

Discussion and recommendationsKey messages, conclusions and any recommendations for further evaluation or other studiesThe project has learned a lot about planning for and conducting work in partnership with UCAS. While this has not seen immediate results in the outcomes of this project, the awareness and strategies developed can be useful to others working in this area. UCAS is now developing ways of working with external projects: in the short-term it is hoped that the benefits of this will be seen in DELIA and EFIFA. The project has also learned some useful lessons about partnership working with institutions: that support is needed at a variety of levels, not just from the top; that working with technology is easier than working with people. There have also been benefits from working closely with other projects in the programme, in terms of sharing emerging outcomes and experience.Further work is needed around support for the fairness agenda: if ePortfolios are to be used, there are implications in maturity of practice, and also in the availability and level of IAG support, especially for WP students and those with accessibility issues, who may still need help in selecting and presenting ePortfolio evidence in the most appropriate way. However the demographic for UK HE admissions is changing, with increasing WP targets and new kinds of learners, including work-based learners and those on 14-19 Diploma programmes. The growing number of learners who will be able to, and expect to, make use of a richer evidence base to support their applications means that further work in this area is imperative. If they are able to use a tool such as the PIFF demonstrator to support multiple applications for education and employment from a single source, or to make an application for work placement once in higher education, issues of transparency, accuracy and verification begin to be addressed. The model of the ‘thin’ ePortfolio, which allows a window to distributed assets for a particular purpose, is being pursued by a number of other projects in the UK and Europe. Initiatives such as MIAP, the new UK national system to support applications for apprenticeships and the regional 14-19 partnerships , many of which are supporting use of a common application form and investigating shared tools and information, are looking to established systems and shared service models, of which the UCAS application

of 40


process is the best-established example. There is rich scope for extending the outputs of PortisHEad in different sectors. To be sustainable and extendable, however, this work must be rooted in common standards. XCRI, HR-XML, IMS and the emerging LEAP 2.0 need to be taken into account in any further development work. More work is needed on the use and impact of ePortfolios on induction and retention. It is recommended that JISC consider a pedagogic study on students who want to use ePortfolio instruments for application, using a small number of institutions or courses with an existing tradition of ePortfolio use, together with HEIs with a similar trend, and that this be continued through to track students through their HE courses and possibly on into employment.

Sandra WinfieldNovember 2008

of 40

archive.alt.ac.ukarchive.alt.ac.uk/.../programmes/elearningcapital/portishe… · web...

Documents