evaluation of academic review in wales · introduction 1 this report outlines the findings of the...

The Quality Assurance Agency for Higher EducationSouthgate HouseSouthgate StreetGloucester GL1 1UB

Tel 01452 557000Fax 01452 557070www.qaa.ac.uk

RG

003 06/03

Evaluation of academic review in Wales 2002

JUNE 2003

Published byQuality Assurance Agency for Higher EducationSouthgate HouseSouthgate StreetGloucester GL1 1UB

Tel 01452 557000Fax 01452 557070Email [email protected] www.qaa.ac.uk

© Quality Assurance Agency for Higher Education 2003

ISBN 1 85824 925 2

All the Agency's publications are available on our web site www.qaa.ac.uk

Printed copies are available from:Linney DirectAdamswayMansfieldNottinghamshire NG18 4FN

Tel 01623 450788Fax 01623 450629Email [email protected]

Contents

Introduction 1

Background to the subject engagements 1

The outcomes of subject engagements 2

Evaluation approach 3

Evaluation results 3

Reports 5

Summary of key issues and recommendations arising from the engagements 6

Conclusion 6

Appendix 1 7

Appendix 2 11

Appendix 3 24

Introduction1 This report outlines the findings of theevaluation of the academic review of subjects inWales, conducted by the Quality AssuranceAgency for Higher Education (the Agency) in2002-03. The reviews, known as subjectengagements, were conducted in fourteen highereducation institutions (HEIs), addressing 19 subjectareas at both undergraduate and postgraduatelevel. As the individual review reports areconfidential to the HEI and the Higher EducationFunding Council for Wales (HEFCW), this reporthas been published to disseminate the results andlearning from the 2002 engagements.

2 This evaluation report begins by setting thecontext for these engagements, the review methodand the outcomes. The evaluation process is brieflydescribed, before outlining the key issues andlearning points from the experiences of participantsin 2002.

3 This report has been written primarily forcolleagues who are involved in higher educationand those who have an interest in the standardsand quality of higher education programmes. The results of the evaluation contribute to thediscussions and consultation that are taking place inthe first part of 2003 in relation to the developmentof the new process of institutional review.

Background to the subject engagements4 The plans for quality assurance in Wales weresuspended in March 2001. This came about whenthe UK-wide arrangements were dissolved and newmethods were considered in England. As the firstHEFCW cycle of quality assessment was aconsiderable distance from the likely implementationof new methods for quality assurance, interimarrangements were needed. The circular WO1/90HEupdated the higher education sector on a range ofissues related to higher education (Appendix 1). Thedocument stated that every institution wouldundertake from one to three engagements at subjectlevel, with the key purpose of assisting institutions todevelop their procedures and processes to reflectAgency mechanisms.

5 The engagements were based on the Handbookfor Academic Review 2000 (the Handbook) but wereintended to be a substantially lighter burden ofassessment on the HEIs. As the aim of theengagements was developmental, the outcomes ofthe reviews are confidential to the HEIs concerned

and to the HEFCW. The Agency developed adocument entitled Academic Review in Wales 2002(Appendix 2). This document outlined a method thatwas greatly restricted in terms of the time required,the judgements made and the overall burden placedon subject providers. The judgements were focusedon academic standards (intended outcomes,curriculum, assessment and student achievement), onstudent progression, including the quality of studentsupport arrangements, and on the quality of thelearning resources supporting the programmes ofstudy. In addition, the reviewers were asked toprovide a commentary on the maintenance andenhancement of the quality of the subject. Theinstitutions were required to prepare a self-evaluationdocument (SED), incorporating programmespecifications, for each review.

6 Teams of three Subject Specialist Reviewers(SSRs) normally carried out the reviews.Management of each review and liaison with theinstitution was the responsibility of a Review Coordinator (CR). The team was normally in theinstitution for a maximum of two and a half days.The reviewers analysed the self-evaluationdocumentation and any other materials providedby the institution off-site, and then normally visitedthe institution for two and a half days.

7 As Agency officers started to work withinstitutions, it became clear that some additionalexplanation of the method of engagement wouldhelp institutional staff. Two briefing seminars wereheld on 26 April and 7 May 2002. HEFCW officersand at least two representatives from eachinstitution attended. The briefings were planned tohelp subject providers understand the requirementsof the engagements. At these seminars, anadditional synoptic paper was distributed(Appendix 3). This addressed operational issues andthe implementation for the processes for theengagements. The participants evaluated bothsessions positively.


page 1

The outcomes of subject engagements

8 The range of subject engagements

Subjects Number of Number of SSRs engagements involved in engagements

Single reviews Computing 5 15Law 2 6Sports Science 2 6Architecture 1 3Theatre Design 1 3Nursing 1 3Archaeology 1 3Occupational Therapy 1 2Business and Management 1 3Social Policy 1 3Aggregate reviews Computing/Electrical Engineering 1 4Business/Accounting 1 4IT/Business and Management 1 3Total 19 58

The outcomes of the subject engagements

Academic Standards Confidence in standards 19

Quality of Learning Opportunities

Student progression Commendable 17*Approved 2Failed 0

Learning resources Commendable 13*Approved 6Failed 0

*including the identification of exemplary features in Student Progression in one institution and exemplaryfeatures in Learning Resources in another.


page 2

Evaluation approach 9 The evaluation is based on the establishedAgency evaluation framework and draws upon anumber of sources:

postal evaluation questionnaires;

visit support monitoring logs: these are recordsmade by Agency officers when attending thereviews;

minutes/notes of meetings including thebriefing seminars;

oral feedback from participants;

the Agency's web folders: these are electronicfiles of the review visits includingdocumentation created by the review team;

the evaluation seminar in January 2003;

review judgements and reports.

10 The evaluation questionnaires were sent to thethree key groups involved in each engagement: theHEIs, including the Institutional Nominee (IN); theCRs and the SSRs. The percentage returned was 74 per cent, 53 per cent, and 72 per centrespectively, with an overall total return of 69 per cent.

11 Most of the responses are given in a four-pointscale ranging from 1 (strongly disagree) to 4 (stronglyagree). The analysis provides insights into the level ofagreement between the three groups and the level ofsatisfaction with the different aspects of the review.The majority of responses were positive, either 3(agree) or 4 (strongly agree), 79 per cent for the HEIs,86 per cent for the CRs and 88 per cent for the SSRs.

Evaluation resultsThe Handbook

12 The participants had four documents forreference: the Handbook for Academic Review, theCircular WO1/90HE, the Academic Review in Wales2002 document and the synoptic paper. Thisnumber of documents increased the likelihood ofstaff, particularly at subject level, becomingconfused and this happened in some institutions.The evaluation data indicated that HEIs would havebenefitted from more preparation and supportfrom the the Agency for writing SEDs.

13 Dissemination from the briefing seminars wasmore effective in some HEIs than others. This hadsome effect on the standard and structure of SEDs. A few of these reflected the process ofsubject review in England 1998-2000. In these

cases, HEI staff had not fully understood the shiftfrom evaluating the quality of student experiencethrough a grading system (1998-2000) to one ofconsideration of standards and quality against self-evaluation and external reference points. Themajority of HEIs considered the seminars helpfuland relevant. The evaluations taken at the end ofboth events confirmed a high level of satisfactionwith the content and approach of the briefings.

Schedule of visits

14 The HEIs were offered three possible modes forthe engagement: an intensive visit of two and a halfdays, with the final half day devoted to meetings toconfirm team judgements which were then conveyedto the institution in an informal oral feedback; or twodays then a later half day for the meetings aboutjudgements; or two and a half days used flexiblywithin a four-week period. The first two approacheshad the advantage of brevity and less disruption forthe institution and the last provided both the subjectstaff and the review team with greater opportunity toreflect upon and consider the information shared.

15 The number of HEIs that chose each of themodes was nine, six and four respectively. MostHEIs preferred a concentrated period of peerreview activity.

Self-evaluation documents

16 These varied in quality and comprehensiveness.The evaluation feedback indicated that institutionswere less than satisfied with the guidelines for thereviews (Appendix 2). This may have been linked tothe multiplicity of documents, some published andsome not. For future methods of review, a singlepublished document, widely available to the HEsector and subject communities, will be preferable.Several institutions took the opportunity to directlyreference key institutional and departmental leveldocuments in the SED. This worked very well inpractice as these references enabled reviewers to finddocumentation and evidence quickly and efficiently.

17 It soon became clear to the Agency that if HEIssupplied additional documentation with the SED,this would help the reviewers to prepare prior tothe visit(s). Where institutions were careful andfocused in their choice of additional material andwhere their SED had clear and direct references tomaterials, the whole process was more efficient andless intrusive.

18 Several HEIs made very good use of informationtechnology to provide reviewers with information,for example, CD-ROM and web-based materials.


page 3

Engagement activity, process issues

19 The analysis of the questionnaires showed thatthe majority of participants were positive andsatisfied with the process. The institutions generallyfound that the review teams were constructive andunderstood the developmental nature of theengagements. The approach of peer review againsta critical and evaluative SED was clearly successful.The feedback indicated that the HEIs found thisprocess more collegial than their early experiencesof external quality assurance.

20 The length of time (2.5 days) for theengagements was sufficient for the review. The CRs and the SSRs expressed a high level ofsatisfaction in the evidence they gathered and thejudgements they made. Institutions were satisfiedwith the communication between the review teamand the HEI.

21 The HEIs provided the reviewers with materialsthat were adequate and appropriate to inform thejudgements and the commentary on themaintenance and enhancement of quality. Thematerial was particularly helpful if it reflected theSED and was focused in nature. This element ofcontrol and restraint was also important in thesample of student work provided by the institution.HEIs and review teams worked best when thesample was relevant and contained no moreexamples of work than were necessary todemonstrate the process of assessment, and thestudents' achievement of the required standard forthe award.

Review teams

22 Review teams were normally composed ofthree reviewers in each subject area, selected toreflect the broad Joint Academic Coding System(JACs) codes of the subject. Each institutionidentified the key aspects that they wereconcerned that the review team should reflect andhad an opportunity to comment on the proposedteam membership.

23 Institutions varied in the extent to which theyrequested that review teams had an awareness ofhigher education in Wales. Wherever possible, atleast one member of the team had direct experienceof working in an HEI in Wales, or indirect experiencesuch as validation panel member or externalexaminer. Many institutions thought it more importantthat the ethos of the HEI was reflected, for example,research-oriented institution, rather than directexperience of providing higher education in Wales.

Review Coordinators

24 CRs manage reviews and liaise with theinstitution, they do not act as subject specialists.The Agency introduced CRs to the key issues ofengagements in Wales at the twice-yearlyconferences which the Agency arranges. All CRsreceived the three guidance papers (Appendix1.2.3). An Agency Assistant Director and anadministrator were named contacts for theengagements. Fifteen CRs managed the 19 reviews.Their coordination of the reviews was generally wellreceived by institutional staff and reviewers, butthere were some concerns about the consistency ofthe conduct of the engagements. This wasparticularly the case where more than oneengagement took place in an HEI and where eachCR adopted a different approach. In each case, theguidance in the Handbook was followed, but theinterpretations of the scope of the role varied.

Institutional Nominee

25 This role was introduced in the later stages ofplanning the engagements (Annex D, Appendix 2).The role was optional for the HEI. The personnominated was expected to have a thoroughknowledge of their institution, its protocols andregulations as well as extensive knowledge andexperience of HE and quality assurance. TheAgency provided written briefing materials for INsand, in addition, the majority of INs attended thebriefing seminars. All HEIs chose to have an IN, butthe extent of their involvement in the engagementand particularly at judgement meetings varied. Thiswas partly due to the interpretation of the role bysome CRs. This was sometimes a reflection of theiranxiety about that level of involvement of amember of the HEI in the process of theengagement. This individual interpretation of sucha role needs careful risk assessment andmanagement for future review processes in Wales.

26 The analysis of the questionnaires and otherfeedback found that the IN role was considered avery positive aspect of this activity and well receivedby review teams and institutions. In the briefingseminars, participants were not sure that INs shouldbe present at the meetings when reviewers werediscussing and confirming judgements. The mainreasons given concerned the potential impact of theIN on the process of making judgements. Secondly,some participants considered that the IN would beunder considerable pressure from colleagues in theirinstitution both before and after judgements,particularly if the judgements were negative. Theevaluation shows that where the HEIs and thereview teams followed the written guidance


page 4

provided by the Agency, the predicted problems ofthe INs' attendance at judgement meetings werenot realised.

Reports27 In all 19 engagements there was confidence instandards. There were 17 judgements ofcommendable for student progression and twoapproved. There were 12 judgements ofcommendable for learning resources. The reviewersalso noted two exemplary features, one in studentprogression and one in learning resources, each ata different institution. As the subjects andprogrammes reviewed were so diverse, noconclusions can be reached about the standard orstate of subjects.

28 The comments below are drawn from ananalysis of the reports:

Academic standards

29 The intended learning outcomes specified bysubject staff were usually clear and relevant,informed by external reference points. The outcomeswere generally well communicated to students andother stakeholders.

30 Review teams found that curricula are generallycurrent and supported by research and otherscholarly activities of staff. The programmes wereconsidered to have appropriate content for thelevel of the awards.

31 Reviewers found that assessments are wellmatched to outcomes, with assessment strategiesand activities that are rigorous and secure. Somereview teams commented that feedback to studentscould be improved and/or be more consistent interms of rigour and helpfulness to the students.Several reports also commented that internalmoderation strategies, including second marking,could be improved. Where good practice wasevident in such strategies, this was of a particularlyhigh standard.

32 Student achievement demonstrated added-value for many students. This aspect ofstandards was generally good, being at anappropriate level for most programmes.

Student progression

33 There was wide spread evidence of goodpractice of supporting students from their entry tothe completion of their programmes of study and

beyond. This positive outcome concerns bothacademic and pastoral support. Students reportedthat staff are consistently accessible, positive and helpful.

34 A clear message from the reports on theengagements is that HEIs in Wales are activelyengaged in widening participation. The reviewersfound that staff understood the importance of support tailored to the needs of students fromnon-traditional backgrounds and met these needs well.

35 The reviewers raised questions about theeffectiveness of the management information systemsin several HEIs. Reviewers found that HEIs need toimprove the quality and breadth of the statisticalinformation on students, and to enable both subjectstaff and the institutional administrative staff to shareand use the information more effectively. This abilityto provide clear and accurate information is importantfor the institutions' own management of standardsand quality, as well as for future review processes.

Learning resources

36 The reviewers commented positively on staffqualifications and expertise. They reported thatacademic, administrative and other support staffare suitably qualified and knowledgeable. Generally,HEIs are able to provide support for staffdevelopment, promoting professional as well aseducational development.

37 There are mixed messages about the learningresources of libraries and computing in the HEIs.While all institutions have a challenge to maintainand improve the quantity and quality of librarystock, some institutions succeeded while others didnot meet their students' needs as well. There wasno discernable pattern within subjects orinstitutions as to why the level and quality ofprovision was so variable. Information andcommunication technology also varied, both forthe students' direct use and when supportingteaching and learning processes.

38 The reviewers clearly identified pressure onstaff and technical resources in several HEIs. They attributed this pressure to the increasingnumbers of students in many subjects, and thelack of a corresponding increase in staffing andother resources.


page 5

Summary of key issues andrecommendations arising from theengagements39 The engagements in Wales consistentlyidentified many positive messages about thestandards and quality of higher educationprogrammes in Wales. All subjects receivedjudgements of confidence in standards andcommendable or approved in the quality of thelearning opportunities.

40 The programmes reviewed demonstratedconsiderable strengths in relation to learningoutcomes, curriculum, assessment andachievement, and particularly to wideningparticipation. Assessment remains the area ofstandards where the reviewers identified the mostscope for improvement. In particular, the reviewershighlighted the need for improvements in secondmarking or internal moderation.

41 The reviewers found consistently very goodlevels of academic and pastoral support forstudents. Increasing student numbers continues tobe a challenge for some HEIs, especially wherethere has not been a corresponding increase instaffing and other learning resources. In thesecases, staff are under pressure to sustain this verygood level of academic and pastoral support.

42 The overall commentary on maintenance andenhancement of quality and standards was positiveand the HEIs were well aware of their responsibilities.Some institutions are advised to review theirmanagement information systems to ensure thatprogramme and subject staff as well as administrativestaff receive information which helps and supportsthem to achieve their institution's mission.

43 The reviewers find the use of references inSEDs very helpful. Where HEIs also cross-referencedto documents available on the institutions' websites, CD-ROM and intranet or in the workroom forreviewers, the engagements with their externalpeers worked particularly well.

44 The role of the IN was welcomed by mostreviewers and HEIs. Many recommended that thisor a similar role should be included in futuremethods of review.

45 Preparation for the method of InstitutionalReview in Wales needs to be thorough and tailoredto meet the needs of HEIs in Wales. The participantsat the evaluation seminar recommended: workshopsto assist with the preparation of SEDs; moreguidance on documentation to be provided by the

HEI both before the review and during the process;and more preparation for the role of the INincluding training shared by reviewers and INs, withthe benefit of ensuring a greater appreciation of theWales higher education context during training.

Conclusion46 The subject engagements in Wales wereapproached with a mix of some reservations andsome enthusiasm by the HEIs. The aims for theseactivities were realised and achieved; the HEFCW,the Agency and the HEIs worked together to ensurethat the subject engagements in Wales were apositive experience. This report has highlighted thekey issues and learning points arising from thereviews and evaluation. There are lessons which areapplicable to the new method of InstitutionalReview under development.

47 The successful completion of the process is ingreat part due to the hard work and enthusiasm ofthe HEIs' subject staff and students, and of thereview teams.


page 6

Appendix 1

Academic Review in Wales, 2002

To: Heads of higher education institutions in Wales

Summary: This Circular updates the sector on a range of issues related to quality assurance in higher education.

Reference: W01/90HE

Publication date: 30 October 2001

Response by: No response required.

Further information: Celia Hunte-mail: [email protected]: 029 20 682224

Address: HEFCWLinden CourtThe OrchardsIlex CloseLlanishenCardiff CF14 5DZ

Appendix 1

page 7

Introduction1 This Circular updates the sector on a range ofissues related to quality assurance in higher education.

Background2 The HEFCW's plans for taking forward theQuality Assurance Agency (Agency's) UK wideproposals for quality assurance within Wales weresuspended in March 2001. This followed theannouncement of new arrangements to beimplemented in England and recognised thepossible impact on quality assurance within highereducation (HE) in Wales.

3 Subsequently, a consultation paper waspublished related to arrangements to be takenforward in England from September 2002 onwards.This document is available on the Agency websitewww.qaa.ac.uk/public/newmethod/newmethod.htm

4 Given the lengthy gap between thecompletion of the first HEFCW five-year cycle ofquality assessment in Wales and the likelyimplementation of new arrangements, it wasnecessary to address interim arrangements for2001/02. This circular gives some informationabout those interim arrangements and also reportsback on a number of related issues which havebeen delayed since March 2001.

Interim arrangements for 20025 After some discussion, and following thepublication of Circular W01/65HE (27 July 2001),the HEFCW has now formally agreed with HigherEducation Wales (HEW) interim arrangements forthe quality assurance of HE in Wales in 2001/02.

These interim arrangements have several purposes:they

meet the requirement of the NationalAssembly for Wales for an end to thecontinuing delay in the implementation of anew method of assuring quality and standardsin higher education provision in Wales;

assist institutions in re-engaging with processesand procedures developed by the Agency,following wide consultation with institutions, asset out in the Agency Handbook for AcademicReview;

and, most importantly, they will

assist institutions in developing their processesand procedures to fit the Agency mechanisms.

They should be seen in the context of assistingthe maximisation of quality enhancementrather than in the negative sense of theimposition of quality control.

6 Arrangements have been agreed as follows:

That each institution should undertake one tothree engagements at subject level with theAgency in 2001/02. The number ofengagements will depend upon the size of theinstitution and will be agreed between theinstitution, the Council and the Agency;

These engagements should be based upon(but not be bound by) the subjects proposedby institutions for scrutiny by the Agencyduring 2001/02 in the Agency's scope andpreference survey;

The arrangements for the engagements will bebased upon the processes and procedures setout in the Agency Handbook for AcademicReview but will see a substantially lighterburden of assessment on institutions during2001/02. They will not be full subject reviewsas previously conceived;

In a revision to the circular W01/65HE, theperiod during which these engagementsshould take place has been extended toDecember 2002 (ie they should take placeduring the calendar year 2002);

Because the aim of these engagements isdevelopmental (viz; to enable institutions to re-familiarise themselves with Agency practicesand expectations), the outcomes will beconfidential and made available only to theinstitutions concerned and to the HEFCW.

7 Attached to this circular is a description of themethod to be adopted by the Agency, entitledAcademic Review in Wales 2002. Any queries relatingto this method should be addressed to the Agency.Any institution wishing to opt for the full Agencysubject review method is free to do so.

8 Within the overall position set out above, theAgency will aim to be flexible in its dealings withinstitutions, in terms of both the timetable forassessments and the structure of the arrangements.Each individual higher education institution will besent a letter shortly which sets out the subjectscurrently agreed with the Agency for inclusion inthe new arrangements and any subsidiary querieswhich will need to be addressed, for example, theextent of Welsh Medium provision being putforward for assessment.


page 8

Arrangements after December 20029 The HEFCW will await the outcome of theconsultation in England and the development ofnew arrangements, planned for introduction inSeptember 2002 or soon after. It has encouragedinstitutions in Wales to respond to the consultation.It will then determine the approach to be adoptedin Wales, in consultation with the sector.Institutions responding to the consultation havebeen asked to copy their responses to the HEFCWand these will be taken into account in thedevelopment of consultation proposals in Wales. Itis expected that details of the arrangements forEngland will be available before the end ofDecember 2001 and HEFCW officers will beinvolved in the development of these proposalswhere possible.

Other issues10 A number of related issues have been delayedsince March 2001.

a The assessment of higher education infurther education (FE) institutions

11 A consultation was conducted in 2000 on theprocesses to be adopted by the Council for theassessment of HE in FE institutions. The outcome ofthis consultation was overtaken by events just priorto publication. This will be published in the nearfuture, updated to reflect the establishment of thenew National Council for Education and Trainingfor Wales.

12 However, given the delays in the publication ofthis outcome, it is not expected that FE providers ofHE courses will wish to be involved in the interimarrangements set out in this circular for 2001/02.They should, however, plan for involvement in thenew Agency framework from 2002/03, inconsultation with the Agency. It is likely that thearrangements for the HE sector, arising from thecurrent consultation with in England, will drawheavily upon the Agency's Institutional Reviewprocess for higher education institutions. Given thatFE providers will not be involved in institutionalreview within Agency procedures, it is expectedthat the arrangements for subject review, as set outin the Agency's Handbook for Academic Review,will continue to apply to HE provision in FEinstitutions for the remainder of the current cycle.

13 However, formal liaison has been establishedbetween the HEFCW, Agency and Estyn and thescope for joint working is under consideration.

HEFCW officers will continue to encourage thereduction of duplication between the Agency andEstyn on these matters wherever possible.

b The inclusion of a institutional response inquality assurance Reports

14 The Council conducted a brief informalconsultation with HE institutions on the above in2000. The purpose of the consultation was toascertain whether institutions would prefer to retainthe option, utilised by HEFCW in the first cycle ofassessment, whereby they were permitted to includea formal response to the interim teaching qualityassessment report in the final version to be published.

15 Nine institutions responded and all indicatedthat the option should be retained. Reasons givenfor this included that it allowed institutions tocomment on significant action taken as a result ofthe review; it enabled institutions to demonstratethe responsiveness of their quality systems and theircommitment to continuous improvement; itpromoted quality enhancement; it improvedinformation available to potential students; and ithelped to close the quality loop. Discussions withAgency officers had finalized arrangements for theinclusion of such a response, based upon an actionplan developed by the institution in answer toissues raised in the published report.

16 These matters were, of course, held up by thedelay in implementation of the new Agencyframework. This option will not be available as partof the interim arrangements in 2001/02, since it isnot intended that reports should be made public.However, given the unanimous response in favourof this option, the Council will consider this matteragain when developing the arrangements to beimplemented in Wales from 2002/3.

c The rewarding of the audit

17 Proposals for new arrangements for thereward of quality were discussed by the Council atits meeting in May 2001, taking account ofrecommendations from its Quality AssessmentCommittee. These will involve a focus upon theenhancement of quality, via the submission ofexpanded Learning and Teaching Strategies, ratherthan the direct reward of 'excellent' outcomes. Aconsultation is planned regarding these proposals,which have also been temporarily suspended, giventhe uncertainty surrounding the new arrangements.

18 A paper detailing these proposals will becirculated in the near future and institutions will beinvited to comment on the proposals put forward.

Appendix 1

page 9

d Quality assessment committee

19 Simultaneously, the Council consideredproposals for a restructuring of its QualityAssessment Committee (QAC), which has astatutory role in advising it on quality assessmentmatters. The proposals considered included theintroduction of a new larger, and more broadlyfocussed, Learning and Teaching Committee, ofwhich members of the the QAC would form a sub-set, whilst retaining their separate status forstatutory purposes.

e Agency liaison in Wales

20 The Council has agreed the establishment of aLiaison Officer for Wales, currently Mike Laugharne,within the terms of its Service Level Agreement withthe Agency. The arrangements also include fundingfor a capacity for dealing with Welsh Mediumenquiries. Institutions are encouraged to make use ofthe Liaison Officer whenever queries arise regardingAgency processes and procedures. It is noted that,despite the uncertainties regarding quality assuranceprocesses and procedures in Wales over the past fewyears, very few institutional queries have beenpassed either to Council or Agency officers.

Queries21 Any queries regarding the content of thiscircular should be directed to Ms Celia Hunt, e-mail: [email protected], tel: 02920 682224.


page 10

Appendix 2

Academic Review in Wales, 2002

'Engagements' with Welsh Higher Education Institutions

Contents

Introduction

The Method

The Evidence

The Judgements

The Reports

Annex A Aide-Mémoire for Academic Review in Wales

Introduction

Section i Aims and Outcomes

Section ii Curricula

Section iii Assessment

Section iv Student Progression

Section v Learning Resources

Annex B Documentation for Academic Review in Wales

Institutional documents

Student work

Annex C Agenda for meeting with students

Introduction

The curriculum and intended learning outcomes

Assessment and achievement

Student progression

Learning resources

Annex D Institutional nominee

Introduction

Role of the nominee

Activities preceding engagements

Activities during engagements

Confidentiality

Appendix 2

page 11

Subject Level 'Engagements'

Introduction

1 This booklet describes a method forconducting subject level 'engagements' in Walesduring the calendar year 2002. It should be used inconjunction with the Handbook for AcademicReview published previously by the Agency. Theduration of the overall review schedule has beenagreed with the Higher Education Funding Councilfor Wales (HEFCW). The precise timing of reviewswill be subject to negotiation between the QualityAssurance Agency (Agency) and the individualWelsh higher education institutions.

2 Although the method described below is basedon the same principles and practices as thosealready described in the Handbook for AcademicReview, it is greatly restricted in terms of the timerequired, the judgements made and the overallburden placed on subject providers. It is mostcertainly not subject review by another name anddoes not include judgements on the quality ofteaching and learning. However, the units of reviewremain the subject categories already devised bythe Agency and outlined in Annex K of theHandbook. Subjects will not normally beaggregated for the purposes of this schedule.

3 A self-evaluation document, incorporatingprogramme specifications, will be required for eachreview and should be prepared in accordance withthe guidance in paragraphs 24-26, 28-30 andAnnex C of the Handbook. Institutions will also findit helpful to use Annex E of the Handbook (the Aide-Mémoire for subject review) for the purposes ofconstructing a self-evaluation. Even though thisacademic review method does not permit reviewersto scrutinise all aspects of provision directly, it isexpected that institutions will provide informationabout quality and standards in full, not leastbecause it is difficult for them to give an adequateaccount of these matters without covering allaspects. It is intended that the self-evaluationdocument will form the basis for the report of therestricted external review, albeit modified by thereviewers' observations.

4 The judgements to be made will be focused onacademic standards (intended outcomes,curriculum, assessment and student achievement),on student progression, including the quality of thestudent support arrangements and on the qualityof the resources supporting the programmes ofstudy. The Aide-Mémoire at Annex A in this bookletprovides guidance on the questions likely to lead to

the collection of the evidence necessary to makethese judgements.

The Method

5 Subject providers should set out clearly thebroad aims of their programmes in theirprogramme specifications and provide a statementof the overall aims of the provision at the start ofthe self-evaluation document. Such statementsshould indicate, in general terms, what the subjectprovider is seeking to achieve and how these aimsrelate to external indicators (such as thequalifications framework, subject benchmarkstatements, professional body requirements, andemployer expectations). These broad aims willprovide the context in which the review takes place.

6 These reviews will normally be carried out bysmall teams of three subject specialists, whose mainresponsibility is to read, analyse and test the self-evaluation produced by the institution, and togather whatever further evidence they need tomake the judgements described later in thisbooklet. Management of each review and liaisonwith the institution is the responsibility of a reviewcoordinator.

7 The review team will normally spend amaximum of two days visiting the institution,preceded by an off-site desk-based analysis of theself-evaluation document and any other materialssupplied by the institution. Such materials normallycomprise only a prospectus and location maps,although it is possible that a review coordinatormay request that copies of the student handbookare supplied in advance of the visits to theinstitution. Documents needed during visits will berequested at the time. If further information isrequired following the visit to enable the report tobe completed, this will be requested by telephoneor email. The two days' direct observation may beconsecutive or separated by a period for reflectionby the reviewers, according to the preference ofthe institution. In either event, the review must becompleted within a period of four weeks.

8 The review coordinator will make initial contactwith the subject provider to agree an outlineprogramme for the review and to establish the:

pattern of visits;

range of student work which can be madeavailable for scrutiny, and the extent to whichthis constitutes a representative sample ofstudent achievement in the subject;


page 12

nature of relevant documentation held by theinstitution and its availability for scrutiny byreviewers;

timing of any related visits by professional orstatutory bodies;

probable agenda and timing of meetings withacademic staff, students and former students;

other practical arrangements for the review.

9 The views of students are important, butshould be treated as one source of evidence amongseveral. Meetings with students enable reviewers toestablish the students':

understanding of the overall aims andintended learning outcomes;

views on the curriculum and the assessmentmethods used;

views on academic support and the resourcesavailable;

views on feedback and representationarrangements.

Arrangements for meetings with students are setout in Annex C of this booklet.

Where appropriate, and if time permits, reviewersmay also see the views of former students, theiremployers, and representatives from relevantindustries or professions.

The Evidence

10 Reviewers will seek to establish that:

intended learning outcomes are clearlyexpressed by the subject provider, and reflectappropriately the qualifications framework,relevant subject benchmark statements and theoverall aims of the programme;

there is effective communication to staff andstudents, so that learners and teachers knowwhat is expected of them;

curricular content supports the intendedoutcomes, and assessments measureappropriately their achievement;

11 Reviewers will seek to establish, by reference tosubject benchmark statements where appropriate,that the design and content of the curriculumfacilitate:

acquisition of knowledge and understanding;

acquisition of cognitive skills;

acquisition of subject-specific skills, includingpractical and professional skills;

acquisition of transferable skills;

progression to employment and/or furtherstudy

12 Reviewers will seek to establish that:

there is an appropriate overall strategy foracademic support;

there are effective arrangements for admissionand induction;

learning is promoted by academic guidance,effective feedback and careful supervision;

students' progression towards successfulcompletion of their programmes is facilitated.

13 Finally, reviewers will also seek to establish that:

the collective expertise of the academic,technical and administrative staff is suitable foreffective delivery of the curricula;

there is an overall strategy for the deploymentof learning resources;

suitable accommodation, equipment and otherlearning resources are available.

The Judgements

14 A single, threshold judgement is made aboutacademic standards. Having regard to all thematters listed below, reviewers will decide whetherthey have confidence in the academic standards ofthe provision under review. A 'confidence'judgement will be made if reviewers are satisfiedboth with current standards, and with the prospectof those standards being maintained in the future.If standards are being achieved, but there is doubtabout the ability of the institution to maintain theminto the future, reviewers will indicate 'limitedconfidence' in the academic standards achieved. If,in relation to any of the matters listed below,reviewers feel that arrangements are inadequate toenable standards to be achieved or demonstrated,then their overall judgement will be that they have'no confidence' in the academic standards of theprovision under review.

15 Reviewers will judge, for each programme,whether there are clear learning outcomes, whichappropriately reflect applicable subject benchmarkstatements and the level of the award. Subjectbenchmark statements represent generalexpectations about standards in an academic

Appendix 2

page 13

discipline, particularly in relation to intellectualdemand and challenge. The qualificationsframework sets expectations for awards at a givenlevel more generally. Reference points are therebyprovided to assist reviewers in determining whetherprovision is meeting the standards expected by theacademic community generally, for awards of aparticular type and level. If the intended learningoutcomes are found not to match thoseexpectations, it would be unlikely that reviewerscould have confidence in the standards of theprovision. An example of potential failure would beif a postgraduate programme has learningoutcomes set at an undergraduate level only.

16 Reviewers will judge whether the content anddesign of the curriculum are effective in achievingthe intended programme outcomes. It is thecurriculum that ensures that students are able tomeet the intended outcomes of the programme.Providers should be able to demonstrate how eachoutcome is supported by the curriculum.'Curriculum' for this purpose includes both thecontent necessary to develop understanding andthe acquisition of knowledge, and the opportunitiesto develop practical skills and abilities where theseare stated as intended outcomes. If significantintended learning outcomes are found to beunsupported by the curriculum, it would beunlikely that reviewers could have confidence in thestandards of the provision.

17 Reviewers will judge whether the curriculumcontent is appropriate to each stage of theprogramme, and to the level of the award.Providers should be able to demonstrate how thedesign of the curriculum secures academic andintellectual progression by imposing increasingdemands on the learner, over time, in terms of theacquisition of knowledge and skills, the capacity forconceptualisation, and increasing autonomy inlearning. Reviewers will have regard to theguidance on programme design in the section ofthe Code of practice on programme approval,monitoring and review.

18 Reviewers will judge whether assessment isdesigned appropriately to measure achievement ofthe intended outcomes. Providers should be able todemonstrate that achievement of intendedoutcomes is assessed, and that, in each case, theassessment method selected is appropriate to thenature of the intended outcome. There must alsobe confidence in the security and integrity of theassessment process, with appropriate involvementof external examiners. An assessment strategy

should also have a formative function, providingstudents with prompt feedback, and assisting themin the development of their intellectual skills. Thereshould be clear and appropriate criteria for differentclasses of performance, and these criteria should becommunicated effectively to students. If significantintended learning outcomes appear not to beassessed, or if there are serious doubts about theintegrity of the assessment procedures, it would beunlikely that reviewers could have confidence in thestandards of the provision. Reviewers will haveregard to the section of the Code of practice onassessment of students.

19 Reviewers will judge whether studentachievement matches the intended outcomes andlevel of the award. Reviewers will consider externalexaminers' reports from the three years prior to thereview, and will themselves sample student work.The balance between reliance upon the reports ofexternal examiners and direct sampling of studentwork will depend on the confidence that reviewershave in the external examining arrangements ofthe institution. Regard will be had to the section ofthe Code of practice on external examining.

20 Where programmes are offered at more thanone level, separate judgements will be made inrespect of each level, if there are significantdifferences between them. In all cases, reports willcontain a narrative commentary on strengths andweaknesses in relation to each aspect of thestandards judgement.

21 Judgements about student progression andlearning resources are made against the broad aimsof the provision and the intended learningoutcomes of the programmes. They normally coverall provision within the scope of the review, but ifperformance is significantly different in a particularprogramme or at a particular level, separatejudgements will be made. These judgements willplace student progression and learning resources inone of three categories, 'failing', 'approved' or'commendable', made on the following basis:

'failing' provision makes a less than adequatecontribution to the achievement of theintended outcomes. Significant improvement isrequired urgently if the provision is to becomeat least adequate;

'approved' provision enables the intendedoutcomes to be achieved, but improvement isneeded to overcome weaknesses. The report ofthe review will normally include a statementcontaining the phrase 'approved, but ….',


page 14

which will set out the areas whereimprovement is needed;

'commendable' provision contributessubstantially to the achievement of theintended outcomes, with most elementsdemonstrating good practice.

22 In making judgements about studentprogression, reviewers will judge whether anappropriate strategy for the academic support ofstudents is in place. Such support should includewritten guidance consistent with the student profileand the overall aims of the provision. It commenceswith recruitment, admission and inductionarrangements; these should be thoroughlyunderstood by academic staff and should betransparent to applicants and to the studentsrecruited. Throughout their programmes of study,students' learning should be facilitated by properacademic guidance, prompt feedback onperformance and effective supervision. Reviewerswill consider these features in general and in thecontext of the arrangements made for academictutorial support. Reviewers will judge whether thesesupport arrangements are understood by staff andstudents, and whether they are effective infacilitating student progression towards successfulcompletion of their programmes.

23 In order to make the judgement about studentprogression, reviewers will scrutinise subject orprogramme handbooks, student questionnaires orsummaries thereof, internal review documents,recruitment data and progression data. They willdiscuss related matters with staff and with students.

24 In making judgements about learningresources, reviewers will judge whether theminimum resource necessary to deliver eachprogramme is available. Consideration will be givento the use of equipment (including IT),accommodation (including laboratories), and thelibrary (including electronic resources). Effectiveutilisation of academic, technical and administrativestaff will be considered, as will the matching of thequalifications, experience and expertise of teachingstaff to the requirements of the programmes.Reviewers will judge whether or not resources areadequate to support achievement of the intendedlearning outcomes of the programme under review,and will look for clear evidence of an overallstrategy which ensures effective deployment. Theywill expect to see evidence that the teachingaccommodation, the book and periodical stocksand the equipment and IT facilities bring abouteffective student learning.

The Reports

25 At the end of each 'engagement', a report thatdescribes the findings of the team of reviewers willbe produced. It is the main documented outcomeof the review process and provides the mainfeedback to the institution and its subject provider.For the purposes of the 2002 reviews in Wales, thereports will not be published and will remainconfidential to the institution, HEFCW and Agency.The reports will include:

a brief description of the review method;

the overall aims of the subject provider asstated in the self-evaluation document;

an evaluation of the academic standardsachieved and of student progression andlearning resources;

the conclusions reached and the judgementsmade;

a one-page summary of the main conclusions.

26 Reporting on standards takes the form of anarrative commentary which addresses strengthsand weaknesses by reference (where appropriate)to the relevant sections of the Code of practice,subject benchmark statements, and thequalifications framework, and leads to the overallthreshold judgement. The narrative may identifymatters for particular commendation or matters ofconcern. Reporting on student progression andlearning resources also takes the form of narrativecommentary on strengths and weaknesses, andleads to graded judgements about their quality.

27 Draft reports are sent to the institution forcomment on matters of factual accuracy. Feedbackon the review is provided to the institution throughthe draft report.

Appendix 2

page 15

Annex A

Aide-Mémoire for Academic Review in Wales

Introduction

1 This Aide-Mémoire consists of questions andprompts to assist academic reviewers andinstitutions. It may be used in:

analysis of the self-evaluation prior to thereview;

collection of evidence during the review;

preparation and compilation of the report ofthe review.

2 The Aide-Mémoire covers the main features ofthe review process, but it is neither prescriptive norexhaustive. The provider's self-evaluation, thestatement of aims, and the intended outcomes ofprogrammes may all raise issues peculiar to theprovision under scrutiny.

3 Specific prompts for reviewers are set outunder a series of headings. The process of reviewfocuses on the setting of academic standards bythe subject provider, student progression andstudent achievement and the quality of thelearning resources provided. The Aide-Mémoireprovides questions and prompts about:

aims and outcomes;

curricula;

assessment;

student progression;

learning resources.

4 Key points of reference for reviewers willinclude the relevant sections of the Code of practice,the qualifications framework, relevant subjectbenchmark statements, and the overall aims of thesubject provider. Notice should also be taken of therequirements of professional and statutory bodiesin respect of programmes that they accredit.

5 The Aide-Mémoire is divided into sections thathelp to set the parameters for the review. Eachsection comprises:

a set of questions, to gather information;

the key issues for evaluation;

an indication of likely sources of information;

an indication of the types of activity likely to beundertaken during a review;

the judgements that should be made.

Section i - Aims and outcomes

Evaluation of the intended learning outcomesin relation to external reference points and tothe broad aims of the provision

6 Reviewers should ask:

what are the intended learning outcomes for aprogramme?

how do they relate to external reference pointsincluding relevant subject benchmarkstatements, the qualifications framework andany professional body requirements?

how do they relate to the overall aims of theprovision as stated by the subject provider?

are they appropriate to the aims?

They should then evaluate the intended learningoutcomes against relevant external reference pointsand against the aims of the provision as describedin the self-evaluation.

Potential sources of information will include theself-evaluation (and its appended programmespecifications), curricular documents, subjectbenchmark statements, the qualificationsframework and details of professional bodyrequirements.

Review activities may include an analysis ofprogramme content and benchmark statements,discussions with members of the teaching staff.

7 As a result of these activities reviewers shouldbe able to judge:

whether the intended learning outcomes areclearly stated;

whether they reflect appropriately relevantbenchmark statements, other externalreferences, and the overall aims of theprovision.

The means by which the intended outcomesare communicated to students, staff andexternal examiners


how are the intended outcomes of aprogramme and its constituent parts

Annex A

page 16

communicated to staff, students and externalexaminers?

do the students know what is expected of them?

They should then evaluate the way in which subjectproviders convey their expectations to staff,students and external examiners.

Sources of information will include programme orsubject handbooks and curricular documents suchas module or unit guides.

Review activities may include discussions withteaching teams, students and external examiners.

The main outcomes should be judgements on theadequacy of arrangements within the subject forcommunicating intended learning outcomes.

Section ii - Curricula

Evaluation of the means by which the subjectprovider creates the conditions for achievementof the intended learning outcomes


do the design and content of the curriculaencourage achievement of the intendedlearning outcomes in terms of knowledge andunderstanding, cognitive skills, subject specificskills (including practical/professional skills),transferable skills, progression to employmentand/or further study, and personal development?

They should then evaluate the design and contentof the curriculum for each programme in relationto its potential for enabling students to achieve theintended learning outcomes.

Sources of information will include subject orprogramme handbooks and curricular documents,such as module or unit guides, practical orplacement handbooks, and further study andemployment statistics.

Review activities may include evaluation of curriculardocuments and discussions with staff and students.

The section of the Code of practice dealing withprogramme approval, monitoring and review willprovide an important point of reference.

As a result of these activities reviewers should beable to judge whether the intended learningoutcomes are adequately supported by the curricula.


Is there evidence that curricular content anddesign is informed by recent developments intechniques of teaching and learning, bycurrent research and scholarship, and by anychanges in relevant occupational orprofessional requirements?

They should then evaluate whether the curriculumis adequately informed by such developments.

Sources of information will include subject orprogramme handbooks, validation or re-validationdocuments, and professional and/or statutory bodyaccreditation reports.

Review activities may include discussions with staffand external examiners, discussions with professionaland/or statutory bodies, and discussions withemployers (where relevant and possible).

As a result of these activities reviewers should beable to assess the currency of the curricula.

Section iii - Assessment

Evaluation of the assessment process and thestandard it demonstrates


does the assessment process enable learners todemonstrate achievement of the intendedoutcomes?

are there criteria that enable internal andexternal examiners to distinguish betweendifferent categories of achievement?

can there be full confidence in the security andintegrity of assessment procedures?

does the assessment strategy have an adequateformative function in developing student abilities?

They should then evaluate whether the overallassessment process and the particular assessmentinstruments chosen are appropriate and effective.

Sources of information will include assessmentcriteria and guidance to markers, externalexaminers' reports and procedures for monitoringand recording achievement.

Review activities may include discussions withteaching teams, students and external examinersand the analysis of the methods for recordingprogress and achievement.

Appendix 2

page 17

The sections of the Code of practice dealing withassessment of students and external examining willbe important points of reference.

As a result of these activities, reviewers should beable to judge whether assessment processes canadequately measure achievement of the intendedprogramme outcomes.


what evidence is there that the standardsachieved by learners meet the minimumexpectations for the award, as measuredagainst relevant subject benchmarks and thequalifications framework?

They should then evaluate whether studentachievement meets such expectations.

Sources of information will include externalexaminers' reports, examination board minutes,and samples of student work.

Review activities may include discussions withteaching teams and external examiners, andobservation of examination boards where possible.

Relevant subject benchmark statements and thelevel descriptors of the qualifications framework willbe important points of reference.

As a result of these activities, reviewers should beable to judge whether appropriate standards arebeing achieved.

Section iv - Student Progression

Evaluation of the quality of the learningopportunities offered by the subject provider:

student progression and academic support


is there an appropriate overall strategy foracademic support, including written guidance,which is consistent with the student profile andthe overall aims of the provision?

are there effective arrangements for admissionand induction which are generally understoodby staff and students?

how effectively is learning facilitated byacademic guidance, feedback and supervisoryarrangements?

are the arrangements for academic tutorialsupport clear and generally understood by staffand students?

They should then evaluate whether thearrangements in place are effective in facilitatingstudent progression towards successful completionof their programmes.

Sources of information will include subject orprogramme handbooks, student questionnaires,internal review documents, recruitment data, andprogression data.

Review activities will include discussions withadmissions staff, discussions with teaching staff,and discussions with students.

As a result of these activities, reviewers should beable to judge the effectiveness of the recruitmentarrangements, the strategy for student support andthe progression of students.

Section v - Learning Resources

Evaluation of the quality of the learningopportunities offered by the subject provider:learning resources and their deployment


is the collective expertise of the academic staffsuitable and available for effective delivery ofthe curricula, for the overall teaching, learningand assessment strategy, and for theachievement of the intended learningoutcomes?

are appropriate staff developmentopportunities available?

is appropriate technical and administrativesupport available?

They should then evaluate the effectiveness of thedeployment of academic and support staff insupport of the intended learning outcomes.

Sources of information will include staff CVs,internal review documents, external examiners'reports, and staff development documents.

Review activities may include discussions withteaching teams, and discussions with students.

As a result of these activities reviewers should beable to judge whether there are appropriatelyqualified staff who are contributing effectively toachievement of the intended outcomes.


is there an overall strategy for the deploymentof learning resources?

Annex A

page 18

how effectively is learning facilitated in termsof the provision of resources?

is suitable teaching and learningaccommodation available?

are the subject book and periodical stocksappropriate and accessible?

are suitable equipment and appropriate ITfacilities available to learners?

They should then evaluate the appropriateness ofthe learning resources available.

Sources of information will include equipment lists,library stocks, and internal review documents.

Review activities may include direct observation ofaccommodation and equipment, discussions withstaff, and discussions with students.

As a result of these activities, reviewers should beable to judge how effectively the learning resourcesare deployed in support of the intended outcomes.

Appendix 2

page 19

Annex B

Documentation for Academic Review in Wales

Institutional documents

1 Apart from the self-evaluation, academicreviewers will not normally expect documents to beprepared especially for review. Subject providersshould direct reviewers, in the self-evaluationand/or by means of a separate list, to theavailability and relevance of documents whichmight assist them to test and verify the statementsmade in the self-evaluation or which are relevant tothe judgements they will make.

2 The following documents will be required inadvance of the review:

the self-evaluation, with the programmespecifications annexed;

relevant prospectuses;

a location map.

3 The availability and relevance of furtherdocumentation will be discussed through the initialcontact between the review coordinator and thesubject provider. The following documents arelikely to be relevant to the review:

subject or programme handbooks;

curricular documents, module or unit guides;

subject or programme monitoring reports,including those from external sources such asprofessional and/or statutory bodies, if theseare available;

external examiners' reports for the previousthree years.

The following documents may also be relevant, butthis list is neither prescriptive nor exhaustive:

minutes of examination boards;

practical or placement handbooks;

programme approval, validation and re-validation documents;

academic staffing list and short profiles

There is no requirement or expectation thatdocuments will be assembled in a 'base room'for the use of reviewers. If reviewers wish tosee a document, they will ask for it.

Student work

4 Reviewers will expect to see a sample ofstudent work. The range and nature of studentwork to be made available to the reviewers will bediscussed during the initial contact between thesubject provider and the review coordintor.Reviewers will look at student work to evaluatewhether:

student achievement matches the intendedoutcomes of the programme(s);

assessment is designed appropriately tomeasure achievement of the intended learningoutcomes;

the assessment instruments provide anadequate basis for discriminating betweendifferent categories of attainment;

the actual outcomes of programmes meet theminimum expectations for the award.

Reviewers will not duplicate or 'second-guess' thework of external examiners and will not normallyexpect to see work which is currently underconsideration by external examiners.

5 Academic reviewers will need to see a broadsample of student work that demonstrates use ofthe full range of assessment instruments deployedin both formative and summative assessments. Toenable them to gain a full understanding of theassessment strategy, reviewers may need to seemarking guides or other assessment criteria, andany guidance on providing feedback to studentsthrough assessment. They will use externalexaminers' reports to triangulate with their ownobservations of work from each level/year of study,samples of work from core modules and specialistoptions and from a representative range ofattainment. Samples of work may include, forexample:

coursework of various types;

practical, laboratory or workshop notebooks;

projects and/or dissertations;

examination scripts.

Annex B

page 20

Annex C

Agenda for Meeting with Students

Introduction

1 Meetings with students enable reviewers toestablish student views on the issues beingconsidered. These meetings provide an opportunitynot only to hear the direct views of those present,but also to establish more generally whether thereare effective arrangements for student feedbackand representation.

2 The meeting is normally chaired by the reviewcoordinator, who will introduce the subjectspecialist reviewers and provide a brief summary ofthe review method. S/he will outline the purpose ofthe meeting and will emphasise the importance oftransparency of the review process. The dialoguewith students will normally start with a question toestablish on what basis the students were selectedto attend the meeting.

The curriculum and intended learning outcomes

are students made aware of the intendedlearning outcomes by programmespecifications or other means?

what is the match between the expectations ofstudents, the intended learning outcomes andthe curricular content?

does the curricular content encourage thedevelopment of knowledge and skills?

what is its relevance to further study andprospective employment?

are timetables and workloads appropriate?

Assessment and achievement

do students understand the criteria forassessment and the methods employed?

is assessment formative as well as summative?

what feedback is there? Is it prompt andeffective?

in their experience, have the intended learningoutcomes been achieved?

do academic staff discuss student achievementwith students?

are further study and career aspirations likely tobe satisfied?

Student progression

what admission and induction arrangementsare in operation?

what are the arrangements for academicsupport?

do these arrangements extend to workexperience, placements, study abroad andother off-site experiences?

what skills are acquired? Do they enhanceemployability?

do students receive effective support?

Learning resources

How good are the library services in terms ofopening hours, access, user support,availability of books and journals?

what IT support is there? Are opening hours,access, user support and availability of workstations and software appropriate?

are there suitable programme-specificmaterials?

are the accommodation and equipmentadequate?

Appendix 2

page 21

Annex D

Institutional nominee

Introduction

1 Each institution may nominate a member ormembers of staff (normally no more than three) totake on the role of institutional nominee, althoughthere is no requirement to do so. The purpose ofthis is to provide the opportunity to the institutionto observe the process in order to gain a fullunderstanding of the engagement and to ensurethat the team obtains accurate and comprehensiveinformation about the educational provision and itsinstitutional context. In due course, the experiencegained by the institutional nominees in dealingwith engagements in several departments shouldenable them to help subject providers prepare forthese events, disseminate good practice within theinstitution, and highlight areas for improvementidentified by each engagement. Institutionalnominees will be briefed for their role by theAgency through written materials.

2 Institutional staff who wish to act asinstitutional nominees should possess:

thorough knowledge of the structure, policies,priorities, procedures and practices of theirinstitution;

extensive knowledge and experience ofworking in HE at a senior level;

extensive experience of quality assuranceprocedures;

knowledge and understanding of the Agency'sreview method for Wales;

qualifications and experience in a subject areaother than that being reviewed;

an ability to maintain confidentiality.

It is also preferable that nominees have either directexperience of teaching in HE or experience as asenior administrator in an HE institution.

3 If the institution puts no nominee forward, adesignated member of staff will normally take onthe liaison functions described in this annex fromthe academic department under review who willnot attend the team meetings.

Role of the nominee

General matters

4 Organisation and management of theengagement is the responsibility of the reviewcoordinator (CR). Responsibility for ensuring thatthe review team is provided with appropriateevidence to allow it to reach its judgements liesprimarily with the subject provider. The providerwill identify a principal subject contact with whomthe CR will liaise. Part of the nominee's role is toobserve the channels of communication betweenthe two and ensure that they work effectively.Discussions between the subject contact and reviewcoordinator should ensure that the subject provideris aware of issues being addressed by the team andthe evidence needed to clarify them. It would behelpful if HE institutions could supply reviewcoordinators with brief outlines of nominees'previous experience and current institutional roles.

5 Throughout the course of an engagement, thenominee may help the reviewers to come to a clearand accurate understanding of the structures,policies, priorities and procedures of the institution.S/he may wish to bring additional information tothe attention of the team and may seek to correctfactual inaccuracy. Assistance in understanding thesubject provision is provided by the subjectcontact. It is for the reviewers to decide how bestto use the information provided. The nominee isnot a member of the team and will not makejudgements about the provision.

6 The role requires the nominee to observeobjectively, to communicate clearly with the teamwhere requested, to respect the protocols onconfidentiality outlined below, and to establisheffective relationships with the review coordinatorand the team, as well as with the subject staff.Nominees should refrain from acting as advocatesfor the subject provision under review. However,they may legitimately:

provide advice on institutional matters.

identify the location of evidence

assist the institution in understanding issues ofconcern to reviewers;

Activities preceding engagements

7 Institutions may find it helpful to involvenominees fully in preparation for a subjectengagement, including the initial meeting with theAgency to discuss the scope and form of the event.

Annex D

page 22

The nominee with responsibility for an engagementshould receive copies of all correspondencebetween the Agency and the institution, and mayattend the first private meeting of the team or bebriefed about it by the review coordinator.

Activities during engagements

8 The extended pattern of engagement requiresnominees to fulfil two main functions in addition tothe general information role outlined above. First,they should monitor the pattern of visits byacademic reviewers. If it appears that there is adeparture from the agreed pattern, the mattershould be discussed immediately with the reviewcoordinator.

9 Second, nominees may attend all thefollowing:

team meetings, including those in whichjudgements are being discussed by the team ofreviewers;

formal meetings held between the reviewersand the institution to investigate mattersspecific to standards and quality, except thosewith current and former students;

'progress' meetings held between the reviewcoordinator and subject staff.

Nominees will not attend review team meetingswith students or employers.

10 In addition, the nominee should be keptinformed of the regular telephone and/or emailcontact between the review coordinator and thesubject contact.

Confidentiality

11 Nominees will observe the same conventionsof confidentiality as subject specialist reviewers. Inparticular, no information gained during anengagement shall be used in a manner that allowsindividuals to be identified. Nominees must exercisecare when reporting back to subject or institutionalstaff to maintain the confidentiality of writtenmaterial produced by reviewers for the initial teammeeting, or at other times during and after theengagement. However, nominees may make theirown notes on team discussions in order to help theinstitution understand the issues addressed byreviewers. This can contribute to the enhancementof standards and quality within the institution.

The QAA16 October 2001

Appendix 2

page 23

Appendix 3

Quality Assurance Agency Engagementsin Wales 2002

A synoptic paper addressing operational issuesand the implementation of the processesdescribed in the HEFCW paper (WO1/90HE)and the Agency Academic Review in Wales2002 Guidance, including Annexes A to D.

The rationale and broad process for theengagements in Wales for 2002 are set out in theHECWE circular W01/90HE and the AcademicReview in Wales 2002 Guidance including annexesA to D. Every Institution had a copy of thisdocumentation (except for Annex D) in December2001. Annex D is a draft outline of the role ofInstitutional Nominee that will be developed overthe next year

Together, these three papers should enable HEI andAgency staff/representatives to work together andensure subject staff and review teams successfullyundertake this developmental peer review process.

Scope

The scope of the review has been discussed andprovisionally agreed through a range of meetingsbetween the Agency and the Higher EducationInstitutions involved. Any changes or concerns canbe discussed with the Assistant Director,Programme Review Directorate (PRD), contactdetails below. Final confirmation will occur at thepreliminary meeting between the Review Co-ordinator and the Institutional staff.

Self Evaluation Document (SED)

This needs to reflect the provision under scope. TheDocument should follow the guidance set out inAnnexes C and E in the Agency Handbook forAcademic Review 2000. All elements includingteaching and learning should be addressed in theevaluation. The

SED will normally be submitted to the Agency (3copies) two months prior to the start of the review(Day One when the review team and the subjectstaff meet). The Agency will be responsive toinstitutional requests, please agree any changeswith the Assistant Director PRD before revisingdates for submission of the SED.

Once the review team has been agreed, theInstitution needs to send the SED to each reviewer(the Review Co-ordinator receives one of the threecopies sent to the Agency).

Review Team

This will normally involve three subject specialistreviewers plus a review co-ordinator. In this peerreview process, the reviewers should be credible tothe Institution's subject staff as competent tocomment on the range of programmes provided.However, the review teams cannot reflect everyJoint Academic Coding System (JACs code) and theintention is to reflect the nature of the provisionand the Institution, as well as subject area. EachInstitution will have opportunity to discuss andagree the proposed reviewers.

The Agency will try to arrange that one of thereviewers either works in Welsh Higher Education orhas had experience of the Welsh HE system.Institutions need to inform the Agency AssistantDirector if they do not wish this to be arranged.

Since no bilingual or Welsh-medium programmesare being offered for review, the Agency will not beseeking to appoint specifically bilingual reviewers.

Preliminary Visit

The Review Co-ordinator will arrange for apreliminary visit to agree arrangements(approximately four weeks prior to Day One) andthe range of documentary evidence that will berequired. The intention is always to make theserequirements as minimal as possible and forreviewers to only request relevant, livedocumentation when necessary. On someoccasions, a Agency officer will join this meeting tohelp answer queries that the Institution may raiseabout the method of engagement.

At this meeting, the Institution needs to confirmwhether they wish to use the role of InstitutionalNominee and who will undertake it. The ReviewCo-ordinator completes the record of thepreliminary meeting by noting who is theInstitutional Nominee, where relevant.

Pattern of an Engagement

Although teaching and learning does not receive aseparate judgement, it is certainly expected thatthe review team and subject staff will engage withthis element. Reference will be made to teachingand learning in sections of the report, (see below)where appropriate.


page 24

Whatever the preferred pattern, it is the intentionthat review team activity in all subjectengagements will not normally extend beyond fourweeks (this does not include the preparation of thereport). The Review Co-ordinator will guide thereviewers to use the Agency Academic Review inWales's 2002 guidance in full when visiting theinstitution.

The Review Co-ordinator will provide oral feedback.The exact timing will depend upon the pattern ofthe engagement, but will be agreed by theInstitution with the Review Co-ordinator. Thefeedback will be brief and outline the judgementsin relation to standards and quality of learningopportunities, only key issues supporting thejudgements will be noted. The judgements will beconfirmed in writing within three weeks of thejudgement meeting.

The pattern will be either a two one day visits tothe institution, preceded by a one-day desk- basedanalysis and followed by one day report drafting, oran intensive visit of two days, preceded by desk-based analysis and followed by one day reportdrafting. The Institution and the Review Co-ordinator may agree to use the report writing timeby staying at the institution for 'Day Three' andagreeing judgements and providing oral feedbackat that time. This is an optional approach, morereflection may be needed prior to the finaljudgements being made by the team. The ReviewCo-ordinator will make sure the Institution knowswhen both judgements have been reached andwhen the oral feedback will be provided. The oralfeedback may be face to face or by telephone ifthat is agreed.

Each reviewer has five days allocated, each ReviewCo-ordinator has ten days allocated per review.

Report

These reports are Agency copyright. The report willbe confidential to the Institution, the Agency andthe HEFCW, the report will not be published. Thereport will include all judgements and sections aslaid out in the Academic Review Handbook 2000,EXCEPT teaching and learning.

The reports will follow the standard Agencyapproach of a first draft being sent to theInstitution for checking for factual accuracy byWeek 7. The Engagement report will normally befinalised by week 18 following the judgementmeeting.

In 2002, it will be the decision of the Institutionwas to whether an action plan in response to thereport findings is produced.

Appendix 3

page 25