10 July 2008

Rafael LlavoriHead of the Institutional and International Relations Unit

Self-assessment of agencies and self-Self-assessment of agencies and self-assessment report. assessment report.

An agency perspectiveAn agency perspective

ENQA Seminar

“First external evaluations of quality assurance agencies – lessons learned”

10-11 July 2008, Paris

Understanding the ESG

ANECA. Internal review of a QA

Agency

Table of contentsTable of contents

1. Understanding the ESG´s

Main steps: Analysing the ESG: rethinking at the Spanish setting

Adaptation of ANECA´s processes to the ESG

Elaborating a self-evaluation guide for ANECA´s staff (traslate the ESG into ANECA´s vocabulary): to be sure that everyone speaks the same ‘language’

Understanding the ESG´s

Part 1

European standards and guidelines for

internal quality assurance within higher education institutions

European standards and guidelines for the external quality assurance of

higher education

European standards for

external quality assurance agencies

Understanding the ESG´s

External quality assurance agencies

3.1 Use external quality for higher education

3.2 Official status

3.3 Activities

3.4 Resources

3.5 Mission statement

3.6 I ndependence

3.7 External quality assurance criteria and processes

used by the agencies

3.8 Accountability procedures

Understanding the ESG´s

European standards and guidelines for the external quality assurance of higher education

2.1 Use of internal quality assurance procedures

2.2 Development of external quality assurance processes

2.3 Criteria for decisions

2.4 Processes fit for purpose

2.5 Reporting

2.6 Follow-up procedures

2.7 Periodic reviews

2.8 System-wide analyses

Understanding the ESG´s

European standards and guidelines for internal quality assurance within higher education institutions

Part 1

1.1 Policy and procedures for quality assurance

1.2 Approval, monitoring and periodic review of

programmes and awards

1.3 Assessment of students

1.4 Quality assurance of teaching staff

1.5 Learning resources and student support

1.6 I nformation systems

1.7 Public information

Understanding the ESG´s

ANECA’s external evaluation programmes /Standards for

quality assurance in the EHEA (part 1)

Institutional assessment Programme

Quality Label Ph.D. Programme

Evaluation programme - Recognised postgraduate programmes

Evaluation programme -

Services

Evaluation programme - Teaching staff / Support programme for teaching

activity assessment (DOCENTIA)

Standards numbered according to the corresponding ANECA programme

1.1 Policy and procedures for quality assurance 2 8 5 * *

1.2 Approval, monitoring and periodic review of programmes and awards

1, 2, 3, 4, 5 and 6

1, 4, 7 and 8 1, 2, 3 and 5

1.3 Assessment of students 1 and 5 1 2

1.4 Quality assurance of teaching staff 3 and 6 2 and 3 5 All standards

1.5 Learning resources and student support 3, 4 and 5 4 and 5

All standards

1.6 Information systems 2 and 6 4 and 5 5

1.7. Public information 1 and 2 5

Correlation between the criteria of the different programmes carried out by ANECA and the standards established in part 1

Understanding the ESG´s

2. ANECA. An internal review of a QA

Agency

Building up a structure to work

Staff Involvement

Leadership (person incharge of the self-evaluation report and 2 assistants)

Self-evaluation committe (12 people)

Time investment to achieve the goals

ANECA. An internal review of a QA agency

ESG

The evaluation agenda: the deadline

Director´s implication

Board of directors and staff of ANECA

Advisory Board

Tools

ANECA. An internal review of a QA agency

Methodology of the work

Evaluation as a strategic objective of the Agency as a whole

Self-evaluation guide

Fixed schedule

Periodical meetings of the S-E C

On-going information and training

process

ANECA. An internal review of a QA agency

Main purpose

Participative processParticipative process

Information Dissemination

ANECA. An internal review of a QA agency

Development of the work

ANECA. An internal review of a QA agency

The Team work was the reference point:

• Strict with the time and work

• Defined the agenda of the

meetings

• Elaborate the self-evaluation

guide

• Analysed the information of the documents used by the Self-evalution Committe

ANECA. An internal review of a QA agency

The Team work was the reference point:

• Collected the evidence and organised them

• Traslated the documentation into English

• Writing the self-evaluation report

• Prepared the site visit

Development of the work

ANECA. An internal review of a QA agency

The Self-evaluation Committee worked with the designed guide and:

• Analysed the information of the guide and possible evidence to be provided in each section

• Attended the different meetings

• Set up and circulated the information on time

Development of the work

ANECA. An internal review of a QA agency

The Self-evaluation Committee worked with the designed guide and:

• Collaborate in the spreading of the information to the rest of the staff

• Studied the first draft of the self-evaltuation report

Development of the work

ANECA. An internal review of a QA agency

The Board of Director and Self-evaluation Committe:

• Studied the draft of the self-evaltuation report

• Breakfast at ENQA’s

Board of Trustees

Advisory Board

Board of Directors

Technical Committee

Evaluation Committees

ANECA staff

Self-evaluation Committee

Working group

Procedure followed during ANECA´s self-evaluationProcedure followed during ANECA´s self-evaluation

Preparation of the self-evaluation

guideSetting upof the self-evaluation

committee

Analysis of criteria and search for

information

First draft of the self-evaluation

report drawn-up

Summary of the information provided

First draft of the report is reviewed

Suggestions incorporated

Report endorsed

Review of the report proposal

Approval of final report

Second draft of the report

reviewed Suggestions incorporated

Suggestions analysed

Third draf of the report reviewed

Suggestions incorporated

Suggestions incorporated

Dates

Sep. 06

Oct. 06

Nov. & Dec. 06

Jan. 07

Mar. 07

Third draft of the report reviewed

Rules to play for

ANECA. An internal review of a QA agency

Information

• Meeting

• General presentation

• Internal newsletter

The truth

• Weaknesses and strengthes are useful for the organization

Precision and clarity

• It was very important to be short and clear

• 20 words for an explanation better that 100 words

Structure of the report

ANECA. An internal review of a QA agency

Introduction

• Background

• Aim of the external evaluation

• Spanish university system

• ANECA

Self-evaluation of ANECA

• Background

• Self-evaluation process

Analysis of the Standards and guidelines

Overall assessments

Overall assessment

Oportunities for improvement

Annexes

Glossary

List of evidence

Organization chart

Conclusions

ANECA. An internal review of a QA agency

The self-evaluation process was important:

To learn about our own organization

To create a new methodology for internal work

To be aware of the weaknessess of ANECA

To be aware of the strengths of ANECA

Conclusions

ANECA. An internal review of a QA agency

We could not do it before Maturity of both organisation and staff

Completion of the natonal legal framework

We should not do it later Just before the implementation of the accreditation

process

Need for an internatonal support and recognition

www.aneca.eswww.aneca.es

Thanks for your attentionThanks for your attention