Evaluating teaching development in higher education: towards impact assessment

Project Summary

Confidential to the Project Team and HE Academy.

This document is shared with the Academy providing additional context and information about the process of the project, and to inform discussion at the final meeting.

It is not intended for wider distribution or publication.

The project team:

PedRIO: Pauline Kneale, Jennie Winter, Rebecca Turner, Lucy Spowart and Reema Muneer

Critical friends: Denise Chalmers, Nancy Turner,

HEDERA: Jane Hughes, Colleen McKenna

Project partners: Jan Smith (Durham) and Chris Smith (University Campus, Suffolk)

2/222

Table of Contents

Terms of reference................................................................................................................................6

Abbreviations used in this report..........................................................................................................6

Acknowledgements...............................................................................................................................6

Executive summary...............................................................................................................................6

Framing of the project...........................................................................................................................8

Work Package 1 (WP 1) - Literature review.........................................................................................12

1.1 Introduction...............................................................................................................................12

1.2 Building on Parsons...................................................................................................................13

1.3 Method......................................................................................................................................15

1.4 Areas of Impact..........................................................................................................................17

1.4.1 Impact on teachers’ attitudes, knowledge and skills..........................................................17

1.4.2 Evidence of impact on teachers’ behaviour and practice..................................................19

1.4.3 Effects of disciplinary or generic programme focus...........................................................20

1.4.4 Compulsory vs non-compulsory.........................................................................................21

1.4.5 Student impact/impact on student learning.......................................................................22

1.4.6 Other references to impact not covered by the above......................................................25

1.5 Research in the broad area of CPD impact that addresses the construction and use of frameworks.....................................................................................................................................28

1.6 Research that goes beyond impact of CPD but which may be relevant more broadly..............32

1.6.1 Reconceptualising academic development........................................................................32

1.6.2 Impact of networks and brokers........................................................................................33

1.6.3 Narratives..........................................................................................................................33

1.6.4 Impact of diverse participation groups..............................................................................33

1.6.5 Identity construction, boundary crossing and impact........................................................34

1.7 Critique of the impact discourse................................................................................................35

1.8 Conclusion.................................................................................................................................37

1.8.1 UKPSF, teaching excellence and CPD frameworks.............................................................37

1.8.2 Gaps in the literature.........................................................................................................39

1.8.3 Key points and implications for future work on evaluating CPD........................................39

1.9 Final thoughts............................................................................................................................40

Work Package 2 (WP 2): Invite institutions operating CPD schemes to provide evidence of current practice in measuring the impact and effectiveness of CPD provision............................................41

3/222

2.1 National audit............................................................................................................................42

2.1.1 Themes addressed within the audit...................................................................................42

2.1.2 Audit audience...................................................................................................................44

2.1.3 Audit development............................................................................................................44

2.1.4 Dissemination of the audit.................................................................................................45

2.1.5 Analysis of audit data.........................................................................................................45

2.2 Results of the National Audit.....................................................................................................46

2.2.1 Overview of respondent profile.........................................................................................46

2.2.2 CPD Provision (N = 109).....................................................................................................50

2.2.3 Evaluation...........................................................................................................................54

2.2.4 Student involvement in CPD (N=108).................................................................................60

2.2.5 Institutional policies and culture around teaching-related CPD.........................................61

2.3 Case study development...........................................................................................................66

2.3.1 Selecting the case study institutions..................................................................................66

2.3.2 Data collection...................................................................................................................67

2.4 Case studies...............................................................................................................................68

2.4.1 Case Study – London Metropolitan University....................................................................68

2.4.2 Case Study – University of Roehampton.............................................................................72

2.4.3 Case Study – Sheffield Hallam University............................................................................76

2.4.4 Case Study – Nottingham Trent University.........................................................................80

2.4.5 Case Study – Queen Margaret University, Edinburgh.........................................................86

Professional development opportunities for staff............................................................................86

Evaluating the effectiveness of teaching-related CPD......................................................................87

Challenges to evaluating CPD..........................................................................................................88

Future Plans.....................................................................................................................................89

Work Package 3 (WP 3): Design and test a tool for the assessment of the effectiveness of CPD........90

3.1 Tool Design and Development...................................................................................................90

3.1.1 Influential factors shaping the development of the tool....................................................91

3.1.2 Tool development...............................................................................................................93

3.2 Tool Implementation and Piloting.............................................................................................95

3.3 Tool Evaluation..........................................................................................................................97

3.4 Results of the national pilot of the evaluation tool....................................................................99

3.4.1 Piloting of the tool..............................................................................................................99

3.4.2 CPD completers experiences of using the tool.................................................................104

4/222

3.4.3 Recommendations for the further development of the tool based on the analysis CPD completer data..........................................................................................................................106

3.4.4 CPD Providers experiences of using the tool...................................................................106

3.4.5 Limitations to the current evaluation work......................................................................112

3.4.6 Implications for the further development of the tool based on the analysis CPD completer data............................................................................................................................................112

3.5 Recommendations...................................................................................................................113

3.5.1 Higher Education Academy...............................................................................................113

3.5.2 HE Providers......................................................................................................................114

3.5.3 Educational Developers / CPD providers..........................................................................114

3.5.4 Individual..........................................................................................................................115

3.5.5 Future work......................................................................................................................115

References.........................................................................................................................................116

Appendix 1: Audit template used in WP2..........................................................................................121

Appendix 2 – Workbook on which the national audit of the tool was based....................................154

Introduction...................................................................................................................................155

Evaluation Protocol.......................................................................................................................156

Step 1: Consider these questions..................................................................................................157

Step 2: Evaluation questions..........................................................................................................158

Step 3: Choose the method you want to use.................................................................................179

Step 4: Develop evaluation instruments to accommodate these requirements...........................184

Step 5: Send the evaluation instruments.......................................................................................184

Step 6: Return annotated workbook.............................................................................................184

Appendix 3: Appendix 3: Key findings from interviews with CPD completers captured through WP3...........................................................................................................................................................185

Summary analysis – teaching courses for new lecturers...............................................................185

Summary analysis - In house accreditation schemes.....................................................................186

Summary analysis across activities................................................................................................187

List of Tables

Table 1.1: stages of the literature review

Table 1.2: tags used to code the literature

Table 2.1: categories of institution and responses

5/222

Table 2.2: percentage of institutions with no minimum teaching-related CPD requirement

Table 2.3: types of CPD from most to least commonly reported

Table 2.4: focus of accredited and non-accredited CPD offers

Table 2.5: overview of the evaluation practices discussed by respondents

Table 2.6: perception of student awareness and involvement in CPD

Table 2.7: percentage of institutions which have an existing policy / expectation around peer review (by institutional category)

Table 2.8: peer review and annual appraisal / performance review

Table 3.1: range of CPD activities included in the pilot

Table 3.2: frequency counts of the themes from which the questions were selected across the range of CPD activities included in the pilot

Table 3.3: timeframes over which the CPD completers had engaged with their CPD activities that were the subject of the pilot

Table 3.4: what the CPD providers were using the tool to evaluate

Table 3.5: the purpose of the evaluation

Table 3.6: the intended uses of the evaluation data

List of Figures

Figure 2.1: the most commonly reported CPD expectations for different roles

Figure 2.2: accredited CPD offers described by respondents

Figure 2.3: non-accredited CPD offers described by respondents

Figure 2.4: persons involved in the delivery of CPD

Figure 2.5: count of evaluation at each stage of the accredited / non-accredited offer

Figure 2.6: focus of evaluation of accredited / non-accredited offers at different stages

Figure 2.7: institutional differences in promotion pathways

Figure 2.8: occurrence of professors of teaching and learning

Figure 2.9: reward and recognition for teaching and learning

List of Boxes

6/222

Box 2.1: overview of findings related to teaching qualifications and minimum CPD requirements

Box 2.2: overview of findings related to accredited / non-accredited CPD offers

Box 2.3: overview of findings related to specific CPD activities

Box 2.4: overview of findings related to reward and recognition for teaching and learning

Terms of reference

CPD provider – this term is used to represent individuals who may perform a staff, educational or academic development function or an individual who was responsibility for the provision of teaching-related CPD.

CPD completer – this term is used to represent individuals who have engaged with teaching-related CPD.

Abbreviations used in this report

Abbreviation DefinitionCBHE College-based HECPD Continuing professional developmentFE Further EducationFTE Full time equivalentsGTA Graduate teaching assistantsHE Higher educationHEA Higher Education AcademyNSS National Students SurveyPDR Performance Development ReviewPTES Postgraduate Taught Experience SurveyQAA Quality Assurance AgencySEDA Staff and Educational Development AssociationSoTL Scholarship of teaching and learningUKPSF UK Professional Standards FrameworkWP Work package

Acknowledgements

This project was funded by the HE Academy.

The project research and toolkit development was undertaken by the authors, through the Pedagogic Research Institute and Observatory (PedRIO) at Plymouth University in collaboration with the Higher Education Development Research Agency (HEDERA) and critical advisors.

7/222

The toolkit for evaluation was developed and piloted with 12 HE institutions including college-based, private provider, teaching focused and research focused HE institutions located in England, Scotland and Wales.

The research team is very grateful to all the partners who gave their time to explain current processes and shared classroom evaluation practice.

8/222

Executive summary

This report is informed by the literature on evaluating higher education teaching and by empirical

research which established common and best practice across the sector. It provides a series of

templates that can be used to create bespoke CPD evaluations aiming to capture the impact through

evaluation before, during and after an activity or event. In essence this follows a five step process:

Analysis of staff and programme development needs in relation to previous feedback,

institutional and departmental priorities;

Establishing learning outcomes for the CPD activity and the aims of the evaluation;

Consider HOW and WHEN to evaluate impact, considering the aims and learning outcomes;

Deliver or facilitate the CPD activity and commence ongoing evaluation. This may be ongoing

over several years;

Disseminate the findings and share the lessons learnt.

The Report is focussed around supporting staff to develop meaningful evaluation processes, using

standard social science research methods, and supported through a suite of 12 Templates that offer

structure to the selection of questions to pose.

The approach advocates evaluation activity before, during and after events, with the emphasis on

repeating evaluation after 6 months and two years to gauge longer term impacts. The annual

university cycle means that opportunities to act on information gained in a CPD activity cannot be

used for eighteen months to two years.

The templates are a primary resource that the authors hope others will contribute to, allowing us to

expand and refine the data base of questions, and the scope of the templates.

These templates are available at http://www1.plymouth.ac.uk/research/pedrio/Pages/HEA-CPD-Framework.aspx

9/222

Framing of the project

Interest in the value of continuing professional development (CPD) for teachers in Higher Education

(HE) has increased in response to a changing landscape motivated in part by policy developments1

and paralleled by changes to fee structures which position students as ‘consumers’ of HE (Clark et al.

2002). The student voice gathers power through instruments such as the NSS and the PTES and HE is

publicly accountable and indeed scrutinised for teaching quality (Stoakes, 2013) Teaching and

learning, traditionally the poor relations of research, are now commonly used by policy makers and

students alike in making distinctions and decisions between HE providers.

These changes have been imposed on an increasingly complex HE sector, with respect to the

diversity of providers, and differing policy levers acting on provision across the four UK nations. In

England in particular, successive government initiatives and policies have led to a clear agenda

emerging with respect to CPD for those leading and supporting teaching and learning (Spowart et al,

in press). Whilst training for new lecturers is commonplace, following the Browne Review (2010),

the spotlight2 has also been placed on CPD for more established staff (HEA, 2012).

CPD in teaching is manifest in a number of guises including (but not exclusively) the Postgraduate

Certificate in Teaching and Learning in the UK (LTHE/ PGCAP or equivalent), short training courses;

in-situ training; consulting, peer review and mentoring; student assessment of teaching; and

intensive staff development (Prebble et al. 2004). Such provision has flourished since the White

Paper: The Future of Higher Education (DfES, 2003) - although there is a much longer history in some

institutions - and has been supported by the ongoing uptake of the UKPSF (Turner et al., 2013; HEA,

2012).

There have been a number of seminal studies which have investigated this area (Gosling, 2008; Ling,

2009; Turner et al., 2013), as well as meta reviews of the literature (Chalmers et al., 2012; Parsons et

1 Including the DfeS (2003) English Education White Paper, HEFCEs Strategic Plan (2003) and the establishment of the of the HEA in 2004, the role of the QAA in monitoring teaching quality (HEA/QAA 2004), the launch of the UKPSF (2006), the BIS White Paper ‘Students at the Heart of the System’ (BIS 2011) and the requirements of HESA (2012) to return teaching qualifications of university staff

2 But not necessarily all HE providers

10/222

al., 2012). These studies found widespread consensus about the value of CPD in teaching but

identified issues with how impact is measured, with respect to content and methodology.

In terms of method, to date, much CPD evaluation relies upon a snapshot which takes place in situ,

post intervention and seeks measurement of satisfaction, rather than of those changes to thinking

and practice which develop over time (Bamber, 2013; Rust, 1998; Sword, 2011). These studies have

provided a useful benchmark in terms of the direction future work should take. Greater

consideration needs to be placed on measuring the breadth of impacts from CPD activities, looking

beyond immediate satisfaction, to how CPD changes practice in the longer term and also shapes

student learning.

Bamber (2013) calls for a greater focus on ‘evidencing value’ which will encourage a move away

from quantitative measures of ‘satisfaction’ with respect to whether intended outcomes have been

achieved, towards more reflexive approaches that consider soft and hard outcomes. Measuring

these so-called ‘soft’ impacts, such as increased confidence, thinking differently, and a willingness to

change practice, all of which benefit student learning, are challenging and require greater attention

to be placed upon the process and the outcomes of CPD (Bamber, 2013). The use of qualitative data

is advocated, as such information when viewed alongside the standard quantitative measures, can

develop a more nuanced understanding about commonalities and differences across different

approaches and contexts for CPD in teaching and learning (Chalmers, 2008; Shavelson, 2010). This

would also allow the scope of future CPD evaluation to look beyond individuals own practice to

impacts they have on their students, their colleagues and institutional ways of working (Bamber,

2013).

It is against this backdrop the current project was funded. The HEA called for proposals around the

area of ‘evaluating teaching development in HE: towards an impact assessment’, outlining the

specific aim of ‘creating state of the art knowledge and understanding relating to how the impact of

CPD schemes on the student learning experience can be measured’. In response to this aim the

project team divided the project into three work packages (WP) to address the aims of the tender:

Work package 1 (WP1): Update the literature review established in the Parsons et al. (2012)

report entitled ‘the impact of teaching development programmes in higher education’.

11/222

Work package 2 (WP2): Invite institutions operating CPD schemes to provide evidence of

current practice in measuring the impact and effectiveness of CPD provision.

Work Package 3 (WP3): Design and test a tool for the assessment of the effectiveness of

CPD.

Below we report on work undertaken and outcomes of each WP sequentially. Detailed accounts are

provided of each WP, which complements the overview of the toolkit produced by the project team

which focuses solely on the outcomes of WP3.

12/222

Work Package 1 (WP 1) - Literature review

1.1 Introduction

Largely established in the 1990s, academic development for teachers in UK Higher Education is still a

relatively young field which is under regular review and revision (Turner et al., 2013). With recent

changes to funding and priorities in the UK HE sector, the impact or effectiveness of CPD

programmes and activities is increasingly under scrutiny as researchers and stakeholders attempt to

understand how such work influences teaching, learning and the broader student experience.

This literature review aims to update the Parsons et al. review commissioned in 2012 by the HEA in

order to inform the Evaluating Teaching Development in HE: Towards Impact Assessment project, in

particular the development of a framework to evaluate the effectiveness of CPD. The review focuses

on research into the impact of CPD in teaching and learning published between 2012 and 2015. We

have broadened the scope and orientation of the original review in order to take account of a wider

range of activities that feature in CPD and to include literature that critically engages with the impact

discourse (e.g. Di Napoli, 2014). We also consider prominent themes in the research on impact of

CPD in relation to work on excellence in teaching in HE (e.g. Gunn and Fisk, 2013).

There are several challenges in assessing the body of research on the impact of CPD. Firstly, the

literature on CPD and impact encompasses a range of activities. For example, De Vries et al. (2013),

define CPD quite broadly as 'updating, reflective and collaborative activities' and point to a growing

interest in collaboration as a valuable teacher development activity. This breadth aligns with the

trend in the UK towards more flexible accreditation schemes (in the wake of the revised UKPSF) in

addition to taught postgraduate programmes; an increased focus on mid-career provision; increased

use of activities such as professional conversations; and peer networks, to name but a few. We

have tried to stay alert, in this review, to research on activities which are part of this broadening of

provision and, to this end, in addition to more conventional CPD work (such as PGCerts, short

courses, workshops) we have included work that addresses identity construction, reflection as a

form of CPD, narrative study and the impact of peer networks. Additionally, bearing in mind

Saunders (2014), we have tried to acknowledge the diversity of contexts and the significance of this

for individual teachers' implementation of professional development learning.

Secondly, ‘impact’ is a complex, often contested, concept, and there is not a consensus about what

constitutes impact in relation to CPD. What is measured under the rubric of impact varies

considerably with studies focusing variously on satisfaction, student performance, teacher self-

efficacy and reflection, among other things. As with Gunn and Fisk’s (2013) observations about

teaching and teacher excellence, impact is contingent on context (institutional and discipline),

educational values and how it is defined.

Finally, there is a dissonance in the discourses surrounding CPD and in order to represent the range

of attitudes towards assessing impact, we have included publications that challenge a narrow,

instrumentalist approach to impact measurement and that call, instead, for a more holistic, creative

attempt to discuss and determine the ‘impact’ of academic development. There is a growing body of

work on this theme and it is important to acknowledge and respond to it in a project that relies on

the participation and support of colleagues in the field.

1.2 Building on Parsons

The 2012 Parsons report pointed to a number of strengths and weaknesses in evidence for the

impact of professional development. The diversity, international base and applied nature of studies

were noted as strengths. A preponderance of small-scale studies and ‘snapshots’ was considered a

weakness. Recent work includes such studies but we also found a number of larger-scale evaluation

studies and several that addressed factors affecting transfer to practice and the impact of

professional development over time. Reliance on self-report as evidence was and still is a concern; it

perhaps accounts for the frequent focus on teacher and student perceptions. Lack of comparability

between studies is still an issue. This is linked with the diversity of frameworks, methods and

measures being used and also with differences in context. Parsons et al. (2012) also note the lack of

baseline data from which to measure teacher or student gains and point to the need for common

tools and frameworks to capture such data and to aid comparability.

We found new work in all of the areas considered by Parsons et al. (2012). In defining the scope for

the review and grouping the literature reviewed, we used similar categories to theirs (1-6 in the list

below) with three additions (7-9):

15/222

The review is organised along the following themes:

1. Impact on teachers’ attitudes, knowledge and skills

2. Impact on teachers’ behaviour and practice

3. Effects of disciplinary or generic programme focus

4. Compulsory vs non-compulsory

5. Impact on student learning

6. Other emergent themes:

a. Motivation

b. Teacher experience

c. Online CPD

d. Social networks/communities of practice

e. Location

f. Time

7. Research into frameworks for evaluating CPD;

8. Related, relevant research, concerned with issues beyond impact, and

9. Papers which offer a critique of impact measurement

Additionally, following comment on the first draft of this review, we have considered potential

synergies between the research on CPD impact and recent work on teaching/teacher excellence and

career progression.

The main aims for this report are to

- update, summarise and analyse literature on the impact of CPD in teaching published since the

2012 Parsons review.

- identify gaps in this body of work

- highlight references to the use of frameworks for evaluating impact

- consider how the findings of the review can be used to inform the project’s evaluation

framework and other outputs.

In this update we are looking particularly for

16/222

- different ways in which impact has been described or defined

- perspectives from which impact has been assessed (e.g. teacher, student, course)

- relationships between impact and groups or networks

- an awareness of impact and contexts (disciplinary, institutional, national, etc.)

- frameworks that offer both theoretical conceptualisations and practical approaches to the

evaluation of CPD.

-

1.3 Method

This review was drafted as a working document, the first purpose of which was to inform the next

stage of the project. It has been further developed in the light of discussion with the project team

and HEA representative.

The methodology for the review arose from the need to synthesise research in diverse educational

contexts, with varying aims, approaches, frameworks and definitions of what constituted

professional development and impact.

Table 1 summarises the stages of the review. Two researchers conducted a search of large education

databases, followed by a manual search of selected journals, a further search using Google Scholar

and a review of more recent work by a small selection of the key writers identified by Parsons et al.

(2012). This process generated over 800 potential sources, and based on initial inclusion criteria

(concerned with the impact or effects of teacher professional development), a review of abstracts

resulted in the elimination of more than three quarters of these documents.

The resulting list of 187 texts was examined with reference to their aims, methods and findings, and

each was tagged using one or more of the tags listed in Table 2. This process led to further

elimination. The 87 remaining texts were then coded and assigned to categories corresponding to

Parsons’ 6 key areas, with the three additional themes listed above. A document could be assigned

to more than one of these. This process again resulted in a reduction, so that the final group of

about 40 sources forms the basis of the findings outlined in the following sections. Additionally, as

we encountered papers of relevance to the broad aims of the project, but outside the immediate

17/222

scope of ‘impact’, we included these and they are considered in sections 8 and 9. Finally, following

consultation on the draft literature review, we have considered our findings in light of recent work

on excellence in teaching and learning and we have included observations on the potential links with

this research.

Table 1.1: Stages of the literature review

Activity Sources Details NotesSearch 1 Major

databasesBritish Education Index, ERIC, Australian Education Index

Search terms based on Stes et al. (2012) and De Rijdt et al. (2013) with addition of terms related to learning technologies.Publication dates 2012-2015

Search 2 Manual journal search

Educational Research and Evaluation, Educational Research Review, Innovations in Education and Teaching International, International Journal of Academic Development, Higher Education Research & Development, Teaching and Teacher Education,Studies in Higher Education, Teaching in Higher Education

Review of titles, reference to abstracts in all volumes published between 2012-2015

Search 3 Personal home pages

Major contributors identified in Parsons

Aim to find any further relevant research or on-going projects

Search 4 Google scholar Further searches Simplified set of search terms

18/222

Publications 2012-2015Inclusion 1 Closer look at

the nature of the study and its relevance to evaluating CPD.

Review of titles and abstracts Aim to exclude documents that are clearly outside the scope of the review

Create single list

Lists of documents generated by Searches 1-4

Researchers’ lists combined and duplicates removed

“Long list” of 188 documents entered in Mendeley

Inclusion 2 Documents in ‘long list’

Review and tagging of long list texts based on abstract and reference to methods and findings; Additional stage: further papers excluded following reading of full texts

“Short list” of 87

Inclusion 3 “Short list” documents

Allocation of full texts to 6 categories representing Parsons’ areas of interest plus 3 additions; further papers excluded

38 documents considered in this review

Table 1.2: Tags used to code the literature

E Empirical I Student impact

C Conceptual L Long-term study

S Schools B Big study

H Higher Education O Other

F Framework U Unsure

D Discipline-based R Reject (Exclude)

1.4 Areas of Impact

1.4.1 Impact on teachers’ attitudes, knowledge and skills

A core area of investigation in the impact literature is the extent to which CPD influences teachers’

awareness and understanding. Evidence of impact in these studies can be found in

attitudes, beliefs and intentions;

self-efficacy;

teachers’ conceptions of learning; and

skill development.

19/222

The extent to which HE teachers gain technical and conceptual skills that enable them to embed

learning technologies into their teaching is measured by Rienties et al. (2013) in a Europe-wide study

in which the CPD was delivered entirely online. Researchers used pre- and post- tests to attempt to

measure impact of an online teacher training programme and to study whether teachers altered

their beliefs and intentions towards student-centred learning and the use of learning technologies.

The framework that underpinned this work (the Technological Pedagogical Content Knowledge or

TPACK model) is addressed further in section 1.4.6 below.

Lau and Yuen (2013) address questions about teachers' perceptions of learning technologies and the

factors that influence their perceptions. Data (self-report) are from post training workshop

questionnaires from a sample of 100 in-service mathematics teachers in Hong Kong secondary

schools. 90 completed all 5 questionnaires. Question areas included both perceptions of and actual

use of technologies in teaching. The questionnaire also asked respondents to give tutor and session

evaluations. Age, gender and prior experience were also included and the result suggested that

these factors did influence the impact of training. Training was found to have a positive influence on

teachers' perceived efficacy with technologies. Impact on beliefs about the pedagogical usefulness of

technologies was less evident.

Enhancement of teacher self-efficacy is a recurrent theme in the research on CPD. Lee, Cawthon and

Dawson (2013) explore the potential relationship between teacher self-efficacy and pedagogical

conceptual change. Teacher self-efficacy was not found to be a predictor of conceptual change.

Similarly, Mowbray and Perry (2015) report improved teacher self-efficacy following a CPD

intervention, and, in contrast to the previous study, they identified a ‘significant increase’ in student

attainment, which they aligned with the increase in self-efficacy. This is, however, a small scale,

relatively short (6-weeks) study (see section 1.4.5 for more details.). In a similar vein, Lumpe et al.

(2012), reporting on a large scale study of intense and sustained CPD for US science schoolteachers,

found that the teachers’ self-efficacy improved significantly and that this was positively correlated to

an improvement in students’ standardized science test scores. The authors conclude that ‘self-

efficacy and professional development hours were positive predictors of student achievement’. The

scale is large in this research, with a sample size of about 500 teachers and between 1300 and 8000

students, varying according to study strand.

20/222

McGee et al. (2013) adapted Guskey’s (2012) framework to investigate how teachers’ perceptions of

CPD and changes in self-efficacy affect transfer and impact. Centred around the implementation of a

new, standards-based mathematics curriculum, the CPD intervention comprised an intensive

summer course followed by a year-long programme, similar to Lumpe et al. (2012). Researchers

state that self-efficacy increased and that teachers’ orientations to CPD at the outset influence its

impact. However, neither of these findings is quantified.

1.4.2 Evidence of impact on teachers’ behaviour and practice

Impact on teachers’ behaviour and practice is an area that has motivated significant research

particularly in terms of student perception of teachers’ practice and ways in which teachers pursue

their own pedagogic development in the period following CPD.

A mixed methods study by Stes et al. (2013a) suggests that students perceive no impact on teachers’

teaching behaviours following engagement with CPD. However Willett et al. (2014) offer evidence of

improvement in teachers’ pedagogical practice as determined by evaluating assignment prompts

and student texts as part of a 3-year mixed methods study. (This study is discussed further in section

1.4.5.)

Rienties and Kinchin (2014) demonstrate that teachers’ social behaviour with professional networks,

that is, the extent to which they engage with and learn from their peers in relation to teaching and

learning, is enhanced by CPD. (See section 1.4.6 for a fuller discussion of this work in light of

communities of practice and CPD impact.)

Brown and Inglis (2013) attempt to identify factors that enable or hinder teachers’ application of

professional development learning in their practice, and the sustainability of their learning over

time. One enabling factor was a continuing dialogue, which might support the findings of research

on teachers’ social behaviours and learning from peers (Rienties and Kinchin, 2014; Stewart, 2014;

De Rijdt et al., 2013). Other factors which encouraged the embedding of CPD learning in day-to-day

practice were, according to Brown and Inglis (2013): seeing resulting change in the students;

21/222

supportive management and leadership; opportunities to discuss and share with peers; and time to

reflect.

Dyment and O’Connell (2014) suggest that CPD programmes ‘model’ good practice, which

participants transfer to their own teaching. It is, however, a small interview study (8 interviewees)

with a very specific focus on the use of learning journals. Similarly, Han and Finkelstein (2013)

explored the relationship between teachers’ professional development in the use of clicker

technology and their adoption of this in formative and summative assessment (see section 1.4.5).

Similarly, Armour and Makopoulou (2012), evaluating a national teacher development programme,

suggest that the programme would have been more effective had it enabled teachers to follow up

particular interests over time and offered opportunities for on-going dialogue.

In a different type of transfer, Trigwell et al. (2012) examining 10 years of data, show that academics

who completed a CPD programme were more likely to be awarded teaching grants from the

university. Similarly, staff who completed the programme were more likely to be recipients of

teaching awards. They also reported a ‘small but significant difference’ in students’ satisfaction with

their course when taught by staff after they had completed the CPD programme. Similarly, the

authors found that students in faculties with a higher proportion of staff who had completed the

CPD reported higher satisfaction rates with their courses.

1.4.3 Effects of disciplinary or generic programme focus

Possible differences in the effectiveness of CPD, depending on whether it had a disciplinary or

generic focus, were considered by Parsons et al. (2012). As in that report, we found little comparison

of disciplinary and ‘generic’ programmes in relation to impact. Throughout this review, there are

studies that look exclusively at a subject or group of cognate disciplines (often mathematics or

sciences), particularly in relation to CPD in schools. However, they do not make claims for the

efficacy of such programmes based on their disciplinary orientation.

22/222

Amundsen and Wilson (2012) address the distinction between disciplinary and generic CPD from a

conceptual perspective in their 6-point framework of educational development; they draw on

theoretical rather than empirical work (See section 7 for a fuller account of this research).

Rienties et al. (2013) find that discipline has an impact upon the retention of participants in an online

CPD course. Additionally, Skelton (2013), considering impact in relation to identity construction of

academic teachers, reminds us that in between ‘disciplinary’ and ‘generic’ is ‘interdisciplinary’ CPD in

which the deliberate combining and interaction of academics from different disciplines plays a role

in the manner in which they can challenge and learn from each other. He argues for the retention of

interdisciplinary CPD and suggests that there are particular affordances of ‘critical interdisciplinarity’,

particularly in terms of the personal and professional changes experienced by the teachers

themselves. (See section 10)

1.4.4 Compulsory vs non-compulsory

As in the previous category, we found scant research that explicitly investigates this comparison,

although there was reference, often in passing, as to whether participants were undertaking CPD

voluntarily or as a compulsory requirement. There is some evidence that staff on temporary

contracts adopted a ‘“defensive” faculty development’ orientation: they felt compelled to participate

in voluntary CPD in order to enhance their chances for contract renewal (Rutz et al. 2012).

Rutz et al. (2012) also found that staff on temporary contracts were less likely to experiment in the

classroom with their learning from CPD for fear of lowering student satisfaction scores; whereas

permanent staff were more willing to experiment with new tasks and techniques even though they

attended fewer CPD events. Of particular interest here is the breakdown along the lines of

temporary and permanent. Other studies tend to distinguish between novice and experienced

teachers and Rutz et al. (2012) are framing the comparison in terms of job security. This paper was

the only one we encountered that explicitly considered job security in relation to engagement with

and impact of CPD, and it would appear to be an area that warrants further attention.

23/222

Trigwell et al. (2012) observe that, over a 10 year period, compulsory CPD programmes had an

impact on teachers’ engagement with the scholarship of teaching’ when compared with those who

had not participated in the CPD.

1.4.5 Student impact/impact on student learning

In relation to assessing impact on students (2012), we have not identified any striking advance on

work reported by Parsons et al. Writers continue to highlight the difficulties of quantifying the

impact of teacher CPD upon student learning, because causality of this nature is hard to isolate

amongst the complex processes of student learning.

Nonetheless, there are studies that make claims about the impact of teacher CPD upon student

behaviour and learning. Furthermore, although self-report is still a major source of data, a wider

range of evidence has been sought in a number of mixed methods studies, such as Antoniou and

Kyriakides (2013). Control groups were also used in two studies (see below), which might lend

weight to their findings.

Not surprisingly, research in a single discipline or around a single student skill or attribute brings

student learning and attainment into focus. Examples include evaluation of teacher professional

development in relation to writing development (e.g. Willett et al., 2014), critical thinking (e.g. Shim

and Walczak, 2012), mathematics teaching and learning (Antoniou and Kyriakides, 2013) and the use

of personal response systems (Han and Finkelstein, 2013).

Trigwell et al. (2012) argue that measuring the impact of CPD programmes on student learning is

difficult and complex and that most studies that have attempted to measure these have found only

small changes. (Much of what has gone before attempts to measure shifts between deep and

surface learning.) The authors found a ‘small but significant’ indication that students’ overall

satisfaction with their course increases when taught by a teacher who has completed the CPD (as

compared with being taught by the same teacher prior to the CPD). Correspondingly, they found

that, in faculties where ‘higher proportions’ of academics have completed the CPD under

24/222

consideration in their study, students report being more satisfied with their degree courses than

students from faculties with lower proportions of teachers engaging in the CPD.

Mowbray and Perry (2015) report on a small scale, mixed methods study with 11 lecturers, which

demonstrates that a particular CPD course in improving lecturing skills has had direct impact on

student learning. The CPD took place over 6 weeks and in assessing its impact, researchers used pre-

and post-intervention questionnaires, interviews, and exam performance of students from 2

cohorts. Results showed an improvement in teachers’ self-efficacy in relation to lecturing skills.

Findings also show a ‘significant increase’ in student attainment for those students whose lecturers

participated in the CPD.

Student learning is a rather broad concept and is often associated with performance on exams or

overall achievement at the end of a course of study. Stes et al. (2013b) look at the impact of a 1-year

CPD programme for new teachers on the students’ approaches to studying, and they find little

effect. Similarly, Stes et al. (2012) reported limited impact upon students’ learning outcomes

(measured using a slightly modified version of Entwistle’s Experience of Teaching and Learning

Questionnaire, ETLQ) amongst students of teachers who had undertaken a year-long CPD course.

However, both of these studies are premised upon a deep vs surface conceptualisation of student

learning which in itself is contested and rather narrow.

Pehmer et al. (2015), in the context of STEM disciplines, claim to have proved the effectiveness of a

CPD programme and shown that students benefit. The study involved 135 student and teacher

participants. Teachers participated in a video-based programme on classroom dialogue and the

research aimed to investigate the impact of this on students' perceptions of their higher order

learning (broken down into ‘situational learning processes’ and ‘cognitive elaboration strategies’).

Likert scale statements were used with students to assess their perceptions of both their own ability

(‘self-concept’) in STEM subjects and these two higher order learning elements. Again, the focus is

on perceptions, with evidence from self-report but there is a control group of 90 in which teachers

followed a similar professional development programme but without video and this may make the

findings more persuasive.

25/222

Antoniou and Kyriakides (2013), in the context of mathematics teaching in Cypriot primary schools,

measure both 'teacher skills' and student achievement before and after a professional development

programme. They followed the study up one year later to investigate sustainability. There are 130

teacher participants. The 'Dynamic Integrated Approach' (DIA) to professional development is used

both as the basis for the CPD course and as an evaluation framework. This is a complex five-stage

model, which sets out teacher development as a hierarchy. The authors claim an association

between teachers operating at the higher DIA levels and higher student attainment. The 'before'

evaluation consisted of observation, teacher questionnaire, student tests, student questionnaire.

'After' evaluation used the same methods and measures plus a feedback meeting with teachers.

Students of teachers in the higher DIA levels made greater gains than those of teachers assessed as

in the lower levels. Students of teachers in the DIA group made statistically significant gains while

those of a control group did not. The follow-up revealed no further improvement and no decline.

In a study over 4 semesters with 74 academics and over 5000 students, Han and Finkelstein (2013)

found evidence that CPD for teachers on using clicker assessment and feedback (CAF) tools

influenced student perceptions of the technology.

Rutz et al. (2012) identify improvement in student critical thinking as evidenced in written

assignments. This mixed methods study investigates the impact of CPD (mainly unaccredited) at two

different US HEIs over 3 years. Willett et al. (2014) is a detailed paper on the same study and

advances a more tentative conclusion that student learning (clearly defined here with reference to

writing, quantitative reasoning and critical thinking) is enhanced. Shim and Walczak (2012) also

suggest a relationship between teacher practices and student development of critical thinking skills.

Additionally, Willett et al. (2014) focus on the development of a methodology for assessing impact

using student work and teacher assignments. The Haswell paired-comparison rubric was used to

analyse student texts and staff assessment prompts before and after CPD events to establish

evidence of impact. The findings indicate clear change/enhancement in academics’ practice and, as

suggested above, some evidence of impact on student learning, although the latter findings are less

strong.

26/222

Finally, as suggested in section 1 above, Lumpe et al. (2012) found that the teachers’ self-efficacy

was positively correlated to an improvement in students’ standardized science test scores. The

authors conclude that ‘self-efficacy and professional development hours were positive predictors of

student achievement’ p. 6.

1.4.6 Other references to impact not covered by the above

The literature also revealed a number of additional indicators that align to impact and transfer. Key

among them are motivation, teacher experience, place and context for CPD, networks, affordances

of online CPD in relation to impact, and time or distance from the course or intervention.

Motivation – De Rijdt et al. (2013) identify motivation as a key variable in the success of

transfer of CPD to the workplace. Han (2012), investigating ‘teacher-driven’ CPD also views

motivation as an important variable. Motivation and orientation or attitude towards CPD is a

recurrent theme the literature. (See Peters et al. (2012) at the end of this section for a

related discussion of ‘enjoyment’ and impact.)

Teacher experience - Stewart (2014) offers a tentative suggestion, based on a narrative

study, that CPD (in this case a PGCert) had more impact for teachers who had some prior

teaching experience than those who did not (This aligns with findings, similarly tentative,

from De Rijdt et al. (2013)). However, Stewart suggests that further research is needed to

ascertain whether a PGCert has a more sustained impact on participants who had some prior

teaching experience than those who are at the start of their career.

Online CPD – Rienties et al. (2013) - While this paper is not explicitly about online CPD and

its effectiveness, the authors imply that there are certain affordances in terms of online

delivery that influence impact. As suggested above, Rienties et al. found that discipline and

institutional culture appeared to significantly influence whether participants successfully

completed online CPD modules.

Creation of social networks/ communities of practice - Rientes and Kinchin (2014) identify

the impact of CPD programmes on HE teachers’ subsequent development of social network

structures and social learning relationships with colleagues. They suggest that socially co-

constructing and sharing knowledge after CPD has finished is evidence of an

undervalued/under-reported type of impact which sees participants entering into

communities of practice around teaching and learning.

27/222

Rientes and Kinchin (2014) suggest that their findings indicate that research into the impact

of CPD should be broadened beyond a consideration of ‘formal programme boundaries’ (p.

123). This research builds on Moolenaar et al. (2012) who argue that cohesive teacher

networks improved self-efficacy and had an indirect impact on student (in this case school

children’s) achievement. Furthermore, knowledge ‘spillover’ and a changing sense of

institutional culture are also identified as a significant finding by Rutz et al. (2012).

Skelton (2013) and Watson (2014) also suggest that community building as a result of CPD

should feature in a consideration of impact. Related to this, Belvis et al. (2013) found that

transfer depended on more than one person from a work context (in this case, schools)

participating in the CPD and they refer to a participant’s ‘sensitivity to impart’ their learning

to others as having an impact on transfer (although this is not described in more depth.)

Viewing the impact of CPD in relation to social and professional networks and communities

of practice introduces an important and complex dimension to the consideration of

effectiveness. As Roxa et al. (2011) suggest, institutional cultures and their dynamics and

complexities must be taken into account when considering the management of change,

especially in relation to academic development. Seemingly, such awareness of organisational

culture is important when analysing and (as Rienties and Kinchin suggest) extending the

impact of CPD.

The location of the CPD in relation to participants’ working environment may have an

impact. De Rijdt et al. (2013) suggest that ‘on the job’ CPD has a greater impact than ‘off the

job’ but indicate that this finding needs further research. Related to this is the ‘learning

climate’ and the extent to which the institutional approach to CPD is framed as

appreciative/developmental or remedial.

Similarly, Han (2012), analysing the impact of pre-school teachers’ professional development

on their ‘instructional strategy development’, addresses transfer in terms of location and

refers to 'on the job' CPD as being preferable to 'shot in the arm' (off the job) training. Han

distinguishes between ‘knowledge for practice’ (off the job) often taught by external

specialists through workshops or similar events, and ‘knowledge in practice’ (on the job)

which takes place within teaching.

The study looked at different features of CPD including ‘teacher-driven’ activities (in which

teachers determined the topics of CPD events and ‘job-embedded’ professional

development (in which the CPD took place as part of teachers’ work). Following analysis of

28/222

the pre and post intervention questionnaires, Han concluded that ‘teacher driven’ and ‘job

embedded’ are significant attributes in relation to impact on teacher behaviour. As above,

these findings address the issue of context for the delivery and engagement with CPD and

further work is warranted to understand more fully ways in which these influence the

impact of CPD.

Time – Many of the studies are evaluating impact soon after CPD has been offered and a

number of researchers indicate that longitudinal research is needed to better understand

impact over a sustained period. Recent exceptions are Trigwell et al. (2012) who look at the

impact of a CPD programme using data collected over a 10 year period and Stewart (2014)

who investigates the impact on teachers after 5–plus years.

Finally, Peters et al. (2012) bring together a number of the above factors in their research

into online training and the impact of participant satisfaction and motivation to transfer.

Satisfaction, and within this ‘enjoyment of learning’ are foci of the research. Citing Axtell et

al. (1997), Peters et al. remind us that ‘motivation to transfer’ predicts the speed and

likelihood of transfer. They also find that opportunities for interaction are linked to

enjoyment and learning outcomes, and, indirectly, to the motivation to transfer. Their

findings suggest that enjoyment as a feature of learning is an important dimension of

motivation to transfer, which in turn, predicts the likelihood of transfer.

1.5 Research in the broad area of CPD impact that addresses the construction and use of frameworks

There are a variety of frameworks designed to describe or evaluate CPD programmes and activity.

This range is due, in part, to the different value systems underpinning the work which, in turn,

influences which characteristics activities, points of view, and indicators of impact are foregrounded

and described by the different frameworks.

The development and application of a framework to evaluate CPD is described in depth in Chalmers

and Gardiner (2015) who argues that the impact of CPD programmes in HE is under-researched with

a few exceptions. Working within an Australian context, where, increasingly, education is scrutinized

for quality, value for money and range of participation, Chalmers and Gardiner (2015) argue that in

the absence of a rigorous and relevant evaluation tool, that CPD programmes will continued to be

assessed with blunt and limited instruments such as participation satisfaction surveys which do not

29/222

report on richer, contextualised impacts of the programmes. Chalmers et al.’s (2012) own

framework is designed to be used in a variety of contexts and is based upon 4 principles: relevance,

rigour, context and reliability. This is a relatively complex framework with a matrix of indicators

focusing on both programme and institutional contexts. It advocates the collection of data along a

number of specific indicators which address processes, outputs and outcomes. The framework(s)

and accompanying account are well-documented; however, potential criticisms are that it describes

CPD activity within a context without sufficient emphasis on collecting evidence to demonstrate

impact. Additionally, the student voice/learning perspective is not considered as prominently as it

might be.

Building on a ‘conceptual’ review of the literature, and with particular reference to Stes et al. (2010),

Amundsen and Wilson (2012) devise a 6-point framework intended to enable new insights into

practice design and to aid evaluation of effectiveness of CPD. Cautioning against being overly

‘narrow’ in questions about impact and effectiveness of CPD (and they argue that many previous

empirical studies are too narrow), Amundsen and Wilson set out a framework that they suggest

offers a broader account of educational development and is better suited to capture its complexity.

Their six ‘foci of practice’ around which the framework is based include skill, method, reflection,

disciplinary, institutional and action research/inquiry.

Farley and Murphy (2013) report on the early stages of an Australian project to develop an

evaluation framework for mobile learning. It would be worth following the progress of this work

since its ultimate goal is an evaluation toolkit that comprises the kinds of elements the current

project might offer: framework, guidelines, resources, examples and a ‘maturity model’. Rienties et

al. (2013) use the TPACK model as a means of gathering data but also as a conceptual framing for

their study. Various discussions of TPACK framing appear in North American schools research that

addresses CPD and impact, particularly in relation to bolstering content knowledge.

Saunders (2014) reports on the evaluation of a programme based on "instructional intelligence",

which is about developing expert behaviour (described in some detail). The conceptual framework is

the Concerns Based Adoption Model (CBAM), a complex model designed to assess teachers'

educational change. This use of CBAM includes a ‘stages of concern’ questionnaire (SoCQ) and levels

of use (LoU) interviews.

30/222

As suggested above, Trigwell et al. (2012), report on four studies into the impact of a year-long CPD

programme at a research institution in Australia, which draw on data collected over a decade.

Building on prior work (e.g. Guskey (2002) and Kirkpatrick (1998)), the researchers developed a

framework to assess satisfaction of CPD participants, the impact upon participants’ practice and

impact on student learners. Like Chalmers and Gardiner (2015), Trigwell et al. (2012) argue for the

affordances of taking into account multiple indicators of impact; in their case they looked at both

staff attainment and student satisfaction. Trigwell et al. suggest that the use of multiple indicators

can more easily identify and demonstrate enhanced impact/significance than studies drawing on

single indicators.

As an alternative to models designed to measure impact quantitatively, Bozalek et al. (2014) use the

Tronto ‘political ethics of care’ framework to evaluate CPD in a South African university. The

framework is a reflective tool which addresses 5 characteristics: attentiveness, responsibility,

competence, responsiveness and trust. This holistic, reflexive approach foregrounds the lived

experiences of teachers and the practices of the team; members of the team reflect on their own

practice in a systematic manner. This paper discusses the application of the framework in detail, and

describes some of the reflections that emerged from working through it. From an exclusively

‘impact’ perspective, this does not yield much in the way of metrics or fixed findings; however, from

a conceptual perspective, the valuing of the individual’s and cohort’s reflexivity and development is

signalled, as is the importance of values. This work offers more of an orientation towards evaluation

of CPD rather than a model for measuring or quantifying impact

Belvis et al. (2013) evaluated a CPD intervention for mathematics teachers (primary and secondary)

by drawing upon the Holistic Model (Pineda, 2002) which addresses dimensions of satisfaction,

learning, pedagogical appropriateness, transfer and impact. The data collection tools of the model

include surveys, reports and interviews with participants, colleagues and managers. The framework

is set out fully in the paper. The programme, with 284 participants from primary and secondary

education, asked teachers to systematically reflect on their teaching practice and to share their

reflections with colleagues in small groups. In this approach, teachers started with observations and

‘ended with strategic planning for performance improvement’. The ultimate aim was to enhance

classroom teaching including the use of ICT.

31/222

The authors conclude that while the ‘study shows that the education programme generates high

levels of satisfaction, pedagogical appropriateness and learning, its achievements in effectiveness

are moderate’ and there was ‘little evidence of its impact on student learning’.

Like Chalmers et al. (2012), Fink (2013) suggests that impact is difficult to ascertain despite good

intentions; evaluation of impact rarely goes beyond the measurement of participant reaction or

satisfaction. This paper explores the reasons why measuring impact is important and from whose

perspectives. It also offers a framework for considering impact (or, as it is termed here, 'assessment')

and identifies 4 perspectives (students, academics, senior managers and 'faculty development

activities') from which data might be gathered. Fink (2013) foregrounds the complexity and range of

CPD within any given institution (or sector) and argues that different approaches to collecting

information about impact are required: CPD is not a singular entity and nor should its evaluation be.

Fink describes a protocol for collecting data about the impact of CPD on teaching practices and on

student learning. Each branch of the protocol has 4 parts and the instruments can be used in any

combination in order to reflect which aspect(s) of the framework are being evaluated. The data

collection tools are based on recent research and employ a combination of qualitative and

quantitative methods. The article closes with guidance on useful analytical approaches; these

include creating comparison data, constructing measurable objectives and considering the ‘multiple

meanings of impact. In closing, Fink cites Hines’ (2009) 8 dimensions of quality program assessment:

Systemic: Creates feedback from systemic and continuous assessment.

Goal-directed: Clear program goals guide the assessment.

Measurable objectives: Objectives are designed in ways that enable measurement.

Criteria for success: Standards have been set that define the desired level of goal

achievement.

Assessment methods measure the objectives: The methods are valid measures of the

objectives.

Multiple measures: The assessment uses multiple measures of program quality.

Summative and formative data: The data gathered can serve the purposes of both

program improvement and a determination of end-of-program effectiveness.

32/222

Evidence of a causal relationship: Uses comparative data to establish a causal

relationship between program activities and its impact(s).

(Hines, 2009)

As suggested above, Guskey’s work (2002) still features strongly in framework development. McGee

et al. (2013) critique and expand Guskey’s framework as a means of evaluating the impact of CPD;

they describe in detail questions and data sources devised to gather evidence for each level of the

resulting framework.

Also drawing upon Guskey and foregrounding reflection, Postareff and Nevgi, (2015), report on a

study that addresses the impact on individual teachers of a CPD course and analyses the pathways

that 18 participants took through the course by researching their reflective diaries in an attempt to

understand the complexity and variety in experiences. They identify 5 development pathways and

analyse findings in light of Guskey. They suggest that teachers need opportunities to experiment

with new pedagogic methods in order for the CPD to have impact. They also find that the motivation

of teachers to ‘develop conceptions and understanding of learning’ aids potential transfer.

Finally, Willett et al. (2014) offer a nuanced, well-illustrated discussion about the affordances and

deficits of their approach to evaluating CPD and the implications for evaluating the impact of CPD in

other HEIs. In particular, they address the challenges of transferring evaluation techniques and

frameworks developed for a particular institution into a new setting.

1.6 Research that goes beyond impact of CPD but which may be relevant more broadly

Recent research offers many observations about academic development and CPD that sits outside of

a consideration of impact, but is potentially relevant for the current study, nonetheless. We also

include here suggestions in recent literature for future research.

1.6.1 Reconceptualising academic development

Boud and Brew (2013) argue for a reconceptualization of academic development and suggest that it

should be viewed more as a social practice with a focus on the extent to which there is a cultural

33/222

shift in the institution as opposed to the development of individual teachers. This gaze might usefully

inform future frameworks. Amundsen and Wilson (2012) also observe this distinction in the

orientation of CPD in their discussion of ‘contextual positioning’.

1.6.2 Impact of networks and brokers

Rienties and Kinchin (2014) suggest that future research should investigate the complex roles of ‘key

brokers inside and outside’ CPD programmes in HEIs who help with ‘knowledge spillover’ and

support the creation of links with colleagues beyond the original CPD programme. There is some

synergy here with De Rijdt et al.’s (2013) tentative findings that novice teachers benefit and show

more transfer of learning when they form collaborations with more experienced teachers. This

thinking is in keeping with work by Roxa et al. (2011) who explore academic development and

change in relation to the power of university cultures.

1.6.3 Narratives

Similarly, Stewart (2014) (building on Stronach, 2010) while recognising the need for large scale

studies into impact, argues that ‘the quest for generalisability’ will necessarily omit ‘significant

personal impacts’. The research, based on analysis of HE teachers' narratives some years after their

participation in a formal CPD programme, is unusual in its focus on long term impact. Significantly,

Stewart (2014) cautions against an over dependence upon the discourse of impact and

measurement, and suggests that a rich understanding of how teachers orient themselves to their

CPD over time (> 5 years) helps illuminate shifts in personal development of teachers, and provides a

‘fuller understanding of teacher change’.

Although Stewart describes this study as moving away from impact evaluation, its consideration of

sense-making, particularly how university teachers make sense of the course as their careers

progress, is highly relevant to a broad view of impact.

1.6.4 Impact of diverse participation groups

Some studies suggest that diversity amongst participants may influence the outcomes of teacher

professional development. Boman (2013) investigates outcomes of teacher development workshops

for graduate teaching assistants in Canada. The mixed methods (self-report, pre- and post-test,

observer coding) enquiry considered self-efficacy, confidence, and teaching behaviours and found

variations between home (Canadian) international participants and between those who had

34/222

previous teaching experience and those who had not. Lau and Yuen (2013), in a survey-based study,

found changes in beliefs about learning technologies were greater in ‘younger’ than ‘more senior’

mathematics teachers.

Herman (2013) suggests that the ratio between teaching and learning development unit staffing and

institutional FTE and student numbers is an indicator of institutional commitment to teaching and

learning. This is questionable but the data is easily obtained so it might be worth considering.

1.6.5 Identity construction, boundary crossing and impact

Postareff and Nevgi (2015) explore identity construction and boundary crossing of CPD participants

looking at four indicators: identification, coordination, reflection and transformation. In this paper,

these dimensions are presented as a feature of successful CPD design and participant orientation;

however, this could be adapted and developed as potential indicators of impact in relation to

teacher development. So, these indicators might be borne in mind when developing future

frameworks. Another implication of the study appears to be that those designing and leading CPD

should be attentive to sociocultural and disciplinary differences of the attendees.

Similarly, Skelton (2013) writes about a two –year Masters course at a research intensive HEI which

takes a critical, reflexive, interdisciplinary approach to CPD. Exploring the implications for

participants through semi-structured interviews, Skelton finds that a key impact is that of

transformation in terms of personal and professional identities with an enhanced sense of self-

confidence in terms of professional practice; stronger theoretical underpinning of teaching and

enhanced self-awareness. A potential negative impact of CPD identified here is the slight suspicion

that a focus on teaching might carry with participants’ peers at a research HEI. With reference to the

recent HEA UKPSF Impact study (Turner et al., 2013), which recommends more discipline-based CPD,

Skelton argues for the affordances and impact of interdisciplinary (as distinguished from disciplinary

or generic) critique and awareness which emerge in this programme.

Van den Bos and Brouwer (2014) take a broadly similar approach to Postareff and Nevgi (2014) by

following ‘novice’ university teacher journeys (albeit over a shorter period – 5 months) during a CPD

programme, using interviews and digital logbooks to gather data. As part of the study, they use

dilemmas (with quantitative and qualitative responses) as a means of evaluating participants’

learning on the course and understanding ‘what’ was learned and ‘how’. They advocate Clarke and

Hollingsworth’s (2002) ‘interconnected model of personal growth’ as a framework for analysing

35/222

research data of this nature and perhaps this would be worth investigating further for the current

project.

Finally, Watson (2014) draws upon social learning theory to analyse a case study of a single teacher;

the context is secondary school teaching and the specific focus is problem solving. The theory, as

described, views learning as involving observation, self-efficacy and ‘reciprocal triadic determinism’,

which is defined here as the relationships between ‘individual thinking and beliefs, the social context

and individual behaviour’. Watson argues that this approach could be a useful evaluation tool, and

aims to contribute to the theorising of professional development.

1.7 Critique of the impact discourse

There is much meta-level analysis and critique of the impact discourse as it relates to measurement

in higher education, particularly academic development, in recent literature.

Stefani (2013) laments the ‘obsession with measuring performance’ across HEIs and especially in

relation to academic development. Nonetheless, she recognises the demand for accountability and

observes that the academic community is increasingly expressing interest in ‘meaningful

frameworks’ for measuring performance. She cites the CADAD 2011 benchmarking statement with

its eight key domains, as areas to consider, and in relation to benchmarking performance and

measuring impact, she asks

‘How do we conceptualise our contribution within the ‘big picture’ of organisational goals

and the overarching purposes of higher education in the twenty-first century?

Would we set our performance measures at the operational level, at the strategic level or

both?

Is it time for a shared, fit-for-purpose narrative for academic development for the twenty-

first century?

Bozalek et al. (2014) argue that many models for measuring impact of CPD, while not explicitly

addressing their underlying values, are nonetheless aligned with a neoliberal approach to evaluation,

focussing on ‘efficiency, measurability and individualism’ (p. 447). As an alternative, they develop a

36/222

‘political ethics of care framework’ based on Tronto (1993) as a means of analysing individuals’

response to professional development (See section 1.5 above).

Similarly, Mockler (2013), critiques the dominant discourse surrounding teacher CPD, particularly the

Australian ‘teacher quality agenda’ which he aligns with a neoliberal framing of education and one

which risks becoming fixated on measurement and regulation which is ultimately damaging to the

teaching profession. Citing Sachs (2003), Mockler distinguishes between CPD that is aligned with

‘managerial professionalism’ and that which is associated with ‘democratic professionalism’. The

latter is determined by teachers, relies on teachers’ professional judgement and is characterised by a

level of trust.

Arguing that conceptualising CPD as ‘identity’ work is a more productive orientation for evaluating it,

Mockler (2013) raises a series of questions and challenges for future work. These would be worth

bearing in mind when developing the framework and will help guard against creating a model which

could be perceived as overly technicist and/or so narrow that it omits the teacher perspective.)

Adopting a broader critical perspective, Di Napoli (2014) explores the tensions inherent in academic

development, arguing that those working in the field often find themselves operating in contexts

that are ‘saturated’ with neoliberal discourses of marketization which are ‘shot through’ with

superficial notions of what ‘good’ learning and teaching are. He reminds us of the complex nature of

academic development and the extent to which it differs according to context and he warns against

unwittingly relinquishing personal and professional values that have traditionally characterised the

profession through internalising a culture of compliance and conformity, rather than that of

challenge. The quest for ‘frameworks, roles and regulations’ to apply and measure activity around

academic development, Di Napoli(2014)suggests, risks eclipsing ‘imagination, reflection and

creativity’. In particular, he argues that it is important to ‘unpack how [the] discourses of care,

support, and criticality meets those of efficiency and accountability’.

This discourse of critique, particularly in relation to frameworks and measurement, is an area not

really explored in Parsons, yet, as Stefani (2013) notes, it is important to acknowledge and address

37/222

this tension and remain alert to it in order to garner and maintain the support of academic

developers. As Jordan (2014) argues, it is important to resist a reductive, ‘instrumental mindset’

when evaluating impact and, instead, to remain alert to a broad range of ways in which the effect of

academic development can be manifest.

1.8 Conclusion

1.8.1 UKPSF, teaching excellence and CPD frameworks

This project is an enquiry into how the effects of CPD can be evaluated over time to help understand

its impact for teachers, students and institutions. Given this, it is useful to consider how the research

on the evaluation of CPD intersects with the broader discourse and policy related to CPD for

teachers, including the UKPSF and recent research on excellence. Professional development,

teaching excellence, career progression and impact of CPD are all areas that are intertwined in

experiences of academics, yet, as Gunn and Fisk (2013) and others have observed, the connections

between them are often insufficiently explored in policy, frameworks and research.

The UKPSF is one such example. This Framework is intended to describe attributes and values

associated with professional practice for those engaged in teaching and supporting learning in UK

HE. The impact of this UKPSF itself is variable across the HE and FE sectors, with senior managers

tending to suggest that it has informed CPD strategies, policy and practice, while subject academics

indicate that the framework has had rather less impact (Turner et al., 2013). The UKPSF does not

prescribe CPD or its intended impact; rather, as Gunn and Fisk (2013) suggest, the UKPSF is largely a

benchmarking tool that does not ‘measure’ or record excellence as people progress in their careers.

Nonetheless, increasingly, CPD courses, processes and events in UK HEIs are mapped against the

UKPSF which is therefore arguably asserting considerable influence in the scope and development of

CPD. Interestingly, however, there is relatively little mention of the UKPSF in the literature reviewed

in this report.

38/222

Furthermore, as Gunn and Fisk (2013) and Turner et al. (2013) argue, it is difficult to align the UKPSF

to excellence at different stages of careers in the way that the Vitae Early Career Research

Development framework enables the mapping of research development. Gunn and Fisk (2013) urge

an alignment between frameworks that articulate excellence at all stages of the academic career,

including those that map research development and those which may map or guide excellence in

teaching. An awareness of the various frameworks, policies and discourses that address the full

spectrum of the academic career will be important for a framework that evaluates the impact of

CPD; in other words, CPD should not be considered in isolation from other elements of the academic

role.

Indeed, the consideration of career progression is implicit in a number of the studies cited above but

not always articulated explicitly in the extent to which it relates to CPD. Throughout, the literature

discussed here is generally addressing CPD as something which enhances teaching and student

learning outcomes. Gunn and Fisk (2013) argue that a significant gap in literature, discourse, policy

and practice is the relationship between models of teaching excellence, CPD frameworks and the

acknowledgement that academics’ careers vary over time. They argue for a robust methodology for

‘analysing the links between teaching excellence and student learning outcomes’ and they suggest

that such approaches should also be cognisant of the various roles and stages of an academic career.

We suggest that the impact of CPD could usefully be included in such methodologies. Likewise,

frameworks which model and guide the evaluation of CPD should also be mindful of career

progression and the ways in which the impact of CPD articulates with teaching excellence.

1.8.2 Gaps in the literature

There are a number of gaps in the literature in relation to studies which evaluate the impact of CPD

in HE teaching. Significant gaps include

Little research that considers the relationship between engagement with the UKPSF and

impact of CPD. As suggested above, other recent research has called for greater integration

between frameworks and discourses which benchmark teaching and research practices and

those that articulate teaching excellence. The same dislocation appears to hold in relation to

CPD and its impact.

There is not a sufficient body of research on the complexities and range of contexts (e.g.

institutional, disciplinary) and the way in which ‘impact’ of CPD might be both determined

39/222

and evaluated in relation to context (Work that might inform further research in this area

includes Roxa et al. (2011) on teaching and learning cultures and Gunn and Fisk, 2013 who

take a critical and nuanced view of the literature on teaching and teacher excellence).

Similarly, there is not a great deal of research on understanding the complexity and

challenges of collecting evidence related to impact of CPD. Cashmore et al. (2013) address

this issue in relation to the amassing of evidence around career progression in UK HE and the

same principles could be applied to identifying and collecting evidence to critically evaluate

the effectiveness of CPD.

There appears to be relatively little work on how students’ directly experience the impact of

CPD undertaken by teachers. Studies that focused on student perspectives and experiences

would be a valuable addition to the literature.

1.8.3 Key points and implications for future work on evaluating CPD

CPD is a wide-ranging term and how it is defined and used has implications for what can be

researched and claimed.

The contexts for CPD – both where and how it is delivered – and the broader institutional

orientation to it, influence its impact.

‘Student learning’ has diverse meanings in the research – ranging from specific evidence of

critical thinking to orientations to deep and surface learning implicit in study behaviours.

Understanding and precisely describing what is meant by student learning is critical in

investigating CPD.

Students’ voices tend not to be heard directly in research on impact.

Assessing the impact of teacher CPD on students can build on existing descriptive

frameworks or established associations between teacher behaviours and student attainment

or perceptions, and even use these to create an analysis framework. Examples include the

use of the DIA (Antoniou and Kyriakides, 2013), and CBAM (Saunders, 2014).

The impact and influence of networks – both online and within institutional contexts – are

worth further investigation.

There is a wide range of frameworks in recent literature – both conceptual and practical and

with different aims and underpinning values – which can potentially inform future

evaluation and research. Established frameworks such as Guskey’s (2002) and Kirkpatrick’s

(1998) are still widely drawn upon. Additionally, a number of studies, as suggested in

section 1.5, have extended these frameworks OR devised alternative ones, many of which

have the potential to help interpret and relate findings across studies.

40/222

Seeking ‘evidence’ is a feature of work in both teaching excellence and the UKPSF, further

supporting the need to link CPD impact evaluation with these.

This framework should relate to the UKPSF and this relationship should be explicitly

described.

Teacher self-efficacy, although disregarded in some research as not demonstrating ‘impact’,

has nonetheless been shown by Lumpe et al. (2012) to be a positive predictor of student

achievement.

Time – although there is much talk about the need for longitudinal studies, these can be

difficult to plan. However, both Trigwell et al. (2012) and Stewart (2014) offer ways of

analysing existing datasets to observe impact over time.

Finally, the use of multiple indicators in analysing impact appears to offer a richer picture of

what is happening as Trigwell et al. (2012) suggest.

1.9 Final thoughts

Impact is a difficult concept to define and measure, and, as with ‘excellence’ there is not consensus

on the meaning of the term (Gunn and Fisk, 2013). For some, impact is about quantifying a change in

student attainment or a change in study behaviours; for others it entails understanding the extent to

which teachers’ reflections of their changing practice over time signal ‘impact’. Another way of

conceptualising impact involves examining teachers’ engagement within peer networks following

CPD.

Additionally, as Fink (2013) observes, an assessment of ‘impact’ can vary according to the standpoint

from which it is measured: the student, teacher, staff development unit, institution, etc. There are

clearly issues of power at play, potentially, when we begin to consider who determines what counts

as ‘impact’ and how it is to be measured and by whom. In other words: Whose impact is it, anyway?

This breadth within the literature lends both complexity and richness to a potential framework for

evaluating CPD. Identifying methods to assess impact from multiple perspectives would be an

important feature of future work.

Finally, as noted by Cashmore et al. (2013) in relation to understanding teaching excellence,

‘evidence’ for understanding impact is complex and challenging to collect. Bearing this in mind, a

41/222

framework for the evaluation of CPD should be alert to this complexity and accommodate, as flexibly

as possible, a range of dimensions (such as time, motivation, experience, point of view, communities

of practice, student learning, etc.) when scoping, exploring and describing impact.

Work Package 2 (WP 2): Invite institutions operating CPD schemes to provide evidence of current practice in measuring the impact and effectiveness of CPD provision.

This WP consisted of two main elements. Firstly, a national audit was conducted to capture data

from HE providers regarding the impact and effectiveness of current CPD provision. We then

identified examples of innovative approaches to the evaluation of CPD activities; these were the

subject of further inquiry that formed the basis of the four case studies presented in section 2..8.

Below we provide an overview of our approach to the development of the data collection

instruments, and the procedures for data collection and analysis.

2.1 National audit

A national audit was identified in the bid as an appropriate mechanism through which we could

gather data regarding teaching-related CPD provision from the varying contexts in which HE may be

delivered. A similar approach was used by Chalmers et al. (2012) to gather data relating to the

evaluation of teaching preparation programmes. Given the timeframe of the project, we felt that

adopting a similar methodology would be an efficient method of garnering insights into the range of

CPD activities and approaches to their evaluation, to inform WP3.

42/222

2.1.1 Themes addressed within the audit

An initial review of research drawn on in developing the proposal (e.g. Cashmore et al., 2013;

Chalmers, 2011; Chalmers et al., 2012; De Rijdt et al., 2012; Guskey, 2002; Kandlbinder & Peseta,

2009, Kreber & Brook, 2001, Parsons et al., 2012 and Turner et al., 2013), underlined the diversity of

CPD provision and highlighted a distinction between CPD provision for those new to university

teaching and the CPD activities with which their established colleagues engage. For early career

lecturers, the majority of CPD is focused around structured teaching preparation programmes

designed to introduce them to the practice of university teaching (Smith, 2011; Parsons et al., 2012).

Established academics more commonly engage with CPD activities such as workshops, conference

attendance, activities related to the scholarship of teaching and learning and, more recently,

schemes to provide recognition of their teaching expertise (Parsons et al., 2012; Spowart et al., in

press). This broad distinction according to career stage highlighted a need to capture data around

both institutions’ formal CPD offer including (e.g. accredited courses) and their informal / non-

accredited CPD offer. From the start of the project we were aware of the need to consider CPD for

teachers at different career points. This was supported by findings from the literature review (WP1),

in which a tendency to overlook relationships between career progression and CPD was noted (see

section 11.1).

Given the overall focus of this project on evaluating the impact of CPD, we were primarily interested

in capturing information regarding the methods used to evaluate provision. We drew on the ideas

of Guskey (2002), Chalmers et al., 2012 and others to ascertain the focus of respondents’ CPD

evaluation activities, for example whether they were interested mainly in participants’ satisfaction or

trying to gain insights into other areas of impact such as student learning, or beliefs around teaching

and learning. Integral to this are the methods used to evaluate provision (e.g. questionnaires,

interviews), the data drawn on as part of the evaluation and the timeframe over which the

evaluation takes place (Bamber, 2013; Rust 1998). We were also interested in ways in which

evaluation data might be used. Recent work by Roni Bamber and colleagues around Evidencing

Value indicates this is an aspect of educational development practice that needs further

consideration. Respondents were therefore asked to identify how the impacts of CPD provision

were disseminated through institutional systems and structures (e.g. policies, conferences, staff

appraisals).

43/222

Much of the existing work around teaching-related CPD is centred on the impact on staff, with the

benefits to students, though implied, rarely evidenced (Parsons et al., 2012). Whilst we were not in

a position to address this gap through the audit, we were interested in the contribution students

made to teaching-related CPD as, through initiatives such as the Student Engagement Partnerships,

we were aware of the increased prominence given to student input to activities such as curriculum

design, staff CPD and policy development (BIS, undated). We felt there was considerable potential

for students to take an active role in CPD and evaluating its impact, and by including this in the audit

we might gain insights regarding the extent of student participation.

Reward and recognition was the final area identified for inclusion in the audit. Reward and

recognition can motivate individuals’ engagement with CPD (Cashmore et al., 2012; Parsons et al.,

2012). Recent studies (e.g. Spowart et al., in press) acknowledge the opportunity to gain

accreditation and recognition for teaching experience as a driver behind established academics’

involvement with CPD activities. More widely, there can be overlap between CPD and some of the

strategies employed to reward an individual’s contribution to teaching and learning, for example

development awards (Turner & Gosling, 2012). Therefore gaining insights into institutional reward

and recognition policies, and possible connections with CPD activities, was seen as essential to

understanding institutional framing of CPD.

2.1.2 Audit audience

The audit targeted those in an educational / staff development role, working within a UK institution

with a remit to provide HE. These staff were likely to have a remit for teaching-related CPD, to be

cognisant of relevant professional bodies (e.g. SEDA / UKPSF) and to be able to provide insights into

evaluation practices and institutional impacts of their work. They were also a readily accessible

sample population due to the presence of a number of national and regional online mailing lists

through which we could disseminate the audit.

2.1.3 Audit development

The audit underwent a rigorous development and piloting procedure, with feedback solicited from a

number of groups including the Project Team, the HEA Project Steering Group and the project’s

44/222

‘critical friend’, Nancy Turner. The audit was revised three times; in each iteration feedback was

carefully considered and responded to.

Ensuring relevance to the diversity of HE providers in our target group had implications for the

length of the questionnaire. The final audit contained 72 questions. To maximise response rates and

minimise completion time we asked mainly closed, multiple choice questions but, where

appropriate, there were options for respondents to provide further detail or to qualify their

responses.

Prior to the final pilot the audit was converted into a digital format using the Survey Monkey online

questionnaire platform which is a widely used and easily accessible survey tool.

The full audit is in Appendix 1. To aid interpretation, it captured contextual data around the

following themes in addition to those areas discussed above:

Demographic information (e.g. institution type)

Format of CPD provision (e.g. modes of delivery, intended outcomes (for students, staff and

the institution), participation (voluntary / compulsory);

Support and guidance for the CPD provision;

Engagement with professional frameworks and bodies (e.g. UKPSF / SEDA);

Respondents were asked to provide this general information about their context; however, the main

focus of the audit was on the evaluation of their accredited (e.g. accredited courses for new

lecturers / recognition frameworks) and non-accredited provision (e.g. action research projects,

conference attendance, peer review etc.). Guidance was offered to respondents to support them in

completing the audit and in identifying what, within their CPD offer, might be relevant to include in

the survey. Respondents were also invited to leave contact details if they were willing to be involved

in the development of case studies of innovative evaluation practices from across the sector.

2.1.4 Dissemination of the audit

The audit was disseminated (primarily using electronic mailing lists) through the HEA and SEDA, and

networks including the Friends of the North PGCert Group, South West Educational Developers

Forum (SWEDF), Scottish Higher Educational Developers (SHED), the Head of Educational Developers

Group (HEDG) and the Association of Learning Technologists (ALT). The project team drew upon

45/222

their membership of the majority of these networks in order to disseminate the survey and

encourage participation. The survey was open from the 4th to the 17th of February 2015.

2.1.5 Analysis of audit data

In total 189 responses were received out of an approximate population of 1150, equating to a

response rate of approximately 16%. 142 of these were included in the analysis after removing

responses which were either not relevant (for instance, from non-UK institutions) or were

incomplete. Respondents were then categorised to facilitate analysis based on institution type, with

the categories listed in Table 2.1 applied to the data set. Similar types of institutions were grouped

into categories to allow for deeper investigation.

Table 2.1: Categories of institutions and responses

Descriptive statistics were used to review the data. Further analysis was conducted to identify

potential relationships between institution type and specific themes within the audit, for instance

reward and recognition. However, this stage was limited to respondents from category A and B

institutions, where there were sufficient responses for a meaningful analysis. The outcomes of this

audit are reported in detail in the next section, and directly informed the development of the tool

through WP3.

46/222

Category

Institution N

A Old/Post-1964/Russell group 47

B Post-1992/Million +/New 2010/University Alliance

55

C FE College 17

D Private Provider 5

E Unaligned/Unspecified 18

Total 142

2.2 Results of the National Audit3

2.2.1 Overview of respondent profile

The aim of this audit was to gain deeper insight in to CPD provision in the HE sector, overall, and

hence results are presented on a sector-wide basis. However, since the number of responses from

the research-intensive and teaching-focused categories of institutions are both significant and

similar, comparisons are made between them where relevant. College- based HE providers are not

included in these comparisons because of their lower response rates and associated issues of

representativeness. Instead, results about colleges are commented on separately. A number of

respondents chose not to provide details about to which group their institution belonged. Others,

categorised themselves as private providers or unaligned with any group but were too small in

number for findings to be generalizable. Hence all three types of responses were combined and

included in the overall analysis as part of an ‘Other’ category.

2.2.1.1 Teaching qualifications and minimum requirement for teaching–related CPD

Even with the introduction of the revised UKPSF, not all institutions link “formal” teaching-related

CPD with gaining teaching qualifications (Parsons et al., 2012). More recently however, there has

been a shift in this trend, especially for new academic staff. Hence, examining where the sector

stands with respect to staff qualifications and policies on mandatory teaching-related CPD, offers a

useful background with which to explore CPD provision in more detail.

Staff with HE-specific teaching qualifications (N = 137)

63% of respondents were aware of the percentage of staff within their institution who held relevant

qualifications to teach higher education. In particular, 49% of the staff from research-intensive

institutions were familiar with these statistics. A much higher percentage, 78% of staff from

teaching-focused institutions, were able to provide these details.

3 Notes: 1. To improve readability, all figures have been rounded to the nearest whole number. 2. The number of respondents to each question of the audit differs. This number has been specified in thesub-headings of the report for better interpretation of percentage values. 3. Response options to many questions in the audit were not mutually exclusive, i.e., participants were free to select more than one option. This may be useful to bear in mind when interpreting percentages.

47/222

A significant point-biserial correlation revealed that higher percentages of qualified staff tended to

be reported at teaching-focused institutions rpb = .466, p < .01. Further, chi-square revealed a

relationship between institutional category and whether qualified/recognised staff were a majority

(over 50%) or minority (less than 50%) within the institution χ2 (1) = 12.8 at p < .05 . Significantly

fewer research-intensive institutions had more that 50% of their staff qualified with an HE-specific

teaching award. However, because of the generally lower levels of awareness about staff’s teaching

qualifications amongst research-intensive institutions (with a higher number “not sure”), these

results need to be interpreted cautiously and confirmed with bigger samples.

41% college-based HE providers (N=17) were able to give details about the percentage of their staff

with teaching qualifications. In 4 of these 7 colleges, a majority of staff held an HE-specific

qualification.

Minimum teaching-related CPD (N = 106 – 113 for different teaching roles)

Respondents also provided information about the minimum teaching-related CPD required for

different roles. Overall, only 3% of institutions reported that they did not have a minimum CPD

requirement for incoming staff with teaching responsibilities and no prior teaching experience. In

contrast, for staff in other teaching and learning roles, a higher number of institutions reported

having no minimum CPD expectation (Table 2.2).

Table 2.2: Percentage of institutions with no minimum teaching-CPD requirement

RolePercentage of institutions with no minimum CPD requirement

Percentage of research-intensive institutions with no minimum CPD requirement

Percentage of teaching-focused institutions with no minimum CPD requirement

Staff in learning support roles

25 37 19

Staff in leadership roles 23 36 17PG students 20 21 15Existing teaching staff 20 31 9Associate lecturers 16 15 19Incoming staff with teaching experience

9 15 2

Incoming staff with no teaching experience

3 5 0

The most commonly reported CPD expectation for different roles is presented in Figure 2.1.

48/222

Figure 2.1: Minimum teaching-related CPD requirement in different teaching roles

Expectations from incoming staff new to teaching is more consistent across the sector with a

requirement to complete an accredited course (79%). Postgraduate students with teaching

responsibilities form the only group where the minimum CPD expectation is commonly a non-

accredited activity (39%). For all other roles, including staff with leadership responsibilities in

teaching and learning (32%), staff in learning support roles (40%), existing teaching staff (31%) and

Associate Lecturers (32%), the expectation for CPD is usually decided on a case-by-case basis.

However, it is almost as common for existing teaching staff (27%) and Associate Lecturers (29%) to

be required to complete an accredited course (both percentages have been included in the figure).

This is reflective of some institutional differences in the sector.

Teaching-focused institutions appear to be more instrumental in setting an expectation around

engagement with CPD. 46% of teaching-focused institutions and 41% of teaching-focused

institutions expected existing staff and staff in teaching related leadership roles, respectively, to

have undertaken an accredited course. Within research-intensive institutions, 18% required existing

staff to have completed an accredited course and 10% required staff in teaching related leadership

roles to have engaged with an accredited course.

49/222

Minimum CPD within college-based providers displays a similar trend to the overall sector.

Requirements tended to be decided on a case-by-case basis to a large extent. More detailed

comments cannot be made because a large proportion of respondents (ranging from 8% to 31%

relative to different roles) reported being unaware of their institution’s expectation around CPD.

Box 2.1: Teaching Qualifications – minimum CPD requirements

2.2.2 CPD Provision (N = 109)

2.2.2.1 Types of teaching-related CPD

Findings revealed that workshops, peer-reviews schemes, conferences and accredited programmes

were the most common forms of teaching-related CPD within the sector (see Table 2.3).

Table 2.3:Types of CPD from most to least commonly reported

Type of CPD Percentage who reported offering this type of CPD

50/222

Staff in teaching-focused institutions appeared to be more cognisant of the percentage of staff with HE-specific qualifications within their institution compared to staff in research-focused institutions. Teaching-focused institutions reported higher percentages of staff with relevant teaching qualifications although these results are to be interpreted cautiously.

Across the sector, there is an expectation for new staff without any teaching experience to engage with an accredited course as part of their CPD.

For more experienced staff and staff in leadership roles, their CPD requirements tended to be decided on a case-by-case basis.

Teaching-focused institutions were comparatively more consistent in setting an expectation around CPD for staff in different teaching and learning roles (including experienced staff), compared to research-focused institutions.

Workshops 92Peer-review and observation schemes 87In-house teaching and learning conferences 86Accredited academic programmes for teaching

85

Mentoring schemes 79Communities of practice around teaching and learning

60

External teaching and learning conferences 50Funding for PG study 46Action research 45Teaching consultations 38

In addition, through free-text comments, respondents also cited printed and online resources,

institutional journals, teaching fellowship schemes, writing retreats, coffee and paper sessions and

informal processes which encourage reflective practice, as a contributor to teaching-related CPD.

2.2.2.2 Examples of accredited and non-accredited CPD

The audit asked respondents to select any one CPD offer at their institution and provide details

about it. 71% selected an accredited CPD offer (N = 100) and 23% described a non-accredited CPD

offer (N=33). Approximately 6% reported that their institution did not provide any teaching-related

CPD. These were discounted.

75% of the examples of accredited CPD offers focused on Postgraduate Certificates in Higher

Education (PGCHE) leading to recognition with the HEA. The remaining examples were about

accreditation with SEDA, GTA and Master’s degrees in Education (MEd). Examples of non-accredited

provision consisted of teaching and learning conferences (32%), workshops (29%), scholarship of

teaching and learning activities (14%) and short courses (11%) (see Figures 2.2 and 2.3)

51/222

GTA 22%

PGCHE 75%

MEd 30%

Recognition with HEA

73%

Recognition with SEDA, 7%

Figure 2.2: Accredited CPD offers described by respondents

Respondents could select more than one option e.g. PGCHE and recognition with the HEA

Workshops 29%

Short courses

11%

Internal teaching and learning conferences32%

Peer review

7%

Mentoring -community of practice

4%

Scholarship of teaching and learn-

ing 14 %

Training / away days, 4%

Figure 2.3: Non-accredited CPD offers described by respondents

Staff targeted for the CPD offer (N=94 for accredited offers and N=29 for non-accredited offers)

Accredited offers were largely targeted at incoming teaching staff (88%), staff without a teaching

qualification (78%) and staff who support teaching and learning (75%). Non-accredited offers largely

52/222

targeted teaching staff with three years and more experience (86%), and to a lesser extent, staff in

other roles (staff without teaching qualifications 66%; incoming teaching staff 62%).

Focus and key aims of CPD offers (N=93 for accredited offers and N=28 for non-accredited offers)

Overall, the majority of CPD offers are cross-institutional. This is encouraging considering the

benefits of inter-disciplinary CPD that have been emerging from the literature, specifically around

sensitising staff to contrasting teaching approaches (see Parsons et al., 2012). However, there is a

significant association between type of CPD offer (accredited or non-accredited) and whether the

offer is cross-institutional or discipline-specific χ² (1) = 12.95 at p < .01. Specifically, more non-

accredited CPD offers were tailored to specific disciplines.

Accredited CPD offers primarily focussed on developing teacher skills and changing teacher practice

(98%). Yet, unlike non-accredited offers, they were also concerned, to quite a large extent, with

enhancing student learning (77%) and developing an understanding of institutional policies and

culture around teaching and learning (66%). This is not surprising considering that they primarily

targeted new staff and staff in learning support roles (see Table 2.4).

Table 2.4: Focus of accredited and non-accredited CPD offers

Focus Accredited offers Non-accredited offersTeacher-focused 98% 93%Student-focused 77% 43%Institutional-focused 66% 46%

Content of CPD offers (N=93 for accredited offers and N=26 for non-accredited offers)

Content for accredited teaching-related CPD was largely informed by external professional

frameworks (UKPSF,SEDA, discipline-specific frameworks etc.) (91%), institutional priorities (82%)

and evidence-based best practice (74%). Content for accredited teaching-related CPD was largely

informed by external professional frameworks (UKPSF,SEDA, discipline-specific frameworks etc.)

(91%), institutional priorities (82%) and evidence-based best practice (74%). Institutional priorities

also play a big role in informing content of non-accredited offers (81%), followed by educational

development unit priorities (54%) and teaching and learning committee priorities (54%).

53/222

Delivery of CPD (N=92 for accredited offers and N=27 for non-accredited offers)

Most often, non-accredited offers tended to be delivered completely face-to-face (73%). 53% of

accredited offers involved blended learning while 45% were completely face-to-face. The offers were

largely delivered by staff dedicated to providing CPD and academic staff within faculties for both

accredited and non-accredited provision (see Figure 2.4). Input from previous participants (54%) and

use of web-based resources (53%) appeared to be commonly integrated in to accredited offers. On

the other hand, talks by external guest speakers (44%) were common in non-accredited offers. Few

respondents indicated some level of student involvement in both accredited (17%) and non-

accredited offers (30%).

Staff ded

icated

to provid

ing CPD

Academ

ic staff

with

in facu

lties

Profes

sional

servic

e staff

Previous p

articip

ants

Extern

al guest

speak

ers

Web-base

d resource

s/onlin

e deliv

ery

Studen

ts0

102030405060708090

100 Accredited

Non-accredited

Figure 2.4: Persons involved in the delivery of CPD

Box 2.2: CPD Provision

54/222

Teaching-related CPD provision most commonly took the form of workshops, peer-review schemes, conferences, mentoring schemes and discussions and collaborative activities around teaching and learning through communities of practice.

Accredited offers aimed at catering to the professional development needs of new staff and staff in learning support roles. There was a strong emphasis on the teacher or participant (developing skills/knowledge/conceptions), but provision also focused on student needs and institutional culture. These offers were driven by external professional frameworks, institutional priorities and evidence-based practice.

Non-accredited offers targeted experienced teaching staff as well as engaged staff in other teaching roles. Their aims focused on the participant, and to a lesser degree on students and the institution. Teaching and learning priorities of the institution, educational development units and teaching and learning committees, informed the content of non-accredited offers.

Both accredited and non-accredited offers were usually delivered by staff dedicated to

2.2.3 Evaluation

2.2.3.1 Longitudinal evaluation and tracking changes to scholarship of teaching and learning in staff

Overall, 15% of respondents (N=109) reported evaluating at least one of their CPD offers

longitudinally. 57% reported they did not evaluate any CPD longitudinally and the remaining were

unsure. Further, 15% (N=108) tracked changes in scholarship of teaching and learning in staff

undertaking teaching- related CPD while 84% reported that they did not.

Programmes leading to HEA accreditation, funded research projects and teaching fellowship

schemes were provided as examples of offers which were evaluated longitudinally and where

participants’ engagement with SoTL was tracked. Evaluations included mentor catch-ups with

participants, questionnaires with participants, feedback from discipline-specific learning and

teaching committees and analysis of data in annual teaching and learning department reports.

Examples of how SoTL was tracked included staff e-portfolios for remaining in good standing and in

one institution, an annual scholarship survey for all staff.

We know from the audit results on the types of teaching CPD (Table 2.3) that communities of

practice around teaching and learning are fairly prevalent (60%). Yet there is no evidence that these

more unstructured, embedded SoTL activities within departments and the institution are tracked

and evaluated.

Box 2.3: Key findings about evaluation of specific CPD activities

55/222

Teaching-related CPD provision most commonly took the form of workshops, peer-review schemes, conferences, mentoring schemes and discussions and collaborative activities around teaching and learning through communities of practice.

Accredited offers aimed at catering to the professional development needs of new staff and staff in learning support roles. There was a strong emphasis on the teacher or participant (developing skills/knowledge/conceptions), but provision also focused on student needs and institutional culture. These offers were driven by external professional frameworks, institutional priorities and evidence-based practice.

Non-accredited offers targeted experienced teaching staff as well as engaged staff in other teaching roles. Their aims focused on the participant, and to a lesser degree on students and the institution. Teaching and learning priorities of the institution, educational development units and teaching and learning committees, informed the content of non-accredited offers.

Both accredited and non-accredited offers were usually delivered by staff dedicated to

11% of respondents who described a specific CPD offer at their institution (accredited or non-accredited) did not evaluate this offer.

21% (N=100) of accredited CPD offers were evaluated only once.In 35% of these offers, evaluation was carried out in two stages. A lower percentage of accredited CPD offers were evaluated in multiple stages:14% evaluated the offer thrice;10% evaluated four times; and 2% evaluated the offer at five different stages.

52% (N=33) of non-accredited CPD offers were evaluated only once. Both accredited and non-accredited offers were generally evaluated at the end of the

activity. Further, with non-accredited offers, no assessment occurred before and during engagement with the activity (Figure 2.5).

Before Event

During Event

At the end of event

Within 6 months

After 6 months

0

10

20

30

40

50

60

70 Recognised/accredited

Non-accredited

Figure 2.5: Count of evaluations at each stage for accredited and non-accredited offers

2.2.3.2 Focus of evaluation

The general trend in evaluations conducted during and immediately after a CPD offer was a focus on

participant satisfaction, changes in beliefs about teaching and learning and changes in teaching

practice, and to a lesser extent, the impact on students or the institution. A difference in this pattern

was observed in subsequent evaluations conducted after six months or more. Figure 2.6 compares

the focus of evaluation at the end of a CPD offer and six months after completion of the offer. At the

end of an event, 81% of the evaluations focused on participant satisfaction and less than half on

student learning (40%) and institutional culture (33%). On the other hand, from all evaluations

conducted after six months, only 38% focused on participant satisfaction but 50% reported

evaluating student learning and 48%, institutional culture.

56/222

Frequency

11% of respondents who described a specific CPD offer at their institution (accredited or non-accredited) did not evaluate this offer.

21% (N=100) of accredited CPD offers were evaluated only once.In 35% of these offers, evaluation was carried out in two stages. A lower percentage of accredited CPD offers were evaluated in multiple stages:14% evaluated the offer thrice;10% evaluated four times; and 2% evaluated the offer at five different stages.

52% (N=33) of non-accredited CPD offers were evaluated only once. Both accredited and non-accredited offers were generally evaluated at the end of the

activity. Further, with non-accredited offers, no assessment occurred before and during engagement with the activity (Figure 2.5).

Participants' satisfaction

Participants' beliefs

aboutT&L

Participants' teaching practice

Students' perceptions

teaching

Student learning

Culture of dept and the

institution

0

10

20

30

40

50

60

70

80

90

81

55

66

17

4033

38

6063

18

50 48

End of event

Six months or more

%

Figure 2.6: Focus of evaluation of accredited and non-accredited offers at different stages

2.2.3.3 Methods

Questionnaires with participants were the most commonly used evaluation. Following this, a mix of

interviews and focus groups, observation of teaching practice, review of participant

journals/teaching material and review of curriculum enhancement practices were consistently used

in different stages of evaluation.

Evaluations conducted six months after completion of the CPD offer tend to differ slightly from those

at other stages. These frequently employed interviews/focus groups with participants rather than

only questionnaires (presumably allowing collection of more rich and descriptive data). Evaluations

after six months were also more likely to consider impact on the student, even if indirectly through

external student surveys and progression and retention statistics. Many institutions appeared to use

a combination of a number of methods in their evaluation although the range seems limited (those

that involve students or line-managers were rarely used).

Other means of evaluation reported by respondents included informal emails and discussions with

participants (including through student reps and liaison committees), external examiner feedback

and feedback from mentors. They also commented on analysis of data around participants’

involvement in teaching and learning activities –for instance, away days, promotion and prize panels,

57/222

publication and dissemination of work etc. – as a useful indicator for evaluation. The use of module

evaluation forms was discussed with the challenges in gleaning useful information from them.

58/222

EvaluationsBefore event

(N = 8)

EvaluationsDuring event

(N = 48)

EvaluationsEnd of event

(N = 83)

EvaluationsWithin 6 months

(N = 37)

EvaluationsAfter 6 months(N = 40)

Respondents who evaluated at this stage( N = 142 )

6% 32% 54% 22% 20%

Focus(Top three at each stage)

Participant satisfaction(75%)

Changes in beliefs about teaching and learning (75%)

Changes in teaching practice (75%)

Changes in teaching practice (75%)

Participant satisfaction (71%)

Changes in beliefs about teaching and learning (56%)

Participant satisfaction (81%)

Changes in teaching practice (66%)

Changes in beliefs about teaching and learning (55%)

Changes in teaching practice (73%)

Participant satisfaction (68%)

Changes in beliefs about teaching and learning (68%)

Changes in teaching practice (63%)

Changes in beliefs about teaching and learning (60%)

Impact on student learning (50%)

59/222

Table 2.5: Overview of evaluation practices discussed by respondents

Methods of evaluation(Top five at each stage)

Questionnaire for previous participants

Review of participant’s journal/teaching material/other statements

Curriculum development/enhancement

Observation of teaching

Student assessments

Questionnaire for current participants

Interview/focus group with current participants

Observation of teaching

Review of participant’s journal/teaching material/other statements

Curriculum development/enhancement.

Questionnaire for current participants

Observation of teaching

Questionnaire for previous participant

Interview/focus group with current participant

Review of participant’s journal/teaching material/other statements

Questionnaire for current participants

Observation of teaching

Interview/focus group with current participants

Questionnaire for previous participants

Curriculum development /enhancement

Questionnaire for current participants

Interview/focus group with current participants

Observation of teaching

PTES/NSS

Review of retention and progression stats

60/222

HEA CPD Pilot Workshop - Workbook

2.2.3.4 Results of CPD evaluation and dissemination (N=105)

Results of CPD evaluation were primarily utilised in QAA and other professional audits (80%) and to

inform future CPD provision (71%). Following this, they were considered in making changes to policy

around teaching and learning (52%) , probation and promotion criteria (48%) and drawing up annual

reports for governing bodies (44%). They were included in marketing materials to a much lesser

extent (12%). 9% of respondents reported that the results of CPD evaluations were not used in any

way in their institutions.

The impacts of teaching-related CPD (N=110) were most frequently disseminated via sharing of best

practices through presentations, training and discussions (84%), and cascading learning through

teaching and learning committees (72%). Pedagogic research was another fairly common route for

dissemination (54%). 8% of respondents reported that impacts of teaching CPD were not cascaded

at their institution.

Other means of dissemination which were reported included changes made to programme design

and curriculum, newsletters, in-house teaching and learning journals and through the university

website or intranet.

2.2.4 Student involvement in CPD (N=108)

Responses from staff involved in CPD provision indicated that a majority of students were aware of

staff participation in teaching-related CPD, although to varying levels (54% ‘somewhat’, 13% ‘yes’,

Table 2.6). The majority of respondents agreed that teaching-related CPD was an important student

agenda (41% ‘somewhat’, 28% ‘yes’). However, opportunities for students to contribute to teaching-

related CPD appear to be limited (Table 2.6)

Table 2.6: Perception of student awareness and involvement in CPD

Yes Somewhat NoAre students aware of staff participation in teaching 13% 54% 10%

61/222

HEA CPD Pilot Workshop - Workbook

related CPD?Is teaching CPD an important student agenda? 28% 41% 11%

Are there opportunities for students to contribute to evaluation of teaching related CPD

13% 20% 50%

Respondents perceived where students were made aware of teaching-related CPD, this was through

the student union (54%), student representative system (51%), schools and faculties (41%) and

central communications (29%). Students also learned about CPD when participants (their lecturers)

spoke to them about their CPD, involved them in projects, and during peer-reviews and observations

in the classroom.

Indirect ways in which students contributed to the evaluation of CPD were through student-led

teaching awards, module evaluations and through providing feedback at teaching and learning

committees. Sometimes, students were involved more directly and participated in workshops and

teaching and learning conferences. A few especially interesting examples of student involvement

emerged through respondents free-text comments, for instance, one where students were involved

in observations of teaching and curriculum design.

There is some evidence that the sector makes an effort to gain student feedback from external

surveys and other forms of feedback (module evaluations and liaison committees) to inform the

content of CPD. However, the robustness of feedback mechanisms and the consistency with which it

is integrated in to teaching-related CPD needs to be further scrutinized.

2.2.5 Institutional policies and culture around teaching-related CPD

Research has highlighted some of the factors which influence the extent to which staff embed their

learnings from professional development activities in their practice. These include opportunities for

continuing dialogue, opportunities for reflection and discussion with peers, observing the impact of

an implemented change in students and having supportive leadership (Brown & Inglis, 2013). It thus

becomes evident that the culture of an institution plays a role in shaping the extent to which staff

might engage with and benefit from CPD. This section examines responses about institutional

policies that could influence teaching-related CPD.

62/222

HEA CPD Pilot Workshop - Workbook

2.2.5.1 Peer-review (N=114)

83% reported that their institution had a policy or expectation around peer-review of teaching. This

trend was found within categories of institutions as well, as shown in Table 2.7.

Research-intensive (N=40)

Teaching-focused (N=44)

College-based (N=13)

95% 80% 77%

Table 2.7: Percentage of institutions which have an existing policy/expectation around peer-review (by institutional category)

34% of respondents reported peer-review was not linked to appraisal in their institution with

another 42% indicating that it was linked to appraisal only in some cases. Comparing types of

institutions, Table 2.7 indicates that CBHE providers appear to be different from the rest of the

sector in this regard. In FE colleges, peer-review was more consistently linked to appraisal. Chi-

square tests did not reveal significant associations between institutional category and whether or

not peer-review was linked to annual appraisal/performance reviews.

Table 2.8: Peer-review and annual appraisal/performance review

2.2.5.2 Promotion pathways

The literature suggests that having separate promotion routes for teaching, research, and ‘teaching

and research’ roles enhance the chances that specific teaching and learning criteria will be included

in promotion policies (Cashmore et al., 2013). This, in turn, impacts the culture of CPD within the

institution as CPD becomes instrumental as a means to gain qualifications and evidence engagement

with teaching and learning.

63/222

Research-intensive (N=39)

Teaching-focused (N=39)

College-based (N=12)

Not linked to appraisal

Linked in some cases

Linked in all cases

Not sure

31%

51%

5%

13%

44%

33%

10%

13%

17%

42%

42%

0%

HEA CPD Pilot Workshop - Workbook

In the research-intensive category of institutions, a majority of the institutions had separate

promotion routes for research and teaching roles, with 35% reporting a three-track and 28% a two-

track path (see Figure 2.7). A three-track pathway might include separate promotion criteria for

teaching, research and ‘teaching and research’ roles. A two-track might, for instance, might have

separate promotion routes for teaching and research roles.

Research-focused (N=40)

Teaching-intensive (N=43)

College-based (N=13)0.05.0

10.015.020.025.030.035.040.045.0

Single-trackTwo-trackThree-trackNot sure

Figure 2.7: Institutional differences in promotion pathways

Amongst teaching-focused institutions, single-track pathways were much more common (35%).

There is a significant association between the category of the institution (research-intensive or

teaching-focused) and the type of promotion pathways it has in place χ² (4) = 10.25 at p < .05. 39% of

respondents from CBHE providers reported having single-track pathway and an equal number were

unsure. Looking at promotion policies of the highest position, that of professor, a large number of

research-intensive and teaching-focused institutions reported having explicit teaching and learning

criteria for promotion to this level (58% and 45% respectively). However, while only 8% of the

research-intensive institutions did not have explicit criteria at this level, this number was higher for

teaching-focused institutions, 15%. This inclusion of explicit teaching and learning criteria in

promotion policies in a higher percentage of research-intensive institutions compared to teaching-

focused institutions could be partly contributed to a higher number of research-intensive institutions

having professors of teaching (50%) compared to teaching-focused institutions (42%) (see Figure

2.8). However, this difference was not found significant by a chi-square test. A percentage of

respondents, including all those from CBHE providers, reported they were unsure about promotion

policies to professor.

64/222

%

HEA CPD Pilot Workshop - Workbook

Research-intensive (N=40)

Teaching-focused (N=43) College-based (N=13)0.0

10.020.030.040.050.060.070.080.090.0

YesNoNot sure

%

Figure 2.8: Occurrence of professors of teaching and learning

2.2.5.3 Research Excellence Framework and educational research (N=113)

Overall, 48% reported that their institution had made a contribution in educational research to the

2014 REF, 33% said they had not and 19% were unsure.Teaching-focused institutions made a higher

return (63%) research-intensive institutions (53%). However, this difference was not found

significant by chi-square. The majority of respondents from CBHE providers reported they did not

make a return to the REF or were unsure (92%).

2.2.5.4 Reward and recognition for teaching (N=112)

Student awards were most commonly reported as a means of rewarding teaching (65%), followed by

organisation of teaching and learning conferences (56%) and institutionally bestowed teaching

awards (54%) (see Figure 2.9).

Student bestowed teaching award

Organisation of T&L conferences

Institutionally bestowed teaching awards

Promotion

Internal teaching fellowship award

Community of practice around research in HE

External teaching fellowship award

Time allocation for teaching related CPD

Pedagogic research grants

Honorary titles

Financial incentives

0 10 20 30 40 50 60 70

65

56

54

47

46

35

31

31

30

9

6

65/222

(%)

HEA CPD Pilot Workshop - Workbook

Figure 2.9: Reward and recognition for teaching and learning

Differences within the sector were not marked. However research-focused institutions reported

more recognition through promotion (53%) (linking to the previous finding that they more

commonly have explicit teaching and learning criteria for promotion) and institutionally-bestowed

teaching awards (60%), compared to teaching-focused institutions (35% and 38% respectively). In

turn, teaching-focused institutions reported more recognition of staff’s CPD engagement through

organisation of teaching and learning conferences (56%), pedagogic research grants (35%),

communities of practice around research (38%) and time allocation for teaching-related CPD (36%)

compared to the research-intensive category.

Box 2.4: Overview of Findings related to institutional policies around teaching-related CPD

66/222

Overall, a majority of the sector has an expectation around peer-review of teaching but it is not consistently linked to the annual appraisal and peer review processes.

Research-intensive institutions currently tend to have two and three track promotion pathways, which allows them to have explicit teaching and learning criteria in promotion policies. Single track promotion pathways tend to be more common in teaching-intensive institutions and colleges.

Motivation plays a big role in staff’s ongoing engagement with CPD. The most prominent mechanism for recognition of teaching quality is currently student awards and to a lesser extent, institutionally bestowed awards.

HEA CPD Pilot Workshop - Workbook

2.3 Case study development

2.3.1 Selecting the case study institutions

In total, 22 audit respondents indicated that they would be willing to be considered as a case study.

A review of their audit responses led to the immediate elimination of ten of these, primarily because

they did not evaluate CPD or, where they did, their evaluation was focused immediately after a CPD

activity with no longer-term follow up which would be of value in measuring impact (Trigwell et al.,

2012; Chalmers et al., 2012; Stewart, 2014). A further two were excluded due to a lack of relevant

CPD provision. Of the remaining 12 institutions, eight were selected for initial follow-up as they

evaluated the impact of their work at least six months after an activity had been implemented.

The project team also undertook a desk-based survey of publications (such as Educational

Developments) which provide insights into contemporary educational development practice.

Following this, a further three potential case studies were identified. In total, 11 institutions were

contacted to explain the context of the current project and to request participation. At this stage we

applied the selection criteria used by Turner et al. (2013) in the development of case studies for their

HEA/SEDA funded project: Measuring the impact of the UK Professional Standards Framework for

Teaching and Supporting Learning. In this project the following criteria were used to select case

study institutions:

1. Participant has agreed to be interviewed in the survey;

2. Survey responses indicate something interesting to say;

3. The group of case studies represents a range of institutions;

4. The group of case studies represents different types of engagement.

Turner et al. (2013) also advocated limiting the number of open-ended interview questions and

allowing time to explore what was ‘interesting’ about the examples specific to individual case

studies. In heeding this, we adopted a narrative approach to the interview schedule. A number of

central prompts ensured that essential contextual information regarding CPD provision was

captured. However, the primary focus of each interview was to obtain information relating to the

development of interviewees’ particular approach to evaluating their CPD provision and why this

was effective.

Naming the institutions and, where relevant, key individuals, was integral to the development of the

case studies. We felt this would afford recognition to those engaged with innovative evaluation

67/222

HEA CPD Pilot Workshop - Workbook

practice and allow further dialogue to take place beyond the life of this project. However, we

recognised the potential implications for participating institutions. Potential participants were

requested to discuss the invitation with their colleagues / unit heads. Following this, five institutions

(University of Roehampton, Sheffield Hallam University, London Metropolitan University, Queen

Margaret University and The Open University) were selected to be developed into case studies. We

also decided to include Plymouth University, as through the process of reviewing the audit

responses and holding preliminary discussions with the case study institutions, it was realised that a

number of innovative approaches to evaluating the impact of CPD activities were being undertaken

here, which would be of wider interest to the Educational Development community.

2.3.2 Data collection

The audit data were reviewed to frame the interview schedule, which was adapted depending on

the institution, the CPD activities that were to be discussed and their approach to evaluating impact.

A telephone interview was held with a key contact in each institution. Each interview was audio

recorded and transcribed verbatim. A member of the project team then analysed the transcript to

identify the information necessary to develop the case study. Extracts of the interview narrative

which could illuminate the case study were also identified, to be used in the final case study. Each

case study was written and reviewed by the project team, and then returned to the participant for a

final review and edit. Giving the participants (and where relevant their colleagues) the opportunity

to view the final text was integral to the development of the case study, in order to ensure they

were happy with the information that was included and its presentation as the case study.

68/222

HEA CPD Pilot Workshop - Workbook

2.4 Case studies

2.4.1 Case Study – London Metropolitan University

London Metropolitan University is a large post-1992 university spread across 3 London

campuses. The university comprises 4 Faculties: The Faculty of Social Sciences and

Humanities and the Faculty of Life Sciences and Computing are based at Holloway Road, the

Faculty of Business and Law at Moorgate, and the Sir John Cass Faculty of Art, Architecture

and Design at Aldgate.

It has 16,000 students, and offers undergraduate and postgraduate degrees across a range

of over 27 subject areas encompassed within the academic domains reflected in the above

Faculties.

The University is committed to widening access to Higher Education, invests in

entrepreneurship and innovation, and places employability at the heart of the curriculum.

Professional development opportunities for staff

The Centre for the Enhancement of Learning and Teaching (CELT) runs a PGCert/MA in Learning and

Teaching in HE (MALTHE) accredited by the HE Academy at UKPSF Descriptor 2, as well as a Learning

& Teaching Practice course accredited at Descriptor 1. These programmes are integrated within the

University’s CPD Framework for Academic Practice, overseen by CELT, which also includes provision

for staff to gain professional recognition from D1-D3, on the basis of experience. The Centre

organises an annual Learning and Teaching conference, and a range of professional development

modules, seminars and workshops, plus online guides and resources. CELT also produces a

registered journal, Investigations in university teaching and learning (ISSN 1740-5106).

Evaluating the effectiveness of teaching-related CPD

CELT routinely gathers statistics and qualitative feedback from attendees at CELT organised activities,

and these are presented in an annual report which is designed to “inform the University and

Faculties of CELT’s range of services…to demonstrate output and outcomes and thereby provide a

basis for consideration of future priorities and funding of CELT work”.4

4 https://metranet.londonmet.ac.uk/fms/MRSite/psd/hr/capd/CELT%20Policy%20Docs/CELT%20Annual%20Report%202012%20to%202013.pdf

69/222

HEA CPD Pilot Workshop - Workbook

Courses, such as the MALTHE and the e-learning modules, are evaluated continuously. At the start of

the CPD activity staff establish what participants hope to gain from the course, and adjust as

appropriate; during the course informal feedback is collected to help clarify expectations, and fine-

tune the content and process. At the end of the courses more formal methods, such as a self-

evaluation questionnaire as well as evidence presented in projects (for example, projects relating to

curriculum evaluation/redesign and assessment practices) are used to evaluate participants’

satisfaction, as well as changes to participants’ beliefs about teaching and learning, and shifts in

teaching practices. Staff within CELT are also interested in the impact on student learning as well as

the impact participation has on the culture of the department and the institution. The impact on

students is measured indirectly by teachers self-reporting on shifts in their teaching approaches e.g.

from a teacher-centred to a more student-centred approach, rather than looking to shifts in student

results or other impact indicators.

In 2008, a substantive impact study of the MALTHE was undertaken which involved staff who had

completed the programme within the previous four years.5 The study explored “the impact of the

formal course on lecturers’ pedagogical thought and practice, and opportunities for and barriers to

implementing acquired ideas or desired changes”, and utilised an open-ended questionnaire which

was emailed to participants.

In 2013, a similar, but smaller scale evaluation was undertaken with the purpose of collecting

feedback for programme revalidation.

‘Every five years we conduct a major review towards updating provision for revalidation. This is when we use questionnaires and focus groups with previous participants, to get a sense their views of the benefits of the CPD and how it has impacted on their educational thinking and practice and on their students' learning’ Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).

There were a lot of similarities between the findings of the 2008 and 2013 evaluations. Respondents

identified cross-disciplinary networks, opportunities to share practice and a boost in confidence as

key features of the course:

‘Participants described the benefits of being part of a whole network where they could share ideas across disciplines, and get new ideas…for some of them it was a really transformative experience in

5 Warren, D. (2008). Benefits and barriers regarding the enhancement of educational practice: an investigation of the impact of an accredited professional development programme for teachers in higher education. Paper presented at the London Scholarship of Teaching and Learning 7th International Conference, London, 15-16 May, 2008.

70/222

HEA CPD Pilot Workshop - Workbook

terms of extending their understanding of teaching and learning, and just really giving them confidence that some of the new teachers need to get them started’

However, participants also encountered impediments to integrating their learning from the MALTHE

into their teaching practice:

‘People left the programme with ideas and inspiration and confidence and keen to maybe try out things but when they got out into the real world of the department they found that change was a little less easy to implement... that there were cultural and institutional inhibitors’ Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).

Staff that progress to the Masters programme (MALTHE) are asked to submit a reflective ‘graduation

statement’ articulating what they have learnt through the process of undertaking the course, and to

think ahead in terms of how they think this would inform or influence their future practice. These

reflective statements are used as a further source of impact evaluation evidence.

Challenges to evaluating CPD

It was recognised that the benefits to practice might well become clearer some time after

completion of the CPD activity. However, carrying out longitudinal in-depth evaluations such as the

post-completion impact evaluation for MALTHE is resource intensive. Another challenge noted was

the fluctuation in institutional agendas and policies. When asked ‘Why is longer term evaluation not

carried out as a matter of course?’, the response was:

‘There are times when it’s important to make visible the quality and impact of what one is doing and so you seize those moments when it is necessary to do that. Because of the kind of fairly common cycle of having programmes reviewed every five years I think people often rely on that more standard periodic review process as a way to reflect on and refresh CPD provision without necessarily doing the more in-depth study that a proper longitudinal impact study requires, I think it’s all these mix of things, it is resources, it is political agendas, institutional agendas’ Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).

The sensitivity of undertaking impact evaluation was also highlighted in the interview. It was felt that

there needed to be a clear division between education enhancement, and gathering data that

71/222

HEA CPD Pilot Workshop - Workbook

ultimately could be used for quality monitoring. This connected to the question of ‘Who undertakes

the evaluation?’ In particular, if an evaluation involves students, ‘Who speaks to the students?’

‘I think there is a fine balance to be struck between asking teaching staff to articulate how they feel both [teachers and students] benefit, and how their thinking and practice has changed, and not making them feel they are being made subject to some kind of appraisal when you come in and start talking to their students. I think it presents some quite tricky research ethical issues’. Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).

72/222

HEA CPD Pilot Workshop - Workbook

2.4.2 Case Study – University of Roehampton

The University of Roehampton is located in south-west London and comprises four colleges.

It had 8,530 students enrolled in the 2013/14 academic year, and offers undergraduate and

postgraduate degrees across a range of subjects including Business and Management,

English Literature, Creative Writing, Film, Anthropology, Sports Science, Sociology,

Education, Dance, Drama, Philosophy, Classics, Criminology, Psychology, and Counselling.

It was recently named in the Sunday Times Good University Guide, as the Best modern

university in London.

Professional development opportunities for staff

There is a growing institutional expectation that staff involved in teaching will have recognition at

Descriptor 2 of the UK Professional Standards Framework(UKPSF). The Department of Academic

Enhancement (DAE) runs an introductory course for PhD students who teach, technicians and

visiting lecturers to supporting learning and teaching in H.E (accredited at D1). It also offers a

mandatory probationary programme for new lecturers: ‘The University of Roehampton’s Certificate

in Learning and Teaching in Higher Education’ (URCert), accredited at D2. In addition, the university

manages an in-house CPD framework for experienced staff, accredited at all four levels of the UKPSF

and this is aligned to hiring and promotion processes; for example, staff seeking promotion to

principal lecturer are expected to have achieved or be working towards D3.

A dedicated DAE staff member acts as the link to each department. In addition, DAE offers a range of

teaching-related workshops, a popular annual learning and Teaching Conference, and has been

particularly successful in gaining funding for teaching-related research projects.6 Individualised

support is also given to staff consistently receiving disappointing module feedback, via a

personalised development programme tailored to their needs. This might include peer reviews of

teaching, professional dialogue and introductions to new teaching approaches.

Student engagement with teaching-related CPD7

Student engagement is high on the University of Roehampton’s agenda, and as such this forms a

specific aspect of interest in this case study. The university has a student senate, which meets

regularly with the Vice Chancellor as a formal committee. Student representatives from across the

6 http://www.roehampton.ac.uk/Services/Learning-and-Teaching/Research/

7 http://www.roehampton.ac.uk/Services/Learning-and-Teaching/Student-Voice/

73/222

HEA CPD Pilot Workshop - Workbook

different programmes and the Students’ Union have an active role in policy making for the

university, with a dedicated budget of around £50,000 to spend on initiatives that they think are

worthwhile. This central drive for student engagement impacts across all departments. DAE involves

students in staff CPD in a number of ways. Students have been involved in peer reviews of academic

staff on the URCert, and this has been hugely successful. Student consultants are trained by DAE

staff to observe teaching and to give feedback. Students observe staff from a department which is

not connected to their own programme. They focus on the pedagogic practices of the lecturer, not

the content.

“They could talk about whether or not the academic was trying to involve and include all of the students, whether all the material seemed to be taken from a sort of Anglo-centric source, whether the lecturer just read from PowerPoint slides and whatever. So they definitely had the wherewithal to comment on the sort of pedagogic practice adopted, and that was very positive. And a lot of the academics said, “We’ve always wanted to ask for sort of direct student feedback, but we know that our own students are never going to be as honest as they might be, because we’re marking their work, and because they’ve got to confront us in the session, so, you know, they’ve got to be there in the session next week, if they’ve said anything negative”, whereas if they have students from outside their departments, you know, no axes to grind anywhere, so they could get this sort of, this direct student feedback’. Jo Peat, Head of Academic Professional Development

As part of the HE Academy funded project ‘Re-imagining Attainment For All’ (RAFA Project), students

at Roehampton have been involved in developing an interactive resource (dialogue sheets) to use

with academic staff teams to help them to review teaching and assessment practices. The project

was set up in response to the attainment gap between Black and minority ethnic students and white

students. The dialogue sheets are generated by students with a student co-ordinator, and include

student reflections on their teaching and learning experiences. These are presented as direct quotes

to the learning and teaching advisory group chairs alongside data from planning, on the degree

attainment differential for their programmes.

Jo Peat, Head of Academic Professional Development explains: ‘Rather than it being us [academic staff developers] facilitating these workshops, we got our student consultants to do it. So we very much took a back seat, and we got them to facilitate the workshops and to guide the direction of the workshops. The academics said that they’re the best workshops they’d ever attended, because they could hear directly from the students. It wasn’t being mediated by anyone, they could say to the students that, “I never knew you felt that, really, the students on my programme feel that?”, and the students were saying, “Well, yes, I’m one of them and, yes, this is exactly how we feel, you know, this is what we’re experiencing, these are the emotions we go through”. So the fact that we weren’t

74/222

HEA CPD Pilot Workshop - Workbook

there, as the sort of the purveyors and the facilitators of these workshops, made them much more powerful, and the academics really did like that direct link with the students.

This approach has been so successful that a number of cross-university workshops have been held

utilising the same approach in examining what the students have said, and what that might mean for

pedagogic practices across the university.

Evaluating the effectiveness of teaching-related CPD

Participants on formal CPD offerings are asked to evaluate the course upon completion, and, in the

case of the URCert, one year afterwards. Both evaluations take the form of an email questionnaire.

The initial feedback is sought using a 22-item survey the bulk of which comprises likert-style

statements about teaching and learning. For example, one statement reads “My ideas on teaching

and learning are substantially the same as before”. 5 open questions are asked at the end, focusing

on participant satisfaction, attendance and participants’ reaction to undertaking a small-scale

research project. A year later, participants are sent 7 further evaluation questions which ask

participants to recall the most and least valuable aspects of the course and to identify changes in

behaviour in relation to teaching and assessment practices.

In addition to formal monitoring via questionnaires, there are numerous opportunities for informal

feedback, helped, in part, by the small size of the institution.

‘We try to ask staff fairly regularly without being sort of overly enthusiastic about badgering them, about elements that have contributed to them changing their practice…We've all got departments we link with, and we'll visit them sort of at least once a month for meetings and things. So we've got lots of opportunities for fairly sort of ad hoc conversations and informal conversations about how things are going, so we do get quite a lot of information that way, but it is informal, you know, it's not a formal way of sort of evaluating practice and impact’. Jo Peat, Head of Academic Professional Development

In relation to the RAFA project, both staff and students complete evaluation sheets that focuse on

reactions to the CPD activity as well as changes in teaching and learning behaviours. In addition,

verbal feedback is sought individually to try and ascertain the impact of the activity upon practice.

75/222

HEA CPD Pilot Workshop - Workbook

Challenges to evaluating CPD

Whilst it is recognised that longitudinal evaluations can provide useful information relating to impact

on teaching practices, in reality this is not straightforward. For example, in the past longitudinal

evaluation had been undertaken to assess the impact of the Annual Teaching and Learning

Conference on practice, 6 months after the event. However, the response rate was so low that these

evaluations have been discontinued.

‘Our annual learning and teaching conference, and that's become very popular, you know, we get an awful lot of staff there now, in fact more than we ever thought we would get there, and they have an evaluation to fill in. One of the questions is something about what they think they might do in terms of their own practice in the light of what they've seen or been part of in the conference, but often that's the bit that's sort of left blank or it's answered, "I don't know", and even if we go back sort of six months later and say, "Did you make any changes?", again, it tends to be sort of, "Oh, I'm not really sure, I didn't really think about it that way’. Jo Peat, Head of Academic Professional Development

Future plans

Involving students with Peer Review is something that Jo Peat (Head of Academic Professional

Development) would like to see developed beyond the URCert. She regards it as important that

teaching staff aren’t primarily receiving their feedback from educational developers. Resources

constrain development in relation to the training of students, and ensuring that both students and

staff are protected.

76/222

HEA CPD Pilot Workshop - Workbook

2.4.3 Case Study – Sheffield Hallam University

Sheffield Hallam is the 4th largest university in the UK, with 34,718 students (25,988

undergraduates and 7,114 postgraduates). As of 2013/14, staff totalled 4,494, including

2,114 academic staff and income generated was £257.1m.

The University has four teaching faculties – including 18 academic departments - based

across two campuses (City and Collegiate).

The University encourages work-based learning, with over half of all courses offering

integrated practice or work placement opportunities.

Approach to CPD for academic staff

The approach to CPD for academic staff at Sheffield Hallam University draws upon a peer

enhancement strategy informed by the following principles: focussed on teaching and learning

practice; authentic; facilitated by using expertise to scaffold learning and application; peer

supported; team based; evidence based; and generates reflection and feedforward. The key

components of appraisal, CPD and Peer Review and Enhancement continue to be central to this

strategy.

“Measuring the impact of this work on the staff and student experience remains a challenge,

although we routinely reflect on our approach, trial new ways of measuring what we do and adapt

our practice in light of experience and feedback” Jackie Cawkwell, CPD Manager

Professional development opportunities for staff

Teaching-related professional development opportunities for academic and learning support staff at

Sheffield Hallam include informal and credit-bearing opportunities; responsibility for these areas of

work lies in different parts of the University.

The central directorate works with faculty leaders and the wider academic community through a

'hub and spoke' approach; events and opportunities are co-designed and co-facilitated by staff from

the Learning Enhancement & Academic Development (LEAD) directorate and faculty learning and

teaching leaders and innovators. The aim is to develop good inspirational learning and teaching

practices across the University by targeting staff groups according to their levels of experience.

77/222

HEA CPD Pilot Workshop - Workbook

Priorities are currently early career teaching staff and academics with leadership roles. The themes

of learner engagement, course belonging and professional recognition are current priorities. The

CPD strategy seeks to reinforce the need to support staff commitment to remaining in good standing

as Fellows and Senior Fellows of the HEA.

Evaluating the effectiveness of teaching-related CPD

The central directorate (LEAD) evaluate the impact of CPD events that they co-ordinate, as well as

broader CPD work. All academic staff were surveyed on their engagement with, and attitude

towards, CPD in March 2012. Participants reported overwhelming support in terms of successful

delivery and achieving stated outcomes of CPD sessions, particularly early career staff who

welcomed the informal and inclusive approach. Anecdotally, colleagues who attended reported that

engaging with CPD extended both their practice and their confidence, whilst presenters

demonstrated a growing capacity for leadership of others in their academic discipline and pedagogy

in the context of HE.

LEAD seeks to identify several very specific elements when they monitor, evaluate and measure

impact:

Monitoring numbers and staff profiles of those attending and facilitating events.

Evaluating process indicators e.g. expectations, quantity/quality of each event;

Measuring outcome indicators (impact) for individual staff (both participants and facilitators).

For example: affective outcomes, such as confidence and motivation as a result of engaging

with CPD activities; development of skills associated with professional practice, such as critical

enquiry, evidence-informed practice, reflective practice and scholarship; development of a

repertoire of 'classroom' practice and tool-kit of resources for teaching, learning and

assessment activities;

Measuring the impact on broader teaching and learning ambitions, i.e. aligning provision

with institutional and teaching team objectives, and informing individual development plans.

“We also reflect more generally on other aspects that influence the achievement of our aims relating to excellent teachers, teaching excellence and improvements to the student experience, for example: appraisal; Peer Review & Enhancement; professional recognition targets and strategies; recording CPD and maintaining good standing” Jackie Cawkwell, CPD Manager

78/222

HEA CPD Pilot Workshop - Workbook

Monitoring and review of Appraisal and Peer Review aims to encourage good practice in recording

and reflecting on outcomes for the individual and their line manager. The only recording beyond

this, however, involves the requirement of an audit at faculty level that captures which activities

took place and identifies any relevant cross-faculty and staff development issues. In reality, the latter

information is often too generalised to inform future CPD activities.

Challenges to evaluating CPD

Monitoring numbers and profiles of all the CPD that actually occurs is a huge task. Further, in some

cases, routine monitoring is not appropriate or even possible (for example, when nurturing special

interest groups or contributing to ad hoc events). In many cases it is more appropriate to encourage

individuals to record their engagement autonomously, for example, using PebblePad. Evaluating

aspects such as meeting expectations, whilst useful for planning, design and delivery purposes, does

not in itself contribute in any meaningful way to our understanding of the actual impact of CPD

activities.

‘CPD activities are often a 'slow burn' and without the chance to practice and reflect on new approaches, impact can be overlooked or disassociated from the initial development activity. Nor do we routinely have access in any detail to reflective discussions between academic colleagues, their peers and their line-managers. Thus the impact of what we do is often lost to us’. Jackie Cawkwell, CPD Manager

Future plans

LEAD seeks more effective integration of monitoring, evaluation and measuring of impact for the

range of professional development activities that staff engage with. These vary between discipline-

specific and teaching and learning pedagogy; between scholarship and research; between credit

bearing courses, non-credit bearing (but formal) and informal events. Aspirations include:

an improved capacity for all academic and learning support staff to act autonomously in:

o recognising their own development needs as part of an evidence-informed approach

to practice,

o working collaboratively with peers in effectively addressing those needs,

o feeling confident in practising and further reflecting on new ways of being and

doing;

79/222

HEA CPD Pilot Workshop - Workbook

more effective use of institutional quality assurance processes such as Course/Periodic

Review, institutional audit, appraisal, and performance management. This would benefit

both the implementation of institutional strategies and inform future policy direction;

extending our understanding of how our growing community of professionally

recognised/certificated staff (and our community of staff engaged in maintaining and further

developing themselves) actually impact on the learning gains and overall experience of our

learners.

80/222

HEA CPD Pilot Workshop - Workbook

2.4.4 Case Study – Nottingham Trent University

Nottingham Trent University (NTU) is one of the largest universities in the UK, with 27,000

students.

The University has recently improved its overall ranking, moving from 73 rd to 57th in the

Guardian University League Table for 2015 (i).

A diverse range of academic disciplines are provided by 9 Academic Schools organised in 3

Colleges: College of Business, Law and Social Sciences; College of Arts and Sciences; and

College of Art, Design and Built Environment.

Professional development opportunities for staff

The Centre for Professional Learning and Development (CPLD) was established in 2008 as part of the

Human Resources Department. Its role is to promote and support effective professional

development policy, strategy and opportunities for all staff groups as appropriate to help meet the

University’s strategic ambitions.

‘Being part of HR has actually helped us in a lot of the work that we do. We have professional development very much embedded in the annual appraisal process, in wider policy and in wider practice around the institution. So it’s really helped from that point of view.’ Alison Stewart, Academic Practice Development Consultant

In 2011, CPLD led the development of institutional CPD frameworks to assist and inform academic

practice professional development. One of these was the NTU Teaching Development Framework

(NTU TDF), designed to support the professional development of all staff who teach and/or support

learning at NTU.

The Framework has 5 elements including the NTU Learning and Teaching Professional Development

Policy. Another Framework element embeds national standards for teaching and supporting learning

as set out in the UKPSF by defining what they mean at NTU; all staff who teach and support learning

at NTU are expected to gain HEA recognition. As such, the Framework demonstrates the University’s

commitment to embedding those standards within professional practice and its staff development

policy and provision.

NTU-wide Teaching-related CPD delivered by CPLD has undergone significant development over the

last 7 years and includes writing workshops and 1:1 support for HEA applicants, on-line distance

81/222

HEA CPD Pilot Workshop - Workbook

learning resources and a number of conferences, course leader workshops, teaching development

sabbaticals and secondments, pedagogic research, and professional development programmes

including:

The new Postgraduate Certificate in Academic Practice (PGCAP) – permanent lecturers new

to teaching in HE are required to complete Module 1 by the end of their first year. In

addition to academic credits, this gives them HEA Associate Fellow recognition, and they go

on to achieve HEA Fellow recognition within 3 years of starting at NTU.

The 100-hour Certificate in Learning and Teaching in HE programme (CiLTiHE) is aimed at

full-, or part-time staff with some prior HE teaching experience, and Hourly Paid Lecturers.

The course comprises 5 Modules each aligned to the five areas of activity of the UKPSF.

Postgraduate Research Students complete a 25-hour Essential Teaching Toolkit programme

(ETTiHE) to achieve their ‘Readiness to Teach’ certificate. It provides them with the practice

skills and knowledge to design and deliver individual sessions.

Evaluating the effectiveness of teaching-related CPD

The overall service provided by CPLD is evaluated with the aim of understanding its impact for the

individual participants, their teams, the wider Schools and Professional Services, and the institution.

Evaluation is regarded as a crucial aspect of all work undertaken, as CPLD is required to report on the

outputs and outcomes it supports.

CPLD have designed an Evaluation Framework to enable different levels of impact to be determined.

CPLD’s remit for staff development across NTU’s academic and professional services falls into three

categories of role:

consultancy or advice to support CPD development

partnership working to help develop and/or deliver CPD

direct provision of courses and resources

The CPLD Evaluation Framework maps those three categories of role against five levels of impact:

Level 0: Establishing foundations and awareness – the work that needs to be in place

before a development intervention

Level 1: Initial responses – the reaction to a development intervention at the time or shortly

after by those involved (including CPLD staff) – the ‘CPD Reactionnaire’

Level 2: Perceived change – the change to an individual’s/team’s practice as reported by

them during follow-up activities

82/222

HEA CPD Pilot Workshop - Workbook

Level 3: Observable change – the change in an individual’s/team’s practice as observed by a

line manager or person

Level 4: Institutional impact – changes to institutional practice (aims, structures, processes)

and their impact

‘We try really hard not to deliver training which is discrete and doesn’t tie in with a bigger organisational development ethos. We try and build our CPD into other drivers so that we can demonstrate wider, observable impact and change. That’s really important to demonstrate our contribution to the wider organisation. Lindsay Davies, Academic Practice Development Consultant

The evidence for the impact of all three roles at the different levels is captured via a suite of tools

and measures, including self-reflective questions, practice and policy reviews, and practice case

studies.

The CPLD Evaluation Framework was created in-house in 2012 to meet the needs of the Centre and

builds on Kirkpatrick and Kirkpatrick’s (2005) evaluation levels and The Rugby Team Impact

Framework (2008) which was designed to measure the effectiveness of researcher development

initiatives.

‘Kirkpatrick’s model focuses on the effect on the participant. What I wanted to do was to widen that. Greater impact is made when the impact shifts to the Department. So we are interested in how far does ownership shift from us out to the target group? So ultimately we should be performing a consultancy role and things should be working either in partnership with us or owned by the Schools for instance. We are interested in: How far has the activity become embedded within the organisation? E.g. has policy changed?.....that’s the high impact stuff. You may be able to get some quantitative statistics around that but others are more qualitative. They are about – what has happened to the institutional structure or who is represented on various bodies’. Lindsay Davies, Academic Practice Development Consultant

All CPD events including conferences, workshops, and briefings are evaluated at Level 1 using an on-

line post event feedback form known as the ‘CPD Reactionnaire’ is used to capture participant

83/222

HEA CPD Pilot Workshop - Workbook

feedback about the effectiveness of the event organisation, booking and publicity, the learning

environment, facilitation, and how well the event matched participants’ expectations. Level 0

evaluation is done prior to the event to evaluate the appropriateness of the event design in line with

the needs of the target audience.

‘We have always tried to avoid just rating the popularity of interventions using a simple ‘tick-box’ approach to evaluate impact. That Level 1 approach has its’ place and benefits, but we are more interested in getting actual evidence of how the intervention has impacted on participants’ practice in the short and longer term’ Alison Stewart, Academic Practice Development Consultant

Teaching and learning professional development courses, such as ETTiHE and CiLTiHE outlined

earlier, are routinely evaluated at Level 2 to determine to what extent to which the CPD has

informed participants’ practice. This is evaluation is done using a number of methods including:

Pre-, and post-course self-diagnostic assessments against learning outcomes and other

skills/knowledge frameworks, to identify initial levels of participant confidence, competence and/or

capability and how these alter as a result of engagement in the course.

Activities during professional development delivery using critically reflective questions are used in a

number of settings including online questionnaires, practice portfolios, presentations, formative and

summative assessments, class discussions, and occasionally – focus groups - related to the CPD

learning outcomes. Both group and individual feedback is collected, and much of it is anonymous. In

discussions and feedback activities, participants are left together in a room (without the facilitator)

to collate their responses. This approach has generated rich information about impact on practice,

knowledge and values, which CPLD have been able to use to demonstrate the impact of professional

development on practice.

Consultancy and bespoke teaching and learning CPD activities are evaluated in two ways. Firstly,

participants are asked to complete a questionnaire, which asks four open questions. These questions

seek to understand: what was most useful; what needs to happen next; whether they are any

further development needs for the individual or for the staff team. Secondly, the relevant Manager

is also asked to complete an evaluation, which asks more searching questions in terms of whether

the CPD activity achieved the agreed objectives and outcomes, and how those align to the needs of

the team and the strategic priorities of the university.

CPLD report that a very rich source of higher levels of impact of teaching-related CPD emerged from

the 250 applications for HEA professional recognition via the HEA-accredited NTU Scheme.

84/222

HEA CPD Pilot Workshop - Workbook

‘The application process at NTU focuses applicants and referees not only on explaining what is done and why, but also identifying impact on practice – and the information they use as evidence of impact. Applicants and referees often found the latter aspects the most challenging parts of the process. Not only did they find it rewarding and satisfying to see what they had achieved over time, but we realised the value of it as evidence of impact at Levels 2 and 3 in our Evaluation Framework. Some have said that it has changed the type of conversations they now have in the peer and line manager reviews. Alison Stewart, Academic Practice Development Consultant

Future plans

There is a strong institutional driver to demonstrate impact of activities on teaching quality. CPLD

have identified the potential of using their Evaluation Framework to support this. This will involve

working with senior managers and academic staff to review and revise, as appropriate, the standards

already defined using the UKPSF in the NTU Teaching Development Framework to articulate an

institutional definition of ‘teaching quality’.

CPLD are working to establish a sustainable method for longitudinal evaluation of the impact of

teaching-related CPD, including the new Postgraduate Certificate in Academic Practice starting in

January 2016 and HEA professional recognition fellowships. In 2015/16, academic staff will be

asked to identify in annual plans, the professional development activities they will complete to

achieve this. This will enable CPLD to evaluate the impact of those activities in contributing to

supporting implementation of University strategic goals and teaching quality.

‘Longitudinal evaluation is one of those areas that is ‘tricky’ and it’s tempting to end up avoiding it because it is just too difficult and very time consuming, or get hung up on particular measures which are not really telling us anything about long term impact. I’m sure we’ll find an approach that’s a sensible compromise!’ Alison Stewart, Academic Practice Development Consultant

CPLD have also started discussions with the Students Union to identify ways of involving students in

teaching-related CPD and evaluating it.

‘The discussions we’ve had with colleagues from Plymouth really brought home to us how we haven’t really thought about involving our students. We approached the Student Union Reps at a recent NTU teaching conference and were delighted to hear them say they have lots of ideas about how they

85/222

HEA CPD Pilot Workshop - Workbook

could get involved. So we’re meeting them over the summer to talk further’. Alison Stewart, Academic Practice Development Consultant

Email: alison.stewart@ntu.ac.uk

1. Kirkpatrick, Donald L., and Kirkpatrick, James D., (2005) Evaluating Training Programs: The Four

Levels (3rd Edition) San Fransisco: Berrett-Koehler Publishers

2. Bromley, T., Metcalfe, J., and Park, C., (2008) ‘The Rugby Team Impact Framework’ ISBN-13: 978-

1-906774-00-4 www.vitae.ac.uk/ieg

86/222

HEA CPD Pilot Workshop - Workbook

2.4.5 Case Study – Queen Margaret University, Edinburgh

Queen Margaret University (QMU) is a small post-1992 university which dates back to 1875,

but was granted full university title in 2007. The University is located to the east of

Edinburgh, and has a portfolio of socially and economically relevant educational

programmes that are organised within 2 Schools.

The School of Arts, Social Sciences and Management comprises of: Media and Performing

Arts; Psychology and Sociology and Business, Events and Hospitality Management. The

School of Health Sciences encompasses the subjects of: Dietetics, Nutrition and Biological

sciences; Nursing; Occupational Therapy and Arts Therapies: Physiotherapy: Podiatry;

Radiography: Speech and Hearing Sciences and The Institute for International Health and

Development.

QMU has around 6000 students and 470 staff. The university also has a broad portfolio of

undergraduate and postgraduate programmes with international partners in Egypt,

Greece, India, Nepal and Singapore.

Professional development opportunities for staff

Staff from the Centre for Academic Practice (CAP) contribute to policy and strategic planning,

collaborate with all departments and units, and provide professional development for staff and post-

graduate students in all aspects of learning, teaching and assessment. The breadth of CAP’s offer is

illustrated in their annual programme handbook, and includes a series a short courses, bespoke

workshops for programme teams or schools, and support for staff wishing to apply for HEA

Fellowship.8 QMU also runs a Master’s in Professional and Higher Education (PGC PHE), of which the

Post Graduate Certificate is HEA accredited. This is an applied programme, founded upon theories of

inclusive learning and reflective practice, relevant and of interest to practitioners across all

subject/practice areas. Since 2013, all new lecturers undertake this course.

The in-house CPD Scheme was piloted at QMU in 2013 and is optional for experienced members of

staff. No university targets have been set for the achievement of Fellowship. The emphasis is very

much on enhancement.

8 http://www.qmu.ac.uk/cap/PdfFiles/ProgrammeCurrent.pdf

87/222

HEA CPD Pilot Workshop - Workbook

Evaluating the effectiveness of teaching-related CPD

Evaluation of credit-bearing CPD programmes (such as the PGC PHE) is relatively ‘straight forward’

because there is a requirement for annual programme monitoring, which includes the involvement

of students and an external examiner. In Scotland, quality monitoring takes the form of

Enhancement-Led Institutional Review (ELIR), within which institutions examine and monitor their

own quality processes, and express this in a Reflective Analysis. The Quality Assurance Agency

undertakes a cycle of quinquennial visits to review the findings of the institutional self-review.”

Questions such as ‘How are we developing the staff who deliver the PGC PHE programme?’, and

‘How are we enhancing teaching practices on the programme?’ are included in the annual

programme monitoring which feeds, eventually, into the ELIR process.

‘With that kind of formal provision, evaluation is embedded and what is really important is that the quality assurance and the quality enhancement agendas do very clearly join up. And then evaluation is linked in to university processes; for example, people will talk at their performance enhancement review [appraisal] about their progress on the programme. So in those formal programmes then evaluation is reasonably straight forward. It’s not innovative, but it is embedded’. Roni Bamber, Director of Centre for Academic Practice

In 2010, a substantive piece of research was undertaken to explore the evaluative practices of

academic staff at QMU, to ascertain what was done beyond formal requirements. Two members of

CAP issued an online survey to around 200 academic staff (see Bamber & Anderson, 2012) and found

that a great deal of evaluation of teaching and learning goes on, but this isn’t always reflected in

formal reporting structures. Much evaluation is informal and local, and often falls below the radar of

formal monitoring.

Roni Bamber suggests that we need to move away from thinking about ‘impact’ and instead consider

‘evidencing value’. Drawing on her experience of running the HEA-accredited CPD Scheme at QMU

she describes a triangular approach that considers three distinct types of data (literature, feedback

from review processes and ‘practice wisdom’) that can be analysed with to demonstrate the value

of CPD activities (See Bamber, 2013).

‘The first element is Research: what is the literature and what are the theoretical frameworks that we are aware of, that would support what we are doing and help us keep developing? Second,

88/222

HEA CPD Pilot Workshop - Workbook

Evaluation: what feedback do we get from externals, from panel members, from participants in the CPD scheme and, also, what might we pick up from their applications that would suggest the value of the Scheme? (And we know there is great value in applying for fellowship – it’s a development activity in itself). And the third type of data is Practice Wisdom: what do we learn from discussions with staff, and from our knowledge of what is appropriate in our context. What do we pick up informally from staff that tells us the value to them of going through that CPD process? Collectively that can be quite rich. It’s quite hard to capture, but that’s one of the areas that I think is most promising, that is probably under-appreciated because it is informal data which may look ‘unscientific’ to anyone taking a very technical-rational stance on evaluation.’

The small size of the institution means that a significant amount of evaluation takes place informally,

within that practice wisdom frame:

‘A lot of what we do happens in the informal domain because everybody knows each other and talks to each other in the corridor and you get to learn all sorts of things about people’s teaching practice and what’s going on with their students, and that comes in to that practice wisdom area. It is really valuable information but it’s not necessarily written down’. Roni Bamber, Director of Centre for Academic Practice

Challenges to evaluating CPD

The biggest challenge to evaluating CPD is that there is no direct cause and effect relationship, and

so impact is always by implication, by interpretation and association.

‘We can only go by what they [teachers] tell us, and usually it’s things like ‘I’m a lot more confident’; ‘I’ve got a greater repertoire of techniques’; ‘I feel as if I am responding better and learning better’; but it is hard to produce concrete evidence of that.’ Roni Bamber, Director of Centre for Academic Practice

Roni has been involved in the evaluation of teaching and learning for many years. In a

previous institution she collected data from PGCert participants over a period of nine years,

and has used a number of tools such as the Approaches to Teaching Inventory(Trigger and

Prosser, 2004), and the Teaching Methods Inventory (Caffey and Gibbs, 2004).

‘We also designed a confidence questionnaire and we did try to find some kind of patterns in the data between entry and exit over that period of time. But there wasn’t, maybe because the numbers weren’t that great. There were no concrete data that we could call significant patterns. So measurable data is hard to establish, but we do get lots of rich evaluation data from discussing with staff their day to day practices, and how they’re developing those. The challenge is to turn that into something meaningful for reporting purposes, in this era of impact and measurement.’ Roni Bamber, Director of Centre for Academic Practice

89/222

HEA CPD Pilot Workshop - Workbook

Future Plans

Bamber suggests that evidencing the value of academic development is a challenging aspect of

evaluating something as intangible as the benefits of developing staff. The staff at CAP will continue

to gather data based on Research, Evaluation and Practice Wisdom, and are looking at how best to

combine formal and non-formal sources of data in ways that suit different purposes (eg for

reporting, for improving our CPD Scheme, for planning future provision).

90/222

HEA CPD Pilot Workshop - Workbook

Work Package 3 (WP 3): Design and test a tool for the assessment of the effectiveness of CPD

This WP was informed by the outcomes of the literature review and national audit, as well as

conversations with interested parties over the course of the project. This was a dynamic WP, which

had to be responsive to the tight timeframe of the project and the complex demands associated

with evaluating the impact of CPD across a diverse HE sector.

3.1 Tool Design and Development

The original intention of this project with respect to ‘tool development’ was to ‘customise,

refine and extend the tool developed by Chalmers et al., (2012)’ in their work on evaluating

teacher preparation programmes. Chalmers et al. (2012) highlighted a number of features of

their tool intended to promote its widespread application. These included flexible design,

the broad evidence base that it encouraged users to engage with and the potential for

capturing reflections over extended timescales (Chalmers et al., 2012). Whilst these were

recognised strengths, we identified several other researchers in this area who had made a

significant contribution to the study of ‘evaluation’ in the context of HE and school-based

education.

Amundsen and Wilson (2013) show that to evaluate the potential impact of CPD activities

effectively you need to consider the rationale underpinning their design. They highlight the

importance of planning the design and evaluation of CPD at the same time.

Cashmore et al. (2012) demonstrate the complexity of the data in career progression

research and the challenges associated with collecting it.

De Ridjt et al. (2013) identify factors that determine the impact of CPD such as participants’

motivation to engage in CPD, and the complexity of ‘transfer’, that is integrating learning

from a CPD activity into practice.

Fink (2013) identifies the range of ‘stakeholders’ that need to be considered in order to

evaluate the impact of CPD. These stakeholders represent institutional leaders, CPD

providers, students and teachers.

Guskey (2002) and Kreber and Brook (2001) both discuss and critique the complexity of CPD

and the varying levels at which we need to evaluate potential impacts. Guskey (2002)

identifies the following levels: participants’ reactions, participants’ learning, organisational

support and change, participants’ use of new knowledge and skills and student learning

outcomes. Kreber and Brook (2001) identify participants’ perceptions, participants’ beliefs

91/222

HEA CPD Pilot Workshop - Workbook

around teaching and learning, participants’ teaching performance, students’ perceptions of

staff’s teaching performance, students’ learning and effects on the culture of the institution.

We drew on the work of Chalmers et al. (2012) and Cashmore et al. (2012) and integrated the ideas

presented by the writers above to inform the philosophy and framing of the tool as well as its

practical use and applicability. The findings of the WP2 national audit were also integral to the

development of the tool.

We originally used the published literature (e.g. Gosling, 2008; Ling, 2009; Parsons et al., 2012;

Prebble et al., 2004; Turner et al., 2013) to determine the scope of the tool with respect to the range

of the CPD activities it would evaluate. At the bidding stage, we anticipated developing resources to

support the evaluation of teaching preparation programmes, teaching accreditation schemes,

conferences, staff development workshops, peer review and teaching development grants.

However, following the national audit (WP2), we extended the scope of this to include mentoring

schemes and collaborative and networking activities. Such activities have been identified in the

published literature as of growing significance in terms of CPD and to date have received limited

attention in the evaluation of teaching-related CPD (see WP1 and, for example, Section 1.4.2)

3.1.1 Influential factors shaping the development of the tool

In order to take account of existing processes and mechanisms to evaluate CPD activities, we used

the academic community as a starting point for our development work. Project team members

drew on the expertise in their networks in order to identify existing CPD evaluation resources. They

also reviewed available resources in the grey literature and resulting from grants awarded by

organisations such as SEDA, the HEA and its former subject centre network. Additionally, we also

drew on examples from our own institutions. In reviewing these existing resources we identified the

following factors as integral to the design of the tool to ensure it promoted a critically reflective

approach to evaluating the impact to CPD:

1) The scope and focus of the tool - Many existing resources seek to capture evaluation data around

satisfaction. Our primary focus would rather be on identifying changes in thinking, practice,

behaviour and action (Bamber, 2013; Rust, 1998; Sword, 2011). These are integral to identifying the

impact of CPD over the longer term and the effects it has on student learning. They also encompass

the so-called ‘soft’ impacts such as increased confidence, thinking differently, and a willingness to

change practice, which benefit student learning but are challenging to measure (Bamber, 2013). We

framed these changes with respect to the following categories identified within published studies on

92/222

HEA CPD Pilot Workshop - Workbook

CPD which discuss the common goals and anticipated outcomes of CPD (e.g. Guskey, 2002; Kreber &

Brook, 2001):

Changes in participant conceptions of teaching and learning

Changes in participant teaching behaviour

Changes to students perception of teaching

Changes to student learning outcomes

Changes to organisational culture around teaching and learning

2) Applicability – Fink (2013) and WP2 (Section 2.4.4) clarified the range of stakeholders who may

use the tool and interact with the outcomes of an evaluation resulting from the tool. We had to

ensure the tool was sufficiently flexible to allow these different groups to evaluate their CPD

activities and result in data on impact relevant to these different groups.

3) Promoting a longitudinal approach to evaluation - The ability to evaluate over time is essential to

identify potential impacts across the categories identified above (Bamber, 2013). The tool would

therefore need to include provision to capture data across extended timeframes (eg during and after

a CPD activity, including prior to the event to provide ‘baseline’ data, if relevant, during i.e. if the

CPD activity is more than one-day in length), immediately afterwards and then six, or 12 months

after the CPD activity).

4) Drawing on qualitative and quantitative data – Drawing on a range of data, and therefore

evidence would ensure a more reflexive approach to evaluation, which is therefore sensitive to the

complex nature of impact (Fink, 2013; WP1 section 1.5. The value of using qualitative data to

evaluate practice has been widely advocated. Considering this alongside quantitative measures, can

develop a more nuanced understanding about commonalities and differences across different

approaches and contexts for CPD in teaching and learning (Chalmers, 2008; Shavelson, 2010).

5) Promoting a flexible approach to methods of data collection. In order to ensure the tool is

responsive to a diverse range of HE contexts and CPD activities, it should enable flexible data

collection. Currently, most evaluation practice is centred on the short answer questionnaire (see

WP2, section 2.4.3) but this tool offers the potential to counter this trend and consider the range of

ways in which evaluation data can be captured. It could, for example, promote the use of focus

groups, interviews, or reflective logs for data collection.

6) Encouraging those engaged in evaluation to reflect on a range of data. Encouraging individuals

and teams to draw on multiple sources of evidence - including existing data - would ensure an

evaluation is robust and captures the range of impacts CPD activities can result in. Such data might

93/222

HEA CPD Pilot Workshop - Workbook

include self-reflections, module evaluations, feedback from colleagues, reviews of data on areas

such as student performance, retention, progression and NSS activities.

3.1.2 Tool development

The first key question concerned who would be conducting the evaluation. We identified the CPD

provider, for example the Educational / Academic Developer or HE manager in FE contexts) as most

likely to be taking the lead in evaluating the impact of CPD activities. These are the groups with the

remit for teaching-related CPD and are well placed to co-ordinate a systematic approach to impact

evaluation using this tool. Their usual central location in institutional structures offers the potential

for the outcomes of evaluation activities to inform future CPD and teaching and learning

development work across an institution (McKenna & Hughes, 2015). We recognise the

organisational structure may be different in an FE college or private provider, though we anticipate

there is someone in these institutions with a remit that encompasses activities such as staff

development or quality enhancement. Therefore the tool targeted this group at the design stage.

The toolkit aims to guide the evaluator through the process, from identifying the focus and purpose

of the evaluation, choosing evaluation questions relevant to stakeholders, identifying ways to collect

evidence relevant to the questions, to conducting the evaluation and making use of the results.

The CPD activity-type was chosen as the starting point for evaluation planning. We identified this as

an accessible entry point on which to focus evaluation activities that would work across a range of

institutions. For each activity, we then developed lists of open questions for the potential

stakeholders. The workbook in App 2 illustrates this approach, with sample questions relating to the

impact of attending a conference. The different stakeholder groups (e.g. providers, completers,

students, etc.) were addressed in the following ways:

1) Prompts were written for the CPD provider to trigger reflection on the design and framing of

the CPD, as well as examining their rationale in undertaking the evaluation and the expected

outcomes / implications for practice;

2) Questions were designed for selection by the CPD provider to evaluate the impact of a CPD

activity on the CPD completer.

3) Questions were themed around the categories outlined above (e.g. changes in conceptions

to teaching and learning). Additional questions were included to capture data around CPD

completers’ motivation and reaction to a CPD activity, since these have been identified as

having an impact on whether the intended outcomes of a CPD activity are realised and/or

whether learning is transferred to practice (e.g. De Ridjt et al., 2013).

94/222

HEA CPD Pilot Workshop - Workbook

4) Questions were also designed for students who may have contributed to the design or

implementation of a CPD activity. These sought to capture the students’ involvement in the

CPD activity and the potential impacts on their academic development. The questions for

the CPD participant and students were presented over an extended timescale, giving the

potential for evaluation to be conducted prior to and during a CPD activity, which could

provide useful data on expectations and motivations, as well as initial reactions to a CPD

activity and potential changes to practice that may result from engagement with a CPD

activity. To prompt on-going reflections on CPD activities and their impacts we identified

common questions, which were asked at different time points to allow a participant to

revisit past reflections and use these as a stimulus for future reflections. Depending on the

CPD provider’s responses to the initial questions regarding the design of the CPD activity and

the purpose of the evaluation, they are then directed to select questions from each of the

recommended timeframes for completion by CPD participants. The CPD provider then

collates the selected questions and identifies the method of answering these.

The toolkit also provides guidance on the following areas:

1) Evaluating CPD completes participants’ reactions and experience of a CPD activity – currently

this is the focus of much evaluation practice, and though it provides limited information

regarding the impact of CPD activities, it does give essential information on areas such as the

content and delivery of a CPD activity, the facilitator and the administration, facilities and

logistics. (This immediate evaluation also provides baseline data against which to consider

longitudinal findings.)

2) Methods of data collection – in order to encourage CPD providers to consider alternate ways

of evaluating practice, the toolkit includes a brief overview of the potential role of

questionnaires, interviews and focus groups in evaluation. Information was provided on the

strengths and challenges of each method of data collection, methods of implementation and

ethical considerations. Additionally, a resources list to assist in these collection methods is

provided.

3) Using evaluation data - The use of evaluation results is potentially sensitive and there may

be tensions between the need to inform further educational development and a desire to

monitor or judge performance. The tool therefore includes guidance relating to

communicating information about CPD impact, guiding understanding, formats and media

for reporting findings, responsibilities to stakeholders and other ethical issues.

95/222

HEA CPD Pilot Workshop - Workbook

3.2 Tool Implementation and Piloting

Initially, the tool seemed best suited to online presentation so that different views could reflect the

variety of stakeholders, perspectives and evaluation aims. However, following the design and review

of the tool by the project team, the tool was presented as a hardcopy ‘workbook’ for piloting

nationally. (See Appendix 2) The decision to present the tool as a workbook rather than online

resource was to ensure that, at this stage, we received feedback on the content and focus of the tool

as this was the primary concern. It was felt that until these aspects of the tool were finalised,

piloting the tool as an online resource was premature. Usability issues could be distracting, with

potential implications on the quality of the feedback.

The national pilot was focused on individuals with a remit for staff and educational development

work across the HE sector. This included representatives from two FE colleges, two private providers

and universities, covering research and teaching focused institutions. Originally we intended to

include eight institutions, in addition to the two project partners, in this pilot but, following receipt

of funding, a further two institutions were invited to be participate as this would increase the

breadth of CPD activities on which the tool would be piloted. The following 12 institutions were

involved in the pilot:

Bangor University

Bath Spa University

Bradford College

Durham University

Greenwich School of Management

Manchester Metropolitan University

Nottingham Trent University

South Devon College

University of Bath

University Campus Suffolk

University of Greenwich

University of Edinburgh

A named contact from each institution was invited to a one-day workshop at the University of

Birmingham on the 20th of March 2015. The purpose of this day was to present progress with the

project to date, to introduce the tool, gather immediate feedback and prepare for the pilot process.

96/222

HEA CPD Pilot Workshop - Workbook

Ten of the pilots were able to attend, with the remaining two contacted at a later date via Skype to

introduce the tool and undertake the necessary preparation.

Following the one-day workshop, the CPD providers had three weeks in which to review the tool and

prepare to evaluate their CPD activities. They were requested to identify CPD activities that could be

evaluated using the tool. Depending on the breadth of their institution’s CPD offer and individual

interests, they chose to focus on either one CPD activity or a range of activities. Using the tool, they

identified the focus of their evaluation, selected evaluation questions and the method by which the

evaluation would be conducted (e.g. as an interview or questionnaire). Following this, they returned

the workbook to the project team. We requested return of the workbook as this represented an

important source of feedback on the tool to inform its on-going development.

The pilot CPD provider also identified three or four members of staff (CPD completers) who had

engaged with the chosen CPD activity in the past 12-24 months and would be willing to assist with

the pilot. The CPD provider and Plymouth team then arranged a mutually convenient date for a

member of the project team to visit the institution, meet the CPD completer(s) and undertake the

evaluation. During this meeting the CPD completer would evaluate their CPD experience, and

potential impacts on practice, using the questions and the method chosen by the CPD provider.

They would also be interviewed about the process of evaluating their CPD activity. This interview

captured the CPD completer’s reflections on the questions they responded to, considering their

accessibility and relevance, as well as contextual information regarding the CPD activity under

consideration. The meeting and subsequent interview was informed by a ‘think aloud’ methodology

(Cotton & Gresty, 2006), and the interviewees were encouraged to reflect on the questions they

were asked. In total 40 interviews were held with CPD completers from across the pilot group.

The meeting with the CPD completer was audio recorded and transcribed in full. Data were

thematically analysed by the project team. Three members of the project team undertook analysis

of the transcripts from the first pilot individually, and then came together to discuss and refine

emerging themes. Following this, an analytical framework was developed and subsequently applied

to the whole data set. The outcomes of this analysis provided insights into completers’ experiences

of the CPD they had engaged with and its impacts on their practice. Insights were also gained into

the process of evaluating their CPD activity using the tool. These are reported in section 3.4.2.

97/222

HEA CPD Pilot Workshop - Workbook

Based on the first category of information, brief summaries were developed of the collective CPD

experiences of CPD completers for return to the CPD provider. Each summary touched upon themes

pertinent to CPD e.g. individuals’ motivation for engaging with the activity, impacts on areas such as

conceptions of teaching and learning, teaching behaviour, future CPD and suggested improvements

to each type of CPD activity based on the data collected. These summaries gave an indication of the

potential outcomes of the tool as well as offering anonymised feedback on the overall breadth of

CPD provision represented by the pilots. These were returned to the CPD provider to review prior to

the final stage of data collection – an interview to capture their experiences of using the tool to

inform the final refinements to the tool.

3.3 Tool Evaluation

The tool was evaluated based on the following sources of data:

1. A review of the workbooks and the questions CPD providers selected to evaluate their

practice;

2. The outcomes of the analysis of the CPD completers’ evaluation data and reflections on the

questions they were asked to respond to;

3. A review of the summaries produced regarding the impacts of CPD activities on participants’

practice.

Based on these data sources, an interview schedule was developed to capture the CPD providers’

perceptions and experiences of using the tool. The schedule specifically explored the CPD

providers’ roles with respect to CPD and its evaluation, their approach to using the tool and rationale

for choosing the evaluation questions and methodology used. We also solicited feedback on the

tool and recommendations for on-going development of the tool. Ten CPD providers were

interviewed via Skype by a member of the Plymouth project team. Interviews were audio recorded

and transcribed in full.

98/222

HEA CPD Pilot Workshop - Workbook

3.4 Results of the national pilot of the evaluation tool

In this section we present the findings of the piloting of the tool within 12 UK-based institutions.

Providers for teaching-related CPD used the workbook to select questions and appropriate methods

of evaluating the impact of CPD activities previously engaged in by members of their teaching staff.

40 members of teaching staff selected by the CPD providers were then asked to answer the

evaluation questions in an interview situation. The interviews were carried out by a research

assistant from Plymouth. We also report on data collected from SU representatives from seven of

the pilot institutions. These data have provided insights in the following areas:

1) CPD completers experiences of using the tool to evaluate the impact of previous CPD

activities they had engaged with9;

2) CPD providers experiences of using the tool to evaluate their CPD provision captured

through an analysis of the workbooks (e.g. questions and methods selected) and subsequent

interview to discuss the evaluation process and the outcomes of the CPD completer

interviews;

3.4.1 Piloting of the tool

The CPD completers involved in the pilot represented the full range of CPD activities we identified in

WP2, Section 2.3.1. In brief, most CPD providers focused their evaluation activities on teaching

courses for new lecturers and in-house accreditation schemes (Table 3.1), therefore focusing on

formal CPD provision aligned with the UKPSF. This was not unanticipated as these are areas of CPD

often perceived as core to educational development work, and activities individuals are most likely

to engage with due to institutional drivers including probation and promotion (Smith, 2011;

McKenna & Hughes, 2015). Also, it is the CPD that affords ‘recognised status’ that institutions are

now required to report as part of their data return to HESA (HESA, 2012). Activities such as teaching

and learning workshops, teaching and learning conferences, peer review, teaching and learning

development projects and mentoring, were included in the pilot by fewer CPD providers (Table 3.1).

Although these are CPD activities that are common across the sector, and serve an important

education development and enhancement function, engagement with these activities tends to be

voluntary and more at the determination of the individual academic rather then due to a central

driver / institutional requirement (Parsons et al., 2012). Therefore in terms of prioritising which

9 As a consequence of piloting the tool the CPD completers reflected on the CPD activity they had engaged with which they were now evaluating. These reflections were collated and short summaries were produced for each CPD activity which summarise the perceived impacts on practice and suggestions of how the CPD activity may be improved based. These summaries were discussed with the CPD providers as part of the evaluation,

and are included in Appendix 3.

99/222

HEA CPD Pilot Workshop - Workbook

activities to evaluate this could have been a clear rationale underpinning provider choice. Likewise,

some HE providers e.g. private providers or FE colleges may have smaller CPD provision compared to

universities, and therefore this automatically determined which activities the CPD provider chose.

Table 3.1: Range of CPD activities included in the pilot

CPD activity Number of CPD completers the tool was tested on*

Teaching courses for new lecturers 12In house accreditation Schemes 14Other CPD activities** (Teaching and learning development projects, teaching and learning workshops, teaching and learning conferences, peer review and mentoring)

14

* Some institutions chose to pilot the tool on a number of different CPD activities; therefore although 12 institutions were included in the pilot, the number of activities the tool was piloted on was greater.

** For the rest of the chosen CPD activities the number of CPD completers piloting the tool was relatively low. Due to our ethics and the requirement to maintain confidentiality we have not stated the exact number of CPD completers piloting the tool across the rest of these activities.

CPD activities have the potential to impact upon a number of different aspects of an individual’s

practice (e.g. their conceptions of teaching and learning, the institutional culture and students

(Guskey, 2002). The impact a CPD activity has on these different areas is influenced by an

individual’s motivation to engage with a CPD activity, their experience in their role and their reaction

or response to it (De Ridjt et al., 2013; Parsons et al., 2012). As many of our CPD completers had

been involved in teaching courses for new lecturers that are often a probationary requirement

completed by those new to HE-level teaching, the level of learning that participants engaged with

may have been highly variable. This could therefore have influenced the evaluation they undertook

of this CPD activity and the learning they identified.

In designing the tool we themed the questions to cover the key areas of impact CPD activities are

recognised to have as identified by Guskey, 2002 (see Appendix 2). Although we discussed the

relevance of each theme to assessing impact of CPD provision, we left it to the pilots to select their

questions according to the aims of their evaluation and the learning outcomes / goals of their CPD

provision. Table 3.2 summarises the number of questions selected from each theme. The data are

based on frequency counts across each CPD activity. It may be that where a CPD provider was

evaluating more than one activity there was repetition in the question, hence the higher frequency

100/222

HEA CPD Pilot Workshop - Workbook

counts for some themes. These data highlight patterns in question and theme choice relevant to the

on-going development of the tool.

The majority of the CPD providers selected questions that would capture data regarding CPD

completer’s motivation / reaction to a CPD activity, measure changes in conceptions of teaching and

learning, changes in teaching and learning behaviour and impacts on institutional culture. These

areas are the focus of much existing evaluation practice (e.g. Lau & Yuen, 2013; Lee et al., 2013) and

align with the recognised benefits of teaching-related CPD (Stes et al., 2013a; Willets, et al., 2014).

Very little emphasis was placed on questions that would provide insights into changes in students’

outcomes or changes in student perceptions of teacher behaviour. This was not unexpected, as

these are widely recognised as the challenging areas in which to evaluate and gaps within current

evaluation practice (Parsons et al., 2012; Trigwell et al., 2012). They do indicate the need for

inclusion of clear guidance within the tool of ways in which they can be measured and how, for

example, existing data sets (NSS / in house student satisfaction surveys) can be used to provide

evidence of impact.

101/222

HEA CPD Pilot Workshop - Workbook

Table 3.2: Frequency counts of the themes from which the questions were selected across the range of CPD activities included in the pilot.

Teaching courses for new lecturers

In house accreditation Framework

T&L workshop

T&L conference

Peer Review

Teaching and learning development projects

Mentoring Total

Motivation 25 21 4 1 1 4 1 57Changes in conceptions of teaching and learning

20 18 1 3 1 2 45

Changes in teaching and learning behaviour

23 15 2 1 4 1 46

Changes in student perceptions of teacher behaviour

2* 1 1 4

Changes in student outcomes 13 8 1 3 25

Impacts on institutional culture 14 26 7 1 1 7 1 57

* These questions were added by a CPD provider

102/222

HEA CPD Pilot Workshop - Workbook

3.4.2 CPD completers experiences of using the tool

The CPD completers had also engaged with their CPD activities over differing timeframes (Table 3.3). This

provides some provisional insights into the potential of the tool to evaluate impact over extended timeframes.

As data were only collected from one point in time rather than over repeated points these findings are tentative

Despite this, we did gain insights into the impact of timeframe upon the extent to which CPD completers were

able to evaluate the impact of their CPD activity on their practice. For example, the CPD completers piloting the

tool often needed prompting with respect to what the content or focus of the CPD activity was. This was

particularly evident where there had been a considerable time lapse between the CPD activity and the

interview. Prompts were included to stimulate these reflections, however, regular engagement with impact

evaluation following completion of a CPD activity, as advocated through this tool, may help overcome issues of

recall.

The difficulties the CPD completers experienced in recalling the CPD activity and its potential impacts could also

be indicative of the positioning of the CPD activity. That is to say, it was frequently perceived as being

undertaken outside of the ‘everyday role’ of being a lecturer and supporting students (Han, 2012).There was

therefore an absence of regular dialogue with colleagues and peers to integrate learning into practice (Brown &

Inlis, 2013). Regular dialogue has been identified as integral in maximising the learning that can take place from

CPD, and ensuring the so-called transfer to practice (Brown & Inlis, 2013). Struggling to reflect on their

experience and identify the learning that had taken place indicates that the opportunities to enact or practice

the knowledge, skills and experiences presented in their CPD may have been limited. Engaging in regular

dialogue and reflecting on their CPD experiences through the tool could create such opportunities.

Table 3.3: Timeframes over which the CPD completers had engaged with their CPD activities that were the subject of the pilot.

CPD activity TimeframePre During Post 6 months 12 months+

Teaching and learning 2 3 4 5 3

103/222

HEA CPD Pilot Workshop - Workbook

development projectIn house accreditation scheme

3 2 5 3 4

Other CPD activities*

1 1 1 3 2

* Again data amalgamated as in Table 3.1

The clarity and accessibility of the questions was an issue for a number of the CPD completers. Educational

development work is recognised for having a language of its own (Green, 2009), and although a CPD completer

may have been cognisant of this at the time of undertaking the CPD activity, the use of such context specific

language was identified as problematic for some. The language in educational development is also highly

variable, often changing across institution types and indeed counties (Green & Huston, 2006). This highlights the

need for the CPD provider to contextualise the tool to their own institutional context.

Accessibility and language was also a key issue for those for whom English was a second language and for one

dyslexic participant who expressed concerns regarding the accessibility of the questions. In some cases the CPD

completers were provided with the questions in advance to the data collection. This was recognised as

advantageous, as the CPD completers felt it allowed them to engage with a more in depth reflection on their

experience and the potential impacts on practice:

“If I’d have had [the questions] beforehand, and had a bit of time to think about them, then I would

have found it easier to answer the questions because I would have had that processing time.” (Pilot 3)

The CPD completers also highlighted the need for a clear and logical flow to the evaluation questions. Integral

to this was the space for discussion and reflection on the specific CPD activity that was the focus of the

evaluation.

“For me yes I think the discussion we have just had has made me realise how far I have come in the last couple

of years, it’s made me think about what I’m doing and why I’m doing it and what it means to me and what it

means to everyone else so yes reflection is always good and that was a good half hour of reflection we had.”

(Pilot 12)

In terms of developing an approach to evaluation that takes place over extended timeframes, it highlights the

need for common questions or themes to allow the CPD completer to revisit past experiences and identify

104/222

HEA CPD Pilot Workshop - Workbook

future learning. This would also lead to a gradual shift in evaluation practice from the widely critiqued focus on

outputs to outcomes, learning and change (Stefani, 2011).

Current evaluation practice commonly involves the use of ‘satisfaction’ questions, for example those that assess

whether the CPD participant was happy with their experience. This approach has been discussed in the

literature (e.g. Amundson & Wilson, 2012; Chalmers and Gardiner 2015), as providing limited insights into the

potential impacts of CPD. We do recognise they are useful in terms of assessing the immediate experience of a

CPD participant with an activity or the quality of learning promoted by a particular facilitator. Therefore in

presenting the tool to the CPD providers we acknowledged the potential limitations of these questions and

guided caution in their use. Consequently only one CPD provider selected such a question. However, CPD

completers were keen to report their level of satisfaction with a CPD activity and indicated this was an important

part of the evaluation process.

3.4.3 Recommendations for the further development of the tool based on the analysis CPD completer data

Consider the needs of both the CPD completer and CPD provider in the recommendations regarding

questions choice. Although the CPD may be more focused on the impacts on practice, questions also

need to be included that allow the completer to reflect on the actual CPD activity and learning that

resulted, and offer feedback on their experience of the CPD activity.

Guidance should be given to the CPD provider on the ordering of questions. ‘Exemplar’ question sets for

each CPD activity may be useful.

Guidance should be given to the CPD provider regarding the need to contextualise the questions to their

institutions, including relevant terms of reference to CPD activity, the Educational Development unit etc.

Giving the CPD completer time to review the questions prior to engaging with the evaluation would

promote inclusivity and allow time for greater reflections to be made on their experiences.

3.4.4 CPD Providers experiences of using the tool

As Table X indicates, most CPD providers focused on evaluating a specific CPD activity, rather than considering

the specific educational development themes (or learning outcomes) that underpinned their design. This use of

the tool does reflect its presentation (see Appendix X - workbook), in that it is focused on the CPD activities.

Within the CPD literature, most evaluation work is centred on the activity itself (Bamber, 2013; Parsons et al.,

2012). However, we have attempted to draw attention to these wider themes through the organisation and

105/222

HEA CPD Pilot Workshop - Workbook

presentation of the questions related to each CPD activity. This was identified as useful by the CPD providers as

they felt it served as a prompt in question choice.

Table 3.4: What the CPD providers were using the tool to evaluate

Purpose of the evaluation N

A named activity 10Changes in conceptions of teaching and learning 2Changes in teaching and learning behaviour 2Changes to student learning 3Changing identity as a teacher 1

The CPD providers identified a number of motivators underpinning their desire to evaluate their practice, all of

which resonate with current practice across the sector (e.g. Bamber, 2013). Educational developers are

expected to provide evidence of impact in numerous forms for a range of audiences (Bamber, 2013). Despite

high numbers reporting that evaluation feeds into future CPD offers there was little evidence or interest in

evaluating the on-going value of CPD on the individual academic. There is clear benefit in tracking individual

academics continued interactions with CPD, teaching and scholarly activity as a way to evidence value and

establish future CPD need. Indeed, given agendas inherent to the Browne Review (2010), such as the publicising

of teaching qualifications by HESA and the agenda for good standing, tracking individuals on-going interaction

with CPD is likely to be of increased relevance to policymakers and institutional managers. This is a potential

outcome from using the tool which may fulfil a gap within current evaluation practice.

Table 3.5: The purpose of the evaluation

Why evaluate? N

To inform future CPD offer 8To inform institutional policy 6To articulate and evidence value 3Evidence for internal auditing 3Evidence for QAA audit 2Individuals’ on-going academic development 2

A range of audiences were identified as recipients of the impact evaluation data (Table 3.6). It is likely that

senior managers and committees will be provided with brief updates which can help to inform decisions about

the value of CPD activities. More nuanced narrative data will better inform future CPD and assist CPD providers

in understanding issues around transfer to practice (De Ridjt et al., 2013). The toolkit could therefore usefully

provide advice about what sorts of data are appropriate to use for different audiences.

Table 3.6: The intended uses of the evaluation data

106/222

HEA CPD Pilot Workshop - Workbook

Use of evaluation data NInforming future CPD including development of metrics 5Reported to senior management 3Presented at committees 2Reported to HR 2Reported to QAA 1Reported to respondents 1

Seven pilots chose to use interviews as their preferred methodology to collect evaluation data from the CPD

completers, with the remaining two using open questionnaires. These approaches follow trends in evaluation

practice which advocate the importance of qualitative approaches to data collection and the potential to

develop greater insights into the impacts of CPD through narrative accounts (Bamber, 2013; Rust, 1998).

However, the intended audiences of the evaluation data are mixed (Table 3.6), and it is likely that some, such as

university committees and senior managers, who may be more interested in a headline summary of satisfaction

or numerical overviews of impacts. Therefore mixed method approaches should be considered in order to be

able to communicate the outcomes of evaluation work in both qualitative and quantitative formats.

The CPD providers identified numerous benefits following the use of the tool to evaluate their provision. They

acknowledged its flexibility and the ease at which it aligned with, but also enhanced, their current evaluation

practices, particularly in terms of promoting a longitudinal approach to evaluation. This was a current gap in the

evaluation provision of most of the pilot institutions.

‘One of the first challenges you often have is coming up with the evaluation questions, so actually

having a readymade resource that has started to categorise things into different evaluation

scenarios, and starting to give you different questions for different levels of evaluation, we found

that really helpful…. you’ll end up with better evaluation tool for whatever you design because

some of that thinking and working through and revision have been done already..’ (Pilot 7)

In line with the concerns from the CPD completers regarding the language used in the tool the CPD providers felt

that where necessary there was the potential to customise the tool to ensure it was sensitive to their own

institutional context. The recognition of the need for this by CPD providers is important, as it could mediate

potential accessibility and language issues raised by the CPD completers.

The comprehensive nature of the tool was also valued, in that it covered the breadth of CPD provision, and

allowed robust, longitudinal evaluations to be conducted across this range of activities:

107/222

HEA CPD Pilot Workshop - Workbook

‘We thought the theme ones are really good, so the themes cover pretty much everything that we

wanted to look at anyway. There were a few ones that we added about the UKPSF and whether the

students know that lecturers are engaging with CPD activities.’ (Pilot 11)

‘Given that it’s a hundred percent more than most people are using ever, just taking it from where

it is at the moment would be really useful.’ (Pilot 7)

They also felt the connection within the tool to academic development themes and / or direction to considerer

the learning outcomes of a CPD activity improved alignment in evaluation practice, overcoming a noted

limitation of much evaluation practice (Guskey, 2000).

The CPD providers identified benefits to those who engaged in the evaluation process, citing the value of

engaging in meaningful dialogue around their previous CPD and the contribution this could make to planning

future CPD. Again this could benefit the growing agenda for ‘remaining in good standing’ as advocated through

the UKPSF. The process of reflecting on their learning was also seen as valuable in terms of making connections

between their individual practice and wider institutional processes and priorities.

Through engagement with the tool, both in selecting questions and reflecting on the completers collated

responses to the CPD activity, providers became aware of their current practice, identifying limitations and also

how they would seek to develop their evaluation practices in the future. CPD providers seemed acutely aware

that their current evaluation was focused on satisfaction rather than evidencing value as noted elsewhere (e.g.

Bamber, 2013). This was attributed partly to historical precedence but also to institutional administration

processes. Providers’ current evaluation methods included some focus groups and exit interviews but centrally

designed module evaluation forms were widely used. In each case there was critique of the extent to which the

module evaluation questions reflected teaching and learning themes and the reliance on self-reporting at a

singular time point, normally immediately post event.

‘We evaluate at the end of the event, but real impact can only be observed after years and years.’

(Pilot 4)

‘The evaluation we do is a judgement on the activity, not of its impact on people’s practices.’ (Pilot

10)

The toolkit was perceived to offer a structure for better alignment between the learning outcomes/intentions of

the CPD activity and its evaluation which could potentially influence the CPD providers understanding of value.

108/222

HEA CPD Pilot Workshop - Workbook

The data from the CPD completers prompted providers to consider the links between CPD provision and other

institutional processes, in particular annual and peer review, probation and promotion. These mechanisms were

seen as having potential to provide ‘systematic and supportive’ (Pilot 10) opportunities for evidencing teaching

quality and remaining in good standing. The success of these links was thought to dependent on the experience

of managers to discuss and promote CPD, the value of CPD and teaching quality within the institution, and the

extent to which annual and peer review are centrally coordinated. Providers suggested that data generated by

the toolkit could be used in conjunction with peer review and other quality mechanisms to inform a

conversation in annual review around remaining in good standing and future CPD needs. This is a future

development of the tool which can be realised through further refinement and piloting. There were however

perceived barriers to enhancing these links. There was recognition that at present, annual review was not in

most cases either focused on teaching development / CPD or linked to CPD provision. Although some

institutions captured individual CPD trajectories this was the minority and only one case linked this to annual

review. The location of the CPD provider within the institution varied and this influences the sphere of influence

they experienced.

‘Annual review….its variable the extent to which people take part in that, take it seriously and use

it developmentally.’ (Pilot 12)

‘We hope that appraisals can pick up CPD needs within specific departments.’ (Pilot 11)

The tool provided evidence that some CPD activities were adding value for the lecturer and wider institution in

terms of confidence changes to conceptions and changes to practice. However, there was less evidence about

value for the student learning experience, as completers were largely unable to answer these questions. CPD

providers were also unsure about how to evidence this. This gap was also evidenced through the analysis of

question choice (see Table 3.2 – frequency counts).

‘It suggests the CPD wasn't impacting on student outcomes…but we just don't know that, and we

can’t in any clear or direct correlated way’ (Pilot 10).

‘I can demonstrate that CPD around the PSF has an impact on teaching practice but to evidence the

same for students is impossible to be honest.’ (Pilot 6)

‘Getting at the important stuff is so much harder, so much more expensive and will take forever. Is

in the too difficult box, we don't go there. But if you want change, real change then that is what you

have to do.’ (Pilot 4).

More widely the challenges experienced by CPD completers in identifying impacts on students outcomes, and

the limited extent to which these questions were integrated in the pilot, indicated further work is needed in this 109/222

HEA CPD Pilot Workshop - Workbook

area. It is clear that CPD completers do not know how to evidence this, and in several cases has prompted

providers to consider why this was and how they might start to have a conversation around this issue. For

example, as a result of the data received, one institution was prompted to contact their student union to discuss

how best to raise awareness of CPD and its value for the student learning experience and all completers

discussed the need for guidance in articulating possible links between CPD, teaching and learning.

‘I haven’t taken it that far in terms of thinking about the impact that CPD has had on student

learning….I wouldn't say my questions are particularly stretching ones, whereas these are. So I think

the workbook did make me think more about that, a bit more deeply about what is it I want to know

and then how is the best way to achieve that, and which questions would help with that or what

other questions do I want to add in. So this exercise has informed me to think about what I will do

with my evaluation in the future.’ (Pilot 9)

Although the tool does draw attention to the need for firstly, consideration of potential impacts on students,

and secondly the use of institutional data sets that can provide a proxy of impacts on students, the inclusions of

questions around these theme and use of data beyond the CPD own reflections on their experiences and

practice, were both limited. In the on-going development of the tool we need to considerably strengthen the

focus placed upon these two elements we identified as integral to evaluating the impact of CPD activities on the

student experience.

There is also a wider issue that is outside the scope of this current work, but relevant to the potential uses of the

tool to determine the impact of CPD on the student experience. Much CPD takes place with limited student

input. The opportunities, or even potential for students to engage with teaching-related CPD are few and far

between, as reflected in the limited discussion of this subject across the Educational Development community.

Although projects (Campbell et al., 2007) have sought to promote student engagement with the CPD of their

lecturers, there are many barriers that need to be overcome before this can happen. Likewise student

awareness of staff CPD is implicit. Discussions that took place as a consequence of undertaking this project

indicated that students have limited awareness or knowledge of the training or development their lecturers

engage with. Therefore, in order to determine an impact on student learning, even at a basic level, we need to

raise student awareness of the CPD lecturing staff engage with and the impacts it can have on their teaching and

learning. This is beyond the scope of this project, however, needs to become part of wider discussions students

are increasingly engaging with through their Student Unions and projects such as the Student Engagement

Partnership.

110/222

HEA CPD Pilot Workshop - Workbook

3.4.5 Limitations to the current evaluation work

We aimed to ensure the evaluation was inclusive of the breadth of teaching-related CPD taking place across the

sector. As discussed, the pilot was centred on specific CPD activities (teaching preparation programmes and in

house accreditation schemes) largely as a consequence of the particular interests of the pilots involved in this

work. Although the tool was piloted on activities such as peer review, teaching development grants, mentoring

etc., the number of CPD completers piloting the tool on these activities was limited. We recognise the need for

further attention to piloting the tool on these activities beyond the life of this current project to ensure the tool

can effectively be used to evaluate the impact of these activities.

The extent to which the tool could be piloted over extended timeframes as advocated was limited by the

duration of the project. Further testing of this is needed, with CPD completers engaging with the evaluation

process from the benchmarking stage through to 12-24 months following a CPD activity. This will provide robust

insights into the effectiveness of the tool for evaluating over extended timeframes, and consider the potential of

the tool in shaping future CPD and promoting dialogue around the learning that can take place through CPD.

3.4.6 Implications for the further development of the tool based on the analysis CPD completer data

1. The toolkit should provide evaluation opportunities for all levels of evaluation expertise, from the novice

to those looking to embed more advanced processes. This will be best expressed through the guidance

for using the toolkit.

2. The role of students will be considered more explicitly by providing guidance for how to evaluate

student learning as well as satisfaction and making suggestions for how to raise awareness of and

contributions to CPD in the student body.

3. Although the CPD providers identified the potential for integration of the UKPSF in the toolkit, the

project team decided against explicitly embedding the UKPSF at this point. It was felt more useful for

the CPD provider to integrate the UKPSF as they select questions to align with the objectives /learning

outcome of the CPD activity.

4. Providing the question sets in quantitative form so there is ‘off the shelf’ provision of this. Quantitative

data perceived as useful for evaluating large numbers of participants and providing overview data for

the wider institution. This approach is also more cost effective which was a factor in most cases

5. The language in the toolkit needs to be accessible so important to mitigate the use of academic

development terminology

111/222

HEA CPD Pilot Workshop - Workbook

6. Using the data generated alongside other existing data sets to develop enhanced understanding of the

value CPD has for the lecturer, student and wider institution.

7. A section outlining guidance on how to collate, present and communicate evaluation data will be

embedded.

3.5 Recommendations

3.5.1 Higher Education Academy

An evaluation tool has been developed that builds on existing work (e.g. Chalmers et al., 2012) and has

been tested across a range of HE providers (e.g. Research / teaching focused HEIs, FE Colleges and

Private Providers). The breadth of the CPD provision across the pilot institutions determined the extent

to which the project team were able to pilot the tool across the full range of activities. As a result we

recommend further testing of the tool across teaching and learning conferences, teaching and learning

workshops, mentoring, peer review and teaching development grants.

The tool is built on the principle of promoting an approach to evaluation that takes place longitudinally.

Due to the limited timeframe of the project we recommend that further piloting takes place which

follows CPD completers through the evaluation process, from collecting pre-entry responses to the

capturing of evaluation data 12-24 months after a CPD event.

The project team have delivered a paper-based tool in line with guidance from the HEA. Future work

should focus on converting this to a fully functioning resource that is hosted online. The tool has been

tested by CPD providers. In addition, there is the potential for teaching staff to use the tool to evaluate

their on-going professional development. At the beginning of this project we developed an online

prototype which can be viewed here – http://www1.plymouth.ac.uk/research/pedrio/Pages/HEA-CPD-

Framework.aspx. (Please note that this is representative of an earlier version of the toolkit, February

2015).

Expertise surrounding evaluation is highly variable across the sector; although the tool has been

designed for use by those with different levels of evaluation expertise, there is an identified need for

further capacity building in this area. The HEA should include training and support for evaluation as part

of its enhancement offer.

There is considerable potential for students to make a contribution to teaching-related CPD, and

through this project we identified examples of best practice in this area. The HEA could usefully take

forward this agenda, perhaps in collaboration with organisations such as SEDA and the NUS, to consider

how students can be supported in making meaningful contributions to teaching-related CPD.

112/222

HEA CPD Pilot Workshop - Workbook

3.5.2 HE Providers

This work has identified connections between individuals’ CPD, annual performance development

reviews and mechanisms of reward and recognition. Commonly these are processes that operate in

isolation and could benefit from greater coherence. Institutions should consider how data regarding

teaching-related CPD is collated, disseminated and used as a forum for discussion in performance

development reviews;

Some institutions have moved toward standard mechanisms of evaluation particularly for CPD that is

institutionally accredited / delivered as part of a course. This can act as a barrier to meaningful

evaluation. Institutions might usefully consider how they can build flexibility in these systems to allow

for evaluation that links explicitly to the aims of the CPD offer.

Students regularly provide feedback on their teaching and learning experiences and have the potential

to input into teaching-related CPD. Institutions could usefully consider how students can be supported

to engage with CPD.

3.5.3 Educational Developers / CPD providers

The design of CPD and plans for evaluating the impact of provision should be considered collectively and

built into the long-term implementation and development of CPD activities;

CPD providers should build appropriate evaluation into all CPD provision. This should include the

capture of pre-entry / baseline information as well as evaluation over extended timeframes, as

advocated in the tool. However, CPD providers need to be mindful of their expectations regarding

evaluation, ensuring they are appropriate to the learning outcomes and the duration of the CPD activity.

CPD providers need to consider how they communicate the outcomes of their evaluation work and with

whom. They need to be mindful of ‘who’ is the best person to be talking with to ensure a clear message

regarding the importance of CPD is communicated across the institution.

CPD providers ought to promote a culture of reflection and evaluation into CPD provision, whereby all

participants become cognisant of the principles and practice of evaluation in order to transfer this into

their own practice.

Evaluation should become a team activity focussed not only on individual outcomes but which considers

CPD collectively, in light of institutional priorities.

CPD providers should introduce teaching staff to a wide range of evaluation tools including small scale

action research so that they are best able to evidence the impact of changes they introduce.

113/222

HEA CPD Pilot Workshop - Workbook

3.5.4 Individual

Individuals with responsibility for teaching and supporting learning should be supported to capture and

reflect on ongoing CPD involvement and its value in line with the UK Professional Standards Framework.

3.5.5 Future work

Most CPD provision is interdisciplinary and cuts across disciplinary boundaries. As a result the tool is

framed in this way. However, given the nuances of disciplinary communities there may be value in

considering how engagement with the tool may be promoted across different disciplines.

The potential for the tool to promote synergies between planning for CPD, performance development

review and mechanisms for reward and recognition was repeatedly highlighted in the data. This is an

area that warrants further investigation.

The contribution students can make to staff CPD is an area that has received limited attention and could

usefully be explored to better align CPD and the student voice.

The tool has been developed to focus on individual CPD activities however, there is considerable

potential for evaluating the impact of the full range of CPD activities individuals engage with over the

academic year, and their careers. Mechanisms need to be considered for how this could be supported

within the tool and how individuals can be supported to engage with this.

114/222

HEA CPD Pilot Workshop - Workbook

References

Amundsen, C., & Wilson, M. (2012). Are we asking the right questions?: A conceptual review of the educational development literature in higher education. Review of Educational Research, 82 (1), 90-126.

Antoniou, P. & Kyriakides, L. (2013). A Dynamic Integrated Approach to teacher professional development: Impact and sustainability of the effects on improving teacher behaviour and student outcomes. Teaching and Teacher Education, 29, 1–12.

Armour, K.M. & Makopoulou, K. (2012). Great expectations: Teacher learning in a national professional development programme. Teaching and Teacher Education, 28(3), 336–346.

Axtell, C.M., Maitlis, S., & Yearta, S.K. (1997). Predicting immediate and long-term transfer of training. Personnel Review, 26(3), 201–213.

Bamber, V. (2013). Evidencing the value of Educational Development, SEDA Special 34.

London: SEDA.

Belvis, E., Pineda, P., Armengol, C. & Moreno, V. (2013). Evaluation of reflective practice in teacher education. European Journal of Teacher Education, 36(3), 279–292.

Boman, J. S. (2013). Graduate Student Teaching Development: Evaluating the Effectiveness of Training in Relation to Graduate Student Characteristics. Canadian Journal of Higher Education, Volume 43 (1), 100-114

Boud, D., & Brew, A. (2013). Reconceptualising academic work as professional practice: implications for academic development. International Journal for Academic Development, 18(3), 208–221.

Bozalek, V., McMillan, W., Marshall, D., November, M., Daniels, A. & Sylvester, T. (2014). Analysing the professional development of teaching and learning from a political ethics of care perspective, Teaching in Higher Education, 19:5, 447-458.

Brown, A., & Inglis, S. (2013). So what happens after the event? Exploring the realisation of professional development with early childhood educators. Australasian Journal of Early Childhood, 38(1), 11-15.

Cashmore, A., Cane, C., & Cane, R. (2013). Rebalancing promotion in the HE sector: is teaching excellence being rewarded?. Genetics Education Networking for Innovation and Excellence: the UK’s Centre for Excellence in Teaching and Learning in Genetics (GENIE CETL), University of Leicester, The Higher Education Academy.

Chalmers, D. & Gardiner, D. (2015). The measurement and impact of university teacher development programs. Educar. 51 (1). 1-28.

Chalmers, D., Stoney, S., Goody, A., Goerke, V., & Gardiner, D. (2012). Identification and implementation of indicators and measures of effectiveness of teaching preparation programs for academics in higher education. Final report.

Chalmers, D. (2011). Progress and challenges to the recognition and reward of the scholarship of teaching in higher education. Higher Education Research and Development, 30, 25-38.

115/222

HEA CPD Pilot Workshop - Workbook

Chalmers, D. (2008) Indicators of university teaching and learning quality. Sydney, NSW: Australian Learning and Teaching Council. Available from: http://www.catl.uwa.edu.au/projects/tqi

Clarke, D., & H. Hollingsworth (2002). Elaborating a Model of Teacher Professional Growth. Teaching and Teacher Education 18 (8): 947–967.

Cotton, D.R.E. & Gresty, K.A. (2006) Reflecting on the think-aloud methodology for evaluating e-learning. British Journal of Educational Technology, 37, 45-54.

De Rijdt, D., Stes, A., van der Vleuten, C, & Dochy, F. (2013) Influencing variables and moderators of transfer of learning to the workplace within the area of staff development in higher education. Educational Research Review. 8. 48-74.

De Vries, S., Jansen, E.P.W.A. & van de Grift, W.J.C.M. (2013). Profiling teachers’ continuing professional development and the relation with their beliefs about learning and teaching. Teaching and Teacher Education, 33, pp.78–89.

Department for Education and Skills. (2003). The Future of Higher Education.

http://www.bis.gov.uk/assets/biscore/corporate/migratedd/publications/f/future_

of_he.pdf

Department for Business, Innovation and Skills (2011) Higher Education. Students at the Heart of the System. http://www.bis.gov.uk/assets/biscore/higher-education/docs/h/11-944-higher-educationstudents-at-heart-of-system.pdf

Di Napoli, R. (2014). Value gaming and political ontology: between resistance and compliance in academic development. International Journal for Academic Development, 19(1), pp.4–11.

Dyment, J. and O'Connell, T. (2014). When the Ink Runs Dry: Implications for Theory and Practice When Educators Stop Keeping Reflective Journal. Innovative Higher Education; 39 (5), 417-429.

Ellis, R., Weyers, M. & Hughes, J. (2013). Campus-based student experiences of learning technologies in a first-year science course. British Journal of Educational Technology, 44(5), 745–757.

Farley, H. and Murphy, A. (2013). Developing a framework for evaluating the impact and sustainability of mobile learning initiatives in higher education. Open and Distance Learning Association of Australia (ODLAA) Distance Education Summit (Sydney).

Field, A. (2009). Discovering statistics using SPSS. Sage publications.

Fink, L.D. (2013). Innovative Ways of Assessing Faculty Development. New Directions for Teaching and Learning, 2013(133), 47–59.

116/222

HEA CPD Pilot Workshop - Workbook

Gibbs, G., Habeshaw, T. and Yorke, M. (2000) Institutional Learning and Teaching Strategies in English Higher Education. Higher Education. 40 (3), 351-372.

Gosling, D. (2008). Educational development in the United Kingdom. Report for the heads of educational development group. London, Heads of Educational Development Group (HEDG) UK. http://www.hedg.ac.uk/documents/HEDG_Report_final.pdf

Green, D. (2009). New academics' perceptions of the language of teaching and learning: identifying and overcoming linguistic barriers. IJAD, 14 (1) 33-45.

Green, D., & Huston, T. (2006). Lost in translation? Developing an inclusive approach to the language of academic development. Session presented at the 6th Conference of the International Consortium for Educational Development, Sheffield, UK.

Gunn, V & Fisk, A. (2013) Considering teaching excellence in higher education: 2007 – 3013” A literature review since the CHERI report. HEA.

Guskey, T. R. (2002) Professional development and teacher change. Teachers and Teaching: Theory and Practice, 8 (3-4), 381–391.

Han J.H. & Finkelstein, A. (2013). Understanding the effects of professors’ pedagogical development with Clicker Assessment and Feedback technologies and the impact on students' engagement and learning in higher education. Computers & Education, 65, 64–76.

Han, S. (2012). Professional Development That Works: Shifting Preschool Teachers' Beliefs and Use of Instructional Strategies to Promote Children's Peer Social Competence, Journal of Early Childhood Teacher Education 33:3, 251-268

Higher Education Academy (2012). The Higher Education Academy Strategic Plan 2012-2016. York: Higher Education Academy. Retrieved 20th October 2014 from: https://www.heacademy.ac.uk/sites/default/files/downloads/strategic-plan-2012-16.pdf

HESA (2012). Changes made to the Staff record for 2012/13. Higher Education Statistics Agency. Retrieved from http://www.hesa.ac.uk/component/option,com_studrec/task,show_file/Itemid,233/mnl,12025/href,Changes.html/

Herman, J. (2013). Staffing of Teaching and Learning Centers in the United States: Indicators of Institutional Support for Faculty Development. Journal of Faculty Development 27 (2), 33-37.

Hine, G. (2013). The importance of action research in teacher education programs. Issues in Educational Research; Special Issue, 23(2), 151-163.

Hines, S. R.( 2009). Investigating Faculty Development Program Assessment Practices: What’s Being Done and How Can It Be Improved?. Journal of Faculty Development 23 (3): 5–19. Cited in Fink,

117/222

HEA CPD Pilot Workshop - Workbook

L.D. (2013). Innovative Ways of Assessing Faculty Development. New Directions for Teaching and Learning, 2013(133), 47–59.

Jordan, L. (2014). ‘Success’ and teaching development programmes: a reflexive study of personal goals and factors influencing achievement. Research proposal as part of EdD submission.

Kandlbinder, P., & Peseta, T. (2009). Key concepts in postgraduate certificates in higher education teaching and learning in Australasia and the United Kingdom. International Journal for Academic Development, 14(1), 19-31.

Kirkpatrick, D. (1998). Evaluating training programs (2nd ed.). San Francisco, CA: Berrett- Koehler.

Kreber, C. & Brook, P. (2001). Impact Evaluation of Educational Development

Programmes. International Journal for Academic Development, 6, 96-107.

Lau, W. W. F., & Yuen, A. H. K. (2013). Educational technology training workshops for mathematics teachers : An exploration of perception changes. Australasian Journal of Educational Technology, 29(4), 595-611.

Lee, B., Cawthon, S. & Dawson, K. (2013). Elementary and secondary teacher self-efficacy for teaching and pedagogical conceptual change in a drama-based professional development program. Teaching and Teacher Education, 30, 84–98.

Ling, P. (2009). Development of academics and higher education futures. Report Vol. 1.

Sydney, ALTC.

Lumpe, A., Czerniak, C., Haney, J., & Beltyukova, S. (2012). Beliefs about Teaching Science: The relationship between elementary teachers’ participation in professional development and student achievement, International Journal of Science Education, 34:2, 153-166.

McGee, J.R., Wang, C. & Polly, D., (2013). Guiding Teachers in the Use of a Standards-Based Mathematics Curriculum: Teacher Perceptions and Subsequent Instructional Practices After an Intensive Professional Development Program. School Science and Mathematics, 113(1), pp.16–28.

McKenna, C. and Hughes, J. (2015). The locations and identity construction of academic developers: Findings from the ExILED (Exploring the Identities and Locations of Educational Developers) project. SEDA Report.

Mockler, N., (2013). Teacher Professional Learning in a Neoliberal Age: Audit, Professionalism and Identity. Australian Journal of Teacher Education, 38(10).

Moolenaar, N. M., Sleegers, P. J. C., & Daly, A. J. (2012). Teaming up: linking collaboration networks, collective efficacy, and student achievement. Teaching and Teacher Education, 28(2), 251e262

Mowbray, R. & Perry, L. (2015). Improving lecture quality through training in public speaking, Innovations in Education and Teaching International, 52:2, 207-217

Parsons, D., Hill, I., Holland, J., & Willis, D. (2012). Impact of teaching development programmes in higher education. York: Higher Education Academy.

118/222

HEA CPD Pilot Workshop - Workbook

Peeraer, J., V.P. (2012). The limits of programmed professional development on integration of information and communication technology in education. Australasian Journal of Educational Technology,1039–1056.

Pehmer, A.K., Gröschner, A. & Seidel, T. (2015). How teacher professional development regarding classroom dialogue affects students’ higher-order learning. Teaching and Teacher Education, 47, pp.108–119. Available at: http://www.sciencedirect.com/science/article/pii/S0742051X14001644 [Accessed January 19, 2015].

Peters, S. , Barbier, M., Faulx, D., & Hansez, I. (2012). Learning and motivation to transfer after an e-learning programme: impact of trainees’ motivation to train, personal interaction and satisfaction. Innovations in Education and Teaching International, 49(4), pp.375–387.

Pineda, P. (2002). Gestión de la formación en las organizaciones [Training management in organizations]. Barcelona: Ariel.

Postareff, L. & Nevgi, A. (2015). Development paths of university teachers during a pedagogical development course. Educar, 51(1).

Rust, C. (1998). The impact of educational development workshops on teachers’

practice. The International Journal for Academic Development. 3, 72-80.

Rienties, B. and Kinchin, I. (2014). Understanding (in)formal learning in an academic development programme: A social network perspective. Teaching and Teacher Education. 39. 123-135.

Rienties, B. Brower, N. & Lygo-Baker, S. (2013). ‘The effects of online professional development on higher education teachers’ beliefs and intentions towards learning facilitation and technology. Teaching and Teacher Education, 29, 122-131.

Roxa, T., Martensson, K., & Alveteg, M. (2011). Understanding and influencing teaching and learning cultures at university: a network approach. Higher Education . 62, 99-111.

Rutz, C. Condon, W., Iverson, E, R. Manduca, C. & Willett, G. (2012). Faculty Professional Development and Student Learning: What is the Relationship? Change. 44 (3), 40-47.

Sachs, J. (2003). The activist teaching profession. Buckingham: Open University Press.

Saunders, R. (2014). Effectiveness of Research-Based Teacher Professional Development: Australian Journal of Teacher Education, 39(4).

Shavelson, R. (2010). Measuring college learning responsibly: Accountability in a New Era. California, Stanford University Press.

Shim, W. & Walczak, K. (2012) The Impact of Faculty Teaching Practices on the Development of Students’ Critical Thinking Skills. International Journal of Teaching and Learning in Higher Education. 24:1, 16-30.

Skelton, A. (2013). Positively transformational or poisoned chalice? The impact of a course on higher education teaching at a research-intensive institution. Teaching in Higher Education, 18(8), pp.908–919.

119/222

HEA CPD Pilot Workshop - Workbook

Smith, J. (2010). Forging identities: the experiences of probationary lecturers in the UK. Studies in Higher Education, 35, 577-591.

Spowart, L., Turner, R., Kneale, P. & Shenton, D. (in press). ‘But I have been teaching for 20 years...’: encouraging teaching accreditation for experience staff working in higher education. International Journal for Academic Development.

Stefani, L. (2013). Performance measurement for academic development: risk or opportunity? International Journal for Academic Development, 18(3), pp.294–296.

Stes, A., Coertjens, L. & Petegem, P. (2013a) Instructional development in higher education: impact on teachers' teaching behaviour as perceived by students. Instructional Science; Vol. 41 Issue 6, 1103-1126.

Stes, A., Min-Leliveld, M., Gijbels, D., & Van Petegem, P. (2010). The impact of instructional development in higher education: The state-of-the-art of the research. Educational Research Review, 5, 25-49.

Stes, A. et al. (2012). Instructional development for teachers in higher education: effects on students’ learning outcomes. Teaching in Higher Education, 17(3), 295–308

Stes, A., De Maeyer, C., Gijbels, D. & Van Petegem, P. (2013b) Effects of teachers' instructional development on students' study approaches in higher education, Studies in Higher Education, 38:1, 2-19

Stewart, M. (2014). Making sense of a teaching programme for university academics: Exploring the longer-term effects. Teaching and Teacher Education, 38, 89–98.

Stronach, I. (2010). Globalizing education, educating the local: How method made us mad. London: Routledge.

Sword, H. (2011). Archiving for the future: A longitudinal approach to evaluating a postgraduate certificate

program. In L. Stefani (ed.), Evaluating the effectiveness of academic development: principles and practice.

New York, Routledge.

Thomas, J. & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC medical research methodology, 8, p.45.

Trigwell, K., Caballero Rodriguez, K., & Han, F. (2012). Assessing the impact of a university teaching development programme. Assessment & Evaluation in Higher Education, 37(4), 499–511.

Tronto, J. (1993). Moral Boundaries: A Political Argument for an Ethic of Care. New York: Routledge. Cited in Bozalek, V., McMillan, W., Marshall, D., November, M., Daniels, A. & Sylvester,T. (2014). Analysing the professional development of teaching and learning from a political ethics of care perspective, Teaching in Higher Education, 19:5, 447-458.

Turner, N., Oliver, M., McKenna, C., Hughes, J., Smith, H., Deepwell, F., & Shrives, L. (2013). Measuring the impact of the UK Professional Standards Framework for teaching and supporting learning. HEA report.

Turner, R. & Gosling, D. (2012). Rewarding excellent teaching: the translation of a policy initiative in the United Kingdom. Higher Education Quarterly, 66, 415-430.

Van den Bos, P. & Brouwer, J. (2014). Learning to teach in higher education: how to link theory and practice, Teaching in Higher Education, 19:7, 772-786

120/222

HEA CPD Pilot Workshop - Workbook

Watson, S. (2014). The impact of professional development on the teaching of problem solving in mathematics: A Social Learning Theory perspective. In Pope, S. (Ed.) Proceedings of the 8th British Congress of Mathematics Education 2014. 351–358.

Willett, G., Iverson, E., Rutz, C., & Manduca, C. (2014). Measures matter: Evidence of faculty development effects on faculty and student learning. Assessing Writing Vo. 20, 19-36.

121/222

HEA CPD Pilot Workshop - Workbook

Appendix 1: Audit template used in WP2

1. Continuing Professional Development in Higher Education

Thank you for taking the time to participate in this audit we are conducting as part of theHEA project “Evaluating teaching development in HE: towards an impact assessment toolkit”, undertaken by P e d R IO at Plymouth University in collaboration withH E D E R A , Durham University and University Campus Suffolk.

P u rpos e o f t h is aud i t

Your inputs are vital to helping us develop an accurate understanding of the range of teaching-related CPD offered to support the delivery of higher education teaching, and current practices in measuring their impact. Specifically, your responses will be used to inform the development of a tool to evaluate teaching-related CPD activities which in turn, will help influence teaching quality across the sector. In addition, we hope you will find completing this audit a useful tool for reflecting on the CPD provision in your institution.

W h o t h is aud i t is a i me d a t

This audit is targeted at education/academic developers, staff dedicated to CPD provision, individuals who may have responsibility for coordinating teaching-related CPD as part of their role and senior managers.

S o m e us e f u l gu i de li ne s

It should take approximately 15 minutes to complete, depending on the level of detail you are able to

122/222

HEA CPD Pilot Workshop - Workbook

provide. We have attempted to design it to be as convenient as possible, and many questions require only closed-answer responses.

Please use the 'prev' and 'next' buttons at the end of each page to navigate through the audit; using the 'back' button on your browser might interfere with your responses being saved.

The audit will be available until 17th February 2015.

C o n f i den t i a l i t y

The survey asks for only generic details about your institution and you have the option of completing it anonymously. Taking part is entirely voluntary and by completing the survey you will be considered to have given consent for its inclusion in this project. The results of this study will be used for scholarly purposes and in the development of a CPD evaluation tool which will be owned by the HEA.

This study is consistent with the British Educational Research Association guidelines on research ethics and has received ethical approval from the Education Research Ethics Committee, Plymouth University.

We do hope you are able to complete this audit and thank you in advance for your time.

If you have any questions about the audit please email evaluatingcpd@plymouth.ac.uk

123/222

HEA CPD Pilot Workshop - Workbook

2. Details about your institution

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

1. Which group does your institution belong to? Please tick all that apply.

Old university

Post-1964

Post-1992

New 2010

FE Colleges

Private provider

Russell group

Million+ group

University Alliance

Unaligned

Other (please specify)

124/222

HEA CPD Pilot Workshop - Workbook

2. Approximately what percentage of staff teaching higher education in your institution have a recognised HE-specific teaching award or qualification?

Not sure

Percentage of recognised teaching staff in HE

125/222

HEA CPD Pilot Workshop - Workbook

A c co r din g t o t h e H i g he r Educ a t i on S t a t i s t i cs A g e nc y ( H ES A ) , t h e f o ll owi ng a re cons i de r e d t o be H E - spec i f ic awa r d s or q ua l i f i c a t i on s

• Completed an institutional provided course in teaching in HE accredited against the UK Professional Standards Framework

• Recognised by the Higher Education Academy (HEA) as an Associate Fellow • Recognised by the HEA as a Fellow

• Recognised by the HEA as a Senior Fellow

• Recognised by the HEA as a Principal Fellow

• Holder of a National Teaching Fellowship Scheme Individual Award • Holder of a PGCert in higher education

• Other UK accreditation or qualification in teaching in the higher education sector • Overseas accreditation or qualification for any level of teaching

• Staff and Educational Development Association (SEDA) professional development framework awards

126/222

HEA CPD Pilot Workshop - Workbook

127/222

HEA CPD Pilot Workshop - Workbook

3. Teaching-related CPD provision at your institution

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

Teaching-related CPD is defined here as any activity ‘targeted to strengthen and extend the knowledge, skills and conceptions of teachers in a way that will lead to changes in their way of thinking and their educational behaviour’ (Fenstermacher & Berliner, 1985: 49).

This includes accredited teacher-training programmes and courses, and non-accredited activities such as workshops, mentoring and informal discussions related to teaching and learning. In this audit, teaching-related CPD includes higher education teachers but also those in support roles (e.g learning technologists, librarians, e-learning advisors etc.).

I. E x ampl es o f r ecogn i s e d/ a cc r ed i t e d t eac h in g - re l a t ed C P D • Postgraduate teaching qualification in higher education

• Masters in Education

• Recognition with Higher Education Academy • SEDA Fellowship

II. Ex a m p l e s o f non - a c cr e d i t e d t each i ng - r e l a t e d C P D • Workshops

• Short courses

• Institutional agendas/projects around teaching and learning agendas e.g. inclusivity/exam feedback/internationalisation

• Internal teaching and learning conferences • Peer-review and consultation

128/222

HEA CPD Pilot Workshop - Workbook

• Mentoring and other activities through participation in communities of practice • Discussion groups/panel sessions

• Action research

• Pedagogic research

• Scholarship of teaching and learning activities • Individual research

• Training / away days

We are interested in examples of recognised/accredited and non-accredited teaching-related CPD for HE. Please select one teaching-related CPD offer (either recognised/accredited or non-accredited) provided in your institution.

129/222

HEA CPD Pilot Workshop - Workbook

* 3. Which teaching-related CPD do you wish to tell us about?

Recognised/accredited

Non-accredited

My institution does not offer teaching-related CPD

130/222

HEA CPD Pilot Workshop - Workbook

131/222

HEA CPD Pilot Workshop - Workbook

4. Recognised/accredited CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

4. How would you describe this CPD offer? Please tick all that apply.

General Teaching Associates or equivalent

Postgraduate Certificate in Higher Education

Masters in Education

Working towards recognition with Higher Education Academy

Working towards recognition with SEDA

Other (please specify)

5. Which groups does this CPD offer target? Please tick all that apply.

Unqualified/unrecognised teaching staff

Incoming teaching staff

Teaching staff with 3 years and more experience

Staff who support teaching (learning technologists, librarians etc.)

132/222

HEA CPD Pilot Workshop - Workbook

Research/postgraduate students

Other

6. Is this offer?

Cross institutional

Tailored for a specific unit/department

Other (please specify)

7. Is participation in this CPD offer? (Please tick all that apply)

Mandatory (linked to probation)

Mandatory (not linked to probation)

Voluntary

133/222

HEA CPD Pilot Workshop - Workbook

8. Approximately how many participants are engaged with this CPD offer per annum?

9. How are the values and ethos for this CPD offer identified?

10. Do any of the following inform the content for this CPD offer? Please tick all that apply.

Institutional priorities

Discipline-specific priorities

Educational/academic development unit priorities

Evidence-based best practice

External professional frameworks (SEDA, UKPSF, Vitae RDF, discipline-specific frameworks)

Teaching and learning committees or equivalent

Course leader priorities

External feedback for example, NSS, KIS, PTES and other student surveys

Student feedback for example, module evaluations, student complaints, student representative systems, student union

External examiners

134/222

HEA CPD Pilot Workshop - Workbook

Other (please specify)

SE D A - Staff and Educational Development Association UK PS F - UK Professional Standards Framework

V i t ae RD F - Vitae Researcher Development Framework N S S – National Student Survey

K IS – Key Information Sets

P T E S – Postgraduate Taught Experience Survey

11. What are the key aims of this CPD offer? Please tick all that apply.

Teacher-focused - Develop pedagogical knowledge, teaching approaches, skills, strategies; Encourage reflective practice;

Change teaching behaviour

Student-focused - Improve students' approach to learning; Engage students' in learning; Enhance student experience

Institutional-focused – Orient participants to understand and implement culture and policies regarding teaching and learning;

Leadership development

135/222

HEA CPD Pilot Workshop - Workbook

12. Over what duration does this CPD offer usually run?

1 day or less

Less than a month

Between 1 - 6 months

Between 7 - 12 months

More than 12 months

13. How is this CPD offer primarily delivered?

Face to face

Flexible/on-line

Both (blended learning)

14. Who is involved in the delivery and support of this CPD offer? Please tick all that apply

Staff dedicated to providing teaching-related CPD

Academic staff within faculties

Professional service staff (student support, librarians, learning technologists)

Previous participants

External guest speakers

Web-based resources/online delivery

Students

Other (please specify)

136/222

HEA CPD Pilot Workshop - Workbook

* 15. Do you evaluate this CPD offer?

Yes

No

137/222

HEA CPD Pilot Workshop - Workbook

5. Non-accredited CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

16. How would you describe this CPD offer?

Workshop(s)

Short course(s)

Institutional projects around teaching and learning agendas e.g. inclusivity/exam feedback/internationalisation

Internal teaching and learning conferences

Peer-review and/or consultation

Panel discussions/group discussions

Mentoring and other activities through participation in communities of practice

Action research

Pedagogic research

Scholarship of teaching and learning activities

Training / away days

17. Which groups does this CPD offer target? Please tick all that apply.

138/222

HEA CPD Pilot Workshop - Workbook

Unqualified/unrecognised teaching staff

Incoming teaching staff

Teaching staff with 3 years+ experience

Staff who support teaching (learning technologists, librarians etc.)

Research/postgraduate students

18. Is participation in this CPD offer? (Please tick all that apply).

Mandatory (linked to probation)

Mandatory (not linked to probation)

Voluntary

139/222

HEA CPD Pilot Workshop - Workbook

19. Is this offer?

Cross institutional

Tailored for a specific unit/department

Other (please specify)

20. Approximately how many participants are engaged with this CPD offer per annum?

21. How are the values and ethos for this CPD offer identified?

22. How is content for this CPD offer identified? Please tick all that apply.

Institutional priorities

Discipline-specific priorities

Educational/academic development unit priorities

Evidence-based best practice

140/222

HEA CPD Pilot Workshop - Workbook

External professional frameworks (SEDA, UKPSF, Vitae RDF, discipline-specific frameworks)

Teaching and learning committees or equivalent

Course leader priorities

External feedback for example, NSS, KIS, PTES and other student surveys

Student feedback for example, module evaluations, student complaints, student representative systems, student union

External examiners

Other (please specify)

SE D A - Staff and Educational Development Association UK PS F - UK Professional Standards Framework

V i t ae RD F - Vitae Researcher Development Framework N S S – National Student Survey

K IS – Key Information Sets

P T E S – Postgraduate Taught Experience Survey

141/222

HEA CPD Pilot Workshop - Workbook

23. What are the key aims of this CPD offer? Please tick all that apply.

Teacher-focused - Develop pedagogical knowledge, teaching approaches, skills, strategies; Encourage reflective practice;

Change teaching behaviour

Student-focused - Improve students' approach to learning; Engage students' in learning; Enhance student experience

Institutional-focused – Orient participants to understand and implement culture and policies regarding teaching and learning;

Leadership development

24. Over what duration does this CPD offer usually run?

1 day or less

Less than a month

Between 1 - 6 months

Between 7 - 12 months

More than 12 months

25. How is this CPD offer primarily delivered?

Face to face

Flexible/on-line

Both (blended learning)

26. Who is involved in the delivery and support of this CPD offer? Please tick all that apply

Staff dedicated to providing teaching-related CPD

Academic staff within faculties

142/222

HEA CPD Pilot Workshop - Workbook

Professional service staff (student support, librarians, learning technologists)

Previous participants

External guest speakers

Web-based resources/online delivery

Students

Other (please specify)

* 27. Do you evaluate this CPD offer?

Yes

No

143/222

HEA CPD Pilot Workshop - Workbook

6. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

28. When is this CPD offer first evaluated? (You will have the chance to tell us about any subsequent evaluations later in this audit).

Before event

During event

At end of event

Within 6 months

After 6 months

144/222

HEA CPD Pilot Workshop - Workbook

145/222

HEA CPD Pilot Workshop - Workbook

7. Initial evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

29. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.

Questionnaire completed by previous participants

Interview/focus group with previous participants

Questionnaire completed by current participants

Interview/focus group with current participants

Postgraduate taught experience survey/National Student Survey

Questionnaire completed by students

Interview/focus group with students

Review of participant journals, teaching materials developed or other questions/statements,

Observations of teaching practice

Observations of changes in student behaviour/approach to learning

Line-manager/ supervisor's report

Student assessment/grades

Review of DLHE (Destination of Leavers in Higher Education)

146/222

HEA CPD Pilot Workshop - Workbook

Review of retention and progression stats

Curriculum development/enhancement

Other (please specify)

P r ev i ou s pa r t i c i pan t s - Persons who have previously undertaken the same CPD offer P a r t i c i pan t s - Persons undertaking this CPD offer

S t uden t s - Undergraduate and postgraduate students in higher education at your institution

147/222

HEA CPD Pilot Workshop - Workbook

30. What is the focus of evaluation for this CPD offer at this time? Please tick all that apply.

Participants' satisfaction with the CPD activity

Changes to participants' beliefs about teaching and learning

Changes to participants' teaching practice

Changes in students' perceptions of staff's teaching performance

Impact on student learning

Impact participation has on culture of departments and the institution

31. Any other comments about the evaluation you think would be useful to us?

32. How were the results of this evaluation used?

148/222

HEA CPD Pilot Workshop - Workbook

149/222

HEA CPD Pilot Workshop - Workbook

8. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

You have told us about evaluation of this CPD offer"[Q28]".

33. Do you evaluate this CPD offer again, at another time?

No

Yes, during event

Yes, at end of event

Yes, within 6 months

Yes, after 6 months

150/222

HEA CPD Pilot Workshop - Workbook

151/222

HEA CPD Pilot Workshop - Workbook

9. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

34. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.

Questionnaire completed by participants

Interview/focus group with participants

Postgraduate taught experience survey/National Student Survey

Questionnaire completed by students

Interview/focus group with students

Review of participant journals, teaching materials developed or other questions/statements,

Observations of teaching practice

Observations of changes in student behaviour/approach to learning

Line-manager/ supervisor's report

Student assessment/grades

Review of DLHE (Destination of Leavers in Higher Education)

Review of retention and progression stats

Other (please specify)

152/222

HEA CPD Pilot Workshop - Workbook

P a r t i c i pan t s - Persons undertaking this CPD offer

S t uden t s - Undergraduate and postgraduate students in higher education at your institution

35. What was the focus of evaluation for this CPD offer at this time? Please tick all that apply.

Participants' satisfaction with the CPD activity

Changes to participants' beliefs about teaching and learning

Changes to participants' teaching practice

Changes in students' perceptions of staff's teaching performance

Impact on student learning

Impact participation has on culture of departments and the institution

153/222

HEA CPD Pilot Workshop - Workbook

36. Any other comments about the evaluation you think would be useful to us?

37. How were the results of this evaluation used?

154/222

HEA CPD Pilot Workshop - Workbook

155/222

HEA CPD Pilot Workshop - Workbook

10. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

Previously you answered "[Q33]" to whether this offer was evaluated a second time.

38. Do you evaluate this CPD offer again, at a later time?

No

Yes, during event

Yes, at end of event

Yes, within 6 months

Yes, after 6 months

156/222

HEA CPD Pilot Workshop - Workbook

157/222

HEA CPD Pilot Workshop - Workbook

11. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

39. What methods are used for evaluation of the CPD offer at this time? Please tick all that apply.

Questionnaire completed by participants

Interview/focus group with participants

Postgraduate taught experience survey/National Student Survey

Questionnaire completed by students

Interview/focus group with students

Review of participant journals, teaching materials developed or other questions/statements,

Observations of teaching practice

Observations of changes in student behaviour/approach to learning

Line-manager/ supervisor's report

Student assessment/grades

Review of DLHE (Destination of Leavers in Higher Education)

Review of retention and progression stats

Other (please specify)

158/222

HEA CPD Pilot Workshop - Workbook

P a r t i c i pan t s - Persons undertaking this CPD offer

S t uden t s - Undergraduate and postgraduate students in higher education at your institution

40. What was the focus of evaluation for this CPD activity? Please tick all that apply.

Participants' satisfaction with the CPD activity

Changes to participants' beliefs about teaching and learning

Changes to participants' teaching practice

Changes in students' perceptions of staff's teaching performance

Impact on student learning

Impact participation has on culture of departments and the institution

159/222

HEA CPD Pilot Workshop - Workbook

41. Any other comments about the evalaution

42. How were the results of this evaluation used?

160/222

HEA CPD Pilot Workshop - Workbook

161/222

HEA CPD Pilot Workshop - Workbook

12. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

Previously, you answered "[Q38]" to whether this CPD offer was evaluated a third time.

43. Do you evaluate this CPD offer again, at a later time?

No

Yes, during event

Yes, at end of event

Yes, within 6 months

Yes, after 6 months

162/222

HEA CPD Pilot Workshop - Workbook

163/222

HEA CPD Pilot Workshop - Workbook

13. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

44. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.

Questionnaire completed by participants

Interview/focus group with participants

Postgraduate taught experience survey/National Student Survey

Questionnaire completed by students

Interview/focus group with students

Review of participant journals, teaching materials developed or other questions/statements,

Observations of teaching practice

Observations of changes in student behaviour/approach to learning

Line-manager/ supervisor's report

Student assessment/grades

Review of DLHE (Destination of Leavers in Higher Education)

Review of retention and progression stats

Other (please specify)

164/222

HEA CPD Pilot Workshop - Workbook

P a r t i c i pan t s - Persons undertaking this CPD offer

S t uden t s - Undergraduate and postgraduate students in higher education at your institution

45. What was the focus of evaluation for this CPD offer at this time? Please tick all that apply.

Participants' satisfaction with the CPD activity

Changes to participants' beliefs about teaching and learning

Changes to participants' teaching practice

Changes in students' perceptions of staff's teaching performance

Impact on student learning

Impact participation has on culture of departments and the institution

165/222

HEA CPD Pilot Workshop - Workbook

46. Any other comments about the evaluation you think would be useful to us?

47. How were the results of this evaluation used?

166/222

HEA CPD Pilot Workshop - Workbook

167/222

HEA CPD Pilot Workshop - Workbook

14. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

Previously, you answered "[Q43]" to whether this CPD offer was evaluated a fourth time.

48. Do you evaluate this CPD offer again, at a later time?

No

Yes, during event

Yes, at end of event

Yes, within 6 months

Yes, after 6 months

168/222

HEA CPD Pilot Workshop - Workbook

24

169/222

HEA CPD Pilot Workshop - Workbook

15. Evaluation of CPD offer

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

49. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.

Questionnaire completed by participants

Interview/focus group with participants

Postgraduate taught experience survey/National Student Survey

Questionnaire completed by students

Interview/focus group with students

Review of participant journals, teaching materials developed or other questions/statements,

Observations of teaching practice

Observations of changes in student behaviour/approach to learning

Line-manager/ supervisor's report

Student assessment/grades

Review of DLHE (Destination of Leavers in Higher Education)

Review of retention and progression stats

Other (please specify)

170/222

HEA CPD Pilot Workshop - Workbook

P a r t i c i pan t s - Persons undertaking this CPD offer

S t uden t s - Undergraduate and postgraduate students in higher education at your institution

50. What was the focus of evaluation for this CPD offer at this time? Please tick all that apply.

Participants' satisfaction with the CPD activity

Changes to participants' beliefs about teaching and learning

Changes to participants' teaching practice

Changes in students' perceptions of staff's teaching performance

Impact on student learning

Impact participation has on culture of departments and the institution

171/222

HEA CPD Pilot Workshop - Workbook

51. Any other comments about the evaluation you think would be useful to us?

52. How were the results of this evaluation used?

172/222

HEA CPD Pilot Workshop - Workbook

173/222

HEA CPD Pilot Workshop - Workbook

16. CPD Evaluation and Impact

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

We are also interested in your broader institutional teaching-related CPD offer and any evaluation of it.

53. Please identify which of the following your institution offers to support teaching-related CPD. Please tick all that apply.

Accredited academic programmes for teaching, e.g. PG Certificates, Masters etc.

Workshops

Institutional teaching and learning conferences

External teaching and learning conferences

Peer-review/observation

Teaching consultation

Mentoring

Identified communities of practice around teaching and learning in higher education

Action research

Funding for postgraduate study

Other (please specify)

174/222

HEA CPD Pilot Workshop - Workbook

175/222

HEA CPD Pilot Workshop - Workbook

54. How are the impacts of these CPD offers disseminated throughout your institution? Please tick all that apply.

Debrief from conference attendances

Disseminating pedagogic research

Changes in policy

Cascading from teaching and learning committees or equivalents

Mentoring

Staff appraisal

Sharing best practices through presentations, training and discussions.

There is no means of institution wide dissemination

Other (please specify)

55. Do you evaluate any of these longitudinally?

Not sure

No

Yes (please provide details below on the type of activity and the stages of evaluation)

56. Are changes in the 'scholarship of teaching' of staff who undertake CPD tracked to 176/222

HEA CPD Pilot Workshop - Workbook

assess impact?

No

Yes (please provide details below)

We define sc ho l ar s hi p o f t ea c hi ng as:

engagement with research on teaching and learning;

reflecting on personal teaching and student learning experiences within a particular discipline; and sharing knowledge and practice about teaching and learning, both within the discipline and more generally (Prosser & Trigwell, 1999)

.

177/222

HEA CPD Pilot Workshop - Workbook

57. Are students in your institution aware of staff participation in teaching-related CPD?

Yes Somewhat No Not sure

58. If you answered 'yes' or 'somewhat' to the question above, how are they made aware? Please tick all that apply.

Through the student union

Through the student representative system

Through the schools and faculties

Central communication

Other (please specify)

59. In your opinion, is teaching-related CPD an important student agenda?

Yes Somewhat No Not sure

60. Are there formal opportunities for students to contribute to the evaluation of teaching-related CPD?

Yes Somewhat No Not sure

If yes and you are happy to share more details, please tell us more about this process below.

178/222

HEA CPD Pilot Workshop - Workbook

61. How are the results of teaching-related CPD evaluation used by the institution? Please tick all that apply.

In marketing materials

As evidence for the Quality Assurance Agency and other professional audits

In annual reports to the governing body

To change policy around teaching and learning

Probation and promotion criteria

To inform future CPD provision

Not at all to my knowledge

Other (please specify)

179/222

HEA CPD Pilot Workshop - Workbook

17. Institutional policies and culture

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

62. What is your institution's minimum teaching-related CPD requirement for teaching at the following levels?

Accredited/recognised CPD

Postgraduate student

Non-accredited

CPD Case by case None Not sure

Associate Lecturer (part-time lecturer)

Incoming staff with teaching experience

Incoming staff with no teaching experience

Existing teaching staff

Staff with leadership roles in teaching

Staff in learning support roles

63. Is there an expectation/policy about peer-observation or peer-review of teaching in your institution?

180/222

HEA CPD Pilot Workshop - Workbook

Yes

No

Not sure

64. If 'yes' to the above, is peer-review of teaching activity linked to annual appraisal and/or performance development review?

Yes, in all cases

Yes, in some cases

No

Not sure

181/222

HEA CPD Pilot Workshop - Workbook

18. Reward and Recognition

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

65. How does your institution recognise and reward teaching? Please tick all that apply.

Internal teaching fellowship award schemes

External teaching fellowship award schemes

Pedagogic research grants

Institutionally bestowed teaching awards

Student bestowed teaching awards

Promotion

Financial incentives

Organisation of Teaching and Learning conferences

Community of practice around research in higher education

Time allocation for teaching-related CPD

Honorary titles

Other (please specify)

182/222

HEA CPD Pilot Workshop - Workbook

66. Are career pathways in your institution

Single-track (for both teaching and research roles, and roles which combine these activities)

Two-track (separate for teaching and research roles)

Three-track (teaching, research, teaching and research)

Not sure

Other (please specify)

A t hr ee – t ra ck s ys t em f o r pr o m o t i o n indicates one track for teaching, one track for research, and one track encompassing teaching and research. These pathways remain independent until professorial level.

183/222

HEA CPD Pilot Workshop - Workbook

67. In your promotion policy, please indicate which of the following positions include explicit criteria relating to teaching and learning?

Yes No Not sure

Professor

Reader/Principal Lecturer or equivalent

Senior Lecturer/Reader or equivalent

Lecturer

68. Did your institution make a return to the 2014 Research Excellence Framework in educational research?

Yes

No

Not sure

69. Does your institution have any Professors of Teaching?

Yes

No

Not sure

184/222

HEA CPD Pilot Workshop - Workbook

185/222

HEA CPD Pilot Workshop - Workbook

19. Next steps

As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.

70. What is the name of your institution? (This will help inform our analysis and will remain confidential but is not essential information).

71. What is your role? (This information will help us to interpret the data we receive).

As part of this project we are seeking innovative examples of teaching-related CPD evaluation right across the HE sector. We are particularly interested in evaluation which seeks to measure:

o The impact of CPD on student learning

o The impact of institutional culture on CPD

o The impact of CPD and its evaluation on institutional culture

72. If you have examples of the above and would be willing for us to

186/222

HEA CPD Pilot Workshop - Workbook

contact you to discuss the possibility of inclusion as a case study then please provide a name and contact details and we will be in touch with you in the next few weeks.

Thank you very much for the time you have spent completing this questionnaire.

Your inputs are immensely valuable to this project and we appreciate your efforts!

If you have any comments/questions about the audit please email evaluatingcpd@plymouth.ac.uk.

Appendix 2 – Workbook on which the national audit of the tool was based

Evaluating teaching development in HE: towards impact assessment

Pilot Workshop

20th March 2015

Winterbourne House, Birmingham

WORKBOOK

187/222

HEA CPD Pilot Workshop - Workbook

Electronic versions of this information can be accessed via this link. http://www1.plymouth.ac.uk/research/pedrio/Pages/HEA-CPD-Framework.aspx

188/222

HEA CPD Pilot Workshop - Workbook

Introduction

This resource will help you to undertake an evaluation of your chosen CPD activities. You will be asked to undertake the following tasks in order to develop appropriate evaluation instruments to use with CPD participants at your institution.

1. Decide what you want to evaluate2. Decide which questions you want to ask 3. Decide what method you want to use4. Develop evaluation instruments to accommodate these requirements5. Send the project team these instruments by 27th March 2015 to

evaluatingcpd@plymouth.ac.uk 6. Return annotated workbook to Dr Jennie Winter, Plymouth University, Rm 114, 3 Endsleigh

Place, Drake Circus, Plymouth, PL4 8AA by 27th March 2015

The Plymouth Team will then visit your institution and use the instruments you have designed to conduct an evaluation with your identified CPD completers. The results of this will be shared with you and a follow-up interview between you and the Plymouth Team will establish the usefulness of the instruments for meeting your evaluation needs.

189/222

HEA CPD Pilot Workshop - Workbook

Evaluation Protocol

20th March

Introduction to the evaluation framework and toolkit, begin to select questions and determine the format for data collection

27th March

Return evaluation instrument to evaluatingcpd@plymouth.ac.uk Return annotated workbook to

Dr Jennie WinterPlymouth University Rm 114, 3 Endsleigh PlaceDrake CircusPlymouthPL48AA

From 13th April until 8th May: Data collection

Stage 1: Member of the Plymouth Team (Reema, project RA) to visit your institution to undertake the evaluation with the 3-4 CPD. Reema will also interview the Student Union representative

Stage 2: Following the visit and data collection, the data from the CPD completers will be transcribed / collated and returned for you to read through and reflect on in preparation for the telephone / skype interview with a member of the Plymouth Team

Stage 3: A member of the Plymouth Team will interview you via skype / phone to capture your experiences of evaluating CPD provision using the CPD toolkit. As noted in stage 2, you will be asked to review the evaluation data captured through the use of the tool and reflect on it with respect to the insights gained regarding current CPD provision, potential impacts on practice and future developments

190/222

HEA CPD Pilot Workshop - Workbook

Step 1: Consider these questions

1. What do you want to evaluate?

Are you interested in participant satisfaction with the course? In changes to their attitudes? In changes to concepts of teaching and learning? In changes to behaviour? Are you interested in impact on student learning? What assumptions are you making about the links between teaching and learning? Are you interested in the context of transfer?

2. Who are you doing the evaluation for?

Is it going to be used to enhance future CPD development? To provide evidence for the QAA? To feed into institutional policies? Or for another reason?

3. What will happen to this data once it is collected? How will it be used to enhance practice? Where will it be disseminated? Who will have access to it?

191/222

HEA CPD Pilot Workshop - Workbook

Step 2: Evaluation questions

Once you have made these decisions, you can use the questions set out below to help inform your evaluation. The questions are presented by activity and have been carefully constructed from considering both the extant literature and current practice in this area. They are presented to reflect the importance of evaluating at different levels and over a longitudinal time period.

For each activity…..

TimeLevels of evaluation

Pre activity

During activity

Immediately post activity

6 months post activity

12 months+ post activity

MotivationChanges in conceptions of teaching & LearningChanges in teaching and learning behaviourChanges in student perceptions of teacher behaviour

Questions

Changes in student outcomesImpacts on institutional culture

Read through the sets of questions. Decide which you activity you would like to evaluate and consider the suggested questions. Use a highlighter to identify which questions you will include in your instrument and make a note of why you want to use these questions on the workbook. You are not restricted to using only these questions, please feel free to develop your own as well. Please note these on the workbook alongside reasons for their inclusion.

192/222

Tends to move towards more qualitative evaluation methods and to include more questions around transfer, student learning and institutional context

HEA CPD Pilot Workshop - Workbook

CPD Activity 1: Evaluation questions for participants of accredited teaching courses for new lecturers

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

Motivation/Reaction

Why have you chosen to undertake the accredited teaching course?

What do you expect to gain from attending this course?

How were these expectations identified, discussed and recorded?

How have your expectations of the course changed since you started?

How have your individual aims been addressed and accommodated in the course?

What would you like to change about the course?

How did your experience of undertaking the accredited teaching course align with your expectations?

What three things from the accredited teaching course which were most useful to you?

What recommendations to you have to improve future courses?

What do you remember most about the accredited teaching course and why?

What do you remember most about the accredited teaching course and why?

Changes in conceptions of teaching & Learning

How long have you been in a teaching and / or supporting learning role within HE?

In what ways may you have you previously engaged with pedagogic literature and theory?

What have you learnt so far?

In what ways have you engaged with pedagogic literature and theory to develop your practice?

What new information or ideas were you introduced to?

What have you done as a result of completing the accredited teaching course?

Has the process of undergoing the accredited teaching course led to new areas of T&L activity / research?

How has your thinking on [XXXX] shifted following the course?

What impact did attending theaccredited teaching course have on your knowledge?

How did the accredited teaching course develop your network of teaching and learning contacts?

193/222

HEA CPD Pilot Workshop - Workbook

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

Changes in teaching and learning behaviour

What impact do you think attending this course will have on your knowledge, skills or practice?

How have you begun to apply any knowledge, skills or ideas in your practice?

In what ways have you been encouraged to reflect on your practice, and what mechanisms are you using to achieve this?

What impact has the accredited teaching course had on your knowledge, skills or practice?

How have you applied the knowledge, skills and ideas gained during the Accredited teaching course to your practice?

What, if anything, have you changed in response to the information or ideas you were introduced to during the course?

Changes in student perceptions of teacher behaviour

N/A N/A N/A N/A N/A

Changes in student outcomes

What impact do you think attending this course will have on your approach to student learning?

How have you changed or altered your approach to student learning?

What impact do you think attending this course will have on your approach to student learning?

How has your engagement with the accredited teaching course impacted on students?

How can you evidence this impact?

How has you’re your engagement in the accredited teaching course impacted on students learning?

How can you evidence this impact?

Impacts on institutional culture

How did you find out about this course?

Are you expected to attend teaching related CPD as part of your academic development?

How well do the aims of this course align with

What support is currently available to you?

What would assist you in completing the course?

How well did the topics covered in the course align with departmental/institutional priorities?

How do you anticipate sharing information within your department?

How have you been recognised and rewarded for completing the accredited teaching course?

Has the process of engaging in the Accredited teaching course led to the

How has this change been supported in your department / institution?

How have the topics covered in the course contributed to pursuing departmental priorities?

How have you

194/222

HEA CPD Pilot Workshop - Workbook

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

departmental priorities?

What support have you been given to attend this course?

What expectations, if any, are there for you to share information gained on the course with peers in your School / Department?

emergence of communities of practice around teaching and learning?

disseminated the information or ideas you were introduced to during the accredited teaching course?

What challenges have you experienced as you undertook these changes?

How did the accredited teaching course develop your network of teaching and learning contacts?

195/222

HEA CPD Pilot Workshop - Workbook

CPD Activity 2: Evaluation questions for participants of an in-house teaching accreditation framework for established academics

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

In-house teaching accreditation framework

Why have you chosen to apply for accreditation?

How long have you been in a teaching and / or supporting learning role within HE?

How did you find out about the accreditation framework?

Are you expected to obtain teaching accreditation as part of your academic development?

What do you expect to gain from this process?

How important do you think it is that staff are accredited to teach in HE?

What is your level of understanding of the UKPSF?

How have your expectations of the accreditation process changed since you started?

What have you learnt so far?

What support is currently available to you?

What would assist you in completing the accreditation process?

In what ways have you engaged with pedagogic literature and theory to develop your application?

In what ways has reflecting on your experience had an impact on your practice?

In what ways has reflecting on your

How did your experience of undergoing the accreditation process align with your expectations?

What new information or ideas were you introduced to whilst preparing your application for accreditation?

What impact do you think undergoing the accreditation process will have on your knowledge, skills or practice?

In what ways have you engaged with pedagogic literature and theory to develop your application?

What support did you access to undergo the accreditation process?

How do you anticipate sharing information gained

What have you done as a result of undergoing the accreditation process?

How have you applied the knowledge, skills and ideas reflected on through the accreditation process?

What have been the impacts of undergoing accreditation on your school / department?

How have you been recognised and rewarded for undergoing accreditation?

How has undergoing accreditation impacted on students?

Has the process of undergoing accreditation led to new areas of T&L activity / research?

What do you remember most about the accreditation process and why?

What impact did undergoing accreditation have on your knowledge, skills or practice?

How has your thinking on [XXXX] shifted following accreditation?

What, if anything, have you changed as a result of reflecting on your experiences of teaching and/or supporting learning?

How has this change been supported in your department / institution?

What challenges have you experienced as you undertook these changes?

196/222

HEA CPD Pilot Workshop - Workbook

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

What impact do you think gaining accreditation will have on your knowledge, skills or practice?

What impact do you think gaining accreditation will have on your career advancement and employability?

What role, if any, have students from your institution had in this workshop?

What support have you been given to work towards accreditation?

What expectations, if any, are there for you to share your ideas, experience and knowledge of the accreditation process?

experience had an impact on your ideas about teaching, learning and the student experience?

through the accreditation process?

How has the process helped you to support other staff?

What do you think the benefits will be for students following your engagement the accreditation process?

How has undergoing the accreditation process been recognised and rewarded within your department / institution?

What three things from undergoing the accreditation process were most useful to you?

Has the process of undergoing accreditation led to the emergence of communities of practice around teaching and learning?

How do these changes align with departmental priorities?

How has the accreditation process helped you to develop your network of teaching and learning contacts?

How has you’re your engagement in the accreditation process impacted on students learning?

How can you evidence this impact?

197/222

HEA CPD Pilot Workshop - Workbook

CPD Activity 3: Evaluation questions for participants of teaching and learning conferences

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

T&L Conference

Open questionsWhy have you chosen to attend this conference?

How did you find out about this conference?

How long have you been in a teaching and / or supporting learning role within HE?

Are you expected to attend teaching related CPD events as part of your academic development?

How well do the aims of this conference align with departmental priorities?

What do you expect to gain from attending this conference?

What impact do you think attending this conference will have on your knowledge, skills or

N/A Open questionsHow did your experience of the conference align with your expectations?

How well did the topics covered in this conference align with departmental priorities?

What new information or ideas were you introduced to at this conference?

What impact do you think attending this conference will have on your knowledge, skills or practice?

What do you think the benefit was for students involved in this conference?

How do you anticipate sharing the conference information?

Open questionsWhat do you remember as being the most useful session and why?

topics covered in this conference contributed to pursuing departmental priorities?

What, if anything, have you done in response to the information or ideas you were introduced to at this conference?

What impact did attending this conference have on your knowledge, skills or practice?

How have you disseminated the information or ideas you were introduced to at this conference?

How has attending this conference had an impact

N/A

198/222

HEA CPD Pilot Workshop - Workbook

practice?

What role, if any, have students from your institution had in this conference?

What support have you been given to attend this conference?

What expectations, if any, are there for you to disseminate conference information?

Closed questions

What three things from the conference were most useful to you?What recommendations to you have to improve future events?

How do you think attending this conference has extended your network of teaching and learning contacts by attending this conference?

on students?

How did the workshop develop your network of teaching and learning contacts?

199/222

HEA CPD Pilot Workshop - Workbook

CPD Activity 4: Evaluation questions for participants of teaching and learning workshops

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

T&L workshop

Why have you chosen to attend this workshop?

How long have you been in a teaching and / or supporting learning role within HE?

How did you find out about this workshop?

Are you expected to attend teaching related CPD events as part of your academic development?

How well do the aims of this workshop align with departmental priorities?

What do you expect to gain from attending this workshop?

What impact do you think attending this workshop will have on your knowledge, skills or practice?

N/A How did your experience of the workshop align with your expectations?

How well did the topics covered in this workshop align with departmental priorities?

What new information or ideas were you introduced to at this workshop?

What impact do you think attending this workshop will have on your knowledge, skills or practice?

What do you think the benefit was for students involved in this workshop?

How do you anticipate sharing the workshop information?

What three things from the workshop were most

What do you remember most about the workshop and why?

What impact did attending this workshop have on your knowledge, skills or practice?

How has your thinking on [XXXX] shifted following this workshop?

What, if anything, have you changed in response to the information or ideas you were introduced to at this workshop?

How has this change been supported in your department / institution?

What challenges have you experienced as you undertook these changes?

How have the topics covered in this workshop

What do you remember most about the workshop and why?

What impact did attending this workshop have on your knowledge, skills or practice?

How has your thinking on [XXXX] shifted following this workshop?

What, if anything, have you changed in response to the information or ideas you were introduced to at this workshop?

How has this change been supported in your department / institution?

What challenges have you experienced as you undertook these changes?

How have the topics covered in this workshop

200/222

HEA CPD Pilot Workshop - Workbook

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

What role, if any, have students from your institution had in this workshop?

What support have you been given to attend this workshop?

What expectations, if any, are there for you to disseminate workshop information?

useful to you? 1

What recommendations to you have to improve future workshops?

contributed to pursuing departmental priorities?

How have you disseminated the information or ideas you were introduced to at this workshop?

How did the workshop develop your network of teaching and learning contacts?

How has attending this workshop impacted on students learning?

How can you evidence this impact?

contributed to pursuing departmental priorities?

How have you disseminated the information or ideas you were introduced to at this workshop?

How did the workshop develop your network of teaching and learning contacts?

How has attending this workshop impacted on students learning?

How can you evidence this impact?

201/222

HEA CPD Pilot Workshop - Workbook

CPD Activity 5: Evaluation questions for participants of peer review activity

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

Peer Review

If a previous review has taken place, what, if any, goals were set at the end of the previous review?

To what extent have your previous goals been achieved?

What is the focus of this review?

What materials are available as a focus for reflection and discussion10?

What are your aims or intentions in this review, or (for example) using technology?

Are there any specific skills or areas of knowledge that you hope to develop over the next 12 months?

What do you hope to gain from this review process?

What, if anything, have you learned:o About your students?o About your teaching or

your role as a teacher?o About teaching and

learning more generally?

Has the review process resulted in any changes in your thinking?

What changes will you make as a result of the review process?

How confident are you that you can address the concerns that you raised at the start?

Has the review process raised any new concerns?

Have you gained what you hoped from the review?

How has your thinking changed as a result of the review process?

Have you made any changes to your teaching approach or methods?

How have students responded? How do you know?

How confident are you that you can address the concerns you raised during the review process?What did you gain from the review? Is your answer to this the same as it was immediately afterwards?

Assuming this is a cyclical process,

probably in annual cycles, return to the “Before” questions

10 For example, previous review documentation or reflective notes, ATI, course and lesson plans, student work, student feedback.

202/222

HEA CPD Pilot Workshop - Workbook

CPD Activity 6: Evaluation questions for recipients of teaching and learning development grants

CPD Activity Timeframe for evaluationPrior to the CPD

activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

T&L Development grants

Before the start of a project but after funding notification received; therefore we assume SOTL / teaching development grants are tied to funding bestowed institutionally. This will make it easier to track the aims of the funding scheme with the impact on practice etc.

We draw on the work of Admunson & Hum (2012: 4) to define the scope of work undertaken by teaching development grants / SoTL-based projects as those that result in ‘systematic inquiry’ into an aspect of teaching practice, which may results in changes in ‘conceptions of self as a teacher and researcher, changing understanding about student learning and the application of project findings to teaching practice’. This definition

How long have you been in a teaching and / or supporting learning role within HE?

Why did you submit an application to undertake the proposed project?

What was the rationale for undertaking the proposed project?

What informed the development of this proposal?

How well do the aims of this project align with departmental / institutional priorities?

What do you expect to gain from undertaking this project?

Is the project on course?

How have your expectations of the project changed?

What challenges have you encountered?

What successes have you experienced?

If relevant, how have students been involved in the project?

What professional development opportunities have you identified that would assist you in undertaking this work?

Are you more aware of scholarship and

How did your experience of the project align with your expectations?

How well did the project align with departmental / institutional priorities?

What new information or ideas did you gain as a consequence of undertaking this project?

What knowledge, experience or skills of pedagogic research and development work do you feel you gained as a consequence of this project?

What knowledge, experience or skills of teaching practice and student learning do you feel you gained as a consequence of this project?

IV question successfully used to open dialogueInitially we would like you to reflect on your experience of undertaking pedagogic research and development work through the teaching fellowship scheme.

How have you applied or used the knowledge gained through the project since its completion?

What were the impacts on your professional development of undertaking this project?

What have been the responses of colleagues to the outcomes of the

Have you applied for funding to continue with this work?

How have you built on the findings of this project?

Have the findings from this project led to new areas of T&L activity / research?

Networking

Impacts on practice

203/222

HEA CPD Pilot Workshop - Workbook

CPD Activity Timeframe for evaluationPrior to the CPD

activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

aligns with the model of SoTL advocated by Trigwell and Shale (2004) where they include student learning as an outcome that can be documented.

What impact do you think undertaking this work will have on your knowledge, skills or practice?

What do you anticipate the impact of the proposed project will be?

What experience do you have of pedagogical research and / or teaching development work?

What role, if any, will students have in the proposed work?

What support / CPD needs have you identified that would assist you in undertaking this work?

What expectations, if any, are there for you to disseminate the

pedagogic literature as a consequence of this project?

What recommendations for teaching, learning and the student experience resulted from this work and is it possible to implement them?

What do you think the benefit was for students involved in this project?

How do you plan to disseminate the outcomes of this work?

Who supported you in completing this work?

How do you think undertaking this project has extended your network of teaching and learning contacts by attending this conference?

Are you more aware of scholarship and pedagogic literature as a consequence of this project?

work?

What have been the impacts of this work on your practice, your school, and where relevant more widely?

If relevant, what have you undertaken since that has followed on from or built on the work that was undertaken through this project?

How has undertaking this project had an impact on students?

204/222

HEA CPD Pilot Workshop - Workbook

CPD Activity Timeframe for evaluationPrior to the CPD

activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

outcomes of this work?

205/222

HEA CPD Pilot Workshop - Workbook

CPD Activity 7: Evaluation questions for participants involved in teaching and learning mentoring

CPD Activity Timeframe for evaluation

Prior to the CPD activity During CPD activity Immediately after the CPD activity

6 months after the CPD activity 12-24 months

T&L MentoringQuestions here assume mentoring will focus on developing an individual's academic career as a whole, not just their teaching. In some institutions, teaching may not be considered the most important area in which to support development.

How long have you been in a teaching and / or supporting learning role within HE?

Are there any targets that you are expected to reach in the next 12-24 months?

What are your own goals in relation to:o Your teaching?o Your academic career?

Are there any milestones you hope to reach in the next 12-24 months (e.g. completing a probationary period)?

What do you hope to achieve in the next 12-24 months?

How will you know whether you have reached your goals?

How confident are you that you understand institutional

Has anything in your practice changed as a result of discussion with your mentor?

Do you feel any more able to address your main concerns?

Are any new issues or concerns being raised?

Has there been any change in what you hope to gain from working with your mentor?

What, if anything, have you learned:o About your students?o About your teaching

or your academic identity?

o About teaching and learning generally?

Has the mentoring process resulted in any changes in your thinking?

Do you think you will make any changes to the way you approach your teaching as a result of mentoring? (Yes)(No).o If Yes, what do you

intend to change?

Has the mentoring process raised any new concerns?

What progress has been made towards targets that you are required to reach?

Is the mentoring relationship continuing?

Has your thinking about your teaching changed as a result of the mentoring process?

Have you made any changes to your teaching approach or methods?

How have students responded? How do you know?

Has there been any other effect of the mentoring process?

How confident are you that you can address the concerns you discussed with your mentor?

What did you gain from the mentoring/working with your mentor? Is your answer to this the

Is the mentoring relationship continuing?

Has your thinking about your teaching changed as a result of the mentoring process?

Have you made any changes to your teaching approach or methods?

How have students responded? How do you know?

Has there been any other effect of the mentoring process?

How confident are you that you can address the concerns you discussed with your mentor?

What did you gain from the mentoring/working with your mentor? Is your answer to this the

206/222

HEA CPD Pilot Workshop - Workbook

and departmental policies and procedures related to teaching and learning?

What materials are available as a focus for reflection and discussion about your teaching11?

Have you gained what you hoped from working with your mentor?

same as it was immediately afterwards?

same as it was immediately afterwards?

11 For example, previous review documentation or reflective notes, ATI, course and lesson plans, student work, student feedback.

207/222

HEA CPD Pilot Workshop - Workbook

General questions that help to establish participant reaction to the CPD activity

Evaluation questions about participant reaction to the CPD activity

Participant reaction does not give the evaluator any information about impact on teaching practice. Instead it provides an overview of the participant’s satisfaction with teacher facilitation, content, delivery, administration, facilities and logistics of the CPD activity. Establishing reaction can be useful because it helps to inform future development of the activity. These questions are typically administered during and after the activity. You may or may not wish to include some of these questions in your evaluation.

Evaluation questions about teacher facilitation

Was your teacher sufficiently knowledgeable in this area?

Did your teacher encourage you to participate in the activity?

Did your teacher welcome your and other students’ ideas?

Did your teacher explain the content of the activity clearly?

Did your teacher use appropriate examples?

Did your teacher use a range of pedagogic activities in the session?

Did your teacher answer questions in a helpful way?

Was your teacher well prepared?

Did your teacher use technology effectively?

Did your teacher use your name?

What were the teacher’s strengths?

What were the teacher’s weaknesses?

How could the teacher improve their teaching?

Evaluation questions about content and delivery

Was the course pitched at the right level for you?

Did the course meet your expectations?

Were the materials useful?

Were the materials accessible?

Were the technologies accessible?

208/222

HEA CPD Pilot Workshop - Workbook

Was the teaching of good quality?

Was the teaching well paced?

Were the teaching methods used appropriate?

Was the content relevant?

Was the content well structured?

Was the content up-to-date?

Was the content interesting?

Were sufficient examples included?

How much of the content was useful to you?

What were the best things about the activity?

What were the worst things about the activity?

Was this activity time well spent?

Would you recommend this activity to others?

Is this activity relevant to other people in your workgroup?

How could the activity be improved?

Did you make any new contacts at this event?

Evaluation questions about the administration, facilities and logistics

Were you satisfied with the standard of the venue?

Were you satisfied with the standard of the food?

Were you satisfied with the standard of the administration?

Was the activity value for money?

Were the joining instructions easy to follow?

Was the date of the activity convenient?

Was the time of the activity convenient?

Was the length of the activity appropriate?

Was the venue location convenient?

How far did you travel to attend this activity?

209/222

HEA CPD Pilot Workshop - Workbook

Were you able to see the activity clearly?

Were you able to hear the activity communications clearly?

Was the room temperature comfortable?

Were the coffee/food breaks conveniently timed?

Were the coffee/food breaks sufficient?

210/222

HEA CPD Pilot Workshop - Workbook

Step 3: Choose the method you want to use

There are pros and cons to using different methods in evaluation. You should consider what it is you want to find out and who you are evaluating for. What sort of data helps to demonstrate this quickly and effectively? It can be a good idea to have a mix of qualitative and quantitative data to help communicate impact to different audiences. The national audit undertaken as part of this project reported questionnaires, interviews and focus groups as commonly used instruments for reporting evaluation. We provide some guidance for using these methods below, read though this information and decide which data collection type will work better for you in your evaluations. You will then need to develop these instruments and send them to us by 27th March 2015 to mailto:evaluatingcpd@plymouth.ac.uk as well as returning this workbook. These will be used as the basis of our data collection when we come to visit your institution.

Questionnaires:

Questionnaires offer an opportunity to gather basic data from groups of people. They are good for measuring responses to specific questions and they enable researchers to make comparisons across groups. They can also help you collect data over time with the same cohort and across different cohorts in order to help ascertain how attitudes and perceptions have changed.

Strengths:

Allow you to survey a large group of people relatively quickly Good at collecting factual information Can be anonymous Can be offered on paper or online Tend to be less time intensive than other methods in terms of administering, completing and

analysing A useful tool for making comparisons across cohorts and year groups Can collect quantitative or qualitative data Quantitative data can be systematically analysed Online surveys packages incorporate tools for analysis You can gather contact details for follow-up interviews or focus groups

Challenges:

Completion rates are often linked to length of time a questionnaire takes and the demands it makes

Additional factors such as tone of voice, facial expressions cannot be taken into account It is harder to gather responses to more complex questions Developing clear, unbiased questions can be challenging and may require training

211/222

HEA CPD Pilot Workshop - Workbook

Things to consider:

Be clear about what you are evaluating Determine what types of questions are most relevant to your study: Closed – T/F; multi-

choice; Likert-style or Open – with free text answers Pilot the questionnaire to test the clarity and validity of the questions

When to use:

Questionnaires are useful to gather baseline data, take a snapshot of participants’ views midway in a course, understand attitudes towards satisfaction at the end of a course or as a means of rating the relative value of components of a CPD programme in a longitudinal study. Additionally, questionnaires can be used as one element of a mixed methods study, and may offer findings which can be explored in more depth using tools such as interviews or focus groups.

Ethical issues:

Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity and withdrawing from answering a questionnaire at any point.

Questionnaire resources:

Graham Gibbs lectures

A series of three video lectures with creative commons licenses. Taken together, they offer quite a comprehensive guide, pitched at an audience that has little experience of using questionnaires:

Video 1: Question types and piloting https://www.youtube.com/watch?v=vjailyWAcJQ This also addresses the point of when a questionnaire is an appropriate method.

Video 2: Questionnaire layout and question wording https://www.youtube.com/watch?v=BZLVI5zae2E

Video 3: Ratings and scales https://www.youtube.com/watch?v=_aOhcGf8EcY

Creative Commons license: CC-BY-NC-SA

Sheffield Learning and Teaching Resource

http://www.shef.ac.uk/lets/strategy/resources/evaluate/general/methods-collection/questionnaire

This University of Sheffield online resource contains a summary of advantages and disadvantages of using questionnaires, some advice about the process and a link to some sample evaluation questions.

212/222

HEA CPD Pilot Workshop - Workbook

University of Wisconsin – Cooperative Extension 2009

http://learningstore.uwex.edu/assets/pdfs/g3658-11.pdf

This online resource provides an excellent introduction and many helpful examples of how to develop an end-of-session evaluation questionnaire for teaching–related CPD.

Interviews:

Interviews offer a way to explore questions in more depth and they tend to yield qualitative data that is potentially richer and more complex than that collected through questionnaires. They are also useful to follow up findings from questionnaires.

Things to consider:

What format of interview would best suit your evaluation? Interview formats may be:

Structured – fixed set of questions asked to all interviewees Semi-structured – set of questions and key themes for exploration, but questions vary

between interviews Unstructured – no predetermined questions; interviewee is encouraged to talk around a set

of issues

How will you keep a record of the interview? You may wish to use a voice recorder so that you can concentrate on asking questions and listening to the responses. If you are using a recording device, ensure that you have consent from your interviewee.

How do you intend to analyse the data? You might wish to transcribe in full and use a grounded approach (that involves coding of the data) as a means of interpretation. This is labour-intensive but may help you make comparisons across groups over time. Or you may prefer to draw out broad themes that you observe in the data.

Strengths:

Topics can be probed more fully Interviews are appropriate for complex, nuanced discussions Questions can be tailored to suit the context The interviewee’s views are brought to the fore and can shape the direction and flow of the

conversation Topics that are unanticipated by the interviewer can be accommodated and explored

213/222

HEA CPD Pilot Workshop - Workbook

A fine-grained perspective of the question under consideration can be obtained

Challenges:

Interviews can be time-consuming to organise, administer and analyse Anonymity is harder to guarantee with interviews Interviews are not practical for large groups where sampling would be required Analysis of interview data can be challenging and less straightforward than survey data They may require transcribing or extensive note-making Participants may be less willing to be interviewed than to complete a survey It can require some practice to guard against ‘leading’ an interviewee towards a particular

response

When to use:

Interviewing might be a suitable technique to use if you are following a set of individuals on a longitudinal study as they develop their practice after completing a CPD course. So, you could arrange to interview them at regular intervals, with a view to building up over time an understanding of their journey as teachers. The data could be reported as a case study.

Ethical issues:

Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity, responsible reporting of data and withdrawing from the interview process at any point. If the interview is being recorded, a specific consent form will need to be provided.

Interview resources:

https://itunes.apple.com/us/book/research-methods-for-business/id862468951?ls=1&mt=13

Djebarni, R., Burnett, S. ; Richards, B. (2014) Research Methods for Business students, Managers and Entrepreneurs. University of South Wales.

This multimodal e-book is an excellent resource ; it is good for interviewing techniques but also covers questionnaires and focus groups. It is freely available, CC-licensed, but has been designed to work with iPads and Macs.

Licence: Attribution-NonCommercial-ShareAlike 4.0 International

What makes a good interview?

https://www.youtube.com/watch?v=9t-_hYjAKww

214/222

HEA CPD Pilot Workshop - Workbook

In this 18-minute video, Graham Gibbs comments on examples of good and bad interview technique and runs through what he considers to be the ten characteristics of a good interviewer. Some people might feel they know most of this but the replay of an interview with commentary is quite engaging.

Focus groups:

Focus groups are effectively group interviews and they can be an efficient way of exploring a set of issues in depth with a cohort. Focus groups are also a useful way of gathering a range of views and offer an opportunity to hear points and counterpoints.

Things to consider:

Having two people to run the focus groups works well - with one person facilitating the discussion and the other taking notes. It is particularly helpful to make a recording of the session.

Ensure that members of the group feel comfortable with each other before the session officially begins. The establishment of a sense of trust, an environment conducive to discussion and a relaxed atmosphere should enable a free flowing conversation.

Strengths:

Focus groups offer a good opportunity for participants to reflect upon and share experiences Participants can stimulate observations or recollections that might not have surfaced in

either surveys or interviews The experience of thinking collaboratively about a topic can be a rewarding activity for

groups of peers. Participants may feel they have gained new perspectives or ideas from the session

The data generated from a focus group has the potential to be rich and multi-voiced

Challenges:

Arranging focus groups can be more time-intensive than surveys or interviews Good facilitation skills are needed to keep discussions focused and ensure everyone is

contributing Dominant, persuasive speakers can steer the discussion in a way that may not be

representative of the group It is difficult to take notes in a focus group, so you may wish to record the session (with

consent of participants) Participants need to feel secure in the presence of peers to express their views. A trusting,

confidential environment should be established

215/222

HEA CPD Pilot Workshop - Workbook

When to use:

Focus groups would be a good way of getting feedback about a programme midway through and at the end. It could be a good way of working with institutional stakeholders to canvas their views about the impact of CPD programme and it would be a useful way of gathering longitudinal data from participants.

Ethical issues

Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity, responsible reporting of data and withdrawing from the focus group at any point. If the focus group is being recorded, a specific consent form will need to be provided. The facilitator should emphasize the importance of confidentiality within the group to establish a sense of trust for all contributors

Focus groups resources:

http://infed.org/mobi/using-focus-groups-in-evaluation-and-research/ - helpful, clear and practical. Addresses evaluation directly and offers useful advice for how to handle data if using for a research project.

Smith, M. K. (2011). ‘Using focus groups in evaluation and research’, the informal education homepage. [http://infed.org/mobi/using-focus-groups-in-evaluation-and-research/. Retrieved: 9 March 2-15].

http://www.joeyanne.co.uk/2012/03/13/facilitating-focus-groups/ This is an excellent CC-licensed blogpost describing the process of setting up and running a focus group. Jo Alcock is a librarian at Birmingham City University. This work by Jo Alcock is licensed under a Creative Commons Attribution-Non-Commercial-Share Alike 2.0 UK: England & Wales License

http://localtrust.org.uk/library/how-to-guides/focus-group - general advice on how to run a focus group by the Big Local charity partnership. CC licensed: ‘All our work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 UK: England and Wales License, unless it says otherwise. - See more at: http://localtrust.org.uk/library/how-to-guides/focus-group#sthash.3tlX8C79.dpuf ‘

Karen Vinall © University of Leeds 2014. This work is made available for reuse under the terms of the Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International Licence.

216/222

HEA CPD Pilot Workshop - Workbook

Other methods:

These examples are not the only ones that can be used. The audit we undertook as part of this work suggested that other methods are also used including looking at participants’ assessment, learning journals, PTES/NSS data , examples of curriculum development, peer reviews and more. We are interested in your ideas about other methods and will ask you about this during the workshop.

Step 4: Develop evaluation instruments to accommodate these requirements

Please use the guidance in this document and the online toolkit to develop evaluation instruments you would like to use with your CPD completers.

Step 5: Send the evaluation instruments

Send the project team these instruments by 27th March 2015 to evaluatingcpd@plymouth.ac.uk

Step 6: Return annotated workbook

Return annotated workbook to Dr Jennie Winter, Plymouth University, Rm 114, 3 Endsleigh Place, Drake Circus, Plymouth, PL4 8AA by 27th March 2015.

Finally many thanks for supporting this project.

We look forwards to meeting you again during our visit to your institution.

If you have any questions please get in touch with us at evaluatingcpd@plymouth.ac.uk

217/222

HEA CPD Pilot Workshop - Workbook

Appendix 3: Appendix 3: Key findings from interviews with CPD completers captured through WP3

Summary analysis – teaching courses for new lecturers

This is a summary of the interview data generated from talking to CPD completers about courses.

Motivation

Often a contractual obligation – this impacted on participants’ intrinsic and extrinsic motivation.

Part time staff were often very enthusiastic to be involved in CPD. The data suggests there is often a disconnect between CPD need and provision – this

impacts on participation and transfer.

Conceptions of teaching and learning

Participants reported shifts in conceptions around :o Acknowledging differences in individual studentso Being exposed to pedagogic literature and theory o Attitude and approaches to practice .

There was evidence that the conceptual shifts depended on the point in their career at which the staff began the course – so teaching experience plays an important role in what staff take on - and so does what their needs are at the time.

These participants were not required to evidence impact on student learning outcomes as part of the curriculum or to pass the course.

Participants noted that the experience of being a student helped shift their assumptions about their own students.

Changes to teaching behaviour

Lots of examples of shifts in practice as a result of participation including:o Giving structured feedback o Writing aims and learning outcomes

Transfer of learning to practice was influenced by the perceptions of students. Reflection dis not necessarily lead to changes in behaviour . Collaboration in departments help enhance transfer to practice.

Changes to student learning

218/222

HEA CPD Pilot Workshop - Workbook

Those new to teaching did not place much emphasis on evidencing changes to student learning, they instead focused on changes to their teaching.

Participants were aware of evaluation instruments such as module evaluations and internal student surveys but were not confident of how these could evidence changes in student learning.

Participants teaching in educational related subjects were more critical of curriculum so were often better versed in discussing (but not necessarily undertaking) student evaluation. The ways they conducted student evaluations seemed more aligned with departmental and institutional processes and values. They found their CPD experiences useful in getting their students to think about and discuss topics around teaching and learning.

This was generally a difficult question for participants to answer. Participants were aware that it is difficult to measure change and to compare cohorts of

students.

Reward and recognition

Generally staff did not feel they were being recognised for participation in teaching –related CPD, particularly where participation was mandatory.

Future CPD

Some participants reported that they were considering moving on to higher qualifications, and many others reported that being involved in a course did make them think about future CPD opportunities including publishing papers and hearing stories of good practice.

Networks and dialogue

There was evidence of a variety of informal networks in existence but there was little evidence of how participation in courses linked to informal networks – these were instead left to chance.

Suggestions for how CPD can be improved

Developing different links with students – staff-student CPD centred conferences - encouraging students to participate in CPD-like activities to develop their understanding

Having a dedicated mentor

Summary analysis - In house accreditation schemes

This is a summary of the interview data generated from talking to CPD completers about ‘in house accreditation schemes’.

219/222

HEA CPD Pilot Workshop - Workbook

Motivation

Participants tended to be better motivated to undertake accreditation schemes where they were doing so voluntarily.

There was a distinction in value between the HE and HE in FE sector: more valued in HE in FE sector.

There were mixed perceptions about whether participants perceived accreditation schemes as CPD or badging. For example, instead of using the accreditation schemes to further individual development some participants reported using it to gain recognition for work already done. Some saw it as CPD - some saw it as badging.

Reflection

Looking back at work done was considered useful however there was critique that the scheme did not foster the whole reflective cycle as there was little sense of next steps.

Conceptions of teaching and learning

Participants have to demonstrate impact so you are forced to consider and evidence changes to conceptions of teaching and learning -this is not so much the case with other activities such as the accredited/non-accredited courses.

Changes to teaching behaviour

There was little evidence that participation motivated changes in teaching and learning behaviour - this seems to be because the process largely encourages looking backwards.

Changes to student learning

This was generally a difficult question for participants to answer. They were aware that it is difficult to measure change and to compare cohorts of students. Some mentioned certain issues in getting access to information that can help to ascertain

possible impact

Reward and recognition

Participants were explicitly being recognised through gaining accredited status but this was not always celebrated within the institution and that impacted on how value of the accreditation scheme was perceived.

220/222

HEA CPD Pilot Workshop - Workbook

Scholarly activity

Participants of the accredited frameworks reported using literature to inform practice (delivery/module development/curriculum development) and using action research to embellish their applications.

Future CPD

Participants reported this process encouraged them to look back rather than look forwards and so there was little evidence of future CPD plans.

Networks and dialogue

There was evidence of a variety of informal networks in existence but there was little evidence of how participation in accredited frameworks linked to informal networks – these were instead largely left to chance.

Suggestions for how CPD can be improved

Emphasis on action research at the micro level as a way to improve the relevance and applicability of the accreditation process.

Having more recognition within the institution for achieving this accreditation route.

Summary analysis across activities

This is a summary of the interview data generated from talking to CPD completers about various activities.

Suggestions for how CPD can be improved

Use of reflective models in everyday practice – looking at CPD as a continual process and thinking “what next?” – not rolling out the red carpet for peer review

Interdisciplinary and team approaches to peer review Interdisciplinary meets and discussions Creating different links with students – staff-student CPD centred conferences - students

participating in CPD-like activities to develop their understanding Having a dedicated mentor Having a mix of practice-based and study – and alternative assessment formats - just

because you cannot write does not mean you cannot teach Considering duration of CPD activity - longer and shorter term interventions Increasing the ways in which we can access the student voice

221/222

HEA CPD Pilot Workshop - Workbook

Emphasis on action research at the micro level as a way to improve the relevance and applicability of the accreditation process

Having more recognition within the institution for CPD Having improved access to training and expertise in specific research-related skills Institutional systems to help staff capture and discuss reflections.

Linking to other institutional processes

Annual review was highlighted as the main way CPD is linked with institutional processes – but there is evidence of frustration with this. Annual review was not always seen as developmental but rather stock taking. CPD needs were not always discussed at annual reviews . Time scales were identified as another factor – an annual process was not seen as conducive to professional development.

Other issues around teaching and how teaching is managed were also highlighted – how teachers often have responsibilities but no rights - and the difficulty in managing people one has no supervisory authority over.

Staff perceived CPD as their own responsibility but looked to the institution for systems to help them capture and discuss reflection.

Better ways to evaluate

Participants made a number of suggestions for how this process could be better evaluated. These included:

Concrete advice about methods of reflection per session and generally Linking to good standing agenda - getting to reflect each year Evaluating multiple points and multiple stakeholders Having the possibility to quickly self-rate - bullet points and check boxes Building evaluation into session plans and peer review Evaluation should identify gaps and plans to fill them Evaluation should be specific and explicit Awareness that constantly asking students for feedback can turn them off – they need to

be convinced about why evaluation is a good and important activity – suggestions emerged that they could be better versed in how to engage in dialogue and feedback

Raised questions about where students have the opportunity to give better feedback on and beyond module evaluation forms

222/222