terms of reference - university of plymouth · web viewcpd in teaching is manifest in a number of...
Post on 30-Jun-2019
213 Views
Preview:
TRANSCRIPT
Evaluating teaching development in higher education: towards impact assessment
Project Summary
Confidential to the Project Team and HE Academy.
This document is shared with the Academy providing additional context and information about the process of the project, and to inform discussion at the final meeting.
It is not intended for wider distribution or publication.
The project team:
PedRIO: Pauline Kneale, Jennie Winter, Rebecca Turner, Lucy Spowart and Reema Muneer
Critical friends: Denise Chalmers, Nancy Turner,
HEDERA: Jane Hughes, Colleen McKenna
Project partners: Jan Smith (Durham) and Chris Smith (University Campus, Suffolk)
2/222
Table of Contents
Terms of reference................................................................................................................................6
Abbreviations used in this report..........................................................................................................6
Acknowledgements...............................................................................................................................6
Executive summary...............................................................................................................................6
Framing of the project...........................................................................................................................8
Work Package 1 (WP 1) - Literature review.........................................................................................12
1.1 Introduction...............................................................................................................................12
1.2 Building on Parsons...................................................................................................................13
1.3 Method......................................................................................................................................15
1.4 Areas of Impact..........................................................................................................................17
1.4.1 Impact on teachers’ attitudes, knowledge and skills..........................................................17
1.4.2 Evidence of impact on teachers’ behaviour and practice..................................................19
1.4.3 Effects of disciplinary or generic programme focus...........................................................20
1.4.4 Compulsory vs non-compulsory.........................................................................................21
1.4.5 Student impact/impact on student learning.......................................................................22
1.4.6 Other references to impact not covered by the above......................................................25
1.5 Research in the broad area of CPD impact that addresses the construction and use of frameworks.....................................................................................................................................28
1.6 Research that goes beyond impact of CPD but which may be relevant more broadly..............32
1.6.1 Reconceptualising academic development........................................................................32
1.6.2 Impact of networks and brokers........................................................................................33
1.6.3 Narratives..........................................................................................................................33
1.6.4 Impact of diverse participation groups..............................................................................33
1.6.5 Identity construction, boundary crossing and impact........................................................34
1.7 Critique of the impact discourse................................................................................................35
1.8 Conclusion.................................................................................................................................37
1.8.1 UKPSF, teaching excellence and CPD frameworks.............................................................37
1.8.2 Gaps in the literature.........................................................................................................39
1.8.3 Key points and implications for future work on evaluating CPD........................................39
1.9 Final thoughts............................................................................................................................40
Work Package 2 (WP 2): Invite institutions operating CPD schemes to provide evidence of current practice in measuring the impact and effectiveness of CPD provision............................................41
3/222
2.1 National audit............................................................................................................................42
2.1.1 Themes addressed within the audit...................................................................................42
2.1.2 Audit audience...................................................................................................................44
2.1.3 Audit development............................................................................................................44
2.1.4 Dissemination of the audit.................................................................................................45
2.1.5 Analysis of audit data.........................................................................................................45
2.2 Results of the National Audit.....................................................................................................46
2.2.1 Overview of respondent profile.........................................................................................46
2.2.2 CPD Provision (N = 109).....................................................................................................50
2.2.3 Evaluation...........................................................................................................................54
2.2.4 Student involvement in CPD (N=108).................................................................................60
2.2.5 Institutional policies and culture around teaching-related CPD.........................................61
2.3 Case study development...........................................................................................................66
2.3.1 Selecting the case study institutions..................................................................................66
2.3.2 Data collection...................................................................................................................67
2.4 Case studies...............................................................................................................................68
2.4.1 Case Study – London Metropolitan University....................................................................68
2.4.2 Case Study – University of Roehampton.............................................................................72
2.4.3 Case Study – Sheffield Hallam University............................................................................76
2.4.4 Case Study – Nottingham Trent University.........................................................................80
2.4.5 Case Study – Queen Margaret University, Edinburgh.........................................................86
Professional development opportunities for staff............................................................................86
Evaluating the effectiveness of teaching-related CPD......................................................................87
Challenges to evaluating CPD..........................................................................................................88
Future Plans.....................................................................................................................................89
Work Package 3 (WP 3): Design and test a tool for the assessment of the effectiveness of CPD........90
3.1 Tool Design and Development...................................................................................................90
3.1.1 Influential factors shaping the development of the tool....................................................91
3.1.2 Tool development...............................................................................................................93
3.2 Tool Implementation and Piloting.............................................................................................95
3.3 Tool Evaluation..........................................................................................................................97
3.4 Results of the national pilot of the evaluation tool....................................................................99
3.4.1 Piloting of the tool..............................................................................................................99
3.4.2 CPD completers experiences of using the tool.................................................................104
4/222
3.4.3 Recommendations for the further development of the tool based on the analysis CPD completer data..........................................................................................................................106
3.4.4 CPD Providers experiences of using the tool...................................................................106
3.4.5 Limitations to the current evaluation work......................................................................112
3.4.6 Implications for the further development of the tool based on the analysis CPD completer data............................................................................................................................................112
3.5 Recommendations...................................................................................................................113
3.5.1 Higher Education Academy...............................................................................................113
3.5.2 HE Providers......................................................................................................................114
3.5.3 Educational Developers / CPD providers..........................................................................114
3.5.4 Individual..........................................................................................................................115
3.5.5 Future work......................................................................................................................115
References.........................................................................................................................................116
Appendix 1: Audit template used in WP2..........................................................................................121
Appendix 2 – Workbook on which the national audit of the tool was based....................................154
Introduction...................................................................................................................................155
Evaluation Protocol.......................................................................................................................156
Step 1: Consider these questions..................................................................................................157
Step 2: Evaluation questions..........................................................................................................158
Step 3: Choose the method you want to use.................................................................................179
Step 4: Develop evaluation instruments to accommodate these requirements...........................184
Step 5: Send the evaluation instruments.......................................................................................184
Step 6: Return annotated workbook.............................................................................................184
Appendix 3: Appendix 3: Key findings from interviews with CPD completers captured through WP3...........................................................................................................................................................185
Summary analysis – teaching courses for new lecturers...............................................................185
Summary analysis - In house accreditation schemes.....................................................................186
Summary analysis across activities................................................................................................187
List of Tables
Table 1.1: stages of the literature review
Table 1.2: tags used to code the literature
Table 2.1: categories of institution and responses
5/222
Table 2.2: percentage of institutions with no minimum teaching-related CPD requirement
Table 2.3: types of CPD from most to least commonly reported
Table 2.4: focus of accredited and non-accredited CPD offers
Table 2.5: overview of the evaluation practices discussed by respondents
Table 2.6: perception of student awareness and involvement in CPD
Table 2.7: percentage of institutions which have an existing policy / expectation around peer review (by institutional category)
Table 2.8: peer review and annual appraisal / performance review
Table 3.1: range of CPD activities included in the pilot
Table 3.2: frequency counts of the themes from which the questions were selected across the range of CPD activities included in the pilot
Table 3.3: timeframes over which the CPD completers had engaged with their CPD activities that were the subject of the pilot
Table 3.4: what the CPD providers were using the tool to evaluate
Table 3.5: the purpose of the evaluation
Table 3.6: the intended uses of the evaluation data
List of Figures
Figure 2.1: the most commonly reported CPD expectations for different roles
Figure 2.2: accredited CPD offers described by respondents
Figure 2.3: non-accredited CPD offers described by respondents
Figure 2.4: persons involved in the delivery of CPD
Figure 2.5: count of evaluation at each stage of the accredited / non-accredited offer
Figure 2.6: focus of evaluation of accredited / non-accredited offers at different stages
Figure 2.7: institutional differences in promotion pathways
Figure 2.8: occurrence of professors of teaching and learning
Figure 2.9: reward and recognition for teaching and learning
List of Boxes
6/222
Box 2.1: overview of findings related to teaching qualifications and minimum CPD requirements
Box 2.2: overview of findings related to accredited / non-accredited CPD offers
Box 2.3: overview of findings related to specific CPD activities
Box 2.4: overview of findings related to reward and recognition for teaching and learning
Terms of reference
CPD provider – this term is used to represent individuals who may perform a staff, educational or academic development function or an individual who was responsibility for the provision of teaching-related CPD.
CPD completer – this term is used to represent individuals who have engaged with teaching-related CPD.
Abbreviations used in this report
Abbreviation DefinitionCBHE College-based HECPD Continuing professional developmentFE Further EducationFTE Full time equivalentsGTA Graduate teaching assistantsHE Higher educationHEA Higher Education AcademyNSS National Students SurveyPDR Performance Development ReviewPTES Postgraduate Taught Experience SurveyQAA Quality Assurance AgencySEDA Staff and Educational Development AssociationSoTL Scholarship of teaching and learningUKPSF UK Professional Standards FrameworkWP Work package
Acknowledgements
This project was funded by the HE Academy.
The project research and toolkit development was undertaken by the authors, through the Pedagogic Research Institute and Observatory (PedRIO) at Plymouth University in collaboration with the Higher Education Development Research Agency (HEDERA) and critical advisors.
7/222
The toolkit for evaluation was developed and piloted with 12 HE institutions including college-based, private provider, teaching focused and research focused HE institutions located in England, Scotland and Wales.
The research team is very grateful to all the partners who gave their time to explain current processes and shared classroom evaluation practice.
8/222
Executive summary
This report is informed by the literature on evaluating higher education teaching and by empirical
research which established common and best practice across the sector. It provides a series of
templates that can be used to create bespoke CPD evaluations aiming to capture the impact through
evaluation before, during and after an activity or event. In essence this follows a five step process:
Analysis of staff and programme development needs in relation to previous feedback,
institutional and departmental priorities;
Establishing learning outcomes for the CPD activity and the aims of the evaluation;
Consider HOW and WHEN to evaluate impact, considering the aims and learning outcomes;
Deliver or facilitate the CPD activity and commence ongoing evaluation. This may be ongoing
over several years;
Disseminate the findings and share the lessons learnt.
The Report is focussed around supporting staff to develop meaningful evaluation processes, using
standard social science research methods, and supported through a suite of 12 Templates that offer
structure to the selection of questions to pose.
The approach advocates evaluation activity before, during and after events, with the emphasis on
repeating evaluation after 6 months and two years to gauge longer term impacts. The annual
university cycle means that opportunities to act on information gained in a CPD activity cannot be
used for eighteen months to two years.
The templates are a primary resource that the authors hope others will contribute to, allowing us to
expand and refine the data base of questions, and the scope of the templates.
These templates are available at http://www1.plymouth.ac.uk/research/pedrio/Pages/HEA-CPD-Framework.aspx
9/222
Framing of the project
Interest in the value of continuing professional development (CPD) for teachers in Higher Education
(HE) has increased in response to a changing landscape motivated in part by policy developments1
and paralleled by changes to fee structures which position students as ‘consumers’ of HE (Clark et al.
2002). The student voice gathers power through instruments such as the NSS and the PTES and HE is
publicly accountable and indeed scrutinised for teaching quality (Stoakes, 2013) Teaching and
learning, traditionally the poor relations of research, are now commonly used by policy makers and
students alike in making distinctions and decisions between HE providers.
These changes have been imposed on an increasingly complex HE sector, with respect to the
diversity of providers, and differing policy levers acting on provision across the four UK nations. In
England in particular, successive government initiatives and policies have led to a clear agenda
emerging with respect to CPD for those leading and supporting teaching and learning (Spowart et al,
in press). Whilst training for new lecturers is commonplace, following the Browne Review (2010),
the spotlight2 has also been placed on CPD for more established staff (HEA, 2012).
CPD in teaching is manifest in a number of guises including (but not exclusively) the Postgraduate
Certificate in Teaching and Learning in the UK (LTHE/ PGCAP or equivalent), short training courses;
in-situ training; consulting, peer review and mentoring; student assessment of teaching; and
intensive staff development (Prebble et al. 2004). Such provision has flourished since the White
Paper: The Future of Higher Education (DfES, 2003) - although there is a much longer history in some
institutions - and has been supported by the ongoing uptake of the UKPSF (Turner et al., 2013; HEA,
2012).
There have been a number of seminal studies which have investigated this area (Gosling, 2008; Ling,
2009; Turner et al., 2013), as well as meta reviews of the literature (Chalmers et al., 2012; Parsons et
1 Including the DfeS (2003) English Education White Paper, HEFCEs Strategic Plan (2003) and the establishment of the of the HEA in 2004, the role of the QAA in monitoring teaching quality (HEA/QAA 2004), the launch of the UKPSF (2006), the BIS White Paper ‘Students at the Heart of the System’ (BIS 2011) and the requirements of HESA (2012) to return teaching qualifications of university staff
2 But not necessarily all HE providers
10/222
al., 2012). These studies found widespread consensus about the value of CPD in teaching but
identified issues with how impact is measured, with respect to content and methodology.
In terms of method, to date, much CPD evaluation relies upon a snapshot which takes place in situ,
post intervention and seeks measurement of satisfaction, rather than of those changes to thinking
and practice which develop over time (Bamber, 2013; Rust, 1998; Sword, 2011). These studies have
provided a useful benchmark in terms of the direction future work should take. Greater
consideration needs to be placed on measuring the breadth of impacts from CPD activities, looking
beyond immediate satisfaction, to how CPD changes practice in the longer term and also shapes
student learning.
Bamber (2013) calls for a greater focus on ‘evidencing value’ which will encourage a move away
from quantitative measures of ‘satisfaction’ with respect to whether intended outcomes have been
achieved, towards more reflexive approaches that consider soft and hard outcomes. Measuring
these so-called ‘soft’ impacts, such as increased confidence, thinking differently, and a willingness to
change practice, all of which benefit student learning, are challenging and require greater attention
to be placed upon the process and the outcomes of CPD (Bamber, 2013). The use of qualitative data
is advocated, as such information when viewed alongside the standard quantitative measures, can
develop a more nuanced understanding about commonalities and differences across different
approaches and contexts for CPD in teaching and learning (Chalmers, 2008; Shavelson, 2010). This
would also allow the scope of future CPD evaluation to look beyond individuals own practice to
impacts they have on their students, their colleagues and institutional ways of working (Bamber,
2013).
It is against this backdrop the current project was funded. The HEA called for proposals around the
area of ‘evaluating teaching development in HE: towards an impact assessment’, outlining the
specific aim of ‘creating state of the art knowledge and understanding relating to how the impact of
CPD schemes on the student learning experience can be measured’. In response to this aim the
project team divided the project into three work packages (WP) to address the aims of the tender:
Work package 1 (WP1): Update the literature review established in the Parsons et al. (2012)
report entitled ‘the impact of teaching development programmes in higher education’.
11/222
Work package 2 (WP2): Invite institutions operating CPD schemes to provide evidence of
current practice in measuring the impact and effectiveness of CPD provision.
Work Package 3 (WP3): Design and test a tool for the assessment of the effectiveness of
CPD.
Below we report on work undertaken and outcomes of each WP sequentially. Detailed accounts are
provided of each WP, which complements the overview of the toolkit produced by the project team
which focuses solely on the outcomes of WP3.
12/222
Work Package 1 (WP 1) - Literature review
1.1 Introduction
Largely established in the 1990s, academic development for teachers in UK Higher Education is still a
relatively young field which is under regular review and revision (Turner et al., 2013). With recent
changes to funding and priorities in the UK HE sector, the impact or effectiveness of CPD
programmes and activities is increasingly under scrutiny as researchers and stakeholders attempt to
understand how such work influences teaching, learning and the broader student experience.
This literature review aims to update the Parsons et al. review commissioned in 2012 by the HEA in
order to inform the Evaluating Teaching Development in HE: Towards Impact Assessment project, in
particular the development of a framework to evaluate the effectiveness of CPD. The review focuses
on research into the impact of CPD in teaching and learning published between 2012 and 2015. We
have broadened the scope and orientation of the original review in order to take account of a wider
range of activities that feature in CPD and to include literature that critically engages with the impact
discourse (e.g. Di Napoli, 2014). We also consider prominent themes in the research on impact of
CPD in relation to work on excellence in teaching in HE (e.g. Gunn and Fisk, 2013).
There are several challenges in assessing the body of research on the impact of CPD. Firstly, the
literature on CPD and impact encompasses a range of activities. For example, De Vries et al. (2013),
define CPD quite broadly as 'updating, reflective and collaborative activities' and point to a growing
interest in collaboration as a valuable teacher development activity. This breadth aligns with the
trend in the UK towards more flexible accreditation schemes (in the wake of the revised UKPSF) in
addition to taught postgraduate programmes; an increased focus on mid-career provision; increased
use of activities such as professional conversations; and peer networks, to name but a few. We
have tried to stay alert, in this review, to research on activities which are part of this broadening of
provision and, to this end, in addition to more conventional CPD work (such as PGCerts, short
courses, workshops) we have included work that addresses identity construction, reflection as a
form of CPD, narrative study and the impact of peer networks. Additionally, bearing in mind
Saunders (2014), we have tried to acknowledge the diversity of contexts and the significance of this
for individual teachers' implementation of professional development learning.
Secondly, ‘impact’ is a complex, often contested, concept, and there is not a consensus about what
constitutes impact in relation to CPD. What is measured under the rubric of impact varies
considerably with studies focusing variously on satisfaction, student performance, teacher self-
efficacy and reflection, among other things. As with Gunn and Fisk’s (2013) observations about
teaching and teacher excellence, impact is contingent on context (institutional and discipline),
educational values and how it is defined.
Finally, there is a dissonance in the discourses surrounding CPD and in order to represent the range
of attitudes towards assessing impact, we have included publications that challenge a narrow,
instrumentalist approach to impact measurement and that call, instead, for a more holistic, creative
attempt to discuss and determine the ‘impact’ of academic development. There is a growing body of
work on this theme and it is important to acknowledge and respond to it in a project that relies on
the participation and support of colleagues in the field.
1.2 Building on Parsons
The 2012 Parsons report pointed to a number of strengths and weaknesses in evidence for the
impact of professional development. The diversity, international base and applied nature of studies
were noted as strengths. A preponderance of small-scale studies and ‘snapshots’ was considered a
weakness. Recent work includes such studies but we also found a number of larger-scale evaluation
studies and several that addressed factors affecting transfer to practice and the impact of
professional development over time. Reliance on self-report as evidence was and still is a concern; it
perhaps accounts for the frequent focus on teacher and student perceptions. Lack of comparability
between studies is still an issue. This is linked with the diversity of frameworks, methods and
measures being used and also with differences in context. Parsons et al. (2012) also note the lack of
baseline data from which to measure teacher or student gains and point to the need for common
tools and frameworks to capture such data and to aid comparability.
We found new work in all of the areas considered by Parsons et al. (2012). In defining the scope for
the review and grouping the literature reviewed, we used similar categories to theirs (1-6 in the list
below) with three additions (7-9):
15/222
The review is organised along the following themes:
1. Impact on teachers’ attitudes, knowledge and skills
2. Impact on teachers’ behaviour and practice
3. Effects of disciplinary or generic programme focus
4. Compulsory vs non-compulsory
5. Impact on student learning
6. Other emergent themes:
a. Motivation
b. Teacher experience
c. Online CPD
d. Social networks/communities of practice
e. Location
f. Time
7. Research into frameworks for evaluating CPD;
8. Related, relevant research, concerned with issues beyond impact, and
9. Papers which offer a critique of impact measurement
Additionally, following comment on the first draft of this review, we have considered potential
synergies between the research on CPD impact and recent work on teaching/teacher excellence and
career progression.
The main aims for this report are to
- update, summarise and analyse literature on the impact of CPD in teaching published since the
2012 Parsons review.
- identify gaps in this body of work
- highlight references to the use of frameworks for evaluating impact
- consider how the findings of the review can be used to inform the project’s evaluation
framework and other outputs.
In this update we are looking particularly for
16/222
- different ways in which impact has been described or defined
- perspectives from which impact has been assessed (e.g. teacher, student, course)
- relationships between impact and groups or networks
- an awareness of impact and contexts (disciplinary, institutional, national, etc.)
- frameworks that offer both theoretical conceptualisations and practical approaches to the
evaluation of CPD.
-
1.3 Method
This review was drafted as a working document, the first purpose of which was to inform the next
stage of the project. It has been further developed in the light of discussion with the project team
and HEA representative.
The methodology for the review arose from the need to synthesise research in diverse educational
contexts, with varying aims, approaches, frameworks and definitions of what constituted
professional development and impact.
Table 1 summarises the stages of the review. Two researchers conducted a search of large education
databases, followed by a manual search of selected journals, a further search using Google Scholar
and a review of more recent work by a small selection of the key writers identified by Parsons et al.
(2012). This process generated over 800 potential sources, and based on initial inclusion criteria
(concerned with the impact or effects of teacher professional development), a review of abstracts
resulted in the elimination of more than three quarters of these documents.
The resulting list of 187 texts was examined with reference to their aims, methods and findings, and
each was tagged using one or more of the tags listed in Table 2. This process led to further
elimination. The 87 remaining texts were then coded and assigned to categories corresponding to
Parsons’ 6 key areas, with the three additional themes listed above. A document could be assigned
to more than one of these. This process again resulted in a reduction, so that the final group of
about 40 sources forms the basis of the findings outlined in the following sections. Additionally, as
we encountered papers of relevance to the broad aims of the project, but outside the immediate
17/222
scope of ‘impact’, we included these and they are considered in sections 8 and 9. Finally, following
consultation on the draft literature review, we have considered our findings in light of recent work
on excellence in teaching and learning and we have included observations on the potential links with
this research.
Table 1.1: Stages of the literature review
Activity Sources Details NotesSearch 1 Major
databasesBritish Education Index, ERIC, Australian Education Index
Search terms based on Stes et al. (2012) and De Rijdt et al. (2013) with addition of terms related to learning technologies.Publication dates 2012-2015
Search 2 Manual journal search
Educational Research and Evaluation, Educational Research Review, Innovations in Education and Teaching International, International Journal of Academic Development, Higher Education Research & Development, Teaching and Teacher Education,Studies in Higher Education, Teaching in Higher Education
Review of titles, reference to abstracts in all volumes published between 2012-2015
Search 3 Personal home pages
Major contributors identified in Parsons
Aim to find any further relevant research or on-going projects
Search 4 Google scholar Further searches Simplified set of search terms
18/222
Publications 2012-2015Inclusion 1 Closer look at
the nature of the study and its relevance to evaluating CPD.
Review of titles and abstracts Aim to exclude documents that are clearly outside the scope of the review
Create single list
Lists of documents generated by Searches 1-4
Researchers’ lists combined and duplicates removed
“Long list” of 188 documents entered in Mendeley
Inclusion 2 Documents in ‘long list’
Review and tagging of long list texts based on abstract and reference to methods and findings; Additional stage: further papers excluded following reading of full texts
“Short list” of 87
Inclusion 3 “Short list” documents
Allocation of full texts to 6 categories representing Parsons’ areas of interest plus 3 additions; further papers excluded
38 documents considered in this review
Table 1.2: Tags used to code the literature
E Empirical I Student impact
C Conceptual L Long-term study
S Schools B Big study
H Higher Education O Other
F Framework U Unsure
D Discipline-based R Reject (Exclude)
1.4 Areas of Impact
1.4.1 Impact on teachers’ attitudes, knowledge and skills
A core area of investigation in the impact literature is the extent to which CPD influences teachers’
awareness and understanding. Evidence of impact in these studies can be found in
attitudes, beliefs and intentions;
self-efficacy;
teachers’ conceptions of learning; and
skill development.
19/222
The extent to which HE teachers gain technical and conceptual skills that enable them to embed
learning technologies into their teaching is measured by Rienties et al. (2013) in a Europe-wide study
in which the CPD was delivered entirely online. Researchers used pre- and post- tests to attempt to
measure impact of an online teacher training programme and to study whether teachers altered
their beliefs and intentions towards student-centred learning and the use of learning technologies.
The framework that underpinned this work (the Technological Pedagogical Content Knowledge or
TPACK model) is addressed further in section 1.4.6 below.
Lau and Yuen (2013) address questions about teachers' perceptions of learning technologies and the
factors that influence their perceptions. Data (self-report) are from post training workshop
questionnaires from a sample of 100 in-service mathematics teachers in Hong Kong secondary
schools. 90 completed all 5 questionnaires. Question areas included both perceptions of and actual
use of technologies in teaching. The questionnaire also asked respondents to give tutor and session
evaluations. Age, gender and prior experience were also included and the result suggested that
these factors did influence the impact of training. Training was found to have a positive influence on
teachers' perceived efficacy with technologies. Impact on beliefs about the pedagogical usefulness of
technologies was less evident.
Enhancement of teacher self-efficacy is a recurrent theme in the research on CPD. Lee, Cawthon and
Dawson (2013) explore the potential relationship between teacher self-efficacy and pedagogical
conceptual change. Teacher self-efficacy was not found to be a predictor of conceptual change.
Similarly, Mowbray and Perry (2015) report improved teacher self-efficacy following a CPD
intervention, and, in contrast to the previous study, they identified a ‘significant increase’ in student
attainment, which they aligned with the increase in self-efficacy. This is, however, a small scale,
relatively short (6-weeks) study (see section 1.4.5 for more details.). In a similar vein, Lumpe et al.
(2012), reporting on a large scale study of intense and sustained CPD for US science schoolteachers,
found that the teachers’ self-efficacy improved significantly and that this was positively correlated to
an improvement in students’ standardized science test scores. The authors conclude that ‘self-
efficacy and professional development hours were positive predictors of student achievement’. The
scale is large in this research, with a sample size of about 500 teachers and between 1300 and 8000
students, varying according to study strand.
20/222
McGee et al. (2013) adapted Guskey’s (2012) framework to investigate how teachers’ perceptions of
CPD and changes in self-efficacy affect transfer and impact. Centred around the implementation of a
new, standards-based mathematics curriculum, the CPD intervention comprised an intensive
summer course followed by a year-long programme, similar to Lumpe et al. (2012). Researchers
state that self-efficacy increased and that teachers’ orientations to CPD at the outset influence its
impact. However, neither of these findings is quantified.
1.4.2 Evidence of impact on teachers’ behaviour and practice
Impact on teachers’ behaviour and practice is an area that has motivated significant research
particularly in terms of student perception of teachers’ practice and ways in which teachers pursue
their own pedagogic development in the period following CPD.
A mixed methods study by Stes et al. (2013a) suggests that students perceive no impact on teachers’
teaching behaviours following engagement with CPD. However Willett et al. (2014) offer evidence of
improvement in teachers’ pedagogical practice as determined by evaluating assignment prompts
and student texts as part of a 3-year mixed methods study. (This study is discussed further in section
1.4.5.)
Rienties and Kinchin (2014) demonstrate that teachers’ social behaviour with professional networks,
that is, the extent to which they engage with and learn from their peers in relation to teaching and
learning, is enhanced by CPD. (See section 1.4.6 for a fuller discussion of this work in light of
communities of practice and CPD impact.)
Brown and Inglis (2013) attempt to identify factors that enable or hinder teachers’ application of
professional development learning in their practice, and the sustainability of their learning over
time. One enabling factor was a continuing dialogue, which might support the findings of research
on teachers’ social behaviours and learning from peers (Rienties and Kinchin, 2014; Stewart, 2014;
De Rijdt et al., 2013). Other factors which encouraged the embedding of CPD learning in day-to-day
practice were, according to Brown and Inglis (2013): seeing resulting change in the students;
21/222
supportive management and leadership; opportunities to discuss and share with peers; and time to
reflect.
Dyment and O’Connell (2014) suggest that CPD programmes ‘model’ good practice, which
participants transfer to their own teaching. It is, however, a small interview study (8 interviewees)
with a very specific focus on the use of learning journals. Similarly, Han and Finkelstein (2013)
explored the relationship between teachers’ professional development in the use of clicker
technology and their adoption of this in formative and summative assessment (see section 1.4.5).
Similarly, Armour and Makopoulou (2012), evaluating a national teacher development programme,
suggest that the programme would have been more effective had it enabled teachers to follow up
particular interests over time and offered opportunities for on-going dialogue.
In a different type of transfer, Trigwell et al. (2012) examining 10 years of data, show that academics
who completed a CPD programme were more likely to be awarded teaching grants from the
university. Similarly, staff who completed the programme were more likely to be recipients of
teaching awards. They also reported a ‘small but significant difference’ in students’ satisfaction with
their course when taught by staff after they had completed the CPD programme. Similarly, the
authors found that students in faculties with a higher proportion of staff who had completed the
CPD reported higher satisfaction rates with their courses.
1.4.3 Effects of disciplinary or generic programme focus
Possible differences in the effectiveness of CPD, depending on whether it had a disciplinary or
generic focus, were considered by Parsons et al. (2012). As in that report, we found little comparison
of disciplinary and ‘generic’ programmes in relation to impact. Throughout this review, there are
studies that look exclusively at a subject or group of cognate disciplines (often mathematics or
sciences), particularly in relation to CPD in schools. However, they do not make claims for the
efficacy of such programmes based on their disciplinary orientation.
22/222
Amundsen and Wilson (2012) address the distinction between disciplinary and generic CPD from a
conceptual perspective in their 6-point framework of educational development; they draw on
theoretical rather than empirical work (See section 7 for a fuller account of this research).
Rienties et al. (2013) find that discipline has an impact upon the retention of participants in an online
CPD course. Additionally, Skelton (2013), considering impact in relation to identity construction of
academic teachers, reminds us that in between ‘disciplinary’ and ‘generic’ is ‘interdisciplinary’ CPD in
which the deliberate combining and interaction of academics from different disciplines plays a role
in the manner in which they can challenge and learn from each other. He argues for the retention of
interdisciplinary CPD and suggests that there are particular affordances of ‘critical interdisciplinarity’,
particularly in terms of the personal and professional changes experienced by the teachers
themselves. (See section 10)
1.4.4 Compulsory vs non-compulsory
As in the previous category, we found scant research that explicitly investigates this comparison,
although there was reference, often in passing, as to whether participants were undertaking CPD
voluntarily or as a compulsory requirement. There is some evidence that staff on temporary
contracts adopted a ‘“defensive” faculty development’ orientation: they felt compelled to participate
in voluntary CPD in order to enhance their chances for contract renewal (Rutz et al. 2012).
Rutz et al. (2012) also found that staff on temporary contracts were less likely to experiment in the
classroom with their learning from CPD for fear of lowering student satisfaction scores; whereas
permanent staff were more willing to experiment with new tasks and techniques even though they
attended fewer CPD events. Of particular interest here is the breakdown along the lines of
temporary and permanent. Other studies tend to distinguish between novice and experienced
teachers and Rutz et al. (2012) are framing the comparison in terms of job security. This paper was
the only one we encountered that explicitly considered job security in relation to engagement with
and impact of CPD, and it would appear to be an area that warrants further attention.
23/222
Trigwell et al. (2012) observe that, over a 10 year period, compulsory CPD programmes had an
impact on teachers’ engagement with the scholarship of teaching’ when compared with those who
had not participated in the CPD.
1.4.5 Student impact/impact on student learning
In relation to assessing impact on students (2012), we have not identified any striking advance on
work reported by Parsons et al. Writers continue to highlight the difficulties of quantifying the
impact of teacher CPD upon student learning, because causality of this nature is hard to isolate
amongst the complex processes of student learning.
Nonetheless, there are studies that make claims about the impact of teacher CPD upon student
behaviour and learning. Furthermore, although self-report is still a major source of data, a wider
range of evidence has been sought in a number of mixed methods studies, such as Antoniou and
Kyriakides (2013). Control groups were also used in two studies (see below), which might lend
weight to their findings.
Not surprisingly, research in a single discipline or around a single student skill or attribute brings
student learning and attainment into focus. Examples include evaluation of teacher professional
development in relation to writing development (e.g. Willett et al., 2014), critical thinking (e.g. Shim
and Walczak, 2012), mathematics teaching and learning (Antoniou and Kyriakides, 2013) and the use
of personal response systems (Han and Finkelstein, 2013).
Trigwell et al. (2012) argue that measuring the impact of CPD programmes on student learning is
difficult and complex and that most studies that have attempted to measure these have found only
small changes. (Much of what has gone before attempts to measure shifts between deep and
surface learning.) The authors found a ‘small but significant’ indication that students’ overall
satisfaction with their course increases when taught by a teacher who has completed the CPD (as
compared with being taught by the same teacher prior to the CPD). Correspondingly, they found
that, in faculties where ‘higher proportions’ of academics have completed the CPD under
24/222
consideration in their study, students report being more satisfied with their degree courses than
students from faculties with lower proportions of teachers engaging in the CPD.
Mowbray and Perry (2015) report on a small scale, mixed methods study with 11 lecturers, which
demonstrates that a particular CPD course in improving lecturing skills has had direct impact on
student learning. The CPD took place over 6 weeks and in assessing its impact, researchers used pre-
and post-intervention questionnaires, interviews, and exam performance of students from 2
cohorts. Results showed an improvement in teachers’ self-efficacy in relation to lecturing skills.
Findings also show a ‘significant increase’ in student attainment for those students whose lecturers
participated in the CPD.
Student learning is a rather broad concept and is often associated with performance on exams or
overall achievement at the end of a course of study. Stes et al. (2013b) look at the impact of a 1-year
CPD programme for new teachers on the students’ approaches to studying, and they find little
effect. Similarly, Stes et al. (2012) reported limited impact upon students’ learning outcomes
(measured using a slightly modified version of Entwistle’s Experience of Teaching and Learning
Questionnaire, ETLQ) amongst students of teachers who had undertaken a year-long CPD course.
However, both of these studies are premised upon a deep vs surface conceptualisation of student
learning which in itself is contested and rather narrow.
Pehmer et al. (2015), in the context of STEM disciplines, claim to have proved the effectiveness of a
CPD programme and shown that students benefit. The study involved 135 student and teacher
participants. Teachers participated in a video-based programme on classroom dialogue and the
research aimed to investigate the impact of this on students' perceptions of their higher order
learning (broken down into ‘situational learning processes’ and ‘cognitive elaboration strategies’).
Likert scale statements were used with students to assess their perceptions of both their own ability
(‘self-concept’) in STEM subjects and these two higher order learning elements. Again, the focus is
on perceptions, with evidence from self-report but there is a control group of 90 in which teachers
followed a similar professional development programme but without video and this may make the
findings more persuasive.
25/222
Antoniou and Kyriakides (2013), in the context of mathematics teaching in Cypriot primary schools,
measure both 'teacher skills' and student achievement before and after a professional development
programme. They followed the study up one year later to investigate sustainability. There are 130
teacher participants. The 'Dynamic Integrated Approach' (DIA) to professional development is used
both as the basis for the CPD course and as an evaluation framework. This is a complex five-stage
model, which sets out teacher development as a hierarchy. The authors claim an association
between teachers operating at the higher DIA levels and higher student attainment. The 'before'
evaluation consisted of observation, teacher questionnaire, student tests, student questionnaire.
'After' evaluation used the same methods and measures plus a feedback meeting with teachers.
Students of teachers in the higher DIA levels made greater gains than those of teachers assessed as
in the lower levels. Students of teachers in the DIA group made statistically significant gains while
those of a control group did not. The follow-up revealed no further improvement and no decline.
In a study over 4 semesters with 74 academics and over 5000 students, Han and Finkelstein (2013)
found evidence that CPD for teachers on using clicker assessment and feedback (CAF) tools
influenced student perceptions of the technology.
Rutz et al. (2012) identify improvement in student critical thinking as evidenced in written
assignments. This mixed methods study investigates the impact of CPD (mainly unaccredited) at two
different US HEIs over 3 years. Willett et al. (2014) is a detailed paper on the same study and
advances a more tentative conclusion that student learning (clearly defined here with reference to
writing, quantitative reasoning and critical thinking) is enhanced. Shim and Walczak (2012) also
suggest a relationship between teacher practices and student development of critical thinking skills.
Additionally, Willett et al. (2014) focus on the development of a methodology for assessing impact
using student work and teacher assignments. The Haswell paired-comparison rubric was used to
analyse student texts and staff assessment prompts before and after CPD events to establish
evidence of impact. The findings indicate clear change/enhancement in academics’ practice and, as
suggested above, some evidence of impact on student learning, although the latter findings are less
strong.
26/222
Finally, as suggested in section 1 above, Lumpe et al. (2012) found that the teachers’ self-efficacy
was positively correlated to an improvement in students’ standardized science test scores. The
authors conclude that ‘self-efficacy and professional development hours were positive predictors of
student achievement’ p. 6.
1.4.6 Other references to impact not covered by the above
The literature also revealed a number of additional indicators that align to impact and transfer. Key
among them are motivation, teacher experience, place and context for CPD, networks, affordances
of online CPD in relation to impact, and time or distance from the course or intervention.
Motivation – De Rijdt et al. (2013) identify motivation as a key variable in the success of
transfer of CPD to the workplace. Han (2012), investigating ‘teacher-driven’ CPD also views
motivation as an important variable. Motivation and orientation or attitude towards CPD is a
recurrent theme the literature. (See Peters et al. (2012) at the end of this section for a
related discussion of ‘enjoyment’ and impact.)
Teacher experience - Stewart (2014) offers a tentative suggestion, based on a narrative
study, that CPD (in this case a PGCert) had more impact for teachers who had some prior
teaching experience than those who did not (This aligns with findings, similarly tentative,
from De Rijdt et al. (2013)). However, Stewart suggests that further research is needed to
ascertain whether a PGCert has a more sustained impact on participants who had some prior
teaching experience than those who are at the start of their career.
Online CPD – Rienties et al. (2013) - While this paper is not explicitly about online CPD and
its effectiveness, the authors imply that there are certain affordances in terms of online
delivery that influence impact. As suggested above, Rienties et al. found that discipline and
institutional culture appeared to significantly influence whether participants successfully
completed online CPD modules.
Creation of social networks/ communities of practice - Rientes and Kinchin (2014) identify
the impact of CPD programmes on HE teachers’ subsequent development of social network
structures and social learning relationships with colleagues. They suggest that socially co-
constructing and sharing knowledge after CPD has finished is evidence of an
undervalued/under-reported type of impact which sees participants entering into
communities of practice around teaching and learning.
27/222
Rientes and Kinchin (2014) suggest that their findings indicate that research into the impact
of CPD should be broadened beyond a consideration of ‘formal programme boundaries’ (p.
123). This research builds on Moolenaar et al. (2012) who argue that cohesive teacher
networks improved self-efficacy and had an indirect impact on student (in this case school
children’s) achievement. Furthermore, knowledge ‘spillover’ and a changing sense of
institutional culture are also identified as a significant finding by Rutz et al. (2012).
Skelton (2013) and Watson (2014) also suggest that community building as a result of CPD
should feature in a consideration of impact. Related to this, Belvis et al. (2013) found that
transfer depended on more than one person from a work context (in this case, schools)
participating in the CPD and they refer to a participant’s ‘sensitivity to impart’ their learning
to others as having an impact on transfer (although this is not described in more depth.)
Viewing the impact of CPD in relation to social and professional networks and communities
of practice introduces an important and complex dimension to the consideration of
effectiveness. As Roxa et al. (2011) suggest, institutional cultures and their dynamics and
complexities must be taken into account when considering the management of change,
especially in relation to academic development. Seemingly, such awareness of organisational
culture is important when analysing and (as Rienties and Kinchin suggest) extending the
impact of CPD.
The location of the CPD in relation to participants’ working environment may have an
impact. De Rijdt et al. (2013) suggest that ‘on the job’ CPD has a greater impact than ‘off the
job’ but indicate that this finding needs further research. Related to this is the ‘learning
climate’ and the extent to which the institutional approach to CPD is framed as
appreciative/developmental or remedial.
Similarly, Han (2012), analysing the impact of pre-school teachers’ professional development
on their ‘instructional strategy development’, addresses transfer in terms of location and
refers to 'on the job' CPD as being preferable to 'shot in the arm' (off the job) training. Han
distinguishes between ‘knowledge for practice’ (off the job) often taught by external
specialists through workshops or similar events, and ‘knowledge in practice’ (on the job)
which takes place within teaching.
The study looked at different features of CPD including ‘teacher-driven’ activities (in which
teachers determined the topics of CPD events and ‘job-embedded’ professional
development (in which the CPD took place as part of teachers’ work). Following analysis of
28/222
the pre and post intervention questionnaires, Han concluded that ‘teacher driven’ and ‘job
embedded’ are significant attributes in relation to impact on teacher behaviour. As above,
these findings address the issue of context for the delivery and engagement with CPD and
further work is warranted to understand more fully ways in which these influence the
impact of CPD.
Time – Many of the studies are evaluating impact soon after CPD has been offered and a
number of researchers indicate that longitudinal research is needed to better understand
impact over a sustained period. Recent exceptions are Trigwell et al. (2012) who look at the
impact of a CPD programme using data collected over a 10 year period and Stewart (2014)
who investigates the impact on teachers after 5–plus years.
Finally, Peters et al. (2012) bring together a number of the above factors in their research
into online training and the impact of participant satisfaction and motivation to transfer.
Satisfaction, and within this ‘enjoyment of learning’ are foci of the research. Citing Axtell et
al. (1997), Peters et al. remind us that ‘motivation to transfer’ predicts the speed and
likelihood of transfer. They also find that opportunities for interaction are linked to
enjoyment and learning outcomes, and, indirectly, to the motivation to transfer. Their
findings suggest that enjoyment as a feature of learning is an important dimension of
motivation to transfer, which in turn, predicts the likelihood of transfer.
1.5 Research in the broad area of CPD impact that addresses the construction and use of frameworks
There are a variety of frameworks designed to describe or evaluate CPD programmes and activity.
This range is due, in part, to the different value systems underpinning the work which, in turn,
influences which characteristics activities, points of view, and indicators of impact are foregrounded
and described by the different frameworks.
The development and application of a framework to evaluate CPD is described in depth in Chalmers
and Gardiner (2015) who argues that the impact of CPD programmes in HE is under-researched with
a few exceptions. Working within an Australian context, where, increasingly, education is scrutinized
for quality, value for money and range of participation, Chalmers and Gardiner (2015) argue that in
the absence of a rigorous and relevant evaluation tool, that CPD programmes will continued to be
assessed with blunt and limited instruments such as participation satisfaction surveys which do not
29/222
report on richer, contextualised impacts of the programmes. Chalmers et al.’s (2012) own
framework is designed to be used in a variety of contexts and is based upon 4 principles: relevance,
rigour, context and reliability. This is a relatively complex framework with a matrix of indicators
focusing on both programme and institutional contexts. It advocates the collection of data along a
number of specific indicators which address processes, outputs and outcomes. The framework(s)
and accompanying account are well-documented; however, potential criticisms are that it describes
CPD activity within a context without sufficient emphasis on collecting evidence to demonstrate
impact. Additionally, the student voice/learning perspective is not considered as prominently as it
might be.
Building on a ‘conceptual’ review of the literature, and with particular reference to Stes et al. (2010),
Amundsen and Wilson (2012) devise a 6-point framework intended to enable new insights into
practice design and to aid evaluation of effectiveness of CPD. Cautioning against being overly
‘narrow’ in questions about impact and effectiveness of CPD (and they argue that many previous
empirical studies are too narrow), Amundsen and Wilson set out a framework that they suggest
offers a broader account of educational development and is better suited to capture its complexity.
Their six ‘foci of practice’ around which the framework is based include skill, method, reflection,
disciplinary, institutional and action research/inquiry.
Farley and Murphy (2013) report on the early stages of an Australian project to develop an
evaluation framework for mobile learning. It would be worth following the progress of this work
since its ultimate goal is an evaluation toolkit that comprises the kinds of elements the current
project might offer: framework, guidelines, resources, examples and a ‘maturity model’. Rienties et
al. (2013) use the TPACK model as a means of gathering data but also as a conceptual framing for
their study. Various discussions of TPACK framing appear in North American schools research that
addresses CPD and impact, particularly in relation to bolstering content knowledge.
Saunders (2014) reports on the evaluation of a programme based on "instructional intelligence",
which is about developing expert behaviour (described in some detail). The conceptual framework is
the Concerns Based Adoption Model (CBAM), a complex model designed to assess teachers'
educational change. This use of CBAM includes a ‘stages of concern’ questionnaire (SoCQ) and levels
of use (LoU) interviews.
30/222
As suggested above, Trigwell et al. (2012), report on four studies into the impact of a year-long CPD
programme at a research institution in Australia, which draw on data collected over a decade.
Building on prior work (e.g. Guskey (2002) and Kirkpatrick (1998)), the researchers developed a
framework to assess satisfaction of CPD participants, the impact upon participants’ practice and
impact on student learners. Like Chalmers and Gardiner (2015), Trigwell et al. (2012) argue for the
affordances of taking into account multiple indicators of impact; in their case they looked at both
staff attainment and student satisfaction. Trigwell et al. suggest that the use of multiple indicators
can more easily identify and demonstrate enhanced impact/significance than studies drawing on
single indicators.
As an alternative to models designed to measure impact quantitatively, Bozalek et al. (2014) use the
Tronto ‘political ethics of care’ framework to evaluate CPD in a South African university. The
framework is a reflective tool which addresses 5 characteristics: attentiveness, responsibility,
competence, responsiveness and trust. This holistic, reflexive approach foregrounds the lived
experiences of teachers and the practices of the team; members of the team reflect on their own
practice in a systematic manner. This paper discusses the application of the framework in detail, and
describes some of the reflections that emerged from working through it. From an exclusively
‘impact’ perspective, this does not yield much in the way of metrics or fixed findings; however, from
a conceptual perspective, the valuing of the individual’s and cohort’s reflexivity and development is
signalled, as is the importance of values. This work offers more of an orientation towards evaluation
of CPD rather than a model for measuring or quantifying impact
Belvis et al. (2013) evaluated a CPD intervention for mathematics teachers (primary and secondary)
by drawing upon the Holistic Model (Pineda, 2002) which addresses dimensions of satisfaction,
learning, pedagogical appropriateness, transfer and impact. The data collection tools of the model
include surveys, reports and interviews with participants, colleagues and managers. The framework
is set out fully in the paper. The programme, with 284 participants from primary and secondary
education, asked teachers to systematically reflect on their teaching practice and to share their
reflections with colleagues in small groups. In this approach, teachers started with observations and
‘ended with strategic planning for performance improvement’. The ultimate aim was to enhance
classroom teaching including the use of ICT.
31/222
The authors conclude that while the ‘study shows that the education programme generates high
levels of satisfaction, pedagogical appropriateness and learning, its achievements in effectiveness
are moderate’ and there was ‘little evidence of its impact on student learning’.
Like Chalmers et al. (2012), Fink (2013) suggests that impact is difficult to ascertain despite good
intentions; evaluation of impact rarely goes beyond the measurement of participant reaction or
satisfaction. This paper explores the reasons why measuring impact is important and from whose
perspectives. It also offers a framework for considering impact (or, as it is termed here, 'assessment')
and identifies 4 perspectives (students, academics, senior managers and 'faculty development
activities') from which data might be gathered. Fink (2013) foregrounds the complexity and range of
CPD within any given institution (or sector) and argues that different approaches to collecting
information about impact are required: CPD is not a singular entity and nor should its evaluation be.
Fink describes a protocol for collecting data about the impact of CPD on teaching practices and on
student learning. Each branch of the protocol has 4 parts and the instruments can be used in any
combination in order to reflect which aspect(s) of the framework are being evaluated. The data
collection tools are based on recent research and employ a combination of qualitative and
quantitative methods. The article closes with guidance on useful analytical approaches; these
include creating comparison data, constructing measurable objectives and considering the ‘multiple
meanings of impact. In closing, Fink cites Hines’ (2009) 8 dimensions of quality program assessment:
Systemic: Creates feedback from systemic and continuous assessment.
Goal-directed: Clear program goals guide the assessment.
Measurable objectives: Objectives are designed in ways that enable measurement.
Criteria for success: Standards have been set that define the desired level of goal
achievement.
Assessment methods measure the objectives: The methods are valid measures of the
objectives.
Multiple measures: The assessment uses multiple measures of program quality.
Summative and formative data: The data gathered can serve the purposes of both
program improvement and a determination of end-of-program effectiveness.
32/222
Evidence of a causal relationship: Uses comparative data to establish a causal
relationship between program activities and its impact(s).
(Hines, 2009)
As suggested above, Guskey’s work (2002) still features strongly in framework development. McGee
et al. (2013) critique and expand Guskey’s framework as a means of evaluating the impact of CPD;
they describe in detail questions and data sources devised to gather evidence for each level of the
resulting framework.
Also drawing upon Guskey and foregrounding reflection, Postareff and Nevgi, (2015), report on a
study that addresses the impact on individual teachers of a CPD course and analyses the pathways
that 18 participants took through the course by researching their reflective diaries in an attempt to
understand the complexity and variety in experiences. They identify 5 development pathways and
analyse findings in light of Guskey. They suggest that teachers need opportunities to experiment
with new pedagogic methods in order for the CPD to have impact. They also find that the motivation
of teachers to ‘develop conceptions and understanding of learning’ aids potential transfer.
Finally, Willett et al. (2014) offer a nuanced, well-illustrated discussion about the affordances and
deficits of their approach to evaluating CPD and the implications for evaluating the impact of CPD in
other HEIs. In particular, they address the challenges of transferring evaluation techniques and
frameworks developed for a particular institution into a new setting.
1.6 Research that goes beyond impact of CPD but which may be relevant more broadly
Recent research offers many observations about academic development and CPD that sits outside of
a consideration of impact, but is potentially relevant for the current study, nonetheless. We also
include here suggestions in recent literature for future research.
1.6.1 Reconceptualising academic development
Boud and Brew (2013) argue for a reconceptualization of academic development and suggest that it
should be viewed more as a social practice with a focus on the extent to which there is a cultural
33/222
shift in the institution as opposed to the development of individual teachers. This gaze might usefully
inform future frameworks. Amundsen and Wilson (2012) also observe this distinction in the
orientation of CPD in their discussion of ‘contextual positioning’.
1.6.2 Impact of networks and brokers
Rienties and Kinchin (2014) suggest that future research should investigate the complex roles of ‘key
brokers inside and outside’ CPD programmes in HEIs who help with ‘knowledge spillover’ and
support the creation of links with colleagues beyond the original CPD programme. There is some
synergy here with De Rijdt et al.’s (2013) tentative findings that novice teachers benefit and show
more transfer of learning when they form collaborations with more experienced teachers. This
thinking is in keeping with work by Roxa et al. (2011) who explore academic development and
change in relation to the power of university cultures.
1.6.3 Narratives
Similarly, Stewart (2014) (building on Stronach, 2010) while recognising the need for large scale
studies into impact, argues that ‘the quest for generalisability’ will necessarily omit ‘significant
personal impacts’. The research, based on analysis of HE teachers' narratives some years after their
participation in a formal CPD programme, is unusual in its focus on long term impact. Significantly,
Stewart (2014) cautions against an over dependence upon the discourse of impact and
measurement, and suggests that a rich understanding of how teachers orient themselves to their
CPD over time (> 5 years) helps illuminate shifts in personal development of teachers, and provides a
‘fuller understanding of teacher change’.
Although Stewart describes this study as moving away from impact evaluation, its consideration of
sense-making, particularly how university teachers make sense of the course as their careers
progress, is highly relevant to a broad view of impact.
1.6.4 Impact of diverse participation groups
Some studies suggest that diversity amongst participants may influence the outcomes of teacher
professional development. Boman (2013) investigates outcomes of teacher development workshops
for graduate teaching assistants in Canada. The mixed methods (self-report, pre- and post-test,
observer coding) enquiry considered self-efficacy, confidence, and teaching behaviours and found
variations between home (Canadian) international participants and between those who had
34/222
previous teaching experience and those who had not. Lau and Yuen (2013), in a survey-based study,
found changes in beliefs about learning technologies were greater in ‘younger’ than ‘more senior’
mathematics teachers.
Herman (2013) suggests that the ratio between teaching and learning development unit staffing and
institutional FTE and student numbers is an indicator of institutional commitment to teaching and
learning. This is questionable but the data is easily obtained so it might be worth considering.
1.6.5 Identity construction, boundary crossing and impact
Postareff and Nevgi (2015) explore identity construction and boundary crossing of CPD participants
looking at four indicators: identification, coordination, reflection and transformation. In this paper,
these dimensions are presented as a feature of successful CPD design and participant orientation;
however, this could be adapted and developed as potential indicators of impact in relation to
teacher development. So, these indicators might be borne in mind when developing future
frameworks. Another implication of the study appears to be that those designing and leading CPD
should be attentive to sociocultural and disciplinary differences of the attendees.
Similarly, Skelton (2013) writes about a two –year Masters course at a research intensive HEI which
takes a critical, reflexive, interdisciplinary approach to CPD. Exploring the implications for
participants through semi-structured interviews, Skelton finds that a key impact is that of
transformation in terms of personal and professional identities with an enhanced sense of self-
confidence in terms of professional practice; stronger theoretical underpinning of teaching and
enhanced self-awareness. A potential negative impact of CPD identified here is the slight suspicion
that a focus on teaching might carry with participants’ peers at a research HEI. With reference to the
recent HEA UKPSF Impact study (Turner et al., 2013), which recommends more discipline-based CPD,
Skelton argues for the affordances and impact of interdisciplinary (as distinguished from disciplinary
or generic) critique and awareness which emerge in this programme.
Van den Bos and Brouwer (2014) take a broadly similar approach to Postareff and Nevgi (2014) by
following ‘novice’ university teacher journeys (albeit over a shorter period – 5 months) during a CPD
programme, using interviews and digital logbooks to gather data. As part of the study, they use
dilemmas (with quantitative and qualitative responses) as a means of evaluating participants’
learning on the course and understanding ‘what’ was learned and ‘how’. They advocate Clarke and
Hollingsworth’s (2002) ‘interconnected model of personal growth’ as a framework for analysing
35/222
research data of this nature and perhaps this would be worth investigating further for the current
project.
Finally, Watson (2014) draws upon social learning theory to analyse a case study of a single teacher;
the context is secondary school teaching and the specific focus is problem solving. The theory, as
described, views learning as involving observation, self-efficacy and ‘reciprocal triadic determinism’,
which is defined here as the relationships between ‘individual thinking and beliefs, the social context
and individual behaviour’. Watson argues that this approach could be a useful evaluation tool, and
aims to contribute to the theorising of professional development.
1.7 Critique of the impact discourse
There is much meta-level analysis and critique of the impact discourse as it relates to measurement
in higher education, particularly academic development, in recent literature.
Stefani (2013) laments the ‘obsession with measuring performance’ across HEIs and especially in
relation to academic development. Nonetheless, she recognises the demand for accountability and
observes that the academic community is increasingly expressing interest in ‘meaningful
frameworks’ for measuring performance. She cites the CADAD 2011 benchmarking statement with
its eight key domains, as areas to consider, and in relation to benchmarking performance and
measuring impact, she asks
‘How do we conceptualise our contribution within the ‘big picture’ of organisational goals
and the overarching purposes of higher education in the twenty-first century?
Would we set our performance measures at the operational level, at the strategic level or
both?
Is it time for a shared, fit-for-purpose narrative for academic development for the twenty-
first century?
Bozalek et al. (2014) argue that many models for measuring impact of CPD, while not explicitly
addressing their underlying values, are nonetheless aligned with a neoliberal approach to evaluation,
focussing on ‘efficiency, measurability and individualism’ (p. 447). As an alternative, they develop a
36/222
‘political ethics of care framework’ based on Tronto (1993) as a means of analysing individuals’
response to professional development (See section 1.5 above).
Similarly, Mockler (2013), critiques the dominant discourse surrounding teacher CPD, particularly the
Australian ‘teacher quality agenda’ which he aligns with a neoliberal framing of education and one
which risks becoming fixated on measurement and regulation which is ultimately damaging to the
teaching profession. Citing Sachs (2003), Mockler distinguishes between CPD that is aligned with
‘managerial professionalism’ and that which is associated with ‘democratic professionalism’. The
latter is determined by teachers, relies on teachers’ professional judgement and is characterised by a
level of trust.
Arguing that conceptualising CPD as ‘identity’ work is a more productive orientation for evaluating it,
Mockler (2013) raises a series of questions and challenges for future work. These would be worth
bearing in mind when developing the framework and will help guard against creating a model which
could be perceived as overly technicist and/or so narrow that it omits the teacher perspective.)
Adopting a broader critical perspective, Di Napoli (2014) explores the tensions inherent in academic
development, arguing that those working in the field often find themselves operating in contexts
that are ‘saturated’ with neoliberal discourses of marketization which are ‘shot through’ with
superficial notions of what ‘good’ learning and teaching are. He reminds us of the complex nature of
academic development and the extent to which it differs according to context and he warns against
unwittingly relinquishing personal and professional values that have traditionally characterised the
profession through internalising a culture of compliance and conformity, rather than that of
challenge. The quest for ‘frameworks, roles and regulations’ to apply and measure activity around
academic development, Di Napoli(2014)suggests, risks eclipsing ‘imagination, reflection and
creativity’. In particular, he argues that it is important to ‘unpack how [the] discourses of care,
support, and criticality meets those of efficiency and accountability’.
This discourse of critique, particularly in relation to frameworks and measurement, is an area not
really explored in Parsons, yet, as Stefani (2013) notes, it is important to acknowledge and address
37/222
this tension and remain alert to it in order to garner and maintain the support of academic
developers. As Jordan (2014) argues, it is important to resist a reductive, ‘instrumental mindset’
when evaluating impact and, instead, to remain alert to a broad range of ways in which the effect of
academic development can be manifest.
1.8 Conclusion
1.8.1 UKPSF, teaching excellence and CPD frameworks
This project is an enquiry into how the effects of CPD can be evaluated over time to help understand
its impact for teachers, students and institutions. Given this, it is useful to consider how the research
on the evaluation of CPD intersects with the broader discourse and policy related to CPD for
teachers, including the UKPSF and recent research on excellence. Professional development,
teaching excellence, career progression and impact of CPD are all areas that are intertwined in
experiences of academics, yet, as Gunn and Fisk (2013) and others have observed, the connections
between them are often insufficiently explored in policy, frameworks and research.
The UKPSF is one such example. This Framework is intended to describe attributes and values
associated with professional practice for those engaged in teaching and supporting learning in UK
HE. The impact of this UKPSF itself is variable across the HE and FE sectors, with senior managers
tending to suggest that it has informed CPD strategies, policy and practice, while subject academics
indicate that the framework has had rather less impact (Turner et al., 2013). The UKPSF does not
prescribe CPD or its intended impact; rather, as Gunn and Fisk (2013) suggest, the UKPSF is largely a
benchmarking tool that does not ‘measure’ or record excellence as people progress in their careers.
Nonetheless, increasingly, CPD courses, processes and events in UK HEIs are mapped against the
UKPSF which is therefore arguably asserting considerable influence in the scope and development of
CPD. Interestingly, however, there is relatively little mention of the UKPSF in the literature reviewed
in this report.
38/222
Furthermore, as Gunn and Fisk (2013) and Turner et al. (2013) argue, it is difficult to align the UKPSF
to excellence at different stages of careers in the way that the Vitae Early Career Research
Development framework enables the mapping of research development. Gunn and Fisk (2013) urge
an alignment between frameworks that articulate excellence at all stages of the academic career,
including those that map research development and those which may map or guide excellence in
teaching. An awareness of the various frameworks, policies and discourses that address the full
spectrum of the academic career will be important for a framework that evaluates the impact of
CPD; in other words, CPD should not be considered in isolation from other elements of the academic
role.
Indeed, the consideration of career progression is implicit in a number of the studies cited above but
not always articulated explicitly in the extent to which it relates to CPD. Throughout, the literature
discussed here is generally addressing CPD as something which enhances teaching and student
learning outcomes. Gunn and Fisk (2013) argue that a significant gap in literature, discourse, policy
and practice is the relationship between models of teaching excellence, CPD frameworks and the
acknowledgement that academics’ careers vary over time. They argue for a robust methodology for
‘analysing the links between teaching excellence and student learning outcomes’ and they suggest
that such approaches should also be cognisant of the various roles and stages of an academic career.
We suggest that the impact of CPD could usefully be included in such methodologies. Likewise,
frameworks which model and guide the evaluation of CPD should also be mindful of career
progression and the ways in which the impact of CPD articulates with teaching excellence.
1.8.2 Gaps in the literature
There are a number of gaps in the literature in relation to studies which evaluate the impact of CPD
in HE teaching. Significant gaps include
Little research that considers the relationship between engagement with the UKPSF and
impact of CPD. As suggested above, other recent research has called for greater integration
between frameworks and discourses which benchmark teaching and research practices and
those that articulate teaching excellence. The same dislocation appears to hold in relation to
CPD and its impact.
There is not a sufficient body of research on the complexities and range of contexts (e.g.
institutional, disciplinary) and the way in which ‘impact’ of CPD might be both determined
39/222
and evaluated in relation to context (Work that might inform further research in this area
includes Roxa et al. (2011) on teaching and learning cultures and Gunn and Fisk, 2013 who
take a critical and nuanced view of the literature on teaching and teacher excellence).
Similarly, there is not a great deal of research on understanding the complexity and
challenges of collecting evidence related to impact of CPD. Cashmore et al. (2013) address
this issue in relation to the amassing of evidence around career progression in UK HE and the
same principles could be applied to identifying and collecting evidence to critically evaluate
the effectiveness of CPD.
There appears to be relatively little work on how students’ directly experience the impact of
CPD undertaken by teachers. Studies that focused on student perspectives and experiences
would be a valuable addition to the literature.
1.8.3 Key points and implications for future work on evaluating CPD
CPD is a wide-ranging term and how it is defined and used has implications for what can be
researched and claimed.
The contexts for CPD – both where and how it is delivered – and the broader institutional
orientation to it, influence its impact.
‘Student learning’ has diverse meanings in the research – ranging from specific evidence of
critical thinking to orientations to deep and surface learning implicit in study behaviours.
Understanding and precisely describing what is meant by student learning is critical in
investigating CPD.
Students’ voices tend not to be heard directly in research on impact.
Assessing the impact of teacher CPD on students can build on existing descriptive
frameworks or established associations between teacher behaviours and student attainment
or perceptions, and even use these to create an analysis framework. Examples include the
use of the DIA (Antoniou and Kyriakides, 2013), and CBAM (Saunders, 2014).
The impact and influence of networks – both online and within institutional contexts – are
worth further investigation.
There is a wide range of frameworks in recent literature – both conceptual and practical and
with different aims and underpinning values – which can potentially inform future
evaluation and research. Established frameworks such as Guskey’s (2002) and Kirkpatrick’s
(1998) are still widely drawn upon. Additionally, a number of studies, as suggested in
section 1.5, have extended these frameworks OR devised alternative ones, many of which
have the potential to help interpret and relate findings across studies.
40/222
Seeking ‘evidence’ is a feature of work in both teaching excellence and the UKPSF, further
supporting the need to link CPD impact evaluation with these.
This framework should relate to the UKPSF and this relationship should be explicitly
described.
Teacher self-efficacy, although disregarded in some research as not demonstrating ‘impact’,
has nonetheless been shown by Lumpe et al. (2012) to be a positive predictor of student
achievement.
Time – although there is much talk about the need for longitudinal studies, these can be
difficult to plan. However, both Trigwell et al. (2012) and Stewart (2014) offer ways of
analysing existing datasets to observe impact over time.
Finally, the use of multiple indicators in analysing impact appears to offer a richer picture of
what is happening as Trigwell et al. (2012) suggest.
1.9 Final thoughts
Impact is a difficult concept to define and measure, and, as with ‘excellence’ there is not consensus
on the meaning of the term (Gunn and Fisk, 2013). For some, impact is about quantifying a change in
student attainment or a change in study behaviours; for others it entails understanding the extent to
which teachers’ reflections of their changing practice over time signal ‘impact’. Another way of
conceptualising impact involves examining teachers’ engagement within peer networks following
CPD.
Additionally, as Fink (2013) observes, an assessment of ‘impact’ can vary according to the standpoint
from which it is measured: the student, teacher, staff development unit, institution, etc. There are
clearly issues of power at play, potentially, when we begin to consider who determines what counts
as ‘impact’ and how it is to be measured and by whom. In other words: Whose impact is it, anyway?
This breadth within the literature lends both complexity and richness to a potential framework for
evaluating CPD. Identifying methods to assess impact from multiple perspectives would be an
important feature of future work.
Finally, as noted by Cashmore et al. (2013) in relation to understanding teaching excellence,
‘evidence’ for understanding impact is complex and challenging to collect. Bearing this in mind, a
41/222
framework for the evaluation of CPD should be alert to this complexity and accommodate, as flexibly
as possible, a range of dimensions (such as time, motivation, experience, point of view, communities
of practice, student learning, etc.) when scoping, exploring and describing impact.
Work Package 2 (WP 2): Invite institutions operating CPD schemes to provide evidence of current practice in measuring the impact and effectiveness of CPD provision.
This WP consisted of two main elements. Firstly, a national audit was conducted to capture data
from HE providers regarding the impact and effectiveness of current CPD provision. We then
identified examples of innovative approaches to the evaluation of CPD activities; these were the
subject of further inquiry that formed the basis of the four case studies presented in section 2..8.
Below we provide an overview of our approach to the development of the data collection
instruments, and the procedures for data collection and analysis.
2.1 National audit
A national audit was identified in the bid as an appropriate mechanism through which we could
gather data regarding teaching-related CPD provision from the varying contexts in which HE may be
delivered. A similar approach was used by Chalmers et al. (2012) to gather data relating to the
evaluation of teaching preparation programmes. Given the timeframe of the project, we felt that
adopting a similar methodology would be an efficient method of garnering insights into the range of
CPD activities and approaches to their evaluation, to inform WP3.
42/222
2.1.1 Themes addressed within the audit
An initial review of research drawn on in developing the proposal (e.g. Cashmore et al., 2013;
Chalmers, 2011; Chalmers et al., 2012; De Rijdt et al., 2012; Guskey, 2002; Kandlbinder & Peseta,
2009, Kreber & Brook, 2001, Parsons et al., 2012 and Turner et al., 2013), underlined the diversity of
CPD provision and highlighted a distinction between CPD provision for those new to university
teaching and the CPD activities with which their established colleagues engage. For early career
lecturers, the majority of CPD is focused around structured teaching preparation programmes
designed to introduce them to the practice of university teaching (Smith, 2011; Parsons et al., 2012).
Established academics more commonly engage with CPD activities such as workshops, conference
attendance, activities related to the scholarship of teaching and learning and, more recently,
schemes to provide recognition of their teaching expertise (Parsons et al., 2012; Spowart et al., in
press). This broad distinction according to career stage highlighted a need to capture data around
both institutions’ formal CPD offer including (e.g. accredited courses) and their informal / non-
accredited CPD offer. From the start of the project we were aware of the need to consider CPD for
teachers at different career points. This was supported by findings from the literature review (WP1),
in which a tendency to overlook relationships between career progression and CPD was noted (see
section 11.1).
Given the overall focus of this project on evaluating the impact of CPD, we were primarily interested
in capturing information regarding the methods used to evaluate provision. We drew on the ideas
of Guskey (2002), Chalmers et al., 2012 and others to ascertain the focus of respondents’ CPD
evaluation activities, for example whether they were interested mainly in participants’ satisfaction or
trying to gain insights into other areas of impact such as student learning, or beliefs around teaching
and learning. Integral to this are the methods used to evaluate provision (e.g. questionnaires,
interviews), the data drawn on as part of the evaluation and the timeframe over which the
evaluation takes place (Bamber, 2013; Rust 1998). We were also interested in ways in which
evaluation data might be used. Recent work by Roni Bamber and colleagues around Evidencing
Value indicates this is an aspect of educational development practice that needs further
consideration. Respondents were therefore asked to identify how the impacts of CPD provision
were disseminated through institutional systems and structures (e.g. policies, conferences, staff
appraisals).
43/222
Much of the existing work around teaching-related CPD is centred on the impact on staff, with the
benefits to students, though implied, rarely evidenced (Parsons et al., 2012). Whilst we were not in
a position to address this gap through the audit, we were interested in the contribution students
made to teaching-related CPD as, through initiatives such as the Student Engagement Partnerships,
we were aware of the increased prominence given to student input to activities such as curriculum
design, staff CPD and policy development (BIS, undated). We felt there was considerable potential
for students to take an active role in CPD and evaluating its impact, and by including this in the audit
we might gain insights regarding the extent of student participation.
Reward and recognition was the final area identified for inclusion in the audit. Reward and
recognition can motivate individuals’ engagement with CPD (Cashmore et al., 2012; Parsons et al.,
2012). Recent studies (e.g. Spowart et al., in press) acknowledge the opportunity to gain
accreditation and recognition for teaching experience as a driver behind established academics’
involvement with CPD activities. More widely, there can be overlap between CPD and some of the
strategies employed to reward an individual’s contribution to teaching and learning, for example
development awards (Turner & Gosling, 2012). Therefore gaining insights into institutional reward
and recognition policies, and possible connections with CPD activities, was seen as essential to
understanding institutional framing of CPD.
2.1.2 Audit audience
The audit targeted those in an educational / staff development role, working within a UK institution
with a remit to provide HE. These staff were likely to have a remit for teaching-related CPD, to be
cognisant of relevant professional bodies (e.g. SEDA / UKPSF) and to be able to provide insights into
evaluation practices and institutional impacts of their work. They were also a readily accessible
sample population due to the presence of a number of national and regional online mailing lists
through which we could disseminate the audit.
2.1.3 Audit development
The audit underwent a rigorous development and piloting procedure, with feedback solicited from a
number of groups including the Project Team, the HEA Project Steering Group and the project’s
44/222
‘critical friend’, Nancy Turner. The audit was revised three times; in each iteration feedback was
carefully considered and responded to.
Ensuring relevance to the diversity of HE providers in our target group had implications for the
length of the questionnaire. The final audit contained 72 questions. To maximise response rates and
minimise completion time we asked mainly closed, multiple choice questions but, where
appropriate, there were options for respondents to provide further detail or to qualify their
responses.
Prior to the final pilot the audit was converted into a digital format using the Survey Monkey online
questionnaire platform which is a widely used and easily accessible survey tool.
The full audit is in Appendix 1. To aid interpretation, it captured contextual data around the
following themes in addition to those areas discussed above:
Demographic information (e.g. institution type)
Format of CPD provision (e.g. modes of delivery, intended outcomes (for students, staff and
the institution), participation (voluntary / compulsory);
Support and guidance for the CPD provision;
Engagement with professional frameworks and bodies (e.g. UKPSF / SEDA);
Respondents were asked to provide this general information about their context; however, the main
focus of the audit was on the evaluation of their accredited (e.g. accredited courses for new
lecturers / recognition frameworks) and non-accredited provision (e.g. action research projects,
conference attendance, peer review etc.). Guidance was offered to respondents to support them in
completing the audit and in identifying what, within their CPD offer, might be relevant to include in
the survey. Respondents were also invited to leave contact details if they were willing to be involved
in the development of case studies of innovative evaluation practices from across the sector.
2.1.4 Dissemination of the audit
The audit was disseminated (primarily using electronic mailing lists) through the HEA and SEDA, and
networks including the Friends of the North PGCert Group, South West Educational Developers
Forum (SWEDF), Scottish Higher Educational Developers (SHED), the Head of Educational Developers
Group (HEDG) and the Association of Learning Technologists (ALT). The project team drew upon
45/222
their membership of the majority of these networks in order to disseminate the survey and
encourage participation. The survey was open from the 4th to the 17th of February 2015.
2.1.5 Analysis of audit data
In total 189 responses were received out of an approximate population of 1150, equating to a
response rate of approximately 16%. 142 of these were included in the analysis after removing
responses which were either not relevant (for instance, from non-UK institutions) or were
incomplete. Respondents were then categorised to facilitate analysis based on institution type, with
the categories listed in Table 2.1 applied to the data set. Similar types of institutions were grouped
into categories to allow for deeper investigation.
Table 2.1: Categories of institutions and responses
Descriptive statistics were used to review the data. Further analysis was conducted to identify
potential relationships between institution type and specific themes within the audit, for instance
reward and recognition. However, this stage was limited to respondents from category A and B
institutions, where there were sufficient responses for a meaningful analysis. The outcomes of this
audit are reported in detail in the next section, and directly informed the development of the tool
through WP3.
46/222
Category
Institution N
A Old/Post-1964/Russell group 47
B Post-1992/Million +/New 2010/University Alliance
55
C FE College 17
D Private Provider 5
E Unaligned/Unspecified 18
Total 142
2.2 Results of the National Audit3
2.2.1 Overview of respondent profile
The aim of this audit was to gain deeper insight in to CPD provision in the HE sector, overall, and
hence results are presented on a sector-wide basis. However, since the number of responses from
the research-intensive and teaching-focused categories of institutions are both significant and
similar, comparisons are made between them where relevant. College- based HE providers are not
included in these comparisons because of their lower response rates and associated issues of
representativeness. Instead, results about colleges are commented on separately. A number of
respondents chose not to provide details about to which group their institution belonged. Others,
categorised themselves as private providers or unaligned with any group but were too small in
number for findings to be generalizable. Hence all three types of responses were combined and
included in the overall analysis as part of an ‘Other’ category.
2.2.1.1 Teaching qualifications and minimum requirement for teaching–related CPD
Even with the introduction of the revised UKPSF, not all institutions link “formal” teaching-related
CPD with gaining teaching qualifications (Parsons et al., 2012). More recently however, there has
been a shift in this trend, especially for new academic staff. Hence, examining where the sector
stands with respect to staff qualifications and policies on mandatory teaching-related CPD, offers a
useful background with which to explore CPD provision in more detail.
Staff with HE-specific teaching qualifications (N = 137)
63% of respondents were aware of the percentage of staff within their institution who held relevant
qualifications to teach higher education. In particular, 49% of the staff from research-intensive
institutions were familiar with these statistics. A much higher percentage, 78% of staff from
teaching-focused institutions, were able to provide these details.
3 Notes: 1. To improve readability, all figures have been rounded to the nearest whole number. 2. The number of respondents to each question of the audit differs. This number has been specified in thesub-headings of the report for better interpretation of percentage values. 3. Response options to many questions in the audit were not mutually exclusive, i.e., participants were free to select more than one option. This may be useful to bear in mind when interpreting percentages.
47/222
A significant point-biserial correlation revealed that higher percentages of qualified staff tended to
be reported at teaching-focused institutions rpb = .466, p < .01. Further, chi-square revealed a
relationship between institutional category and whether qualified/recognised staff were a majority
(over 50%) or minority (less than 50%) within the institution χ2 (1) = 12.8 at p < .05 . Significantly
fewer research-intensive institutions had more that 50% of their staff qualified with an HE-specific
teaching award. However, because of the generally lower levels of awareness about staff’s teaching
qualifications amongst research-intensive institutions (with a higher number “not sure”), these
results need to be interpreted cautiously and confirmed with bigger samples.
41% college-based HE providers (N=17) were able to give details about the percentage of their staff
with teaching qualifications. In 4 of these 7 colleges, a majority of staff held an HE-specific
qualification.
Minimum teaching-related CPD (N = 106 – 113 for different teaching roles)
Respondents also provided information about the minimum teaching-related CPD required for
different roles. Overall, only 3% of institutions reported that they did not have a minimum CPD
requirement for incoming staff with teaching responsibilities and no prior teaching experience. In
contrast, for staff in other teaching and learning roles, a higher number of institutions reported
having no minimum CPD expectation (Table 2.2).
Table 2.2: Percentage of institutions with no minimum teaching-CPD requirement
RolePercentage of institutions with no minimum CPD requirement
Percentage of research-intensive institutions with no minimum CPD requirement
Percentage of teaching-focused institutions with no minimum CPD requirement
Staff in learning support roles
25 37 19
Staff in leadership roles 23 36 17PG students 20 21 15Existing teaching staff 20 31 9Associate lecturers 16 15 19Incoming staff with teaching experience
9 15 2
Incoming staff with no teaching experience
3 5 0
The most commonly reported CPD expectation for different roles is presented in Figure 2.1.
48/222
Figure 2.1: Minimum teaching-related CPD requirement in different teaching roles
Expectations from incoming staff new to teaching is more consistent across the sector with a
requirement to complete an accredited course (79%). Postgraduate students with teaching
responsibilities form the only group where the minimum CPD expectation is commonly a non-
accredited activity (39%). For all other roles, including staff with leadership responsibilities in
teaching and learning (32%), staff in learning support roles (40%), existing teaching staff (31%) and
Associate Lecturers (32%), the expectation for CPD is usually decided on a case-by-case basis.
However, it is almost as common for existing teaching staff (27%) and Associate Lecturers (29%) to
be required to complete an accredited course (both percentages have been included in the figure).
This is reflective of some institutional differences in the sector.
Teaching-focused institutions appear to be more instrumental in setting an expectation around
engagement with CPD. 46% of teaching-focused institutions and 41% of teaching-focused
institutions expected existing staff and staff in teaching related leadership roles, respectively, to
have undertaken an accredited course. Within research-intensive institutions, 18% required existing
staff to have completed an accredited course and 10% required staff in teaching related leadership
roles to have engaged with an accredited course.
49/222
Minimum CPD within college-based providers displays a similar trend to the overall sector.
Requirements tended to be decided on a case-by-case basis to a large extent. More detailed
comments cannot be made because a large proportion of respondents (ranging from 8% to 31%
relative to different roles) reported being unaware of their institution’s expectation around CPD.
Box 2.1: Teaching Qualifications – minimum CPD requirements
2.2.2 CPD Provision (N = 109)
2.2.2.1 Types of teaching-related CPD
Findings revealed that workshops, peer-reviews schemes, conferences and accredited programmes
were the most common forms of teaching-related CPD within the sector (see Table 2.3).
Table 2.3:Types of CPD from most to least commonly reported
Type of CPD Percentage who reported offering this type of CPD
50/222
Staff in teaching-focused institutions appeared to be more cognisant of the percentage of staff with HE-specific qualifications within their institution compared to staff in research-focused institutions. Teaching-focused institutions reported higher percentages of staff with relevant teaching qualifications although these results are to be interpreted cautiously.
Across the sector, there is an expectation for new staff without any teaching experience to engage with an accredited course as part of their CPD.
For more experienced staff and staff in leadership roles, their CPD requirements tended to be decided on a case-by-case basis.
Teaching-focused institutions were comparatively more consistent in setting an expectation around CPD for staff in different teaching and learning roles (including experienced staff), compared to research-focused institutions.
Workshops 92Peer-review and observation schemes 87In-house teaching and learning conferences 86Accredited academic programmes for teaching
85
Mentoring schemes 79Communities of practice around teaching and learning
60
External teaching and learning conferences 50Funding for PG study 46Action research 45Teaching consultations 38
In addition, through free-text comments, respondents also cited printed and online resources,
institutional journals, teaching fellowship schemes, writing retreats, coffee and paper sessions and
informal processes which encourage reflective practice, as a contributor to teaching-related CPD.
2.2.2.2 Examples of accredited and non-accredited CPD
The audit asked respondents to select any one CPD offer at their institution and provide details
about it. 71% selected an accredited CPD offer (N = 100) and 23% described a non-accredited CPD
offer (N=33). Approximately 6% reported that their institution did not provide any teaching-related
CPD. These were discounted.
75% of the examples of accredited CPD offers focused on Postgraduate Certificates in Higher
Education (PGCHE) leading to recognition with the HEA. The remaining examples were about
accreditation with SEDA, GTA and Master’s degrees in Education (MEd). Examples of non-accredited
provision consisted of teaching and learning conferences (32%), workshops (29%), scholarship of
teaching and learning activities (14%) and short courses (11%) (see Figures 2.2 and 2.3)
51/222
GTA 22%
PGCHE 75%
MEd 30%
Recognition with HEA
73%
Recognition with SEDA, 7%
Figure 2.2: Accredited CPD offers described by respondents
Respondents could select more than one option e.g. PGCHE and recognition with the HEA
Workshops 29%
Short courses
11%
Internal teaching and learning conferences32%
Peer review
7%
Mentoring -community of practice
4%
Scholarship of teaching and learn-
ing 14 %
Training / away days, 4%
Figure 2.3: Non-accredited CPD offers described by respondents
Staff targeted for the CPD offer (N=94 for accredited offers and N=29 for non-accredited offers)
Accredited offers were largely targeted at incoming teaching staff (88%), staff without a teaching
qualification (78%) and staff who support teaching and learning (75%). Non-accredited offers largely
52/222
targeted teaching staff with three years and more experience (86%), and to a lesser extent, staff in
other roles (staff without teaching qualifications 66%; incoming teaching staff 62%).
Focus and key aims of CPD offers (N=93 for accredited offers and N=28 for non-accredited offers)
Overall, the majority of CPD offers are cross-institutional. This is encouraging considering the
benefits of inter-disciplinary CPD that have been emerging from the literature, specifically around
sensitising staff to contrasting teaching approaches (see Parsons et al., 2012). However, there is a
significant association between type of CPD offer (accredited or non-accredited) and whether the
offer is cross-institutional or discipline-specific χ² (1) = 12.95 at p < .01. Specifically, more non-
accredited CPD offers were tailored to specific disciplines.
Accredited CPD offers primarily focussed on developing teacher skills and changing teacher practice
(98%). Yet, unlike non-accredited offers, they were also concerned, to quite a large extent, with
enhancing student learning (77%) and developing an understanding of institutional policies and
culture around teaching and learning (66%). This is not surprising considering that they primarily
targeted new staff and staff in learning support roles (see Table 2.4).
Table 2.4: Focus of accredited and non-accredited CPD offers
Focus Accredited offers Non-accredited offersTeacher-focused 98% 93%Student-focused 77% 43%Institutional-focused 66% 46%
Content of CPD offers (N=93 for accredited offers and N=26 for non-accredited offers)
Content for accredited teaching-related CPD was largely informed by external professional
frameworks (UKPSF,SEDA, discipline-specific frameworks etc.) (91%), institutional priorities (82%)
and evidence-based best practice (74%). Content for accredited teaching-related CPD was largely
informed by external professional frameworks (UKPSF,SEDA, discipline-specific frameworks etc.)
(91%), institutional priorities (82%) and evidence-based best practice (74%). Institutional priorities
also play a big role in informing content of non-accredited offers (81%), followed by educational
development unit priorities (54%) and teaching and learning committee priorities (54%).
53/222
Delivery of CPD (N=92 for accredited offers and N=27 for non-accredited offers)
Most often, non-accredited offers tended to be delivered completely face-to-face (73%). 53% of
accredited offers involved blended learning while 45% were completely face-to-face. The offers were
largely delivered by staff dedicated to providing CPD and academic staff within faculties for both
accredited and non-accredited provision (see Figure 2.4). Input from previous participants (54%) and
use of web-based resources (53%) appeared to be commonly integrated in to accredited offers. On
the other hand, talks by external guest speakers (44%) were common in non-accredited offers. Few
respondents indicated some level of student involvement in both accredited (17%) and non-
accredited offers (30%).
Staff ded
icated
to provid
ing CPD
Academ
ic staff
with
in facu
lties
Profes
sional
servic
e staff
Previous p
articip
ants
Extern
al guest
speak
ers
Web-base
d resource
s/onlin
e deliv
ery
Studen
ts0
102030405060708090
100 Accredited
Non-accredited
Figure 2.4: Persons involved in the delivery of CPD
Box 2.2: CPD Provision
54/222
Teaching-related CPD provision most commonly took the form of workshops, peer-review schemes, conferences, mentoring schemes and discussions and collaborative activities around teaching and learning through communities of practice.
Accredited offers aimed at catering to the professional development needs of new staff and staff in learning support roles. There was a strong emphasis on the teacher or participant (developing skills/knowledge/conceptions), but provision also focused on student needs and institutional culture. These offers were driven by external professional frameworks, institutional priorities and evidence-based practice.
Non-accredited offers targeted experienced teaching staff as well as engaged staff in other teaching roles. Their aims focused on the participant, and to a lesser degree on students and the institution. Teaching and learning priorities of the institution, educational development units and teaching and learning committees, informed the content of non-accredited offers.
Both accredited and non-accredited offers were usually delivered by staff dedicated to
2.2.3 Evaluation
2.2.3.1 Longitudinal evaluation and tracking changes to scholarship of teaching and learning in staff
Overall, 15% of respondents (N=109) reported evaluating at least one of their CPD offers
longitudinally. 57% reported they did not evaluate any CPD longitudinally and the remaining were
unsure. Further, 15% (N=108) tracked changes in scholarship of teaching and learning in staff
undertaking teaching- related CPD while 84% reported that they did not.
Programmes leading to HEA accreditation, funded research projects and teaching fellowship
schemes were provided as examples of offers which were evaluated longitudinally and where
participants’ engagement with SoTL was tracked. Evaluations included mentor catch-ups with
participants, questionnaires with participants, feedback from discipline-specific learning and
teaching committees and analysis of data in annual teaching and learning department reports.
Examples of how SoTL was tracked included staff e-portfolios for remaining in good standing and in
one institution, an annual scholarship survey for all staff.
We know from the audit results on the types of teaching CPD (Table 2.3) that communities of
practice around teaching and learning are fairly prevalent (60%). Yet there is no evidence that these
more unstructured, embedded SoTL activities within departments and the institution are tracked
and evaluated.
Box 2.3: Key findings about evaluation of specific CPD activities
55/222
Teaching-related CPD provision most commonly took the form of workshops, peer-review schemes, conferences, mentoring schemes and discussions and collaborative activities around teaching and learning through communities of practice.
Accredited offers aimed at catering to the professional development needs of new staff and staff in learning support roles. There was a strong emphasis on the teacher or participant (developing skills/knowledge/conceptions), but provision also focused on student needs and institutional culture. These offers were driven by external professional frameworks, institutional priorities and evidence-based practice.
Non-accredited offers targeted experienced teaching staff as well as engaged staff in other teaching roles. Their aims focused on the participant, and to a lesser degree on students and the institution. Teaching and learning priorities of the institution, educational development units and teaching and learning committees, informed the content of non-accredited offers.
Both accredited and non-accredited offers were usually delivered by staff dedicated to
11% of respondents who described a specific CPD offer at their institution (accredited or non-accredited) did not evaluate this offer.
21% (N=100) of accredited CPD offers were evaluated only once.In 35% of these offers, evaluation was carried out in two stages. A lower percentage of accredited CPD offers were evaluated in multiple stages:14% evaluated the offer thrice;10% evaluated four times; and 2% evaluated the offer at five different stages.
52% (N=33) of non-accredited CPD offers were evaluated only once. Both accredited and non-accredited offers were generally evaluated at the end of the
activity. Further, with non-accredited offers, no assessment occurred before and during engagement with the activity (Figure 2.5).
Before Event
During Event
At the end of event
Within 6 months
After 6 months
0
10
20
30
40
50
60
70 Recognised/accredited
Non-accredited
Figure 2.5: Count of evaluations at each stage for accredited and non-accredited offers
2.2.3.2 Focus of evaluation
The general trend in evaluations conducted during and immediately after a CPD offer was a focus on
participant satisfaction, changes in beliefs about teaching and learning and changes in teaching
practice, and to a lesser extent, the impact on students or the institution. A difference in this pattern
was observed in subsequent evaluations conducted after six months or more. Figure 2.6 compares
the focus of evaluation at the end of a CPD offer and six months after completion of the offer. At the
end of an event, 81% of the evaluations focused on participant satisfaction and less than half on
student learning (40%) and institutional culture (33%). On the other hand, from all evaluations
conducted after six months, only 38% focused on participant satisfaction but 50% reported
evaluating student learning and 48%, institutional culture.
56/222
Frequency
11% of respondents who described a specific CPD offer at their institution (accredited or non-accredited) did not evaluate this offer.
21% (N=100) of accredited CPD offers were evaluated only once.In 35% of these offers, evaluation was carried out in two stages. A lower percentage of accredited CPD offers were evaluated in multiple stages:14% evaluated the offer thrice;10% evaluated four times; and 2% evaluated the offer at five different stages.
52% (N=33) of non-accredited CPD offers were evaluated only once. Both accredited and non-accredited offers were generally evaluated at the end of the
activity. Further, with non-accredited offers, no assessment occurred before and during engagement with the activity (Figure 2.5).
Participants' satisfaction
Participants' beliefs
aboutT&L
Participants' teaching practice
Students' perceptions
teaching
Student learning
Culture of dept and the
institution
0
10
20
30
40
50
60
70
80
90
81
55
66
17
4033
38
6063
18
50 48
End of event
Six months or more
%
Figure 2.6: Focus of evaluation of accredited and non-accredited offers at different stages
2.2.3.3 Methods
Questionnaires with participants were the most commonly used evaluation. Following this, a mix of
interviews and focus groups, observation of teaching practice, review of participant
journals/teaching material and review of curriculum enhancement practices were consistently used
in different stages of evaluation.
Evaluations conducted six months after completion of the CPD offer tend to differ slightly from those
at other stages. These frequently employed interviews/focus groups with participants rather than
only questionnaires (presumably allowing collection of more rich and descriptive data). Evaluations
after six months were also more likely to consider impact on the student, even if indirectly through
external student surveys and progression and retention statistics. Many institutions appeared to use
a combination of a number of methods in their evaluation although the range seems limited (those
that involve students or line-managers were rarely used).
Other means of evaluation reported by respondents included informal emails and discussions with
participants (including through student reps and liaison committees), external examiner feedback
and feedback from mentors. They also commented on analysis of data around participants’
involvement in teaching and learning activities –for instance, away days, promotion and prize panels,
57/222
publication and dissemination of work etc. – as a useful indicator for evaluation. The use of module
evaluation forms was discussed with the challenges in gleaning useful information from them.
58/222
EvaluationsBefore event
(N = 8)
EvaluationsDuring event
(N = 48)
EvaluationsEnd of event
(N = 83)
EvaluationsWithin 6 months
(N = 37)
EvaluationsAfter 6 months(N = 40)
Respondents who evaluated at this stage( N = 142 )
6% 32% 54% 22% 20%
Focus(Top three at each stage)
Participant satisfaction(75%)
Changes in beliefs about teaching and learning (75%)
Changes in teaching practice (75%)
Changes in teaching practice (75%)
Participant satisfaction (71%)
Changes in beliefs about teaching and learning (56%)
Participant satisfaction (81%)
Changes in teaching practice (66%)
Changes in beliefs about teaching and learning (55%)
Changes in teaching practice (73%)
Participant satisfaction (68%)
Changes in beliefs about teaching and learning (68%)
Changes in teaching practice (63%)
Changes in beliefs about teaching and learning (60%)
Impact on student learning (50%)
59/222
Table 2.5: Overview of evaluation practices discussed by respondents
Methods of evaluation(Top five at each stage)
Questionnaire for previous participants
Review of participant’s journal/teaching material/other statements
Curriculum development/enhancement
Observation of teaching
Student assessments
Questionnaire for current participants
Interview/focus group with current participants
Observation of teaching
Review of participant’s journal/teaching material/other statements
Curriculum development/enhancement.
Questionnaire for current participants
Observation of teaching
Questionnaire for previous participant
Interview/focus group with current participant
Review of participant’s journal/teaching material/other statements
Questionnaire for current participants
Observation of teaching
Interview/focus group with current participants
Questionnaire for previous participants
Curriculum development /enhancement
Questionnaire for current participants
Interview/focus group with current participants
Observation of teaching
PTES/NSS
Review of retention and progression stats
60/222
HEA CPD Pilot Workshop - Workbook
2.2.3.4 Results of CPD evaluation and dissemination (N=105)
Results of CPD evaluation were primarily utilised in QAA and other professional audits (80%) and to
inform future CPD provision (71%). Following this, they were considered in making changes to policy
around teaching and learning (52%) , probation and promotion criteria (48%) and drawing up annual
reports for governing bodies (44%). They were included in marketing materials to a much lesser
extent (12%). 9% of respondents reported that the results of CPD evaluations were not used in any
way in their institutions.
The impacts of teaching-related CPD (N=110) were most frequently disseminated via sharing of best
practices through presentations, training and discussions (84%), and cascading learning through
teaching and learning committees (72%). Pedagogic research was another fairly common route for
dissemination (54%). 8% of respondents reported that impacts of teaching CPD were not cascaded
at their institution.
Other means of dissemination which were reported included changes made to programme design
and curriculum, newsletters, in-house teaching and learning journals and through the university
website or intranet.
2.2.4 Student involvement in CPD (N=108)
Responses from staff involved in CPD provision indicated that a majority of students were aware of
staff participation in teaching-related CPD, although to varying levels (54% ‘somewhat’, 13% ‘yes’,
Table 2.6). The majority of respondents agreed that teaching-related CPD was an important student
agenda (41% ‘somewhat’, 28% ‘yes’). However, opportunities for students to contribute to teaching-
related CPD appear to be limited (Table 2.6)
Table 2.6: Perception of student awareness and involvement in CPD
Yes Somewhat NoAre students aware of staff participation in teaching 13% 54% 10%
61/222
HEA CPD Pilot Workshop - Workbook
related CPD?Is teaching CPD an important student agenda? 28% 41% 11%
Are there opportunities for students to contribute to evaluation of teaching related CPD
13% 20% 50%
Respondents perceived where students were made aware of teaching-related CPD, this was through
the student union (54%), student representative system (51%), schools and faculties (41%) and
central communications (29%). Students also learned about CPD when participants (their lecturers)
spoke to them about their CPD, involved them in projects, and during peer-reviews and observations
in the classroom.
Indirect ways in which students contributed to the evaluation of CPD were through student-led
teaching awards, module evaluations and through providing feedback at teaching and learning
committees. Sometimes, students were involved more directly and participated in workshops and
teaching and learning conferences. A few especially interesting examples of student involvement
emerged through respondents free-text comments, for instance, one where students were involved
in observations of teaching and curriculum design.
There is some evidence that the sector makes an effort to gain student feedback from external
surveys and other forms of feedback (module evaluations and liaison committees) to inform the
content of CPD. However, the robustness of feedback mechanisms and the consistency with which it
is integrated in to teaching-related CPD needs to be further scrutinized.
2.2.5 Institutional policies and culture around teaching-related CPD
Research has highlighted some of the factors which influence the extent to which staff embed their
learnings from professional development activities in their practice. These include opportunities for
continuing dialogue, opportunities for reflection and discussion with peers, observing the impact of
an implemented change in students and having supportive leadership (Brown & Inglis, 2013). It thus
becomes evident that the culture of an institution plays a role in shaping the extent to which staff
might engage with and benefit from CPD. This section examines responses about institutional
policies that could influence teaching-related CPD.
62/222
HEA CPD Pilot Workshop - Workbook
2.2.5.1 Peer-review (N=114)
83% reported that their institution had a policy or expectation around peer-review of teaching. This
trend was found within categories of institutions as well, as shown in Table 2.7.
Research-intensive (N=40)
Teaching-focused (N=44)
College-based (N=13)
95% 80% 77%
Table 2.7: Percentage of institutions which have an existing policy/expectation around peer-review (by institutional category)
34% of respondents reported peer-review was not linked to appraisal in their institution with
another 42% indicating that it was linked to appraisal only in some cases. Comparing types of
institutions, Table 2.7 indicates that CBHE providers appear to be different from the rest of the
sector in this regard. In FE colleges, peer-review was more consistently linked to appraisal. Chi-
square tests did not reveal significant associations between institutional category and whether or
not peer-review was linked to annual appraisal/performance reviews.
Table 2.8: Peer-review and annual appraisal/performance review
2.2.5.2 Promotion pathways
The literature suggests that having separate promotion routes for teaching, research, and ‘teaching
and research’ roles enhance the chances that specific teaching and learning criteria will be included
in promotion policies (Cashmore et al., 2013). This, in turn, impacts the culture of CPD within the
institution as CPD becomes instrumental as a means to gain qualifications and evidence engagement
with teaching and learning.
63/222
Research-intensive (N=39)
Teaching-focused (N=39)
College-based (N=12)
Not linked to appraisal
Linked in some cases
Linked in all cases
Not sure
31%
51%
5%
13%
44%
33%
10%
13%
17%
42%
42%
0%
HEA CPD Pilot Workshop - Workbook
In the research-intensive category of institutions, a majority of the institutions had separate
promotion routes for research and teaching roles, with 35% reporting a three-track and 28% a two-
track path (see Figure 2.7). A three-track pathway might include separate promotion criteria for
teaching, research and ‘teaching and research’ roles. A two-track might, for instance, might have
separate promotion routes for teaching and research roles.
Research-focused (N=40)
Teaching-intensive (N=43)
College-based (N=13)0.05.0
10.015.020.025.030.035.040.045.0
Single-trackTwo-trackThree-trackNot sure
Figure 2.7: Institutional differences in promotion pathways
Amongst teaching-focused institutions, single-track pathways were much more common (35%).
There is a significant association between the category of the institution (research-intensive or
teaching-focused) and the type of promotion pathways it has in place χ² (4) = 10.25 at p < .05. 39% of
respondents from CBHE providers reported having single-track pathway and an equal number were
unsure. Looking at promotion policies of the highest position, that of professor, a large number of
research-intensive and teaching-focused institutions reported having explicit teaching and learning
criteria for promotion to this level (58% and 45% respectively). However, while only 8% of the
research-intensive institutions did not have explicit criteria at this level, this number was higher for
teaching-focused institutions, 15%. This inclusion of explicit teaching and learning criteria in
promotion policies in a higher percentage of research-intensive institutions compared to teaching-
focused institutions could be partly contributed to a higher number of research-intensive institutions
having professors of teaching (50%) compared to teaching-focused institutions (42%) (see Figure
2.8). However, this difference was not found significant by a chi-square test. A percentage of
respondents, including all those from CBHE providers, reported they were unsure about promotion
policies to professor.
64/222
%
HEA CPD Pilot Workshop - Workbook
Research-intensive (N=40)
Teaching-focused (N=43) College-based (N=13)0.0
10.020.030.040.050.060.070.080.090.0
YesNoNot sure
%
Figure 2.8: Occurrence of professors of teaching and learning
2.2.5.3 Research Excellence Framework and educational research (N=113)
Overall, 48% reported that their institution had made a contribution in educational research to the
2014 REF, 33% said they had not and 19% were unsure.Teaching-focused institutions made a higher
return (63%) research-intensive institutions (53%). However, this difference was not found
significant by chi-square. The majority of respondents from CBHE providers reported they did not
make a return to the REF or were unsure (92%).
2.2.5.4 Reward and recognition for teaching (N=112)
Student awards were most commonly reported as a means of rewarding teaching (65%), followed by
organisation of teaching and learning conferences (56%) and institutionally bestowed teaching
awards (54%) (see Figure 2.9).
Student bestowed teaching award
Organisation of T&L conferences
Institutionally bestowed teaching awards
Promotion
Internal teaching fellowship award
Community of practice around research in HE
External teaching fellowship award
Time allocation for teaching related CPD
Pedagogic research grants
Honorary titles
Financial incentives
0 10 20 30 40 50 60 70
65
56
54
47
46
35
31
31
30
9
6
65/222
(%)
HEA CPD Pilot Workshop - Workbook
Figure 2.9: Reward and recognition for teaching and learning
Differences within the sector were not marked. However research-focused institutions reported
more recognition through promotion (53%) (linking to the previous finding that they more
commonly have explicit teaching and learning criteria for promotion) and institutionally-bestowed
teaching awards (60%), compared to teaching-focused institutions (35% and 38% respectively). In
turn, teaching-focused institutions reported more recognition of staff’s CPD engagement through
organisation of teaching and learning conferences (56%), pedagogic research grants (35%),
communities of practice around research (38%) and time allocation for teaching-related CPD (36%)
compared to the research-intensive category.
Box 2.4: Overview of Findings related to institutional policies around teaching-related CPD
66/222
Overall, a majority of the sector has an expectation around peer-review of teaching but it is not consistently linked to the annual appraisal and peer review processes.
Research-intensive institutions currently tend to have two and three track promotion pathways, which allows them to have explicit teaching and learning criteria in promotion policies. Single track promotion pathways tend to be more common in teaching-intensive institutions and colleges.
Motivation plays a big role in staff’s ongoing engagement with CPD. The most prominent mechanism for recognition of teaching quality is currently student awards and to a lesser extent, institutionally bestowed awards.
HEA CPD Pilot Workshop - Workbook
2.3 Case study development
2.3.1 Selecting the case study institutions
In total, 22 audit respondents indicated that they would be willing to be considered as a case study.
A review of their audit responses led to the immediate elimination of ten of these, primarily because
they did not evaluate CPD or, where they did, their evaluation was focused immediately after a CPD
activity with no longer-term follow up which would be of value in measuring impact (Trigwell et al.,
2012; Chalmers et al., 2012; Stewart, 2014). A further two were excluded due to a lack of relevant
CPD provision. Of the remaining 12 institutions, eight were selected for initial follow-up as they
evaluated the impact of their work at least six months after an activity had been implemented.
The project team also undertook a desk-based survey of publications (such as Educational
Developments) which provide insights into contemporary educational development practice.
Following this, a further three potential case studies were identified. In total, 11 institutions were
contacted to explain the context of the current project and to request participation. At this stage we
applied the selection criteria used by Turner et al. (2013) in the development of case studies for their
HEA/SEDA funded project: Measuring the impact of the UK Professional Standards Framework for
Teaching and Supporting Learning. In this project the following criteria were used to select case
study institutions:
1. Participant has agreed to be interviewed in the survey;
2. Survey responses indicate something interesting to say;
3. The group of case studies represents a range of institutions;
4. The group of case studies represents different types of engagement.
Turner et al. (2013) also advocated limiting the number of open-ended interview questions and
allowing time to explore what was ‘interesting’ about the examples specific to individual case
studies. In heeding this, we adopted a narrative approach to the interview schedule. A number of
central prompts ensured that essential contextual information regarding CPD provision was
captured. However, the primary focus of each interview was to obtain information relating to the
development of interviewees’ particular approach to evaluating their CPD provision and why this
was effective.
Naming the institutions and, where relevant, key individuals, was integral to the development of the
case studies. We felt this would afford recognition to those engaged with innovative evaluation
67/222
HEA CPD Pilot Workshop - Workbook
practice and allow further dialogue to take place beyond the life of this project. However, we
recognised the potential implications for participating institutions. Potential participants were
requested to discuss the invitation with their colleagues / unit heads. Following this, five institutions
(University of Roehampton, Sheffield Hallam University, London Metropolitan University, Queen
Margaret University and The Open University) were selected to be developed into case studies. We
also decided to include Plymouth University, as through the process of reviewing the audit
responses and holding preliminary discussions with the case study institutions, it was realised that a
number of innovative approaches to evaluating the impact of CPD activities were being undertaken
here, which would be of wider interest to the Educational Development community.
2.3.2 Data collection
The audit data were reviewed to frame the interview schedule, which was adapted depending on
the institution, the CPD activities that were to be discussed and their approach to evaluating impact.
A telephone interview was held with a key contact in each institution. Each interview was audio
recorded and transcribed verbatim. A member of the project team then analysed the transcript to
identify the information necessary to develop the case study. Extracts of the interview narrative
which could illuminate the case study were also identified, to be used in the final case study. Each
case study was written and reviewed by the project team, and then returned to the participant for a
final review and edit. Giving the participants (and where relevant their colleagues) the opportunity
to view the final text was integral to the development of the case study, in order to ensure they
were happy with the information that was included and its presentation as the case study.
68/222
HEA CPD Pilot Workshop - Workbook
2.4 Case studies
2.4.1 Case Study – London Metropolitan University
London Metropolitan University is a large post-1992 university spread across 3 London
campuses. The university comprises 4 Faculties: The Faculty of Social Sciences and
Humanities and the Faculty of Life Sciences and Computing are based at Holloway Road, the
Faculty of Business and Law at Moorgate, and the Sir John Cass Faculty of Art, Architecture
and Design at Aldgate.
It has 16,000 students, and offers undergraduate and postgraduate degrees across a range
of over 27 subject areas encompassed within the academic domains reflected in the above
Faculties.
The University is committed to widening access to Higher Education, invests in
entrepreneurship and innovation, and places employability at the heart of the curriculum.
Professional development opportunities for staff
The Centre for the Enhancement of Learning and Teaching (CELT) runs a PGCert/MA in Learning and
Teaching in HE (MALTHE) accredited by the HE Academy at UKPSF Descriptor 2, as well as a Learning
& Teaching Practice course accredited at Descriptor 1. These programmes are integrated within the
University’s CPD Framework for Academic Practice, overseen by CELT, which also includes provision
for staff to gain professional recognition from D1-D3, on the basis of experience. The Centre
organises an annual Learning and Teaching conference, and a range of professional development
modules, seminars and workshops, plus online guides and resources. CELT also produces a
registered journal, Investigations in university teaching and learning (ISSN 1740-5106).
Evaluating the effectiveness of teaching-related CPD
CELT routinely gathers statistics and qualitative feedback from attendees at CELT organised activities,
and these are presented in an annual report which is designed to “inform the University and
Faculties of CELT’s range of services…to demonstrate output and outcomes and thereby provide a
basis for consideration of future priorities and funding of CELT work”.4
4 https://metranet.londonmet.ac.uk/fms/MRSite/psd/hr/capd/CELT%20Policy%20Docs/CELT%20Annual%20Report%202012%20to%202013.pdf
69/222
HEA CPD Pilot Workshop - Workbook
Courses, such as the MALTHE and the e-learning modules, are evaluated continuously. At the start of
the CPD activity staff establish what participants hope to gain from the course, and adjust as
appropriate; during the course informal feedback is collected to help clarify expectations, and fine-
tune the content and process. At the end of the courses more formal methods, such as a self-
evaluation questionnaire as well as evidence presented in projects (for example, projects relating to
curriculum evaluation/redesign and assessment practices) are used to evaluate participants’
satisfaction, as well as changes to participants’ beliefs about teaching and learning, and shifts in
teaching practices. Staff within CELT are also interested in the impact on student learning as well as
the impact participation has on the culture of the department and the institution. The impact on
students is measured indirectly by teachers self-reporting on shifts in their teaching approaches e.g.
from a teacher-centred to a more student-centred approach, rather than looking to shifts in student
results or other impact indicators.
In 2008, a substantive impact study of the MALTHE was undertaken which involved staff who had
completed the programme within the previous four years.5 The study explored “the impact of the
formal course on lecturers’ pedagogical thought and practice, and opportunities for and barriers to
implementing acquired ideas or desired changes”, and utilised an open-ended questionnaire which
was emailed to participants.
In 2013, a similar, but smaller scale evaluation was undertaken with the purpose of collecting
feedback for programme revalidation.
‘Every five years we conduct a major review towards updating provision for revalidation. This is when we use questionnaires and focus groups with previous participants, to get a sense their views of the benefits of the CPD and how it has impacted on their educational thinking and practice and on their students' learning’ Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).
There were a lot of similarities between the findings of the 2008 and 2013 evaluations. Respondents
identified cross-disciplinary networks, opportunities to share practice and a boost in confidence as
key features of the course:
‘Participants described the benefits of being part of a whole network where they could share ideas across disciplines, and get new ideas…for some of them it was a really transformative experience in
5 Warren, D. (2008). Benefits and barriers regarding the enhancement of educational practice: an investigation of the impact of an accredited professional development programme for teachers in higher education. Paper presented at the London Scholarship of Teaching and Learning 7th International Conference, London, 15-16 May, 2008.
70/222
HEA CPD Pilot Workshop - Workbook
terms of extending their understanding of teaching and learning, and just really giving them confidence that some of the new teachers need to get them started’
However, participants also encountered impediments to integrating their learning from the MALTHE
into their teaching practice:
‘People left the programme with ideas and inspiration and confidence and keen to maybe try out things but when they got out into the real world of the department they found that change was a little less easy to implement... that there were cultural and institutional inhibitors’ Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).
Staff that progress to the Masters programme (MALTHE) are asked to submit a reflective ‘graduation
statement’ articulating what they have learnt through the process of undertaking the course, and to
think ahead in terms of how they think this would inform or influence their future practice. These
reflective statements are used as a further source of impact evaluation evidence.
Challenges to evaluating CPD
It was recognised that the benefits to practice might well become clearer some time after
completion of the CPD activity. However, carrying out longitudinal in-depth evaluations such as the
post-completion impact evaluation for MALTHE is resource intensive. Another challenge noted was
the fluctuation in institutional agendas and policies. When asked ‘Why is longer term evaluation not
carried out as a matter of course?’, the response was:
‘There are times when it’s important to make visible the quality and impact of what one is doing and so you seize those moments when it is necessary to do that. Because of the kind of fairly common cycle of having programmes reviewed every five years I think people often rely on that more standard periodic review process as a way to reflect on and refresh CPD provision without necessarily doing the more in-depth study that a proper longitudinal impact study requires, I think it’s all these mix of things, it is resources, it is political agendas, institutional agendas’ Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).
The sensitivity of undertaking impact evaluation was also highlighted in the interview. It was felt that
there needed to be a clear division between education enhancement, and gathering data that
71/222
HEA CPD Pilot Workshop - Workbook
ultimately could be used for quality monitoring. This connected to the question of ‘Who undertakes
the evaluation?’ In particular, if an evaluation involves students, ‘Who speaks to the students?’
‘I think there is a fine balance to be struck between asking teaching staff to articulate how they feel both [teachers and students] benefit, and how their thinking and practice has changed, and not making them feel they are being made subject to some kind of appraisal when you come in and start talking to their students. I think it presents some quite tricky research ethical issues’. Digby Warren, Head of the Centre for the Enhancement of Learning & Teaching (CELT).
72/222
HEA CPD Pilot Workshop - Workbook
2.4.2 Case Study – University of Roehampton
The University of Roehampton is located in south-west London and comprises four colleges.
It had 8,530 students enrolled in the 2013/14 academic year, and offers undergraduate and
postgraduate degrees across a range of subjects including Business and Management,
English Literature, Creative Writing, Film, Anthropology, Sports Science, Sociology,
Education, Dance, Drama, Philosophy, Classics, Criminology, Psychology, and Counselling.
It was recently named in the Sunday Times Good University Guide, as the Best modern
university in London.
Professional development opportunities for staff
There is a growing institutional expectation that staff involved in teaching will have recognition at
Descriptor 2 of the UK Professional Standards Framework(UKPSF). The Department of Academic
Enhancement (DAE) runs an introductory course for PhD students who teach, technicians and
visiting lecturers to supporting learning and teaching in H.E (accredited at D1). It also offers a
mandatory probationary programme for new lecturers: ‘The University of Roehampton’s Certificate
in Learning and Teaching in Higher Education’ (URCert), accredited at D2. In addition, the university
manages an in-house CPD framework for experienced staff, accredited at all four levels of the UKPSF
and this is aligned to hiring and promotion processes; for example, staff seeking promotion to
principal lecturer are expected to have achieved or be working towards D3.
A dedicated DAE staff member acts as the link to each department. In addition, DAE offers a range of
teaching-related workshops, a popular annual learning and Teaching Conference, and has been
particularly successful in gaining funding for teaching-related research projects.6 Individualised
support is also given to staff consistently receiving disappointing module feedback, via a
personalised development programme tailored to their needs. This might include peer reviews of
teaching, professional dialogue and introductions to new teaching approaches.
Student engagement with teaching-related CPD7
Student engagement is high on the University of Roehampton’s agenda, and as such this forms a
specific aspect of interest in this case study. The university has a student senate, which meets
regularly with the Vice Chancellor as a formal committee. Student representatives from across the
6 http://www.roehampton.ac.uk/Services/Learning-and-Teaching/Research/
7 http://www.roehampton.ac.uk/Services/Learning-and-Teaching/Student-Voice/
73/222
HEA CPD Pilot Workshop - Workbook
different programmes and the Students’ Union have an active role in policy making for the
university, with a dedicated budget of around £50,000 to spend on initiatives that they think are
worthwhile. This central drive for student engagement impacts across all departments. DAE involves
students in staff CPD in a number of ways. Students have been involved in peer reviews of academic
staff on the URCert, and this has been hugely successful. Student consultants are trained by DAE
staff to observe teaching and to give feedback. Students observe staff from a department which is
not connected to their own programme. They focus on the pedagogic practices of the lecturer, not
the content.
“They could talk about whether or not the academic was trying to involve and include all of the students, whether all the material seemed to be taken from a sort of Anglo-centric source, whether the lecturer just read from PowerPoint slides and whatever. So they definitely had the wherewithal to comment on the sort of pedagogic practice adopted, and that was very positive. And a lot of the academics said, “We’ve always wanted to ask for sort of direct student feedback, but we know that our own students are never going to be as honest as they might be, because we’re marking their work, and because they’ve got to confront us in the session, so, you know, they’ve got to be there in the session next week, if they’ve said anything negative”, whereas if they have students from outside their departments, you know, no axes to grind anywhere, so they could get this sort of, this direct student feedback’. Jo Peat, Head of Academic Professional Development
As part of the HE Academy funded project ‘Re-imagining Attainment For All’ (RAFA Project), students
at Roehampton have been involved in developing an interactive resource (dialogue sheets) to use
with academic staff teams to help them to review teaching and assessment practices. The project
was set up in response to the attainment gap between Black and minority ethnic students and white
students. The dialogue sheets are generated by students with a student co-ordinator, and include
student reflections on their teaching and learning experiences. These are presented as direct quotes
to the learning and teaching advisory group chairs alongside data from planning, on the degree
attainment differential for their programmes.
Jo Peat, Head of Academic Professional Development explains: ‘Rather than it being us [academic staff developers] facilitating these workshops, we got our student consultants to do it. So we very much took a back seat, and we got them to facilitate the workshops and to guide the direction of the workshops. The academics said that they’re the best workshops they’d ever attended, because they could hear directly from the students. It wasn’t being mediated by anyone, they could say to the students that, “I never knew you felt that, really, the students on my programme feel that?”, and the students were saying, “Well, yes, I’m one of them and, yes, this is exactly how we feel, you know, this is what we’re experiencing, these are the emotions we go through”. So the fact that we weren’t
74/222
HEA CPD Pilot Workshop - Workbook
there, as the sort of the purveyors and the facilitators of these workshops, made them much more powerful, and the academics really did like that direct link with the students.
This approach has been so successful that a number of cross-university workshops have been held
utilising the same approach in examining what the students have said, and what that might mean for
pedagogic practices across the university.
Evaluating the effectiveness of teaching-related CPD
Participants on formal CPD offerings are asked to evaluate the course upon completion, and, in the
case of the URCert, one year afterwards. Both evaluations take the form of an email questionnaire.
The initial feedback is sought using a 22-item survey the bulk of which comprises likert-style
statements about teaching and learning. For example, one statement reads “My ideas on teaching
and learning are substantially the same as before”. 5 open questions are asked at the end, focusing
on participant satisfaction, attendance and participants’ reaction to undertaking a small-scale
research project. A year later, participants are sent 7 further evaluation questions which ask
participants to recall the most and least valuable aspects of the course and to identify changes in
behaviour in relation to teaching and assessment practices.
In addition to formal monitoring via questionnaires, there are numerous opportunities for informal
feedback, helped, in part, by the small size of the institution.
‘We try to ask staff fairly regularly without being sort of overly enthusiastic about badgering them, about elements that have contributed to them changing their practice…We've all got departments we link with, and we'll visit them sort of at least once a month for meetings and things. So we've got lots of opportunities for fairly sort of ad hoc conversations and informal conversations about how things are going, so we do get quite a lot of information that way, but it is informal, you know, it's not a formal way of sort of evaluating practice and impact’. Jo Peat, Head of Academic Professional Development
In relation to the RAFA project, both staff and students complete evaluation sheets that focuse on
reactions to the CPD activity as well as changes in teaching and learning behaviours. In addition,
verbal feedback is sought individually to try and ascertain the impact of the activity upon practice.
75/222
HEA CPD Pilot Workshop - Workbook
Challenges to evaluating CPD
Whilst it is recognised that longitudinal evaluations can provide useful information relating to impact
on teaching practices, in reality this is not straightforward. For example, in the past longitudinal
evaluation had been undertaken to assess the impact of the Annual Teaching and Learning
Conference on practice, 6 months after the event. However, the response rate was so low that these
evaluations have been discontinued.
‘Our annual learning and teaching conference, and that's become very popular, you know, we get an awful lot of staff there now, in fact more than we ever thought we would get there, and they have an evaluation to fill in. One of the questions is something about what they think they might do in terms of their own practice in the light of what they've seen or been part of in the conference, but often that's the bit that's sort of left blank or it's answered, "I don't know", and even if we go back sort of six months later and say, "Did you make any changes?", again, it tends to be sort of, "Oh, I'm not really sure, I didn't really think about it that way’. Jo Peat, Head of Academic Professional Development
Future plans
Involving students with Peer Review is something that Jo Peat (Head of Academic Professional
Development) would like to see developed beyond the URCert. She regards it as important that
teaching staff aren’t primarily receiving their feedback from educational developers. Resources
constrain development in relation to the training of students, and ensuring that both students and
staff are protected.
76/222
HEA CPD Pilot Workshop - Workbook
2.4.3 Case Study – Sheffield Hallam University
Sheffield Hallam is the 4th largest university in the UK, with 34,718 students (25,988
undergraduates and 7,114 postgraduates). As of 2013/14, staff totalled 4,494, including
2,114 academic staff and income generated was £257.1m.
The University has four teaching faculties – including 18 academic departments - based
across two campuses (City and Collegiate).
The University encourages work-based learning, with over half of all courses offering
integrated practice or work placement opportunities.
Approach to CPD for academic staff
The approach to CPD for academic staff at Sheffield Hallam University draws upon a peer
enhancement strategy informed by the following principles: focussed on teaching and learning
practice; authentic; facilitated by using expertise to scaffold learning and application; peer
supported; team based; evidence based; and generates reflection and feedforward. The key
components of appraisal, CPD and Peer Review and Enhancement continue to be central to this
strategy.
“Measuring the impact of this work on the staff and student experience remains a challenge,
although we routinely reflect on our approach, trial new ways of measuring what we do and adapt
our practice in light of experience and feedback” Jackie Cawkwell, CPD Manager
Professional development opportunities for staff
Teaching-related professional development opportunities for academic and learning support staff at
Sheffield Hallam include informal and credit-bearing opportunities; responsibility for these areas of
work lies in different parts of the University.
The central directorate works with faculty leaders and the wider academic community through a
'hub and spoke' approach; events and opportunities are co-designed and co-facilitated by staff from
the Learning Enhancement & Academic Development (LEAD) directorate and faculty learning and
teaching leaders and innovators. The aim is to develop good inspirational learning and teaching
practices across the University by targeting staff groups according to their levels of experience.
77/222
HEA CPD Pilot Workshop - Workbook
Priorities are currently early career teaching staff and academics with leadership roles. The themes
of learner engagement, course belonging and professional recognition are current priorities. The
CPD strategy seeks to reinforce the need to support staff commitment to remaining in good standing
as Fellows and Senior Fellows of the HEA.
Evaluating the effectiveness of teaching-related CPD
The central directorate (LEAD) evaluate the impact of CPD events that they co-ordinate, as well as
broader CPD work. All academic staff were surveyed on their engagement with, and attitude
towards, CPD in March 2012. Participants reported overwhelming support in terms of successful
delivery and achieving stated outcomes of CPD sessions, particularly early career staff who
welcomed the informal and inclusive approach. Anecdotally, colleagues who attended reported that
engaging with CPD extended both their practice and their confidence, whilst presenters
demonstrated a growing capacity for leadership of others in their academic discipline and pedagogy
in the context of HE.
LEAD seeks to identify several very specific elements when they monitor, evaluate and measure
impact:
Monitoring numbers and staff profiles of those attending and facilitating events.
Evaluating process indicators e.g. expectations, quantity/quality of each event;
Measuring outcome indicators (impact) for individual staff (both participants and facilitators).
For example: affective outcomes, such as confidence and motivation as a result of engaging
with CPD activities; development of skills associated with professional practice, such as critical
enquiry, evidence-informed practice, reflective practice and scholarship; development of a
repertoire of 'classroom' practice and tool-kit of resources for teaching, learning and
assessment activities;
Measuring the impact on broader teaching and learning ambitions, i.e. aligning provision
with institutional and teaching team objectives, and informing individual development plans.
“We also reflect more generally on other aspects that influence the achievement of our aims relating to excellent teachers, teaching excellence and improvements to the student experience, for example: appraisal; Peer Review & Enhancement; professional recognition targets and strategies; recording CPD and maintaining good standing” Jackie Cawkwell, CPD Manager
78/222
HEA CPD Pilot Workshop - Workbook
Monitoring and review of Appraisal and Peer Review aims to encourage good practice in recording
and reflecting on outcomes for the individual and their line manager. The only recording beyond
this, however, involves the requirement of an audit at faculty level that captures which activities
took place and identifies any relevant cross-faculty and staff development issues. In reality, the latter
information is often too generalised to inform future CPD activities.
Challenges to evaluating CPD
Monitoring numbers and profiles of all the CPD that actually occurs is a huge task. Further, in some
cases, routine monitoring is not appropriate or even possible (for example, when nurturing special
interest groups or contributing to ad hoc events). In many cases it is more appropriate to encourage
individuals to record their engagement autonomously, for example, using PebblePad. Evaluating
aspects such as meeting expectations, whilst useful for planning, design and delivery purposes, does
not in itself contribute in any meaningful way to our understanding of the actual impact of CPD
activities.
‘CPD activities are often a 'slow burn' and without the chance to practice and reflect on new approaches, impact can be overlooked or disassociated from the initial development activity. Nor do we routinely have access in any detail to reflective discussions between academic colleagues, their peers and their line-managers. Thus the impact of what we do is often lost to us’. Jackie Cawkwell, CPD Manager
Future plans
LEAD seeks more effective integration of monitoring, evaluation and measuring of impact for the
range of professional development activities that staff engage with. These vary between discipline-
specific and teaching and learning pedagogy; between scholarship and research; between credit
bearing courses, non-credit bearing (but formal) and informal events. Aspirations include:
an improved capacity for all academic and learning support staff to act autonomously in:
o recognising their own development needs as part of an evidence-informed approach
to practice,
o working collaboratively with peers in effectively addressing those needs,
o feeling confident in practising and further reflecting on new ways of being and
doing;
79/222
HEA CPD Pilot Workshop - Workbook
more effective use of institutional quality assurance processes such as Course/Periodic
Review, institutional audit, appraisal, and performance management. This would benefit
both the implementation of institutional strategies and inform future policy direction;
extending our understanding of how our growing community of professionally
recognised/certificated staff (and our community of staff engaged in maintaining and further
developing themselves) actually impact on the learning gains and overall experience of our
learners.
80/222
HEA CPD Pilot Workshop - Workbook
2.4.4 Case Study – Nottingham Trent University
Nottingham Trent University (NTU) is one of the largest universities in the UK, with 27,000
students.
The University has recently improved its overall ranking, moving from 73 rd to 57th in the
Guardian University League Table for 2015 (i).
A diverse range of academic disciplines are provided by 9 Academic Schools organised in 3
Colleges: College of Business, Law and Social Sciences; College of Arts and Sciences; and
College of Art, Design and Built Environment.
Professional development opportunities for staff
The Centre for Professional Learning and Development (CPLD) was established in 2008 as part of the
Human Resources Department. Its role is to promote and support effective professional
development policy, strategy and opportunities for all staff groups as appropriate to help meet the
University’s strategic ambitions.
‘Being part of HR has actually helped us in a lot of the work that we do. We have professional development very much embedded in the annual appraisal process, in wider policy and in wider practice around the institution. So it’s really helped from that point of view.’ Alison Stewart, Academic Practice Development Consultant
In 2011, CPLD led the development of institutional CPD frameworks to assist and inform academic
practice professional development. One of these was the NTU Teaching Development Framework
(NTU TDF), designed to support the professional development of all staff who teach and/or support
learning at NTU.
The Framework has 5 elements including the NTU Learning and Teaching Professional Development
Policy. Another Framework element embeds national standards for teaching and supporting learning
as set out in the UKPSF by defining what they mean at NTU; all staff who teach and support learning
at NTU are expected to gain HEA recognition. As such, the Framework demonstrates the University’s
commitment to embedding those standards within professional practice and its staff development
policy and provision.
NTU-wide Teaching-related CPD delivered by CPLD has undergone significant development over the
last 7 years and includes writing workshops and 1:1 support for HEA applicants, on-line distance
81/222
HEA CPD Pilot Workshop - Workbook
learning resources and a number of conferences, course leader workshops, teaching development
sabbaticals and secondments, pedagogic research, and professional development programmes
including:
The new Postgraduate Certificate in Academic Practice (PGCAP) – permanent lecturers new
to teaching in HE are required to complete Module 1 by the end of their first year. In
addition to academic credits, this gives them HEA Associate Fellow recognition, and they go
on to achieve HEA Fellow recognition within 3 years of starting at NTU.
The 100-hour Certificate in Learning and Teaching in HE programme (CiLTiHE) is aimed at
full-, or part-time staff with some prior HE teaching experience, and Hourly Paid Lecturers.
The course comprises 5 Modules each aligned to the five areas of activity of the UKPSF.
Postgraduate Research Students complete a 25-hour Essential Teaching Toolkit programme
(ETTiHE) to achieve their ‘Readiness to Teach’ certificate. It provides them with the practice
skills and knowledge to design and deliver individual sessions.
Evaluating the effectiveness of teaching-related CPD
The overall service provided by CPLD is evaluated with the aim of understanding its impact for the
individual participants, their teams, the wider Schools and Professional Services, and the institution.
Evaluation is regarded as a crucial aspect of all work undertaken, as CPLD is required to report on the
outputs and outcomes it supports.
CPLD have designed an Evaluation Framework to enable different levels of impact to be determined.
CPLD’s remit for staff development across NTU’s academic and professional services falls into three
categories of role:
consultancy or advice to support CPD development
partnership working to help develop and/or deliver CPD
direct provision of courses and resources
The CPLD Evaluation Framework maps those three categories of role against five levels of impact:
Level 0: Establishing foundations and awareness – the work that needs to be in place
before a development intervention
Level 1: Initial responses – the reaction to a development intervention at the time or shortly
after by those involved (including CPLD staff) – the ‘CPD Reactionnaire’
Level 2: Perceived change – the change to an individual’s/team’s practice as reported by
them during follow-up activities
82/222
HEA CPD Pilot Workshop - Workbook
Level 3: Observable change – the change in an individual’s/team’s practice as observed by a
line manager or person
Level 4: Institutional impact – changes to institutional practice (aims, structures, processes)
and their impact
‘We try really hard not to deliver training which is discrete and doesn’t tie in with a bigger organisational development ethos. We try and build our CPD into other drivers so that we can demonstrate wider, observable impact and change. That’s really important to demonstrate our contribution to the wider organisation. Lindsay Davies, Academic Practice Development Consultant
The evidence for the impact of all three roles at the different levels is captured via a suite of tools
and measures, including self-reflective questions, practice and policy reviews, and practice case
studies.
The CPLD Evaluation Framework was created in-house in 2012 to meet the needs of the Centre and
builds on Kirkpatrick and Kirkpatrick’s (2005) evaluation levels and The Rugby Team Impact
Framework (2008) which was designed to measure the effectiveness of researcher development
initiatives.
‘Kirkpatrick’s model focuses on the effect on the participant. What I wanted to do was to widen that. Greater impact is made when the impact shifts to the Department. So we are interested in how far does ownership shift from us out to the target group? So ultimately we should be performing a consultancy role and things should be working either in partnership with us or owned by the Schools for instance. We are interested in: How far has the activity become embedded within the organisation? E.g. has policy changed?.....that’s the high impact stuff. You may be able to get some quantitative statistics around that but others are more qualitative. They are about – what has happened to the institutional structure or who is represented on various bodies’. Lindsay Davies, Academic Practice Development Consultant
All CPD events including conferences, workshops, and briefings are evaluated at Level 1 using an on-
line post event feedback form known as the ‘CPD Reactionnaire’ is used to capture participant
83/222
HEA CPD Pilot Workshop - Workbook
feedback about the effectiveness of the event organisation, booking and publicity, the learning
environment, facilitation, and how well the event matched participants’ expectations. Level 0
evaluation is done prior to the event to evaluate the appropriateness of the event design in line with
the needs of the target audience.
‘We have always tried to avoid just rating the popularity of interventions using a simple ‘tick-box’ approach to evaluate impact. That Level 1 approach has its’ place and benefits, but we are more interested in getting actual evidence of how the intervention has impacted on participants’ practice in the short and longer term’ Alison Stewart, Academic Practice Development Consultant
Teaching and learning professional development courses, such as ETTiHE and CiLTiHE outlined
earlier, are routinely evaluated at Level 2 to determine to what extent to which the CPD has
informed participants’ practice. This is evaluation is done using a number of methods including:
Pre-, and post-course self-diagnostic assessments against learning outcomes and other
skills/knowledge frameworks, to identify initial levels of participant confidence, competence and/or
capability and how these alter as a result of engagement in the course.
Activities during professional development delivery using critically reflective questions are used in a
number of settings including online questionnaires, practice portfolios, presentations, formative and
summative assessments, class discussions, and occasionally – focus groups - related to the CPD
learning outcomes. Both group and individual feedback is collected, and much of it is anonymous. In
discussions and feedback activities, participants are left together in a room (without the facilitator)
to collate their responses. This approach has generated rich information about impact on practice,
knowledge and values, which CPLD have been able to use to demonstrate the impact of professional
development on practice.
Consultancy and bespoke teaching and learning CPD activities are evaluated in two ways. Firstly,
participants are asked to complete a questionnaire, which asks four open questions. These questions
seek to understand: what was most useful; what needs to happen next; whether they are any
further development needs for the individual or for the staff team. Secondly, the relevant Manager
is also asked to complete an evaluation, which asks more searching questions in terms of whether
the CPD activity achieved the agreed objectives and outcomes, and how those align to the needs of
the team and the strategic priorities of the university.
CPLD report that a very rich source of higher levels of impact of teaching-related CPD emerged from
the 250 applications for HEA professional recognition via the HEA-accredited NTU Scheme.
84/222
HEA CPD Pilot Workshop - Workbook
‘The application process at NTU focuses applicants and referees not only on explaining what is done and why, but also identifying impact on practice – and the information they use as evidence of impact. Applicants and referees often found the latter aspects the most challenging parts of the process. Not only did they find it rewarding and satisfying to see what they had achieved over time, but we realised the value of it as evidence of impact at Levels 2 and 3 in our Evaluation Framework. Some have said that it has changed the type of conversations they now have in the peer and line manager reviews. Alison Stewart, Academic Practice Development Consultant
Future plans
There is a strong institutional driver to demonstrate impact of activities on teaching quality. CPLD
have identified the potential of using their Evaluation Framework to support this. This will involve
working with senior managers and academic staff to review and revise, as appropriate, the standards
already defined using the UKPSF in the NTU Teaching Development Framework to articulate an
institutional definition of ‘teaching quality’.
CPLD are working to establish a sustainable method for longitudinal evaluation of the impact of
teaching-related CPD, including the new Postgraduate Certificate in Academic Practice starting in
January 2016 and HEA professional recognition fellowships. In 2015/16, academic staff will be
asked to identify in annual plans, the professional development activities they will complete to
achieve this. This will enable CPLD to evaluate the impact of those activities in contributing to
supporting implementation of University strategic goals and teaching quality.
‘Longitudinal evaluation is one of those areas that is ‘tricky’ and it’s tempting to end up avoiding it because it is just too difficult and very time consuming, or get hung up on particular measures which are not really telling us anything about long term impact. I’m sure we’ll find an approach that’s a sensible compromise!’ Alison Stewart, Academic Practice Development Consultant
CPLD have also started discussions with the Students Union to identify ways of involving students in
teaching-related CPD and evaluating it.
‘The discussions we’ve had with colleagues from Plymouth really brought home to us how we haven’t really thought about involving our students. We approached the Student Union Reps at a recent NTU teaching conference and were delighted to hear them say they have lots of ideas about how they
85/222
HEA CPD Pilot Workshop - Workbook
could get involved. So we’re meeting them over the summer to talk further’. Alison Stewart, Academic Practice Development Consultant
Email: alison.stewart@ntu.ac.uk
1. Kirkpatrick, Donald L., and Kirkpatrick, James D., (2005) Evaluating Training Programs: The Four
Levels (3rd Edition) San Fransisco: Berrett-Koehler Publishers
2. Bromley, T., Metcalfe, J., and Park, C., (2008) ‘The Rugby Team Impact Framework’ ISBN-13: 978-
1-906774-00-4 www.vitae.ac.uk/ieg
86/222
HEA CPD Pilot Workshop - Workbook
2.4.5 Case Study – Queen Margaret University, Edinburgh
Queen Margaret University (QMU) is a small post-1992 university which dates back to 1875,
but was granted full university title in 2007. The University is located to the east of
Edinburgh, and has a portfolio of socially and economically relevant educational
programmes that are organised within 2 Schools.
The School of Arts, Social Sciences and Management comprises of: Media and Performing
Arts; Psychology and Sociology and Business, Events and Hospitality Management. The
School of Health Sciences encompasses the subjects of: Dietetics, Nutrition and Biological
sciences; Nursing; Occupational Therapy and Arts Therapies: Physiotherapy: Podiatry;
Radiography: Speech and Hearing Sciences and The Institute for International Health and
Development.
QMU has around 6000 students and 470 staff. The university also has a broad portfolio of
undergraduate and postgraduate programmes with international partners in Egypt,
Greece, India, Nepal and Singapore.
Professional development opportunities for staff
Staff from the Centre for Academic Practice (CAP) contribute to policy and strategic planning,
collaborate with all departments and units, and provide professional development for staff and post-
graduate students in all aspects of learning, teaching and assessment. The breadth of CAP’s offer is
illustrated in their annual programme handbook, and includes a series a short courses, bespoke
workshops for programme teams or schools, and support for staff wishing to apply for HEA
Fellowship.8 QMU also runs a Master’s in Professional and Higher Education (PGC PHE), of which the
Post Graduate Certificate is HEA accredited. This is an applied programme, founded upon theories of
inclusive learning and reflective practice, relevant and of interest to practitioners across all
subject/practice areas. Since 2013, all new lecturers undertake this course.
The in-house CPD Scheme was piloted at QMU in 2013 and is optional for experienced members of
staff. No university targets have been set for the achievement of Fellowship. The emphasis is very
much on enhancement.
8 http://www.qmu.ac.uk/cap/PdfFiles/ProgrammeCurrent.pdf
87/222
HEA CPD Pilot Workshop - Workbook
Evaluating the effectiveness of teaching-related CPD
Evaluation of credit-bearing CPD programmes (such as the PGC PHE) is relatively ‘straight forward’
because there is a requirement for annual programme monitoring, which includes the involvement
of students and an external examiner. In Scotland, quality monitoring takes the form of
Enhancement-Led Institutional Review (ELIR), within which institutions examine and monitor their
own quality processes, and express this in a Reflective Analysis. The Quality Assurance Agency
undertakes a cycle of quinquennial visits to review the findings of the institutional self-review.”
Questions such as ‘How are we developing the staff who deliver the PGC PHE programme?’, and
‘How are we enhancing teaching practices on the programme?’ are included in the annual
programme monitoring which feeds, eventually, into the ELIR process.
‘With that kind of formal provision, evaluation is embedded and what is really important is that the quality assurance and the quality enhancement agendas do very clearly join up. And then evaluation is linked in to university processes; for example, people will talk at their performance enhancement review [appraisal] about their progress on the programme. So in those formal programmes then evaluation is reasonably straight forward. It’s not innovative, but it is embedded’. Roni Bamber, Director of Centre for Academic Practice
In 2010, a substantive piece of research was undertaken to explore the evaluative practices of
academic staff at QMU, to ascertain what was done beyond formal requirements. Two members of
CAP issued an online survey to around 200 academic staff (see Bamber & Anderson, 2012) and found
that a great deal of evaluation of teaching and learning goes on, but this isn’t always reflected in
formal reporting structures. Much evaluation is informal and local, and often falls below the radar of
formal monitoring.
Roni Bamber suggests that we need to move away from thinking about ‘impact’ and instead consider
‘evidencing value’. Drawing on her experience of running the HEA-accredited CPD Scheme at QMU
she describes a triangular approach that considers three distinct types of data (literature, feedback
from review processes and ‘practice wisdom’) that can be analysed with to demonstrate the value
of CPD activities (See Bamber, 2013).
‘The first element is Research: what is the literature and what are the theoretical frameworks that we are aware of, that would support what we are doing and help us keep developing? Second,
88/222
HEA CPD Pilot Workshop - Workbook
Evaluation: what feedback do we get from externals, from panel members, from participants in the CPD scheme and, also, what might we pick up from their applications that would suggest the value of the Scheme? (And we know there is great value in applying for fellowship – it’s a development activity in itself). And the third type of data is Practice Wisdom: what do we learn from discussions with staff, and from our knowledge of what is appropriate in our context. What do we pick up informally from staff that tells us the value to them of going through that CPD process? Collectively that can be quite rich. It’s quite hard to capture, but that’s one of the areas that I think is most promising, that is probably under-appreciated because it is informal data which may look ‘unscientific’ to anyone taking a very technical-rational stance on evaluation.’
The small size of the institution means that a significant amount of evaluation takes place informally,
within that practice wisdom frame:
‘A lot of what we do happens in the informal domain because everybody knows each other and talks to each other in the corridor and you get to learn all sorts of things about people’s teaching practice and what’s going on with their students, and that comes in to that practice wisdom area. It is really valuable information but it’s not necessarily written down’. Roni Bamber, Director of Centre for Academic Practice
Challenges to evaluating CPD
The biggest challenge to evaluating CPD is that there is no direct cause and effect relationship, and
so impact is always by implication, by interpretation and association.
‘We can only go by what they [teachers] tell us, and usually it’s things like ‘I’m a lot more confident’; ‘I’ve got a greater repertoire of techniques’; ‘I feel as if I am responding better and learning better’; but it is hard to produce concrete evidence of that.’ Roni Bamber, Director of Centre for Academic Practice
Roni has been involved in the evaluation of teaching and learning for many years. In a
previous institution she collected data from PGCert participants over a period of nine years,
and has used a number of tools such as the Approaches to Teaching Inventory(Trigger and
Prosser, 2004), and the Teaching Methods Inventory (Caffey and Gibbs, 2004).
‘We also designed a confidence questionnaire and we did try to find some kind of patterns in the data between entry and exit over that period of time. But there wasn’t, maybe because the numbers weren’t that great. There were no concrete data that we could call significant patterns. So measurable data is hard to establish, but we do get lots of rich evaluation data from discussing with staff their day to day practices, and how they’re developing those. The challenge is to turn that into something meaningful for reporting purposes, in this era of impact and measurement.’ Roni Bamber, Director of Centre for Academic Practice
89/222
HEA CPD Pilot Workshop - Workbook
Future Plans
Bamber suggests that evidencing the value of academic development is a challenging aspect of
evaluating something as intangible as the benefits of developing staff. The staff at CAP will continue
to gather data based on Research, Evaluation and Practice Wisdom, and are looking at how best to
combine formal and non-formal sources of data in ways that suit different purposes (eg for
reporting, for improving our CPD Scheme, for planning future provision).
90/222
HEA CPD Pilot Workshop - Workbook
Work Package 3 (WP 3): Design and test a tool for the assessment of the effectiveness of CPD
This WP was informed by the outcomes of the literature review and national audit, as well as
conversations with interested parties over the course of the project. This was a dynamic WP, which
had to be responsive to the tight timeframe of the project and the complex demands associated
with evaluating the impact of CPD across a diverse HE sector.
3.1 Tool Design and Development
The original intention of this project with respect to ‘tool development’ was to ‘customise,
refine and extend the tool developed by Chalmers et al., (2012)’ in their work on evaluating
teacher preparation programmes. Chalmers et al. (2012) highlighted a number of features of
their tool intended to promote its widespread application. These included flexible design,
the broad evidence base that it encouraged users to engage with and the potential for
capturing reflections over extended timescales (Chalmers et al., 2012). Whilst these were
recognised strengths, we identified several other researchers in this area who had made a
significant contribution to the study of ‘evaluation’ in the context of HE and school-based
education.
Amundsen and Wilson (2013) show that to evaluate the potential impact of CPD activities
effectively you need to consider the rationale underpinning their design. They highlight the
importance of planning the design and evaluation of CPD at the same time.
Cashmore et al. (2012) demonstrate the complexity of the data in career progression
research and the challenges associated with collecting it.
De Ridjt et al. (2013) identify factors that determine the impact of CPD such as participants’
motivation to engage in CPD, and the complexity of ‘transfer’, that is integrating learning
from a CPD activity into practice.
Fink (2013) identifies the range of ‘stakeholders’ that need to be considered in order to
evaluate the impact of CPD. These stakeholders represent institutional leaders, CPD
providers, students and teachers.
Guskey (2002) and Kreber and Brook (2001) both discuss and critique the complexity of CPD
and the varying levels at which we need to evaluate potential impacts. Guskey (2002)
identifies the following levels: participants’ reactions, participants’ learning, organisational
support and change, participants’ use of new knowledge and skills and student learning
outcomes. Kreber and Brook (2001) identify participants’ perceptions, participants’ beliefs
91/222
HEA CPD Pilot Workshop - Workbook
around teaching and learning, participants’ teaching performance, students’ perceptions of
staff’s teaching performance, students’ learning and effects on the culture of the institution.
We drew on the work of Chalmers et al. (2012) and Cashmore et al. (2012) and integrated the ideas
presented by the writers above to inform the philosophy and framing of the tool as well as its
practical use and applicability. The findings of the WP2 national audit were also integral to the
development of the tool.
We originally used the published literature (e.g. Gosling, 2008; Ling, 2009; Parsons et al., 2012;
Prebble et al., 2004; Turner et al., 2013) to determine the scope of the tool with respect to the range
of the CPD activities it would evaluate. At the bidding stage, we anticipated developing resources to
support the evaluation of teaching preparation programmes, teaching accreditation schemes,
conferences, staff development workshops, peer review and teaching development grants.
However, following the national audit (WP2), we extended the scope of this to include mentoring
schemes and collaborative and networking activities. Such activities have been identified in the
published literature as of growing significance in terms of CPD and to date have received limited
attention in the evaluation of teaching-related CPD (see WP1 and, for example, Section 1.4.2)
3.1.1 Influential factors shaping the development of the tool
In order to take account of existing processes and mechanisms to evaluate CPD activities, we used
the academic community as a starting point for our development work. Project team members
drew on the expertise in their networks in order to identify existing CPD evaluation resources. They
also reviewed available resources in the grey literature and resulting from grants awarded by
organisations such as SEDA, the HEA and its former subject centre network. Additionally, we also
drew on examples from our own institutions. In reviewing these existing resources we identified the
following factors as integral to the design of the tool to ensure it promoted a critically reflective
approach to evaluating the impact to CPD:
1) The scope and focus of the tool - Many existing resources seek to capture evaluation data around
satisfaction. Our primary focus would rather be on identifying changes in thinking, practice,
behaviour and action (Bamber, 2013; Rust, 1998; Sword, 2011). These are integral to identifying the
impact of CPD over the longer term and the effects it has on student learning. They also encompass
the so-called ‘soft’ impacts such as increased confidence, thinking differently, and a willingness to
change practice, which benefit student learning but are challenging to measure (Bamber, 2013). We
framed these changes with respect to the following categories identified within published studies on
92/222
HEA CPD Pilot Workshop - Workbook
CPD which discuss the common goals and anticipated outcomes of CPD (e.g. Guskey, 2002; Kreber &
Brook, 2001):
Changes in participant conceptions of teaching and learning
Changes in participant teaching behaviour
Changes to students perception of teaching
Changes to student learning outcomes
Changes to organisational culture around teaching and learning
2) Applicability – Fink (2013) and WP2 (Section 2.4.4) clarified the range of stakeholders who may
use the tool and interact with the outcomes of an evaluation resulting from the tool. We had to
ensure the tool was sufficiently flexible to allow these different groups to evaluate their CPD
activities and result in data on impact relevant to these different groups.
3) Promoting a longitudinal approach to evaluation - The ability to evaluate over time is essential to
identify potential impacts across the categories identified above (Bamber, 2013). The tool would
therefore need to include provision to capture data across extended timeframes (eg during and after
a CPD activity, including prior to the event to provide ‘baseline’ data, if relevant, during i.e. if the
CPD activity is more than one-day in length), immediately afterwards and then six, or 12 months
after the CPD activity).
4) Drawing on qualitative and quantitative data – Drawing on a range of data, and therefore
evidence would ensure a more reflexive approach to evaluation, which is therefore sensitive to the
complex nature of impact (Fink, 2013; WP1 section 1.5. The value of using qualitative data to
evaluate practice has been widely advocated. Considering this alongside quantitative measures, can
develop a more nuanced understanding about commonalities and differences across different
approaches and contexts for CPD in teaching and learning (Chalmers, 2008; Shavelson, 2010).
5) Promoting a flexible approach to methods of data collection. In order to ensure the tool is
responsive to a diverse range of HE contexts and CPD activities, it should enable flexible data
collection. Currently, most evaluation practice is centred on the short answer questionnaire (see
WP2, section 2.4.3) but this tool offers the potential to counter this trend and consider the range of
ways in which evaluation data can be captured. It could, for example, promote the use of focus
groups, interviews, or reflective logs for data collection.
6) Encouraging those engaged in evaluation to reflect on a range of data. Encouraging individuals
and teams to draw on multiple sources of evidence - including existing data - would ensure an
evaluation is robust and captures the range of impacts CPD activities can result in. Such data might
93/222
HEA CPD Pilot Workshop - Workbook
include self-reflections, module evaluations, feedback from colleagues, reviews of data on areas
such as student performance, retention, progression and NSS activities.
3.1.2 Tool development
The first key question concerned who would be conducting the evaluation. We identified the CPD
provider, for example the Educational / Academic Developer or HE manager in FE contexts) as most
likely to be taking the lead in evaluating the impact of CPD activities. These are the groups with the
remit for teaching-related CPD and are well placed to co-ordinate a systematic approach to impact
evaluation using this tool. Their usual central location in institutional structures offers the potential
for the outcomes of evaluation activities to inform future CPD and teaching and learning
development work across an institution (McKenna & Hughes, 2015). We recognise the
organisational structure may be different in an FE college or private provider, though we anticipate
there is someone in these institutions with a remit that encompasses activities such as staff
development or quality enhancement. Therefore the tool targeted this group at the design stage.
The toolkit aims to guide the evaluator through the process, from identifying the focus and purpose
of the evaluation, choosing evaluation questions relevant to stakeholders, identifying ways to collect
evidence relevant to the questions, to conducting the evaluation and making use of the results.
The CPD activity-type was chosen as the starting point for evaluation planning. We identified this as
an accessible entry point on which to focus evaluation activities that would work across a range of
institutions. For each activity, we then developed lists of open questions for the potential
stakeholders. The workbook in App 2 illustrates this approach, with sample questions relating to the
impact of attending a conference. The different stakeholder groups (e.g. providers, completers,
students, etc.) were addressed in the following ways:
1) Prompts were written for the CPD provider to trigger reflection on the design and framing of
the CPD, as well as examining their rationale in undertaking the evaluation and the expected
outcomes / implications for practice;
2) Questions were designed for selection by the CPD provider to evaluate the impact of a CPD
activity on the CPD completer.
3) Questions were themed around the categories outlined above (e.g. changes in conceptions
to teaching and learning). Additional questions were included to capture data around CPD
completers’ motivation and reaction to a CPD activity, since these have been identified as
having an impact on whether the intended outcomes of a CPD activity are realised and/or
whether learning is transferred to practice (e.g. De Ridjt et al., 2013).
94/222
HEA CPD Pilot Workshop - Workbook
4) Questions were also designed for students who may have contributed to the design or
implementation of a CPD activity. These sought to capture the students’ involvement in the
CPD activity and the potential impacts on their academic development. The questions for
the CPD participant and students were presented over an extended timescale, giving the
potential for evaluation to be conducted prior to and during a CPD activity, which could
provide useful data on expectations and motivations, as well as initial reactions to a CPD
activity and potential changes to practice that may result from engagement with a CPD
activity. To prompt on-going reflections on CPD activities and their impacts we identified
common questions, which were asked at different time points to allow a participant to
revisit past reflections and use these as a stimulus for future reflections. Depending on the
CPD provider’s responses to the initial questions regarding the design of the CPD activity and
the purpose of the evaluation, they are then directed to select questions from each of the
recommended timeframes for completion by CPD participants. The CPD provider then
collates the selected questions and identifies the method of answering these.
The toolkit also provides guidance on the following areas:
1) Evaluating CPD completes participants’ reactions and experience of a CPD activity – currently
this is the focus of much evaluation practice, and though it provides limited information
regarding the impact of CPD activities, it does give essential information on areas such as the
content and delivery of a CPD activity, the facilitator and the administration, facilities and
logistics. (This immediate evaluation also provides baseline data against which to consider
longitudinal findings.)
2) Methods of data collection – in order to encourage CPD providers to consider alternate ways
of evaluating practice, the toolkit includes a brief overview of the potential role of
questionnaires, interviews and focus groups in evaluation. Information was provided on the
strengths and challenges of each method of data collection, methods of implementation and
ethical considerations. Additionally, a resources list to assist in these collection methods is
provided.
3) Using evaluation data - The use of evaluation results is potentially sensitive and there may
be tensions between the need to inform further educational development and a desire to
monitor or judge performance. The tool therefore includes guidance relating to
communicating information about CPD impact, guiding understanding, formats and media
for reporting findings, responsibilities to stakeholders and other ethical issues.
95/222
HEA CPD Pilot Workshop - Workbook
3.2 Tool Implementation and Piloting
Initially, the tool seemed best suited to online presentation so that different views could reflect the
variety of stakeholders, perspectives and evaluation aims. However, following the design and review
of the tool by the project team, the tool was presented as a hardcopy ‘workbook’ for piloting
nationally. (See Appendix 2) The decision to present the tool as a workbook rather than online
resource was to ensure that, at this stage, we received feedback on the content and focus of the tool
as this was the primary concern. It was felt that until these aspects of the tool were finalised,
piloting the tool as an online resource was premature. Usability issues could be distracting, with
potential implications on the quality of the feedback.
The national pilot was focused on individuals with a remit for staff and educational development
work across the HE sector. This included representatives from two FE colleges, two private providers
and universities, covering research and teaching focused institutions. Originally we intended to
include eight institutions, in addition to the two project partners, in this pilot but, following receipt
of funding, a further two institutions were invited to be participate as this would increase the
breadth of CPD activities on which the tool would be piloted. The following 12 institutions were
involved in the pilot:
Bangor University
Bath Spa University
Bradford College
Durham University
Greenwich School of Management
Manchester Metropolitan University
Nottingham Trent University
South Devon College
University of Bath
University Campus Suffolk
University of Greenwich
University of Edinburgh
A named contact from each institution was invited to a one-day workshop at the University of
Birmingham on the 20th of March 2015. The purpose of this day was to present progress with the
project to date, to introduce the tool, gather immediate feedback and prepare for the pilot process.
96/222
HEA CPD Pilot Workshop - Workbook
Ten of the pilots were able to attend, with the remaining two contacted at a later date via Skype to
introduce the tool and undertake the necessary preparation.
Following the one-day workshop, the CPD providers had three weeks in which to review the tool and
prepare to evaluate their CPD activities. They were requested to identify CPD activities that could be
evaluated using the tool. Depending on the breadth of their institution’s CPD offer and individual
interests, they chose to focus on either one CPD activity or a range of activities. Using the tool, they
identified the focus of their evaluation, selected evaluation questions and the method by which the
evaluation would be conducted (e.g. as an interview or questionnaire). Following this, they returned
the workbook to the project team. We requested return of the workbook as this represented an
important source of feedback on the tool to inform its on-going development.
The pilot CPD provider also identified three or four members of staff (CPD completers) who had
engaged with the chosen CPD activity in the past 12-24 months and would be willing to assist with
the pilot. The CPD provider and Plymouth team then arranged a mutually convenient date for a
member of the project team to visit the institution, meet the CPD completer(s) and undertake the
evaluation. During this meeting the CPD completer would evaluate their CPD experience, and
potential impacts on practice, using the questions and the method chosen by the CPD provider.
They would also be interviewed about the process of evaluating their CPD activity. This interview
captured the CPD completer’s reflections on the questions they responded to, considering their
accessibility and relevance, as well as contextual information regarding the CPD activity under
consideration. The meeting and subsequent interview was informed by a ‘think aloud’ methodology
(Cotton & Gresty, 2006), and the interviewees were encouraged to reflect on the questions they
were asked. In total 40 interviews were held with CPD completers from across the pilot group.
The meeting with the CPD completer was audio recorded and transcribed in full. Data were
thematically analysed by the project team. Three members of the project team undertook analysis
of the transcripts from the first pilot individually, and then came together to discuss and refine
emerging themes. Following this, an analytical framework was developed and subsequently applied
to the whole data set. The outcomes of this analysis provided insights into completers’ experiences
of the CPD they had engaged with and its impacts on their practice. Insights were also gained into
the process of evaluating their CPD activity using the tool. These are reported in section 3.4.2.
97/222
HEA CPD Pilot Workshop - Workbook
Based on the first category of information, brief summaries were developed of the collective CPD
experiences of CPD completers for return to the CPD provider. Each summary touched upon themes
pertinent to CPD e.g. individuals’ motivation for engaging with the activity, impacts on areas such as
conceptions of teaching and learning, teaching behaviour, future CPD and suggested improvements
to each type of CPD activity based on the data collected. These summaries gave an indication of the
potential outcomes of the tool as well as offering anonymised feedback on the overall breadth of
CPD provision represented by the pilots. These were returned to the CPD provider to review prior to
the final stage of data collection – an interview to capture their experiences of using the tool to
inform the final refinements to the tool.
3.3 Tool Evaluation
The tool was evaluated based on the following sources of data:
1. A review of the workbooks and the questions CPD providers selected to evaluate their
practice;
2. The outcomes of the analysis of the CPD completers’ evaluation data and reflections on the
questions they were asked to respond to;
3. A review of the summaries produced regarding the impacts of CPD activities on participants’
practice.
Based on these data sources, an interview schedule was developed to capture the CPD providers’
perceptions and experiences of using the tool. The schedule specifically explored the CPD
providers’ roles with respect to CPD and its evaluation, their approach to using the tool and rationale
for choosing the evaluation questions and methodology used. We also solicited feedback on the
tool and recommendations for on-going development of the tool. Ten CPD providers were
interviewed via Skype by a member of the Plymouth project team. Interviews were audio recorded
and transcribed in full.
98/222
HEA CPD Pilot Workshop - Workbook
3.4 Results of the national pilot of the evaluation tool
In this section we present the findings of the piloting of the tool within 12 UK-based institutions.
Providers for teaching-related CPD used the workbook to select questions and appropriate methods
of evaluating the impact of CPD activities previously engaged in by members of their teaching staff.
40 members of teaching staff selected by the CPD providers were then asked to answer the
evaluation questions in an interview situation. The interviews were carried out by a research
assistant from Plymouth. We also report on data collected from SU representatives from seven of
the pilot institutions. These data have provided insights in the following areas:
1) CPD completers experiences of using the tool to evaluate the impact of previous CPD
activities they had engaged with9;
2) CPD providers experiences of using the tool to evaluate their CPD provision captured
through an analysis of the workbooks (e.g. questions and methods selected) and subsequent
interview to discuss the evaluation process and the outcomes of the CPD completer
interviews;
3.4.1 Piloting of the tool
The CPD completers involved in the pilot represented the full range of CPD activities we identified in
WP2, Section 2.3.1. In brief, most CPD providers focused their evaluation activities on teaching
courses for new lecturers and in-house accreditation schemes (Table 3.1), therefore focusing on
formal CPD provision aligned with the UKPSF. This was not unanticipated as these are areas of CPD
often perceived as core to educational development work, and activities individuals are most likely
to engage with due to institutional drivers including probation and promotion (Smith, 2011;
McKenna & Hughes, 2015). Also, it is the CPD that affords ‘recognised status’ that institutions are
now required to report as part of their data return to HESA (HESA, 2012). Activities such as teaching
and learning workshops, teaching and learning conferences, peer review, teaching and learning
development projects and mentoring, were included in the pilot by fewer CPD providers (Table 3.1).
Although these are CPD activities that are common across the sector, and serve an important
education development and enhancement function, engagement with these activities tends to be
voluntary and more at the determination of the individual academic rather then due to a central
driver / institutional requirement (Parsons et al., 2012). Therefore in terms of prioritising which
9 As a consequence of piloting the tool the CPD completers reflected on the CPD activity they had engaged with which they were now evaluating. These reflections were collated and short summaries were produced for each CPD activity which summarise the perceived impacts on practice and suggestions of how the CPD activity may be improved based. These summaries were discussed with the CPD providers as part of the evaluation,
and are included in Appendix 3.
99/222
HEA CPD Pilot Workshop - Workbook
activities to evaluate this could have been a clear rationale underpinning provider choice. Likewise,
some HE providers e.g. private providers or FE colleges may have smaller CPD provision compared to
universities, and therefore this automatically determined which activities the CPD provider chose.
Table 3.1: Range of CPD activities included in the pilot
CPD activity Number of CPD completers the tool was tested on*
Teaching courses for new lecturers 12In house accreditation Schemes 14Other CPD activities** (Teaching and learning development projects, teaching and learning workshops, teaching and learning conferences, peer review and mentoring)
14
* Some institutions chose to pilot the tool on a number of different CPD activities; therefore although 12 institutions were included in the pilot, the number of activities the tool was piloted on was greater.
** For the rest of the chosen CPD activities the number of CPD completers piloting the tool was relatively low. Due to our ethics and the requirement to maintain confidentiality we have not stated the exact number of CPD completers piloting the tool across the rest of these activities.
CPD activities have the potential to impact upon a number of different aspects of an individual’s
practice (e.g. their conceptions of teaching and learning, the institutional culture and students
(Guskey, 2002). The impact a CPD activity has on these different areas is influenced by an
individual’s motivation to engage with a CPD activity, their experience in their role and their reaction
or response to it (De Ridjt et al., 2013; Parsons et al., 2012). As many of our CPD completers had
been involved in teaching courses for new lecturers that are often a probationary requirement
completed by those new to HE-level teaching, the level of learning that participants engaged with
may have been highly variable. This could therefore have influenced the evaluation they undertook
of this CPD activity and the learning they identified.
In designing the tool we themed the questions to cover the key areas of impact CPD activities are
recognised to have as identified by Guskey, 2002 (see Appendix 2). Although we discussed the
relevance of each theme to assessing impact of CPD provision, we left it to the pilots to select their
questions according to the aims of their evaluation and the learning outcomes / goals of their CPD
provision. Table 3.2 summarises the number of questions selected from each theme. The data are
based on frequency counts across each CPD activity. It may be that where a CPD provider was
evaluating more than one activity there was repetition in the question, hence the higher frequency
100/222
HEA CPD Pilot Workshop - Workbook
counts for some themes. These data highlight patterns in question and theme choice relevant to the
on-going development of the tool.
The majority of the CPD providers selected questions that would capture data regarding CPD
completer’s motivation / reaction to a CPD activity, measure changes in conceptions of teaching and
learning, changes in teaching and learning behaviour and impacts on institutional culture. These
areas are the focus of much existing evaluation practice (e.g. Lau & Yuen, 2013; Lee et al., 2013) and
align with the recognised benefits of teaching-related CPD (Stes et al., 2013a; Willets, et al., 2014).
Very little emphasis was placed on questions that would provide insights into changes in students’
outcomes or changes in student perceptions of teacher behaviour. This was not unexpected, as
these are widely recognised as the challenging areas in which to evaluate and gaps within current
evaluation practice (Parsons et al., 2012; Trigwell et al., 2012). They do indicate the need for
inclusion of clear guidance within the tool of ways in which they can be measured and how, for
example, existing data sets (NSS / in house student satisfaction surveys) can be used to provide
evidence of impact.
101/222
HEA CPD Pilot Workshop - Workbook
Table 3.2: Frequency counts of the themes from which the questions were selected across the range of CPD activities included in the pilot.
Teaching courses for new lecturers
In house accreditation Framework
T&L workshop
T&L conference
Peer Review
Teaching and learning development projects
Mentoring Total
Motivation 25 21 4 1 1 4 1 57Changes in conceptions of teaching and learning
20 18 1 3 1 2 45
Changes in teaching and learning behaviour
23 15 2 1 4 1 46
Changes in student perceptions of teacher behaviour
2* 1 1 4
Changes in student outcomes 13 8 1 3 25
Impacts on institutional culture 14 26 7 1 1 7 1 57
* These questions were added by a CPD provider
102/222
HEA CPD Pilot Workshop - Workbook
3.4.2 CPD completers experiences of using the tool
The CPD completers had also engaged with their CPD activities over differing timeframes (Table 3.3). This
provides some provisional insights into the potential of the tool to evaluate impact over extended timeframes.
As data were only collected from one point in time rather than over repeated points these findings are tentative
Despite this, we did gain insights into the impact of timeframe upon the extent to which CPD completers were
able to evaluate the impact of their CPD activity on their practice. For example, the CPD completers piloting the
tool often needed prompting with respect to what the content or focus of the CPD activity was. This was
particularly evident where there had been a considerable time lapse between the CPD activity and the
interview. Prompts were included to stimulate these reflections, however, regular engagement with impact
evaluation following completion of a CPD activity, as advocated through this tool, may help overcome issues of
recall.
The difficulties the CPD completers experienced in recalling the CPD activity and its potential impacts could also
be indicative of the positioning of the CPD activity. That is to say, it was frequently perceived as being
undertaken outside of the ‘everyday role’ of being a lecturer and supporting students (Han, 2012).There was
therefore an absence of regular dialogue with colleagues and peers to integrate learning into practice (Brown &
Inlis, 2013). Regular dialogue has been identified as integral in maximising the learning that can take place from
CPD, and ensuring the so-called transfer to practice (Brown & Inlis, 2013). Struggling to reflect on their
experience and identify the learning that had taken place indicates that the opportunities to enact or practice
the knowledge, skills and experiences presented in their CPD may have been limited. Engaging in regular
dialogue and reflecting on their CPD experiences through the tool could create such opportunities.
Table 3.3: Timeframes over which the CPD completers had engaged with their CPD activities that were the subject of the pilot.
CPD activity TimeframePre During Post 6 months 12 months+
Teaching and learning 2 3 4 5 3
103/222
HEA CPD Pilot Workshop - Workbook
development projectIn house accreditation scheme
3 2 5 3 4
Other CPD activities*
1 1 1 3 2
* Again data amalgamated as in Table 3.1
The clarity and accessibility of the questions was an issue for a number of the CPD completers. Educational
development work is recognised for having a language of its own (Green, 2009), and although a CPD completer
may have been cognisant of this at the time of undertaking the CPD activity, the use of such context specific
language was identified as problematic for some. The language in educational development is also highly
variable, often changing across institution types and indeed counties (Green & Huston, 2006). This highlights the
need for the CPD provider to contextualise the tool to their own institutional context.
Accessibility and language was also a key issue for those for whom English was a second language and for one
dyslexic participant who expressed concerns regarding the accessibility of the questions. In some cases the CPD
completers were provided with the questions in advance to the data collection. This was recognised as
advantageous, as the CPD completers felt it allowed them to engage with a more in depth reflection on their
experience and the potential impacts on practice:
“If I’d have had [the questions] beforehand, and had a bit of time to think about them, then I would
have found it easier to answer the questions because I would have had that processing time.” (Pilot 3)
The CPD completers also highlighted the need for a clear and logical flow to the evaluation questions. Integral
to this was the space for discussion and reflection on the specific CPD activity that was the focus of the
evaluation.
“For me yes I think the discussion we have just had has made me realise how far I have come in the last couple
of years, it’s made me think about what I’m doing and why I’m doing it and what it means to me and what it
means to everyone else so yes reflection is always good and that was a good half hour of reflection we had.”
(Pilot 12)
In terms of developing an approach to evaluation that takes place over extended timeframes, it highlights the
need for common questions or themes to allow the CPD completer to revisit past experiences and identify
104/222
HEA CPD Pilot Workshop - Workbook
future learning. This would also lead to a gradual shift in evaluation practice from the widely critiqued focus on
outputs to outcomes, learning and change (Stefani, 2011).
Current evaluation practice commonly involves the use of ‘satisfaction’ questions, for example those that assess
whether the CPD participant was happy with their experience. This approach has been discussed in the
literature (e.g. Amundson & Wilson, 2012; Chalmers and Gardiner 2015), as providing limited insights into the
potential impacts of CPD. We do recognise they are useful in terms of assessing the immediate experience of a
CPD participant with an activity or the quality of learning promoted by a particular facilitator. Therefore in
presenting the tool to the CPD providers we acknowledged the potential limitations of these questions and
guided caution in their use. Consequently only one CPD provider selected such a question. However, CPD
completers were keen to report their level of satisfaction with a CPD activity and indicated this was an important
part of the evaluation process.
3.4.3 Recommendations for the further development of the tool based on the analysis CPD completer data
Consider the needs of both the CPD completer and CPD provider in the recommendations regarding
questions choice. Although the CPD may be more focused on the impacts on practice, questions also
need to be included that allow the completer to reflect on the actual CPD activity and learning that
resulted, and offer feedback on their experience of the CPD activity.
Guidance should be given to the CPD provider on the ordering of questions. ‘Exemplar’ question sets for
each CPD activity may be useful.
Guidance should be given to the CPD provider regarding the need to contextualise the questions to their
institutions, including relevant terms of reference to CPD activity, the Educational Development unit etc.
Giving the CPD completer time to review the questions prior to engaging with the evaluation would
promote inclusivity and allow time for greater reflections to be made on their experiences.
3.4.4 CPD Providers experiences of using the tool
As Table X indicates, most CPD providers focused on evaluating a specific CPD activity, rather than considering
the specific educational development themes (or learning outcomes) that underpinned their design. This use of
the tool does reflect its presentation (see Appendix X - workbook), in that it is focused on the CPD activities.
Within the CPD literature, most evaluation work is centred on the activity itself (Bamber, 2013; Parsons et al.,
2012). However, we have attempted to draw attention to these wider themes through the organisation and
105/222
HEA CPD Pilot Workshop - Workbook
presentation of the questions related to each CPD activity. This was identified as useful by the CPD providers as
they felt it served as a prompt in question choice.
Table 3.4: What the CPD providers were using the tool to evaluate
Purpose of the evaluation N
A named activity 10Changes in conceptions of teaching and learning 2Changes in teaching and learning behaviour 2Changes to student learning 3Changing identity as a teacher 1
The CPD providers identified a number of motivators underpinning their desire to evaluate their practice, all of
which resonate with current practice across the sector (e.g. Bamber, 2013). Educational developers are
expected to provide evidence of impact in numerous forms for a range of audiences (Bamber, 2013). Despite
high numbers reporting that evaluation feeds into future CPD offers there was little evidence or interest in
evaluating the on-going value of CPD on the individual academic. There is clear benefit in tracking individual
academics continued interactions with CPD, teaching and scholarly activity as a way to evidence value and
establish future CPD need. Indeed, given agendas inherent to the Browne Review (2010), such as the publicising
of teaching qualifications by HESA and the agenda for good standing, tracking individuals on-going interaction
with CPD is likely to be of increased relevance to policymakers and institutional managers. This is a potential
outcome from using the tool which may fulfil a gap within current evaluation practice.
Table 3.5: The purpose of the evaluation
Why evaluate? N
To inform future CPD offer 8To inform institutional policy 6To articulate and evidence value 3Evidence for internal auditing 3Evidence for QAA audit 2Individuals’ on-going academic development 2
A range of audiences were identified as recipients of the impact evaluation data (Table 3.6). It is likely that
senior managers and committees will be provided with brief updates which can help to inform decisions about
the value of CPD activities. More nuanced narrative data will better inform future CPD and assist CPD providers
in understanding issues around transfer to practice (De Ridjt et al., 2013). The toolkit could therefore usefully
provide advice about what sorts of data are appropriate to use for different audiences.
Table 3.6: The intended uses of the evaluation data
106/222
HEA CPD Pilot Workshop - Workbook
Use of evaluation data NInforming future CPD including development of metrics 5Reported to senior management 3Presented at committees 2Reported to HR 2Reported to QAA 1Reported to respondents 1
Seven pilots chose to use interviews as their preferred methodology to collect evaluation data from the CPD
completers, with the remaining two using open questionnaires. These approaches follow trends in evaluation
practice which advocate the importance of qualitative approaches to data collection and the potential to
develop greater insights into the impacts of CPD through narrative accounts (Bamber, 2013; Rust, 1998).
However, the intended audiences of the evaluation data are mixed (Table 3.6), and it is likely that some, such as
university committees and senior managers, who may be more interested in a headline summary of satisfaction
or numerical overviews of impacts. Therefore mixed method approaches should be considered in order to be
able to communicate the outcomes of evaluation work in both qualitative and quantitative formats.
The CPD providers identified numerous benefits following the use of the tool to evaluate their provision. They
acknowledged its flexibility and the ease at which it aligned with, but also enhanced, their current evaluation
practices, particularly in terms of promoting a longitudinal approach to evaluation. This was a current gap in the
evaluation provision of most of the pilot institutions.
‘One of the first challenges you often have is coming up with the evaluation questions, so actually
having a readymade resource that has started to categorise things into different evaluation
scenarios, and starting to give you different questions for different levels of evaluation, we found
that really helpful…. you’ll end up with better evaluation tool for whatever you design because
some of that thinking and working through and revision have been done already..’ (Pilot 7)
In line with the concerns from the CPD completers regarding the language used in the tool the CPD providers felt
that where necessary there was the potential to customise the tool to ensure it was sensitive to their own
institutional context. The recognition of the need for this by CPD providers is important, as it could mediate
potential accessibility and language issues raised by the CPD completers.
The comprehensive nature of the tool was also valued, in that it covered the breadth of CPD provision, and
allowed robust, longitudinal evaluations to be conducted across this range of activities:
107/222
HEA CPD Pilot Workshop - Workbook
‘We thought the theme ones are really good, so the themes cover pretty much everything that we
wanted to look at anyway. There were a few ones that we added about the UKPSF and whether the
students know that lecturers are engaging with CPD activities.’ (Pilot 11)
‘Given that it’s a hundred percent more than most people are using ever, just taking it from where
it is at the moment would be really useful.’ (Pilot 7)
They also felt the connection within the tool to academic development themes and / or direction to considerer
the learning outcomes of a CPD activity improved alignment in evaluation practice, overcoming a noted
limitation of much evaluation practice (Guskey, 2000).
The CPD providers identified benefits to those who engaged in the evaluation process, citing the value of
engaging in meaningful dialogue around their previous CPD and the contribution this could make to planning
future CPD. Again this could benefit the growing agenda for ‘remaining in good standing’ as advocated through
the UKPSF. The process of reflecting on their learning was also seen as valuable in terms of making connections
between their individual practice and wider institutional processes and priorities.
Through engagement with the tool, both in selecting questions and reflecting on the completers collated
responses to the CPD activity, providers became aware of their current practice, identifying limitations and also
how they would seek to develop their evaluation practices in the future. CPD providers seemed acutely aware
that their current evaluation was focused on satisfaction rather than evidencing value as noted elsewhere (e.g.
Bamber, 2013). This was attributed partly to historical precedence but also to institutional administration
processes. Providers’ current evaluation methods included some focus groups and exit interviews but centrally
designed module evaluation forms were widely used. In each case there was critique of the extent to which the
module evaluation questions reflected teaching and learning themes and the reliance on self-reporting at a
singular time point, normally immediately post event.
‘We evaluate at the end of the event, but real impact can only be observed after years and years.’
(Pilot 4)
‘The evaluation we do is a judgement on the activity, not of its impact on people’s practices.’ (Pilot
10)
The toolkit was perceived to offer a structure for better alignment between the learning outcomes/intentions of
the CPD activity and its evaluation which could potentially influence the CPD providers understanding of value.
108/222
HEA CPD Pilot Workshop - Workbook
The data from the CPD completers prompted providers to consider the links between CPD provision and other
institutional processes, in particular annual and peer review, probation and promotion. These mechanisms were
seen as having potential to provide ‘systematic and supportive’ (Pilot 10) opportunities for evidencing teaching
quality and remaining in good standing. The success of these links was thought to dependent on the experience
of managers to discuss and promote CPD, the value of CPD and teaching quality within the institution, and the
extent to which annual and peer review are centrally coordinated. Providers suggested that data generated by
the toolkit could be used in conjunction with peer review and other quality mechanisms to inform a
conversation in annual review around remaining in good standing and future CPD needs. This is a future
development of the tool which can be realised through further refinement and piloting. There were however
perceived barriers to enhancing these links. There was recognition that at present, annual review was not in
most cases either focused on teaching development / CPD or linked to CPD provision. Although some
institutions captured individual CPD trajectories this was the minority and only one case linked this to annual
review. The location of the CPD provider within the institution varied and this influences the sphere of influence
they experienced.
‘Annual review….its variable the extent to which people take part in that, take it seriously and use
it developmentally.’ (Pilot 12)
‘We hope that appraisals can pick up CPD needs within specific departments.’ (Pilot 11)
The tool provided evidence that some CPD activities were adding value for the lecturer and wider institution in
terms of confidence changes to conceptions and changes to practice. However, there was less evidence about
value for the student learning experience, as completers were largely unable to answer these questions. CPD
providers were also unsure about how to evidence this. This gap was also evidenced through the analysis of
question choice (see Table 3.2 – frequency counts).
‘It suggests the CPD wasn't impacting on student outcomes…but we just don't know that, and we
can’t in any clear or direct correlated way’ (Pilot 10).
‘I can demonstrate that CPD around the PSF has an impact on teaching practice but to evidence the
same for students is impossible to be honest.’ (Pilot 6)
‘Getting at the important stuff is so much harder, so much more expensive and will take forever. Is
in the too difficult box, we don't go there. But if you want change, real change then that is what you
have to do.’ (Pilot 4).
More widely the challenges experienced by CPD completers in identifying impacts on students outcomes, and
the limited extent to which these questions were integrated in the pilot, indicated further work is needed in this 109/222
HEA CPD Pilot Workshop - Workbook
area. It is clear that CPD completers do not know how to evidence this, and in several cases has prompted
providers to consider why this was and how they might start to have a conversation around this issue. For
example, as a result of the data received, one institution was prompted to contact their student union to discuss
how best to raise awareness of CPD and its value for the student learning experience and all completers
discussed the need for guidance in articulating possible links between CPD, teaching and learning.
‘I haven’t taken it that far in terms of thinking about the impact that CPD has had on student
learning….I wouldn't say my questions are particularly stretching ones, whereas these are. So I think
the workbook did make me think more about that, a bit more deeply about what is it I want to know
and then how is the best way to achieve that, and which questions would help with that or what
other questions do I want to add in. So this exercise has informed me to think about what I will do
with my evaluation in the future.’ (Pilot 9)
Although the tool does draw attention to the need for firstly, consideration of potential impacts on students,
and secondly the use of institutional data sets that can provide a proxy of impacts on students, the inclusions of
questions around these theme and use of data beyond the CPD own reflections on their experiences and
practice, were both limited. In the on-going development of the tool we need to considerably strengthen the
focus placed upon these two elements we identified as integral to evaluating the impact of CPD activities on the
student experience.
There is also a wider issue that is outside the scope of this current work, but relevant to the potential uses of the
tool to determine the impact of CPD on the student experience. Much CPD takes place with limited student
input. The opportunities, or even potential for students to engage with teaching-related CPD are few and far
between, as reflected in the limited discussion of this subject across the Educational Development community.
Although projects (Campbell et al., 2007) have sought to promote student engagement with the CPD of their
lecturers, there are many barriers that need to be overcome before this can happen. Likewise student
awareness of staff CPD is implicit. Discussions that took place as a consequence of undertaking this project
indicated that students have limited awareness or knowledge of the training or development their lecturers
engage with. Therefore, in order to determine an impact on student learning, even at a basic level, we need to
raise student awareness of the CPD lecturing staff engage with and the impacts it can have on their teaching and
learning. This is beyond the scope of this project, however, needs to become part of wider discussions students
are increasingly engaging with through their Student Unions and projects such as the Student Engagement
Partnership.
110/222
HEA CPD Pilot Workshop - Workbook
3.4.5 Limitations to the current evaluation work
We aimed to ensure the evaluation was inclusive of the breadth of teaching-related CPD taking place across the
sector. As discussed, the pilot was centred on specific CPD activities (teaching preparation programmes and in
house accreditation schemes) largely as a consequence of the particular interests of the pilots involved in this
work. Although the tool was piloted on activities such as peer review, teaching development grants, mentoring
etc., the number of CPD completers piloting the tool on these activities was limited. We recognise the need for
further attention to piloting the tool on these activities beyond the life of this current project to ensure the tool
can effectively be used to evaluate the impact of these activities.
The extent to which the tool could be piloted over extended timeframes as advocated was limited by the
duration of the project. Further testing of this is needed, with CPD completers engaging with the evaluation
process from the benchmarking stage through to 12-24 months following a CPD activity. This will provide robust
insights into the effectiveness of the tool for evaluating over extended timeframes, and consider the potential of
the tool in shaping future CPD and promoting dialogue around the learning that can take place through CPD.
3.4.6 Implications for the further development of the tool based on the analysis CPD completer data
1. The toolkit should provide evaluation opportunities for all levels of evaluation expertise, from the novice
to those looking to embed more advanced processes. This will be best expressed through the guidance
for using the toolkit.
2. The role of students will be considered more explicitly by providing guidance for how to evaluate
student learning as well as satisfaction and making suggestions for how to raise awareness of and
contributions to CPD in the student body.
3. Although the CPD providers identified the potential for integration of the UKPSF in the toolkit, the
project team decided against explicitly embedding the UKPSF at this point. It was felt more useful for
the CPD provider to integrate the UKPSF as they select questions to align with the objectives /learning
outcome of the CPD activity.
4. Providing the question sets in quantitative form so there is ‘off the shelf’ provision of this. Quantitative
data perceived as useful for evaluating large numbers of participants and providing overview data for
the wider institution. This approach is also more cost effective which was a factor in most cases
5. The language in the toolkit needs to be accessible so important to mitigate the use of academic
development terminology
111/222
HEA CPD Pilot Workshop - Workbook
6. Using the data generated alongside other existing data sets to develop enhanced understanding of the
value CPD has for the lecturer, student and wider institution.
7. A section outlining guidance on how to collate, present and communicate evaluation data will be
embedded.
3.5 Recommendations
3.5.1 Higher Education Academy
An evaluation tool has been developed that builds on existing work (e.g. Chalmers et al., 2012) and has
been tested across a range of HE providers (e.g. Research / teaching focused HEIs, FE Colleges and
Private Providers). The breadth of the CPD provision across the pilot institutions determined the extent
to which the project team were able to pilot the tool across the full range of activities. As a result we
recommend further testing of the tool across teaching and learning conferences, teaching and learning
workshops, mentoring, peer review and teaching development grants.
The tool is built on the principle of promoting an approach to evaluation that takes place longitudinally.
Due to the limited timeframe of the project we recommend that further piloting takes place which
follows CPD completers through the evaluation process, from collecting pre-entry responses to the
capturing of evaluation data 12-24 months after a CPD event.
The project team have delivered a paper-based tool in line with guidance from the HEA. Future work
should focus on converting this to a fully functioning resource that is hosted online. The tool has been
tested by CPD providers. In addition, there is the potential for teaching staff to use the tool to evaluate
their on-going professional development. At the beginning of this project we developed an online
prototype which can be viewed here – http://www1.plymouth.ac.uk/research/pedrio/Pages/HEA-CPD-
Framework.aspx. (Please note that this is representative of an earlier version of the toolkit, February
2015).
Expertise surrounding evaluation is highly variable across the sector; although the tool has been
designed for use by those with different levels of evaluation expertise, there is an identified need for
further capacity building in this area. The HEA should include training and support for evaluation as part
of its enhancement offer.
There is considerable potential for students to make a contribution to teaching-related CPD, and
through this project we identified examples of best practice in this area. The HEA could usefully take
forward this agenda, perhaps in collaboration with organisations such as SEDA and the NUS, to consider
how students can be supported in making meaningful contributions to teaching-related CPD.
112/222
HEA CPD Pilot Workshop - Workbook
3.5.2 HE Providers
This work has identified connections between individuals’ CPD, annual performance development
reviews and mechanisms of reward and recognition. Commonly these are processes that operate in
isolation and could benefit from greater coherence. Institutions should consider how data regarding
teaching-related CPD is collated, disseminated and used as a forum for discussion in performance
development reviews;
Some institutions have moved toward standard mechanisms of evaluation particularly for CPD that is
institutionally accredited / delivered as part of a course. This can act as a barrier to meaningful
evaluation. Institutions might usefully consider how they can build flexibility in these systems to allow
for evaluation that links explicitly to the aims of the CPD offer.
Students regularly provide feedback on their teaching and learning experiences and have the potential
to input into teaching-related CPD. Institutions could usefully consider how students can be supported
to engage with CPD.
3.5.3 Educational Developers / CPD providers
The design of CPD and plans for evaluating the impact of provision should be considered collectively and
built into the long-term implementation and development of CPD activities;
CPD providers should build appropriate evaluation into all CPD provision. This should include the
capture of pre-entry / baseline information as well as evaluation over extended timeframes, as
advocated in the tool. However, CPD providers need to be mindful of their expectations regarding
evaluation, ensuring they are appropriate to the learning outcomes and the duration of the CPD activity.
CPD providers need to consider how they communicate the outcomes of their evaluation work and with
whom. They need to be mindful of ‘who’ is the best person to be talking with to ensure a clear message
regarding the importance of CPD is communicated across the institution.
CPD providers ought to promote a culture of reflection and evaluation into CPD provision, whereby all
participants become cognisant of the principles and practice of evaluation in order to transfer this into
their own practice.
Evaluation should become a team activity focussed not only on individual outcomes but which considers
CPD collectively, in light of institutional priorities.
CPD providers should introduce teaching staff to a wide range of evaluation tools including small scale
action research so that they are best able to evidence the impact of changes they introduce.
113/222
HEA CPD Pilot Workshop - Workbook
3.5.4 Individual
Individuals with responsibility for teaching and supporting learning should be supported to capture and
reflect on ongoing CPD involvement and its value in line with the UK Professional Standards Framework.
3.5.5 Future work
Most CPD provision is interdisciplinary and cuts across disciplinary boundaries. As a result the tool is
framed in this way. However, given the nuances of disciplinary communities there may be value in
considering how engagement with the tool may be promoted across different disciplines.
The potential for the tool to promote synergies between planning for CPD, performance development
review and mechanisms for reward and recognition was repeatedly highlighted in the data. This is an
area that warrants further investigation.
The contribution students can make to staff CPD is an area that has received limited attention and could
usefully be explored to better align CPD and the student voice.
The tool has been developed to focus on individual CPD activities however, there is considerable
potential for evaluating the impact of the full range of CPD activities individuals engage with over the
academic year, and their careers. Mechanisms need to be considered for how this could be supported
within the tool and how individuals can be supported to engage with this.
114/222
HEA CPD Pilot Workshop - Workbook
References
Amundsen, C., & Wilson, M. (2012). Are we asking the right questions?: A conceptual review of the educational development literature in higher education. Review of Educational Research, 82 (1), 90-126.
Antoniou, P. & Kyriakides, L. (2013). A Dynamic Integrated Approach to teacher professional development: Impact and sustainability of the effects on improving teacher behaviour and student outcomes. Teaching and Teacher Education, 29, 1–12.
Armour, K.M. & Makopoulou, K. (2012). Great expectations: Teacher learning in a national professional development programme. Teaching and Teacher Education, 28(3), 336–346.
Axtell, C.M., Maitlis, S., & Yearta, S.K. (1997). Predicting immediate and long-term transfer of training. Personnel Review, 26(3), 201–213.
Bamber, V. (2013). Evidencing the value of Educational Development, SEDA Special 34.
London: SEDA.
Belvis, E., Pineda, P., Armengol, C. & Moreno, V. (2013). Evaluation of reflective practice in teacher education. European Journal of Teacher Education, 36(3), 279–292.
Boman, J. S. (2013). Graduate Student Teaching Development: Evaluating the Effectiveness of Training in Relation to Graduate Student Characteristics. Canadian Journal of Higher Education, Volume 43 (1), 100-114
Boud, D., & Brew, A. (2013). Reconceptualising academic work as professional practice: implications for academic development. International Journal for Academic Development, 18(3), 208–221.
Bozalek, V., McMillan, W., Marshall, D., November, M., Daniels, A. & Sylvester, T. (2014). Analysing the professional development of teaching and learning from a political ethics of care perspective, Teaching in Higher Education, 19:5, 447-458.
Brown, A., & Inglis, S. (2013). So what happens after the event? Exploring the realisation of professional development with early childhood educators. Australasian Journal of Early Childhood, 38(1), 11-15.
Cashmore, A., Cane, C., & Cane, R. (2013). Rebalancing promotion in the HE sector: is teaching excellence being rewarded?. Genetics Education Networking for Innovation and Excellence: the UK’s Centre for Excellence in Teaching and Learning in Genetics (GENIE CETL), University of Leicester, The Higher Education Academy.
Chalmers, D. & Gardiner, D. (2015). The measurement and impact of university teacher development programs. Educar. 51 (1). 1-28.
Chalmers, D., Stoney, S., Goody, A., Goerke, V., & Gardiner, D. (2012). Identification and implementation of indicators and measures of effectiveness of teaching preparation programs for academics in higher education. Final report.
Chalmers, D. (2011). Progress and challenges to the recognition and reward of the scholarship of teaching in higher education. Higher Education Research and Development, 30, 25-38.
115/222
HEA CPD Pilot Workshop - Workbook
Chalmers, D. (2008) Indicators of university teaching and learning quality. Sydney, NSW: Australian Learning and Teaching Council. Available from: http://www.catl.uwa.edu.au/projects/tqi
Clarke, D., & H. Hollingsworth (2002). Elaborating a Model of Teacher Professional Growth. Teaching and Teacher Education 18 (8): 947–967.
Cotton, D.R.E. & Gresty, K.A. (2006) Reflecting on the think-aloud methodology for evaluating e-learning. British Journal of Educational Technology, 37, 45-54.
De Rijdt, D., Stes, A., van der Vleuten, C, & Dochy, F. (2013) Influencing variables and moderators of transfer of learning to the workplace within the area of staff development in higher education. Educational Research Review. 8. 48-74.
De Vries, S., Jansen, E.P.W.A. & van de Grift, W.J.C.M. (2013). Profiling teachers’ continuing professional development and the relation with their beliefs about learning and teaching. Teaching and Teacher Education, 33, pp.78–89.
Department for Education and Skills. (2003). The Future of Higher Education.
http://www.bis.gov.uk/assets/biscore/corporate/migratedd/publications/f/future_
of_he.pdf
Department for Business, Innovation and Skills (2011) Higher Education. Students at the Heart of the System. http://www.bis.gov.uk/assets/biscore/higher-education/docs/h/11-944-higher-educationstudents-at-heart-of-system.pdf
Di Napoli, R. (2014). Value gaming and political ontology: between resistance and compliance in academic development. International Journal for Academic Development, 19(1), pp.4–11.
Dyment, J. and O'Connell, T. (2014). When the Ink Runs Dry: Implications for Theory and Practice When Educators Stop Keeping Reflective Journal. Innovative Higher Education; 39 (5), 417-429.
Ellis, R., Weyers, M. & Hughes, J. (2013). Campus-based student experiences of learning technologies in a first-year science course. British Journal of Educational Technology, 44(5), 745–757.
Farley, H. and Murphy, A. (2013). Developing a framework for evaluating the impact and sustainability of mobile learning initiatives in higher education. Open and Distance Learning Association of Australia (ODLAA) Distance Education Summit (Sydney).
Field, A. (2009). Discovering statistics using SPSS. Sage publications.
Fink, L.D. (2013). Innovative Ways of Assessing Faculty Development. New Directions for Teaching and Learning, 2013(133), 47–59.
116/222
HEA CPD Pilot Workshop - Workbook
Gibbs, G., Habeshaw, T. and Yorke, M. (2000) Institutional Learning and Teaching Strategies in English Higher Education. Higher Education. 40 (3), 351-372.
Gosling, D. (2008). Educational development in the United Kingdom. Report for the heads of educational development group. London, Heads of Educational Development Group (HEDG) UK. http://www.hedg.ac.uk/documents/HEDG_Report_final.pdf
Green, D. (2009). New academics' perceptions of the language of teaching and learning: identifying and overcoming linguistic barriers. IJAD, 14 (1) 33-45.
Green, D., & Huston, T. (2006). Lost in translation? Developing an inclusive approach to the language of academic development. Session presented at the 6th Conference of the International Consortium for Educational Development, Sheffield, UK.
Gunn, V & Fisk, A. (2013) Considering teaching excellence in higher education: 2007 – 3013” A literature review since the CHERI report. HEA.
Guskey, T. R. (2002) Professional development and teacher change. Teachers and Teaching: Theory and Practice, 8 (3-4), 381–391.
Han J.H. & Finkelstein, A. (2013). Understanding the effects of professors’ pedagogical development with Clicker Assessment and Feedback technologies and the impact on students' engagement and learning in higher education. Computers & Education, 65, 64–76.
Han, S. (2012). Professional Development That Works: Shifting Preschool Teachers' Beliefs and Use of Instructional Strategies to Promote Children's Peer Social Competence, Journal of Early Childhood Teacher Education 33:3, 251-268
Higher Education Academy (2012). The Higher Education Academy Strategic Plan 2012-2016. York: Higher Education Academy. Retrieved 20th October 2014 from: https://www.heacademy.ac.uk/sites/default/files/downloads/strategic-plan-2012-16.pdf
HESA (2012). Changes made to the Staff record for 2012/13. Higher Education Statistics Agency. Retrieved from http://www.hesa.ac.uk/component/option,com_studrec/task,show_file/Itemid,233/mnl,12025/href,Changes.html/
Herman, J. (2013). Staffing of Teaching and Learning Centers in the United States: Indicators of Institutional Support for Faculty Development. Journal of Faculty Development 27 (2), 33-37.
Hine, G. (2013). The importance of action research in teacher education programs. Issues in Educational Research; Special Issue, 23(2), 151-163.
Hines, S. R.( 2009). Investigating Faculty Development Program Assessment Practices: What’s Being Done and How Can It Be Improved?. Journal of Faculty Development 23 (3): 5–19. Cited in Fink,
117/222
HEA CPD Pilot Workshop - Workbook
L.D. (2013). Innovative Ways of Assessing Faculty Development. New Directions for Teaching and Learning, 2013(133), 47–59.
Jordan, L. (2014). ‘Success’ and teaching development programmes: a reflexive study of personal goals and factors influencing achievement. Research proposal as part of EdD submission.
Kandlbinder, P., & Peseta, T. (2009). Key concepts in postgraduate certificates in higher education teaching and learning in Australasia and the United Kingdom. International Journal for Academic Development, 14(1), 19-31.
Kirkpatrick, D. (1998). Evaluating training programs (2nd ed.). San Francisco, CA: Berrett- Koehler.
Kreber, C. & Brook, P. (2001). Impact Evaluation of Educational Development
Programmes. International Journal for Academic Development, 6, 96-107.
Lau, W. W. F., & Yuen, A. H. K. (2013). Educational technology training workshops for mathematics teachers : An exploration of perception changes. Australasian Journal of Educational Technology, 29(4), 595-611.
Lee, B., Cawthon, S. & Dawson, K. (2013). Elementary and secondary teacher self-efficacy for teaching and pedagogical conceptual change in a drama-based professional development program. Teaching and Teacher Education, 30, 84–98.
Ling, P. (2009). Development of academics and higher education futures. Report Vol. 1.
Sydney, ALTC.
Lumpe, A., Czerniak, C., Haney, J., & Beltyukova, S. (2012). Beliefs about Teaching Science: The relationship between elementary teachers’ participation in professional development and student achievement, International Journal of Science Education, 34:2, 153-166.
McGee, J.R., Wang, C. & Polly, D., (2013). Guiding Teachers in the Use of a Standards-Based Mathematics Curriculum: Teacher Perceptions and Subsequent Instructional Practices After an Intensive Professional Development Program. School Science and Mathematics, 113(1), pp.16–28.
McKenna, C. and Hughes, J. (2015). The locations and identity construction of academic developers: Findings from the ExILED (Exploring the Identities and Locations of Educational Developers) project. SEDA Report.
Mockler, N., (2013). Teacher Professional Learning in a Neoliberal Age: Audit, Professionalism and Identity. Australian Journal of Teacher Education, 38(10).
Moolenaar, N. M., Sleegers, P. J. C., & Daly, A. J. (2012). Teaming up: linking collaboration networks, collective efficacy, and student achievement. Teaching and Teacher Education, 28(2), 251e262
Mowbray, R. & Perry, L. (2015). Improving lecture quality through training in public speaking, Innovations in Education and Teaching International, 52:2, 207-217
Parsons, D., Hill, I., Holland, J., & Willis, D. (2012). Impact of teaching development programmes in higher education. York: Higher Education Academy.
118/222
HEA CPD Pilot Workshop - Workbook
Peeraer, J., V.P. (2012). The limits of programmed professional development on integration of information and communication technology in education. Australasian Journal of Educational Technology,1039–1056.
Pehmer, A.K., Gröschner, A. & Seidel, T. (2015). How teacher professional development regarding classroom dialogue affects students’ higher-order learning. Teaching and Teacher Education, 47, pp.108–119. Available at: http://www.sciencedirect.com/science/article/pii/S0742051X14001644 [Accessed January 19, 2015].
Peters, S. , Barbier, M., Faulx, D., & Hansez, I. (2012). Learning and motivation to transfer after an e-learning programme: impact of trainees’ motivation to train, personal interaction and satisfaction. Innovations in Education and Teaching International, 49(4), pp.375–387.
Pineda, P. (2002). Gestión de la formación en las organizaciones [Training management in organizations]. Barcelona: Ariel.
Postareff, L. & Nevgi, A. (2015). Development paths of university teachers during a pedagogical development course. Educar, 51(1).
Rust, C. (1998). The impact of educational development workshops on teachers’
practice. The International Journal for Academic Development. 3, 72-80.
Rienties, B. and Kinchin, I. (2014). Understanding (in)formal learning in an academic development programme: A social network perspective. Teaching and Teacher Education. 39. 123-135.
Rienties, B. Brower, N. & Lygo-Baker, S. (2013). ‘The effects of online professional development on higher education teachers’ beliefs and intentions towards learning facilitation and technology. Teaching and Teacher Education, 29, 122-131.
Roxa, T., Martensson, K., & Alveteg, M. (2011). Understanding and influencing teaching and learning cultures at university: a network approach. Higher Education . 62, 99-111.
Rutz, C. Condon, W., Iverson, E, R. Manduca, C. & Willett, G. (2012). Faculty Professional Development and Student Learning: What is the Relationship? Change. 44 (3), 40-47.
Sachs, J. (2003). The activist teaching profession. Buckingham: Open University Press.
Saunders, R. (2014). Effectiveness of Research-Based Teacher Professional Development: Australian Journal of Teacher Education, 39(4).
Shavelson, R. (2010). Measuring college learning responsibly: Accountability in a New Era. California, Stanford University Press.
Shim, W. & Walczak, K. (2012) The Impact of Faculty Teaching Practices on the Development of Students’ Critical Thinking Skills. International Journal of Teaching and Learning in Higher Education. 24:1, 16-30.
Skelton, A. (2013). Positively transformational or poisoned chalice? The impact of a course on higher education teaching at a research-intensive institution. Teaching in Higher Education, 18(8), pp.908–919.
119/222
HEA CPD Pilot Workshop - Workbook
Smith, J. (2010). Forging identities: the experiences of probationary lecturers in the UK. Studies in Higher Education, 35, 577-591.
Spowart, L., Turner, R., Kneale, P. & Shenton, D. (in press). ‘But I have been teaching for 20 years...’: encouraging teaching accreditation for experience staff working in higher education. International Journal for Academic Development.
Stefani, L. (2013). Performance measurement for academic development: risk or opportunity? International Journal for Academic Development, 18(3), pp.294–296.
Stes, A., Coertjens, L. & Petegem, P. (2013a) Instructional development in higher education: impact on teachers' teaching behaviour as perceived by students. Instructional Science; Vol. 41 Issue 6, 1103-1126.
Stes, A., Min-Leliveld, M., Gijbels, D., & Van Petegem, P. (2010). The impact of instructional development in higher education: The state-of-the-art of the research. Educational Research Review, 5, 25-49.
Stes, A. et al. (2012). Instructional development for teachers in higher education: effects on students’ learning outcomes. Teaching in Higher Education, 17(3), 295–308
Stes, A., De Maeyer, C., Gijbels, D. & Van Petegem, P. (2013b) Effects of teachers' instructional development on students' study approaches in higher education, Studies in Higher Education, 38:1, 2-19
Stewart, M. (2014). Making sense of a teaching programme for university academics: Exploring the longer-term effects. Teaching and Teacher Education, 38, 89–98.
Stronach, I. (2010). Globalizing education, educating the local: How method made us mad. London: Routledge.
Sword, H. (2011). Archiving for the future: A longitudinal approach to evaluating a postgraduate certificate
program. In L. Stefani (ed.), Evaluating the effectiveness of academic development: principles and practice.
New York, Routledge.
Thomas, J. & Harden, A. (2008). Methods for the thematic synthesis of qualitative research in systematic reviews. BMC medical research methodology, 8, p.45.
Trigwell, K., Caballero Rodriguez, K., & Han, F. (2012). Assessing the impact of a university teaching development programme. Assessment & Evaluation in Higher Education, 37(4), 499–511.
Tronto, J. (1993). Moral Boundaries: A Political Argument for an Ethic of Care. New York: Routledge. Cited in Bozalek, V., McMillan, W., Marshall, D., November, M., Daniels, A. & Sylvester,T. (2014). Analysing the professional development of teaching and learning from a political ethics of care perspective, Teaching in Higher Education, 19:5, 447-458.
Turner, N., Oliver, M., McKenna, C., Hughes, J., Smith, H., Deepwell, F., & Shrives, L. (2013). Measuring the impact of the UK Professional Standards Framework for teaching and supporting learning. HEA report.
Turner, R. & Gosling, D. (2012). Rewarding excellent teaching: the translation of a policy initiative in the United Kingdom. Higher Education Quarterly, 66, 415-430.
Van den Bos, P. & Brouwer, J. (2014). Learning to teach in higher education: how to link theory and practice, Teaching in Higher Education, 19:7, 772-786
120/222
HEA CPD Pilot Workshop - Workbook
Watson, S. (2014). The impact of professional development on the teaching of problem solving in mathematics: A Social Learning Theory perspective. In Pope, S. (Ed.) Proceedings of the 8th British Congress of Mathematics Education 2014. 351–358.
Willett, G., Iverson, E., Rutz, C., & Manduca, C. (2014). Measures matter: Evidence of faculty development effects on faculty and student learning. Assessing Writing Vo. 20, 19-36.
121/222
HEA CPD Pilot Workshop - Workbook
Appendix 1: Audit template used in WP2
1. Continuing Professional Development in Higher Education
Thank you for taking the time to participate in this audit we are conducting as part of theHEA project “Evaluating teaching development in HE: towards an impact assessment toolkit”, undertaken by P e d R IO at Plymouth University in collaboration withH E D E R A , Durham University and University Campus Suffolk.
P u rpos e o f t h is aud i t
Your inputs are vital to helping us develop an accurate understanding of the range of teaching-related CPD offered to support the delivery of higher education teaching, and current practices in measuring their impact. Specifically, your responses will be used to inform the development of a tool to evaluate teaching-related CPD activities which in turn, will help influence teaching quality across the sector. In addition, we hope you will find completing this audit a useful tool for reflecting on the CPD provision in your institution.
W h o t h is aud i t is a i me d a t
This audit is targeted at education/academic developers, staff dedicated to CPD provision, individuals who may have responsibility for coordinating teaching-related CPD as part of their role and senior managers.
S o m e us e f u l gu i de li ne s
It should take approximately 15 minutes to complete, depending on the level of detail you are able to
122/222
HEA CPD Pilot Workshop - Workbook
provide. We have attempted to design it to be as convenient as possible, and many questions require only closed-answer responses.
Please use the 'prev' and 'next' buttons at the end of each page to navigate through the audit; using the 'back' button on your browser might interfere with your responses being saved.
The audit will be available until 17th February 2015.
C o n f i den t i a l i t y
The survey asks for only generic details about your institution and you have the option of completing it anonymously. Taking part is entirely voluntary and by completing the survey you will be considered to have given consent for its inclusion in this project. The results of this study will be used for scholarly purposes and in the development of a CPD evaluation tool which will be owned by the HEA.
This study is consistent with the British Educational Research Association guidelines on research ethics and has received ethical approval from the Education Research Ethics Committee, Plymouth University.
We do hope you are able to complete this audit and thank you in advance for your time.
If you have any questions about the audit please email evaluatingcpd@plymouth.ac.uk
123/222
HEA CPD Pilot Workshop - Workbook
2. Details about your institution
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
1. Which group does your institution belong to? Please tick all that apply.
Old university
Post-1964
Post-1992
New 2010
FE Colleges
Private provider
Russell group
Million+ group
University Alliance
Unaligned
Other (please specify)
124/222
HEA CPD Pilot Workshop - Workbook
2. Approximately what percentage of staff teaching higher education in your institution have a recognised HE-specific teaching award or qualification?
Not sure
Percentage of recognised teaching staff in HE
125/222
HEA CPD Pilot Workshop - Workbook
A c co r din g t o t h e H i g he r Educ a t i on S t a t i s t i cs A g e nc y ( H ES A ) , t h e f o ll owi ng a re cons i de r e d t o be H E - spec i f ic awa r d s or q ua l i f i c a t i on s
• Completed an institutional provided course in teaching in HE accredited against the UK Professional Standards Framework
• Recognised by the Higher Education Academy (HEA) as an Associate Fellow • Recognised by the HEA as a Fellow
• Recognised by the HEA as a Senior Fellow
• Recognised by the HEA as a Principal Fellow
• Holder of a National Teaching Fellowship Scheme Individual Award • Holder of a PGCert in higher education
• Other UK accreditation or qualification in teaching in the higher education sector • Overseas accreditation or qualification for any level of teaching
• Staff and Educational Development Association (SEDA) professional development framework awards
126/222
HEA CPD Pilot Workshop - Workbook
127/222
HEA CPD Pilot Workshop - Workbook
3. Teaching-related CPD provision at your institution
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
Teaching-related CPD is defined here as any activity ‘targeted to strengthen and extend the knowledge, skills and conceptions of teachers in a way that will lead to changes in their way of thinking and their educational behaviour’ (Fenstermacher & Berliner, 1985: 49).
This includes accredited teacher-training programmes and courses, and non-accredited activities such as workshops, mentoring and informal discussions related to teaching and learning. In this audit, teaching-related CPD includes higher education teachers but also those in support roles (e.g learning technologists, librarians, e-learning advisors etc.).
I. E x ampl es o f r ecogn i s e d/ a cc r ed i t e d t eac h in g - re l a t ed C P D • Postgraduate teaching qualification in higher education
• Masters in Education
• Recognition with Higher Education Academy • SEDA Fellowship
II. Ex a m p l e s o f non - a c cr e d i t e d t each i ng - r e l a t e d C P D • Workshops
• Short courses
• Institutional agendas/projects around teaching and learning agendas e.g. inclusivity/exam feedback/internationalisation
• Internal teaching and learning conferences • Peer-review and consultation
128/222
HEA CPD Pilot Workshop - Workbook
• Mentoring and other activities through participation in communities of practice • Discussion groups/panel sessions
• Action research
• Pedagogic research
• Scholarship of teaching and learning activities • Individual research
• Training / away days
We are interested in examples of recognised/accredited and non-accredited teaching-related CPD for HE. Please select one teaching-related CPD offer (either recognised/accredited or non-accredited) provided in your institution.
129/222
HEA CPD Pilot Workshop - Workbook
* 3. Which teaching-related CPD do you wish to tell us about?
Recognised/accredited
Non-accredited
My institution does not offer teaching-related CPD
130/222
HEA CPD Pilot Workshop - Workbook
131/222
HEA CPD Pilot Workshop - Workbook
4. Recognised/accredited CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
4. How would you describe this CPD offer? Please tick all that apply.
General Teaching Associates or equivalent
Postgraduate Certificate in Higher Education
Masters in Education
Working towards recognition with Higher Education Academy
Working towards recognition with SEDA
Other (please specify)
5. Which groups does this CPD offer target? Please tick all that apply.
Unqualified/unrecognised teaching staff
Incoming teaching staff
Teaching staff with 3 years and more experience
Staff who support teaching (learning technologists, librarians etc.)
132/222
HEA CPD Pilot Workshop - Workbook
Research/postgraduate students
Other
6. Is this offer?
Cross institutional
Tailored for a specific unit/department
Other (please specify)
7. Is participation in this CPD offer? (Please tick all that apply)
Mandatory (linked to probation)
Mandatory (not linked to probation)
Voluntary
133/222
HEA CPD Pilot Workshop - Workbook
8. Approximately how many participants are engaged with this CPD offer per annum?
9. How are the values and ethos for this CPD offer identified?
10. Do any of the following inform the content for this CPD offer? Please tick all that apply.
Institutional priorities
Discipline-specific priorities
Educational/academic development unit priorities
Evidence-based best practice
External professional frameworks (SEDA, UKPSF, Vitae RDF, discipline-specific frameworks)
Teaching and learning committees or equivalent
Course leader priorities
External feedback for example, NSS, KIS, PTES and other student surveys
Student feedback for example, module evaluations, student complaints, student representative systems, student union
External examiners
134/222
HEA CPD Pilot Workshop - Workbook
Other (please specify)
SE D A - Staff and Educational Development Association UK PS F - UK Professional Standards Framework
V i t ae RD F - Vitae Researcher Development Framework N S S – National Student Survey
K IS – Key Information Sets
P T E S – Postgraduate Taught Experience Survey
11. What are the key aims of this CPD offer? Please tick all that apply.
Teacher-focused - Develop pedagogical knowledge, teaching approaches, skills, strategies; Encourage reflective practice;
Change teaching behaviour
Student-focused - Improve students' approach to learning; Engage students' in learning; Enhance student experience
Institutional-focused – Orient participants to understand and implement culture and policies regarding teaching and learning;
Leadership development
135/222
HEA CPD Pilot Workshop - Workbook
12. Over what duration does this CPD offer usually run?
1 day or less
Less than a month
Between 1 - 6 months
Between 7 - 12 months
More than 12 months
13. How is this CPD offer primarily delivered?
Face to face
Flexible/on-line
Both (blended learning)
14. Who is involved in the delivery and support of this CPD offer? Please tick all that apply
Staff dedicated to providing teaching-related CPD
Academic staff within faculties
Professional service staff (student support, librarians, learning technologists)
Previous participants
External guest speakers
Web-based resources/online delivery
Students
Other (please specify)
136/222
HEA CPD Pilot Workshop - Workbook
* 15. Do you evaluate this CPD offer?
Yes
No
137/222
HEA CPD Pilot Workshop - Workbook
5. Non-accredited CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
16. How would you describe this CPD offer?
Workshop(s)
Short course(s)
Institutional projects around teaching and learning agendas e.g. inclusivity/exam feedback/internationalisation
Internal teaching and learning conferences
Peer-review and/or consultation
Panel discussions/group discussions
Mentoring and other activities through participation in communities of practice
Action research
Pedagogic research
Scholarship of teaching and learning activities
Training / away days
17. Which groups does this CPD offer target? Please tick all that apply.
138/222
HEA CPD Pilot Workshop - Workbook
Unqualified/unrecognised teaching staff
Incoming teaching staff
Teaching staff with 3 years+ experience
Staff who support teaching (learning technologists, librarians etc.)
Research/postgraduate students
18. Is participation in this CPD offer? (Please tick all that apply).
Mandatory (linked to probation)
Mandatory (not linked to probation)
Voluntary
139/222
HEA CPD Pilot Workshop - Workbook
19. Is this offer?
Cross institutional
Tailored for a specific unit/department
Other (please specify)
20. Approximately how many participants are engaged with this CPD offer per annum?
21. How are the values and ethos for this CPD offer identified?
22. How is content for this CPD offer identified? Please tick all that apply.
Institutional priorities
Discipline-specific priorities
Educational/academic development unit priorities
Evidence-based best practice
140/222
HEA CPD Pilot Workshop - Workbook
External professional frameworks (SEDA, UKPSF, Vitae RDF, discipline-specific frameworks)
Teaching and learning committees or equivalent
Course leader priorities
External feedback for example, NSS, KIS, PTES and other student surveys
Student feedback for example, module evaluations, student complaints, student representative systems, student union
External examiners
Other (please specify)
SE D A - Staff and Educational Development Association UK PS F - UK Professional Standards Framework
V i t ae RD F - Vitae Researcher Development Framework N S S – National Student Survey
K IS – Key Information Sets
P T E S – Postgraduate Taught Experience Survey
141/222
HEA CPD Pilot Workshop - Workbook
23. What are the key aims of this CPD offer? Please tick all that apply.
Teacher-focused - Develop pedagogical knowledge, teaching approaches, skills, strategies; Encourage reflective practice;
Change teaching behaviour
Student-focused - Improve students' approach to learning; Engage students' in learning; Enhance student experience
Institutional-focused – Orient participants to understand and implement culture and policies regarding teaching and learning;
Leadership development
24. Over what duration does this CPD offer usually run?
1 day or less
Less than a month
Between 1 - 6 months
Between 7 - 12 months
More than 12 months
25. How is this CPD offer primarily delivered?
Face to face
Flexible/on-line
Both (blended learning)
26. Who is involved in the delivery and support of this CPD offer? Please tick all that apply
Staff dedicated to providing teaching-related CPD
Academic staff within faculties
142/222
HEA CPD Pilot Workshop - Workbook
Professional service staff (student support, librarians, learning technologists)
Previous participants
External guest speakers
Web-based resources/online delivery
Students
Other (please specify)
* 27. Do you evaluate this CPD offer?
Yes
No
143/222
HEA CPD Pilot Workshop - Workbook
6. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
28. When is this CPD offer first evaluated? (You will have the chance to tell us about any subsequent evaluations later in this audit).
Before event
During event
At end of event
Within 6 months
After 6 months
144/222
HEA CPD Pilot Workshop - Workbook
145/222
HEA CPD Pilot Workshop - Workbook
7. Initial evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
29. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.
Questionnaire completed by previous participants
Interview/focus group with previous participants
Questionnaire completed by current participants
Interview/focus group with current participants
Postgraduate taught experience survey/National Student Survey
Questionnaire completed by students
Interview/focus group with students
Review of participant journals, teaching materials developed or other questions/statements,
Observations of teaching practice
Observations of changes in student behaviour/approach to learning
Line-manager/ supervisor's report
Student assessment/grades
Review of DLHE (Destination of Leavers in Higher Education)
146/222
HEA CPD Pilot Workshop - Workbook
Review of retention and progression stats
Curriculum development/enhancement
Other (please specify)
P r ev i ou s pa r t i c i pan t s - Persons who have previously undertaken the same CPD offer P a r t i c i pan t s - Persons undertaking this CPD offer
S t uden t s - Undergraduate and postgraduate students in higher education at your institution
147/222
HEA CPD Pilot Workshop - Workbook
30. What is the focus of evaluation for this CPD offer at this time? Please tick all that apply.
Participants' satisfaction with the CPD activity
Changes to participants' beliefs about teaching and learning
Changes to participants' teaching practice
Changes in students' perceptions of staff's teaching performance
Impact on student learning
Impact participation has on culture of departments and the institution
31. Any other comments about the evaluation you think would be useful to us?
32. How were the results of this evaluation used?
148/222
HEA CPD Pilot Workshop - Workbook
149/222
HEA CPD Pilot Workshop - Workbook
8. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
You have told us about evaluation of this CPD offer"[Q28]".
33. Do you evaluate this CPD offer again, at another time?
No
Yes, during event
Yes, at end of event
Yes, within 6 months
Yes, after 6 months
150/222
HEA CPD Pilot Workshop - Workbook
151/222
HEA CPD Pilot Workshop - Workbook
9. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
34. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.
Questionnaire completed by participants
Interview/focus group with participants
Postgraduate taught experience survey/National Student Survey
Questionnaire completed by students
Interview/focus group with students
Review of participant journals, teaching materials developed or other questions/statements,
Observations of teaching practice
Observations of changes in student behaviour/approach to learning
Line-manager/ supervisor's report
Student assessment/grades
Review of DLHE (Destination of Leavers in Higher Education)
Review of retention and progression stats
Other (please specify)
152/222
HEA CPD Pilot Workshop - Workbook
P a r t i c i pan t s - Persons undertaking this CPD offer
S t uden t s - Undergraduate and postgraduate students in higher education at your institution
35. What was the focus of evaluation for this CPD offer at this time? Please tick all that apply.
Participants' satisfaction with the CPD activity
Changes to participants' beliefs about teaching and learning
Changes to participants' teaching practice
Changes in students' perceptions of staff's teaching performance
Impact on student learning
Impact participation has on culture of departments and the institution
153/222
HEA CPD Pilot Workshop - Workbook
36. Any other comments about the evaluation you think would be useful to us?
37. How were the results of this evaluation used?
154/222
HEA CPD Pilot Workshop - Workbook
155/222
HEA CPD Pilot Workshop - Workbook
10. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
Previously you answered "[Q33]" to whether this offer was evaluated a second time.
38. Do you evaluate this CPD offer again, at a later time?
No
Yes, during event
Yes, at end of event
Yes, within 6 months
Yes, after 6 months
156/222
HEA CPD Pilot Workshop - Workbook
157/222
HEA CPD Pilot Workshop - Workbook
11. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
39. What methods are used for evaluation of the CPD offer at this time? Please tick all that apply.
Questionnaire completed by participants
Interview/focus group with participants
Postgraduate taught experience survey/National Student Survey
Questionnaire completed by students
Interview/focus group with students
Review of participant journals, teaching materials developed or other questions/statements,
Observations of teaching practice
Observations of changes in student behaviour/approach to learning
Line-manager/ supervisor's report
Student assessment/grades
Review of DLHE (Destination of Leavers in Higher Education)
Review of retention and progression stats
Other (please specify)
158/222
HEA CPD Pilot Workshop - Workbook
P a r t i c i pan t s - Persons undertaking this CPD offer
S t uden t s - Undergraduate and postgraduate students in higher education at your institution
40. What was the focus of evaluation for this CPD activity? Please tick all that apply.
Participants' satisfaction with the CPD activity
Changes to participants' beliefs about teaching and learning
Changes to participants' teaching practice
Changes in students' perceptions of staff's teaching performance
Impact on student learning
Impact participation has on culture of departments and the institution
159/222
HEA CPD Pilot Workshop - Workbook
41. Any other comments about the evalaution
42. How were the results of this evaluation used?
160/222
HEA CPD Pilot Workshop - Workbook
161/222
HEA CPD Pilot Workshop - Workbook
12. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
Previously, you answered "[Q38]" to whether this CPD offer was evaluated a third time.
43. Do you evaluate this CPD offer again, at a later time?
No
Yes, during event
Yes, at end of event
Yes, within 6 months
Yes, after 6 months
162/222
HEA CPD Pilot Workshop - Workbook
163/222
HEA CPD Pilot Workshop - Workbook
13. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
44. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.
Questionnaire completed by participants
Interview/focus group with participants
Postgraduate taught experience survey/National Student Survey
Questionnaire completed by students
Interview/focus group with students
Review of participant journals, teaching materials developed or other questions/statements,
Observations of teaching practice
Observations of changes in student behaviour/approach to learning
Line-manager/ supervisor's report
Student assessment/grades
Review of DLHE (Destination of Leavers in Higher Education)
Review of retention and progression stats
Other (please specify)
164/222
HEA CPD Pilot Workshop - Workbook
P a r t i c i pan t s - Persons undertaking this CPD offer
S t uden t s - Undergraduate and postgraduate students in higher education at your institution
45. What was the focus of evaluation for this CPD offer at this time? Please tick all that apply.
Participants' satisfaction with the CPD activity
Changes to participants' beliefs about teaching and learning
Changes to participants' teaching practice
Changes in students' perceptions of staff's teaching performance
Impact on student learning
Impact participation has on culture of departments and the institution
165/222
HEA CPD Pilot Workshop - Workbook
46. Any other comments about the evaluation you think would be useful to us?
47. How were the results of this evaluation used?
166/222
HEA CPD Pilot Workshop - Workbook
167/222
HEA CPD Pilot Workshop - Workbook
14. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
Previously, you answered "[Q43]" to whether this CPD offer was evaluated a fourth time.
48. Do you evaluate this CPD offer again, at a later time?
No
Yes, during event
Yes, at end of event
Yes, within 6 months
Yes, after 6 months
168/222
HEA CPD Pilot Workshop - Workbook
24
169/222
HEA CPD Pilot Workshop - Workbook
15. Evaluation of CPD offer
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
49. What methods are used for evaluation of this CPD offer at this time? Please tick all that apply.
Questionnaire completed by participants
Interview/focus group with participants
Postgraduate taught experience survey/National Student Survey
Questionnaire completed by students
Interview/focus group with students
Review of participant journals, teaching materials developed or other questions/statements,
Observations of teaching practice
Observations of changes in student behaviour/approach to learning
Line-manager/ supervisor's report
Student assessment/grades
Review of DLHE (Destination of Leavers in Higher Education)
Review of retention and progression stats
Other (please specify)
170/222
HEA CPD Pilot Workshop - Workbook
P a r t i c i pan t s - Persons undertaking this CPD offer
S t uden t s - Undergraduate and postgraduate students in higher education at your institution
50. What was the focus of evaluation for this CPD offer at this time? Please tick all that apply.
Participants' satisfaction with the CPD activity
Changes to participants' beliefs about teaching and learning
Changes to participants' teaching practice
Changes in students' perceptions of staff's teaching performance
Impact on student learning
Impact participation has on culture of departments and the institution
171/222
HEA CPD Pilot Workshop - Workbook
51. Any other comments about the evaluation you think would be useful to us?
52. How were the results of this evaluation used?
172/222
HEA CPD Pilot Workshop - Workbook
173/222
HEA CPD Pilot Workshop - Workbook
16. CPD Evaluation and Impact
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
We are also interested in your broader institutional teaching-related CPD offer and any evaluation of it.
53. Please identify which of the following your institution offers to support teaching-related CPD. Please tick all that apply.
Accredited academic programmes for teaching, e.g. PG Certificates, Masters etc.
Workshops
Institutional teaching and learning conferences
External teaching and learning conferences
Peer-review/observation
Teaching consultation
Mentoring
Identified communities of practice around teaching and learning in higher education
Action research
Funding for postgraduate study
Other (please specify)
174/222
HEA CPD Pilot Workshop - Workbook
175/222
HEA CPD Pilot Workshop - Workbook
54. How are the impacts of these CPD offers disseminated throughout your institution? Please tick all that apply.
Debrief from conference attendances
Disseminating pedagogic research
Changes in policy
Cascading from teaching and learning committees or equivalents
Mentoring
Staff appraisal
Sharing best practices through presentations, training and discussions.
There is no means of institution wide dissemination
Other (please specify)
55. Do you evaluate any of these longitudinally?
Not sure
No
Yes (please provide details below on the type of activity and the stages of evaluation)
56. Are changes in the 'scholarship of teaching' of staff who undertake CPD tracked to 176/222
HEA CPD Pilot Workshop - Workbook
assess impact?
No
Yes (please provide details below)
We define sc ho l ar s hi p o f t ea c hi ng as:
engagement with research on teaching and learning;
reflecting on personal teaching and student learning experiences within a particular discipline; and sharing knowledge and practice about teaching and learning, both within the discipline and more generally (Prosser & Trigwell, 1999)
.
177/222
HEA CPD Pilot Workshop - Workbook
57. Are students in your institution aware of staff participation in teaching-related CPD?
Yes Somewhat No Not sure
58. If you answered 'yes' or 'somewhat' to the question above, how are they made aware? Please tick all that apply.
Through the student union
Through the student representative system
Through the schools and faculties
Central communication
Other (please specify)
59. In your opinion, is teaching-related CPD an important student agenda?
Yes Somewhat No Not sure
60. Are there formal opportunities for students to contribute to the evaluation of teaching-related CPD?
Yes Somewhat No Not sure
If yes and you are happy to share more details, please tell us more about this process below.
178/222
HEA CPD Pilot Workshop - Workbook
61. How are the results of teaching-related CPD evaluation used by the institution? Please tick all that apply.
In marketing materials
As evidence for the Quality Assurance Agency and other professional audits
In annual reports to the governing body
To change policy around teaching and learning
Probation and promotion criteria
To inform future CPD provision
Not at all to my knowledge
Other (please specify)
179/222
HEA CPD Pilot Workshop - Workbook
17. Institutional policies and culture
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
62. What is your institution's minimum teaching-related CPD requirement for teaching at the following levels?
Accredited/recognised CPD
Postgraduate student
Non-accredited
CPD Case by case None Not sure
Associate Lecturer (part-time lecturer)
Incoming staff with teaching experience
Incoming staff with no teaching experience
Existing teaching staff
Staff with leadership roles in teaching
Staff in learning support roles
63. Is there an expectation/policy about peer-observation or peer-review of teaching in your institution?
180/222
HEA CPD Pilot Workshop - Workbook
Yes
No
Not sure
64. If 'yes' to the above, is peer-review of teaching activity linked to annual appraisal and/or performance development review?
Yes, in all cases
Yes, in some cases
No
Not sure
181/222
HEA CPD Pilot Workshop - Workbook
18. Reward and Recognition
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
65. How does your institution recognise and reward teaching? Please tick all that apply.
Internal teaching fellowship award schemes
External teaching fellowship award schemes
Pedagogic research grants
Institutionally bestowed teaching awards
Student bestowed teaching awards
Promotion
Financial incentives
Organisation of Teaching and Learning conferences
Community of practice around research in higher education
Time allocation for teaching-related CPD
Honorary titles
Other (please specify)
182/222
HEA CPD Pilot Workshop - Workbook
66. Are career pathways in your institution
Single-track (for both teaching and research roles, and roles which combine these activities)
Two-track (separate for teaching and research roles)
Three-track (teaching, research, teaching and research)
Not sure
Other (please specify)
A t hr ee – t ra ck s ys t em f o r pr o m o t i o n indicates one track for teaching, one track for research, and one track encompassing teaching and research. These pathways remain independent until professorial level.
183/222
HEA CPD Pilot Workshop - Workbook
67. In your promotion policy, please indicate which of the following positions include explicit criteria relating to teaching and learning?
Yes No Not sure
Professor
Reader/Principal Lecturer or equivalent
Senior Lecturer/Reader or equivalent
Lecturer
68. Did your institution make a return to the 2014 Research Excellence Framework in educational research?
Yes
No
Not sure
69. Does your institution have any Professors of Teaching?
Yes
No
Not sure
184/222
HEA CPD Pilot Workshop - Workbook
185/222
HEA CPD Pilot Workshop - Workbook
19. Next steps
As a reminder, please do not use the back button on your browser. Instead, use the 'prev' and 'next' buttons at the end of this page.
70. What is the name of your institution? (This will help inform our analysis and will remain confidential but is not essential information).
71. What is your role? (This information will help us to interpret the data we receive).
As part of this project we are seeking innovative examples of teaching-related CPD evaluation right across the HE sector. We are particularly interested in evaluation which seeks to measure:
o The impact of CPD on student learning
o The impact of institutional culture on CPD
o The impact of CPD and its evaluation on institutional culture
72. If you have examples of the above and would be willing for us to
186/222
HEA CPD Pilot Workshop - Workbook
contact you to discuss the possibility of inclusion as a case study then please provide a name and contact details and we will be in touch with you in the next few weeks.
Thank you very much for the time you have spent completing this questionnaire.
Your inputs are immensely valuable to this project and we appreciate your efforts!
If you have any comments/questions about the audit please email evaluatingcpd@plymouth.ac.uk.
Appendix 2 – Workbook on which the national audit of the tool was based
Evaluating teaching development in HE: towards impact assessment
Pilot Workshop
20th March 2015
Winterbourne House, Birmingham
WORKBOOK
187/222
HEA CPD Pilot Workshop - Workbook
Electronic versions of this information can be accessed via this link. http://www1.plymouth.ac.uk/research/pedrio/Pages/HEA-CPD-Framework.aspx
188/222
HEA CPD Pilot Workshop - Workbook
Introduction
This resource will help you to undertake an evaluation of your chosen CPD activities. You will be asked to undertake the following tasks in order to develop appropriate evaluation instruments to use with CPD participants at your institution.
1. Decide what you want to evaluate2. Decide which questions you want to ask 3. Decide what method you want to use4. Develop evaluation instruments to accommodate these requirements5. Send the project team these instruments by 27th March 2015 to
evaluatingcpd@plymouth.ac.uk 6. Return annotated workbook to Dr Jennie Winter, Plymouth University, Rm 114, 3 Endsleigh
Place, Drake Circus, Plymouth, PL4 8AA by 27th March 2015
The Plymouth Team will then visit your institution and use the instruments you have designed to conduct an evaluation with your identified CPD completers. The results of this will be shared with you and a follow-up interview between you and the Plymouth Team will establish the usefulness of the instruments for meeting your evaluation needs.
189/222
HEA CPD Pilot Workshop - Workbook
Evaluation Protocol
20th March
Introduction to the evaluation framework and toolkit, begin to select questions and determine the format for data collection
27th March
Return evaluation instrument to evaluatingcpd@plymouth.ac.uk Return annotated workbook to
Dr Jennie WinterPlymouth University Rm 114, 3 Endsleigh PlaceDrake CircusPlymouthPL48AA
From 13th April until 8th May: Data collection
Stage 1: Member of the Plymouth Team (Reema, project RA) to visit your institution to undertake the evaluation with the 3-4 CPD. Reema will also interview the Student Union representative
Stage 2: Following the visit and data collection, the data from the CPD completers will be transcribed / collated and returned for you to read through and reflect on in preparation for the telephone / skype interview with a member of the Plymouth Team
Stage 3: A member of the Plymouth Team will interview you via skype / phone to capture your experiences of evaluating CPD provision using the CPD toolkit. As noted in stage 2, you will be asked to review the evaluation data captured through the use of the tool and reflect on it with respect to the insights gained regarding current CPD provision, potential impacts on practice and future developments
190/222
HEA CPD Pilot Workshop - Workbook
Step 1: Consider these questions
1. What do you want to evaluate?
Are you interested in participant satisfaction with the course? In changes to their attitudes? In changes to concepts of teaching and learning? In changes to behaviour? Are you interested in impact on student learning? What assumptions are you making about the links between teaching and learning? Are you interested in the context of transfer?
2. Who are you doing the evaluation for?
Is it going to be used to enhance future CPD development? To provide evidence for the QAA? To feed into institutional policies? Or for another reason?
3. What will happen to this data once it is collected? How will it be used to enhance practice? Where will it be disseminated? Who will have access to it?
191/222
HEA CPD Pilot Workshop - Workbook
Step 2: Evaluation questions
Once you have made these decisions, you can use the questions set out below to help inform your evaluation. The questions are presented by activity and have been carefully constructed from considering both the extant literature and current practice in this area. They are presented to reflect the importance of evaluating at different levels and over a longitudinal time period.
For each activity…..
TimeLevels of evaluation
Pre activity
During activity
Immediately post activity
6 months post activity
12 months+ post activity
MotivationChanges in conceptions of teaching & LearningChanges in teaching and learning behaviourChanges in student perceptions of teacher behaviour
Questions
Changes in student outcomesImpacts on institutional culture
Read through the sets of questions. Decide which you activity you would like to evaluate and consider the suggested questions. Use a highlighter to identify which questions you will include in your instrument and make a note of why you want to use these questions on the workbook. You are not restricted to using only these questions, please feel free to develop your own as well. Please note these on the workbook alongside reasons for their inclusion.
192/222
Tends to move towards more qualitative evaluation methods and to include more questions around transfer, student learning and institutional context
HEA CPD Pilot Workshop - Workbook
CPD Activity 1: Evaluation questions for participants of accredited teaching courses for new lecturers
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
Motivation/Reaction
Why have you chosen to undertake the accredited teaching course?
What do you expect to gain from attending this course?
How were these expectations identified, discussed and recorded?
How have your expectations of the course changed since you started?
How have your individual aims been addressed and accommodated in the course?
What would you like to change about the course?
How did your experience of undertaking the accredited teaching course align with your expectations?
What three things from the accredited teaching course which were most useful to you?
What recommendations to you have to improve future courses?
What do you remember most about the accredited teaching course and why?
What do you remember most about the accredited teaching course and why?
Changes in conceptions of teaching & Learning
How long have you been in a teaching and / or supporting learning role within HE?
In what ways may you have you previously engaged with pedagogic literature and theory?
What have you learnt so far?
In what ways have you engaged with pedagogic literature and theory to develop your practice?
What new information or ideas were you introduced to?
What have you done as a result of completing the accredited teaching course?
Has the process of undergoing the accredited teaching course led to new areas of T&L activity / research?
How has your thinking on [XXXX] shifted following the course?
What impact did attending theaccredited teaching course have on your knowledge?
How did the accredited teaching course develop your network of teaching and learning contacts?
193/222
HEA CPD Pilot Workshop - Workbook
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
Changes in teaching and learning behaviour
What impact do you think attending this course will have on your knowledge, skills or practice?
How have you begun to apply any knowledge, skills or ideas in your practice?
In what ways have you been encouraged to reflect on your practice, and what mechanisms are you using to achieve this?
What impact has the accredited teaching course had on your knowledge, skills or practice?
How have you applied the knowledge, skills and ideas gained during the Accredited teaching course to your practice?
What, if anything, have you changed in response to the information or ideas you were introduced to during the course?
Changes in student perceptions of teacher behaviour
N/A N/A N/A N/A N/A
Changes in student outcomes
What impact do you think attending this course will have on your approach to student learning?
How have you changed or altered your approach to student learning?
What impact do you think attending this course will have on your approach to student learning?
How has your engagement with the accredited teaching course impacted on students?
How can you evidence this impact?
How has you’re your engagement in the accredited teaching course impacted on students learning?
How can you evidence this impact?
Impacts on institutional culture
How did you find out about this course?
Are you expected to attend teaching related CPD as part of your academic development?
How well do the aims of this course align with
What support is currently available to you?
What would assist you in completing the course?
How well did the topics covered in the course align with departmental/institutional priorities?
How do you anticipate sharing information within your department?
How have you been recognised and rewarded for completing the accredited teaching course?
Has the process of engaging in the Accredited teaching course led to the
How has this change been supported in your department / institution?
How have the topics covered in the course contributed to pursuing departmental priorities?
How have you
194/222
HEA CPD Pilot Workshop - Workbook
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
departmental priorities?
What support have you been given to attend this course?
What expectations, if any, are there for you to share information gained on the course with peers in your School / Department?
emergence of communities of practice around teaching and learning?
disseminated the information or ideas you were introduced to during the accredited teaching course?
What challenges have you experienced as you undertook these changes?
How did the accredited teaching course develop your network of teaching and learning contacts?
195/222
HEA CPD Pilot Workshop - Workbook
CPD Activity 2: Evaluation questions for participants of an in-house teaching accreditation framework for established academics
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
In-house teaching accreditation framework
Why have you chosen to apply for accreditation?
How long have you been in a teaching and / or supporting learning role within HE?
How did you find out about the accreditation framework?
Are you expected to obtain teaching accreditation as part of your academic development?
What do you expect to gain from this process?
How important do you think it is that staff are accredited to teach in HE?
What is your level of understanding of the UKPSF?
How have your expectations of the accreditation process changed since you started?
What have you learnt so far?
What support is currently available to you?
What would assist you in completing the accreditation process?
In what ways have you engaged with pedagogic literature and theory to develop your application?
In what ways has reflecting on your experience had an impact on your practice?
In what ways has reflecting on your
How did your experience of undergoing the accreditation process align with your expectations?
What new information or ideas were you introduced to whilst preparing your application for accreditation?
What impact do you think undergoing the accreditation process will have on your knowledge, skills or practice?
In what ways have you engaged with pedagogic literature and theory to develop your application?
What support did you access to undergo the accreditation process?
How do you anticipate sharing information gained
What have you done as a result of undergoing the accreditation process?
How have you applied the knowledge, skills and ideas reflected on through the accreditation process?
What have been the impacts of undergoing accreditation on your school / department?
How have you been recognised and rewarded for undergoing accreditation?
How has undergoing accreditation impacted on students?
Has the process of undergoing accreditation led to new areas of T&L activity / research?
What do you remember most about the accreditation process and why?
What impact did undergoing accreditation have on your knowledge, skills or practice?
How has your thinking on [XXXX] shifted following accreditation?
What, if anything, have you changed as a result of reflecting on your experiences of teaching and/or supporting learning?
How has this change been supported in your department / institution?
What challenges have you experienced as you undertook these changes?
196/222
HEA CPD Pilot Workshop - Workbook
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
What impact do you think gaining accreditation will have on your knowledge, skills or practice?
What impact do you think gaining accreditation will have on your career advancement and employability?
What role, if any, have students from your institution had in this workshop?
What support have you been given to work towards accreditation?
What expectations, if any, are there for you to share your ideas, experience and knowledge of the accreditation process?
experience had an impact on your ideas about teaching, learning and the student experience?
through the accreditation process?
How has the process helped you to support other staff?
What do you think the benefits will be for students following your engagement the accreditation process?
How has undergoing the accreditation process been recognised and rewarded within your department / institution?
What three things from undergoing the accreditation process were most useful to you?
Has the process of undergoing accreditation led to the emergence of communities of practice around teaching and learning?
How do these changes align with departmental priorities?
How has the accreditation process helped you to develop your network of teaching and learning contacts?
How has you’re your engagement in the accreditation process impacted on students learning?
How can you evidence this impact?
197/222
HEA CPD Pilot Workshop - Workbook
CPD Activity 3: Evaluation questions for participants of teaching and learning conferences
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
T&L Conference
Open questionsWhy have you chosen to attend this conference?
How did you find out about this conference?
How long have you been in a teaching and / or supporting learning role within HE?
Are you expected to attend teaching related CPD events as part of your academic development?
How well do the aims of this conference align with departmental priorities?
What do you expect to gain from attending this conference?
What impact do you think attending this conference will have on your knowledge, skills or
N/A Open questionsHow did your experience of the conference align with your expectations?
How well did the topics covered in this conference align with departmental priorities?
What new information or ideas were you introduced to at this conference?
What impact do you think attending this conference will have on your knowledge, skills or practice?
What do you think the benefit was for students involved in this conference?
How do you anticipate sharing the conference information?
Open questionsWhat do you remember as being the most useful session and why?
topics covered in this conference contributed to pursuing departmental priorities?
What, if anything, have you done in response to the information or ideas you were introduced to at this conference?
What impact did attending this conference have on your knowledge, skills or practice?
How have you disseminated the information or ideas you were introduced to at this conference?
How has attending this conference had an impact
N/A
198/222
HEA CPD Pilot Workshop - Workbook
practice?
What role, if any, have students from your institution had in this conference?
What support have you been given to attend this conference?
What expectations, if any, are there for you to disseminate conference information?
Closed questions
What three things from the conference were most useful to you?What recommendations to you have to improve future events?
How do you think attending this conference has extended your network of teaching and learning contacts by attending this conference?
on students?
How did the workshop develop your network of teaching and learning contacts?
199/222
HEA CPD Pilot Workshop - Workbook
CPD Activity 4: Evaluation questions for participants of teaching and learning workshops
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
T&L workshop
Why have you chosen to attend this workshop?
How long have you been in a teaching and / or supporting learning role within HE?
How did you find out about this workshop?
Are you expected to attend teaching related CPD events as part of your academic development?
How well do the aims of this workshop align with departmental priorities?
What do you expect to gain from attending this workshop?
What impact do you think attending this workshop will have on your knowledge, skills or practice?
N/A How did your experience of the workshop align with your expectations?
How well did the topics covered in this workshop align with departmental priorities?
What new information or ideas were you introduced to at this workshop?
What impact do you think attending this workshop will have on your knowledge, skills or practice?
What do you think the benefit was for students involved in this workshop?
How do you anticipate sharing the workshop information?
What three things from the workshop were most
What do you remember most about the workshop and why?
What impact did attending this workshop have on your knowledge, skills or practice?
How has your thinking on [XXXX] shifted following this workshop?
What, if anything, have you changed in response to the information or ideas you were introduced to at this workshop?
How has this change been supported in your department / institution?
What challenges have you experienced as you undertook these changes?
How have the topics covered in this workshop
What do you remember most about the workshop and why?
What impact did attending this workshop have on your knowledge, skills or practice?
How has your thinking on [XXXX] shifted following this workshop?
What, if anything, have you changed in response to the information or ideas you were introduced to at this workshop?
How has this change been supported in your department / institution?
What challenges have you experienced as you undertook these changes?
How have the topics covered in this workshop
200/222
HEA CPD Pilot Workshop - Workbook
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
What role, if any, have students from your institution had in this workshop?
What support have you been given to attend this workshop?
What expectations, if any, are there for you to disseminate workshop information?
useful to you? 1
What recommendations to you have to improve future workshops?
contributed to pursuing departmental priorities?
How have you disseminated the information or ideas you were introduced to at this workshop?
How did the workshop develop your network of teaching and learning contacts?
How has attending this workshop impacted on students learning?
How can you evidence this impact?
contributed to pursuing departmental priorities?
How have you disseminated the information or ideas you were introduced to at this workshop?
How did the workshop develop your network of teaching and learning contacts?
How has attending this workshop impacted on students learning?
How can you evidence this impact?
201/222
HEA CPD Pilot Workshop - Workbook
CPD Activity 5: Evaluation questions for participants of peer review activity
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
Peer Review
If a previous review has taken place, what, if any, goals were set at the end of the previous review?
To what extent have your previous goals been achieved?
What is the focus of this review?
What materials are available as a focus for reflection and discussion10?
What are your aims or intentions in this review, or (for example) using technology?
Are there any specific skills or areas of knowledge that you hope to develop over the next 12 months?
What do you hope to gain from this review process?
What, if anything, have you learned:o About your students?o About your teaching or
your role as a teacher?o About teaching and
learning more generally?
Has the review process resulted in any changes in your thinking?
What changes will you make as a result of the review process?
How confident are you that you can address the concerns that you raised at the start?
Has the review process raised any new concerns?
Have you gained what you hoped from the review?
How has your thinking changed as a result of the review process?
Have you made any changes to your teaching approach or methods?
How have students responded? How do you know?
How confident are you that you can address the concerns you raised during the review process?What did you gain from the review? Is your answer to this the same as it was immediately afterwards?
Assuming this is a cyclical process,
probably in annual cycles, return to the “Before” questions
10 For example, previous review documentation or reflective notes, ATI, course and lesson plans, student work, student feedback.
202/222
HEA CPD Pilot Workshop - Workbook
CPD Activity 6: Evaluation questions for recipients of teaching and learning development grants
CPD Activity Timeframe for evaluationPrior to the CPD
activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
T&L Development grants
Before the start of a project but after funding notification received; therefore we assume SOTL / teaching development grants are tied to funding bestowed institutionally. This will make it easier to track the aims of the funding scheme with the impact on practice etc.
We draw on the work of Admunson & Hum (2012: 4) to define the scope of work undertaken by teaching development grants / SoTL-based projects as those that result in ‘systematic inquiry’ into an aspect of teaching practice, which may results in changes in ‘conceptions of self as a teacher and researcher, changing understanding about student learning and the application of project findings to teaching practice’. This definition
How long have you been in a teaching and / or supporting learning role within HE?
Why did you submit an application to undertake the proposed project?
What was the rationale for undertaking the proposed project?
What informed the development of this proposal?
How well do the aims of this project align with departmental / institutional priorities?
What do you expect to gain from undertaking this project?
Is the project on course?
How have your expectations of the project changed?
What challenges have you encountered?
What successes have you experienced?
If relevant, how have students been involved in the project?
What professional development opportunities have you identified that would assist you in undertaking this work?
Are you more aware of scholarship and
How did your experience of the project align with your expectations?
How well did the project align with departmental / institutional priorities?
What new information or ideas did you gain as a consequence of undertaking this project?
What knowledge, experience or skills of pedagogic research and development work do you feel you gained as a consequence of this project?
What knowledge, experience or skills of teaching practice and student learning do you feel you gained as a consequence of this project?
IV question successfully used to open dialogueInitially we would like you to reflect on your experience of undertaking pedagogic research and development work through the teaching fellowship scheme.
How have you applied or used the knowledge gained through the project since its completion?
What were the impacts on your professional development of undertaking this project?
What have been the responses of colleagues to the outcomes of the
Have you applied for funding to continue with this work?
How have you built on the findings of this project?
Have the findings from this project led to new areas of T&L activity / research?
Networking
Impacts on practice
203/222
HEA CPD Pilot Workshop - Workbook
CPD Activity Timeframe for evaluationPrior to the CPD
activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
aligns with the model of SoTL advocated by Trigwell and Shale (2004) where they include student learning as an outcome that can be documented.
What impact do you think undertaking this work will have on your knowledge, skills or practice?
What do you anticipate the impact of the proposed project will be?
What experience do you have of pedagogical research and / or teaching development work?
What role, if any, will students have in the proposed work?
What support / CPD needs have you identified that would assist you in undertaking this work?
What expectations, if any, are there for you to disseminate the
pedagogic literature as a consequence of this project?
What recommendations for teaching, learning and the student experience resulted from this work and is it possible to implement them?
What do you think the benefit was for students involved in this project?
How do you plan to disseminate the outcomes of this work?
Who supported you in completing this work?
How do you think undertaking this project has extended your network of teaching and learning contacts by attending this conference?
Are you more aware of scholarship and pedagogic literature as a consequence of this project?
work?
What have been the impacts of this work on your practice, your school, and where relevant more widely?
If relevant, what have you undertaken since that has followed on from or built on the work that was undertaken through this project?
How has undertaking this project had an impact on students?
204/222
HEA CPD Pilot Workshop - Workbook
CPD Activity Timeframe for evaluationPrior to the CPD
activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
outcomes of this work?
205/222
HEA CPD Pilot Workshop - Workbook
CPD Activity 7: Evaluation questions for participants involved in teaching and learning mentoring
CPD Activity Timeframe for evaluation
Prior to the CPD activity During CPD activity Immediately after the CPD activity
6 months after the CPD activity 12-24 months
T&L MentoringQuestions here assume mentoring will focus on developing an individual's academic career as a whole, not just their teaching. In some institutions, teaching may not be considered the most important area in which to support development.
How long have you been in a teaching and / or supporting learning role within HE?
Are there any targets that you are expected to reach in the next 12-24 months?
What are your own goals in relation to:o Your teaching?o Your academic career?
Are there any milestones you hope to reach in the next 12-24 months (e.g. completing a probationary period)?
What do you hope to achieve in the next 12-24 months?
How will you know whether you have reached your goals?
How confident are you that you understand institutional
Has anything in your practice changed as a result of discussion with your mentor?
Do you feel any more able to address your main concerns?
Are any new issues or concerns being raised?
Has there been any change in what you hope to gain from working with your mentor?
What, if anything, have you learned:o About your students?o About your teaching
or your academic identity?
o About teaching and learning generally?
Has the mentoring process resulted in any changes in your thinking?
Do you think you will make any changes to the way you approach your teaching as a result of mentoring? (Yes)(No).o If Yes, what do you
intend to change?
Has the mentoring process raised any new concerns?
What progress has been made towards targets that you are required to reach?
Is the mentoring relationship continuing?
Has your thinking about your teaching changed as a result of the mentoring process?
Have you made any changes to your teaching approach or methods?
How have students responded? How do you know?
Has there been any other effect of the mentoring process?
How confident are you that you can address the concerns you discussed with your mentor?
What did you gain from the mentoring/working with your mentor? Is your answer to this the
Is the mentoring relationship continuing?
Has your thinking about your teaching changed as a result of the mentoring process?
Have you made any changes to your teaching approach or methods?
How have students responded? How do you know?
Has there been any other effect of the mentoring process?
How confident are you that you can address the concerns you discussed with your mentor?
What did you gain from the mentoring/working with your mentor? Is your answer to this the
206/222
HEA CPD Pilot Workshop - Workbook
and departmental policies and procedures related to teaching and learning?
What materials are available as a focus for reflection and discussion about your teaching11?
Have you gained what you hoped from working with your mentor?
same as it was immediately afterwards?
same as it was immediately afterwards?
11 For example, previous review documentation or reflective notes, ATI, course and lesson plans, student work, student feedback.
207/222
HEA CPD Pilot Workshop - Workbook
General questions that help to establish participant reaction to the CPD activity
Evaluation questions about participant reaction to the CPD activity
Participant reaction does not give the evaluator any information about impact on teaching practice. Instead it provides an overview of the participant’s satisfaction with teacher facilitation, content, delivery, administration, facilities and logistics of the CPD activity. Establishing reaction can be useful because it helps to inform future development of the activity. These questions are typically administered during and after the activity. You may or may not wish to include some of these questions in your evaluation.
Evaluation questions about teacher facilitation
Was your teacher sufficiently knowledgeable in this area?
Did your teacher encourage you to participate in the activity?
Did your teacher welcome your and other students’ ideas?
Did your teacher explain the content of the activity clearly?
Did your teacher use appropriate examples?
Did your teacher use a range of pedagogic activities in the session?
Did your teacher answer questions in a helpful way?
Was your teacher well prepared?
Did your teacher use technology effectively?
Did your teacher use your name?
What were the teacher’s strengths?
What were the teacher’s weaknesses?
How could the teacher improve their teaching?
Evaluation questions about content and delivery
Was the course pitched at the right level for you?
Did the course meet your expectations?
Were the materials useful?
Were the materials accessible?
Were the technologies accessible?
208/222
HEA CPD Pilot Workshop - Workbook
Was the teaching of good quality?
Was the teaching well paced?
Were the teaching methods used appropriate?
Was the content relevant?
Was the content well structured?
Was the content up-to-date?
Was the content interesting?
Were sufficient examples included?
How much of the content was useful to you?
What were the best things about the activity?
What were the worst things about the activity?
Was this activity time well spent?
Would you recommend this activity to others?
Is this activity relevant to other people in your workgroup?
How could the activity be improved?
Did you make any new contacts at this event?
Evaluation questions about the administration, facilities and logistics
Were you satisfied with the standard of the venue?
Were you satisfied with the standard of the food?
Were you satisfied with the standard of the administration?
Was the activity value for money?
Were the joining instructions easy to follow?
Was the date of the activity convenient?
Was the time of the activity convenient?
Was the length of the activity appropriate?
Was the venue location convenient?
How far did you travel to attend this activity?
209/222
HEA CPD Pilot Workshop - Workbook
Were you able to see the activity clearly?
Were you able to hear the activity communications clearly?
Was the room temperature comfortable?
Were the coffee/food breaks conveniently timed?
Were the coffee/food breaks sufficient?
210/222
HEA CPD Pilot Workshop - Workbook
Step 3: Choose the method you want to use
There are pros and cons to using different methods in evaluation. You should consider what it is you want to find out and who you are evaluating for. What sort of data helps to demonstrate this quickly and effectively? It can be a good idea to have a mix of qualitative and quantitative data to help communicate impact to different audiences. The national audit undertaken as part of this project reported questionnaires, interviews and focus groups as commonly used instruments for reporting evaluation. We provide some guidance for using these methods below, read though this information and decide which data collection type will work better for you in your evaluations. You will then need to develop these instruments and send them to us by 27th March 2015 to mailto:evaluatingcpd@plymouth.ac.uk as well as returning this workbook. These will be used as the basis of our data collection when we come to visit your institution.
Questionnaires:
Questionnaires offer an opportunity to gather basic data from groups of people. They are good for measuring responses to specific questions and they enable researchers to make comparisons across groups. They can also help you collect data over time with the same cohort and across different cohorts in order to help ascertain how attitudes and perceptions have changed.
Strengths:
Allow you to survey a large group of people relatively quickly Good at collecting factual information Can be anonymous Can be offered on paper or online Tend to be less time intensive than other methods in terms of administering, completing and
analysing A useful tool for making comparisons across cohorts and year groups Can collect quantitative or qualitative data Quantitative data can be systematically analysed Online surveys packages incorporate tools for analysis You can gather contact details for follow-up interviews or focus groups
Challenges:
Completion rates are often linked to length of time a questionnaire takes and the demands it makes
Additional factors such as tone of voice, facial expressions cannot be taken into account It is harder to gather responses to more complex questions Developing clear, unbiased questions can be challenging and may require training
211/222
HEA CPD Pilot Workshop - Workbook
Things to consider:
Be clear about what you are evaluating Determine what types of questions are most relevant to your study: Closed – T/F; multi-
choice; Likert-style or Open – with free text answers Pilot the questionnaire to test the clarity and validity of the questions
When to use:
Questionnaires are useful to gather baseline data, take a snapshot of participants’ views midway in a course, understand attitudes towards satisfaction at the end of a course or as a means of rating the relative value of components of a CPD programme in a longitudinal study. Additionally, questionnaires can be used as one element of a mixed methods study, and may offer findings which can be explored in more depth using tools such as interviews or focus groups.
Ethical issues:
Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity and withdrawing from answering a questionnaire at any point.
Questionnaire resources:
Graham Gibbs lectures
A series of three video lectures with creative commons licenses. Taken together, they offer quite a comprehensive guide, pitched at an audience that has little experience of using questionnaires:
Video 1: Question types and piloting https://www.youtube.com/watch?v=vjailyWAcJQ This also addresses the point of when a questionnaire is an appropriate method.
Video 2: Questionnaire layout and question wording https://www.youtube.com/watch?v=BZLVI5zae2E
Video 3: Ratings and scales https://www.youtube.com/watch?v=_aOhcGf8EcY
Creative Commons license: CC-BY-NC-SA
Sheffield Learning and Teaching Resource
http://www.shef.ac.uk/lets/strategy/resources/evaluate/general/methods-collection/questionnaire
This University of Sheffield online resource contains a summary of advantages and disadvantages of using questionnaires, some advice about the process and a link to some sample evaluation questions.
212/222
HEA CPD Pilot Workshop - Workbook
University of Wisconsin – Cooperative Extension 2009
http://learningstore.uwex.edu/assets/pdfs/g3658-11.pdf
This online resource provides an excellent introduction and many helpful examples of how to develop an end-of-session evaluation questionnaire for teaching–related CPD.
Interviews:
Interviews offer a way to explore questions in more depth and they tend to yield qualitative data that is potentially richer and more complex than that collected through questionnaires. They are also useful to follow up findings from questionnaires.
Things to consider:
What format of interview would best suit your evaluation? Interview formats may be:
Structured – fixed set of questions asked to all interviewees Semi-structured – set of questions and key themes for exploration, but questions vary
between interviews Unstructured – no predetermined questions; interviewee is encouraged to talk around a set
of issues
How will you keep a record of the interview? You may wish to use a voice recorder so that you can concentrate on asking questions and listening to the responses. If you are using a recording device, ensure that you have consent from your interviewee.
How do you intend to analyse the data? You might wish to transcribe in full and use a grounded approach (that involves coding of the data) as a means of interpretation. This is labour-intensive but may help you make comparisons across groups over time. Or you may prefer to draw out broad themes that you observe in the data.
Strengths:
Topics can be probed more fully Interviews are appropriate for complex, nuanced discussions Questions can be tailored to suit the context The interviewee’s views are brought to the fore and can shape the direction and flow of the
conversation Topics that are unanticipated by the interviewer can be accommodated and explored
213/222
HEA CPD Pilot Workshop - Workbook
A fine-grained perspective of the question under consideration can be obtained
Challenges:
Interviews can be time-consuming to organise, administer and analyse Anonymity is harder to guarantee with interviews Interviews are not practical for large groups where sampling would be required Analysis of interview data can be challenging and less straightforward than survey data They may require transcribing or extensive note-making Participants may be less willing to be interviewed than to complete a survey It can require some practice to guard against ‘leading’ an interviewee towards a particular
response
When to use:
Interviewing might be a suitable technique to use if you are following a set of individuals on a longitudinal study as they develop their practice after completing a CPD course. So, you could arrange to interview them at regular intervals, with a view to building up over time an understanding of their journey as teachers. The data could be reported as a case study.
Ethical issues:
Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity, responsible reporting of data and withdrawing from the interview process at any point. If the interview is being recorded, a specific consent form will need to be provided.
Interview resources:
https://itunes.apple.com/us/book/research-methods-for-business/id862468951?ls=1&mt=13
Djebarni, R., Burnett, S. ; Richards, B. (2014) Research Methods for Business students, Managers and Entrepreneurs. University of South Wales.
This multimodal e-book is an excellent resource ; it is good for interviewing techniques but also covers questionnaires and focus groups. It is freely available, CC-licensed, but has been designed to work with iPads and Macs.
Licence: Attribution-NonCommercial-ShareAlike 4.0 International
What makes a good interview?
https://www.youtube.com/watch?v=9t-_hYjAKww
214/222
HEA CPD Pilot Workshop - Workbook
In this 18-minute video, Graham Gibbs comments on examples of good and bad interview technique and runs through what he considers to be the ten characteristics of a good interviewer. Some people might feel they know most of this but the replay of an interview with commentary is quite engaging.
Focus groups:
Focus groups are effectively group interviews and they can be an efficient way of exploring a set of issues in depth with a cohort. Focus groups are also a useful way of gathering a range of views and offer an opportunity to hear points and counterpoints.
Things to consider:
Having two people to run the focus groups works well - with one person facilitating the discussion and the other taking notes. It is particularly helpful to make a recording of the session.
Ensure that members of the group feel comfortable with each other before the session officially begins. The establishment of a sense of trust, an environment conducive to discussion and a relaxed atmosphere should enable a free flowing conversation.
Strengths:
Focus groups offer a good opportunity for participants to reflect upon and share experiences Participants can stimulate observations or recollections that might not have surfaced in
either surveys or interviews The experience of thinking collaboratively about a topic can be a rewarding activity for
groups of peers. Participants may feel they have gained new perspectives or ideas from the session
The data generated from a focus group has the potential to be rich and multi-voiced
Challenges:
Arranging focus groups can be more time-intensive than surveys or interviews Good facilitation skills are needed to keep discussions focused and ensure everyone is
contributing Dominant, persuasive speakers can steer the discussion in a way that may not be
representative of the group It is difficult to take notes in a focus group, so you may wish to record the session (with
consent of participants) Participants need to feel secure in the presence of peers to express their views. A trusting,
confidential environment should be established
215/222
HEA CPD Pilot Workshop - Workbook
When to use:
Focus groups would be a good way of getting feedback about a programme midway through and at the end. It could be a good way of working with institutional stakeholders to canvas their views about the impact of CPD programme and it would be a useful way of gathering longitudinal data from participants.
Ethical issues
Finally, it is important that respondents understand their rights and the researcher’s responsibilities in relation to storage of data, anonymity, responsible reporting of data and withdrawing from the focus group at any point. If the focus group is being recorded, a specific consent form will need to be provided. The facilitator should emphasize the importance of confidentiality within the group to establish a sense of trust for all contributors
Focus groups resources:
http://infed.org/mobi/using-focus-groups-in-evaluation-and-research/ - helpful, clear and practical. Addresses evaluation directly and offers useful advice for how to handle data if using for a research project.
Smith, M. K. (2011). ‘Using focus groups in evaluation and research’, the informal education homepage. [http://infed.org/mobi/using-focus-groups-in-evaluation-and-research/. Retrieved: 9 March 2-15].
http://www.joeyanne.co.uk/2012/03/13/facilitating-focus-groups/ This is an excellent CC-licensed blogpost describing the process of setting up and running a focus group. Jo Alcock is a librarian at Birmingham City University. This work by Jo Alcock is licensed under a Creative Commons Attribution-Non-Commercial-Share Alike 2.0 UK: England & Wales License
http://localtrust.org.uk/library/how-to-guides/focus-group - general advice on how to run a focus group by the Big Local charity partnership. CC licensed: ‘All our work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 UK: England and Wales License, unless it says otherwise. - See more at: http://localtrust.org.uk/library/how-to-guides/focus-group#sthash.3tlX8C79.dpuf ‘
Karen Vinall © University of Leeds 2014. This work is made available for reuse under the terms of the Creative Commons Attribution-NonCommercial- ShareAlike 4.0 International Licence.
216/222
HEA CPD Pilot Workshop - Workbook
Other methods:
These examples are not the only ones that can be used. The audit we undertook as part of this work suggested that other methods are also used including looking at participants’ assessment, learning journals, PTES/NSS data , examples of curriculum development, peer reviews and more. We are interested in your ideas about other methods and will ask you about this during the workshop.
Step 4: Develop evaluation instruments to accommodate these requirements
Please use the guidance in this document and the online toolkit to develop evaluation instruments you would like to use with your CPD completers.
Step 5: Send the evaluation instruments
Send the project team these instruments by 27th March 2015 to evaluatingcpd@plymouth.ac.uk
Step 6: Return annotated workbook
Return annotated workbook to Dr Jennie Winter, Plymouth University, Rm 114, 3 Endsleigh Place, Drake Circus, Plymouth, PL4 8AA by 27th March 2015.
Finally many thanks for supporting this project.
We look forwards to meeting you again during our visit to your institution.
If you have any questions please get in touch with us at evaluatingcpd@plymouth.ac.uk
217/222
HEA CPD Pilot Workshop - Workbook
Appendix 3: Appendix 3: Key findings from interviews with CPD completers captured through WP3
Summary analysis – teaching courses for new lecturers
This is a summary of the interview data generated from talking to CPD completers about courses.
Motivation
Often a contractual obligation – this impacted on participants’ intrinsic and extrinsic motivation.
Part time staff were often very enthusiastic to be involved in CPD. The data suggests there is often a disconnect between CPD need and provision – this
impacts on participation and transfer.
Conceptions of teaching and learning
Participants reported shifts in conceptions around :o Acknowledging differences in individual studentso Being exposed to pedagogic literature and theory o Attitude and approaches to practice .
There was evidence that the conceptual shifts depended on the point in their career at which the staff began the course – so teaching experience plays an important role in what staff take on - and so does what their needs are at the time.
These participants were not required to evidence impact on student learning outcomes as part of the curriculum or to pass the course.
Participants noted that the experience of being a student helped shift their assumptions about their own students.
Changes to teaching behaviour
Lots of examples of shifts in practice as a result of participation including:o Giving structured feedback o Writing aims and learning outcomes
Transfer of learning to practice was influenced by the perceptions of students. Reflection dis not necessarily lead to changes in behaviour . Collaboration in departments help enhance transfer to practice.
Changes to student learning
218/222
HEA CPD Pilot Workshop - Workbook
Those new to teaching did not place much emphasis on evidencing changes to student learning, they instead focused on changes to their teaching.
Participants were aware of evaluation instruments such as module evaluations and internal student surveys but were not confident of how these could evidence changes in student learning.
Participants teaching in educational related subjects were more critical of curriculum so were often better versed in discussing (but not necessarily undertaking) student evaluation. The ways they conducted student evaluations seemed more aligned with departmental and institutional processes and values. They found their CPD experiences useful in getting their students to think about and discuss topics around teaching and learning.
This was generally a difficult question for participants to answer. Participants were aware that it is difficult to measure change and to compare cohorts of
students.
Reward and recognition
Generally staff did not feel they were being recognised for participation in teaching –related CPD, particularly where participation was mandatory.
Future CPD
Some participants reported that they were considering moving on to higher qualifications, and many others reported that being involved in a course did make them think about future CPD opportunities including publishing papers and hearing stories of good practice.
Networks and dialogue
There was evidence of a variety of informal networks in existence but there was little evidence of how participation in courses linked to informal networks – these were instead left to chance.
Suggestions for how CPD can be improved
Developing different links with students – staff-student CPD centred conferences - encouraging students to participate in CPD-like activities to develop their understanding
Having a dedicated mentor
Summary analysis - In house accreditation schemes
This is a summary of the interview data generated from talking to CPD completers about ‘in house accreditation schemes’.
219/222
HEA CPD Pilot Workshop - Workbook
Motivation
Participants tended to be better motivated to undertake accreditation schemes where they were doing so voluntarily.
There was a distinction in value between the HE and HE in FE sector: more valued in HE in FE sector.
There were mixed perceptions about whether participants perceived accreditation schemes as CPD or badging. For example, instead of using the accreditation schemes to further individual development some participants reported using it to gain recognition for work already done. Some saw it as CPD - some saw it as badging.
Reflection
Looking back at work done was considered useful however there was critique that the scheme did not foster the whole reflective cycle as there was little sense of next steps.
Conceptions of teaching and learning
Participants have to demonstrate impact so you are forced to consider and evidence changes to conceptions of teaching and learning -this is not so much the case with other activities such as the accredited/non-accredited courses.
Changes to teaching behaviour
There was little evidence that participation motivated changes in teaching and learning behaviour - this seems to be because the process largely encourages looking backwards.
Changes to student learning
This was generally a difficult question for participants to answer. They were aware that it is difficult to measure change and to compare cohorts of students. Some mentioned certain issues in getting access to information that can help to ascertain
possible impact
Reward and recognition
Participants were explicitly being recognised through gaining accredited status but this was not always celebrated within the institution and that impacted on how value of the accreditation scheme was perceived.
220/222
HEA CPD Pilot Workshop - Workbook
Scholarly activity
Participants of the accredited frameworks reported using literature to inform practice (delivery/module development/curriculum development) and using action research to embellish their applications.
Future CPD
Participants reported this process encouraged them to look back rather than look forwards and so there was little evidence of future CPD plans.
Networks and dialogue
There was evidence of a variety of informal networks in existence but there was little evidence of how participation in accredited frameworks linked to informal networks – these were instead largely left to chance.
Suggestions for how CPD can be improved
Emphasis on action research at the micro level as a way to improve the relevance and applicability of the accreditation process.
Having more recognition within the institution for achieving this accreditation route.
Summary analysis across activities
This is a summary of the interview data generated from talking to CPD completers about various activities.
Suggestions for how CPD can be improved
Use of reflective models in everyday practice – looking at CPD as a continual process and thinking “what next?” – not rolling out the red carpet for peer review
Interdisciplinary and team approaches to peer review Interdisciplinary meets and discussions Creating different links with students – staff-student CPD centred conferences - students
participating in CPD-like activities to develop their understanding Having a dedicated mentor Having a mix of practice-based and study – and alternative assessment formats - just
because you cannot write does not mean you cannot teach Considering duration of CPD activity - longer and shorter term interventions Increasing the ways in which we can access the student voice
221/222
HEA CPD Pilot Workshop - Workbook
Emphasis on action research at the micro level as a way to improve the relevance and applicability of the accreditation process
Having more recognition within the institution for CPD Having improved access to training and expertise in specific research-related skills Institutional systems to help staff capture and discuss reflections.
Linking to other institutional processes
Annual review was highlighted as the main way CPD is linked with institutional processes – but there is evidence of frustration with this. Annual review was not always seen as developmental but rather stock taking. CPD needs were not always discussed at annual reviews . Time scales were identified as another factor – an annual process was not seen as conducive to professional development.
Other issues around teaching and how teaching is managed were also highlighted – how teachers often have responsibilities but no rights - and the difficulty in managing people one has no supervisory authority over.
Staff perceived CPD as their own responsibility but looked to the institution for systems to help them capture and discuss reflection.
Better ways to evaluate
Participants made a number of suggestions for how this process could be better evaluated. These included:
Concrete advice about methods of reflection per session and generally Linking to good standing agenda - getting to reflect each year Evaluating multiple points and multiple stakeholders Having the possibility to quickly self-rate - bullet points and check boxes Building evaluation into session plans and peer review Evaluation should identify gaps and plans to fill them Evaluation should be specific and explicit Awareness that constantly asking students for feedback can turn them off – they need to
be convinced about why evaluation is a good and important activity – suggestions emerged that they could be better versed in how to engage in dialogue and feedback
Raised questions about where students have the opportunity to give better feedback on and beyond module evaluation forms
222/222
top related