final report - oxford deanery evaluation final report... · 2011-10-25 · process. contents list...

AN EVALUATION OF THE NESC DEVELOPING DENTAL EDUCATORS COURSE

FINAL REPORT

March 2011

Dr Vickie Firmstone

Centre for Research in Medical and Dental Education (CRMDE) School of Education

University of Birmingham Birmingham

B15 2TT

© 2011 University of Birmingham All rights reserved. No part of this publication may be reproduced or transmitted in any forms or by any means, without permission from the University of Birmingham (contact Vickie Firmstone). Published in the United Kingdom by The University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK. V Firmstone asserts her rights under the Copyright, Designs and Patent Act 1988 to be identified as the author of this work.

ISBN 9780704428119 Copies (£7.50, including p & p) available from: Magdalena Skrybant CRMDE administrator School of Education University of Birmingham Birmingham B15 2TT. Tel: 0121 414 4855

Acknowledgements The NHS Education South Central (NESC) Dental School is acknowledged for funding this evaluation study, and specifically Shalin Mehra and Branwen Thomas for their support in various ways towards its completion. Particular thanks are extended to the Developing Dental Educator Course participants who willingly and generously gave their time to be interviewed for the evaluation. Thanks also to the rest of the course participants for their response to the survey element of the evaluation. Acknowledgement is given to Magdalena Skrybant at the School of Education, University of Birmingham for her high quality administrative support, and to Vernise Daji for her crucial role in the processing of the interview data. Finally, special thanks to Dr Katy Newell Jones for her enthusiasm and support in the evaluation process.

CONTENTS

List of Appendices List of Tables List of Figures Executive Summary 1. INTRODUCTION ........................................................................................ 1 1.1 Background to the DDE course ................................................................................ 1 2. DESIGN AND METHODS .................................................................................. 4 2.1 Aims and objectives ............................................................................................. 4 2.2 Design .................................................................................................................... 5 2.3 Phases of the research process .............................................................................. 6 2.3.1 Phase 1: Preparation ......................................................................................... 6 2.3.2 Phase 2: Profiling participants and interviewing ............................................... 6 2.3.3 Phase 3: Questionnaire .................................................................................... 7 2.3.4 Phase 3: Analysis and report preparation ....................................................... 7 3. RESULTS AND DISCUSSION: ..................................................................... 8 3.1 Analysis of the NESC database ................................................................................. 8 3.1.1 The profile of participants: NESC data ............................................................. 8 3.2 Response to the DDE course ................................................................................. 10 3.2.1 Motivations ................................................................................. 10 3.2.2 Participation ................................................................................ 11 3.2.3 Content ................................................................................. 13 3.2.4 Delivery ................................................................................. 16 3.2.5 Learning gains ......................................................................................... 20 3.2.6 Impact on practice ......................................................................................... 22 3.2.7 Intentions for further study of dental education ............................................. 24 3.2.8 Recommendations ................................................................................. 28 4. CONCLUSIONS ........................................................................................... 29 REFERENCES ......................................................................................................... 33

APPENDICES Appendix 1: Interview Schedule with DDE Course Participants Appendix 2: Interview Schedule with Course Lead Appendix 3: Questionnaire Tool for DDE Course Participants TABLES TABLE 1: Number of course participants at each DDE course ......................................... 8 TABLE 2: Professional background and educator role ..................................................... 9 TABLE 3: Postgraduate qualifications ............................................................................. 9 TABLE 4: Ratings towards ease of attending the course ................................................. 12 TABLE 5: Comments about ease or difficulty in attending the required days ................. 12 TABLE 6: Ratings towards the content ........................................................................... 13 TABLE 7: Ratings towards the topics (four-day programme) .......................................... 14 TABLE 8: Ratings towards the topics (two-day programme) .......................................... 15 TABLE 9: Ratings towards the delivery ............................................................................ 16 TABLE 10: Ratings towards the extent of new learning (four-day programme) ............... 20 TABLE 11: Ratings towards the extent of new learning (two-day programme) ................ 21 TABLE 12: Overall views on the impact of the programme ............................................... 22 TABLE 13: Views towards CPD needs and provision .......................................................... 24 TABLE 14: Comments about further training and support they would like ....................... 25 TABLE 15: Motivation levels to study for CertMedEd ........................................................ 26 TABLE 16: Ratings towards aspects of the CertMedEd ..................................................... 26 FIGURES FIGURE 1: The objectives of the four-day DDE course ...................................................... 3 FIGURE 2: Levels of effect .................................................................................................. 4 FIGURE 3: Phases of the research process ......................................................................... 5

Executive Summary

i

EXECUTIVE SUMMARY Introduction This report details the findings of an evaluation study conducted by the Centre for Research in Medical and Dental Education (CRMDE), based at the University of Birmingham, commissioned by the NHS Education South Central (NESC) Dental School. The study, undertaken between June 2010 and March 2011, was funded to provide an educational evaluation of an innovative, new dental educator programme, the Developing Dental Educators Course. Fieldwork was undertaken by Dr Vickie Firmstone, Research Fellow, CRMDE.

Project Aims The overall aim of this study was to assess the effect of the Developing Dental Educators (DDE) Course on the participants, and specifically on their role as dental educators. The objectives were:

• to report the levels of participation in the DDE course; • to describe the educator/professional profile of participants; • to explore motivations to attend; • to investigate views of the content and pedagogy; • to identify learning gains in knowledge, skills and attitudes; • to explore the impact of the DDE course on dental educator practice; • to investigate perceptions of the course’s relevance for targeted dental educators; and • to explore intentions, uptake and views of further study in dental education undertaken

by DDE participants (e.g. CertMedEd/PGDipMedEd/Masters in Medical Education).

The data Three main methods of data collection were used:

• Analysis of the NESC database of DDE course participants; • Interviews: dental educators (n=14) and course lead; • Questionnaires: all DDE participants (n=149).

Main findings DDE participants 1. The NESC database showed that 149 new and experienced dental educators had attended

one of the 10 DDE courses delivered between August 2008 and August 2010. The majority had attended one of the four-day courses (82%), as opposed to one of the two-day DCP courses (18%). The gender breakdown of course participants overall was nearly even: 47% were female and 51% were male. Most (81%) were dentists; two-thirds (65%) were either already a DF1 trainer or planning to become one.

2. About a third (30%) of DDE course participants had a postgraduate qualification. This

proportion was split evenly between those that had a medical/dental educator qualification and those that had a non-educator related (e.g. a clinical masters). The two most popular postgraduate qualifications which had been achieved (or which were in progress) amongst DDE participants were the PGCertMedEd (9%), and the MJDF or MFGDP/MFDS (10%).

Executive Summary

ii

Motivations 3. The DDE course is compulsory for all current and new DF1 trainers and therapist VT (TVT)

trainers. The value and imperative of attending the course had been clearly communicated by NESC, and about two-thirds of interviewees commented that it had been compulsory to attend. The other third reported that it had been ‘recommended’. Attitudes towards attending the DDE course were mixed: about half of interviewees had been keen and positive about going on it; the other half had been concerned about their suitability and/or negative about how much they would learn. Without NESC’s expectation for dental educators to go on the DDE course, perhaps only half of the participants would have proactively chosen to attend. However, NESC’s clear message about attending this training could be accompanied by even clearer and detailed sources of information about the content of the course. A fuller discussion about individuals’ prior qualifications and suitability for the DDE course could also ameliorate some of the reticence to attend.

Participation 4. Overall, the course had not been difficult for most DDE participants to attend. Nearly three

quarters (72%) of respondents had found the DDE course relatively easy to attend – indeed 54% had found it very/easy. However, a significant minority (28%) reported that attendance had found difficult, and 14% had found it very inconvenient. The main complaint was about the geographical location of the course, and the distance to travel.

Content 5. Most respondents strongly/agreed that the DDE course had been relevant to all

dental educators, including more and less experienced educators, that attending had not been a “waste of time” (82%), and strongly/agreed that it had been relevant to their own specific educator role (63%). An ANOVA test showed that these views did not differ significantly according to years’ experience as educators (3 years or less/more than 3 years).

6. In terms of topics, the learning styles and effective feedback were favoured in both DDE courses (four- and two-day); and tutorials, learning theories and evaluation were positively rated too. Managing challenging or poor performing learners had also been well received and several of the interviewees drew attention to the value of these aspects.

Delivery 7. Respondents endorsed the mix of interactive and didactic styles used in the DDE course

delivery, and about half were really positive about the use of role play. The vast majority (82%) strongly/agreed that the contact with other participants was one of the “best things about the course” signalling that the way the course had been delivered and organised had worked well for participants. Most felt that the spread of days across several weeks and the residential component had given them time to reflect on the course, assimilate the information gained, and been a practical way of planning time away from clinical practice.

8. The majority of interviewees considered that the course lead, Dr Katy Newell Jones,

facilitated the course excellently. They described her style as “very relaxed, informal and approachable”, “a gifted educator” and “good at getting feedback”.

9. Most respondents considered that it was not necessary for Dr Katy Newell Jones to have

experience in dentistry; they were pleased that the focus had been on her expertise in ‘education’ and ‘interpersonal’ or ‘generic communication’ issues. On balance, the evaluation data also supports the use of mixed groups of educators on the DDE course. The

Executive Summary

iii

benefits for most, seemed to outweigh the disadvantages for some. However, the course lead could consider splitting participants into ‘experienced’ and ‘more novice/less experienced’ subgroups for some of the group discussions. And for courses with a large number of participants, particularly many very experienced educators, it would be beneficial to offer Katy the support of an additional facilitator – preferably an experienced dental educator.

Learning gains 10. Most (about two-thirds) of the evaluation respondents reported new learning across all

• Tension and conflict (strategies to work towards resolution)

key topics on the DDE course. The greatest new learning was gained from:

• Poor performance (analysing and managing issues) • Learning theories (e.g. Bloom, Kolb, Grow, adult learning principles, reflective cycle). • Tutorials (planning and approaching them) • Managing challenging learners (strategies to support them) • Effective feedback

11. For some topics on the four-day programme, the extent of new learning was rated

significantly higher by less experienced educators (those who had been an educator for 3 years or less) compared with more experienced educators (ANOVA test). These topics included: learning styles, assessment, effective feedback, and most significantly, tutorials.

Impact on practice 12. About two-thirds of respondents strongly/agreed that they had made several changes to

their educator practice as a result of the DDE course and nearly all those interviewed could readily provide examples of how they had changed what they do. These examples mainly included: new techniques or ideas for teaching; a shift towards a more adult-oriented learning style with their trainee; and greater dedicated effort to take part in reflection (both for them as an educator and their trainee). Barriers to implementing the DDE course ideas were not significant. The extra time and effort required to adopt a more adult-learning style, with a more open questioning style, rather than offering answers and solutions, was the main problem mentioned.

Intentions and views towards further study in dental education 13. Most respondents (65%) indicated they were aware of aspects of their educator practice

they needed to improve. In addition, nearly half (41%) felt that there were too few courses to support dental educators, suggesting an appetite from many for even further training and support for dental educators. Suggestions included: more on the dental foundation portfolio and its required assessments; mentoring/coaching; wanting more opportunities for contact with other educators; and any other practical courses or training that might be relevant.

14. About a quarter (26%; 19/71) of respondents were found to be highly/motivated to study

for a CertMedEd. If this proportion were to be extrapolated, up to as many as 35 of the DDE participants could currently be very interested to undertake a CertMedEd (the NESC database shows 135 individuals had attended the DDE course but had not yet started a Cert Med). This suggests a genuine interest in postgraduate educator study. Further discussion with lead educators in the deanery may help these individuals make the next step. So too could offering them discussion with individuals who have already achieved the CertMedEd.

Executive Summary

iv

Recommendations 15. Suggestions for improving the course were limited. Ideas included: joint facilitation,

preferably with an experienced senior dental educator for at least some aspects of the course; separation between experienced educators and novices in some of the small group discussions; condensing some of the group work activities; accrediting the DDE course with part of the Oxford Brookes CertMedEd; and more time for individuals to bring their own ‘educator challenges’ to the discussions. The course lead had some additional suggestions including strengthening ‘assessment’ in the course, and ways of separating out ‘clinical teaching’ from ‘teaching communication’ skills in the structuring of the sessions.

16. Ideas for future developments from the dental educators included: • To discuss specific scenarios/issues; • To provide an ‘update’ on anything new; • To share experiences, since the DDE course (e.g. new trainers together); and • To provide greater depth on specific topics; those mentioned were tutorials, specific

difficulties (like dyslexia). Providing opportunities for DDE participants to share experiences with each other at a follow-up seminar or exchange session were also endorsed by the course lead.

Conclusions The DDE educational programme has proved a good way of reaching new and current dental educators and an impressive number of nearly 150 individuals had taken part over the two year period studied. Participant feedback suggests this was a high quality dental educator training event, and the content of the programme can be readily mapped against COPDEND’s Guidelines for Dental Educators (2008). This COPDEND guidance document sets out for deaneries the breadth of topics in which different dental educators (including DF1 trainers) could be expected to be knowledgeable and skilled. The programme successfully achieved its principal purpose of supporting dental educators to be more aware about learning theory, and specifically ways to approach trainees (or other learners) from a more adult-oriented perspective. Katy Newell Jones’ teaching style was pivotal in reinforcing this message throughout the course. The focus on tutorials was well received, particularly by less experienced educators, but more could perhaps have been made of other creative approaches to teaching. The participants at the two-day course had really valued the novel teaching ideas that had been included. Finally, although assessment was a key topic on the course, the course lead had some excellent suggestions focussed on the dental foundation portfolio which could be expanded for future four-day DDE courses. This will be an excellent addition to the programme.

Finally, the vast majority (87%) of respondents agreed that the course had made them feel more invigorated as a dental educator. This is very positive – especially since a key goal of the DDE course was to motivate participants to continue with their professional development as educators. The evaluation suggests that this was important for DDE course respondents too: a quarter was very motivated to pursue a CertMedEd. Further discussion with lead educators in the deanery may help these individuals make the next step.

Introduction

1

1.0 INTRODUCTION With the launch of the Developing Dental Educators Guidelines in 2008 (COPDEND, 2008), a framework was provided for postgraduate dental deaneries to review and develop the educational provision for dental educators working at all levels. In response, NESC Dental School commissioned and funded the Developing Dental Educators course which is an extended short course (4 days) for all dental educators in the Oxford and Wessex deaneries; a shorter (3 day) course was later commissioned for DCPs in an educator role. Dr Katy Newell Jones, a consultant and trainer in education, was commissioned to develop and deliver the programme. This evaluation of the Developing Dental Educator (DDE) Course was commissioned by NESC Dental School at a time when it had been running for two years (Aug 2008 – Aug 2010) and an impressive number of about 150 dental educators had attended one of the ten courses to date (8 four-day, and 2 two-day courses). The overall purpose of the evaluation was to provide a data driven account of the content, format and impact of the four- and two-day DDE courses on participants, so that NESC can be better informed about the future development of such a programme.

1.1 Background to the DDE course In postgraduate dental deaneries across the UK, the conventional means of preparing dental foundation year 1 (DF1) trainers1

for their educational role is to offer targeted ‘training the trainer’ sessions. Expectations that appointed DF1 trainers attend such courses are evident in nearly all postgraduate deanery websites: the content varies from specifics about the trainer requirements, updates about new clinical and/or education developments, through to developing educator skills. Other education providers also provide training the trainer courses: for example, the Royal College of Surgeons has a long-established two-day ‘Training the Trainers’ course, which aims to build knowledge of teaching and learning for those in an educator role (see www.rcseng.ac.uk).

In addition, there is a growing demand for award-bearing postgraduate programmes in medical education, specifically postgraduate certificates, diplomas and masters. In response to such demand, new courses have emerged, which complement long-standing medical education provision elsewhere (e.g. University of Dundee). In the Oxford and Wessex geographical areas the provision of such courses include: the University of Bedfordshire postgraduate certificate (PGCert) in dental education (with direct progression to a postgraduate diploma (PGDip) and masters in medical education), University of Winchester studies in medical education (PGCert, PGDip, masters), and Oxford Brookes (PgCertMedEd, with pathways to progress to tailor-made PG Diploma and masters programmes). The Oxford Brookes course is delivered in collaboration with Oxford Postgraduate Medical and Dental Education (Oxford PGMDE) and for example, on a part-time basis is a minimum 9 month commitment, with further study to postgraduate diploma or masters level taking considerably longer. However, such a sustained period of continuing professional development is not appropriate to all those in a dental educator role. Dental educators are often employed part-time, and their educator role is only one aspect of a busy clinical post. Moreover, whilst some have been in post for many years and have a wealth of experience in a dental educator role, others are new to the role. With diversity in capacity and readiness to pursue formal qualifications in medical education, NESC Dental School commissioned a new dental educator course, the Developing Dental Education Course which

1 formerly known as vocational trainers.

Introduction

2

was developed by Dr Katy Newell Jones who is a consultant and trainer in education, specialising in health and social care and supporting non-formal adult learning. The DDE course built on previous provision such as the NESC vocational trainer preparation courses and was triggered by interest in the Oxford Brookes New Trainers Course. The Developing Dental Education Course was launched in August 2008. It aims to build the capacity of dental educators by providing experienced and new trainers in dental education with a foundation of knowledge and skills in some of the core areas detailed in the Guidelines for Dental Educators (COPDEND, 2008). The evidence suggests that the development of trainer’s educational capacity is particularly important given that the experience, commitment and teaching ability of a trainer is considered by many to be pivotal to the quality of the training experience for a VDP (Gibson, 2005; Cabot et al, 2007; Baldwin et al 1998; Ralph et al, 2000). To encourage participation, NESC has fully funded this programme and made attendance compulsory mandatory for current and new DF1 trainers and therapist VT trainers. It is also intended to be suitable for dental tutors, those involved in dental foundation year 2 (DF2) such as educational supervisors, dental care professional tutors, others who may be interested in training in the future, such as general dental practitioners (GDPs) or dental nurses. The course is intended to prepare participants to undertake a postgraduate certificate in medical education (CertMedEd) and then hopefully a postgraduate diploma in medical education (PGDipMedEd) and masters in medical education. Between 2008 and 2010, ten Developing Dental Educator Courses were delivered and approximately 150 participants attended. The four-day course runs over a 6 - 8 week period, rather than a one-off, or two-session course. It is organised to include a block of two days (Days 1 and 2 with a residential in between) and then Days 3 and 4, at one and two months respectively. The DCP tailored course is slightly shorter and delivered over two-days: two days with a residential between them, and a separate teaching day some weeks later. The purpose of the four-day DDE programme, as specified in the course materials, is to explore:

• the theory of adult learning and its application to dental education • creative approaches to one-to-one teaching • practical and effective tools for assessing learning • tips for successful tutorials

Specifically, by the end of the course, participants will:

• have increased their understanding of the theory of adult learning and its application to dental education

• feel more confident in selecting appropriate methods of promoting learning • be able to use a wider range of practical and effective tools for assessing learning • be able to use the process of giving and receiving feedback more effectively

For the two-day DCP course, the learning outcomes were to understand more about:

• the learning styles of themselves and others and how this can improve learning relationships

• developing, planning and delivering CPD sessions (e.g. induction, evening and weekend sessions)

• giving and receiving feedback to learners

Introduction

3

Like the four-day course, the emphasis of the two-day course was about facilitating change, evaluating, giving and receiving feedback and the importance of reflection, but there was a more group-teaching focus to the content, as opposed to one-to-one teaching. The formats of the sessions aim to be interactive: course materials specify that participants use case studies and are encouraged to use examples from their own work. The intention is that experienced trainers have the opportunity to experiment with different facilitation styles. Figure 1 provides the objectives of the four-day course and sets out the assignment required.

FIGURE 1: The objectives of the four-day DDE course

Objectives Day 1: Adult learning, roles and responsibilities

To discuss the skills, knowledge and attitudes required in dental education To identify the complexity of roles which dental educators adopt and the factors influencing these To explore the different learning styles of trainer and trainees and the impact of these on the trainer:

trainee relationship

Objectives Day 2: Effective tutorials

To explore a range of different types of tutorial To devise aims and intended outcomes of different tutorials To plan interactive and challenging tutorials Assignment: Between Days 2 and 3, participants are required to complete a piece of reflective writing (500-700words). This will analyse an incident from their practice using one of the models introduced during the course.

Objectives Day 3: Assessment and feedback

To explore the principles behind assessment and the complexity of assessment in practice To identify personal strengths and tendencies when giving and receiving feedback To provide practical opportunities to give and receive feedback in difficult situations

Objectives Day 4: Relationships, challenges and conflict

To identify good practice in chairside teaching, using a clinical case study To explore the causes of tension and conflict in dental practices and methods of managing them To discuss the practical management of poor performance To determine the future funding and development of such a programme, however it is important and timely to critically reflect on its content, format and impact: how is the programme received by its participants and what is its impact on their dental educator role? Do they feel motivated and/or prepared for further study such as a CertMedEd or PGDipMedEd? This evaluation study provides a formative evaluation of this programme, drawing together evidence of attendance rates, and feedback from participants who have taken part in the course. It is important to explore receptiveness to the programme by those in different educator roles, and those with differing levels of educator experience. A comprehensive evaluation is central to ensuring this course is ‘fit for purpose’ and that it meets the learning needs of those in a range of dental educator roles. The next section of this report sets out the Design and Methods and details the tools used for each phase of the evaluation.

Design and Methods

4

2.0 DESIGN AND METHODS

2.1 Aims and objectives The overall aim of this study is to assess the effect of the Developing Dental Educators (DDE) Course on the participants, and specifically on their role as dental educators. The objectives recognise that the full effect of the programme will only be clear once the participants have completed the full programme, and had chance to implement any skills or knowledge gained. For this reason, the study has sampled participants who attended each of the ten courses over a two-year period. The objectives are: 1. to report the levels of participation in the DDE course; 2. to describe the educator/professional profile of participants; 3. to explore motivations to attend; 4. to investigate views of the content and pedagogy; 5. to identify learning gains in knowledge, skills and attitudes; 6. to explore the impact of the DDE course on dental educator practice; 7. to investigate perceptions of the course’s relevance for targeted dental educators; and 8. to explore intentions, uptake and views of further study in dental education undertaken by DDE

participants (e.g. CertMedEd/PGDipMedEd/Masters in Medical Education). These objectives relate to levels of evaluation described by Kirkpatrick (1967) and modified by CRMDE. He suggests that evaluation to determine the effectiveness of a training programme or intervention can take place at five levels, shown in Figure 2.

At the pinnacle - Level 5 - are health care outcomes; the ultimate test of CPD – does it lead to improvements in the health of patients? It is, however very hard to attribute health gains to educational activity. And, as an outcome measure, it is rarely used in medical education studies. At the next level (4) is impact-on-practice: does CPD make a difference to the practice of professionals? Level 3 is about learning: do gains in learning result from CPD? Level 2 is reaction: for example, end of course evaluations. Although these say nothing about the consequences of the CPD they are valuable in terms of feedback to tutors and organisers. The bottom level is participation: this is useful in auditing use of funds, for example, and monitoring the types of professional groups represented on courses.

Figure 2: Levels of Effect (adapted from Kirkpatrick, 1967). Level 5 Outcomes: what were the tangible results of the programme in terms of healthcare outcomes? Level 4 Performance: what changes in practice or behaviour resulted from the programme? Level 3 Learning or knowledge: what principles, facts and techniques were learned? Level 2 Reaction: how did participants react to the programme? Level 1 Participation: numbers and types of participants. Evaluation is progressively more complex as it moves through the levels. However, evaluation at levels 4 and 5 would provide more valuable information on the effectiveness of a training intervention. Thus an evaluation of the Developing Dental Educators Course should attempt to investigate impact-on-practice i.e. impact on the participants’ educator activities. Such an investigation is not straightforward: outcome measures for change in practice are not readily available. Self-reported perceptions of change from the educators offer the best way forward. It is

Design and Methods

5

also important to recognise relationships between these levels – for example, as a result of a course, a trainer may learn a new approach (level 3) but various constraints may prevent him/her from actually changing his/her practice with their trainee (level 4). For this reason, barriers to implementing change will be explored.

2.2 Design This evaluation study has adopted a case study design. The case study design was appropriate for two main reasons. First, the nature of the specific objectives (detailed above). These were descriptive, exploratory and evaluative: to describe the profile of course participants; to explore the views towards the course; and to evaluate the impact of the course on the dental educators’ practice. Second, the expectation of case study design is that a range of methods (or techniques), different sources, and theories are used to shed light on the research question (Denzin, 1970). For this case study evaluation, a mix of interviews and a questionnaire was used, drawing mainly on the perspectives of course participants (dental educators), but also the course lead (Dr Katy Newell Jones). In summary, the data collection entailed:

• Interviews with a sample of 13 course participants, plus 1 dental educator who had been exempt from the course because of prior achievement of the CertMedEd. These interviews were conducted by telephone (approx 40 mins each)

• Interview with course lead • Postal questionnaire to all course participants 2008 – 2010 (n=149 individuals)

This is essentially a formative evaluation: the course is ongoing and this study provides an opportunity to suggest modifications to improve its delivery. It should be noted that this has not been a cost-effectiveness study; the study did not intend to include the costs involved in setting up and delivering this course, or to consider the effectiveness of the course for the investment incurred.

The evaluation was conducted over 10 months (1 June 2010 and 4 March 2011) in four overlapping phases (see Figure 3). FIGURE 3: Phases of the research process Phase 1 1 June – 12 August 2010 Preparation: securing ethics permission, designing draft tools Course observation: attended two of the four days in Course C Phase 2 16 August – 4 November 2010 Profiling participants: secondary analysis of NESC database Interviews a sample of course participants (n=14) Phase 3 28 October – 26 November 2010 Questionnaire to all course participants (n=149) Phase 4 29 November – 4 March 2011 Analysis and report preparation

The following section outlines the four research phases and the data collection tools used in each.

Design and Methods

6

2.3 PHASES OF THE RESEARCH PROCESS

2.3.1 Phase 1: Preparation In the first phase (June – August 2010), preparation for the study was undertaken. Ethics approval was secured from the University of Birmingham (ERN – 10-0521), and to fulfil the ethical requirements, data collection instruments were drafted. To gain an insight into the delivery of the programme, attendance and participant observation was undertaken at two of the days (the first and last day) in Course C (17th June, 12th August 2010). Contemporaneous notes were taken and these will be used to inform the analysis and discussion in this report.

2.3.2 Phase 2: Profiling participants and interviewing In the second phase (August – November 2010) a profile of course participants was made using the NESC database of course participants. The principal purpose of this secondary analysis was to describe the numbers, professional breakdown of course participants, demographic details (gender), and postgraduate qualifications. Permission and cooperation from the course commissioner, Shalin Mehra, was required to secure this information, and the up-to-date database was sent in July 2010. Interviews with a sample of course participants were conducted by telephone for a 30 – 45 minute period. To identify the sample of individuals to be interviewed, a purposive approach was used. The intention was to include a range of individuals who could fulfil the following criteria: all the different courses (the 8 four-day and 2 two-day, DCP courses), male/female, differing educator roles and experience levels, different professional backgrounds (nurse, dentist, hygienist) and those with experience of CertMedEd in different local Higher Education Institutions (HEIs) (Winchester, Oxford Brookes, Bedfordshire). A total of 14 were interviewed; this included 13 who had attended the DDE course plus 1 individual who had been exempted to attend because he had already started his CertMedEd. Reaching this goal of 14 interviewees in the end required inviting 19 individuals. This is because 5 of the original sample were unable to take part, and substitute individuals had to be then identified who met the designated criteria. The main topics for the interview included:

• Background and experience of being an educator • Motivations towards the course • Response to the DDE course, including views on the content, delivery and impact • Further training and development plans • Overall reflections and suggestions for improvement

With permission, audio recordings were made, and these were transcribed for thematic analysis. The interview instrument tool is enclosed as Appendix 1. The same process was used for the interview with the course lead, and the interview schedule is enclosed as Appendix 2. The purpose of the course lead interview was to explore more about the design and purpose of the course, and the DDE course lead’s (Katy Newell Jones) reflections on its impact on course participants.

Design and Methods

7

2.3.3 Phase 3: Questionnaire The questionnaire tool (see Appendix 3) was designed by CRMDE with feedback and comments from key stakeholders, including the course lead, Dr Katy Newell Jones and course commissioner, Shalin Mehra, Associate Dean. The overall purpose of distributing a questionnaire was to complement the limited sample of interviews, with a wider survey of views from all course participants. Its specific objective was to explore dentists’ and DCPs’ responses to the DDE course, including their views on the content, delivery and impact of it on their educator practice. In addition, the survey was used to explore motivations and reflections on pursuing further formal qualifications, notably, the CertMedEd. The semi-structured questionnaire tool (see Appendix 3) was distributed to all 149 participants by post on 28th October 2010 and they were given two weeks to respond (deadline 11th November). Then they were sent a reminder with a second copy of the questionnaire, and given another 10 days to respond (deadline 26th November). To protect anonymity, all questionnaire returns were non-identifiable, thus all participants were sent a reminder letter and questionnaire regardless of whether they had already returned a questionnaire. Phase 3 was completed between October and November 2010.

2.3.4 Phase 4: Analysis and report preparation The purpose of Phase 4 (Nov 2010 – March 2011) was to prepare the Final Report. In terms of analysis, in Phase 1, secondary data analysis was conducted using the NESC database of course participants. The fields on the database contained mainly quantitative data and, for the purpose of this study, simple descriptive analysis was undertaken, providing summary figures. In Phase 2, qualitative interview data were analysed thematically, integrating where possible the sub-heads and specific questions in the survey (Phase 3). The questionnaire used a combination of closed questions (such as dichotomous [e.g. yes/no], rating scales [e.g. 1 – 6]) and open (free text questions). Again, quantitative data from the completed questionnaires was mainly analysed using descriptive statistics and qualitative data, from the open comments, was analysed thematically. The feedback about the DDE course (qualitative interview data and the questionnaire data) was synthesised into key themes addressing the objectives for the evaluation. A draft report was sent for comment to the funders by 4 March 2011. The final report was completed and ready for distribution later in March 2011. The results section is in two main parts:

(1) Analysis of the NESC database – the details of participants of the DDE course; (2) Response to the DDE course – the feedback from the questionnaires and interviews.

The Conclusion then provides a synthesis of the findings from the evaluation.

Results and Discussion

8

3.0 RESULTS AND DISCUSSION

3.1 Analysis of the NESC database The NESC database provided was an MSExcel spreadsheet with 11 fields, including gender, profession, dental educator role, postgraduate qualifications, contact details, and DDE course attendance; this information has been used to map out the available details about the 149 participants who have already attended the DDE course. The analysis of this data is presented below.

3.1.1 The profile of participants: NESC data Number of participants The NESC Dental School database was interrogated to provide a profile of the course participants. Numbers at each course ranged from 9 to 23, with an average (mean) of 14.9 and median of 15.5 (see Table 1). Course 1 and 3 with 19 and 23 participants respectively were very large groups to facilitate and arguably must have made tailored discussion much more challenging. Most participants had attended one of the four-day courses (82%), as opposed to the two-day DCP course (18%). The gender breakdown of course participants overall was nearly even: 47% were female (70/1462) and 51% were male (76/146). Not surprisingly, most were dentists (81%; 120/149); much smaller proportions were practising dental nurses (15%; 22/149), practice managers3

(3%; 5/149) or hygienists (1%; 2/149). Nearly all were primary care dentists; a very small proportion was hospital specialists – mainly oral maxillo-facial surgeons, and one dentist was a senior dental officer.

TABLE 1: Number of course participants at each DDE course Course Frequency Percent

1 19 12.8

2 12 8.1

3 23 15.4

4 9 6.0

5 9 6.0

A 16 10.7

B 16 10.7

C 18 12.1

Four-day subtotal (122) (81.9%)

DCP 2009 15 10.1

DCP 2010 12 8.1

Two-day subtotal (27) (18.1%)

Total 149 100.0%

Participants’ dental educator roles In terms of their dental educator role, by far the biggest group (53%) were DF1 trainers at the time of their attendance at the DDE course and a further 4% had a combined therapist vocational trainer and a DF1 trainer. Another small group (8%) attended as ‘potential’ DF1 trainers. This means that nearly two-thirds of all DDE participants had attended because of their role (or intention) as a DF1 trainer. The rest comprised small numbers of DF2 trainers, training programme directors, therapist vocational trainers, and ‘others’ (which included an associate dean). DCPs who attended the course

2 For three individuals, their gender details were missing from the database. 3 All the practice managers were dental nurses by professional background


9

were mainly DCP facilitators or training coordinators, or in some kind of nurse training or mentor role in the practice. The many ‘others’ for DCPs included oral health promoters, those in an examiner or assessor role, and a senior dental nurse. TABLE 2: Professional background and educator role

Frequency

Percent

DF1 trainers 79 53.0% Potential DF1 trainers 12 8.1% Tutors 6 4.0% DF2 trainers 6 4.0% Therapist vocational trainers (TVT) 5 3.4% Training programme directors 4 2.7% DF1 and TVT trainers 6 4.0% Others 2 1.3% Dentists subtotal (120) (80.5%) DCP tutor/facilitator/training coordinators 11 7.4% Nurse mentor/trainer roles 6 4.0% Training programme director 1 0.7% Others 11 7.4% DCPs subtotal (29) (19.5%) Total 149 100.0%

NB: Two DCPs attended one of the four-day courses, rather than one of the DCP targeted courses.

Participants’ postgraduate qualifications On the NESC database, limited information was available about the postgraduate qualifications of the course participants. However, crude analysis suggests that about a third of the DDE course participants had a postgraduate qualification (30%; 45/149) and these were split between those that had an education-related qualification (15%; 22/149) and those that had other qualifications not-education specific (15%; 23/149%) (Table 3). Non-education related qualifications included clinical (e.g. MSc in General Practice) and non-clinical studies (e.g. PGCert Leadership and Management). The two most popular postgraduate qualifications which had been achieved (or which were in progress) were the PGCertMedEd, and the MJDF or MFGDP/MFDS (see Table 3). TABLE 3: Postgraduate qualifications

Frequency

Percent

Postgraduate Certificate in Medical Education 14 31.1% In progress (11) Completed (3) Postgraduate Diploma in Medical Education Completed

1 (1)

2.2%

Certificate in Education 4 8.9% PTTLS teaching certificate 3 6.7% Educator-related award bearing further qualifications 22 48.9% Certificate 1 2.2% Diploma level (e.g. Diploma in Dental Studies) 3 6.7% MJDS/MFDS/MFGDP 15 33.3% Postgraduate Certificate (e.g. Leadership and Management)

1

2.2%

MSc (e.g. Advanced General Practice) 1 2.2% Other 2 Not an ‘educator-related’ award bearing qualification 23 51.1% Total 45 100.0%


10

3.2 Response to the DDE course A key aspect of the evaluation was to explore participants’ responses to the course. In this section of the report, the interview data and survey responses are combined together in several key themes related to the content, delivery and impact of the course. First, it is important to outline the key features of the interview data and questionnaire responses. Interview data with DDE participants Fourteen individuals were interviewed on the telephone. The gender split of the sample was deliberately even: 7 male, 7 female. Four were DCPs (1 attended the four-day course), and 10 were dentists. In terms of role and experience, there were four DF1 trainers who had been new to the role when they started the course, two established DF1 or TVT trainers, four in an experienced trainer or educator position (e.g. enhanced trainer, training programme director – either for DF1, DF2 or TVT) and four were either tutors, DCP facilitators/trainers or oral health promoters. Five also had direct experience of the CertMedEd – either doing or completed – at each of the three main HEIs in the area: Winchester (2), Oxford Brookes (2), and Bedfordshire (1). The sample included 10 indivuduals who had attended a four-day course, 3 who had attended a two-day, plus one individual who had been exempt from the course, because of prior completion of a CertMedEd.

Questionnaire responses from DDE participants After two mail-outs, the total number of survey responses was 82 (response rate 55%; 82/149). This includes 65% males (53/82), 35% females (29/82). Most were working as dentists (85%, 70/82) in the GDS (86%, 60/70), though had a range of total years’ experience, qualifying since 2000 (25%, 17/69), in the 1990s (31%, 22/69), in the 1980s (30%, 21/69) and in the 1970s (13%, 9/69). The survey responses also indicate that this was a group of educators who were diverse in their years’ experience in a dental educator role: just over half had 3 years or less experience in a dental educator role (51%, 42/82) and the remainder had more than 3 years (49%, 40/82) - indeed, most of these (40%, 33/82) self-reported that they had 6 or more years’ experience as a dental educator. When completing the survey, most were currently employed in a dental educator role (85%, 70/82) as DF1/VT trainers (74%, 49/66). It is also positive that the 82 respondents were drawn from the cross-section of the courses: number of respondents from each course ranged from 1 to 14, mean 7.5 across the ten courses. Not surprisingly, given the greater number of four-day courses, nearly all responses were from indivuduals who had attended one of the four-day courses (89%; 74/82) as opposed to one of the two DCP two-day courses (11%; 8/82). So, it is fair to conclude that the questionnaire responses are dominated by respondents who had attended one of the four-day courses (89%), and also by DF1 trainers (74%).

3.2.1 Motivations In the first main part of the interviews about the DDE course, thirteen4

of the interviewees were asked: “Can you tell me the background of how you came to get signed up for the course? And how did you feel about going on it?”

It was clear from the interviewees’ comments that this course was recommended by the deanery, and for many, was clearly viewed as compulsory. Indeed, most of the interviewees (8 of the 13) reported that it was definitely a condition of them being accepted in their post as a dental educator

4 One of the interviewees had not attended the course.


11

(eg. as a trainer, tutor): “I applied and was accepted, but subject to doing the course”. Moreover, one added that dentists interested in becoming a trainer were also atending the course because it was a recognised expectation of becoming a VT trainer: “..when we were doing it there were actually people thinking about doing training in a couple of years time and were doing it then just so they’d tick a box and it looks better when it comes for selection to be a trainer”. However, the other five (of the 13) were softer in their understanding of how they had become signed up for the course. They felt it had been ‘recommended’ by the deanery and this had then been reinforced by colleagues who had attended in similar roles. Only one had taken his/her own initiative to attend and had personally paid for the course. This individual was not employed by the deanery but had seen it advertised and after talking to a deanery colleague had been motivated to attend the course. Thus, the deanery had been very effective in communicating the value and imperative of attending the course for those in a dental educator role, and on the whole, most interviewees felt that it had been compulory to attend. The supplementary question, which asked ‘how did you feel about going on it?’ raised more varied responses. About half (7/13) of the indviduals were either happy to attend (n=1), or very positive and keen about the course (n=6), for example, “It was a great idea, yes”;“I was very pleased to be able to do it …I was pleased to go on the course, yes”. This group included dentists and DCPs in a variety of roles, and were a mix of established and new educators – though it is noteworthy that most of the new DF1 educators (2/3) were in this group. The other half (6/13) were more negative at the recommendation or perceived compulsion to go on the course. Two dentists simply felt they would not

have chosen to attend: one responded that s/he was only doing it to “get the piece of paper” whilst the other was concerned it would be a “waste of time”. A further dentist was completely apprehensive and intimidated about the course, and this had created negative feelings about going on it. For the other three, their reticence was more about their concerns about suitability. Two were concerned that their other educator-related qualifications had not been properly taken into account, resulting in perceived overlap with the DDE course content. The other was a DF2 educator and commented that s/he thought that s/he “…realistically probably wouldn’t learn that much”.

Such findings demonstrate that without the deanery’s expectation to go on the DDE course, perhaps only half of the participants would have chosen to attend. What is clear, however, is that the NESC’s clear message about attending this training should be accompanied by even clearer and detailed information and advice about the value of the course – perhaps from some who have been on it. This might help with some of the negative expectations. Also, it is important to discuss fully with individuals their prior qualifications – although this might not affect their need to attend the course, a fuller discussion about this would have been very much welcomed.

3.2.2 Participation In the questionnaire, respondents were asked “How easy or difficult was it for your to attend all the days required for the course?” and were given a six-point rating scale, ranging from very difficult, 1 through to very easy, 6, as shown below.


12

TABLE 4: Ratings towards ease of attending the DDE course How easy or difficult was it for you to attend all the days required for the course? Please circle where 1=very difficult and 6=very easy, no difficulty at all

Frequency

Percent

Very difficult, 1 3 3.7

2 8 9.9

3 12 14.8

Subtotal (23) (28.4%)

4 14 17.3

5 30 37.0

Very easy, no difficulty at all, 6 14 17.3

Subtotal (58) (71.6%)

Total responses 81 100%

Interpretation of the data in Table 4 shows that nearly three quarters (72%, 58/81) found the DDE course relatively easy to attend and rated 4, 5 or 6 – indeed 54% found it very/easy and rated 5 or 6. However, there was a significant minority of about a quarter who had found it more difficult; just over 28% (23/81) rated 1,2 or 3. And more importantly, 14% (11/81), which is about 2 individuals in an average course of 15 participants, reported to have found it very/difficult to attend, rating 1 or 2. When asked in the questionnaire whether they had “any comments about the ease or difficulty [they] experienced in attending the required days”, only 31 respondents (38%) provided a response. This suggests that, for most respondents, the barriers experienced in participating in the course were not significant issues on which they had felt the need to to elaborate. The comments of those that did

respond have been coded into thematic categories (Table 5 below) and separated out in to those who rated they had ‘difficulty’ (rated 1, 2 or 3) and those who felt it was relatively ‘easy’ to attend (rated 4, 5 or 6).

TABLE 5: Comments about ease or difficulty in attending the required days

Total

(% of respondents)

(n=31)

(Rated 1, 2, 3)

Difficult (n=13)

(Rated 4, 5, 6)

Easy (n=18)

Location, i.e. travel time, distance 10 (32%) 4 6

Double booked 5 (16%) 4 1

No problem with attending 5 (16%) 0 5

Rescheduling patients/loss of practice time 5 (16%) 2 3

Snow 4 (13%) 2 2

Number of separate days 2 (15%) 2 0

Leaving the VT 1 (3%) 0 1

Childcare 1 (3%) 1 0

Total responses 33 15 18

The first thing to note from Table 5 is that there were not many significant differences in the types of comments made by the two groups (‘difficult’, or ‘easy’). The same kinds of comments were mentioned by both groups – suggesting that individual perceptions of specific issues (like the location) were, for some, interpreted as making their attendance ‘difficult’, whilst for others had not been a major issue.


13

Overall, by far the main complaint was about the geographical location of the course, and the distance to travel (e.g. “I live in Portsmouth; the course location, Oxford”) with about a third of the comments about this issue. This is understandable given the geographical spread of the participants. Another comment made by several was the difficulty of being ‘double-booked’ on one of the required days; reasons given included work-based and personal issues. The third main difficulty was the time away from practice and needing to reschedule patients, for example: “Four days out of practice is difficult given PCT UDA target still has to be met”. One individual who rated that it was difficult to attend, neatly summarised the main issues, but obviously had found it worth overcoming the barriers: “I felt it was worth the difficulty – loss of income and practice time, child care costs, and transport”. On a positive note, several individuals commented that they had no problem in attending, for example: Just had to book time off work. It was easy, venue was appropriate. It was easier to do full days than part days.

3.2.3 Content Most of the respondents to the questionnaire strongly/agreed that the content of the DDE course had been relevant to all

dental educators, including more and less experienced educators, that attending had definitely not been a “waste of time” (82%), and that it had been relevant to their own specific educator role (63%). A one way analysis of variance (ANOVA) was calculated for experience levels as a dental educator (3 years or less/more than 3 years) and no statistically significant results were found. In other words, views towards the content of the course did not statistically differ according to the educator experience of the respondent.

Such findings have been extracted from responses to Question 11 of the questionnaire (see Appendix 1) where ten statements were given and respondents were asked to indicate on a 6-point scale the extent to which they strongly agreed (6) through to strongly disagreed (1). Three of the ten statements were asking for their views on the content of the course, as shown in Table 6. TABLE 6: Ratings towards the content To what extent do you agree or disagree with the following statements? Please circle one number per line where 1=strongly disagree and 6=strongly agree.

Strongly disagree

Strongly agree

Mean Valid number

(1,2) (3) (4) (5,6)

The course was relevant to all dental educators, including more experienced and less experienced educators

3 (4%) 8 (10%) 16 (20%) 55 (67%) 4.8 82

Attending this course was a waste of time 67 (82%) 7 (9%) 5 (6%) 3 (4%) 1.7 82

Most of the course content was relevant for me in my current role.

2 (2%) 8 (10%) 20 (24%) 52 (63%) 4.6 82

In the questionnaire, respondents were also asked to consider each of the core topics for their four (or two-day) course and rate the overall quality of each topic. A rating scale was provided where 1 represented very poor and 6 was excellent.


14

TABLE 7: Ratings towards the topics (four-day programme) Overall quality Very poor Excellent Mean Valid number

(1,2) (3) (4) (5,6)

Introductions: understanding trainees and the trainer role

1 (1%) 4 (6%) 16 (23%) 48 (70%) 4.8 69

Learning styles and their relevance for learners and trainers

3 (4%) 17 (25%) 49 (71%) 4.9 69

Learning theories e.g. Bloom, Kolb, Grow, Adult Learning Theory

5 (7%) 5 (7%) 7 (10%) 51 (75%) 4.7 68

Tutorials: planning and approaching them 2 (3%) 12 (18%) 54 (79%) 5.0 68

Assessment in theory and in the dental practice

2 (3%) 7 (10%) 10 (15%) 49 (71%) 4.8 68

Feedback: giving and receiving feedback to maximise learning

2 (3%) 3 (5%) 9 (14%) 52 (79%) 4.9 66

Poor performance: analysing and managing issues

1 (1%) 6 (9%) 14 (20%) 48 (70%) 4.7 69

Tension and conflict – strategies to work towards resolution

1 (1%) 8 (12%) 9 (13%) 51 (74%) 4.8 69

The main message from the respondents about the topics on the four-day programme was that most rated each of the eight topics positively: more than two-thirds rated 5 or 6 for each of the topics and this is evident in the mean scores ranging from 4.7 through to 5.0. Interestingly, the interviews with the ten participants from the four-day course reinforced this message: 7 of the 10 interviewees were broadly positive about the overall content of the course. The three interviewees who were more negative provided critical comments about the topics and

the extent to which the topics had met their needs. Specifically, one of the three felt that the topics were not adequately targeted for an experienced educator (like him/herself) for example, insufficient time to discuss particular issues like managing a difficult trainee. Similarly, another experienced trainer who was critical reported that there was not enough opportunity for question and answer time on difficult issues, like conflict resolution, and that the coverage of topics was too broad. Having said that, this individual reported to have been motivated to look into the CertMedEd following the course. The third individual, again an experienced trainer, considered that experienced trainers’ expertise had not been sufficiently used in the topics covered, and was disappointed that there was not enough time on assessment, the portfolio, and the structure of vocational training. Doing more on the practical side of assessments was also a suggestion for improving the course that was made by the course lead in interview.

Moving onto the specific feedback about the topics on the DDE course, the questionnaire data in Table 6 show that tutorials were rated highest; a mean score of 5.0 and 79% (54/68) rated the tutorials session as excellent. Learning styles and effective feedback were also highly rated, with mean scores of 4.9 for both. Not far behind, 75% and 74% rated the learning theories and resolution and conflict as excellent (5 or 6). These are comprehensively high ratings; the ‘overall quality’ of these particular topics had been overwhelmingly well received and there was no statistically significant difference in terms of educator experience. Specifically, using a one-way ANOVA test, we know that those who had had more than 3 years’ experience as an educator did not rate the topics significantly differently than those with 3 years or less experience. Again, the interviews broadly supported this data and this is reassuring given the limited response rate to this particular question on the questionnaire (as noted above). From the interviews, most commonly identified ‘highlights’ in an open question about the content of the course showed that learning styles, tutorials, poor performance and the difficult trainee were most frequently mentioned


15

(4 of the 75

who identified specific highlights. Educational theory and feedback were the other two topics most frequently highlighted (3 of the 7 identified these as one of their ‘highlights’). For example, one of the new DF1 trainers expressed how the tutorial topic on the course had helped him change his questioning approach:

“It’s very easy for me to go ‘yep that’s the answer’ and then go way, rather than turn the question back onto them and get them to come up with the answer or get close to what the answer is themselves.”

Indeed, the shift towards this open style of questioning was a recurrent theme which interviewees returned to in the interviews, and is addressed further in the Impact on Practice section (3.2.6). For the two-day DCP course, respondents were (nearly) universally positive in their ratings, with all bar one individual scoring 4, 5 or 6 on the scale for each of the seven topics. Learning styles was again rated highest with a mean score of 5.3 and 83% rating excellent (5 or 6). This was followed by: evaluation (80% rated it as excellent), the introduction (67% excellent), learning theories session (67%) and effective feedback (67%). Interviews with the three DCP participants supported this positive attitude towards the content of the course. However, it is difficult to disentangle particular themes about which topics were considered to be their ‘highlight’: two of the three mentioned the managing challenging learning was a highlight; one mentioned the ‘highlight’ was understanding more about feedback and how it links with the cycles of assessment and evaluation, two mentioned that the course had helped them to reflect on their strengths and weaknesses as educators, one also emphasised their highlight had been learning about a more open questioning approach to adopt with learners. TABLE 8: Ratings towards the topics (two-day programme) Overall quality Very poor Excellent Mean Valid number

(1,2) (3) (4) (5,6)

Introductions: understanding the role of DCPs as trainers

2 (33%) 4 (67%) 5.0 6

Learning styles and their relevance for your own role in CPD

1 (17%) 5 (83%) 5.3 6

Learning and teaching theories e.g. reflective cycle, adult learning

2 (33%) 4 (67%) 4.7 6

Planning CPD/training – devising sessions for different learners

4 (67%) 2 (33%) 4.3 6

How to evaluate the effectiveness of CPD/training

1 (20%) 4 (80%) 5.0 5

Effective feedback - giving and receiving feedback using scenarios

2 (33%) 4 (67%) 4.8 6

Managing challenging learners: strategies to support them

1 (17%) 3 (50%) 2 (33%) 4.2 6

Thus, overall, the questionnaire and interview data indicates that learning styles and effective feedback were very well received in both courses, and that tutorials, theories and evaluation were positively rated for the four- and the two-day course participants respectively. Managing challenging or poor performing learners had also been welcomed and several of the interviewees drew attention to the value of this aspect.

5 The other three gave very general comments about the content – identifying no specific topics.


16

3.2.4 Delivery Format The course was delivered with an interactive emphasis in its delivery style. On the days observed, didactic teaching was minimal and was mainly used to introduce a topic, or to cover theoretical ideas or conceptual ideas. For the majority of the topics, after a short introduction, the whole group was divided into smaller groups to undertake a task or discuss a topic; the whole group later reconvened for feedback and discussion. This process was reiterated frequently throughout the days. One of the key techniques the facilitator used was role play. In their small groups, participants were encouraged to improvise various scenarios, and prepare to act out the role of the ‘foundation dentist trainee’, for example, and the ‘trainer’ in front of the whole group. The course lead strongly advocated the importance of role play as a way of simulating communication skills learning:

I wouldn’t teach communication skills without people actually needing to do it, so from that perspective there is a part of me that’s a bit hard and says ‘like it or not, you are only going to develop communications skills by doing, reflecting and getting feedback from others’

Two of the attitude statements in the questionnaire asked for respondents’ views towards the delivery and specifically the interactive aspects of the programme (Table 9). TABLE 9: Ratings towards the delivery To what extent do you agree or disagree with the following statements? Please circle one number per line where 1=strongly disagree and 6=strongly agree.

Strongly disagree

Strongly agree

Mean Valid number

(1,2) (3) (4) (5,6)

The way the course was delivered helped me learn (e.g. discussion, role play)

3 (4%) 7 (9%) 14 (17%) 58 (71%) 4.8 82

One of the best things about the course was the contact with the other participants.

1 (1%) 5 (6%) 9 (11%) 66 (82%) 5.1 81

By far the majority (71%) responded that they strongly/agreed that the way the course had been delivered had helped them to learn. For only a minority (10/82; 12%) the course delivery had not been well received (rated 1, 2 or 3). The vast majority (82%) strongly/agreed that the contact with other participants was one of the “best things about the course” signalling that the way the course was delivered and organised had worked well for participants. This is discussed further in the section below, organisational issues. Again, a one way analysis of variance (ANOVA) was calculated for experience levels as a dental educator (3 years or less/more than 3 years) and no statistically significant results were found. In other words, views towards the delivery of the course did not statistically differ according to the educator experience of the respondent. All those interviewed, without exception, were positive about the mix of interactive and didactic styles used in the delivery. A key thread in the interviews was that several (5/13) specifically identified that it was refreshing to have less didactic time than expected at CPD sessions. They commented that it had helped keep them “focussed” “awake”, that there was a “good balance” and that it suits members of the dental team, who are practical by nature, to be practically-oriented in their learning:

“…there was a lot interaction getting into groups and the fact that you had to present something really did make you focussed on the topic at hand, you knew you’d got to present it in front of everybody else.”


17

The role play element of the course was a feature of its delivery about which 11 of the 13 interviewees particularly drew attention. Just under half (5/11) were really positive about the use of role play; they had enjoyed it from the outset, thought it was “fun”, “a real strength [of the course]” “useful because it was in a safe environment”. Another third (4/11) had been initially “intimidated”, had thought “oh no, role play!”, commented that “role play strikes fear”, but had gained confidence and been converted in their views. All four of these individuals reinforced the idea that it had taken them out of their “comfort zone”, “but [they] had got into it”. In contrast, a minority of those interviewed (2/11) considered that the role play did not help their learning; one felt that it was too artificial and that participants “pretended” how they would respond in a non-real situation, the other individual categorically stated s/he was “not a role play person” and acknowledged that s/he had been destructive in the role play activities at times. Written assignment Only participants who attended the four-day course were required to complete a written assignment. Overall, the interviews suggest that the written assignment was well received; individuals could see its benefit in helping them reflect on the programme. However, the data suggests that new trainers (with limited experience to draw on) and those already studying for a CertMedEd were more likely to have viewed this written assignment more negatively. Specifically, ten6

of the 13 interviewees were asked about the value of the written assignment in the DDE four-day course. Only seven responded, and most of these (5/7) were positive about its importance in the programme. S/he reported that the assignment had encouraged their reflection on the learning gained in the course:

“Very enjoyable process and, much to my surprise, very helpful. Gave us chance to practically apply something a short time after we’d learnt it”

Another of the five who was positive commented that the assignment had overlapped with elements of the CertMedEd s/he was studying, but stated that the comments Katy had given him/her were “valuable for the certificate studies”. Finally, the most positive commendation about the written assignment was from an individual who had since developed his/her educator role. S/he felt that the written assignment had been pivotal in his/her change of career direction:

“I actually thought right this is an opportunity for me to put down on paper where I am in my career and where I want to be. I think it was very cathartic when I spoke to Katie about it…I have to say Katie gave me the confidence to think I can I can apply for this [new] job”

Only one of the seven interviewees who commented about the written assignment did not find it valuable. S/he reported that it had been “very tricky” because as a new educator who had only just started training, s/he felt that s/he did not have sufficient educator experience to draw on. Another of the seven stated that s/he had not done the written assignment because s/he had just started a CertMedEd. Organisational issues Drawing on the interview data, views towards two main organisational issues are addressed here:

• The mix of educators participating in each course;

6 Only those who had attended the four-day DDE course.


18

• Spread of the DDE days across a month or more, including a residential. A mix of educators A principal feature of this course was the inclusion of a range of different types of educators, and the breadth in years’ experience. As the course lead acknowledged: “There was huge diversity and that has been something I’ve been aware of all the time”. Overall, most interviewees’ views towards the inclusion of mix of participants with diverse educator experience were positive; only a minority felt that it had not worked out well (2 of the 117

). Most (9/13) thought it was broadly a good idea and six of the 11 (about half) were overwhelmingly positive. As one of these very positive dentists stated:

“It’s a good thing to have a spread of people with different experiences…we get stuck in our own ways and I’m sure it’ll always benefit everyone”

Three of the 11 who felt it was a good idea mentioned some minor reservations. One, a new trainer, talked about being “initially intimidated because I didn’t feel I knew anything” but that this had passed off. Another mentioned needing to adapt the content and thread of conversation to his/her particular area, but had managed this ably. And the third interviewee mentioned that it “would have been good” to have had more

experienced trainers at his/her course, but also noted that the experienced trainers at the course had shown “some resistance” to the ideas discussed.

The two individuals who were entirely negative about the breadth of experience at the DDE course both felt that the experienced educators could have been used more in discussions, and, as experienced educators themselves, reported that some discussions had been difficult. For example, one mentioned that conflict resolution had been insufficient for experienced trainers, but had “ended up worrying new trainers”. In addition, one of these considered that the individual who had attended from a secondary setting had such a different day-to-day working environment that some of his/her examples were not easy to integrate into the group discussion. On balance, the evaluation data supports the use of mixed groups of educators in the DDE course. Arguably, the benefits for most, seemed to outweigh the disadvantages for some. From an external evaluation point of view, the facilitator could consider breaking into ‘experienced’ and ‘more novice/less experienced’ subgroups for some of the group discussions, and providing even more opportunity for ‘question and answer’ time for more experienced educators to respond to queries from less experienced educators. This would serve the needs of all individuals, as a few interviewees talked about “enjoying being the expert” and sharing with others. Spread of days across a month or more All bar two of the 13 interviewees were very positive about the delivery of the DDE course across a month or more. The principle advantage was seen to be that it had given them time to “assimilate”, “to get thought into gear”, “reflect” and “to apply what had been learnt”. As one dentist commented: “It chuntered away in my subconscious…and I had certainly grown by the time I went on January’s course”. Another, more practical advantage mentioned by three of these interviewees was that four days out of practice would have been very difficult to arrange. Finally, it is noteworthy that several also highlighted the value of the residential component of the course: “good to have a social”, “residential was good”. So, overall the plan for the sessions had been well received by nearly all individuals interviewed. Only one dentist specifically noted that s/he would have preferred a block

7 2 did not talk openly about this issue.


19

of four days “to get into it more”; another DCP stated that it wouldn’t have made much difference to her if it had been arranged in a block. Facilitation From observation at two of the DDE four-day sessions, I consider Katy Newell-Jones was excellent at steering the whole group discussions and the plenary feedback time. In my view, she also made a commendable effort to be inclusive in her approach and encourage a voice from all delegates throughout; this included making time for verbal feedback, comments and criticism at the end of the course. The 13 interviews with course participants reinforced my observations. The majority of interviewees (11/13) considered that Katy had excellent facilitation skills, and described her style as “very relaxed, informal and approachable”; “handled the large group well”; “fulfilled her role very, very well”; “very positive”; “pushed everyone”; “a gifted educator” and “good at getting feedback”. Several (n=4) drew specific attention to her skill at drawing the ideas from them, rather than being prescriptive in her advice. One referred to this as “adult learning in practice” similarly another noted that she had modelled an open questioning style, encouraging the learner to take responsibility for their own learning. As s/he commented: “It was delivered the way it should be taught…as we should be teaching”. In interview with the course lead, she endorsed that this had been her intention: “I want to model support, learning and listening. I want to model that you don’t necessarily have to know it all to teach it and that you can get other people to problem solve”. When the interviewees were asked for the views towards her background as an ‘educator’, rather than a ‘dental educator’ (with a clinical background), again participants were generally positive. Most (10/13) considered that it did not matter or was not needed for her to have experience in dentistry; the consensus view was that they were pleased that the focus had been on ‘education’ and ‘interpersonal’ or ‘generic communication’ issues. Indeed, in interview with the course lead, she emphasised that this had been her intention in the course – to focus on what Katy called “the softer skills”, rather than teaching clinical skills “which seemed to be the area that was much stronger anyway”. Many stated that her education expertise had provided a different perspective: as one highlighted, dentistry “can be very narrow-minded”, and that they themselves have the specific dental context knowledge, and can give their own experiences in the course of discussion.

“Basically, though she doesn’t know much about dentistry she doesn’t need to really cause we are the ones going to learn about education, rather than the other way round”

A minority (3/13) provided some critical comment about her lack of clinical experience; two simply said that “it would have been helpful” because it was such a large group with considerable diversity of experience if she had had in a joint facilitator arrangement with an experienced dental educator. Only one interviewee (from the 13) considered it had been a “big problem” for him/her that Katy had no direct familiarity with the dental field. S/he felt that she approached issues from a theoretical perspective, rather than being experienced in educating in challenging clinical situations and patient management. For courses with a large number of participants, particularly very experienced educators, it would be beneficial to consider some additional facilitation from an individual in an established senior dental educator role, perhaps for some of the sessions. The course lead indicated that she had been able to use some of the course participants who were experienced adult learners to support her in some of the group facilitation and this had worked well; they had brought the “dental context/content element” to the group work discussion.


20

3.2.5 Learning Gains The principal purpose of the course is to enhance the knowledge and skills of dental educators. In the questionnaire, Question 11 provided an opportunity to ask respondents the extent of new learning from each of the key topic areas within the programme. As the topics were devised to suit the four-day or two-day DCP programme, Question 11 was divided into two tables so that respondents only commented about their specific course (see Appendix 3). The results to the learning gained from each course topics are set out in Tables 10 and 11: specifically, respondents were asked ‘What are your views on the content of the course? For each of the days in the programme you attended please indicate the extent of new learning on a scale of 1 to 6 where 1= no new learning/reassurance/refresher only and 6=a great deal new learning. TABLE 10: Ratings towards the extent of new learning (four-day programme) Extent of new learning No new

learning/ reassurance/

refresher only

Great deal of new learning

Mean Valid number

(1,2) (3) (4) (5,6)

Introductions: understanding trainees and the trainer role

11 (17%) 6 (9%) 23 (35%) 26 (39%) 4.6 66

Learning styles and their relevance for learners and trainers

7 (10%) 11 (16%) 14 (21%) 35 (52%) 4.8* 67

Learning theories e.g. Bloom, Kolb, Grow, Adult Learning Theory

6 (9%) 6 (9%) 17 (25%) 39 (57%) 4.6 68

Tutorials: planning and approaching them 3 (4%) 9 (13%) 20 (29%) 36 (53%) 4.6*** 68

Assessment in theory and in the dental practice

4 (6%) 13 (19%) 20 (30%) 30 (45%) 4.4* 67

Feedback: giving and receiving feedback to maximise learning

4 (6%) 8 (12%) 23 (34%) 32 (48%) 4.5** 67

Poor performance: analysing and managing issues

3 (4%) 5 (7%) 21 (31%) 39 (57%) 4.5 68

Tension and conflict – strategies to work towards resolution

2 (3%) 6 (9%) 17 (25%) 42 (63%) 4.6 67

*p< 0.05 ; ** p<0.01; *** p<0.001 The data in Table 10 shows that most (more than two-thirds) of the respondents from the four-day DDE course gained new learning (rated a 4, 5 or 6) across each of the topics listed. Most highly rated new learning was in the topics of: tension and conflict (63% rated 5 or 6 ‘excellent’), poor performance (57%), learning theories (57%) tutorials (53%) and learning styles (52%). Here, however, there were

statistically significant differences amongst those who were less experienced defined as (3 years of less dental educator experience) and those with more experience as educators. A one way ANOVA showed that less experienced educators indicated that they had gained significantly more new learning from: learning styles and assessment (both with p values p<0.05), feedback (p<0.01), but particularly more from tutorials (p<0.001). The tutorial session had been particularly useful and relevant to those who were less experienced as educators.

In Table 11 overleaf, it is evident that respondents from the two-day course gained most from the effective feedback (mean score of 4.2), evaluation (mean score of 4.0) and managing challenging learners (mean score of 4.0). Due to relatively low numbers, it has not been appropriate to perform a one-way ANOVA test.


21

TABLE 11: Ratings towards the extent of new learning (two-day programme) Extent of new learning No new

learning/ reassurance/

refresher only

Great deal of new learning

Mean Valid number

(1,2) (3) (4) (5,6)

Introductions: understanding the role of DCPs as trainers

2 (40%) 1 (20%) 2 (40%) 3.4 5

Learning styles and their relevance for your own role in CPD

2 (40%) 1 (20%) 2 (40%) 3.4 5

Learning and teaching theories e.g. reflective cycle, adult learning

2 (40%) 1 (20%) 1 (20%) 1 (20%) 3.2 5

Planning CPD/training – devising sessions for different learners

2 (40%) 2 (40%) 1 (20%) 3.4 5

How to evaluate the effectiveness of CPD/training

1 (25%) 1 (25%) 2 (50%) 4.0 4

Effective feedback - giving and receiving feedback using scenarios

2 (20%) 2 (40%) 2 (40%) 4.2 5

Managing challenging learners: strategies to support them

1 (20%) 1 (20%) 1 (20%) 2 (40%) 4.0 5

So overall the questionnaire data from the two types of DDE course suggest that the greatest new learning was gained from:

• Tension and conflict (strategies to work towards resolution) • Poor performance (analysing and managing issues) • Learning theories (e.g. Bloom, Kolb, Grow, adult learning principles, reflective cycle). • Tutorials (planning and approaching them) • Managing challenging learners (strategies to support them) • Effective feedback

To a considerable extent this data was endorsed with the findings from the interviews. In talking to sample of individuals who had attended the courses, the main new learning mentioned was in relation to learning theory. Nearly half (8/13) commented that they had learnt considerably more about educational theory, with six specifically drawing attention to the principles of adult learning. This was described by all six as a process of learning about the importance of encouraging the learner to learn for themselves, of the value of “drawing out the answers from the trainee” rather than giving them the answers, “not to talk so much to the Foundation Dentists and to step back and let them do the talking”. As one characterised adult learning:

“…not giving the answers and solving things for the trainee so readily…being more student-centred, so that they do more than the work than me.”

Other learning gains mentioned by interviews were: learning styles (mentioned by 3/13), teaching styles (3/13 – for example how to use practical activities with groups), three mentioned greater understanding about their own limitations. Notably, three highlighted that it was difficult to articulate their learning from the DDE course because they had also been studying for an educational qualification, such as the CertMedEd at a similar time. Only two interviewees concluded that their new learning from the course had been fairly limited. Both of these were experienced trainers, and did not consider that they had needed to attend for their own personal learning, but had hoped it would be valuable for understanding more about what other educators in the deanery were being taught.


22

3.2.6 Impact on practice We know from research literature that sustained continuing education taken over a period of time, and using multi-methods, is more effective than a one-off event (Mansouri and Lockyer, 2007). The impact of the course on the participants’ educator role was therefore a key theme which was explored in the questionnaire, but was also discussed at some length in the interviews. One of the ways in which this issue was investigated in the survey was the inclusion of three statements in Question 10. These are set out in Table 12. TABLE 12: Overall views on the impact of the programme To what extent do you agree or disagree with the following statements? Please circle one number per line where 1=strongly disagree and 6=strongly agree.

Strongly disagree

Strongly agree Mean Valid number

(1,2) (3) (4) (5,6)

The course made me feel more invigorated as a dental educator

4 (5%) 7 (9%) 26 (32%) 45 (55%) 4.5 82 (100%)

As a result of the course, I have made several changes to how I approach my dental educator or teaching role.

3 (4%) 6 (7%) 18 (22%) 55 (67%) 4.8 82 (100%)

I learnt great ideas during the days but have not implemented them yet.

31 (38%) 27 (33%) 16 (20%) 7 (9%) 2.9 81 (100%)

As can be seen above, it is extremely positive that for about half (55%) of respondents they were very satisfied with the course: the course had made a considerable impact on their motivation levels as a dental educator. Another third (32%) were fairly

positive about its impact on their enthusiasm for the role, rating ‘4’ on the 6-point scale. However, 14% were much more negative – which given that we know that the average mean number of course participants was 15, equates to approximately 2 individuals at each course. Notably, again, an ANOVA test showed that experience as a dental educator showed no statistically significant results. In other words, views towards the impact of the course did not statistically differ according to the educator experience of the respondent.

At the heart of this evaluation has to be whether the DDE course impacts on the practice of dental educators. From responses provided in Table 12, about two-thirds (67%) strongly/agreed that they had made several changes to their educator practice. To find out more about this issue, the 13 interviewees were asked: “Can you think of any concrete examples of ways in which your approach to your dental educator role has changed as a result of your involvement in the DDE course?” Here, the interview data was very positive: nearly all (11/13) could readily offer ways in which they had changed their approach to their educational role. For many (6 of the 13) they highlighted how the course had given them new ideas or techniques in teaching or educating. For example, two of the DCPs had implemented different educational games (ice breakers) and assessment games which they had covered in the DDE course e.g. a noughts and crosses game for questions and answers about an issue. Another of the dentists talked about how s/he now uses role play with his/her FD1 trainee (e.g. role playing a difficult patient getting frustrated about his treatment) so that the FD1 “gets a varied mix of teaching”. Another DF1 trainer talked about how the course had encouraged him/her to think more about “why”, “how” and “what” and to try breaking up his/her teaching of clinical skills:

“I think it did make me start thinking about things a lot more I mean I’m in the position of being very much at the unconscious competence level of learning and having to stop and think ‘how’ I do things and ‘why’ I do things cause I do


23

so much of the dental stuff, so much on autopilot… made me a little bit more aware of how to break up what I did”

Another key way in which several interviewees (4/13) talked about the course impacting on their practice was their shift towards a more adult learning style with their trainees. Again, as mentioned earlier, this had taken the guise of trainers moving towards a more open questioning style, “drawing answers out of them”, rather than offering answers and solutions too readily:

“…the fundamental change is that I go in and I try and draw out from the trainee rather than going in and telling them what I think”.

Others highlighted that they now encouraged the trainee to take more responsibility for their own learning e.g. one trainer talked about his/her trainee helping plan and directly lead a school oral health session (rather than supporting the trainer), another highlighted that his/her trainee was now doing more of the preparation ahead of his/her own tutorials. Greater dedicated effort to reflect on practice was another key theme highlighted by four of the interviewees. For some, this was about reflecting on their own performance as a trainer; for others it was about encouraging their trainee to reflect and learn from their performance in a clinical situation. Finally, three mentioned that they were now using some of the educational theory in their day to day teaching; another two highlighted they were now more confident in their teaching role; and one drew attention to their renewed vigour to tackle a situation of conflict in his/her workplace. Barriers to implementation When interviewees were asked: “Were there any barriers to implementing change?” they were not particularly talkative. Only seven (of the 13) gave some feedback to this question. Four of them highlighted that the barriers were mainly personal, and related to finding the extra time and effort required to change the way you educate. For example:

“If the trainee says ‘oh this impression is wrong’ it takes me 2 minutes to do a new impression but it takes me 10 minutes to work them round… it takes longer to educate someone else rather than just to going in and doing it yourself” “It’s more to do with me as an individual whether I would spend that little bit of time developing it because it is always difficult at first to change techniques but once you do it becomes a lot easier the second time round

Another trainer commented that a planned approach to a tutorial can turn out differently in practice, because a tutorial is not a very ‘formal environment’. Only one educator mentioned the barriers which can be experienced by other staff being resistant to new ideas. Finally, one (an experienced trainer), cautioned that new trainers might have had greater difficulty implementing some of the ideas, particularly those with the models, because so much is new for them. However, this was not a message borne out by new trainers’ comments.


24

3.2.7 Intentions for further study of dental education Enhancing participants’ motivation and preparation for further study is a key goal of this course. The extent to which this course provided a realistic springboard into a postgraduate certificate and/or diploma in medical education was explored in interviews and in the survey. Learning needs Effective continuing professional development will be well matched to learning needs (Grant, 2002) and for this reason, one of the statements in Question 10 of the questionnaire asked respondents to indicate their views towards their learning needs in their dental educator role (as shown below). TABLE 13: Views towards CPD needs and provision To what extent do you agree or disagree with the following statements? Please circle one number per line where 1=strongly disagree and 6=strongly agree.

Strongly disagree

Strongly agree

Mean Valid number

(1,2) (3) (4) (5,6)

There are too few courses to support dental educators

11 (14%) 16 (20%) 21 (26%) 33 (41%) 4.0 81

I’m aware of aspects of my educator role that I need to improve

3 (4%) 4 (5%) 22 (27%) 53 (65%) 4.7 82

Views towards the availability of educator courses (There are too few courses to support dental educators) were mixed – clearly a minority (14%) perceived that provision was good, nearly another half (46%, rated 3 or 4) considered that it was adequate. However, as many as 41% strongly/agreed that there were too few courses for dental educators. But can respondents readily identify their learning needs? Clearly, the questionnaire data suggests that most (65%) feel that they are

aware of aspects of [their] educator role that [they] need to improve. ANOVA calculations for experience levels as a dental educator (3 years or less/more than 3 years) showed no no statistically significant results. Views towards their CPD needs and provision did not statistically differ according to the educator experience of the respondent.

This was explored further in Question 17 of the questionnaire which asked respondents to specify whether ‘…there are any specific areas you’d like further training and support in relation to being an educator’. An open comments box was provided and 25 of the 82 individuals provided a detailed response (40%). Several others (n=8) simply noted that they had ‘none’ or ‘not at the moment’ comments about training and support. The comments from the 25 individuals who provided a detailed response have been coded into four main categories:

1) specific topics they would like; 2) strategies or ways of accessing educational support that they would like; and 3) other miscellaneous comments about ‘education’ more broadly.

The frequency of these categories is set out in Table 14 overleaf:


25

TABLE 14: Comments about further training and support they would like

Total

(% of respondents) (n=25)

Topics Portfolio requirements/assessments Mentoring/coaching skills Planning tutorials Presentation skills

4 (16%) 4 (16%)

2 (8%) 1 (4%)

Strategies More opportunities for reassurance/ feedback/contact with others Practical courses/training Further education and training generally Background reading

5 (20%)

4 (16%)

2 (8%) 1 (4%)

Other 5 (20%)

Total responses 28

As shown in Table 14, several individuals made a more lengthy comment which addressed more than one of the coding categories, hence the number of responses equals 28 from 25 individuals. Comments made about ‘topics’ they wanted more training were about: 1) the electronic dental foundation training portfolio, and the assessments within it (e.g. ‘More training in the changing paperwork/electronic PDP used for the evaluation of the trainees’) and 2) wanting more mentoring and coaching training (‘Perhaps in future more mentoring training’). One of the main suggested ‘strategies’ for further training and support included wanting greater opportunity for more feedback and contact with others (both peers and leaders) about their educator role. Comments in this respect included: ‘A website forum for discussing issues confidentially’ ‘An optional de-briefing session at the end of FD/VT year would be useful’ ‘Assessments of teaching skills in practice would be useful but difficult to achieve’ ‘Review, meet-up with course participants for feedback now we are training’ ‘Feedback would be valuable from an assessor’ Clearly there is an interest amongst some in being part of a more supportive trainer community and having ‘continuous updated reassurance’ from those in an educational role for dental educators. Notably, three of the five were in their first few years in an educator role. The provision of more ‘practical’ courses was also important for several respondents. Refresher dental educator courses were mentioned by two, for example: ”Mostly practical courses similar to the dental educators course – not qualifications for qualifications sake”. One mentioned that s/he would like a “refresher course yearly”. These comments were endorsed by many of the interviewees when they were asked for recommendations for the future (see 3.2.8, p.29). Motivations To find out DDE course participants’ levels of motivation towards studying for a postgraduate certificate in medical education, the questionnaire (Question 15) asked them to rate their motivation levels on a six point scale, where 1 represented very low motivation and 6 represented very high motivation (see Table 16).


26

TABLE 15: Motivation levels to study for CertMedEd How motivated are you to study for a postgraduate certificate in medical education (CertMedEd)

Frequency

Percent

Very low motivation, 1 12 16.7

2 8 11.1

3 11 15.3

subtotal (31) (43.1%)

4 22 30.6

5 12 16.7

Very high motivation, 6 7 9.7

subtotal (34) (47.2%)

Total responses

72 100%

As shown in Table 15, attitudes amongst respondents towards studying for a CertMedEd were quite varied with nearly even proportions who reporting they were motivated (rated 4, 5, 6) or not motivated (1, 2, 3). This is reflected in the mean score 3.5, and is not surprising: this was a diverse group of educators with very different levels of experience and we know that studying for a PGCertMedEd is not a practical idea or even an aspiration for all individuals. But in terms of planning for future uptake of a CertMedEd, there was a pocket of real interest in pursuing a CertMedEd. About a quarter (26%; 19/72) were found to be highly/motivated and rated 5 or 6 on the rating scale. If this proportion were to be extrapolated, this data suggests there could be as many as 35 DDE participants currently very interested in undertaking a CertMedEd (the NESC database shows 135 individuals who had attended the DDE course had not yet started a CertMedEd). The interviews enabled greater exploration of their motivations towards beginning a CertMedEd. Of the 98

DDE course participants interviewed who had not done a CertMedEd, 2 of them had definite plans to pursue this qualification (one had a start date, the other had already decided the HEI) and another 2 considered that they were very motivated to do it, but cited the immediate constraints related to time and family commitments. These comments endorse the broad findings from the questionnaire data, approximately 2 in 10 dental educators who took part in this evaluation and had not yet started a CertMedEd reported that they were seriously interested in starting one.

Views towards the CertMedEd Sadly, only 8 individuals who responded to the survey provided their views towards the completion of their CertMedEd. For this reason, the 6-point rating scale data has been collapsed into two categories: those rating 1,2 or 3 (low to mid point on the scale) and 4,5 and 6 (mid to high point on the scale). TABLE 16: Ratings towards aspects of the CertMedEd

If you have already started or completed a CertMed, how would you rate the following:

Low (- mid) (mid -) High Mean Valid number

(1,2,3) (4,5,6) The overall quality of the CertMedEd 1 (12.5%) 7 (87.5%) 4.0 8 The extent of new learning you gained/are gaining from the CertMedEd

1 (12.5%) 7 (87.5%) 4.5 8

The impact of the CertMedEd on your dental educator or teaching role

3 (37.5%) 5 (62.5%) 4.0 8

8 4 had already completed their CertMedEd.


27

Amongst these 8 people, and across these three aspects, views towards the CertMedEd were positive. Although it may be interesting to explore whether views differ according to which course is/was attended, this was not appropriate given the small numbers of respondents. First, in terms of the overall quality, most of them (75%; 6/8) gave a rating score of 4.0, suggesting that the quality was viewed as ‘good’ by most individuals, and only ‘excellent’ by a minority (13%, 1/8 rated a score of 5.0). This is reflected n the mean score of 4.0. The highest ratings were given for new learning where more than a third (38%; 3/8) rated that their new learning was ‘high’, rating 5 or 6. The mean was highest for this aspect, with a score of 4.5. Views towards the impact of the CertMedEd on their dental educator or teaching role were most varied amongst these aspects: half of respondents provided scores in the mid-range: 3.0 (25%, 2/8) and 4.0 (25%, 2/8) – with most of the rest of respondents at the very positive end of the scale. Five individuals who were interviewed had already started (or completed) their CertMedEd (2 in Winchester 2, Bedfordshire 1, Oxford Brookes 2). They were asked for their feedback about the CertMedEd and their motivation for pursuing a Diploma in Medical Education. The first point to make is that all five were positive about their completion of the CertMedEd – although one didn’t elaborate much about it. They each had their specific reasons for choosing to study the certificate at their host higher educational institution (HEI): one drew attention to the provision of a Certificate in Dental

Education (Bedfordshire); the particular facilitator (Katy Newell Jones) was known to him/her from the DDE course (Oxford Brookes); positive feedback from others (Oxford Brookes); timing of the course in the working week (Winchester); had been motivated by the DDE course (Oxford Brookes); another dentist was doing it from his/her area (Oxford Brookes); wanted a clear pathway through to the Masters (Bedfordshire); and the interactive style (Bedfordshire).

The limited feedback from these five individuals showed that two had been motivated to start their CertMedEd after a previous course - one individual by the two-day trainer preparation course and one by the DDE course. Although numbers here are small, this does suggest that the DDE course can influence participants to look further to build their educator qualifications. Indeed, one of the interviewee’s commented that it would be helpful if the DDE course had an award bearing component such as the credits gained could be offset against part of the CertMedEd at Oxford Brookes. Finally, it is notable that two of these five were already studying beyond the certificate level: one was completing their diploma in medical education, and the other was doing his/her masters level qualification. Two of the others had the motivation to study at diploma level but considered they had insufficient time at present.


28

3.2.8 Recommendations In the interviews, participants were asked: “Are there any suggestions you would make if this programme were to be further developed?” In response, there was a genuine spread of feedback. Six individuals basically provided comments or ideas for how the existing course could be improved, 6 made suggestions for a future follow-up course for his/herself, and three people addressed both – provided a general comment about the existing course and also gave ideas for the future. First, the nine people who made comments or suggestions about how the DDE course could be improved: two simply said that it was “very complete” and “more of the same”. Out of the other seven, several were concerned that the course would be better facilitated by two people, in particular someone with a clinical background or an “experienced educator who has been on the course” (n=3). Two of these elaborated that the link between theory and practical application could have been maximised if an experienced dental educator had been co-facilitating, the other was concerned that the group was too large for one person to facilitate all day alone. On another issue, two (both experienced educators) noted that it might have been better if experienced and new trainers could have been split. Two felt the four-day course was too long and some parts of the programme could be “condensed”: both mentioned the group work activities which involved building a tower. Another commented that s/he thought there was insufficient opportunity to bring his/her own issues for discussion – this links with suggestions for the future which many made (see below). One would have liked greater opportunity for interaction between dentists and DCPs – for at least some of the sessions. Finally, as noted previously, one suggested that it would be good if the DDE course formally linked with the CertMedEd so that credits could be awarded which could be offset against aspects of the CertMedEd. The course lead, Katy Newell Jones, also made some insightful comments about some of the ways in which the DDE course could be revised. Suggestions included: separating out some of the sessions as ‘teaching clinical skills’ from ‘teaching about communication’ so that there was greater distinction in the programme about different types of one-to-one teaching. She hopes also to develop the assessment session so that it had greater orientation around the practical tools, such as the case based discussion, and other tools in the portfolio and might include some videos illustrating different types of assessments in practice. Ideas for the future were made by 9 DDE course interviewees and nearly all of these (n=8) wanted a follow-up course to provide a chance to:

• Discuss specific scenarios/issues • Provide an ‘update’ on anything new • Share experiences, since the DDE course (e.g. new trainers together) • Provide greater depth on specific topics; those mentioned were tutorials, specific difficulties

(like dyslexia) The other individual simply commented that s/he would be happy to go on similar future courses. The course lead could also see the value in providing periodic opportunities for DDE participants to share their educator experiences at a later date: “there could be value in an opportunity for sharing, or seminars or something, not very often”. In addition, she felt that some of the DDE course participants could lead or contribute to a dental educator session, and would benefit from the opportunity to take a more active role in teaching about dental education.

Conclusions

29

CONCLUSIONS In this Conclusion, we draw together the data from the different phases of the research to distil three main conclusions. The first considers the extent to which the programme has met its defined purpose (as set out on pg 2), the second considers whether the DDE course had energised individuals to educate themselves beyond the immediate programme, and third, the overall conclusions and recommendations are considered.

Achieving the programme’s purpose Four main goals or purposes of the DDE programme9

were set out in the DDE course materials. The evaluation has not sought to provide a ‘check-list’ as to whether these have been achieved, but rather be more inclusive and consider the learning gained across a range of topics and the broader educational benefits of taking part. However, it would be remiss not to at least consider the extent to which the programme achieved its intended purpose.

1. To explore the theory of adult learning and its application to dental education was a key goal for the DDE course. Without doubt, the evaluation has demonstrated that this was achieved. The questionnaire responses and those who were interviewed show that the value of employing adult learning approaches was a clear and central message that permeated the programme. Some described the course lead, Katy Newell Jones, as modelling adult learning in her approach towards the facilitation of the DDE course. Many others recognised that learning how to approach the trainee with an open questioning style, adopting a student-centred approach, and not seeking to solve questions, but facilitate their own resolution, had been an important aspect of the learning they had gained from the course. A few of those interviewed drew attention to examples of adopting a more adult oriented approach with their trainee, and in this sense, the message had started to impact on their educator practice. Overall, the inclusion of ‘learning theories’ was highly regarded and many respondents at both the four-and the two-day courses reported that they had achieved a great deal of new learning in this area.

2. To explore creative approaches to one-to-one teaching was another purpose for the DDE

course. Interviewees from the DCP course readily identified ideas which they had implemented from the course e.g. ice breaker games, quizzes etc. They each talked about ways in which they had gathered new interactive ideas for teaching groups of learners. Learning more about ‘learning styles’ was an integral part of them considering different ways of teaching, and this was a well received topic by DDE respondents at both the two- and four-day programmes. In terms of one-to-one teaching, the emphasis on role play was a key tool for learning which was emphasised in the DDE course. On the whole, this was received positively by participants – though many had initial reservations – and many felt that it had taken them out of their own comfort zone. The data suggests that the DDE course was successful at encouraging participants to move towards greater interactivity in planning the content of a learning event, and shifting away from didactic-style solutions to their trainee’s learning needs. One interviewee even talked about using role play with his/her own trainee, such was his/her enthusiasm for this approach to learning. However, only one interviewee talked at length about how she had been more creative in a learning event with her trainee; whilst she would normally deliver an oral health session in a school herself, following the DDE course she had worked closely with her trainee to plan and

9 For simplicity, this section of the report is structured around the purposes of the four-day programme, since this is the course format which 82% of course participants had attended.

Conclusions

30

deliver aspects of the session so that the trainee could deliver some of the oral health session. Examples such as this, however, were limited; further exploration of creative approaches to one-to-one teaching could be extended in future courses. Using videoed case studies of real teaching scenarios may be one way of doing this.

3. Adopting practical and effective tools for assessing learning was a topic that was well

received by more than two-thirds of the participants, and nearly half (45%) of those on the four-day course considered they had learnt a ‘great deal’ of new learning about assessment by taking part in the course. This was particularly so for less experienced educators who rated their new learning from the assessment topic significantly higher than those who were more experienced. On the two-day course, ‘how to evaluate the effectiveness of CPD training’ was also rated highly in terms of learning gained. Giving and receiving feedback too was a topic which many (particularly more novice educators) had gained a great deal of new learning, and was considered by the course lead as an integral part of teaching, learning and assessment. On the face of it, such findings are positive; however, the course lead considered that the assessment theme could be expanded in the future courses, with greater discussion of the assessments incorporated in the dental foundation portfolio. This was also emphasised strongly by one of those interviewed and would be a practical way of embedding participants’ learning about assessment into a real and practical context in which they are expected to function.

4. To explore tips for effective tutorials was a topic which featured on the four-day course,

and more than three-quarters (79%) of questionnaire respondents considered its inclusion was excellent. Indeed, this was the highest rated topic on the programme, and more than half (53%) went on to report that they had gained a great deal of new learning about tutorials. Less experienced educators significantly rated this session the most positively. Indeed, many of the interviewees’ illustrations about how the course had impacted on their practice were about the changes that they had made to their tutorials. These focussed on changes to their approach, and adopting a more adult education style.

Overall, the programme successfully achieved its principal purpose of supporting the dental educators to be more aware about learning theory, and specifically ways to approach trainees (or other learners) from a more adult-oriented perspective. For the course lead, this had been her one “overarching aim…getting the learner learning and them taking responsibility and doing the work” The focus on tutorials was also particularly well received, but more could perhaps have been made of other creative approaches to teaching, particularly in the four-day course. The participants at the two-day course had particularly valued the novel teaching ideas that had been included. Finally, although assessment was a key topic for the course, the course lead had some excellent suggestions about how this could be expanded for future DDE courses, particularly in the four-day course, and these will be an excellent addition to the programme.

Stimulating continuing education amongst dental educators On the whole, the vast majority (87%) of respondents agreed that the course had made them feel more invigorated as a dental educator. This is very positive – especially since about half of those interviewed had had reservations about attending. Clearly, the data suggests that most of those who had been initially sceptical about what they would gain from the course were more positive about it afterwards. This is, in part, testimony to most respondents’ positive feedback about the quality of the facilitation by Katy Newell Jones.

Conclusions

31

A key goal of the DDE course was to motivate participants to continue with their professional development as educators. The evaluation suggests that this was important for DDE course respondents too: two thirds of questionnaire respondents (65%) were aware of aspects of their educator role that they needed to improve, and there were several suggestions from them for further targeted training or support. Many (47%) went onto express interest in undertaking a CertMedEd, and a quarter (26%; 19/72) was very motivated. This was endorsed by the interview data. Given that we know from the NESC database that about 10% (14/149) of all DDE participants had already started (or completed) a CertMedEd, it would be an achievement if another cohort (about 19 from the 82 respondents) could be encouraged to pursue a CertMedEd. Further discussion with lead educators in the deanery may help individuals interested in a CertMedEd make the next step. So too could discussion with individuals who have already achieved the CertMedEd; feedback from the interviews suggested individuals would happily talk about the core HEI providers across the Oxford and Wessex deaneries, and it may be good to signpost interested individuals to some of these people. However, it is notable that it is still only a minority of DDE course participants who are seriously interested in planning to undertake the CertMedEd. For most, this DDE course is a stand-alone CPD training course that needs to equip dental educators to progress as learners (themselves) and teachers – whether new or experienced in their educator role. Further opportunities for follow-up, whereby course participants could meet up and discuss particular challenges would be welcomed by many. It would also provide even greater opportunity for dental educators to revisit and reinforce the learning gained from the DDE course in their educator practice. This would fit well with the research evidence that effective CPD is undertaken over a period of time, interactive and includes on-the-job opportunities to reinforce learning in practice (Eraut, 2007; Davis, 1999; Cantillon and Jones, 1999).

Conclusion and recommendations Overall, the DDE course was well received by most of those that responded to the evaluation. This is extremely positive – especially since up to about half of them may have not proactively chosen to attend this course. The topics were all well rated, and examples of new learning and impact on practice could be illustrated by most of those interviewed. The delivery, with an emphasis on role play and group work, was successful, although a few felt that some of the group activities could be more condensed. The key message of supporting dental educators to adopt a more student-focussed, adult-learning approach with reflection as central was received by participants. Katy Newell Jones’ teaching style was pivotal in reinforcing this message throughout the course. The content of each DDE course was particularly relevant to dental educators in a direct, face-to-face educational role with other members of the dental team, rather than those in an educational or coordinating role with other dental educators (e.g. training programme directors). Not surprisingly, less experienced educators gained significantly more learning from some of the topics. It is important to highlight the tension, however, in bringing together experienced and novice educators into one course. For most, this proved a strength of the format (e.g. sharing experiences), but for some (particularly experienced educators) they would have liked greater opportunity for separate discussion on particular issues. On balance, mixed groups of educators should continue for the DDE course, but greater effort could be made to separate into subgroups of differing experience for some of the group work. For those who have already started (or have committed to start) a CertMedEd, or an equivalent educational qualification, it is appropriate that they should continue to be exempted from compulsory attendance at the DDE course.

Conclusions

32

Two topics merit particular focus in future courses: tutorials, and assessment. Practical tips and planning for tutorials was particularly welcomed by respondents and the course lead endorsed how important it is that this emphasis is retained (or even developed) in future courses. Assessment, and in particular, how assessment is undertaken in the dental foundation portfolio is also a topic that could be strengthened. Again, the course lead had particular ideas how this could be achieved. Finally, NESC Dental School should be congratulated for funding an extended short course for dental educators which maps well onto several of the domains incorporated in the Guidelines for Dental Educators (2008). It is very positive that many respondents expressed an appetite for further learning about dental education, and some will continue to a CertMedEd, however, many of the rest would benefit from further opportunities to follow-up the course and discuss their experiences as trainers.

References

33

REFERENCES Baldwin, P J, Dodd, M and Rennie, J S ( 1998) Postgraduate dental education and the new graduate

British Dental Journal; 185: 591 – 594. Cabot, L B, Patel, H M and Kinchin I M (2007) Dental vocational training: identifying and developing

trainer expertise British Dental Journal; 203 (6): 339-345. Cantillon, P, Jones, R (1999) Does continuing medical education in general practice make a

difference? Brtish Medical Journal; 318 (7193): 1276-9. COPDEND (2008) Guidelines for Dental Educators: A framework for developing standards for

educators of the dental team. COPDEND, UK. Davis, D, Tomson, M A, Freemantle, N, Wolf, F M, Mazmanian, P, Taylor-Vaisey, A (1999) Impact of

formal continuing medical education. JAMA; 282 (9): 867-74. Denzin, N K (1970) The Research Act in Sociology. Aldine, Chicago. Eraut, M (2007) Learning from people in the workplace. Oxford Review of Education 33 (4): 403-22. Gibson, C J (2005) The assessment of dental vocational training: a personal view. Dental Update; 32:

552-555. Grant, J (2002) Learning needs assessment: assessing the need. British Medical Journal; 324 (7330):

156 – 9. Kirkpatrick, D L (1967) ‘Evaluation of training’ in Craig, R L and Bittel, L R (eds) Training and

Development Handbook. McGraw-Hill, London. Mansouri M, Lockyer, J (2007) A meta-analysis of continuing medical education effectiveness Journal

of Continuing Education for the Health Professions, 27 (1): 6 – 15. Ralph, J P, Mercer P E and Bailey H (2000) A comparison of the experiences of newly-qualified

dentists and vocational dental practitioners during their first year of general dental practice British Dental Journal; 189: 101-106.

Appendix 1 Interview Schedule with DDE Course Participants

An Evaluation of the NHS Education South Central (NESC) Developing Dental Educators (DDE) Course

Interview schedule

Thank you for agreeing to be interviewed for the Developing Dental Educators Course evaluation study which has been funded by NESC. As I mentioned in my letter, and you will have read in the Participant Information Sheet, CRMDE has been funded to evaluate the Developing Dental Educators Course, which was first introduced in August 2008.

Thank for completion of the consent form, but reiterate that the interviewee still has the right to withdraw at any point in the interview, or to pass on any question that is asked. Reiterate that there is no right or wrong answer, and that confidentiality is assured. No individual will be identified in any reporting of the data to funders or the course tutor.

Explain purpose of the interview, which is to explore your views and experiences of taking part in the DDE course. Should take about 30 minutes. Ask for permission to record the interview. Clarify that this record will be securely stored at the university, and any information used will not be attributed to the interviewee.

Background and experience of being a dental educator

o Can you tell me about your dental background? o Dentist/DCP, hospital/general dental practitioner o Length qualified o If GDP - single/multi, length qualified, practice owner, NHS commitment.

o What type of dental educator are you/were you when you attended the DDE course? [e.g. vocational trainer/therapist vocational trainer/educational supervisor]

o Are you still a dental educator? o How long have you been a dental educator? o In the same practice? Same deanery? o Every year?

o Do you have experience of any other educational roles in the deanery or elsewhere? o Any relevant medical/ dental education qualifications that have complemented the

DDE course? [e.g. PGCertMedEd) If yes – going to talk about this later.

Motivations • Can you tell me the background of how you came to get signed up for the DDE course?

o How did you feel about going on it? • When you started the course, what did you specifically hope to gain?

• How difficult was it to organise and sustain your involvement over the 4 day DDE course?

Response to the DDE course • Thinking about the study day programme

o Was it what you expected? Any particular days of more value than others? What was the highlight? And the least useful aspect?

content:

o Did it cover the areas you wanted it to cover? o Any overlap with other dental educator courses you have attended? o Relevant to your role as a dental educator? Your learning needs?

• Thinking about the study day o What was the main format (didactic, small group, individual reflection)? What did

you think of this balance?

delivery:

o I understand you needed to complete a written assignment, what are your thoughts about that aspect of your involvement in the course?

o How many people were on your course? New/experienced dental educators? How did the number and profile of other participants affect your learning?

o How did it work out, with the course content spread across 6 – 8 weeks? Advantages/disadvantages?

o How would you rate the facilitation of the course? The facilitator was an ‘educator’, rather than a ‘dentist’ – what are your views?

• What do you think was the extent of new learning through taking part in the DDE course?

o Gains in knowledge, skills, attitudes? o Have you shared any of those learning gains with other colleagues outside the

course? o Any other gains – on your confidence, your preparedness for your dental educator

role?

Impact on your role as a dental educator

• Can you think of any concrete examples of ways in which your approach to your dental educator role changed as a result of your involvement in DDE course?

- Are there any things you did differently as a result of the course? - Are there any ways in which it made a difference to your teaching/educator role? In what ways? - Were there any barriers to implementing change? Any examples? Further training and development A key purpose of the DDE course is to prepare participants to undertake further medical or dental education study, such as a Postgraduate Certificate in Medical Education (CertMedEd) or Postgraduate Diploma in MedEd. • Are you aware of the available local provision for further study of medical/dental

education? o If yes, do you have any views of the available provision? o Heard anything from others who have undertaken them? o If no, and you were interested, who would you talk to about finding out

more?

• Would you consider starting further study in medical/dental education, such as a PGCert in Med Ed? Or have you already started/completed such further study?

o If yes or no, has this decision been affected by the DDE course influenced your motivation?

o How did it differ?

Overall Reflections • What are your overall reflections of the DDE course?

• Are there any suggestions you would make if this programme were to be further developed?

o If you were to attend a follow-up course in a few years, what would you suggest for the content, format?

• Any other comments?

Many thanks for your time today. A short questionnaire will follow in a few months as the second part of this evaluation study. This will be sent to all participants of the Developing Dental Educators Course and will pick up

some similar questions as we have discussed today about the course. Please can I ask you if you would mind completing it – even though you have been interviewed today. It is important to

ensure as good a response rate as possible for the questionnaire and it will only take 10 minutes of your time. The final report of the evaluation will then be available in early 2011.

Appendix 2 Interview Schedule with Course Lead

An Evaluation of the NESC Developing Dental Educators Course

Interview schedule with the Course Lead

Thank for willingness to be interviewed. Reiterate that there is no right or wrong answer. Explain purpose of the interview. Should take about 45 minutes. Ask for permission to record the interview. Clarify that this record will be securely stored at the university, and any information used. Discuss fact that the course lead’s comments will be attributed to the interviewee in the report to funders as she is the only course lead. It is important to discuss this issue and explain that a copy of the transcript can be sent for her review. Reassure that comments can be withdrawn after the interview.

Background Details • What is your background and role as a dental educator?

o Length of time worked as an educator?

• How and why did you get involved in the Developing Dental Educators Course? • How would you describe your role in setting up the course?

o Responsibilities – devising the programme? Target participants? Length? • And what about the DCP course? How did this separate course arise? • What is the fit of this course with other courses (e.g. trainer preparation, CertMedED)? The Developing Dental Educators Course: Aims • As a key player in setting up this programme, what do you see as the key purpose of the course?

o For the trainer? How will s/he be better educated by having attended it? o What has been the learning gained? Impact of the course? o Have experienced dental educators gained differently from the course than novice trainers?

o For the trainee? What are the intended knock on effects for trainees?

• Overall, what are your views on the impact of the course on participants? Content • Thinking about the study day programme

o How did you plan the programme? content:

o Any particular days of more value than others? o What’s your reflection on the content? Does it cover the areas you wanted it to cover? Gaps? Or

things you’d drop? o Feedback from participants?

• Thinking about the study day

o The format of the course varied between didactic and small group, what do you think of this balance? And why did you craft it in this way?

delivery:

o Role play seems to be a contentious aspect of the course from the feedback so far, why was there a particular emphasis on this in the course?

o New/experienced dental educators? How did the number and profile of other participants affect their learning?

o What’s your view on the value of the written assignment? Follow-up

• What would you see as the appropriate way for participants to follow-up this course? o Would you see another short course as being useful? o CertMedED?

Overall Reflections • How is your overall impression of the pilot?

o Any feedback from DTs, or from dentists working with them?

• What have been your experiences of being involved in delivering this course? o Highlights/lowlights?

• Any thoughts about how it might be improved?

• Has it worked out as expected? Any particular challenges?

• Any other comments?

Many thanks for your time.

Appendix 3 Questionnaire Tool for DDE Course Participants

The Developing Dental Educators Course: Your Views

The Centre for Research in Medical and Dental Education (CRMDE) at the University of Birmingham has been asked by Mr Shalin Mehra on behalf of the Oxford and Wessex Deanery to evaluate the ‘Developing Dental Educators Course’ facilitated by Dr Katy Newell-Jones. The deanery database indicates that you are one of the practitioners who have attended the course and this questionnaire seeks your views about it. It will take just a few minutes to complete. Completed questionnaires will be confidential to the researcher (Vickie Firmstone). Please return using the pre-paid envelope provided. Many thanks.

Dr Vickie Firmstone, Research Fellow, CRMDE, School of Education, University of Birmingham Tel: 0121 414 4404

About You 1. Are you? Male Female

2. When did you first qualify? ____________________year

3. What is your current role? Dentist Therapist/Hygienist Dental Nurse Other:_____________ 4. In what type of setting do you mainly work? GDS Salaried HDS Other, please specify: _________________________________________ About Your Dental Educator Role 5. Are you currently employed as a dental educator? This includes any educational role for members of the dental team. Yes No

5a. If yes, how would you best describe your current educator role? DF1/VT trainer DF2 trainer Therapist Vocational Trainer Training Programme Director CPD tutor Nurse educator Oral health promoter Other, please specify: ___________________________________________________________________________ 5b. If no, have you been employed as a dental educator since August 2008? Yes, please specify role: ___________________________________ No

6. Have you completed (or recently started) or any other specific qualifications related to being an educator? (e.g. CertMedEd, teaching skills, nurse educator training)

Yes, please specify:_______________________________ location/organiser: _________________________ No 7. Over your career, how many years’ experience have you had in a dental educator role? We know these roles are usually undertaken part-time and that your years could also reflect different positions or employers (e.g. if you worked half time for two years, count this as two years) none less than 1 year 1 - 3 years 4 - 6 years 6 years or more

Please turn over

Your Attendance at the Developing Dental Educator Course Ten ‘Developing Dental Educators’ courses have been delivered since 2008. Courses 1 – 5, and A to C were for 4 days; DCP courses were for 2 days. 8. When did you attend your ‘Developing Dental Educators Course’? Please tick.

Attended

Started in 2008 Course 1 August 2008 – February 2009 Course 2 November 2008 – April 2009

Started in 2009 DCP 2009 February 2009 – March 2009 Course 3 May 2009 – July 2009 Course 4 September 2009 – January 2010 Course 5 November 2009 – February 2010

Started in 2010 DCP 2010 February 2010 – March 2010 Course A March 2010 – April 2010 Course B March 2010 – May 2010 Course C June 2010 – August 2010

9. How easy or difficult was it for you to attend all the days required for the course? Please circle where 1 = very difficult and 6= very easy, no difficulty at all.

Very difficult Very easy, no difficulty at all 1 2 3 4 5 6 Do you have any comments about the ease or difficulty you experienced in attending the required days?

Your Overall Views 10. To what extent do you agree or disagree with the following statements? Please circle one number per line where 1=strongly disagree and 6=strongly agree.

Strongly Strongly disagree agree

One of the best things about the course was the contact with the other participants.

1 2 3 4 5 6

The course was relevant to all dental educators, including more experienced and less experienced educators.

1 2 3 4 5 6

As a result of the course, I have made several changes to how I approach my dental educator or teaching role.

1 2 3 4 5 6

There are too few courses to support dental educators.

1 2 3 4 5 6

I learnt great ideas during the days but have not implemented them yet.

1 2 3 4 5 6

Attending this course was a waste of time for me. 1 2 3 4 5 6 The course made me feel more invigorated as a dental educator. 1 2 3 4 5 6 Most of the course content was relevant for me in my current role.

1 2 3 4 5 6

The way the course was delivered helped me learn (e.g.discussion, role play).

1 2 3 4 5 6

I’m aware of aspects of my educator role that I need to improve. 1 2 3 4 5 6

Do you have any comments about the statements above? Please specify.

Your Views on the Content of the Developing Dental Educator Course

11. What are your views on the content of the course? For each of the days in the programme you attended please indicate:

(a) Its overall quality on a scale of 1 to 6 where 1 = very poor and 6 = excellent. (b) Extent of new learning on a scale of 1 to 6 where 1 = no new learning/reassurance/refresher only

and 6 =a great deal of new learning. If you attended the four day course, please complete Table 1. If you are a DCP who attended the two day course, please complete Table 2.

Table 1 – Dentists’ four day programme Core topics

Overall quality (1=very poor; 6=excellent)

Extent of new learning (1=no new learning/ reassurance/ refresher only; 6=a great deal of new learning

Day 1: Introductions: understanding trainees and the trainer role 1 2 3 4 5 6 1 2 3 4 5 6

Learning styles and their relevance for learners and trainers 1 2 3 4 5 6 1 2 3 4 5 6

Day 2: Learning theories e.g. Bloom, Kolb, Grow, Adult Learning Theory 1 2 3 4 5 6 1 2 3 4 5 6

Tutorials: planning and approaching them 1 2 3 4 5 6 1 2 3 4 5 6

Day 3: Assessment in theory and in the dental practice 1 2 3 4 5 6 1 2 3 4 5 6

Feedback: giving & receiving feedback to maximise learning 1 2 3 4 5 6 1 2 3 4 5 6

Day 4: Poor performance: analysing and managing issues 1 2 3 4 5 6 1 2 3 4 5 6

Tension and conflict - strategies to work towards resolution 1 2 3 4 5 6 1 2 3 4 5 6

Table 2 - DCPs’ two day programme Cope topics

Overall quality (1=very poor; 6=excellent)

Extent of new learning (1=no new learning/ reassurance/ refresher only; 6=a great deal of new learning

Day 1: Introductions: understanding the role of DCPs as trainers 1 2 3 4 5 6 1 2 3 4 5 6

Learning styles and their relevance for your own role in CPD 1 2 3 4 5 6 1 2 3 4 5 6

Learning & teaching theories e.g. reflective cycle, adult learning 1 2 3 4 5 6 1 2 3 4 5 6

Planning CPD/training – devising sessions for different learners 1 2 3 4 5 6 1 2 3 4 5 6

How to evaluate the effectiveness of CPD/training 1 2 3 4 5 6 1 2 3 4 5 6

Day 2: Effective feedback: giving & receiving feedback using scenarios 1 2 3 4 5 6 1 2 3 4 5 6

Managing ‘challenging’ learners : strategies to support them 1 2 3 4 5 6 1 2 3 4 5 6

12. What impact did the course have on you? Please be as specific as possible (e.g. changed something you do with your trainee, how you plan your sessions,

how you respond to the demands of your role, how you evaluate your sessions, your motivation or confidence levels).

Please turn over

Best and Worst 13. What were the best and worst things about the ‘Developing Dental Educators’ course?

Best: Worst:

Recommendations 14. How might the course be improved? (e.g. particular changes that you think should be made for this course?)

Additional education and training 15. How motivated are you to study for a postgraduate certificate in medical education (CertMedEd)? If you

have already started (or completed a CertMedEd), please go direct to Q.15a and tell us what you thought of it. Very low motivation Very high motivation 1 2 3 4 5 6

Q15a. If you have already started or completed a CertMedEd, how would you rate the following?

Low High The overall quality of the CertMedEd. 1 2 3 4 5 6

The extent of new learning you gained/are gaining from the CertMedEd. 1 2 3 4 5 6

The impact of the CertMedEd on your dental educator or teaching role. 1 2 3 4 5 6

16. Do you have any comments about studying for a Certificate in Medical Education? For all respondents.

17. Are there any specific areas you’d like further training and support in relation to being an educator?

Any other comments?

MANY THANKS for completing this form. Please return using the pre-paid envelope by Thursday 11th November 2010

final report - oxford deanery evaluation final report... · 2011-10-25 · process. contents list...

Documents