Download - Critical Appraisal on 24.10.12
-
7/30/2019 Critical Appraisal on 24.10.12
1/36
- Critical Appraisal for Qualitative Article -
Wednesday 24th Oct 2012
Akira NAITO MD PhD
CT2 to Dr Kezia Lange
Mixed-methods evaluation of
a leadership development programme
for higher specialist trainees
-
7/30/2019 Critical Appraisal on 24.10.12
2/36
Background
Newly appointed consultant
psychiatrists have reported that
management and leadership is anarea for which they are unprepared.
-
7/30/2019 Critical Appraisal on 24.10.12
3/36
Background 2
Hewson et al and Spurgeon & Vassilas
introduced management courses.
Feedback suggested that these initiativeswere well received, but the samples in both
studies were made up of newly appointed
consultants only
As the evaluations took place straight after
training the authors were unable to examine
any lasting impact.
-
7/30/2019 Critical Appraisal on 24.10.12
4/36
The Mersey Care NHS Trust specialist
registrar leadership development programme
This aimed to raise awareness and to
increase engagement in medical leadership
and management.
This consisted of bi-monthly meetings for
trainees in their final 2 years of training.
Each meeting was facilitated by two of the
authors and included a guest speaker whowould describe their area of work.
A more structured action learning set would
take place to take forward ideas.
-
7/30/2019 Critical Appraisal on 24.10.12
5/36
Key principles of action learning
a. learning starts from not knowing;
b. People, who take responsibility in a
situation, have the best chance of takingactions that will make a difference;
c. learning involves both what is taught and
read and a questioning insight; and
d. learning should be greater than the rate of
change
-
7/30/2019 Critical Appraisal on 24.10.12
6/36
Study aim
Study aim was to evaluate the impact
of a leadership development
programme based on the principles ofaction learning for higher trainees.
-
7/30/2019 Critical Appraisal on 24.10.12
7/36
1. Was there a clear statement of theaims of the research?
Consider:
What was the goal of the research?
Why is it important?
Its relevance?
-
7/30/2019 Critical Appraisal on 24.10.12
8/36
Questionnaire survey
A questionnaire was drawn up to
examine the impact of the leadership
development programme based on thefindings of previous papers.
Sent electronically and by post to 54
trainees & consultants who hadattended the programme.
-
7/30/2019 Critical Appraisal on 24.10.12
9/36
Qualitative interview
A semi-structured interview schedule was
drawn up on the basis of the results from
the questionnaire. The schedule consisted of 19 items
covering the domains of perspectives on:
a) Leadership development programme;
b) Management and leadership; and
c) Transition to consultant psychiatrist.
-
7/30/2019 Critical Appraisal on 24.10.12
10/36
Qualitative interview (Method)
Face to face interviews were conducted.
Audio-recorded and the transcripts were subject to
an inductive thematic analysis.
By maximum difference sampling, a minimum of 15participants from the 54 people was aimed to be
recruited: sample include at least one consultant
who had completed the programme:
a) 36 months previously;b) 24 months previously;
c) 12 months previously;
d) currently acting consultant;
e) specialist trainee year 6 (ST6) and year 5 (ST5)
-
7/30/2019 Critical Appraisal on 24.10.12
11/36
Sample distribution for interview
-
7/30/2019 Critical Appraisal on 24.10.12
12/36
2. Qualitative methodology appropriate?
Consider:
If the research seeks to interpret or
illuminate the actions and/or subjectiveexperiences of research participants?
-
7/30/2019 Critical Appraisal on 24.10.12
13/36
3. Was the research design appropriateto address the aims of the research?
Consider:
If the researcher has justified the
research design?
(e.g. have they discussed how theydecided which method to use?)
-
7/30/2019 Critical Appraisal on 24.10.12
14/36
4. Was the recruitment strategyappropriate to the aims of the research?
Consider: If the researcher has explained how the
participants were selected? If they explained why the participants they
selected were the most appropriate toprovide access to the type of knowledgesought by the study?
If there are any discussions aroundrecruitment (e.g. why some people chosenot to take part)?
-
7/30/2019 Critical Appraisal on 24.10.12
15/36
5.Were the data collected in a way thataddressed the research issue?
Consider: If the setting for data collection was justified? If it is clear how data were collected (e.g. focus
group, semi-structured interview etc.)? If the researcher has justified the methods chosen? If the researcher has made the methods explicit
(e.g. for interview method, is there an indication ofhow interviews were conducted, or did they use a
topic guide)? If methods were modified during the study. If so,
has the researcher explained how and why? If the form of data is clear (e.g. tape recordings,
video material, notes etc.)?
If the researcher has discussed saturation of data?
-
7/30/2019 Critical Appraisal on 24.10.12
16/36
6. Relationship between researcher andparticipants been considered?
Consider: If the researcher critically examined their
own role, potential bias and influence
during: Formulation of the research questions Data collection, including sample recruitment
and choice of location
How the researcher responded to eventsduring the study and whether theyconsidered the implications of any changesin the research design?
-
7/30/2019 Critical Appraisal on 24.10.12
17/36
7. Ethical issues been considered?
Consider:
If there are sufficient details of how the researchwas explained to participants for the reader to
assess whether ethical standards were maintained If the researcher has discussed issues raised by the
study (e.g. issues around informed consent orconfidentiality or how they have handled the effectsof the study on the participants during and after thestudy)?
If approval has been sought from the ethicscommittee?
-
7/30/2019 Critical Appraisal on 24.10.12
18/36
Data analysis 1(first 5 interviews)
First five transcripts were coded by anauthor and sorted into initial themes.
The transcripts were then passed on to theresearch team for feedback. The teaminvolved different disciplinary including: two consultant psychiatrists,
a clinical psychologist,
a speech therapist and
a social worker
Modifications were made to the 2nd interview
schedule on the basis of the feedbacks
-
7/30/2019 Critical Appraisal on 24.10.12
19/36
The next five transcripts along with initialcategories and confirmatory or dis-confirmatory evidence for initial themeswere circulated to the team beforeembarking on the final five interviews.
At the end of the analysis the themes wereorganised into four major categories throughconsensus between all researchers.
Data analysis 2(2nd & 3rd 5 interviews)
-
7/30/2019 Critical Appraisal on 24.10.12
20/36
8. Data analysis sufficiently rigorous?
Consider: If there is an in-depth description of the analysis
process If thematic analysis is used, is it clear how the
categories/themes were derived from the data? Whether the researcher explains how the data
presented were selected from the original sample todemonstrate the analysis process
If sufficient data are presented to support thefindings
To what extent contradictory data are taken intoaccount
Whether the researcher critically examined theirown role, potential bias and influence duringanalysis and selection of data for presentation
-
7/30/2019 Critical Appraisal on 24.10.12
21/36
Results (Questionnaire survey)
The response rate was 21 responses
from a possible 41 respondents (51%);
67% were higher specialist traineesand 33% were consultant psychiatrists.
All questions had a 100% response
rate.
-
7/30/2019 Critical Appraisal on 24.10.12
22/36
Table 1: Survey results 1
-
7/30/2019 Critical Appraisal on 24.10.12
23/36
Table 2: Survey results 2
The trainees rated questions 5, 7, 13 & 15
higher than the consultants
-
7/30/2019 Critical Appraisal on 24.10.12
24/36
Results (Interview)
Thematic analysis organised themes into
four major categories of:
i. a lack of engagement with managementduring training
ii. the lasting impact of the leadership
development programme
iii. understanding the larger organisationalcontext
iv. making the transition to consultant
psychiatrist
-
7/30/2019 Critical Appraisal on 24.10.12
25/36
A lack of engagement with
management during training
The early years of training were described as being dominated bythe need to pass exams and build clinical skills.
Until you pass your exams management feels a million miles away.(trainee)
The trainees felt that the programme should have been introducedearlier, whereas the consultants felt it would have beenmore beneficial during their first year in post.
Until you actually take a post and you sit down there, you dont reallyknow whats going to come. I wish I could do the action learningagain now. The sessions would have a real practical applicationnow. (consultant)
The need for more hands-on practical management experiencewas a constant theme. Opportunities for this seemed todepend on individual placements and supervisors.
If the person mentoring you is quite influential you will get a project whereyou will be able to develop and flourish in many ways. (acting
consultant)
-
7/30/2019 Critical Appraisal on 24.10.12
26/36
A lack of engagement with
management during training
Shadowing a manager was useful but only to a point. Those whohad been given the opportunity to locum before taking aconsultant post said that it had really helped and suggestedbuilding this into the final stages of training, along with formalsupervision on management.
Because as an SpR you know all about the clinical issues, staffing issues.But once you take a locum post, then you know about all thepressures, about management issues. You have to balanceeverything. (consultant)
The Medical Leadership Competency Framework and MedicalLeadership Curriculum too prescriptive and based too much
on competencies. The reduction in special interest sessionsfrom four a week to two in which to achieve a number ofcompetencies was viewed as difficult.
We need all these things in our portfolio, but its not explicit how weshould go about it. There is a sense that people are just ticking offactivities. The role of supervision on placements should be aboutlearning something you can apply on a wider basis. (trainee)
-
7/30/2019 Critical Appraisal on 24.10.12
27/36
Lasting impact of the leadership
development programme
There were many examples of action learning being incorporatedinto everyday practice.
Much less of a traditional model here where the consultant has all theanswers. We share problems and solutions. Shared responsibilityof the decision-making. It consolidates team support. (consultant)
But this was not always possible because of the expectationsplaced on consultants from junior staff.
When I go to the ward, that sense of hierarchy is much more obvious.People can be quite deferential to you which I find uncomfortable. Ifits the wrong decision you make, you can be left feeling quiteexposed. If you have debate and other people have an opinion it
feels a lot safer. (consultant)
Action learning was viewed positively by the majority ofparticipants.
It was a good exercise [action learning sets]. They are the type of skillsyou need as a manager. Being diplomatic and putting things across
in a way that helps you find solutions, instead of just complainingall the time. (trainee)
-
7/30/2019 Critical Appraisal on 24.10.12
28/36
Lasting impact of the leadership
development programme
The programme had inspired many trainees to pursuehigher degrees in health and social caremanagement.
I was inspired to take this MBA. My special interest has beenmanagement. The MBA supplements the introductionprovided by the action learning. (trainee)
Being a future leader of an NHS organisation was notfor everyone. It was felt that some people wereperhaps not seen as natural leaders.
Im never going to be clinical director material. I still get frustratedwith juggling targets and clinical need. I prefer to be. Imokay with the operational type level of management. I couldnever be involved at a more strategic level. (consultant)
-
7/30/2019 Critical Appraisal on 24.10.12
29/36
Understanding the larger
organisational context
Sessions involving people in higher management fromwithin the local trust were particularly valued. Thismade the people in higher management moreaccessible to the trainees and gave them
personalities. It also challenged manyassumptions that many trainees held.Id previously been much more quick to jump to conclusions about
higher management without really having all the facts. I wasfinding out that managers were a lot more clued in to whatwas going on. Quelled some false assumptions.(consultant)
Respondents were also very impressed with externalnon-clinical managers.
The saddest things for doctors, they just talk too much to otherdoctors. What really helped was people from all differentwalks of life. The issues to do with managing were common
to all professionals, not just psychiatry. (trainee)
-
7/30/2019 Critical Appraisal on 24.10.12
30/36
Making the transition to consultant
psychiatrist
Becoming a consultant generated widespread anxiety, but this tended toreduce over time.
That was a big shock. Peoples expectations and the power you have. People lookto you for everything. But we can all do it and thats how we survive. After 3years you just know where you stand. (consultant)
Being aware that there is a limitation on how much you can prepare forbeing a consultant and being open to continual learning on the jobseemed to create a degree of resilience.
Theres a limitation on how much you can prepare for it. Youre never fullyprepared for the challenges you will face.You need different skills atdifferent stages. Being a manager and leader takes many years to learn.(consultant)
One of the most difficult aspects of the transition for all the consultantswas going from being supervised to supervising junior doctors.Most felt totally unprepared for this and they found it one of theirgreatest challenges.
Youre not responsible for a trainees behaviour. They are. But thats been difficultfor me. Feeling like I have to apologise for things. Like its a reflection on
me. Its an on-going steep learning curve. (consultant)
-
7/30/2019 Critical Appraisal on 24.10.12
31/36
9. Is there a clear statement of findings?
Consider:
If the findings are explicit
If there is adequate discussion of theevidence both for and against theresearchers arguments
If the researcher has discussed thecredibility of their findings (e.g. triangulation,respondent validation, more than oneanalyst)
If the findings are discussed in relation tothe original research question?
-
7/30/2019 Critical Appraisal on 24.10.12
32/36
Conclusion
The findings suggest that leadership developmentprogrammes can be successful in raisingawareness and increasing engagement in medical
leadership and management.
The findings from the qualitative interviews supportthe view that the trainees have changed and thatthe nonclinical management role of the consultant is
becoming more widely embraced.
Many were still finding it difficult to gain hands-onmanagement experience.
-
7/30/2019 Critical Appraisal on 24.10.12
33/36
10. How valuable is the research?
Consider:
If the researcher discusses the contributionthe study makes to existing knowledge or
understanding e.g. do they consider thefindings in relation to current practice orpolicy, or relevant research-basedliterature?
If they identify new areas where research is
necessary? If the researchers have discussed whether
or how the findings can be transferred toother populations or considered other ways
the research may be used?
-
7/30/2019 Critical Appraisal on 24.10.12
34/36
References for critical appraisal
Critically appraising qualitative researchhttp://www.bmj.com/highwire/section-pdf/9027/11/1
Critical Appraisal Skills Programme
http://www.casp-uk.net/ EBM Tools for Critical Appraisal
http://www.cebm.net/index.aspx?o=1157
Cochrane Handbook for Systematic Reviews of
Interventions http://www.cochrane-handbook.org./ Clinical Study Design and Methods Terminology
http://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htm
http://www.bmj.com/highwire/section-pdf/9027/11/1http://www.casp-uk.net/http://www.cebm.net/index.aspx?o=1157http://www.cochrane-handbook.org./http://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.cochrane-handbook.org./http://www.cochrane-handbook.org./http://www.cochrane-handbook.org./http://www.cebm.net/index.aspx?o=1157http://www.casp-uk.net/http://www.casp-uk.net/http://www.casp-uk.net/http://www.bmj.com/highwire/section-pdf/9027/11/1http://www.bmj.com/highwire/section-pdf/9027/11/1http://www.bmj.com/highwire/section-pdf/9027/11/1 -
7/30/2019 Critical Appraisal on 24.10.12
35/36
Reference for Leadership & DH docs
Health and Social Care Bill 2011http://www.publications.parliament.uk/pa/cm201011/cmbills/132/11132.pdf
High Quality Care for All: NHS Next Stage Review
http://www.official-documents.gov.uk/document/cm74/7432/7432.pdf
New Ways of Working for Everyonehttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdf
Medical Leadership Competency Frameworkhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Com
petency%20Framework%203rd%20ed.pdf Leadership and management for all doctors http://www.gmc-
uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.
pdf
DIY Handbook for Action Learners
http://www.actionlearningassociates.co.uk/resources/diyhandbook.php#
http://www.publications.parliament.uk/pa/cm201011/cmbills/132/11132.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.actionlearningassociates.co.uk/resources/diyhandbook.phphttp://www.actionlearningassociates.co.uk/resources/diyhandbook.phphttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.publications.parliament.uk/pa/cm201011/cmbills/132/11132.pdf -
7/30/2019 Critical Appraisal on 24.10.12
36/36
Thank you!