critical appraisal on 24.10.12

7/30/2019 Critical Appraisal on 24.10.12

1/36

- Critical Appraisal for Qualitative Article -

Wednesday 24th Oct 2012

Akira NAITO MD PhD

CT2 to Dr Kezia Lange

Mixed-methods evaluation of

a leadership development programme

for higher specialist trainees


2/36

Background

Newly appointed consultant

psychiatrists have reported that

management and leadership is anarea for which they are unprepared.


3/36

Background 2

Hewson et al and Spurgeon & Vassilas

introduced management courses.

Feedback suggested that these initiativeswere well received, but the samples in both

studies were made up of newly appointed

consultants only

As the evaluations took place straight after

training the authors were unable to examine

any lasting impact.


4/36

The Mersey Care NHS Trust specialist

registrar leadership development programme

This aimed to raise awareness and to

increase engagement in medical leadership

and management.

This consisted of bi-monthly meetings for

trainees in their final 2 years of training.

Each meeting was facilitated by two of the

authors and included a guest speaker whowould describe their area of work.

A more structured action learning set would

take place to take forward ideas.


5/36

Key principles of action learning

a. learning starts from not knowing;

b. People, who take responsibility in a

situation, have the best chance of takingactions that will make a difference;

c. learning involves both what is taught and

read and a questioning insight; and

d. learning should be greater than the rate of

change


6/36

Study aim

Study aim was to evaluate the impact

of a leadership development

programme based on the principles ofaction learning for higher trainees.


7/36

1. Was there a clear statement of theaims of the research?

Consider:

What was the goal of the research?

Why is it important?

Its relevance?


8/36

Questionnaire survey

A questionnaire was drawn up to

examine the impact of the leadership

development programme based on thefindings of previous papers.

Sent electronically and by post to 54

trainees & consultants who hadattended the programme.


9/36

Qualitative interview

A semi-structured interview schedule was

drawn up on the basis of the results from

the questionnaire. The schedule consisted of 19 items

covering the domains of perspectives on:

a) Leadership development programme;

b) Management and leadership; and

c) Transition to consultant psychiatrist.


10/36

Qualitative interview (Method)

Face to face interviews were conducted.

Audio-recorded and the transcripts were subject to

an inductive thematic analysis.

By maximum difference sampling, a minimum of 15participants from the 54 people was aimed to be

recruited: sample include at least one consultant

who had completed the programme:

a) 36 months previously;b) 24 months previously;

c) 12 months previously;

d) currently acting consultant;

e) specialist trainee year 6 (ST6) and year 5 (ST5)


11/36

Sample distribution for interview


12/36

2. Qualitative methodology appropriate?

Consider:

If the research seeks to interpret or

illuminate the actions and/or subjectiveexperiences of research participants?


13/36

3. Was the research design appropriateto address the aims of the research?

Consider:

If the researcher has justified the

research design?

(e.g. have they discussed how theydecided which method to use?)


14/36

4. Was the recruitment strategyappropriate to the aims of the research?

Consider: If the researcher has explained how the

participants were selected? If they explained why the participants they

selected were the most appropriate toprovide access to the type of knowledgesought by the study?

If there are any discussions aroundrecruitment (e.g. why some people chosenot to take part)?


15/36

5.Were the data collected in a way thataddressed the research issue?

Consider: If the setting for data collection was justified? If it is clear how data were collected (e.g. focus

group, semi-structured interview etc.)? If the researcher has justified the methods chosen? If the researcher has made the methods explicit

(e.g. for interview method, is there an indication ofhow interviews were conducted, or did they use a

topic guide)? If methods were modified during the study. If so,

has the researcher explained how and why? If the form of data is clear (e.g. tape recordings,

video material, notes etc.)?

If the researcher has discussed saturation of data?


16/36

6. Relationship between researcher andparticipants been considered?

Consider: If the researcher critically examined their

own role, potential bias and influence

during: Formulation of the research questions Data collection, including sample recruitment

and choice of location

How the researcher responded to eventsduring the study and whether theyconsidered the implications of any changesin the research design?


17/36

7. Ethical issues been considered?

Consider:

If there are sufficient details of how the researchwas explained to participants for the reader to

assess whether ethical standards were maintained If the researcher has discussed issues raised by the

study (e.g. issues around informed consent orconfidentiality or how they have handled the effectsof the study on the participants during and after thestudy)?

If approval has been sought from the ethicscommittee?


18/36

Data analysis 1(first 5 interviews)

First five transcripts were coded by anauthor and sorted into initial themes.

The transcripts were then passed on to theresearch team for feedback. The teaminvolved different disciplinary including: two consultant psychiatrists,

a clinical psychologist,

a speech therapist and

a social worker

Modifications were made to the 2nd interview

schedule on the basis of the feedbacks


19/36

The next five transcripts along with initialcategories and confirmatory or dis-confirmatory evidence for initial themeswere circulated to the team beforeembarking on the final five interviews.

At the end of the analysis the themes wereorganised into four major categories throughconsensus between all researchers.

Data analysis 2(2nd & 3rd 5 interviews)


20/36

8. Data analysis sufficiently rigorous?

Consider: If there is an in-depth description of the analysis

process If thematic analysis is used, is it clear how the

categories/themes were derived from the data? Whether the researcher explains how the data

presented were selected from the original sample todemonstrate the analysis process

If sufficient data are presented to support thefindings

To what extent contradictory data are taken intoaccount

Whether the researcher critically examined theirown role, potential bias and influence duringanalysis and selection of data for presentation


21/36

Results (Questionnaire survey)

The response rate was 21 responses

from a possible 41 respondents (51%);

67% were higher specialist traineesand 33% were consultant psychiatrists.

All questions had a 100% response

rate.


22/36

Table 1: Survey results 1


23/36

Table 2: Survey results 2

The trainees rated questions 5, 7, 13 & 15

higher than the consultants


24/36

Results (Interview)

Thematic analysis organised themes into

four major categories of:

i. a lack of engagement with managementduring training

ii. the lasting impact of the leadership

development programme

iii. understanding the larger organisationalcontext

iv. making the transition to consultant

psychiatrist


25/36

A lack of engagement with

management during training

The early years of training were described as being dominated bythe need to pass exams and build clinical skills.

Until you pass your exams management feels a million miles away.(trainee)

The trainees felt that the programme should have been introducedearlier, whereas the consultants felt it would have beenmore beneficial during their first year in post.

Until you actually take a post and you sit down there, you dont reallyknow whats going to come. I wish I could do the action learningagain now. The sessions would have a real practical applicationnow. (consultant)

The need for more hands-on practical management experiencewas a constant theme. Opportunities for this seemed todepend on individual placements and supervisors.

If the person mentoring you is quite influential you will get a project whereyou will be able to develop and flourish in many ways. (acting

consultant)


26/36

A lack of engagement with

management during training

Shadowing a manager was useful but only to a point. Those whohad been given the opportunity to locum before taking aconsultant post said that it had really helped and suggestedbuilding this into the final stages of training, along with formalsupervision on management.

Because as an SpR you know all about the clinical issues, staffing issues.But once you take a locum post, then you know about all thepressures, about management issues. You have to balanceeverything. (consultant)

The Medical Leadership Competency Framework and MedicalLeadership Curriculum too prescriptive and based too much

on competencies. The reduction in special interest sessionsfrom four a week to two in which to achieve a number ofcompetencies was viewed as difficult.

We need all these things in our portfolio, but its not explicit how weshould go about it. There is a sense that people are just ticking offactivities. The role of supervision on placements should be aboutlearning something you can apply on a wider basis. (trainee)


27/36

Lasting impact of the leadership


There were many examples of action learning being incorporatedinto everyday practice.

Much less of a traditional model here where the consultant has all theanswers. We share problems and solutions. Shared responsibilityof the decision-making. It consolidates team support. (consultant)

But this was not always possible because of the expectationsplaced on consultants from junior staff.

When I go to the ward, that sense of hierarchy is much more obvious.People can be quite deferential to you which I find uncomfortable. Ifits the wrong decision you make, you can be left feeling quiteexposed. If you have debate and other people have an opinion it

feels a lot safer. (consultant)

Action learning was viewed positively by the majority ofparticipants.

It was a good exercise [action learning sets]. They are the type of skillsyou need as a manager. Being diplomatic and putting things across

in a way that helps you find solutions, instead of just complainingall the time. (trainee)


28/36

Lasting impact of the leadership


The programme had inspired many trainees to pursuehigher degrees in health and social caremanagement.

I was inspired to take this MBA. My special interest has beenmanagement. The MBA supplements the introductionprovided by the action learning. (trainee)

Being a future leader of an NHS organisation was notfor everyone. It was felt that some people wereperhaps not seen as natural leaders.

Im never going to be clinical director material. I still get frustratedwith juggling targets and clinical need. I prefer to be. Imokay with the operational type level of management. I couldnever be involved at a more strategic level. (consultant)


29/36

Understanding the larger

organisational context

Sessions involving people in higher management fromwithin the local trust were particularly valued. Thismade the people in higher management moreaccessible to the trainees and gave them

personalities. It also challenged manyassumptions that many trainees held.Id previously been much more quick to jump to conclusions about

higher management without really having all the facts. I wasfinding out that managers were a lot more clued in to whatwas going on. Quelled some false assumptions.(consultant)

Respondents were also very impressed with externalnon-clinical managers.

The saddest things for doctors, they just talk too much to otherdoctors. What really helped was people from all differentwalks of life. The issues to do with managing were common

to all professionals, not just psychiatry. (trainee)


30/36

Making the transition to consultant

psychiatrist

Becoming a consultant generated widespread anxiety, but this tended toreduce over time.

That was a big shock. Peoples expectations and the power you have. People lookto you for everything. But we can all do it and thats how we survive. After 3years you just know where you stand. (consultant)

Being aware that there is a limitation on how much you can prepare forbeing a consultant and being open to continual learning on the jobseemed to create a degree of resilience.

Theres a limitation on how much you can prepare for it. Youre never fullyprepared for the challenges you will face.You need different skills atdifferent stages. Being a manager and leader takes many years to learn.(consultant)

One of the most difficult aspects of the transition for all the consultantswas going from being supervised to supervising junior doctors.Most felt totally unprepared for this and they found it one of theirgreatest challenges.

Youre not responsible for a trainees behaviour. They are. But thats been difficultfor me. Feeling like I have to apologise for things. Like its a reflection on

me. Its an on-going steep learning curve. (consultant)


31/36

9. Is there a clear statement of findings?

Consider:

If the findings are explicit

If there is adequate discussion of theevidence both for and against theresearchers arguments

If the researcher has discussed thecredibility of their findings (e.g. triangulation,respondent validation, more than oneanalyst)

If the findings are discussed in relation tothe original research question?


32/36

Conclusion

The findings suggest that leadership developmentprogrammes can be successful in raisingawareness and increasing engagement in medical

leadership and management.

The findings from the qualitative interviews supportthe view that the trainees have changed and thatthe nonclinical management role of the consultant is

becoming more widely embraced.

Many were still finding it difficult to gain hands-onmanagement experience.


33/36

10. How valuable is the research?

Consider:

If the researcher discusses the contributionthe study makes to existing knowledge or

understanding e.g. do they consider thefindings in relation to current practice orpolicy, or relevant research-basedliterature?

If they identify new areas where research is

necessary? If the researchers have discussed whether

or how the findings can be transferred toother populations or considered other ways

the research may be used?


34/36

References for critical appraisal

Critically appraising qualitative researchhttp://www.bmj.com/highwire/section-pdf/9027/11/1

Critical Appraisal Skills Programme

http://www.casp-uk.net/ EBM Tools for Critical Appraisal

http://www.cebm.net/index.aspx?o=1157

Cochrane Handbook for Systematic Reviews of

Interventions http://www.cochrane-handbook.org./ Clinical Study Design and Methods Terminology

http://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htm
http://www.bmj.com/highwire/section-pdf/9027/11/1http://www.casp-uk.net/http://www.cebm.net/index.aspx?o=1157http://www.cochrane-handbook.org./http://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.vetmed.wsu.edu/courses-jmgay/glossclinstudy.htmhttp://www.cochrane-handbook.org./http://www.cochrane-handbook.org./http://www.cochrane-handbook.org./http://www.cebm.net/index.aspx?o=1157http://www.casp-uk.net/http://www.casp-uk.net/http://www.casp-uk.net/http://www.bmj.com/highwire/section-pdf/9027/11/1http://www.bmj.com/highwire/section-pdf/9027/11/1http://www.bmj.com/highwire/section-pdf/9027/11/1


35/36

Reference for Leadership & DH docs

Health and Social Care Bill 2011http://www.publications.parliament.uk/pa/cm201011/cmbills/132/11132.pdf

High Quality Care for All: NHS Next Stage Review

http://www.official-documents.gov.uk/document/cm74/7432/7432.pdf

New Ways of Working for Everyonehttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdf

Medical Leadership Competency Frameworkhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Com

petency%20Framework%203rd%20ed.pdf Leadership and management for all doctors http://www.gmc-

uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.

pdf

DIY Handbook for Action Learners

http://www.actionlearningassociates.co.uk/resources/diyhandbook.php#
http://www.publications.parliament.uk/pa/cm201011/cmbills/132/11132.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.actionlearningassociates.co.uk/resources/diyhandbook.phphttp://www.actionlearningassociates.co.uk/resources/diyhandbook.phphttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.gmc-uk.org/Leadership_and_management_for_all_doctors_FINAL.pdf_47234529.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.nwpgmd.nhs.uk/sites/default/files/Medical%20Leadership%20Competency%20Framework%203rd%20ed.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.nmhdu.org.uk/silo/files/nww-implementation-guidance.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.official-documents.gov.uk/document/cm74/7432/7432.pdfhttp://www.publications.parliament.uk/pa/cm201011/cmbills/132/11132.pdf


36/36

Thank you!

critical appraisal on 24.10.12

Documents