Peer review process

Trottie KirwanChair APA Peer Review

Committee

SPAN Meeting 24 April 2009

Original process

Original Peer Review Process• APA has run interdepartmental peer review since 1999• Started in the big specialist children’s hospitals• Aim

– raise standards of clinical practice within a paediatric anaesthetic department. Allow for experiences to be shared, with the dissemination of good or innovative practice (Review document)

– to assess departments in relation to standards which should characterise anaesthetic departments throughout the country (Peter Crean in Anaesthesia 2003)

• Based on Good Practice – a guide for departments of anaesthesia. RCA and AAGBI. London, 1998

• 2 day visit by a team of four including a lay member• All departments, visitors and visited, found it valuable

BUT• The process is very demanding of both the

visited department and the visitors– 2 day visit including interviews with 16 different staff

members as well as parents– Detailed report averaging 30 pages

• APA decided to extend the process to be available to all anaesthetic departments with paediatric practice

• Original process too great a commitment for smaller departments with fewer paediatric anaesthetists, particularly as visitors to write the reports

New Process

Simplified Peer Review Scheme

• New scheme developed by Peer Review Committee– Alistair Cranston, Peter Crean, Trottie Kirwan, Madeleine

Wang

• Intended as a tool for any department treating children to– Check compliance with existing criteria of RCA, NSF,

HCC/CQC– Demonstrate that standards of practice are good– Help in discussions with management and colleagues

• Simplify the process for departments with a smaller paediatric workload and fewer paediatric anaesthetists

• Reduce the demands on the reviewing team

Simplified Peer Review Scheme

• Provide a structure for local self-appraisal to help departments in raising standards of clinical practice. – Self assessment– Peer verification

• Based on – Good Practice Guide 2006 (JCGP RCA & AA), – RCA guidelines for the provision of anaesthetic

services 2004– NSF for Children Standard for Hospital Services 2003– Criteria in the proposal come from RCA guidelines

Self-assessment

Self-assessment

• Departments self check against criteria in a freely available template

– Department’s perceived strengths and weaknesses compiled by the paediatric anaesthetists

– Check-list of criteria for good practice (from RCA guidelines)

– 360° appraisal by colleagues and patients/families

• Tool to monitor their own practice, without any external input

Self-assessment Checklist(from RCA guidelines)

• Organisation• Theatres and day surgery

facilities• Recovery • Equipment • Emergency department• Support services• Critical care• Transfer • Staff• Training • Clinical governance• Child and family experience

Self-assessment ChecklistRCoA guidelines

Equipment

2.1 A full range of monitoring devices, paediatric anaesthetic equipment and disposable items for general and regional anaesthesia should be available in theatres and all other areas where children are anaesthetised. This should include a full range of disposable equipment including the following which should be appropriate for use in children of all sizes and ages:blood pressure cuffsintravenous cannulaetemperature probespulse oximetry.

2.2 Resuscitation drugs and equipment, including an appropriate defibrillator, should be routinely available at all sites where children are to be anaesthetised.

2.3 Anaesthetic machines should incorporate ventilators, which have controls and bellows permitting their use over the entire age range together with the facility to provide pressure controlled ventilation.

2.4 There should be appropriate thermostatic control of the operating room; temperature monitoring and patient warming devices should be available in both the operating room and recovery area.

2.5 Intravenous fluids should normally be administered only by volumetric infusion pumps.

Checklist

Wherever children are anaesthetised:

i. There is appropriate paediatric anaesthetic equipment

ii. Monitoring equipment meets the current standards of the Association of Anaesthetists

iii. Ventilators are suitable for children and can deliver pressure controlled mode

iv. There are resuscitation drugs and equipment

v. There are suitable temperature monitoring, and warming devices

There are volumetric infusion pumps for IV fluid administration

Strengths and Weaknesses

• The paediatric anaesthetists, separately and collectively, list of those aspects of their work which they perceive as good and those they think need improving – The aspects which need improving are considered, any

which can be fixed from within the department are addressed

– The process is repeated until eventually the list consists only of strengths, and of weaknesses over which the group has no control

• Found to be a useful tool for discussing problems people have got used to but could actually deal with

360° Appraisal by Colleagues• Colleagues are invited to help in the department’s self-

review – Chief Executive – Medical Director– Head of Department of Anaesthesia, anaesthetic trainees – Consultant paediatricians, paediatric surgeons, ED, PICU &

transfer – Theatre Manager, Senior ODP / anaesthetic nurse– Directorate Business Manager, Anaesthetic Secretary– Senior paediatric ward nurses, play specialist, pain team

• Explanatory letter to say that this is a voluntary process

• Process focussed on the department’s organisation and practice – concerns about individuals should be reported through the normal channels

360° Appraisal by Colleagues• How the paediatric anaesthetic department is

perceived – Concerns with the management of the department or the

service

• Anaesthetists working as a team– Formal working relationships with colleagues– Informal collaboration for lists and individual patients

• Effective management of the service – Theatre time, emergency support – Cover for absent colleagues

• Staffing – Staff levels, recruitment and retention problems– System to deal sensitively with colleagues below an

acceptable level

• Effective planning for equipment, staffing and budget

Patient/Parent Appraisal• Areas covered

– Pre-admission – Pre-operative preparation– Anaesthetic room– Recovery– Analgesia

• Design of questionnaire– Open questions elicit more information– Closed questions give more consistency

• Timing– At discharge easy and cheap– After return home more informative

Peer review

Peer Review• Department completes self-assessment and

volunteers for peer verification• Visiting team

– 2 doctors and a lay visitor, one from outside the region – Lead reviewer with peer review experience– Could include an observer

• Self-assessment paperwork sent to the visiting team

• One day visit – Review the self-assessment evidence– meet paediatric anaesthetists – visit to clinical areas – debriefing session

• Review team send a report within a month

Peer Review

• External APA report – confirmation of good work– suggestions for consideration to improve organisation &

practice– recommendations for use in negotiating plans with

colleagues and management

• Problems arising at visit– visiting team cannot substantiate the self-assessment: this

will be stated at the debriefing meeting and no report sent – a potentially serious problem is unearthed during a review:

the review team leader will contact the regional co-ordinator and an APA peer review committee member immediately

Peer review process

Organisation of the Process• Peer review is organised on a regional basis

– within local paediatric anaesthetic networks where they exist

– regional co-ordinator arranges teams of reviewers – each team has an out-of-region member – national co-ordinator for hospitals not in a local

network

• Extending the process nationally includes– smaller departments of fewer paediatric anaesthetists – less experience of review visiting and report writing– more support is needed

• Standard job description documents– task lists for regional co-ordinators, review teams, and

departments being reviewed– criteria for eligibility for Peer Review Team

membership, anaesthetic and lay

Peer Review

APA peer review version2009

Regional co-ordinator

1. Compile list of anaesthetists to be local review team members. Criteria for eligibility: a. consultant anaesthetist b. member of APA c. experience of peer review d. for team leaders, experience of 2 reviews

2. Recruit new team members. Criteria: a. consultant anaesthetist b. member of APA

People who are interested in getting involved but have no experience of peer review should attend one review as an observer

3. List of local people to be lay visitors. Criteria for eligiblilty: a. Experience of hospital visiting, receiving hospital paediatric and/or anaesthetic care, or working with

children and understanding their needs b. Ability to be objective – should not work within the visited Trust, a neighbouring Trust, PCT or SHA c. Willing to contribute to the report writing as a team member – so not to be visiting in the role of a

report writer for another institution Suitable people might be found from voluntary groups listed with PALS, with former Public Patient Involvement Forum administrators or from the new networks of voluntary organisations known as Local Involvement Networks (LINKs). Local teachers or other workers with children who have experience with their own children in hospital have been enlisted.

4. Receive self assessment from departments wanting to be reviewed, and set up a reviewing team consisting of: a. a lead, who has at least experience as a reviewer, b. another consultant anaesthetist who has at least experience as an observer c. a lay member At least one team member should be from outside the region and there may also be observers from other regions and the APA peer review committee

5. The co-ordinator must ensure that the reviewing team members can write a fair report. Colleagues with a professional interest in a good or bad report cannot serve as review team members for that visit

6. Be aware of communication and arrangements between the department to be visited and the review team, and advise in case of problems

7. Receive the report from the review team, and the feedback on the process from the visited department and the review team, and chase them up if they are not done in 6 weeks

8. Forward the report and feedback to the APA committee chair and discuss any problems in either the report or the feedback

Peer Review

Guidance for departments being reviewed

1. The department needs to agree that peer review will be useful, and carry out the self-evaluation, using it as a tool for self-improvement (unless you are perfect already)

2. Carry out the SWOT process, getting all the paediatric anaesthetists to agree to a consensus document

3. Once the self-evaluation is completed and you are ready to be reviewed, contact the local paediatric anaesthesia peer review co-ordinator, where there is one, or peer review committee chair.

4. Make contact with the review team leader

5. Inform relevant individuals, wards and departments in the Trust, emphasising that this is a voluntary process undertaken by the anaesthetic department to recognise and improve the quality of the service, and not because of a complaint.

6. Collate the short portfolio, self evaluation documents and supporting evidence paperwork and send a copy to each member of the reviewing team, as much as possible in electronic format, 4 weeks before the visit to send to the visiting team:

Paperwork to send to the review team members

• The short portfolio

• SWOT consensus document

• The results of the colleague appraisals

• The self-assessment check-list

Check-list criteria evidence which should be sent:

Reports of incident reviews

Department policies and guidelines

Summaries of completed audits from the last 2 years

Minutes of meetings which contain relevant information

Copies of patient information

Evidence of any research – facilities, support (ethics), trainee input

Critical incident reports and presentations to M&M meetings

List of any complaints and the management of them

Peer Review

Organisation of the Process

• Confidentiality– Reports, and the information they are based on, are

confidential and will not be passed to anyone by the visiting team members

– The visited department may use the report as they choose

• Quality control– individual performance monitoring by formal feedback

after each review by both the visitors and visited– consistency monitoring by an external visitor

• Support for the process – peer review committee– e-network on APA website – meetings at APA ASM and linkmen meetings

Feedback Self-assessment Surprisingly useful. We all thought that we knew the problems in

our hospital but when we discussed our thoughts we found some important concerns that I had not considered before. Additionally it encouraged us to have a detailed meeting at which I could disseminate useful information that others paed anaesthetists had been unaware.

Arranging the review Time consuming. Very difficult to arrange reviewers. People need a lot of notice.

Clear requirements for evidence

I may not have read the documentation closely enough but I missed the requirement for evidence. Would have been helpful to give a long list of possible sources.

Visit discussions useful and professional

Visit discussion was very useful, there were no great surprises

Report received promptly, and contents as expected from visit discussions

Not received yet ( only 2 weeks ago)

Process helpful and fair Process very helpful. Stimulated discussion and action within the hospital before the review and added political muscle to help implement long overdue change in some areas.

Other comments Time consuming but completely worthwhile process. A continuous cycle of review of the process will help to make it less time consuming and more rewarding

Process development

• Peer review intended to be a collaboration with users – Proposals on APA website – colleagues invited to feed back – SWACA pilot

• Changes to self-assessment – drop case studies – flexibility in training and experience criteria

• Changes to process – better guidance on how to do it– standard templates for questionnaires, reports, feedback– addition of out-of-region review team member

• Good ideas – children included in reviewing team

Benefits

• Visited department– Incentive to look critically at local practice and make

changes before writing self-assessment– Team building within the department and with other

colleagues– Reports have been useful in getting organisational

change and improvements in facilities and equipment• Visitors

– Good ideas to manage common problems: meeting life support criteria, managing scheduling of lists

• Region – Clinical links reinforced– Recognition that our colleagues in other hospitals are

good doctors and friendly humans

Misgivings

• Conspiracy– HCC approach – choose inappropriate criteria and

stop good departments doing good work– Predator approach – local big hospital stealing work– Forced marriage approach – departments merge and

one gets the cream

• Cock-up– Huge amount of work for no outcome– Report which will be ignored– Review will have no power to compel change

• May be overcome if some hospitals make a start while others wait to see how it turns out

SPAN

• Some interest• Some cynicism

• Risk/benefit balance in favour of giving peer review a try

• Perhaps start with the psychopathically keen

References

• Draft proposal for revised peer review scheme http://www.apagbi.org.uk/docs/APA_PeerReview(Draft).pdf

• Good Practice: A Guide for Departments of Anaesthesia. The Royal College of Anaesthetists and Association of Anaesthetists of Great Britain and Ireland, 3rd edition 2006 http://www.rcoa.ac.uk/docs/goodpractice(oct2006).pdf

• Department of Health (2003) Getting the right start: National Service Framework for Children - A Standard for Hospital Services http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_4006182

• Guidelines for the Provision of Anaesthetic Services. Royal College of Anaesthetists. 2004 http://www.rcoa.ac.uk/docs/GPAS.pdf

• Quality in paediatric anaesthesia: a pilot study of interdepartmental peer review. Crean PM, Stokes MA, Williamson C, Hatch DJ Anaesthesia. 2003 Jun;58(6):543-8.