BEME Systematic Review Protocol Checklist

Please note that in order to help in forming an opinion on the review, all reviewer comments sections must be completed.

TITLE

LEAD AUTHOR

NAME:

INSTITUTION:

RESPONSIBLE BICC

BICC:

HEAD OF BICC:

REVIEWER

NAME:

INSTITUTION:

EMAIL:

1

Collaboration

INFORMATION ON TITLE, LEAD AUTHOR AND AUTHORS

Please rate each item using Y (yes), N (no) or * if neither Y or N apply (for example partially). If the code is N or * please justify in the comments. Please refer to specific page / line # where appropriate.

1. COVER SHEET

YES/NO/*1.1 Title is clear and understandable for the non-expert.1.2 Title reflects the review topic and aim(s) (i.e. the research question(s)), mentions ‘BEME’

review and type of review (e.g., systematic, realist, scoping, etc.)1.3 An abstract is provided. 1.4 The structure of the team is appropriate for the review.REVIEWERS COMMENTS [Required Field]

METHODOLOGICAL STEPS

2. BACKGROUND TO THE TOPIC

YES/NO/*2.1 Background should address issues that are particularly important for teachers, students or

other health professional education stakeholders (including policy makers). Background should state the ‘problem’ the review plans to address, and the ‘gap’ in the literature the review is trying to fill.

2..2

The rational ( i.e. the ‘hook’ or ‘so what’ of the review) is clearly articulated – how will the findings of the review explicitly impact practice or research?

2.3 Background describes the educational issue under scrutiny, including any conceptual/theoretical framing related to the research topic.

2.4 Definitions of ‘specific terms’ (including the word effectiveness) used by the authors is provided, as well as any well understood terms that have multiple meanings (high fidelity, simulation, technical skills, etc.)

2.5 The review is situated in the current literature and any systematic reviews already published on the same topic are described. Justification should be given for any duplication that arises.

2.6 Background is clear, concise (yet informative) and easy to understand for all, including for non-experts and international colleagues.

REVIEWERS COMMENTS [Required Field]

3. REVIEW QUESTION(S)/OBJECTIVES, TYPE OF REVIEW AND KEYWORDS

YES/NO/*3.1

The review question(s) are well formulated with specified key components using PICO (Population or participants, Intervention, issue or activity under investigation, Comparator (if applicable), Outcomes (e.g., Kirkpatrick’s)); SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type; or CAPS (Current state of knowledge, Area of interest, Potential impact for education, Suggestions from experts in the field.) (Sharma et al. 2015)

3. The review questions address local, national or international needs linked to the background.

2

23.3

The objective(s) are meaningful to relevant stakeholders.

3.4

The objective(s) are clear, realistic, feasible and focused.

3.5

Information on the type of review (e.g., systematic, realist, scoping), as well as rationale for selecting that type of review is provided. If a well-recognised tradition of review is used the questions and subsequent methods must match this review type following published guidance (e.g., Wong et al. 2013 and Wong et al. 2016 (RAMESES I and II); Arksey and O’Malley 2015, or Levac et al. 2010)

3.6

Appropriate key words are identified.

REVIEWERS COMMENTS [Required Field]

4. STUDY SELECTION CRITERIA

YES/NO/*4.1 Appropriate (pertinent and justified) inclusion/exclusion criteria to select articles are outlined.

Types of studies to be included are articulated (Note: Editorials, perspectives and opinion pieces are often excluded unless there is justification for including. Reviews are typically used to identify related articles, but not typically included.)

Types of education or assessment to be included are clearly described. Study participants are adequately described. Any language or geographic restrictions are articulated.

4.2

The reasons for any restrictions (e.g., only undergraduate learners) are appropriate and clearly described.

4.3

The inclusion / exclusion criteria are appropriate to the review questions under evaluation.

REVIEWERS COMMENTS [Required Field]

5. SEARCH SOURCES AND STRATEGIES

YES/NO/*5.1 A pilot search was performed to help authors refine the research question(s) aim(s) and

objectives 5.2 Results of the pilot search are provided and support both the viability* and feasibility** of the

review. A pilot search should be used to determine if a reasonable number of papers are likely to be identified after application of inclusion / exclusion criteria to the full data set. Authors should apply the full inclusion / exclusion criteria to the first 50 -100 citations to obtain an estimate of the likely number of papers in the review.

5.3 At least 3 papers that meet the full inclusion criteria should be cited to help reviewers understand other elements of the protocol.

5.4 The main search strategy envisaged is exhaustive and rigorous regarding the process of localising and selecting the evidence to ensure the objectives of the review are met.

5.5

The protocol identifies the full range of sources to be consulted (including databases and specialist registers to be accessed, journals to be hand-searched, the grey literature to be accessed and methods that will be used to contact experts in the field).

3

5.6

The protocol indicates the time limit for the search and this appears realistic and is appropriately justified.

REVIEWERS COMMENTS - INCLUDING SPECIFICS ON PILOT SEARCH [Required Field]

* To be viable, enough papers need to be identified to justify a review. Final manuscripts with < 10 included papers (or <20 included papers for Scoping reviews) are unlikely to be accepted.** To be feasible, the number of included papers identified needs to be manageable by the review team. Reviews with several hundred papers can be challenging to complete in a timely manner, depending on the construction of the review team.

6. SCREENING ARTICLES AND EXTRACTING DATA

YES/NO/*6.1 To assure a rigorous and systematic process, title, abstract and full text screening will be

performed by at least 2 coders. 6.2 A procedure for establishing inter-rater reliability during screening is described (e.g., Cohen’s

kappa).6.3 All data extraction will be performed by at least two coders, or specific justification will be

provided as to why this will not be achieved. 6.4 Consistency between coders will be assessed during data extraction (describe how and when).6.5

An established procedure to resolve differences between coders and reach consensus is stated.

6.6

Description is made of relevant methodological data to be extracted and presented methods, participants, outcomes, conclusions.

REVIEWERS COMMENTS [Required Field]

7. QUALITY APPRAISAL OF STUDIES

YES/NO/*7.1 Description is given on how the quality of primary study design will be assessed. Risk of bias in

study methodology should be assessed, similar to the Cochrane tradition. 7.2 Description is provided on how the quality of reporting on the educational intervention or

assessment will be assessed. Risk of bias in reporting should be assessed as adequate descriptions of theoretical underpinning, resources, setting, education and context promote replication in education.

7.2. If quality assessment of both study design and reporting are not undertaken, the reasons for not doing so are justified (e.g., scoping reviews that aim to map a wide body of literature may not undertake a quality assessment.)

7.3 In case of disagreement concerning quality assessments, it is clear how consensus will be reached between the two coders.

REVIEWERS COMMENTS [Required Field]

4

8. SYNTHESIS OF EVIDENCE AND TRANSFER TO RESEARCH AND PRACTICE

YES/NO/*8.1 The method(s) to synthesise evidence from quantitative data (if appropriate) are described. 8.2 The method(s) to synthesise evidence from qualitative data (if appropriate) are described 8.3 If another research tradition has been used, such as realist or scoping, an appropriate and

recognised framework is cited and employed (e.g., Wong et al. 2013 and Wong et al. 2016 (RAMESES I and II); Arksey and O’Malley 2015, or Levac et al. 2010)

8.4 The potential expected outcomes and implications for education practice, research and / or policy are envisaged.

8.5 Information on how the review will contribute to the field is provided. Please note that this is an essential requirement of the Protocol.

REVIEWERS COMMENTS [Required Field]

OVERALL ASSESSMENT

9. OVERALL ASSESSMENT

YES/NO/*9.1 The focus of this review is relevant for educational practice

9.2 The important research questions have been considered. Please indicate if research questions are missing

9.3 Protocol is clearly written (overall style of writing is easy to understand and interesting) 9.4 Protocol is globally well structured 9.5 Protocol reveals the coherence of all process (title , aims, questions, objectives, methods,

discussion/conclusions and practice points) REVIEWERS COMMENTS [Required Field]

RECOMMENDATION

10. RECOMMENDATION / FINAL DECISION (Please tick just one box)

YES/NO/*10.1 The protocol is acceptable for submission in its present form10.2 The protocol must undergo minor amendments/changes10.3 The protocol must undergo major amendments/changes10.4 The protocol is not acceptable for submissionREVIEWERS COMMENTS - this is vital justification to support the BRC editor in making their decision and must be completed [Required Field]

5

References:Arksey H, O’Malley L. 2005. Scoping studies: towards a methodological framework. Int J Soc Res Method. 8(1):19-32. https://doi.org/10.1080/1364557032000119616Levac D, Colquhoun H, O’Brien KK. 2010. Scoping studies: advancing the methodology. Implement Sci. 5(69):1-9. https://doi.org/10.1186/1748-5908-5-69

Sharma R, Gordon M, Dharamsi S, Gibbs T. 2015. Systematic reviews in medical education: a practical approach: AMEE guide 94. Med Teach. 37(2):108-24. https://doi.org/10.3109/0142159X.2014.970996

Wong G, Greenhalgh T, Westhorpe G, Buckingham J, Pawson R. 2013. RAMESES publication standards: realist synthesis. BMC Medicine 11(21). https://doi.org/10.1111/jan.12095

Wong G, Westhorpe G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. 2016. RAMESES II reporting standards for realist evaluations. BMC Medicine 14(96). https://doi.org/10.1186/s12916-016-0643-1

6