advancing the who-integrate framework as a tool for
TRANSCRIPT
Advancing the WHO-INTEGRATE Framework as a Tool for Evidence-Informed, Deliberative Decision-Making Processes: Exploring the Views of Developers and Users of WHO GuidelinesJan M. Stratil1,2* ID , Deepak Paudel3 ID , Karen E. Setty4 ID , Carlos E. Menezes de Rezende5,6 ID , Aline A. Monroe6 ID , Jimmy Osuret7 ID , Inger B. Scheel8 ID , Manfred Wildner9,2 ID , Eva A. Rehfuess1,2 ID
AbstractBackground: Decision-making on matters of public health and health policy is a deeply value-laden process. The World Health Organization (WHO)-INTEGRATE framework was proposed as a new evidence-to-decision (EtD) framework to support guideline development from a complexity perspective, notably in relation to public health and health system interventions, and with a foundation in WHO norms and values. This study was conducted as part of the development of the framework to assess its comprehensiveness and usefulness for public health and health policy decision-making.Methods: We conducted a qualitative study comprising nine key informant interviews (KIIs) with experts involved in WHO guideline development and four focus group discussions (FGDs) with a total of forty health decision-makers from Brazil, Germany, Nepal and Uganda. Transcripts were analyzed using MAXQDA12 and qualitative content analysis. Results: Most key informants and participants in the FGDs appreciated the framework for its relevance to real-world decision-making on four widely differing health topics. They praised its broad perspective and comprehensiveness with respect to new or expanded criteria, notably regarding societal implications, equity considerations, and acceptability. Some guideline developers questioned the value of the framework beyond current practice and were concerned with the complexity of applying such a broad range of criteria in guideline development processes. Participants made concrete suggestions for improving the wording and definitions of criteria as well as their grouping, for covering missing aspects, and for addressing overlap between criteria.Conclusion: The framework was well-received by health decision-makers as well as the developers of WHO guidelines and appears to capture all relevant considerations discussed in four distinct real-world decision processes that took place on four different continents. Guidance is needed on how to apply the framework in guideline processes that are both transparent and participatory. A set of suggestions for improvement provides a valuable starting point for advancing the framework towards version 2.0. Keywords: Decision-Making, Priority Setting, Resource Allocation, Guideline Development, World Health Organization, FrameworkCopyright: © 2020 The Author(s); Published by Kerman University of Medical Sciences. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.Citation: Stratil JM, Paudel D, Setty KE, et al. Advancing the WHO-INTEGRATE framework as a tool for evidence-informed, deliberative decision-making processes: exploring the views of developers and users of WHO guidelines. Int J Health Policy Manag. 2020;x(x):x–x. doi:10.34172/ijhpm.2020.193
*Correspondence to:Jan M. Stratil Email: [email protected]
Article History:Received: 30 March 2020Accepted: 29 September 2020ePublished: 27 October 2020
Original Article
Full list of authors’ affiliations is available at the end of the article.
http://ijhpm.comInt J Health Policy Manag 2020, x(x), 1–13 doi 10.34172/ijhpm.2020.193
BackgroundMaking evidence-informed decisions about public health and health system interventions and policies is complex.1-3 On the one hand, producing and assessing evidence eg, on the effectiveness of public health and health policy interventions is challenging due to the complexity of the interventions themselves (eg, the number of components, or the pathway leading to multiple outcomes).4 Furthermore, due to interactions with the system in which these are implemented (eg, system changes due to emergent properties, adaptivity, or feedback mechanisms) as well as due to the high context-
dependency of the effects of the intervention.2,5 On the other hand, simply producing more and stronger evidence eg, on the efficacy or cost-effectiveness of an intervention is in itself not sufficient to make better choices, as evidence-informed decision-making is a deeply value-laden process.6-8 Decision-makers must balance numerous and often conflicting normative and technical aspects for a decision-making process,9-11 which represents an additional source of complexity (eg, which criteria should be considered and how should these be weighed against each other)? This holds true for all forms of structured decision-making processes
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–132
Implications for policy makers• Public health and health policy processes are complex and deeply value-laden. This includes different types of decision-making processes at
national or sub-national levels as well as the development of guidelines at a global level. • The various affected stakeholder groups have their own reasons and principles guiding their decisions. Involving diverse stakeholders and taking
their views into account in a structured way can ensure transparency, legitimacy, and acceptability of the decision, and increase the likelihood of implementation. Evidence-to-Decision (EtD) frameworks, such as the World Health Organization (WHO)-INTEGRATE framework, can serve as helpful tools.
• The WHO-INTEGRATE framework could be a valuable tool to support decision-making processes and, with regard to WHO guideline development, could enhance the relevance and applicability of WHO recommendations in public health and health policy. Suggestions provided will help to further advance the framework and to develop concrete guidance on how to apply it in practice.
Implications for the publicWhen making public health or health policy decisions, decision-makers should consider the best available scientific evidence and other factors (eg, cost, feasibility, or acceptability). They should also ensure that members in the committee preparing or making decisions is sufficiently diverse and represents all relevant viewpoints. This applies to political decisions at national or subnational levels and to more technical processes, eg, development of guidelines. When not adequately or transparently considered, decisions may not lead to the desired impacts or may not be considered acceptable and legitimate. Decision frameworks, such as the World Health Organization (WHO)-INTEGRATE framework, can support decision-makers and help ensure that all factors of relevance are considered. We discussed this framework with developers of guidelines at the WHO and with groups of decision-makers across four continents. They reported that the factors (called criteria) included in this framework are both comprehensive and relevant to real world public health and health policy decisions. This suggests that the WHO-INTEGRATE framework can be a valuable tool for application from global to local levels.
Key Messages
in health, notably priority setting,6,12,13 health technology assessments,14 and the development of guidelines.15,16
To support decision-makers in making informed decisions on matters of public health and health policy, the World Health Organization (WHO) provides systematically developed guidelines.15 The recommendations set forth in these guidelines are particularly important for policy-makers and program managers in low- and middle-income countries who often have limited resources for conducting comprehensive processes of evidence gathering and analysis. Those responsible for developing WHO guidelines are challenged to balance the need for a comprehensive approach – which is indicated due to the complexity of the intervention, the challenges of evidence generation, and the multiplicity of values affected – with the necessity to provide recommendations in a timely manner and often under considerable resource constraints.
In guidelines and beyond, various approaches to integrate a range of specific considerations have been proposed. These approaches – the Accountability for Reasonableness (A4R) framework by Daniels and Sabin17-21 among others6,14,22,23 – emphasize the importance of transparency throughout the process, the inclusion of relevant stakeholders, the appropriate composition of the decision-making panel, and the identification and weighting of criteria to be considered, as well as the possibility of revisions and appeals. According to A4R,17 one key aspect is the condition of relevance: the decision or recommendation must rest on evidence, reasons, and principles that all fair-minded parties can agree are relevant to deciding how to meet the diverse needs of affected stakeholders under the imposed resource constraints.17 Such processes can increase the acceptability and perceived legitimacy of a decision19,24,25 even if – given varying and sometimes contradictory interests – no consensus regarding the right selection and weighting of criteria can be achieved.19
Involving representatives of all relevant stakeholder groups, including community representatives (eg, citizens, patients) in the process of identifying these reasons and principles is considered ideal.6,14,26,27 However, it is often difficult to meet this ideal due to time and resource constraints; this increases the risk of relevant criteria being overlooked. While not intended to nor able to replace stakeholder participation, Evidence-to-Decision (EtD) frameworks can support decision-makers and guideline developers in this balancing act.28,29 EtD frameworks tend to comprise sets of criteria as well as procedural guidance. They are intended to ensure that all relevant criteria are considered, the best available evidence is assessed, and the underlying rationale is made explicit and transparent.30 When applied well, these EtD frameworks can help identify and integrate the criteria of relevance for a given decision-making process, even if the voices of all relevant stakeholders were not heard. Their use should, however, not be misinterpreted as a justification for an unbalanced or incomplete composition of the committee preparing or making decisions. Furthermore, structured processes guided by EtD frameworks can lead to better, more rational decisions by counteracting inadequate (cognitive, emotional or social) heuristics, cognitive biases, or in-group dynamics.31-35 Therefore, EtD frameworks should be as comprehensive as possible, which often is at odds with the constraints, and needs to be balanced against the resources and time available for developing an informed decision.
WHO uses EtD frameworks in their process to develop guidelines.15,27 Given the reach and potential impact of the recommendations set forth in WHO guidelines, the nature of the guideline development process and the criteria used to inform recommendations set a benchmark for other uses. Both are described in the WHO guideline handbook for guideline development.27 In formulating recommendations WHO Guideline Development Groups (GDG)27 are asked to consider not only evidence of effectiveness but also a range
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–13 3
of other criteria (eg, resource implications, acceptability, feasibility27) derived from an early version of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) EtD framework.27,30,35 Methods for guideline development were originally tailored to clinical interventions, and are still profoundly influenced by this field.36,37 WHO recently commissioned a series of papers to make guideline development methods more applicable to complex public health and health system challenges.15 In this context, a new EtD framework, the WHO-INTEGRATE framework version 1.0,16 was developed with a strong conceptual and normative foundation16 primarily based on an evaluation of WHO norms and values derived from key WHO documents (eg, the WHO constitution) and widely used public health ethics frameworks. To ensure the relevance of the framework, this normative approach was combined with a literature review of decision criteria used in real world decision-making,38 and an assessment of complexity features.16
The WHO-INTEGRATE framework comprises six criteria – balance of health benefits and harms; human rights and sociocultural acceptability; health equity, equality and non-discrimination; societal implications; financial and economic considerations; and feasibility and health system considerations – as well as the meta-criterion quality of evidence. Each criterion encompasses a detailed definition, a set of sub-criteria, example questions to assess these sub-criteria and a methodological toolbox with suggested methods for collecting, synthesizing and appraising evidence. Applicability and benefit of the WHO-INTEGRATE framework to real-world decision-making situations remain to be tested.
The objective of this study was to assess the WHO-INTEGRATE framework version 1.0 through a wider participatory process with experts involved in developing WHO guidelines on an international level as well as decision-makers developing national guidelines and/or adapting and implementing WHO guidelines on a national level. Specifically, it served to review the framework in terms of its overall structure and specific criteria and to shed light on the comprehensiveness, relevance, and usefulness in real-world decision-making contexts.
MethodsWe conducted nine key informant interviews (KIIs) with experts involved in WHO guideline development on an international level as well as four focus group discussions (FGDs) with health decision-makers in Brazil, Germany, Nepal, and Uganda.
Participants and Data CollectionKey Informant InterviewsThe KIIs were conducted with experts who had recently participated in a WHO guideline development process, either as the coordinating WHO staff, GDG chair, or the methodologist supporting the process. In consultation with the Secretariat of the WHO Guideline Review Committee, three WHO guidelines – on sexual and reproductive health and rights of women living with HIV,39 on communicating risk in public health emergencies40 and, on antenatal care for a
positive pregnancy experience41 – were selected purposefully, with the aim to cover distinct types of interventions and to capture positive as well as challenging experiences with applying the GRADE EtD framework.27
In the face-to-face or telephone/video interviews carried out between June and October 2017, we used a semi-structured, pre-tested interview guide, developed based on the guiding research questions (Supplementary file 1). The first interview part focused on the experience of using the GRADE EtD framework to formulate recommendations and decide on their strength. In the second part, the interviewees received an interim version of the WHO-INTEGRATE framework and were asked to comment on practical considerations (eg, understandability), the framework content (eg, missing criteria), and the implications of using the framework in WHO guideline development processes.
The audio files of the recorded interviews were transcribed, reviewed, pseudonymized, and then deleted.
Focus Group DiscussionsThe FGDs were conducted with decision-makers across four countries and continents, as detailed in Table 1 and Supplementary file 2. To capture the diversity of views, we sought to maximize heterogeneity among countries (ie, country income group, region) and topics of discussion (ie, type of intervention/approach).
Local researchers undertook the FGDs in close collaboration with the developers of the framework between August 2017 and October 2018. A topic of current importance was suggested by the local researchers. Decision-makers were identified by the local contact, who reached out to a purposive sample of experts involved with making national recommendations, and potential WHO guideline users, ie, those responsible for adapting and implementing recommendations locally, regionally, or nationally. The recruitment strategy and composition of decision-makers varied across the four FGDs regarding the nature of the committee (providing advice vs. making recommendations), the level of decision-making (national vs. local) and exact composition (representative sub-set of committee vs. ad hoc assembly of participants) (see Table 1 for details).
We developed a preliminary interview guide based on the KII interview guide and adapted it to the setting and topic of the FGDs (Supplementary file 1). The FGDs were set up as a thought experiment: First, the participants conducted a guided brainstorming session on criteria of relevance for the decision-making process on their chosen topic, at which point they were unaware about the content of the WHO-INTEGRATE framework. This was one in order to not have the discussion of criteria and considerations of relevance be “contaminated” or framed by the content of the framework. Second, they were presented with an interim version of the framework and asked to review whether the framework covered the previously discussed criteria, whether aspects were missing, and whether there might be specific suggestions for improvement.
The files of the audio-recorded FGDs were transcribed and the transcripts reviewed by the local researchers. Transcripts
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–134
Table 1. Countries, Thematic Areas and Topics of FGDs
Country Brazil Germany Nepal Uganda
Thematic area Infectious diseases; healthcare system Public health nutrition; non-communicable diseases
Sexual and reproductive health; health services research WaSH; infectious diseases
Topic of FGD Tuberculosis guidelines and decentralized actions related to tuberculosis control
(Health) implications of an elimination of an EU quota system on isoglucose and considerations regarding countermeasures (eg, labelling, taxation and/or prohibition of products)
Health services related to sexual reproductive health and rights of adolescents
Management of untreated wastewater, including sewage from septic tanks and fecal sludge from pit latrines
Country income group Middle-income country High-income country Low-income country Low-income country
WHO region Latin American Region European Region South East Asian Region African Region
Researcher(s) conducting FGD AAM, CEMR JMS DP JO, KS
Date of FGD June 2018 June 2018 October 2017 August 2017
Number of participants n = 17 n = 7 n = 8 n = 8
Characteristics of participants and rationale
Multidisciplinary staff and invited members of National Coordination for Tuberculosis Control Program directly dealing with national policies implementation, decision-making and public health protocols design and adaptation
Staff of the Bavarian Health and Food Safety Authority across several departments advising on and preparing decisions regarding food safety and food regulation on the level of a German federal state
National level experts from governmental institutions (eg, divisions of ministry of health) and Nepali experts from national and international NGOs working on development and implementation of programs of and rights ASRH
Members of a national workgroup on water and sanitation from diverse organizational sectors, including national and sub-national government, nongovernmental/civil society, NGO network, and private actors with expertise in water and sanitation guideline implementation; including Ugandan representative of the Sanitation and Water for All network and WHO sanitation guideline development; (note: foreign aid and multilateral organizations were excluded)
Recruitment approach
Direct contact with implementation science expert and researchers working on the topic to identify a diverse set of key experts; invitation of identified experts by the local researchers directly via email.
Direct contact (personal, via telephone or email) of staff members involved with analyzing the implications of food safety or providing guidance on such countermeasures within the Bavarian Health and Food Safety Authority
Direct contact (in person, telephone) to individual experts from the federal ministry of health and individual national and international NGOs; snowballing recruitment of additional experts in the field through the directly contacted experts
In-person recruitment of key contacts at professional conferences followed by more extensive email recruitment within national water and sanitation work group; snowballing recruitment through referral to other potential participants in their networks of professionals doing work on national WaSH issues
Duration FGD 130 minutes 95 minutes 100 minutes 120 minutes
Setting of data collection
Brasília, BrazilConference room in the building of the National Coordination for the National Tuberculosis Control Program
Munich, GermanyConference room in the building of the Bavarian Health and Food Safety Authority (1 participant via video link)
Kathmandu, NepalConference room at centrally located hotel
Kampala, UgandaConference room at WHO offices
Language of FGD/analysis Portuguese/English German/German Nepali/English English/English
Abbreviations: WaSH, Water, sanitation, and hygiene; FGD, focus group discussion; EU, European Union; WHO, World Health Organization; ASRH, adolescent sexual and reproductive health and rights; NGO, non-governmental organization.
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–13 5
were subsequently pseudonymized, translated into English (Brazil and Nepal) and reviewed after translation in the light of the audio-records. Audio-records were then deleted.
Data AnalysisA two-person team (JMS and IS, ST or KK) analyzed the pseudonymized transcripts (KIIs and FGDs) through qualitative content analysis, following the approach by Mayring.42 Findings for the FGDs were provided to local contacts for feedback, clarification, and discussion. The analysis of the transcript of the FGD conducted in Germany was not translated but rather analyzed in German by two native speakers (JMS and KK). We followed mixed deductive and inductive approaches to develop the coding frames (Supplementary file 3) using the software MAXQDA12 (VERBI Software GmbH, Berlin). Furthermore, one researcher (JMS) assessed coded passages of the FGD transcripts with respect to whether aspects considered relevant for decision-making on the topic and theme in question were covered by the criteria and sub-criteria of the WHO-INTEGRATE framework.
ResultsWe conducted nine KIIs with WHO guideline staff (n = 2), GDG chairs (n = 4), and methodologists (n = 3) with a median duration of 62 minutes (range 57-69 minutes). Two additionally intended interviews did not take place, as one participant had retired (and therefore declined) and one interview could not be scheduled despite repeated attempts.The duration of the four FGDs with between seven and seventeen participants ranged from 95-150 minutes (Table 1, Supplementary file 2). The topics were: tuberculosis guidelines and decentralized actions related to tuberculosis control (Brazil), the (health) effects of an increase of isoglucose in food and potential countermeasures (Germany), health services related to sexual reproductive health and rights of adolescents (Nepal), and the management of untreated wastewater, including sewage from septic tanks and fecal sludge from pit latrines (Uganda).
Overview of Focus Group DiscussionsFor conciseness, we aim to provide a synthesis rather than detailed account of all four FGDs here. Supplemental data is available by contacting the authors. The next section outlines the first phase of the discussions, as this varied, and summarizes selected main themes.
Focus Group Discussion in BrazilThe FGD in Brazil was concerned with the development of tuberculosis guidelines and manuals within the Brazilian national plan to fight tuberculosis.
During the first phase, participants discussed their experiences with the development of this plan, covering both general challenges in guideline development (eg, alignment with national and supranational strategies) and specific considerations in making recommendations; most often they referred to testing strategies as an example. Criteria of relevance brought up during this phase included affordability,
availability, acceptability of healthcare access, adherence to treatments, economic and financial feasibility, cost-effectiveness, and political importance.
A central consensual theme was the need to accommodate the realities of a heterogeneous country (“multiple Brazils”) comprising municipalities with high- versus low-disease burdens, and the related needs to address subsidiarity and empowerment of municipalities to develop locally adapted approaches and to target social determinants within a health in all policies approach.
A second major theme was the criterion of acceptability: participants agreed on the importance of socio-cultural acceptability, especially among those intended to implement the intervention and the intended beneficiaries, stressing that acceptability could vary greatly even on a local level (eg, between healthcare institutions) and across population groups (eg, across different ethnicities or religious groups). Participants thought it unlikely to achieve socio-cultural acceptability overall. Several stated that they were unsure about how to handle the acceptability criterion based on a lack of or false knowledge among those rejecting an intervention (eg, the belief that a vaccination was developed to kill elderly).
An important point of controversy related to the question whether a separate sub-criterion regarding the right to health needed to be added. The argument in favour of adding a sub-criterion was the framework’s focus on health, the argument against doing so was that the right to health is already covered within the broader sub-criterion on human rights.
Focus Group Discussion in GermanyThe FGD in Germany focused on food safety and food regulation, notably the expiration of an quota system of the European Union (EU) on the market share of isoglucose, which is expected to increase high-fructose corn syrup (HFCS) in foods and beverages. Due to concerns about adverse health effects of HFCS among the general public and parliamentarians, participants discussed whether the Bavarian Health and Food Safety Authority should issue a recommendation on countermeasures.
During a first phase, participants were presented with a rapid literature review, concluding that there is an ongoing controversy regarding adverse health effects of HFCS. The participants then engaged in lively discussions on whether countermeasures such as labelling, taxation and/or prohibition of products should be taken.
A central theme in the debate was the legal feasibility of such measures and lack of clarity regarding the responsible political level (ie, federal state, nation state, EU): participants discussed whether it would be in line with EU and national regulations and law if the Bavarian government would adopt and implement regulations.
A major point of controversy was the need to balance the expected avoidance of harm, the (lack of) certainty of evidence, and the intrusiveness of the intervention. Some participants argued that in view of inconclusive evidence of harm the government does not have a mandate to act. Others argued with the precautionary principle, which allows enacting regulations to protect public health despite
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–136
unresolved uncertainties. A further controversy was on the role of evidence in
decision-making: methodological and ethical challenges can be prohibitive in proving harm beyond a reasonable doubt. Conflicts of interest can distort the evidence, both with respect to industry-funded research and evidence generated by researchers with ideological or personal interests (white hat bias43).
Focus Group Discussion in NepalThe FGD in Nepal focused on adolescent sexual and reproductive health and rights (ASRH). During the first phase, participants voiced their views on important criteria for developing and implementing guidelines focused on ASRH. These included: the capacity of healthcare providers, privacy, and user-friendliness. The discussion then shifted towards experience with adapting international guidelines (eg, from WHO) and implementing ASRH programs. While international guidelines were considered useful for procuring resources from the government, the group agreed that simply transferring global guideline recommendations to local realities can be challenging (eg, due to limited acceptability or resources).
A major topic of discussion was the issue of socio-cultural acceptability. Importantly, guideline recommendations and programs cannot achieve their intended goals if they do not meet the needs and expectations of adolescents. Participants suggested that this could be achieved by engaging adolescents in developing guidelines and thus creating ownership of the program. Acceptability also encompassed community norms and values, eg, regarding gender issues. If these are not considered, a program’s effectiveness and implementation would suffer.
Another central theme was the need to take local realities into account with respect to feasibility considerations. This covers locally available infrastructure as well as financial, technical and human resources. Local coordination within and beyond the health sector was regarded as essential.
Focus Group Discussion in UgandaThe FGD in Uganda focused on the management of untreated wastewater, including sewage from septic tanks and fecal sludge from pit latrines within the larger context of the WHO guidelines on sanitation and health.
During the first phase, participants discussed the importance of managing wastewater, septage and sludge, as well as the reasons for developing (international) guidelines on the topic and for potentially opposing such guidelines. Participants brought up considerations with respect to health implications of the measure, implementation and maintenance costs, and feasibility considerations.
A central theme was the interlinkage between financial costs, resource availability, feasibility, and acceptability of the intervention. Participants agreed that these aspects needed to be reflected both from the perspective of the national as well as the local government and end users. Guideline recommendations might be rejected if they were regarded as
too imposing in terms of cost and resource claims or regarded as unachievable under local circumstances.
A related central topic was the need to consider resource requirements broadly in guideline recommendations: These need to reflect the required institutional infrastructure beyond the immediate needs for implementing and maintaining the intervention, such as infrastructure for planning, budgeting, or procuring resources as well as monitoring and evaluation.
Differences and Commonalities Across the FGDs Health implications of interventions were discussed in all four FGDs, as was the need for multisectoral collaboration, implications for the health system, and consequences beyond the health sector.
While all FGDs emphasized the importance of evidence (regarding effectiveness), the debate in Germany focused on the trustworthiness of the evidence, while the other FGDs emphasized transferability and generalizability from an international level to local realities.
Socio-cultural acceptability of the intervention was discussed across all FGDs but the focus on different stakeholder groups varied: the FGD in Nepal concentrated on intended beneficiaries and the general population, the FGDs in Brazil and Uganda discussed the acceptability among those implementing and those intended to benefit from the intervention, and the German FGD was focused on political acceptability.
Furthermore, the need to consider local perspectives was raised in all FGDs. In Brazil, discussions primarily regarded the different needs and realities of municipalities, in Uganda and Nepal this focus lay on acceptability, resources availability, and feasibility, and in Germany with a concern was legal feasibility (of passing laws on the level of federal states). In this context, participants in the FGDs in Brazil, Nepal and Uganda emphasized procedural considerations in guideline development, highlighting the need to involve affected stakeholders in the process.
The importance of international guidelines and recommendations (eg, from WHO) were discussed in the FGDs in Brazil, Nepal, and Uganda, eg, regarding their usefulness in developing local guidelines or gaining political support. These were not addressed in the German FGD.In Nepal and Uganda, there was limited controversy: FGD participants seemed to strive for consensus, eg, on regarding missing criteria. In contrast, within the FGD in Germany and, to a lesser extent, in Brazil, discussions were more controversial. To some degree this may be due to cultural norms (eg, regarding deference to authority or conflict tolerance).
General Reception of the WHO-INTEGRATE Framework The majority of interviewees in the KIIs made positive remarks about the framework, notably its usefulness and comprehensiveness. One participant remarked that the new framework covers many important issues and expressed a clear preference for this framework compared to the one in the WHO guideline handbook.27 Several interviewees
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–13 7
explicitly stated that the criteria covered in the framework are important and none could or should be dropped in order to reduce the workload. Explicit positive statements were made regarding the new or expanded criteria Societal implications, Balance of health benefits and harms, Equity, equality and non-discrimination, Human rights and socio-cultural acceptability, and Feasibility and health system considerations and their sub-criteria, for example:
“I think that the framework in its new form with this additional guidance is really informative, and useful, and helpful to participants in these panels and hopefully leads to good recommendations” (KII_Methodologist).One interviewee felt the WHO-INTEGRATE framework
did not go far enough by following the same approach as the GRADE EtD framework, namely starting with a defined intervention, gathering evidence and deciding whether a recommendation should be made; accordingly, a more appropriate approach might be to focus on beneficiaries and ask what should be done to improve health and well-being. Another interviewee remarked that focusing on high quality, quantitative evidence of effectiveness for a clearly defined intervention and outcome may not be feasible for complex interventions.
It was also noted that many aspects in the framework were highly context-dependent and may therefore be less applicable in the development of global guidelines. Furthermore, two interviewees questioned the added value beyond current practice and implied that all newly added sub-criteria could also be addressed as part of the GRADE EtD framework. One interviewee, however, recognized that more explicit sub-criteria could function as “signposts” for less experienced methodologists:
“I like the idea of making it more explicit so that you do think of these things. But if you’re quite a high-level expert, you would automatically do that […]” (KII_Methodologist).Participants in all four FGDs made positive remarks
regarding the framework and its criteria, notably their comprehensiveness. They explicitly mentioned the importance of separating individual and population perspectives regarding health benefits and harms, the range of feasibility considerations, and the broad perspective beyond mere health implications of an intervention. No general critical remarks about the framework were made in the FGDs.
Suggestions Towards Modifying the WHO-INTEGRATE Framework Table 2 provides an overview of suggestions for improvement derived from the KIIs or FGDs.
Wording and DefinitionsParticipants in the KIIs and FGDs made several specific suggestions to expand upon and offer more guidance on selected criteria and sub-criteria; notably the criterion Societal implications was described as “fuzzy and vague” along with the sub-criteria Accordance with human rights, Environmental implications and Intrusiveness of the intervention.
“These [criteria] [...] ‘impact on health system,’ ‘social
impact,’ they are very vague” (KII_WHO-Staff).Other suggestions included rewording “impact” to
“implications,” distinguishing affordability more clearly from financial considerations, and clarifying the types of stakeholders that should be considered with respect to acceptability.
Missing AspectsSeveral interviewees stated explicitly that no criterion seemed to be missing in the framework; others suggested that the framework might not be sufficiently conducive for reflecting on underserved populations and vulnerable groups. They further recommended that a legal expert should assess whether supportive legal environments were sufficiently covered.
FGD discussants noted several potentially missing aspects including intervention sustainability, reliability and quality of an intervention, and outcomes related to well-being, for instance:
“The benefits […] we define as […] professional[s] and [..] that adolescent[s] would define [..] is different: The pleasure of being together with a partner, physical contacts, enjoying beer and cigarette for them is special. I am not sure if [these] benefits [are] considered” (FGD_Nepal).Furthermore, participants in the FGDs discussed whether
political feasibility (eg, political and administrative facilitators and barriers) was sufficiently covered, in particular in regard to political feasibility on the local administrative and political level.
Order and Grouping of Criteria and Sub-criteriaSeveral interviewees commented on the classification of Patients’/beneficiaries’ values in relation to health outcomes as a sub-criterion. As this is a (main) criterion in the current EtD framework,27 some interviewees were concerned that this aspect may not receive enough attention if only addressed as a sub-criterion.
“I feel like [patients’/beneficiaries’ values in relation to health outcomes] is not really balance of benefits and harms. [...] So I wonder if maybe this can be part of the acceptability and values. Or something like that” (KII_WHO-Staff).Discussants recommended a separation of human rights
and acceptability considerations into two distinct criteria. Also, non-discrimination could be framed as a human rights consideration, rather than an aspect under Equity and equality. Furthermore, they suggested combining societal impact and health impact into one broad impact-oriented criterion.
Overlap, Redundancies and Delineation of Criteria and Sub-criteriaSeveral interviewees commented on blurred boundaries between criteria and sub-criteria, eg, between the criterion Health equity, equality and non-discrimination and the sub-criterion Social impact, between the sub-criterion Interaction with and impact on the health system and the criterion Financial and economic considerations, as well as between acceptability considerations and the sub-criterion Patients’/
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–138
Table 2. Overview of Suggestions for Modifications of Framework, Criteria or Sub-criteria Based on FGDs and KIIs
Criteria and sub-criteria
Suggestions for Modifications of Framework, Criteria or Sub-criteria, Based on Statements in One or More FGDs and/or KIIs
Wording and Definition Missing Aspects Order and
Grouping Overlap, Redundancy and
Delineation
Balance of health benefits and harms FGD
Efficacy or effectiveness on health of individuals FGD FGD
Efficacy or effectiveness on health of population
Patients’/beneficiaries’ values in relation to health outcomes KII, FGD KII KII
Safety-risk-profile of intervention FGD
Broader positive or negative health-related impacts
Human rights and socio-cultural acceptability FGD FGD
Accordance with universal human rights standards KII, FGD
Socio-cultural acceptability to beneficiaries and those KII, FGD KII KII, FGD
Socio-cultural acceptability of intervention to the public and other stakeholders KII, FGD FGD
Impact on autonomy of concerned stakeholders
Intrusiveness of intervention FGD FGD
Equity, equality and non-discrimination FGD KII, FGD FGD
Impact on health equality and/or health equity
Distribution of benefits and harms of intervention
Affordability of intervention KII FGD KII
Accessibility of intervention
Lack of a suitable alternative
Societal implications KII, FGD FGD
Social impact KII, FGD KII
Environmental impact KII, FGD
Financial and economic considerations
Financial impact KII FGD FGD
Impact on economy FGD
Ratio of costs and benefits FGD
Feasibility and health system considerations
Legislation KII FGD
Leadership and governance FGD FGD
Interaction with and impact on health system FGD FGD KII
Need for, usage of and impact on health workforce and human resources FGD
Need for, usage of and impact on infrastructure FGD FGD
Quality of evidence (meta-criterion)
Suggestions regarding missing criteria FGD
Suggestions regarding the order of criteria FGD
Abbreviations: FGD, focus group discussion; KII, key informant interview.An expanded version of this table is provided as a supplement. Supplementary file 4 details the suggested changes to the WHO-INTEGRATE framework based on KII and FGD, and Supplementary file 5 provides exemplary quotes based on the FGDs.
beneficiaries’ values in relation to health outcomes. “I think it is just equity and non-discrimination and
societal impact, there are some things that are overlapping. […] How would you really delineate?” (KII_WHO-Staff).Participants in the FGDs whether the financial and
economic as well as the resource considerations were adequately delineated in light of multiple payers on several geographical levels.
Relevance of Criteria and Sub-criteria Based on Focus Group Discussions OnlyDepending on the theme and topic of the FGD, different criteria dominated the discussions; nevertheless, references to all six criteria were identified in all four FGDs (Table 3, Supplementary file 6). For example, the FGD in Germany was dominated by the challenge to balance the intrusiveness of interventions and the resultant limitations inflicted on
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–13 9
individual liberties with the potential health impacts and the available evidence. However, not all sub-criteria were discussed in every FGD. For example, implications for the (natural) environment were only explicitly discussed in the FGD in Uganda (focused on wastewater management). The most discussed themes included various (health) implications of the intervention, acceptability, accessibility, and autonomy and feasibility considerations. When asked, participants from all four FGD judged the framework to cover their reasoning well.
“I think everything here needs to be kept. You’d rather furnish the decision-makers with more information than they need than less. And as far as I’m concerned, whatever’s in here would be really relevant” (FGD_Uganda).
Implications for Using the WHO-INTEGRATE Framework in the WHO Guideline Development Process Based on Key Informant Interviews OnlySeveral interviewees were concerned that the complexity
and the additional workload associated with the WHO-INTEGRATE framework might be overwhelming for the guideline development process. This may lead to the process merely paying lip-service to criteria, such as Societal implications, and skipping over important domains. Budget constraints and limited time had to be considered when applying the framework.
“I think that the guideline panels will find it [the expanded criteria and sub-criteria] more burdensome because to discuss all of these things will take longer. […] I think the panels get exhausted. They get tired and then they start skipping over, and they skip quite a lot” (KII_Staff).In contrast, one interviewee stated that “cutting corners”
to reduce the workload would diminish the value of the final product. This participant emphasized the need to raise the appropriate resources for a guideline to be “done right” and that using the framework as part of a well-coordinated process would not necessarily lead to a more expensive endeavor.
Several participants stressed the need for additional
Table 3. Overview of Passages in the FGDs Containing a Reference to a Criterion or Sub-criteria Covered by the WHO-INTEGRATE Framework or Passages Mentioning a Criterion or Sub-criteria as Relevant for a Decision-Making Process
Criteria and Sub-criteria Brazil Germany Nepal Uganda
Balance of health benefits and harms Yes Yes Yes YesEfficacy or effectiveness on health of individuals Yes
Efficacy or effectiveness on health of population Yes Yes
Patients’/beneficiaries’ values in relation to health outcomes Yes Yes
Safety-risk-profile of intervention Yes
Broader positive or negative health-related impacts Yes Yes
Human rights and socio-cultural acceptability Yes Yes Yes Yes
Accordance with universal human rights standards Yes Yes
Socio-cultural acceptability to beneficiaries and those Yes Yes Yes
Socio-cultural acceptability of intervention to the public and other stakeholders Yes Yes Yes
Impact on autonomy of concerned stakeholders Yes Yes Yes Yes
Intrusiveness of intervention Yes Yes
Equity, equality and non-discrimination Yes Yes Yes Yes
Impact on health equality and/or health equity Yes Yes
Distribution of benefits and harms of intervention Yes Yes
Affordability of intervention Yes Yes Yes
Accessibility of intervention Yes Yes Yes Yes
Lack of a suitable alternative Yes
Societal implications Yes Yes Yes Yes
Social impact Yes
Environmental impact Yes
Financial and economic considerations Yes Yes Yes Yes
Financial impact Yes Yes Yes
Impact on economy Yes Yes
Ratio of costs and benefits Yes Yes
Feasibility and health system considerations Yes Yes Yes Yes
Legislation Yes Yes Yes Yes
Leadership and governance Yes Yes
Interaction with and impact on health system Yes
Need for, usage of and impact on health workforce and human resources Yes Yes
Need for, usage of and impact on infrastructure Yes YesQuality of evidence (meta-criterion) Yes Yes Yes Yes
Abbreviations: FGD, focus group discussion; WHO, World Health Organization.
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–1310
guidance, including general guidance on how to apply the WHO-INTEGRATE framework and more specific guidance on how to select and interpret criteria and sub-criteria.
Several interviewees further remarked that identifying appropriate evidence eg, for Health system and feasibility considerations, Financial and Economic considerations, or Societal impact might be challenging due to limited availability, low certainty and high context-dependency of evidence.
DiscussionDiscussion of Key FindingsIn this qualitative study, we received feedback on the WHO-INTEGRATE framework from WHO guideline developers as well as national public health and health policy decision-makers, and identified suggestions for modifications (Table 2, Supplementary file 5). Overall, the framework, its underlying conceptualization and its comprehensive nature, as well as the detailed criteria and sub-criteria were positively received. Some key informants voiced concerns regarding the implications of applying this framework with its very comprehensive set of criteria in everyday guideline development processes. A need for practical guidance was emphasized.
All criteria and sub-criteria of the WHO-INTEGRATE framework were discussed or mentioned as relevant in at least one of the four FGDs (Table 3). Moreover, interviewees and participants in FGDs commented positively on the framework’s comprehensiveness. The developers of the WHO-INTEGRATE framework considered the few criteria highlighted to be missing by KII or FGD participants to be covered, pointing to a need to revise wording and provide clarification (see below). Since the FGDs and KIIs did not identify relevant gaps, this outcome aligns with the A4R framework’s17 condition of relevance, which depends on decision-makers using the framework properly. Our findings therefore suggest that the criteria included in the WHO-INTEGRATE framework can be considered both comprehensive and relevant for real-world public health decision-making.
These findings are noteworthy in view of the heterogeneity of topics and settings in the FGDs and the diverse WHO guidelines selected for the KIIs (Table 3, Supplementary file 5). Likely, the overview of systematic reviews of real world decision criteria,16,38 undertaken to develop the WHO-INTEGRATE EtD framework played an important role: within the included systematic reviews,9,10,44,45 KIIs9,46,47 and FGDs9,47-49 were employed and the criteria used or suggested for use in these studies were similar to those discussed or mentioned in our KIIs and FGDs.16,38
Beyond the aspects explicitly mentioned in KIIs and FGDs, the developers of the WHO-INTEGRATE framework (JMS, ER, IBS) noted some additional domains where a modification of the framework may be warranted. For example, while “individual well-being” was reported as missing in one FGD, the developers considered this aspect covered by the broad WHO concept of health, which was a key building block towards the WHO INTEGRATE framework. Similarly, the broad conceptualization of health systems50 (beyond healthcare systems) may not have been
clear to all participants and may warrant changes in wording and definitions. Furthermore, in a future revision process the sub-criteria legislation and leadership and governance may need to be more fully described and the criteria relating to availability, accessibility or lack of a suitable alternative may need to be refined with reference to complex public health interventions (eg, labeling interventions).
A concern voiced in the KIIs was the potential additional burden that the use of the WHO-INTEGRATE framework could impose on the guideline development process. This concern was not expressed in the FGDs. These different perceptions are also reflected in the literature: On the one hand, not adequately considering relevant criteria and the views of public health and health policy decision-makers as end-users was found to be a barrier to guideline adherence and implementation.51,52 On the other hand, the balancing act between rigorous methods and finite resources, notably limited time, repeatedly emerged as an obstacle in structured decision-making processes such as guideline development,53,54 and the necessity of pragmatic approaches is frequently emphasized.53-55 As highlighted in the publication of the WHO-INTEGRATE framework version 1.0,16 one solution to resolving the conflict between the comprehensiveness and granularity of the framework (both well-received) and the implications for guideline development (viewed with some concern) is to insist on a broad approach by considering all six criteria as well as the meta-criterion quality of evidence while allowing for much flexibility in terms of sub-criteria to be considered and evidence to be collected towards these (or not).56
While EtD frameworks can support decision-makers in identifying relevant criteria for public health and health policy decisions, this does not supersede the need to address the value-laden nature of these decisions in other parts of the process.16 The substantive criteria put forward in the WHO-INTEGRATE framework need to be integrated with procedural considerations that address issues of fairness, participation, transparency and the right to appeal,6,14,17,22 eg, as suggested in what is referred to as evidence-informed deliberative processes.6,14,57 Major efforts should be made to achieve a balanced composition of the committee preparing for or making decisions, and to ensure representation of affected stakeholders. A balanced committee using an appropriate framework is poised to produce fair and reasonable decisions that are perceived as acceptable and legitimate – a point emphasized by interviewees and participants in the FGDs alike.
Strengths and LimitationsThis study followed a comprehensive and rigorous approach to capture the perspectives of those developing WHO guidelines and those potentially adapting and implementing WHO guidelines and/or developing national recommendations across four different countries and continents. Participants represented diverse roles and backgrounds, and two modes of obtaining insights (KIIs and FGDs) were pursued. While the development of other frameworks for structured decision-making58-62 mostly employed confirmatory approaches (eg,
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–13 11
surveys to rate the importance of criteria63), our FGDs pursued a more open-ended approach stimulating participants to reflect deeply on issues of relevance for a given topic and to discuss the framework, its structure and criteria more freely.
While the analysis was led by a researcher of German origin, data collection and analysis were conducted with local researchers to allow for an inter-cultural and interdisciplinary perspective. A thematic area and topic of relevance were proposed by local experts as the basis for the FGD in each country. Despite these strengths, the discussion represented a theoretical decision-making scenario, and criteria other than those discussed might arise in real-world decision-making. Two FGDs (Nepal and Brazil) were translated and it cannot be ruled out that nuances of the discussions were lost in translation, although efforts have been made to preserve the meaning in the process (eg, through contextualization of statements by local researchers and involvement of the local researchers in the analysis). A further limitation is that we did not systematically assess whether participants had financial or non-financial conflicts of interest in relation to the topic in all four FGDs, and analyzed how this may have affected their positioning in the discussion.
Given the multiple dimensions of heterogeneity present in the FGDs (notably in terms of countries and topics), it is difficult to assess whether saturation was reached. Due to the differences in the nature of the topics, the discussions focused on different aspects and criteria (eg, the FGD in Nepal concerned with sexual and reproductive health did not address environmental implications, while this was an important consideration in the FGD in Uganda concerned with the management of untreated wastewater).64 Additional FGDs (among others in the WHO Western Pacific region) on an even broader set of topics might provide further considerations for advancing the framework. While we captured the perspective of a relatively small sample of users (four FGDs) and developers (nine KIIs), as discussed above, the WHO-INTEGRATE framework itself 65 builds on a much more comprehensive sample of similar real-world events.38
ConclusionOur study suggests that the WHO-INTEGRATE framework can be a valuable resource for better-informed public health and health systems decisions. Reacting to the suggestions for improvement made by potential end-users, the developers are in the process of developing practical guidance for applying the WHO-INTEGRATE framework. Moreover, the applicability and added value of the framework will need to be tested in real-world guideline development and other decision-making processes, as planned with several upcoming WHO guidelines. The WHO-INTEGRATE framework was explicitly published as a living document. The findings presented here provide a valuable starting point to advance the framework towards a version 2.0.
AcknowledgementsThe authors would like to thank all focus group participants and all key informants for their time and engagement. We are very grateful to Susan L. Norris, Anayda Portela, and Rebekah
Thomas Bosco as well as other WHO staff for helping with the identification of suitable guidelines and key informants. Moreover, we would like to thank John Ssempebwa for facilitating the FGD in Uganda, and local assistants Monica Lima and Pedro Bossonario involved with supporting the authors AAM and CEMR in conducting the FGDs in Brazil. We are also grateful to the student assistants Sarah Thumbeck and Katja Krug for contributing to data analysis.
Ethical issues Ethical approval was obtained from the Research Ethics Review Committees (RERC) of the WHO and the LMU Munich, as well as from local RERCs in Brazil, Nepal, and Uganda. All participants provided written and verbal consent prior to participation.
Competing interests JMS and IBS: Developer of the WHO-INTEGRATE EtD framework. EAR: Developer of the WHO-INTEGRATE EtD framework, member of the GRADE Working Group. JMS reports grants from WHO Department of Maternal, Newborn, Child and Adolescent Health, personal fees from Bavarian Health and Food Safety Authority, other from Norwegian Agency for Development Cooperation (NORAD), during the conduct of the study.
Authors’ contributions JMS and EAR designed and directed the project with contribution from IBS. JMS developed with interview guide for KIIs with input from EAR and IBS. JMS and EAR recruited the participants with the help from Susan L. Norris, Anayda Portela, Rebekah Thomas Bosco, and other WHO staff. JMS conducted the KIIs. KII-transcripts were analyzed by JMS with contribution from IBS, ST, and KK. DP, JO; KES, CEMdR, AAM, MF suggested topic and recruited participants for the FGDs as well as prepared and facilitated focus group discussions. DP, JO, KES, CEMR, AAM, MF revised and adapted the discussion guide drafted by JMS. JS supported this process in Uganda and ML and PB supported the facilitation of the FGDs in Brazil. JMS, DP, SES, CEMR, and AAM conducted the FGDs. DP, JO, KES, CEMR, AAM transcribed, translated and contextualized the FGD-audio records. JMS analyzed the FGD transcripts with contribution with contribution from IBS, ST, and KK, as well as input from all authors. JMS took the lead in writing the manuscript with input from all authors. All authors provided critical feedback and helped shape the research, analysis and manuscript.
FundingThe WHO-INTEGRATE EtD Framework Version 1.0 was developed with funding provided by the WHO Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation. We also gratefully acknowledge that JMS’s position was funded by the Bavarian Health and Food Safety Authority, and that IBS’s input was funded by the Norwegian Agency for Development Cooperation (NORAD). The United States Agency for International Development, the Bavarian Health and Food Safety Authority and NORAD did not have any influence on the research process or content of this manuscript.
Authors’ affiliations1Institute for Medical Information Processing, Biometry, and Epidemiology – IBE, LMU Munich, Munich, Germany. 2Pettenkofer School of Public Health, Munich, Germany. 3Save the Children, Kathmandu, Nepal. 4The Water Institute, Department of Environmental Sciences and Engineering, The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. 5National Supplementary Health Agency, Ministry of Health, Brasília, Brazil. 6Department of Maternal-Infant Nursing and Public Health, College of Nursing, University of São Paulo, Ribeirão Preto, Brazil. 7Department of Disease Control and Environmental Health, Makerere University School of Public Health, Kampala, Uganda. 8Department of Global Health, Norwegian Institute of Public Health, Oslo, Norway. 9Bavarian Health and Food Safety Authority, Munich, Germany.
Supplementary filesSupplementary file 1. Interview Guide for KIIs and FGDs.Supplementary file 2. Details on KIIs and FGDs. Supplementary file 3. Coding Frames Used to Code Transcripts of KIIs and FGDs.
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–1312
Supplementary file 4. Suggested Changes to the WHO-INTEGRATE Framework Based on KIIs and FGDs.Supplementary file 5. Suggested Changes to the WHO-INTEGRATE Framework Based on FGDs With Exemplary Quotes.Supplementary file 6. Coverage of Decision Aspects Stated in the FGDs by Criteria and Sub-criteria of the WHO-INTEGRATE Framework.
References1. Sanderson I. Intelligent policy making for a complex world:
pragmatism, evidence and learning. Polit Stud. 2009;57(4):699-719. doi:10.1111/j.1467-9248.2009.00791.x
2. Petticrew M, Knai C, Thomas J, et al. Implications of a complexity perspective for systematic reviews and guideline development in health decision making. BMJ Glob Health. 2019;4(Suppl 1):e000899. doi:10.1136/bmjgh-2018-000899
3. Rutter H, Savona N, Glonti K, et al. The need for a complex systems model of evidence for public health. Lancet. 2017;390(10112):2602-2604. doi:10.1016/s0140-6736(17)31267-9
4. Lewin S, Hendry M, Chandler J, et al. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol. 2017;17(1):76. doi:10.1186/s12874-017-0349-x
5. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol. 2009;43(3-4):267-276. doi:10.1007/s10464-009-9229-9
6. Baltussen R, Jansen MP, Mikkelsen E, et al. Priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. Int J Health Policy Manag. 2016;5(11):615-618. doi:10.15171/ijhpm.2016.83
7. Baltussen R, Mikkelsen E, Tromp N, et al. Balancing efficiency, equity and feasibility of HIV treatment in South Africa - development of programmatic guidance. Cost Eff Resour Alloc. 2013;11(1):26. doi:10.1186/1478-7547-11-26
8. Kapiriri L, Lee NM, Wallace LJ, Kwesiga B. Beyond cost-effectiveness, morbidity and mortality: a comprehensive evaluation of priority setting for HIV programming in Uganda. BMC Public Health. 2019;19(1):359. doi:10.1186/s12889-019-6690-8
9. Guindo LA, Wagner M, Baltussen R, et al. From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc. 2012;10(1):9. doi:10.1186/1478-7547-10-9
10. Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health. 2009;14(8):930-939. doi:10.1111/j.1365-3156.2009.02311.x
11. United Nations. Transforming our World: The 2030 Agenda for Sustainable Development. New York: United Nations; 2015.
12. Holm S. The second phase of priority setting. Goodbye to the simple solutions: the second phase of priority setting in health care. BMJ. 1998;317(7164):1000-1002.
13. Kapiriri L, Martin DK. A strategy to improve priority setting in developing countries. Health Care Anal. 2007;15(3):159-167. doi:10.1007/s10728-006-0037-1
14. Baltussen R, Jansen MPM, Bijlmakers L, et al. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health. 2017;20(2):256-260. doi:10.1016/j.jval.2016.11.019
15. Norris SL, Rehfuess EA, Smith H, et al. Complex health interventions in complex systems: improving the process and methods for evidence-informed health decisions. BMJ Glob Health. 2019;4(Suppl 1):e000963. doi:10.1136/bmjgh-2018-000963
16. Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, Baltussen R. The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective. BMJ Glob Health. 2019;4(Suppl 1):e000844. doi:10.1136/bmjgh-2018-000844
17. Daniels N, Sabin J. The ethics of accountability in managed care reform. Health Aff (Millwood). 1998;17(5):50-64. doi:10.1377/hlthaff.17.5.50
18. Daniels N. Accountability for reasonableness. BMJ. 2000;321:1300-1301. doi:10.1136/bmj.321.7272.1300
19. Daniels N, Sabin JE. Accountability for reasonableness: an update
BMJ. 2008;337:a1850. doi:10.1136/bmj.a185020. Daniels N, Sabin J. Just Health: Meeting Health Needs Fairly. 1st ed.
Cambridge: Cambridge University Press; 2008.21. Daniels N, Sabin J. The ethics of accountability in managed care
reform. Health Aff (Millwood). 1998;17(5):50-64. doi:10.1377/hlthaff.17.5.50
22. Friedman A. Beyond accountability for reasonableness. Bioethics. 2008;22(2):101-112. doi:10.1111/j.1467-8519.2007.00605.x
23. Singer PA, Martin DK, Giacomini M, Purdy L. Priority setting for new technologies in medicine: qualitative case study. BMJ. 2000; 321(7272):1316-1318. doi:10.1136/bmj.321.7272.1316
24. Tyler TR. Psychological perspectives on legitimacy and legitimation. Annu Rev Psychol. 2006;57:375-400. doi:10.1146/annurev.psych.57.102904.190038
25. Greenberg J, Folger R. Procedural justice, participation, and the fair process effect in groups and organizations. In: Paulus PB, eds. Basic Group Processes. New York, NY: Springer; 1983:235-256. doi:10.1007/978-1-4612-5578-9_10
26. Saltelli A, Giampietro M. What is wrong with evidence based policy, and how can it be improved? Futures. 2017;91:62-71. doi:10.1016/j.futures.2016.11.012
27. World Health Organization (WHO). WHO Handbook for Guideline Development. 2nd ed. Geneva, Switzerland: WHO; 2014.
28. Ham C. Priority setting in health care: learning from international experience. Health Policy. 1997;42(1):49-66. doi:10.1016/s0168-8510(97)00054-7
29. Niessen LW, Bridges J, Lau BD, et al. Assessing the Impact of Economic Evidence on Policymakers in Health Care-A Systematic Review. Rockville, MD: Agency for Healthcare Research and Quality (US); 2012.
30. Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016. doi:10.1136/bmj.i2016
31. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc. 2006;4:14. doi:10.1186/1478-7547-4-14
32. Nutt PC. Surprising but true: half the decisions in organizations fail. Acad Manag Perspect. 1999;13(4):75-90. doi:10.5465/ame.1999.2570556
33. Robinson R. Limits to rationality: economics, economists and priority setting. Health Policy. 1999;49(1-2):13-26. doi:10.1016/s0168-8510(99)00040-8
34. Gigerenzer G. Decision making: nonrational theories. In: Smelser NJ, Baltes PB, eds. International Encyclopedia of the Social & Behavioral Sciences. Oxford: Elsevier Science; 2001:3304-3309.
35. Moberg J, Oxman AD, Rosenbaum S, et al. The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst. 2018;16(1):45. doi:10.1186/s12961-018-0320-2
36. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet. 2017;390(10092):415-423. doi:10.1016/s0140-6736(16)31592-6
37. Gopinathan U, Hoffman SJ. Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization. BMJ Glob Health. 2018;3(5):e000716. doi:10.1136/bmjgh-2018-000716
38. Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Eff Resour Alloc. 2020;18:8. doi:10.1186/s12962-020-0203-6
39. World Health Organization (WHO). Consolidated Guideline on Sexual and Reproductive Health and Rights of Women Living with HIV. Geneva, Switzerland: WHO; 2017.
40. World Health Organization (WHO). Communicating Risk in Public Health Emergencies: A WHO Guideline for Emergency Risk Communication (ERC) Policy and Practice. Geneva, Switzerland: WHO; 2018.
41. World Health Organization (WHO). WHO Recommendations on Antenatal Care for a Positive Pregnancy Experience. Geneva, Switzerland: WHO; 2014.
Stratil et al
International Journal of Health Policy and Management, 2020, x(x), 1–13 13
42. Mayring P. Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution. Klagenfurt, Austria: Social Science Open Access Repository; 2014.
43. Cope MB, Allison DB. White hat bias: examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting. Int J Obes (Lond). 2010;34(1):84-88. doi:10.1038/ijo.2009.239
44. Cromwell I, Peacock SJ, Mitton C. ‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the literature. BMC Health Serv Res. 2015;15:164. doi:10.1186/s12913-015-0814-3
45. Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care. 2007; 23(3):310-315. doi:10.1017/s026646230707050x
46. Asante AD, Zwi AB. Factors influencing resource allocation decisions and equity in the health system of Ghana. Public Health. 2009; 123(5):371-377. doi:10.1016/j.puhe.2009.02.006
47. Lasry A, Carter MW, Zaric GS. Allocating funds for HIV/AIDS: a descriptive study of KwaDukuza, South Africa. Health Policy Plan. 2011;26(1):33-42. doi:10.1093/heapol/czq022
48. Baltussen R, Stolk E, Chisholm D, Aikins M. Towards a multi-criteria approach for priority setting: an application to Ghana. Health Econ. 2006;15(7):689-696. doi:10.1002/hec.1092
49. Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan. 2007;22(3):178-185. doi:10.1093/heapol/czm010
50. World Health Organization (WHO). Everybody’s Business -- Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework for Action. Geneva: WHO; 2007.
51. Ryan MA. Adherence to clinical practice guidelines. Otolaryngol Head Neck Surg. 2017;157(4):548-550. doi:10.1177/0194599817718822
52. Fischer F, Lange K, Klose K, Greiner W, Kraemer A. Barriers and strategies in guideline implementation-a scoping review. Healthcare (Basel). 2016;4(3). doi:10.3390/healthcare4030036
53. McDonald S, Elliott JH, Green S, Turner T. Towards a new model for producing evidence-based guidelines: a qualitative study of current approaches and opportunities for innovation among Australian guideline developers. F1000Res. 2019;8:956. doi:10.12688/f1000research.19661.1
54. Turner TJ. Developing evidence-based clinical practice guidelines in hospitals in Australia, Indonesia, Malaysia, the Philippines and Thailand: values, requirements and barriers. BMC Health Serv Res. 2009;9:235. doi:10.1186/1472-6963-9-235
55. Florez ID, Morgan RL, Falavigna M, et al. Development of rapid guidelines: 2. A qualitative study with WHO guideline developers. Health Res Policy Syst. 2018;16(1):62. doi:10.1186/s12961-018-0329-6
56. Setty K, Jiménez A, Willetts J, Leifels M, Bartram J. Global water, sanitation and hygiene research priorities and learning challenges under Sustainable Development Goal 6. Dev Policy Rev. 2020; 38(1):64-84. doi:10.1111/dpr.12475
57. Tromp N, Prawiranegara R, Siregar A, et al. Translating international HIV treatment guidelines into local priorities in Indonesia. Trop Med Int Health. 2018;23(3):279-294. doi:10.1111/tmi.13031
58. Heintz E, Lintamo L, Hultcrantz M, et al. Framework for systematic identification of ethical aspects of healthcare technologies: the SBU approach. Int J Technol Assess Health Care. 2015;31(3):124-130. doi:10.1017/s0266462315000264
59. Norheim OF, Baltussen R, Johri M, et al. Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Eff Resour Alloc. 2014;12:18. doi:10.1186/1478-7547-12-18
60. Johnson AP, Sikich NJ, Evans G, et al. Health technology assessment: a comprehensive framework for evidence-based recommendations in Ontario. Int J Technol Assess Health Care. 2009;25(2):141-150. doi:10.1017/s0266462309090199
61. Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6. doi:10.1186/1748-5908-8-6
62. Alonso-Coello P, Schünemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016. doi:10.1136/bmj.i2016
63. Vogel JP, Oxman AD, Glenton C, et al. Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey. Health Res Policy Syst. 2013; 11:19. doi:10.1186/1478-4505-11-19
64. Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are enough? Qual Health Res. 2017;27(4):591-608. doi:10.1177/1049732316665344
65. Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Eff Resour Alloc. 2020;18:8. doi:10.1186/s12962-020-0203-6