USING QUALITATIVE RESEARCH FINDINGS TO INFORM QUESTION DEVELOPMENT IN SYSTEMATIC REVIEWS

K U Leuven Systematic Review Workshop 4-6 June 2012

Janet Harris

Cochrane Qualitative Research Methods Group

Session aims• To demonstrate how and when qualitative research

findings can be used to inform the development of questions for reviews of effectiveness

• To identify the issues and challenges in formulating different types of qualitative questions for qualitative evidence synthesis

Developing questions for systematic reviews: When should qualitative research be considered?

• Qualitative research should be considered if your review question is about a complex intervention.

What is a complex intervention?• Complex interventions in health care, whether therapeutic

or preventative, comprise a number of separate elements which seem essential to the proper functioning of the interventions although the 'active ingredient' of the intervention that is effective is difficult to specify… Complex interventions are built up from a number of components, which may act both independently and interdependently. The components usually include behaviors, parameters of behaviors (e.g. frequency, timing), and methods of organizing and delivering those behaviors (e.g. type(s) of practitioner, setting and location).’

• Medical Research Council: A framework for development and evaluation of complex interventions to improve health. London: Medical Research Council; 2000.

Most effectiveness questions identify a limited number of causes and effects

Refine definition of

problem

Identify possible

causes and effects

Identify problem

Test solutions

Margerunm-Leys http://www-personal.umich.edu/~jmargeru/conceptmap/types.htm

The linear cause and effect model is reproduced in the effectiveness review

• Effectiveness reviews are conducted when • a body of research has accumulated in a particular field, and

findings need to be aggregated • there is uncertainty about a common strategy or treatment• there is uncertainty about relative effectiveness of two or more

commonly offered interventions so a comparison is needed

• The first step in developing the review protocol is a scoping search to determine how much literature exists and what type of research has been conducted

Scoping searches• A scoping search for a systematic review “should typically

search for existing reviews and major trials and other studies. Results from these searches can refocus or focus the review.... This process can involve several iterations. Scoping searches are also used to estimate the size of the literature and by extension the cost of the review.”

• Centre for Reviews and Dissemination http://www.york.ac.uk/inst/crd/revsint.htm

Limiting your review to existing research is sometimes risky

• Published research reflects• The predominant interest of the funder – the funder’s definition of

the problem, the cause and the effect• Responses from researchers to the requirements of the funder• Researchers’ and funders’ formulation of the problem may be

limited to a particular perspective, neglecting factors in the surrounding environment or characteristics of the participants that can affect outcomes

• The research question will also be bounded by the particular discipline of the researchers. Many health problems, for example, are investigated within the confines of a randomised controlled trial conducted under artificial conditions. As a result, it is difficult to successfully transfer the trial intervention to other contexts

Problems with replication• Successful replication of the results of a trial depends on

the amount of contextual information that was captured during the original trial

• In order to promote transferability, we need to know• Important characteristics of participants• Important characteristics of the setting• Important characteristics of the people who delivered the

intervention

• If these characteristics are similar to our situation, then we can be reasonably confident that we can obtain a similar effect

Collecting adequate data to explain the effects of an RCT

• Some studies collect qualitative information either within the study. This is called a ‘process evaluation’

• In other cases, qualitative research is conducted alongside an RCT

• When looking for qualitative research to inform a review question, always look for this data first• It is directly related to the trials that are going to be

included in your review

Qualitative research that enhances a trial

Qualitative data may be collected • Within the

trial• In a Parallel

study • In a Similar

study

Enhancing question explores issues of patient acceptability, the process of implementing the intervention, and patient-defined outcomes

Enhancing data: Issues and challenges

• Adequate data within the study about PICO characteristics?

• Adequate data in embedded qualitative studies?• Adequate data in parallel sibling studies?• Can ‘similar’ qualitative studies be found?• What does ‘similar’ mean – across embedded, sibling,

and independently conducted studies? • Comparability of data - how to synthesize data across

studies?

What if there aren’t any process evaluations or parallel qualitative studies?

• We’ll use an example to illustrate how this can be done• For people who suffer from low back pain, are back

schools effective in reducing pain, improving functioning, and promoting return to work?• A search for enhancing data showed that there were no qualitative

studies on patients’ experiences of participating in back schools

• So, a qualitative scoping search, in areas related to the phenomenon of interest, needs to be conducted to inform development of the question

Back pain schools for non-specific back pain (Heymanns et al, 2010)

• This Cochrane review was unable to perform a meta analysis due to the heterogeneity of the studies included in the review.

• ‘A back school was defined as consisting of an educational and skills acquisition program, including exercises, in which all lessons were given to groups of patients and supervised by a paramedical therapist or medical specialist. Additional interventions were allowed.’

• What are the possible sources of heterogeneity in this definition?

What do we know about possible sources of heterogeneity that could influence the results?

• The descriptions of participating patients and the content of interventions was ‘thin’ or non-existent

• ‘It is essential to be not only informed about effectiveness, but also about • the characteristics of the included patients and• the content of the programs, to determine the clinical relevance of

the studies

• The majority of RCTs did not score sufficiently on the questions, especially Questions 2 and 4, which described the intervention and treatment settings and the clinically relevant effect size.’

Problems with poor reporting

• ‘RCTs either reported briefly about the content of the intervention or failed to report essential information about the type, intensity or performance of the exercises. ‘

• If the components of the intervention are not described, then we cannot know which components were responsible for the effect

How important are the variations

• ‘Also apparent were the widely variations in the content and components of the interventions. This may explain the differences in interpretation of the items between the two authors, reflected by the disagreement score of 34%

• Can variations be statistically controlled?• Or should heterogeneity be dealt with by ‘lumping and

splitting’?• These questions are especially important when dealing

with complex interventions

How do we define important components of a complex intervention?

• Qualitative research can help to identify what is important in terms of• Implementing the intervention• Practitioners attitudes toward the condition and the treatment or

prevention strategy• Patients’ attitudes toward the intervention and experiences of the

condition

Effectiveness question: In adults with Low Back Pain does provision of education via Back Schools reduce pain, improve functioning and promote return to work?

P: Adults with back pain (BP)

O: Reduce pain, improve functioning,

return to work

I: Back School containing

Information and exercises to

promote self-management

Assumed solution:

Patients need written and/or verbal

education and support delivered by

a health professional

Problem and solution (as defined by

the researchers)

Explore assumptions with informing questions

S: Settings: in work; outside work

I: Receive information on

managing LBP from doctor; from

physiotherapist; from community health worker

P: Adults with LBPInforming questions

(Qualitative questions)

Relevant characteristics, for example literacy

Consistency of implementation;

peer support

Who delivers; how delivered;

credibility and relevance to everyday life

What are adults’ experiences of receiving information on how to manage Low Back Pain in different settings, from different providers?

A qualitative informing question

Back pain patients’ experiences of receiving education for self management

Patients’ experiences of receiving back pain information in a physiotherapy-led back school

‘Before, I had pain so I would lie down. Then I had more pain. But now after the course, when I have pain, I no longer lie down. I see that I must continue to go on moving and the pain will be gone. if I lie down the pain will comeback. Before I was ill, now I am not.’  

‘It is important to be taken seriously.’

Here in the physiotherapy clinic, you can bring up all

sorts of weird things, thoughts and

theories because there’s no time limit’.

‘My husband wasn’t believed. [at the

hospital].’

Quotes from patients with chronic pain

‘The doctor gave me some medication and told me to ‘pop along’ to the gym. Well, telling me what to do doesn’t mean I can just go and do it!’

‘I’ve been stuck inside for 2 years.’

‘I thought I was disabled.’

‘[The Health Trainer] didn’t tell me what to do. I thought about what I’d like to be able to do, and she helped me figure out how to do it.’

Informing questions may identify theories

• Theories are explanations for a phenomenon or an event e.g. Why something happens in relation to another event

• They can be grand theories, they can be middle range theories, or they can be single-case theories

• Middle range theory goes beyond individual studies, providing a cross cutting explanation for patterns that are seen across settings and different types of people

• Whenever possible, interventions testing effectiveness should be based on theory

Emerging theory from qualitative research

• Sender: People need to receive health information from ‘someone like them’ e.g. a peer, someone who understands their situation and has something in common with them

• Message: People need an opportunity to discuss whether the health information applies to them. Time to reflect and opportunities to discuss are critical.

• Receiver: People need to accept that their condition is chronic, understand that a passive approach to treatment will not be effective, and be motivated to actively manage their own condition (Michie, 2008)

Can theories from qualitative research be used to inform the question?

• There may be ‘essential ingredients’ that need to be included in interventions• For example, the search on back pain education identified an

issue with appropriateness and relevance of information

• Informing questions need to produce convincing evidence about these ingredients to justify changing an effectiveness question.

• How much evidence is needed to change the original effectiveness question?

Revising the effectiveness question: An example

• In adults with Back Pain does information plus peer support reduce pain, improve functioning and promote return to work?• P = adults with Low Back Pain • I = information explained using peer support • C= information delivered by health professional• O = reduce pain, improve functioning and promote

return to work

Additional ingredients that theoretically increase relevance

Patient-defined outcomes?• Most of the research defines outcomes as reduced pain,

improved functioning and promoting return to work• Qualitative research with patients who have chronic pain

indicates that they define important outcomes as• Better able to manage pain by recognising onset and timely use of

medication – patients note that this is not the same as reducing pain

• Ability to manage some of the everyday things that I did before – this is not the same as improved physical functioning

• Able to participate in social activities, volunteer activities• Able to make hob adjustments – this is different from returning to

work

• The currently used biopsychosocial outcomes may not be measuring what is important

If you use a refined, theory-based question, what will you find?

• Is the quality of reporting for effectiveness studies adequate?• For example, do the interventions describe how information is

explained by the health professionals? • Is the amount of time for the explanation documented? • Does the comparison describe the type of peer support offered?

• Although there is a move toward requiring theory-based interventions in RCTs, this is a relatively new phenomenon

• Qualitative evidence can indicate how a question should be refined, but if effectiveness studies do not report the relevant information then the end product could be an empty review

Decisions about when to use qualitative research to refine a review question

• Weight of qualitative evidence• Qualitative evidence synthesis?• Large number of qualitative studies?• Convergence of theory?

• Cross-paradigm evidence• Qualitative research in more than one discipline produces

complimentary findings• In the Back Pain example, qualitative research in health psychology,

chronic pain, peer support, and health literacy are all finding similar factors that promote self management

• Quality of reporting in the effectiveness literature• Do effectiveness studies document any of the characteristics that

qualitative research deemed important?

Using qualitative data to inform review questions: Summary of the process

• Search for enhancing studies – qualitative studies conducted within or alongside trials

• Search for ‘similar studies’ that are directly relevant to explaining the intervention

• If these studies don not exist, or the evidence is thin, consider searching on phenomena related to the components of the complex intervention

• If there is qualitative data on various components, what is the weight of the qualitative evidence?

• Can the qualitative evidence be used to further refine the PICO?

References • Daykin AR & Richardson B (2004) Physiotherapists’ pain beliefs

and their influence on the management of patients with chronic low back pain. Spine 29: 783-795.

• HeymansMW, van TulderMW, Esmail R, Bombardier C, Koes BW. Back schools for non-specific low-back pain.. Cochrane

• Database of Systematic Reviews 2004, Issue 4. Art. No.: CD000261. DOI: 10.1002/14651858.CD000261.pub2.

• Medical Research Council: A framework for development and evaluation of complex interventions to improve health. London: Medical Research Council; 2000.

• Michie S et al (2008) • Walker J, Holloway I, Sofaer B. (1999) In the system: the lived

experience of chronic back pain from the perspectives of those seeking help from pain clinics. Pain 80, 621.