question-led systematic reviews: implications for searching methods festival 2006 sandy oliver...

Question-led systematic reviews: implications for

searchingMethods Festival 2006

Sandy Oliver

Institute of Education, University of London

Question-led approach• Starts with the nature of the research questions rather

than the mechanisms for finding the answers. • artificial boundaries between ‘qualitative’ and

‘quantitative’ research methods;• potential for bias in asking questions as well as finding

answers. • identify the most pressing conceptual and technological

issues and what aspects of MRS are common across review questions and which are question specific.

• resonates with the way in which users tend to view research. They consult research evidence in order to inform decisions relating to their policy or practice.

The proposal

• Develop a typology of research questions for synthesis

• Further develop a question-driven Methods for Research Synthesis framework for all types of review questions, specifying the generic issues and issues specific to types of review question and research method.

The presentation• Three approaches to exploring question-

led review methodology– a) Existing typologies of questions– b) Analysis of Social Science questions– c) Questions and methods of diverse types of

reviews

• Model for question-led syntheses• Implications for searching

Analysis of social science

research questions

Analysis of diverse

systematic reviews

(a)12 existing typologies

of questions

Questions?

What are the characteristics of X (and Y etc)?

Is X related to Y (and Z etc)?

How and why are X and Y (and Z etc) related? etc

What types of questions?

(b) Analysis of social science

research questions

Analysis of diverse

systematic reviews

(a) 12 existing typologies

of questions

Questions?




Elements of different questions: What? When? How much? Etc.

Asked of single, multiple and related items.

In order to generate, explore, test knowledge.

What types of questions?


research questions

(c) Analysis of diverse

systematic reviews


of questions

Questions?







Spectrum of quantitative, qualitative and mixed methods

What types of questions


research questions

(c) Analysis of diverse

systematic reviews


of questions

Question-led synthesis methods







Spectrum of quantitative, qualitative and mixed methods

Question-led approach

(a) Typologies of questions

Education, Coe (2004)

Tabulating common and unique types of questions

Descriptive: What exactly is X? What are the characteristics of X?

Historical: what happened at X?

Evaluative: How good is X? What is its impact?

Philosophical: what is the justification/ value of X?

Theoretical: what is the relationship between X and Y?

Causal: what are the effects/ results/ benefits of X?

Emancipatory: how can X be empowered?

Ameliorative: How can X be improved?


Public health, Petticrew and Roberts (2005)


Salience: does it matter?

Process of service delivery: how does it work?

Acceptability: Will people be willing to or want to take up the service offered?

Appropriateness: Is this the right service for these people?

Satisfaction: Are users, providers, and other stakeholders satisfied with the service?

Effectiveness: Does this work? Does doing this work better than doing that?

Safety: will it do more harm than good?


Diffusion, dissemination,& sustainability of innovations in health services delivery and organisation, Greenhalgh, Robert & Bate (2004)


What are the beliefs, perceptions, experiences of group G?

What is the nature of process P and is it transferable to context Q?

Do attributes A, B, C etc account for event D?

Does intervention X produce pre-defined outcome Y (and how large is the effect)?

Cost effectiveness: Is it worth buying this service?

What research has been done into complex field F?


Sociology, Lofland & Lofland (1995)


What are the topic’s types? What is this thing? What type of some already identified unit or aspect is it?

What are the topic’s frequencies?What are the topic’s magnitudes?What are the topic’s structures?What are the topic’s processes?What are the topic’s causes?What are the topic’s consequences? What is human agency?

What is the situation being dealt with?What strategies are being employed in dealing with that situation?


Johnston, 2003

Teaching and Learning, University College Dublin

Education and psychology, University of Utah (2004)

Education, Coe (2004)

Community Nursing, Meadows (2003)

Statistical methods, Morgan and Griego

Public health, Petticrew and Roberts (2005)

Campbell Collaboration: education, crime and justice, and social welfare, Cottingham et al (2006)

Sociology, Lofland & Lofland (1995)

Methods of inquiry syllabus, Rutgers University

Diffusion, dissemination,& sustainability of innovations in health services delivery and organisation, Greenhalgh, Robert & Bate (2004)

News research (journalism), University of Canberra (2004)


(a) Analysis of existing question typologies

What is X? (and Y etc)?

How many? How much?

How similar or different are X and Y (and z etc)?

How do X and Y fit together structurally?

How do X and Y fit together procedurally?

How do X and Y influence each other?

What are people’s situations and strategies?

What do we know?

And, how do we know it?

(b) Analysis of social science questions

70 articles spanning 16 social science categories used to:

1) Collect sample of social science questions

2) Develop typology of research questions

(b) Source of social science questions

ESRC Social Science CategoryNumber of Journal

ArticlesArea Studies and Development Studies 5

Economic and Social History 5

Economics 5

Education 5

Human Geography 5

Interdisciplinary studies of Science, Technology and Innovation 4

Linguistics 4

Management and Business Studies 4

Planning, Environmental Studies & Housing Studies 4

Political Science, International Studies and International Relations 4

Psychology & Cognitive Science 4

Social Anthropology 4

Social Policy, Social Work and Health Studies 4

Socio-legal Studies and Criminology 5

Sociology 4

Statistics, Research Methods & Computing 4

(b) Social Science Questions

RESEARCH ITEMS: e.g. variables, concepts

Single item Multiple items Related items

RESEARCH QUESTIONS

What is it? How much? How many?

How different? How related?

PURPOSE:

To Generate Explore Test KNOWLEDGE

(c) Analysis of diverse reviews• Purposive sample of diverse reviews

– EPPI-reviews• process, impact, agency and meaning

– Dixon-Woods• process, structure, impact, agency and meaning

– WWW• cause

• Compared questions and methods– Eligible studies, sources of studies, search

methods

Question led research activity

Questions Research activities

What is this/ are these? Describe: characterise, categorise, typologies

How many/ much? Measure: frequency and magnitude

How similar or different? Compare: descriptions and measures

How do they fit together structurally?

Analyse: structures

How do they fit together procedurally?

Analyse: processes

How do they influence each other?

Analyse: influences


Analyse: agency and meaning

What do we know? Synthesise: studies

And how do we know it? Study: methodology

Questions, research activity and items


What is this/ these items? Describe: characterise, categorise, typologies



How do they fit together structurally? Analyse: structures

How do they fit together procedurally? Analyse: processes

How do they influence each other? Analyse: influences


Analyse: agency

Research Items

Single item

Related Items

Multiple Items



Questions, purpose, researchactivity and items


What is this/ these items? Describe: characterise, categorise, typologies



How do they fit together structurally? Analyse: structures

How do they fit together procedurally? Analyse: processes

How do they influence each other? Analyse: influences


Analyse: agency

Research Items

Single item

Related Items

Multiple Items



Purpose: ToTest

Explore

Generate

Theory

Implications for searchingQuestions

Process/ structure

Influence: Cause

Influence: Impact

Agency

Implications for searchingQuestions Eligible

studiesStudy sources Search

method

Process/ structure

Influence: Cause

Influence: Impact

Agency



method

Process/ structure

Influence: Cause

Influence: Impact

Agency ‘Qualitative’ design

Academic literature Electronic, contacts & snowballing



method

Process/ structure

Influence: Cause

Influence: Impact

Quantitative design

Academic literature Exhaustive: Electronic, contacts & snowballing

Agency Qualitative design Academic literature Electronic, contacts & snowballing



method

Process/ structure

Influence: Cause

Quantitative design


Influence: Impact

Quantitative design





method

Process/ structure

Mixed design Public services, academic literature

Contacts, electronic & snowballing

Influence: Cause

Quantitative design


Influence: Impact

Quantitative design



Conclusions• Reviews addressing different questions (impact,

cause, process, agency) have sought different types of studies

• Reviews testing theory have employed exhaustive searches to avoid a biased answer

• Some reviews generating or exploring theory also use exhaustive searching

• Does exhaustive searching benefit reviews generating or exploring theory?

question-led systematic reviews: implications for searching methods festival 2006 sandy oliver...

Documents

characteristics of x

types of review questions

intervention x

typology of research

topics types

justification value

approach slide

related items