Rigour and ethics

IHPSR Presentation 8

www.hpsa-africa.org

@hpsa_africa

www.slideshare.net/hpsa_africa

Introduction to Health Policy and

Systems Research

Four key steps in HPSR

1. Identify research focus

(problem/concern/ opportunity) and

question

2. Design study

3. Ensure quality and rigour

4. Apply ethical principles

HPSR rigour

• Health policies and systems are politically and

socially constructed – issues of focus in HPSR are

complex phenomena

• HPSR demands:

– An active process of questioning and checking during the

inquiry

– A constant process of conceptualising and reconceptualising

– Crafting interpretive judgements

– Researcher reflexivity

‘Researcher as instrument’

• Enquiring mind

• Good listening

• Adaptiveness and flexibility

• Grasp of the issues

• Lack of bias (reflexivity)

Researcher reflexivityRobson, 2002

• Identify your personal issues in relation to topic

• Clarify your value system

• Identify areas of possible role conflict

• Identify gatekeepers and how they will influence

you

• Identify where you are not neutral

Study validity/trustworthiness

Fixed designs

• Are the findings valid?

• Are the findings

statistically

generalisable?

Flexible designs

• Are the findings

plausible?

• Do the findings provide

theoretical insights that

have a sufficient degree

of generality to be

projected to other

contexts? (analytic

generalisability)

Rigour in fixed design

• Validity

– does the study measure what is intended?

– internal validity: cause-effect relationships

• Threats to validity

– bias: selection, measurement, confounding

• Reliability

– participant

– observer

• Generalisability (external validity)

– representative sample

– valid and reliable methods

Rigour in RCTs(randomised control trials)

• Pre-specified study outcomes

• Control groups

• Random allocation to intervention and control

(randomisation)

• Intervention fidelity

• Contamination and co-intervention

• Blinding

• CONSORT guidelines

Challenges to RCTs in HPSR

• Sampling units in health systems research (costs!)

• Control groups not always possible

• Complex adaptive systems

– intervention process not under researcher control

– context specificity

• Improved performance requires combined (complex)

interventions

• Cause-effect relationships not easy to establish

• Often have to construct models of complex systems and human

behaviour based on theoretical or practical assumptions

Strategies to enhance rigour in

fixed HPSR design

• Focus on plausibility rather than probability

• Mixed methods and triangulation

• Sensitivity analyses in modelling

• Drawing on theory

• Statistical techniques

– multi-level analysis

– structural equation modelling

– quantitative modelling of system dynamics (causal

loops)

Rigour in flexible design Robson, 2002

Threats lie in:

– description

– interpretation

– theory (rival explanations)

Bias from:

– reactivity (e.g. Hawthorne effect)

– respondent (e.g. social desirability bias)

– researcher (e.g. preconceived ideas)

Strategies to enhance rigour in

flexible HPSR design

• Use theory (at different stages)

• Use literature and a careful protocol

• Clear exposition of methods

• Multiple sources of data collection

• Triangulation (data, observer, methodological, theory)

• Prolonged involvement

• Negative/’deviant’ case analysis

• Counterfactuals (rival explanations)

• Member checking (respondent validation)

• Peer debriefing/support

• Reflexivity

Case study

Validity in case study work requires

paying attention to the trustworthiness of

the interpretive analysis and the resulting

generalisable claim

Critical case study issues

• Case selection and sampling

• Data collection approach and procedures

– prolonged engagement with cases

• Analytic procedures:

– respondent validation (member checking)

– triangulation across data sets and with theory

– negative case analysis

– peer debriefing and support

• Clear report of methods of data collection and analysis

(audit trail)

Analytic/theoretical

generalisation

• Develop ‘theoretical’ insights’ (generalisable

claims) by:

– building or testing theory and/or

– comparative analysis across multiple cases

• These insights are universal enough to have

relevance in other settings

Four ‘tests’ Yin, 2009 (adapted with Robson, 2002)

Case study tactic Phase of research

Confirmability

(the usual

flexible

strategies)

Literature review, identify key concepts

Multiple sources evidence

Establish chain of evidence

Have key informants review draft (member

checking)

Research design

Data collection

Data collection

Write up

Credibility

(the usual

flexible

strategies)

Do pattern matching

Do explanation building

Do rival explanations

Use logic models

Triangulation

Negative cases

Data analysis

Transferability Use theory in single case studies

Use replication logic in multiple case studies

Research design

Dependability Use case study protocol

Develop case study data base (audit trail)

Data collection

Interpretive analysis

in HPSR

• Remember the question/s you are asking

• Analysis begins during data collection

– looking for patterns and possible explanations

• Always need to reduce and display data

– narratives, examples of broader phenomena,

diagrams, figures, pictures, tables

• Ask questions of the data:

– Is explanation plausible? Can you find

evidence confirming it? Can a finding be

replicated in another data set?

Be reflective!

• Contextualise in analysis

– think about and present relevant features of

the specific social and physical setting,

including historical factors, that support

explanation: identify from theory and in

analysis

Role of theory in ensuring

HPSR rigour?

To support generalisable claims:

• Testing hypotheses generated from theory

with empirical evidence

• Triangulation with theory

• Generating theory as a generalisable claim

Four key steps in HPSR

1. Identify research focus

(problem/concern/ opportunity) and

question

2. Design study

3. Ensure quality and rigour

4. Apply ethical principles

Principles of bio-medical

research ethics

• Are these relevant to HPSR?

How?/Why not?

– Scientific validity and collaborative

partnerships

– Social value and risk benefit ratios

– Informed consent and respect for

participants and communities

– Independent reviewEmanuel et al. 2004

What practices of real world research are

(most) important in HPSR and why? Robson, 2002

• Involving people without consent

• Coercing them to participate

• Withholding information about true nature of research

• Otherwise deceiving participants

• Inducing participants to commit acts diminishing of their self-

esteem

• Violating rights of self-determination

• Exposing participants to physical or mental stress

• Invading privacy

• Withholding benefits from some participants

• Not treating participants fairly or with respect

Examples of HPSR ethical

challenges

• Being informed about or observing unethical

behaviour of providers, managers, community

members

• How to present findings in ways that balance

confidentiality and give voice to those not usually

heard

• How to act in ways that have potential to stimulate

change in (rather than exacerbate) unequal

relationships between people

Other HPSR issues

• Who determines the questions?

• Gaining consent and relative power:

– from policy makers to marginalised groups

• Anonymity and small sample sizes

• Acting ethically towards field workers

• Recognising researchers’ privilege and power

• Generating social value: From substantive relevance

to where? Where does the researcher’s role end?

Copyright

Funding

You are free:

To Share – to copy, distribute and transmit the work

To Remix – to adapt the work

Under the following conditions:

Attribution You must attribute the work in the manner

specified by the author or licensor (but not in any way that

suggests that they endorse you or your use of the work).

Non-commercial You may not use this work for commercial

purposes.

Share Alike If you alter, transform, or build upon this work,

you may distribute the resulting work but only under the same

or similar license to this one.

Other conditionsFor any reuse or distribution, you must make clear to

others the license terms of this work.

Nothing in this license impairs or restricts the authors’

moral rights.

Nothing in this license impairs or restricts the rights of

authors whose work is referenced in this document.

Cited works used in this document must be cited following

usual academic conventions.

Citation of this work must follow normal academic

conventions. Suggested citation:

Introduction to Health Policy and Systems Research,

course presentation, Presentation 8. Copyright

CHEPSAA (Consortium for Health Policy & Systems

Analysis in Africa) 2014, www.hpsa-africa.org

www.slideshare.net/hpsa_africa

This document is an output from a project funded by the European Commission (EC) FP7-Africa (Grant no.

265482). The views expressed are not necessarily those of the EC.

The CHEPSAA partners

University of Dar Es SalaamInstitute of Development Studies

University of the WitwatersrandCentre for Health Policy

University of GhanaSchool of Public Health, Department of Health Policy, Planning and Management

University of LeedsNuffield Centre for International Health and Development

University of Nigeria Enugu Health Policy Research Group & the Department of Health Administration and Management

London School of Hygiene and Tropical MedicineHealth Economics and Systems Analysis Group, Depart of Global Health & Dev.

Great Lakes University of KisumuTropical Institute of Community Health and Development

Karolinska InstitutetHealth Systems and Policy Group, Department of Public Health Sciences

University of Cape TownHealth Policy and Systems Programme, Health Economics Unit

Swiss Tropical and Public Health InstituteHealth Systems Research Group

University of the Western CapeSchool of Public Health