Intervention Studies: Principles, Opportunities and Pitfalls

Prof. Dr. phil. Gabriele Meyer

Martin Luther University Halle-Wittenberg, Germany

• 20 English language nursing journals with highest IF (1.221-2.103)

Inclusion: 223 studies from 21 EU-European countries

Results: 34% report on nursing interventions

45% observational studies

39% qualitative

12% experimental

4% randomised controlled trials

Confirmed by: Mantzoukas Int J Nurs Stud 2009; Forbes Int J Nurs Stud 2009; Yarcheski et al. Int J Nurs Stud 2012

LINKING EVIDENCE TO ACTION

• Researchers in nursing should design, undertake, and report fewer descriptive studies and more experimental research into the effectiveness of nursing interventions to ensure a more balanced proportion of intervention and descriptive research in nursing.

• Researchers should structure their studies to explicitly link the development, testing, evaluation, and implementation of nursing interventions in coherent programs of research activity rather than as stand-alone projects.

LINKING EVIDENCE TO ACTION

• Nursing researchers should consider using the UK Medical Research Council’s “Complex Interventions Research Framework” to organize studies that will deliver an increased evidence base for nursing interventions.

• Doctoral education programs for nurses should encourage students to undertake experimental work into the efficacy and effectiveness of nursing interventions.

(Craig et al. 2012; IJNS)

Examples

• Organisation

– Stroke unit

• Health professions

– Guideline implementation

• Defined Groups

– School based programmes for smoking cessation

• Individuals

– Lifestyle intervention in diabetes

Terms used for complex interventions

• Multicomponent

• Multifaceted

• Multifactorial

Introduction of such a system did not significantly reduce the incidence of our study outcomes. Possible explanations for our findings are that the MET system is an ineffective intervention; the MET is potentially effective but was inadequately implemented in our study; we studied the wrong outcomes; control hospitals were contaminated as a result of being in the study; the hospitals we studied were unrepresentative; or our study did not have adequate statistical power to detect important treatment effects.

Interventional study designs

• Also called „experimental study designs“

• Where the researcher intervenes at some point throughout the study

• To evaluate study questions related to either treatment (prophylactic agents, treatments, surgical approaches, diagnostic tests) or prevention (protective equipment, engineering controls, management, policy or any element that should be evaluated as to a potential cause of disease or injury)

• Strongest design: randomised controlled trial

• Other: Pre-post study design, non-randomised controlled trials, quasi-experiments

Most RCTs aim to determine whether one intervention is superior to another. Equivalence trials aim to determine whether one (typically new) intervention is therapeutically similar to another, usually an existing treatment. A non-inferiority trial seeks to determine whether a new treatment is no worse than a reference treatment. Proof of exact equality is impossible, therefore, a pre-stated margin of non-inferiority for the treatment effect in a primary patient outcome is defined.

Heal et al. BMJ 2006; 332

Explanatory trial Pragmatic trial

Benefit of a treatment under ideal conditions (efficacy)

Benefit of a treatment in routine clinical practice (effectiveness)

Homogeneous population as possible; aims primarily to further scientific knowledge

Variations between patients as in real clinical practice; aims to inform choices between treatments

Standardised intervention

Protocol based administration of different treatment to IG

Intermediate outcomes often used Full range of health gains

Bias

„Any process at any stage of inference

tending to produce results that differ

systematically from true values.“

(Murphy 1976)

Internal validity

Threated by

Selection bias

Performance bias

Detection bias

Attrition bias

Blinding …

• minimizes • cointervention bias (differential use of cointerventions), • attrition bias (differential patient dropout) • response bias (differential reporting of symptoms)

• ensures a similar degree of placebo effects in compared groups • protects against observer/detection/ascertainment bias

Trials that are not double blind exaggerate treatment effects (odds ratios) by 13%, and by 23% when outcomes are subjective (Savović J et al. Ann Intern Med 2012)

Schulz KF & Grimes DA Lancet 2002

Conclusions:

The risk of selection bias could not be ascertained for most trials due to poor reporting. Many trials which did provide details on the randomisation procedure were at risk of selection bias due to a poorly chosen randomisation methods. Techniques to reduce the risk of selection bias should be more widely implemented.

n=475 RCTs with mITT

Modified ITT (Anraha et al. BMJ 2010)

• Treatment: „mITT consisted of patients who received at least six doses of study drug.“

• Baseline assesment: „mITT included patients with at least one baseline observation.“

• Target condition: „The mITT population consisted of those patients who were randomly assigned to study treatment minus those who were not H pylori positive.“

Modified ITT (Anraha et al. BMJ 2010)

• Post-baseline assessment: „mITT includes all randomized patients who have . . . at least one post-baseline measurement.”

• Follow-up: “All participants who completed follow-up were analyzed as a part of the group to which they were randomized. This was not a strict intent-to-treat analysis as some study participants were lost to follow-up.”

Time for a short exercise

Stages of waste in the production and reporting of research evidence

• Regulation bodies are not in charge (e.g. FDA, EMA)

• Prospective regulation voluntary; recommendations by International Committee of Medical Journal Editors, WHO Statement on Public Disclosure of Clinical Trial Results, World Medical Association’s Declaration of Helsinki, Reporting Guidelines (e.g. CONSORT, SPIRIT), AllTrials campaign, United Nations

non-regulated RCTs less often registered as regulated RCTs, although 40% of all published RCTs

• Survey, 220 RCTs from clinical geriatric journals

• Published RCT registered in a publicly accessible register?

• Prospectively registered before participants‘ recruitment?

• Agreement between registration, published study protocol and published report of results?

• 140/220 RCTs registered.

• Registration in only 15% of RCTs prospectively.

• Half of RCTs report on results of registered outcomes or at least refer to registered outcomes.

• Time of enrolment of participants remains unclear in one third of registered trials and half of non-registered studies.

•

Sample: 133 trials with 137 interventions Outcome instrument: 8-item checklist Results: 53/137 (39%) were adequately described; after contact to authors: n=81 (59%) according to 63 answers of 88 contacted authors.

Transparency in reporting of complex interventions (Möhler et al. 2011)

No studies (out of 5)

Intervention …

Theoretical basis 3

Piloting 0

Costs 0

Education…

Description of participants 3

Curriculum 2

Access to education material 0

Process evaluation 0

Description standard care 0

Implementation of study protocol 0

Staff turnover 2

CReDECI

• Designed to integrate relevant aspects of the complete research process of development and evaluation of a complex intervention

• 13 Items in 3 sections:

– development (n=4 items)

– feasibility and piloting (n=1)

– introduction of the intervention and evaluation (n=8)

• Not focussing on a particular study design

What are cluster randomised controlled trials?

cRTCs are experiments in which (interacting) social units rather than individuals are randomly allocated to study groups: communities, schools, families, hospitals, nursing homes …

Reasons for cRCT?

• How many levels are involved?

– General practice - patient

– Nursing home - ward - resident

– Single person (with limbs, teeth, eyes)

Trials with one cluster per arm?

• Minimum number of clusters per arm to ensure a valid analysis should be at least four (Hayes & Moulton, 2008)

Challenges of cRCT

• Outcome for each participant cannot be assumed to be independent of that for any other participant since those within the same cluster are more likely to have similar outcomes (clustering effect).

• The reduction in effective sample size depends on average cluster size and the degree of correlation within clusters, ρ, also known as the intracluster (or intraclass) correlation coefficient (ICC).

Challenges of cRCT

• Standard sample size formulas will lead to underpowered studies larger sample sizes are required

• Cluster adjustment has to be taken into account for statistical analysis to avoid unit of analysis bias

Sample size calculation cRCT - example

• Incidence: 60 hip fractures/ 1000 residents/ year

• Effect size: Relative risk reduction 50%

• Observation time: 18 months

• β = 80%, = 5%

• For cluster design: Number and size of clusters (eligible for recruitment and participation, patients/residents per cluster)

Participant randomisation

n = 384

Cluster randomisation

Design effect = 1+(m-1)p=1.24

n = 384 x 1.24 = 477

Sample size calculation cRCT - example

Unfeasable trial

Provided an ICCC of 0.04 and a mean cluster size of 200, more than 900 clusters with a total of 186,000 patients were required to demonstrate a relative risk reduction of 33%.

A football stadium holds about 40 000 fans. Our trial requires 4.7 times the number of participants shown.

The problem of cluster baseline imbalance

• Many cRCTs do not have enough clusters to create a reasonable expectation for cluster-level balance.

• Ivers et al. (BMJ 2011); n=300 cRCTs, published between 2000 and 2008

– Median number of clusters 21

– 25% had fewer than 12 clusters

– 14% had less than 4 clusters

• Different techniques of restricted randomisation have been discussed to reduce baseline imbalance: stratification, pair-wise matching, etc.

Example: Köpke et al. JAMA 2012

• Computer-generated randomization lists were used for allocation of clusters in blocks of 4, 6, and 8 nursing homes. Randomization was stratified by region, ie, Hamburg and Witten.

Example: Köpke et al. JAMA 2012

Nursing homes were eligible if they had a self-reported rate of at least 20% of residents with physical restraints, assessed by a short questionnaire completed by the head nurses.

Example: Köpke et al. JAMA 2012

Other challenges

• Poor design of cluster trials risks bias in selection of participants post-randomisation recruitment bias

– Clusters are randomly allocated and participants are consecutively recruited

– Risk of biased application of eligibility and recruitment criteria selective inclusion

• How to solve this problem?

Solution

• Ideally participants should be identified before the cluster is randomised

• When this is not possible recruitment should be by someone masked to the cluster allocation

How could a user of a publication on a cRCT detect a possible recruitment bias?

• More people recruited when in intervention group

• Differential refusal to consent rate between the conditions

• Baseline characteristics’ imbalance

Attrition bias

Why worse in cRCTs than in iRCTs?

Cluster attrition (data from Puffer et al. BMJ 2003)

Study No of clusters

No of participants

Description Clusters lost after randomisation

Flottorp 2002 142 12 369 To improve general practice management of sore throat and urinary tract infections

22

King 2002 116 410 Behavioural therapy to treat patients with depression

32

Morrison 2001 221 689 Infertility guidelines for general practitioners

7

Olivarius 2001 311 1470 Structured personal care of type 2 diabetes mellitus

10

How to avoid attrition bias or reduce ist impact?

1 nursing home in the control group denied to collect data on accidental falls (secondary outcome: fall events and hip protector use during fall) after randomisation. Primary outcome: hip fracture

Other challenge of cRCTs: Blinding

• Participant blindness (nurses, physicans, patients/residents) to intervention assignment (Cave: proven criterion of internal validity) is difficult or impossible to maintain in cRCTs.

• What could be done?

• Outcome assessors should be blinded.

Please discuss with your neighbour how blinding of outcome assessor - could be guaranteed - controlled for whether successfully conducted Example: Köpke et al. JAMA 2012

Intervention component Rationale

Declaration Proven strategy in own previous studies

Structured 90-minute information program for all nursing staff

Cochrane review; Theory of planned behavior

External structured 1-day intensive training workshop for nominated key nurses

Proven strategy in previous studies

Structured support for key nurses Proven strategy in previous studies

Practice guideline Guideline implementation recommendations

Printed supportive material for nurses, relatives, legal guardians

Guideline implementation recommendations Publications on involvement of significant others

Other supportive material Guideline implementation recommendations

To check for effective blinding, during the second measurement point at 3 months, external researchers were asked about their perception of the visited cluster’s group allocation using a short questionnaire. If raters visited the cluster at 2 or all 3 time points, the questionnaire was completed at the last visit.

During 69 visits at the 3-month follow-up, 37 ratings (53.6%; 95% CI, 41.2%-65.7%) were correct in identifying clusters as an intervention group or control group cluster.

Example: Köpke et al. JAMA 2012

Stepped wedge design

Randomisation in terms of the period for receipt of the intervention type of cluster crossover trial if the unit of randomisation is a cluster.

Thank you very much for your attention!