Evaluation of effects

Sandra Zwakhalen

June 2018, EDCNS

Health Services Research, Maastricht University

• Research program Ageing and Long-Term care

• Disciplines (>100 employees)- Physicians, Health Economy, Health Policy, Nursing Science,

Care Thechnology, Social Gerontology, Health Technology Assessment etc….

Ageing and Long-Term Careaims to:

1. Develop of knowledge on psychosocial and environmental determinants of health problems, need of care, social participation, independency and quality of life

2. Evaluate of innovative health care programs and interventions for older persons to slow down processes of disablement

3. Implement health care programs and interventions

Major themes

• Effectiveness of care (health care problems), e.g.:- Pain in dementia, Incontinence, Malnutrition, Falls and

injuries, Fear of falling, Mobility, Heart failure, Pressure ulcers, Physical restraints, Disability

• Organisation of care, e.g.:- Patient centered care, Quality of care, Care for frail older

people, Redesigning health care, Small scale nursing homes, Geriatric rehabilitation

Living lab, how

Staff working at both; university and LTC organization

Interdisciplinary; nursing, medicine, psychology…. etc.

Assessing effectiveness

• Research designs• Feasibility to apply designs in

evaluation of complex interventions• Bias• Examples en group work

Assessing effectiveness …. (10 minutes brainstorm ABC exercise to start)

• Thinking about effectiveness, - Brainstorm ABC………

• How can we assess the effectiveness of an intervention?- What research design is viewed as the

‘gold standard’?- Experimental – randomized controlled trials

(Quasi)-Experimental Designs

RANDOMIZED DESIGNS1. Individual-patient randomized controlled trials2. Cluster randomized trials3. Factorial design4. Repeated measures design

SEMI or NON-RANDOMIZED DESIGNS e.g1. Preference trial2. Stepped Wedge Design (Wait-list design)3. Other designs e.g. Time Series Design, Before and After

Design (case control)

Randomized Controlled Trial (RCT)

1. Randomization2. Control3. Manipulation

Key idea of a randomized controlled trial is that we compare groups of participants who differ only with respect to their intervention.

RCT• Example: Effectiveness of morphine following

adenotonsillectomies (T&A) in children

- Randomization- Children admitted for T&A

- Two groups- Experimental/intervention group- Control group

- Intervention- Paracetamol + morphine

In sum:1. Paracetamol + morphine2. Paracetamol + placebo

Why experimental design?

• Internal validity- The extent to which the observed effects

are true for the people in a study- The degree to which a result (of a

measurement or study) is likely to be true and free of bias (systematic errors)

SELECTION BIAS(systematic differences in comparison groups)

PERFORMANCE BIAS(systematic differences in care provided apart from intervention under evaluation)

ATTRITION BIAS(systematic differences in withdrawals)

DETECTION BIAS(systematic differences in outcome assessment)

SOURCES OF BIAS

Target Population (baseline)

Allocation

Intervention Group Control Group

Exposed to intervention Not exposed

Follow up Follow up

OUTCOMES OUTCOMES

Critisism experimental design

• External validity- The extent to which the effects observed in a

study truly reflect what can be expected in a target population beyond the people included in the study

- Generalizability

• Explanatory versus pragmatical RCT’s- Efficacy (does it work in most favourable

circumstances?) versus effectiveness (does it work in everyday practice?)

Why are RCT’s not always suitable

Why are RCT’s not always suitable

• Experimentation may be unnecessary- Efficacy of parachutes, anaesthesia for surgical

operations

• Experimentation may be inappropriate- To measure infrequent adverse events, to prevent rare

events, or measure outcomes in far future

• Experimentation may be impossible- Refusal of clinicians to participate or ethical objections

• Experimentation may be inadequate- Patients in trial are atypical, professionals are

unrepresentative or treatment is atypical

Cluster RCT

• Randomization not on individual level, but on organizational level- E.g., midwife, hospital wards, nursing

homes

• Data collection on individual level- E.g., age, gender, pain, quality of life

Disadvantages?

• Complex (multi-level) analyses, less power

• Complex organisation, time consuming • Large sample needed• Contamination

Factorial design

In which two active treatments A and B, are simultaneously compared to each other and to a control.

Participants are randomized into four groups, who receive either A only, B only, both A and B, or neither.

Factorial design

• Effectiveness morphine and nurses’ systematic pain assessments (SPA)

1. Paracetamol + morphine + SPA2. Paracetamol + placebo+ SPA3. Paracetamol + morphine + no SPA4. Paracetamol + placebo+ no SPA

Repeated measurement designor within subject

• Instead of having one score per subjectmultiple scores are generated for eachcase

• Every subject is his own control

(Quasi)-Experimental Designs

RANDOMIZED DESIGNS1. Individual-patient randomized controlled trials2. Cluster randomized trials3. Factorial design4. Repeated measures design

SEMI or NON-RANDOMIZED DESIGNS e.g1. Preference trial2. Stepped Wedge Design (Wait-list design)3. Other designs e.g. Time Series Design, Before and After

Design (case control)

Preference trial

AssessOutcomes

Receive A

Prefer A

AssessOutcomes

Receive A

AssessOutcomes

Receive B

IndifferentRandomised

AssessOutcomes

Receive B

Prefer B

Stepped Wedge Design

Group task

• Use an example of an intervention study of one of participants.

• CreateResearch question(s)……

(Alternative) design: …..Define participants:……Sampling method: …..Methods of data collection: …..Outcomes and meaures:

Advantages and disadvantages (think of possible bias):…….

Reporting statements

CONSORT for randomised controlled trials STARD for diagnostic accuracy studies STROBE for observational studies PRISMA for systematic reviews of trialsMOOSE for meta-analyses of observational studies

EQUATOR networkhttp://www.equator-network.org/library/translations-of-

reporting-guidelines/

General conclusion

• Experimental designs as the ‘golden standard’

• Not always feasible to use for assessing effectiveness of complex interventions

• Look for the best alternative- Explanatory versus pragmatic trials- Non randomized designs controlling for attrition,

performance and detection bias

• Complex nursing interventions can be evaluated adequately

Thank you for your attention!

www.maastrichtuniversity/hsr

s.zwakhalen@maastrichtuniversity.nl

Partners: