building the evidential quality of evaluations (tony redmond, uni manchester)
TRANSCRIPT
Building the evidential quality of evaluations
ALNAP March 5, 2013
Tony Redmond
Humanitarian vs Development
Hierarchy of Evidence
Most evidence in the humanitarian sphere is in the weakest categories or outside
the pyramid altogether
Expert Opinion
• Consensus Statements Regarding the Multidisciplinary Care of Limb Amputation Patients in Disasters or Humanitarian Emergencies: Report of the 2011 Humanitarian Action Summit Surgical Working Group on Amputations Following Disasters or Conflict
• Delphi Studiens
Gathering the Evidence
bias
• Incentive• Observer• Systematic (non random) error
– Confidence intervals
RCTSibbald B, BMJ 1998;316:201
• the most rigorous way of determining whether a cause-effect relation exists between intervention and outcome and
• for assessing the cost effectiveness of an intervention.
Dangerous Data
• Wrong inference• Randomisation• Open and transparent
HERR
• Medical teams are 200 times as cost effective as UKISAR
• That is they cost less, stay longer, treat more people and save more lives
sampling
Convenience sampling
• Using those who are willing to volunteer, or cases which are presented to you as a sample.
Judgement sampling
• Based on deliberate choice and excludes any random process.
• Normal application is for small samples from a population that is well understood and there is a clear method for picking the sample.
• Is used to provide illustrative examples or case studies.
Simple random sampling
• Ensures every member of the population has an equal chance of selection.
• Produces defensible estimates of the population and sampling error.
• Simple sample design and interpretation.
Cluster sampling
Units in the population can often be found in geographical groups or clusters. A random sample of clusters is taken, then all units within those clusters are examined.• Quicker, easier and cheaper than other forms of random sampling.• Does not require complete population information.• Useful for face-to-face interviews.• Works best when each cluster can be regarded as a microcosm of the
population.• Larger sampling error than other forms of random sampling.• If clusters are not small it can become expensive.• A larger sample size may beneeded to compensate forgreater sampling error.
Why is evidence ignored?
• Evidence influencing policy• Policy influencing evidence• We've always done it this way…• ……and it’s done no harm (regardless of
the evidence)• Maybe the evidence was always there….
Epidemiology/statistics
• Observation• Pattern Recognition• Cause and effect or coincidence• Change of practice
CONFIDENCE INTERVALS (CI)
MEAN VALUE
You have a 95% confidence that the TRUE VALUEshould fall between the CI if the study is repeated multiple times in a similar population
95% CONFIDENCE INTERVAL
assessments
• Early warning• Risk analysis• Trending• Needs assessment
Evidence based medicine
• Clinical trials in the field – in an emergency?
• Multiple disciplines
Evaluation
• What was done?• What was the reason for doing it?• What did they find?
• SO WHAT?
Benefit cost
What you have to evaluateadapted from DFID WASH Evaluation 2011
• Internal validity – was what they did good enough?
• External validity – relevance to you?• Can you implement it ?• Sustainability?• Wider impact?• COST
Is good enough, good enough?
Humanitarian
• Historically draws on concept of compassion, individuality and reform (transformation)
• It is evidence and evaluation that will help us reform and lead us to prevention