evaluation methods – where can predictive risk models help?
DESCRIPTION
Evaluation methods – where can predictive risk models help?. Adam Steventon Nuffield Trust 8 July 2013. The problem with observational studies. Intervention patients. Eligible patients. Source: Steventon et al (2012). Solutions, 1) before-after study. Solutions, 2) regression adjustment. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/1.jpg)
© Nuffield Trust
Evaluation methods – where can predictive risk models help?
Adam Steventon
Nuffield Trust
8 July 2013
![Page 2: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/2.jpg)
© Nuffield Trust
The problem with observational studies
Eligible patients
All patients Intervention patients
n 54,990 556
% aged 85+ 21.6 46.2
Prior emergency admissions 0.5 1.4
Number chronic conditions 1.0 1.1
Predictive risk score 22.3 33.6
Intervention
patients
Source: Steventon et al (2012)
![Page 3: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/3.jpg)
© Nuffield Trust
Solutions, 1) before-after study
![Page 4: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/4.jpg)
© Nuffield Trust
Solutions, 2) regression adjustment
Y = f(age, number of chronic conditions,
prior emergency admissions,
intervention status)
![Page 5: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/5.jpg)
© Nuffield Trust
Eligible patients
Intervention
patients
Matched
controls
All patients Matched controls
Intervention patients
n 54,990 556 556
% aged 85+ 21.6 46.2 46.2
Prior emergency admissions
0.5 1.4 1.4
Number chronic conditions
1.0 1.3 1.1
Predictive risk score 22.3 33.5 33.6
Solutions, 3) Matched controls
Source: Steventon et al (2012)
![Page 6: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/6.jpg)
© Nuffield Trust
How to select matched controls
Propensity score (Rosenbaum and Rubin 1983)
- Predictive risk of receiving the intervention
Prognostic score (Hansen 2008)
- Predictive risk of experiencing the outcome (e.g. emergency
hospitalisation), in the absence of the intervention
Genetic matching (Sekhon and Grieve 2012)
- computer-intensive search algorithm
![Page 7: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/7.jpg)
© Nuffield Trust
Advantages / disadvantages
Disadvantage – only allows for observed variables
But
Matching as ‘data pre-processing’ – reduces dependence of estimated
intervention effects on regression model specification
Intuitive?
Good for routine monitoring – once controls found, data can be updated
![Page 8: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/8.jpg)
© Nuffield Trust
Overcoming regression to the mean using a control group
Start of intervention
![Page 9: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/9.jpg)
© Nuffield Trust
Overcoming regression to the mean using a control group
Start of intervention
![Page 10: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/10.jpg)
© Nuffield Trust
Overcoming regression to the mean using a control group
Start of intervention
![Page 11: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/11.jpg)
© Nuffield Trust
Overcoming regression to the mean using a control group
Start of intervention
![Page 12: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/12.jpg)
© Nuffield Trust
Solutions, 4) regression discontinuityW
inn
ing th
e n
ext
ele
ctio
n
Fraction of votes awarded to Democrats in the previous election
Source: Lee and Lemieux (2009)
![Page 13: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/13.jpg)
© Nuffield Trust
What is being done at the moment?Telehealth studies in Pubmed, 2006-2012
Descriptive
Before after
Dose response
Controlled
All
Number of studies 3 24 1 16 44
Median number of patients in telemonitored group (range)
45(40 to 851)
35 (7 to
17,025)246
102 (19 to
1,767)*
45 (7 to
17,025)*
Endpoints
Mortality 2 - 1 3 6
Hospital use (or costs) 3 6 - 12 21
Clinical (e.g. HbA1c) - 17 - 4 21
Patient reported outcomes (e.g. quality of life)
1 10 - 3 14
Source: Steventon, Krief and Grieve (work in progress)
![Page 14: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/14.jpg)
© Nuffield Trust
References
Lee DS, Lemieux T. Regression discontinuity designs in economics. 2009.
Available from: http://www.nber.org/papers/w14723.pdf?new_window=1
Sekhon JS, Grieve RD. A matching method for improving covariate balance in
cost-effectiveness analyses. Health economics 2012;21:695–714.
Rosenbaum P, Rubin D. The central role of the propensity score in observational
studies for causal effects. Biometrika 1983;70:41–55.
Hansen BB. The prognostic analogue of the propensity score. Biometrika
2008;95:481–8.
Steventon A, Bardsley M, Billings J, Georghiou T, Lewis GH. The role of
matched controls in building an evidence base for hospital-avoidance schemes:
a retrospective evaluation. Health services research 2012;47:1679–98.
![Page 15: Evaluation methods – where can predictive risk models help?](https://reader036.vdocument.in/reader036/viewer/2022081516/56813134550346895d97a560/html5/thumbnails/15.jpg)
© Nuffield Trust8 July 2013
www.nuffieldtrust.org.uk
Sign-up for our newsletter:www.nuffieldtrust.org.uk/newsletter
Follow us on Twitter: http://twitter.com/NuffieldTrust
© Nuffield Trust