1
Presenter: Paule Poulin, PhDAuthors: P. Poulin, L. Austen et al.,
Health Technology & InnovationDepartment of Surgery
Calgary Research Portfolio
Multiple CriteriaHTA Into Practice
Local HTA Program
CADTH Symposium April 3-5, 2011Vancouver, BC
InnovationCREATE
OR DIE!
2
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
3
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
4
Local decision-makers need to consider
• Local priority-setting• Local operations• Local population health needs• Local alternatives• Local presence of trained personnel• Local infrastructure impact• Local budget impact……..• As well as HTAs by HTA agencies
HTA into Practice“Local HTA Decision-Support Program”
5
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Local HTA Committeeevaluation
MINOR CHANGENo significant clinical, contractual,
cost or resource issues
Referredfor further assessment
Forms to gather information
+ +
Clinical, Financial, or Economic Impact
Forms E-G
TechnologyRequest
Form A
Support of Technology Request
Form B
Contract-Costing Check
Form C
Local HTACheck
Form D
Approvedfor purchase/implementation
6
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Referredfor further assessment
Tools for Decision Making
RecommendationMade by Local HTA Committee
Evaluation Criteria
Tool 2
Evaluation Worksheet
Tool 3Decision
Made by Executive Committee
MINOR CHANGENo significant clinical, contractual, cost or
resource issues
ApprovedFor purchase/implementation
Screening Guide
Tool 1
Recommendation and Decision Guide
Tool 4
Local HTA Committee evaluation
Prioritization Tool
Tool 5
7
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
8
Criteria Development (supported by CADTH grant)
Criteria list from literature
Criteria revision by selected group
Group retreat (criteria revision)
Member survey (criteria ranking)
Final List of Criteria
Revised
Forms
Revised
Tools
Criteria cross-checked withinformation-gathering Forms
Criteria used to formalizeTechnology Evaluation and Decision
Tools
Mod
ified
Del
phi P
roce
ss
9
Tool 2: Evaluation Criteria
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
10
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
11
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
12
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
“CREATE” OR
“DIE”
13
Domain (5)
Health Gain
Service Delivery
Strategic Fit
Innovation
Financial
Tool 2: Evaluation Criteria
14
Domain (5) Criteria (12) Example Clarifying Questions
Health Gain Efficacy (patient) • evidence for health gain?
Population Health • addresses burden of disease?
Standard of Care • new Standard of Care?
Service Delivery Safety • as safe as current practice?
Training • requires training or credentialing?
Access • improves access to the under-served?
Service Coordination • impact on other health services?
Strategic Fit Strategic Fit • aligned with strategic goals?
Innovation Knowledge & Research • knowledge generation or transfer?
Financial Cost • start-up and on-going costs?
Sustainability • can we afford it ?
Economic Analysis • cost-effective?
Tool 2: Evaluation Criteria
15
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Referredfor further assessment
Tools 2 & 3: Evaluation Criteria and Worksheet
RecommendationMade by Local HTA Committee
Evaluation Criteria
Tool 2
Evaluation Worksheet
Tool 3Decision
Made by Executive Committee
MINOR CHANGENo significant clinical, contractual, cost or
resource issues
ApprovedFor purchase/implementation
Screening Guide
Tool 1
Local HTA Committee evaluation
16
For each criterion:
A. Score the QUALITY and COMPLETENESS of the information provided by the applicant: Adequate, Inadequate, NA
B. Score the SIGNIFICANCE and IMPACT of the technology on a scale of 0-5
C. Make a RECOMMENDATION
Tool 3: Evaluation Worksheet
17
Tool 3A: “ Device X” – Quality and Completeness of Information
Domain Criteria Example Clarifying Questions SCORE
Health Gain Efficacy (patient) • Evidence for efficacy? Indications for use? Inadequate
Population Health • Adequate projections of prevalence? Adequate
Standard of Care • Potential for new standard of care? Inadequate
Service Delivery Safety • Complications and risks addressed? Inadequate
Training • Training implications described? Adequate
Access • Improves access? NA
Service Coordination • Impact on other health services described? Adequate
Strategic Fit Strategic Fit • Aligned with strategic goals? Adequate
Innovation Knowledge & Research
• Innovative characteristics described?• Outcome measures described?
Inadequate
Financial Cost • Information on resources and impact complete? Adequate
Sustainability
Economic Analysis • Evidence to support cost-benefits? Adequate
18
For each criterion:
A. Score the QUALITY and COMPLETENESS of the information provided by the applicant: Adequate, Inadequate, NA
B. Score the SIGNIFICANCE and IMPACT of the technology on a scale of 0-5
C. Make a RECOMMENDATION
Tool 3: Evaluation Worksheet
19
Domain (5) Criteria (12) Example Clarifying Questions
Health Gain Efficacy (patient) • evidence for health gain?
Population Health • addresses burden of disease?
Standard of Care • new Standard of Care?
Service Delivery Safety • as safe as current practice?
Training • requires training or credentialing?
Access • improves access to the under-served?
Service Coordination • impact on other health services?
Strategic Fit Strategic Fit • aligned with strategic goals?
Innovation Knowledge & Research • knowledge generation or transfer?
Financial Cost • start-up and on-going costs?
Sustainability • can we afford it ?
Economic Analysis • cost-effective?
Tool 2: Evaluation Criteria
20
Tool 2: Criteria - Scoring Significance and Impact
CRITERIA 0 POINTS 1 POINT 3 POINTS 5 POINTS
Efficacy No improvement in patient health gain compared with current practices
Minimal improvement
Moderate improvement
Vast improvement
Population Health
Addresses a condition with very low prevalence (rate/100,000 < 1)
Low prevalence (rate/100,000 is 1-10)
Moderate prevalence (rate/100,000 is 10-1000)
High prevalence (rate/100,000 is 1,000-10,000)
Standard of Care
Does not represent Standard of Care in Alberta
Represents Standard of Care in some health regions in Alberta
Represents Standard of Care in most regions in Alberta
Represents a new Standard of Care in our region or Alberta
21
Tool 3B: “ Device X” – Scoring Significance and Impact
POTENTIAL IMPROVEMENT
Domain Criteria None: 0 Minimal: 1 Moderate: 3 Vast: 5
Health Gain Efficacy (patient) X
Population Health X
Standard of Care X
Service Delivery Safety X
Training (vast to minimal) X
Access X
Service Coordination X
Strategic Fit Strategic Fit X
Innovation Knowledge & Research X
Outcome Measures (quality) X
Financial Cost (viability) X
Sustainability
Economic Analysis X
22
SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness
and/or cost or resources impact
Referredfor further assessment
Tool 4: Recommendation and Decision Guide
RecommendationMade by Local HTA Committee
Evaluation Criteria
Tool 2
Evaluation Worksheet
Tool 3Decision
Made by Executive Committee
MINOR CHANGENo significant clinical, contractual, cost or
resource issues
ApprovedFor purchase/implementation
Screening Guide
Tool 1
Local HTA Committee evaluation
Recommendation and Decision Guide
Tool 4
23
Tool 4: Recommendation and Decision Guide
DecisionMade by Executive Committee
1. Rejected (No)
2. Approved (Yes)
3. Conditional (Yes but…a. Single caseb. Clinical auditc. Other (e.g. training)
4. Research Usea. Clinical trialb. Support of project
5. Referred
Recommendation and Decision Guide
Tool 4
RecommendationMade by Local HTA Committee
C r
i t e
r i
a
24
Tool 4: Recommendation and Decision Guide
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE(e.g., may include)
1. Rejected• Negative, poor, or no data on efficacy• Insufficient evidence of safety• Decreases or worsens service delivery
2. Approved • Efficacy and Safety well established• Enhanced population health is likely• Strategic fit is strong• Will likely improve service delivery• Financial impact is the same or better than current
practice (Cost Neutral)• Cost-effectiveness is likely the same or better than
current practice
25
Tool 4: Recommendation and Decision Guide
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE
3a. Conditional –Single Case
• Urgent/emergent approval of a single case where options are limited
3b. Conditional –Clinical Audit
• Evidence promising for efficacy, but limited for effectiveness
• Good evidence for safety• Cost-effectiveness may be uncertain• Advantage over current practice needs to be established• Cost needs to be established
3c. Conditional –Other
• May require new training protocols• Patient population may need to be defined
26
Tool 4: Recommendation and Decision Guide
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE
4a. Research Use –Clinical Trial
• Efficacy uncertain or controversial• Safety uncertain• Population health is uncertain• Effect on service delivery is uncertain• May be innovative• Clinical trials protocols must be observed
4b. Research Use –Support of Project
• Innovative project proposed by applicant• Funding and research protocols in place• Acceptable use of resources
5. Referred • Technology has potential impact beyond the scope of surgery
• Working group required
27
Tool 4: “Device X” - Decision
RECOMMENDATIONOR DECISION
CRITERIA & RATIONALE
4a. Research Use –Clinical Trial
• Efficacy uncertain or controversial• Safety uncertain• Population health is uncertain• Effect on service delivery is uncertain• May be innovative• Clinical trials protocols must be observed
28
Prioritize Technologies for funding or purchaseRecommendationsRecommendationsRecommendationsRecommendations
Prioritize Technologies for funding or purchase
DecisionsDecisionsDecisionsDecisions
Prioritization Tool
Tool 5
Proposal forTechnology OneTechnology OneTechnology OneTechnology One
Proposal forTechnology TwoTechnology TwoTechnology TwoTechnology Two
Proposal forTechnology ThreeTechnology ThreeTechnology ThreeTechnology Three
Tool 5: Prioritization Tool
29
Step 1. Screening Guide
Step 2. Criteria Review
Step 3. Criteria Weighting
Step 4. Criteria Rating Scales
Step 5. Technology Scoring
Step 6. Technology Ranking
Step 7. Additional Checks
Adapted from Drs. Craig Mitton and Cam Donaldson
Tool 5: Prioritization Tool
30
Objectives:
1. Local HTA Program Overview
2. Multiple Criteria for Technology Evaluation
3. Lessons Learned
31
Lessons Learned:
WEAKNESSES
•Criteria have not been weighted
•Prioritization Tool has not been tested
•Patient interests not well represented
•Administrators and practitioners sometimes resistant to change
32
Lessons Learned:
STRENGTHS
•Bottom-Up approach – User develop ensures buy-in
•Reviewers satisfied with criteria and technology evaluation tools
•Provides a systematic, consistent and transparent approach to technology evaluation and decision-making
33
Acknowledgements
Supported by:• Canadian Agency for Drugs and Technology in Health• The Department of Surgery & Surgical Services – Calgary• The Calgary Health Research Portfolio – Calgary• Surgery Clinical Network HTAI - AHS
Participating Departments from the former Calgary H ealth Region:• Surgery & Surgical Services• Anesthesia • Cardiac Sciences• Critical Care Medicine• Internal Medicine• Professional Practice & Development• Medical Device Safety & Risk Management• Pharmacy• Physician Leadership Portfolio, and the • Calgary Health Research Portfolio
34
Multiple CriteriaHTA Into Practice
Local HTA Program
InnovationCREATE
OR DIE!
Contact information:
Dr. Paule Poulin
Alberta Health Services - Calgary
(403) 944-1652