Systematising the process : the role of PSMs in informing model structure

Jim ChilcottTechnical Director, ScHARR-TAG

Suzy PaisleyDoH Research Scientist in Evidence Synthesis

Paul TappendenSenior OR Analyst

Consensus Working Group on the Use of Evidence in Economic Decision Models MRC HSRC Workshop 26 September 2005

Before we start…

Consensus Working Group on the Use of Evidence in Economic Decision Models

Today’s presentation

Define ‘structuring’ within the modelling process

Describe current practice– Evidence used– Tasks undertaken

Discuss ‘What role problem structuring methods (PSMs)?’

Consensus Working Group on the Use of Evidence in Economic Decision Models

Defining model structuring

Disease and treatment pathway

Model scope Modelling method

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:HTA Guidelines

HTA guidelines (n=32)

None 17 (53%)

Description 2 (6%)

Justification 4 (13%)

Sensitivity analysis 5 (16%)

Not available 4 (13%)

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:Modelling guidelines (Philips et al.,2004)

Statement of problem

Justification of approach

Scope / perspective Structural

assumptions Consistency

Strategies / comparators

Model type Time horizon Health states Cycle length Parsimony

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:Reported (Cooper et al., 2005)

HTA monographs (n=42)

Diagram of model 26(62%)

Model structure

and assumptions discussed 5 (12%)

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:Observed

8 case study projects Description model structuring process Short, semi-structured interviews Examination of corresponding study

reports

Consensus Working Group on the Use of Evidence in Economic Decision Models

Evidence consulted

Subject Clinical literature

Clinical expert (one-off)

Clinical expert (ongoing)

Existing models

1 2 3 4 5 6 7 8

TOTAL 8 (100%) 1 (13%) 5 (63%) 8 (100%)

Consensus Working Group on the Use of Evidence in Economic Decision Models

Reporting

Subject Methodology justified Health states /pathways justified?

Source of structural assumptions reported?

1 Report not yet available

2 3 4 5 6 7 8 TOTAL 5 (71%) 2 (29%) 3 (43%)

Consensus Working Group on the Use of Evidence in Economic Decision Models

Summary of themes

Model structure amended according to data availability

Clinician input (1st order validation) Existing models consulted as 1st step Iterative structuring process Complexity of structure shifted both ways Very limited reporting process of development MSc dissertation provided most transparent

reporting

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:Evidence used

Clinical expertise Published evidence Existing models

Natural history Treatment pathway Modelling method Effectiveness

evidence

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:Tasks undertaken

Iteration Consultation Evaluation and comparison Feedback and clarification Decisions regarding simplicity,

complexity, feasibility and inconsistency

Consensus Working Group on the Use of Evidence in Economic Decision Models

Why PSMs?

Lack of transparency Data modelling vs problem modelling Validity of model (unquestioned

question) Issues that ‘can’t be modelled’ ‘Not qualified to comment’

Consensus Working Group on the Use of Evidence in Economic Decision Models

Why PSMs?

Data led models vs evidence informed models

Use of expert opinion Implicit use of PSM methods Observed usefulness of action research

process

Consensus Working Group on the Use of Evidence in Economic Decision Models

PSMs / MSMs

Messes Problems

SODA

SSM

SCA

Robustness Analysis

Drama Theory

Decision Conferencing

?

Decision analysis

Markov

Queuing theory

Simulation

Game Theory

Consensus Working Group on the Use of Evidence in Economic Decision Models

Characteristics of PSMs

Techniques to facilitate exploration / definition of a problem situation

Group based Uncovering options Enabling group action Accepting uncertainty Managing conflict Facilitator - analyst

Consensus Working Group on the Use of Evidence in Economic Decision Models

SODA

Individual interviews =>

Merge maps =>

Group workshops =>

Commitment to action

Cognitive mapsGroup mapExpand/refine map Group ownership

Consensus Working Group on the Use of Evidence in Economic Decision Models

SSM

Enter situation Take action to improve theconsidered problematic problem situation

Define possible changes which areboth desirable and feasible

Express the problem situation

Compare models with the real worldReal world

Systems thinking about the real world

Formulate root Build conceptual models definitions of the systems named in

the root definitions

Consensus Working Group on the Use of Evidence in Economic Decision Models

Pros Problem structuring, not problem

solving Gathering information, generating ideas Retain flexibility in early stages Demonstrate understanding of

complexity, diversity and inconsistency Prioritising and simplifying

Consensus Working Group on the Use of Evidence in Economic Decision Models

Cons

Methods development required Adaptation to HTA context New skills Ownership Facilitation of group decision making Reproducibility

Consensus Working Group on the Use of Evidence in Economic Decision Models

Developing a mixed method

‘Soft’ qualitative structuring process

‘Hard’ analytic

methods

Qualitative model structuring process to inform quantitative decision-making process.