modeling framework to support evidence-based decisions
DESCRIPTION
Describes a framework for modelling in a regulatory environment founded on sound scientific and knowledge management concepts. It includes 1) demand (isue-driven) and supply (model driven) approaches to modelling, 2) balancing modeler, manager, and user perspectives, 3) documentation to demonstrate due diligence, and a 700-term glossary.TRANSCRIPT
![Page 1: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/1.jpg)
A Modelling Framework:
Supporting Evidence-Based Decisions
ModSimWorld
Montreal, Quebec June 8-9, 2009
![Page 2: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/2.jpg)
2
Main MessagesMain Messages
• Models are needed to understand and predict the behavior of complex systems.
• Models are needed to fulfill an agency’s mandate and support its core business.
• Inadequate or incorrect use of models wastes resources, results in errors, and exposes an agency to liability.
• Models are needed to understand and predict the behavior of complex systems.
• Models are needed to fulfill an agency’s mandate and support its core business.
• Inadequate or incorrect use of models wastes resources, results in errors, and exposes an agency to liability.
Models should be used wisely
![Page 3: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/3.jpg)
3
OutlineOutline
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
![Page 4: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/4.jpg)
4
About ModelsAbout Models
• What are they?
– Simplified representations of reality.
– Transform data, information, and knowledge into outputs.
• Why do we use them?
– Reality is too complex
– Experiments are infeasible
– Predict consequences
– Increase understanding
• What are they?
– Simplified representations of reality.
– Transform data, information, and knowledge into outputs.
• Why do we use them?
– Reality is too complex
– Experiments are infeasible
– Predict consequences
– Increase understanding
....
Nonaka (2000)
Concepts
![Page 5: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/5.jpg)
5
What is a Framework?What is a Framework?
“Structural outline of the components of an organization, system, or process and the relationships among them.”
Understanding Knowledge Services NRCan (2006)
Concepts
![Page 6: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/6.jpg)
6
Framework ObjectivesFramework Objectives
• Support needs-driven and science-driven analysis.
• Promote dialogue among modelers, managers, & users.
• Reduce wasted time, effort, & money.
• Provide a basis for planning and action.
• Document and justify decisions.
• Support needs-driven and science-driven analysis.
• Promote dialogue among modelers, managers, & users.
• Reduce wasted time, effort, & money.
• Provide a basis for planning and action.
• Document and justify decisions.
Concepts
![Page 7: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/7.jpg)
7
Framework DesignFramework Design
• Reflect modelling, management, and user perspectives.
• Balance efficiency and effectiveness with cost and effort.
• Applicable to both demand and supply approaches to modelling.
• Applicable to both logical and computational models
• Reflect modelling, management, and user perspectives.
• Balance efficiency and effectiveness with cost and effort.
• Applicable to both demand and supply approaches to modelling.
• Applicable to both logical and computational models
Concepts
![Page 8: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/8.jpg)
8
Different PerspectivesDifferent Perspectives
What developers proposed
What managers funded
What stakeholders wanted
What users needed
Concepts
![Page 9: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/9.jpg)
9
Supply & DemandSupply & Demand
Concepts
Supply: I have a model that solves your problem.
Demand: I have a problem that needs a model.
![Page 10: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/10.jpg)
10
Modelling SystemModelling System
External Models
DevelopDevelop
Nature, Society
Internal Models
UseUseManageManage
Lost Models
ShareShare
PreservePreserve
Knowledge Management
Concepts
![Page 11: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/11.jpg)
11
Modelling ProcessModelling Process
Modelling combines science & computers; judgement & experience; insight & intuition.
• Principles: effort, simple, data, knowledge, transparent, understandable.
• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.
• Development: techniques are well-understood; management less understood and practiced.
• Use: Decision making under uncertainty, unknown elements, outcome probabilities.
Modelling combines science & computers; judgement & experience; insight & intuition.
• Principles: effort, simple, data, knowledge, transparent, understandable.
• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.
• Development: techniques are well-understood; management less understood and practiced.
• Use: Decision making under uncertainty, unknown elements, outcome probabilities.
Concepts
![Page 12: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/12.jpg)
12
Systems HierarchySystems Hierarchy
Data
ModelsDecision SupportInformatio
nKnowledge
ManagementManagement
Mandate
Business
Policies
Processes
Concepts
![Page 13: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/13.jpg)
13
Data Data
A model and its data are inseparable; they succeed or fail as one.
– Data Needs: Situation may involve nature, the system, and/or intervention.
– Sampling: Statistics are essential to determine how much data is needed.
– Source: Ownership? Use rights? Privacy & security concerns?
– Scale: Time, space, and process scale must match the situation.
– Quality: Level of accuracy, detail, and completeness are needed?
A model and its data are inseparable; they succeed or fail as one.
– Data Needs: Situation may involve nature, the system, and/or intervention.
– Sampling: Statistics are essential to determine how much data is needed.
– Source: Ownership? Use rights? Privacy & security concerns?
– Scale: Time, space, and process scale must match the situation.
– Quality: Level of accuracy, detail, and completeness are needed?
Concepts
![Page 14: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/14.jpg)
14
OutputsAcquisition ProcessingStorage
Knowledge
Organization
Society
Environment
Events
Economy
Channels
Access Search Retrieval
InterfaceProcessingDatabase
Inputs Audience UseMedia
InterfaceSystemDataModel
Outputs
Concepts
Information SystemInformation System
Interoperability IntegrationAvailability Utility
![Page 15: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/15.jpg)
151. Common
•Flow-through (-)•Fixed (1:1)•Planning•Mechanistic•Automated•Data, facts
2. Complicated •Feedback•Linear (1:n)•Mathematics•Deterministic•Certainty•Explicit knowledge
3. Complex •Predictive feedback (+)•Non-linear (1:?)•Simulation•Stochastic•Uncertainty •Tacit knowledge
4. Chaotic
•Emergent•Disorganized•Scenario analysis•Mental•Reaction•Intuition
System:Behavior:Approach:Model:Decision:Basis:
Concepts
Models and Knowledge Models and Knowledge
![Page 16: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/16.jpg)
16
OutlineOutline
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
![Page 17: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/17.jpg)
17
Decision Guide - HierarchyDecision Guide - Hierarchy
• Phase: (3) demand, supply, project
• Stage: (7) approach, design, establish, develop, evaluate, implement, conclude
• Step: (34) screening, problem definition, suitability, knowledge, data
• Consideration (132): recurrence, importance, problem space, existence
• Phase: (3) demand, supply, project
• Stage: (7) approach, design, establish, develop, evaluate, implement, conclude
• Step: (34) screening, problem definition, suitability, knowledge, data
• Consideration (132): recurrence, importance, problem space, existence
![Page 18: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/18.jpg)
18
Design
Acquire Data
Generate Knowledge
Approach
Out
Evaluation
IssueStart: (Demand)
Implementation
Outputs
Conclusion
End
Development(Modeller)
(Manager)
(User)
D1
D2
3
Applicability
S2
ModelStart (Supply) identification
S1
5
7
6
Establishment
4
(Manager)
(All)
Decision Guide - StagesGuide
![Page 19: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/19.jpg)
19
Issue Model
Demand Supply
Project
End
Decision Guide PhasesDecision Guide Phases
Guide
![Page 20: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/20.jpg)
20
Start (use)
Demand-drivenbackward chaining, closed question
Model
Supply-drivenforward chaining, open question
Start (model) Uses
Supply & DemandSupply & Demand
Guide
![Page 21: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/21.jpg)
21
Issue
D1. Approach
D2. Design
Development
Acquire Data
Generate Knowledge
Out
Demand PhaseDemand Phase
Guide
![Page 22: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/22.jpg)
22
Issue
Initial Screening
Problem Definition
Suitability
Knowledge Evaluation
Data Availability
Design
Recurrence ImportanceProblem spaceExistence
Business FunctionIntended use
Time available Situation
NeedsExistingGap
NeedsAttributesAccessibilityProcessing
Continue
Continue
Continue
Continue
Continue
Below threshold
Can’t define
Unsuitable
Excess gap
Inadequate
Generate?
Acquire ?
Yes
Yes
Out
No
No
D1.1
D1.2
D1.3
D1.4
D1.5
Guide Approach Stage
![Page 23: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/23.jpg)
23
Decision Guide ConsiderationsDecision Guide Considerations
• Explains the question.
• Classify a situation or write a short description.
• Complete a statement template.
• Decide where to go next.
• Not a cookbook to be followed without interpretation.
• Compliments experience & judgement; doesn’t replace them.
• Explains the question.
• Classify a situation or write a short description.
• Complete a statement template.
• Decide where to go next.
• Not a cookbook to be followed without interpretation.
• Compliments experience & judgement; doesn’t replace them.
Guide
![Page 24: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/24.jpg)
24
Development
Model
S1. Identification
S2. Applicability
Evaluation
Conclusion
Supply PhaseSupply Phase
Guide
![Page 25: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/25.jpg)
25
Existing Model
Knowledge base
Suitability
Evaluation
Specification criteria (4)
Knowledge criteria (3)
Business lineFunction Development
S2.2
S2.3
S2.4
IdentificationSearchDescription
S1
modify
Data AvailabilityData criteria (5)
Data Acquisition
inaccessible
Conclusion
unsuitable
unjustified
Specifications
S2.1
Development criteria (3) Development
S2.5
Applicability Stage
Guide
![Page 26: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/26.jpg)
26
Project
Phase
End
Design
Applicability
4. Development
5. Evaluation
6. Implementation
7. Project Conclusion
Out3. Project Establishment
Situation
Guide
![Page 27: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/27.jpg)
27
Project Establishment
Hierarchy Relationships Indicators Review
Logic Computation Debugging Review
Attributes ConsistencyReview
Uncertainty Representation Review
Conceptualization
Construction
Evaluation
Verification
Validation
Inventory
exit
exit
exit
exit
4.2
4.3
4.4
4.5
continue
continue
continue
continue
Interaction
Awareness Understanding Consensus
4.1
Development Stage
Guide
![Page 28: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/28.jpg)
28
OutlineOutline
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
I. Underlying Concepts
Scientific underpinning
II. Decision Guide
Decision making
III. Glossary
Common understanding
![Page 29: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/29.jpg)
29
GlossaryGlossary
• Background (introduction, methods, references).
• Taxonomy (organization, nature, risk analysis, content, modelling, concepts).
• Definitions - 650 terms from six sources.
• Links to taxonomy and related terms.
• Background (introduction, methods, references).
• Taxonomy (organization, nature, risk analysis, content, modelling, concepts).
• Definitions - 650 terms from six sources.
• Links to taxonomy and related terms.
Glossary
![Page 30: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/30.jpg)
30
Sample DefinitionSample Definition
Model: Abstract and simplified construct or representation of reality in the form of a pattern, description, or definition that shows the essential structure, relationships, and workings of a concept, process, or system.
(see modelling approach, function, modelling methods, process, relationship, representation, system)
Glossary
![Page 31: Modeling Framework to Support Evidence-Based Decisions](https://reader036.vdocument.in/reader036/viewer/2022062704/55574e07d8b42aa8378b50c4/html5/thumbnails/31.jpg)
31
Modelling Framework:
• Supports an agency’s business
• Facilitates horizontal integration
• Minimizes waste & inefficiency
• Maximizes likely success
• Documents & justifies decisions
“Using a clear blueprint first prevents chaos latter.”
Carla O’Dell (1998)