analytics in context: modelling in a regulatory environment

Analytics in Context:

Modelling in a Regulatory

EnvironmentPresented to:

International Conference on Analytics Solutions Ottawa, Ontario, September 29-30, 2014

Presented to:

International Conference on Analytics Solutions Ottawa, Ontario, September 29-30, 2014

Albert SimardIntegrated Knowledge Services

albert.simard@outlook.com

2

OutlineOutline

Modelling Context

Human, Technical

Framework

Structure, Hierarchy

Modelling Context

Human, Technical

Framework

Structure, Hierarchy

3

Diverse PerspectivesDiverse Perspectives

What Analysts proposed

What managers funded

What stakeholders wanted

What users needed

Manager DiversityManager Diversity

• Accountable for actions• Situational pressure• Broader view than analysis• Organizational infrastructure • Depth of understanding • Confidence in results• Risk tolerance• Previous experience• Belief system

• Accountable for actions• Situational pressure• Broader view than analysis• Organizational infrastructure • Depth of understanding • Confidence in results• Risk tolerance• Previous experience• Belief system

Analyst DiversityAnalyst

Diversity

• Objective Criteria–Problem space–Type of problem–Available techniques

• Subjective Criteria–Awareness–Diversity –Skill–Experience–Expertise–Mental model

• Objective Criteria–Problem space–Type of problem–Available techniques

• Subjective Criteria–Awareness–Diversity –Skill–Experience–Expertise–Mental model

User Diversity

User Diversity

GoalsCompatible Conflicting

Interests

Mutual

Autonomous

Competitiondefence or victory aggressive approach no trust secretive, hostile

Collaborationjoint or peer production partnership approach high trust diverse, synergistic

Negotiationmutual agreement adversarial approach nominal trust structured, formal

Sharingleverage knowledge passive approach moderate trust benign, supportive

The Challenge

The Challenge

Integrating the diverse views to increase understanding and achieve a mutually agreed solution!

Integrating the diverse views to increase understanding and achieve a mutually agreed solution!

7

8

Modelling ProcessModelling Process

• Modelling combines science & computers; judgement & experience; insight & intuition.

• Principles: effort, simple, data, knowledge, transparent, understandable.

• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.

• Development: techniques are well-understood; management less understood and practiced.

• Use: Decision making under uncertainty, unknown elements, outcome probabilities.

• Modelling combines science & computers; judgement & experience; insight & intuition.

• Principles: effort, simple, data, knowledge, transparent, understandable.

• Complexity: Modelling is a dynamic feedback process with delays and uncertainty.

• Development: techniques are well-understood; management less understood and practiced.

• Use: Decision making under uncertainty, unknown elements, outcome probabilities.

Spectrum of Evidence

Spectrum of Evidence

• mathematics, logic, proof

• science, engineering, technology

• statistics, data, facts, measurement

• collaboration, validation, management

• expertise, experience, judgement

• opinion, perception, bias

• belief, emotion, values

• mathematics, logic, proof

• science, engineering, technology

• statistics, data, facts, measurement

• collaboration, validation, management

• expertise, experience, judgement

• opinion, perception, bias

• belief, emotion, values

Quantitative (irrefutable evidence)

Qualitative (no evidence)

10

Data Attribute

s

Data Attribute

s A model and its data are inseparable;

they succeed or fail as one.• Data Needs: Situation may involve nature, the

system, and/or intervention.• Sampling: Statistics are essential to determine

how much data is needed.• Source: Ownership? Use rights? Privacy &

security concerns?• Scale: Time, space, and process scale must

match the situation.• Quality: What level of accuracy, detail, and

completeness are needed?

A model and its data are inseparable; they succeed or fail as one.

• Data Needs: Situation may involve nature, the system, and/or intervention.

• Sampling: Statistics are essential to determine how much data is needed.

• Source: Ownership? Use rights? Privacy & security concerns?

• Scale: Time, space, and process scale must match the situation.

• Quality: What level of accuracy, detail, and completeness are needed?

Knowledge Domains

Knowledge Domains

Adapted from Kurtz and Snowden (2003)

12

OutlineOutline

Modelling Context

Human, Technical

Framework

Structure, Hierarchy

Modelling Context

Human, Technical

Framework

Structure, Hierarchy

13

Framework ObjectivesFramework Objectives

• Support needs-driven and science-driven analysis.

• Promote dialogue among modelers, managers, & users.

• Reduce wasted time, effort, & money.

• Provide a basis for planning and action.

• Document and justify decisions.

• Support needs-driven and science-driven analysis.

• Promote dialogue among modelers, managers, & users.

• Reduce wasted time, effort, & money.

• Provide a basis for planning and action.

• Document and justify decisions.

14

Framework DesignFramework Design

• Reflect modelling, management, and user perspectives.

• Balance efficiency and effectiveness with cost and effort.

• Applicable to both demand and supply approaches to modelling.

• Applicable to both logical and computational models

• Reflect modelling, management, and user perspectives.

• Balance efficiency and effectiveness with cost and effort.

• Applicable to both demand and supply approaches to modelling.

• Applicable to both logical and computational models

15

Framework HierarchyFramework Hierarchy

• Phase: (4) demand, supply, project, use

• Stage: (9) approach, design, establish, develop, evaluate, identify, applicability, implement, conclude

• Step: (34) screening, problem definition, suitability, knowledge, data

• Consideration (132): classification, statements, decision

• Phase: (4) demand, supply, project, use

• Stage: (9) approach, design, establish, develop, evaluate, identify, applicability, implement, conclude

• Step: (34) screening, problem definition, suitability, knowledge, data

• Consideration (132): classification, statements, decision

16

Framework PhasesFramework Phases

Issue Model

Demand Supply

Project

End

Implement

Supply & DemandSupply & Demand

Supply: What can I do with an available model?

(Intelligence)

Demand: What model do I need to solve a problem?

(Regulation)

18

Start (methods)

Demand-drivenbackward chaining, closed question

Model

Supply-drivenforward chaining,open question

Start (model)

Uses

Supply & DemandSupply & Demand

19

Issue

Approach

Design

Development

Acquire Data

Generate Knowledge

Out

Demand PhaseDemand Phase

20

Design

Acquire Data

Generate Knowledge

Approach

Out

Evaluation

IssueStart: (Demand)

Implementation

Outputs

Conclusion

End

Development(Modeller)

(Manager)

(User)

Applicability

ModelStart (Supply) identification

Establishment

(Manager)

Framework Stages

21

Issue

Initial Screening

Problem Definition

Suitability

Knowledge Evaluation

Data Availability

Design

Recurrence ImportanceProblem spaceExistence

Business FunctionIntended use

Time available Situation

NeedsExistingGap

NeedsAttributesAccessibilityProcessing

Continue

Continue

Continue

Continue

Continue

Below threshold

Can’t define

Unsuitable

Excess gap

Inadequate

Generate?

Acquire ?

Yes

Yes

Out

No

No

Approach Stage - Steps

Considerations

22

ConsiderationsConsiderations

• Explain what is being considered• Classify a situation or write a short

description. (categories)• Complete a statement. (template)• Decide where to go next. (decision guide)• Not a cookbook to be followed without

interpretation.• Compliments experience & judgement;

doesn’t replace them.

• Explain what is being considered• Classify a situation or write a short

description. (categories)• Complete a statement. (template)• Decide where to go next. (decision guide)• Not a cookbook to be followed without

interpretation.• Compliments experience & judgement;

doesn’t replace them.

23

• [Specified data] are needed to [develop, run] a model of [function, process] because [explanation].

• The [model name] needs [specified knowledge] about [specified function, process] because [explanation].

• [Specified data] are needed to [develop, run] a model of [function, process] because [explanation].

• The [model name] needs [specified knowledge] about [specified function, process] because [explanation].

Consideration Template

Consideration Template

24

• If a substantial majority of the data needed by the model exist, go to Data Accessibility.

• If a majority of the data needed by the model exist, go to Data Accessibility, subject to limits or conditions; describe the limits.

• If existing data are adequate for a simpler or partial model, go to problem definition; redefine the problem.

• If existing data are not adequate for a simpler or partial model, go to Data Acquisition.

• If a substantial majority of the data needed by the model exist, go to Data Accessibility.

• If a majority of the data needed by the model exist, go to Data Accessibility, subject to limits or conditions; describe the limits.

• If existing data are adequate for a simpler or partial model, go to problem definition; redefine the problem.

• If existing data are not adequate for a simpler or partial model, go to Data Acquisition.

Decision Guide

Decision Guide

The Modelling Framework

• Supports business objectives• Facilitates horizontal integration• Minimizes waste & inefficiency • Maximizes likely success • Documents & justifies decisions

• Supports business objectives• Facilitates horizontal integration• Minimizes waste & inefficiency • Maximizes likely success • Documents & justifies decisions

“Using a clear blueprint first prevents chaos latter.” Carla O’Dell (1998)