Modeling UncertaintyModeling UncertaintyFarrokh Alemi, Ph.D.Farrokh Alemi, Ph.D.

Saturday, February 21, 2004Saturday, February 21, 2004

Why Make a Model?Why Make a Model? To forecast likelihood for complex To forecast likelihood for complex

eventsevents To predict events that rarely occurTo predict events that rarely occur To understand and communicate the To understand and communicate the

nature of uncertaintynature of uncertainty

Online HMOOnline HMO MD – Pt email and phone contactMD – Pt email and phone contact

– Triage before visitTriage before visit Lab done before visitLab done before visit Visits postponedVisits postponed

– Follow up automatedFollow up automated

No historical precedence

Step 1. Select Target EventStep 1. Select Target Event Usually two eventsUsually two events

– Die/liveDie/live– Purchase/not purchasePurchase/not purchase– Join HMO/Not joinJoin HMO/Not join

Mutually exclusive eventsMutually exclusive events Exhaustive Exhaustive

Step 2. Divide & ConquerStep 2. Divide & Conquer

Posterior Odds

Likelihood RatiosPrior Odds

Step 3. Identify Clues Step 3. Identify Clues Would you tell me a little about your Would you tell me a little about your

experience with employee choice of health experience with employee choice of health plans? plans? – Suppose you were to decide whether an Suppose you were to decide whether an

employee is likely to join but you could not employee is likely to join but you could not contact the employee. I was chosen to be your contact the employee. I was chosen to be your eyes and your ears. What should I look for?eyes and your ears. What should I look for? What is an example of a characteristic that would What is an example of a characteristic that would

increase the chance of joining the HMO? increase the chance of joining the HMO? Describe an employee who is unlikely to join the Describe an employee who is unlikely to join the

proposed HMO proposed HMO

Example of Clues IdentifiedExample of Clues Identified Age Age Income and value of time to the Income and value of time to the

employee employee Gender Gender Computer literacy Computer literacy Current membership in an HMO Current membership in an HMO

Step 4. Describe Levels of Step 4. Describe Levels of CluesClues

Measures the extent to which clue is Measures the extent to which clue is present present – Usually present or absentUsually present or absent

Combine levels that are similarCombine levels that are similarAnalyst: What age would favor joining the HMO?Expert: Young people are more likely to joinAnalyst: How do you define young employees?Expert: It all depends. Below 30 is different from above 30.

Levels for HMO ExampleLevels for HMO Example AgeAge (younger than 30, 31‑40, older than 41) (younger than 30, 31‑40, older than 41) Value of time to the employeeValue of time to the employee (income (income

over $50,000, income between $30,000 and over $50,000, income between $30,000 and $50,000, income less than $30,000) $50,000, income less than $30,000)

GenderGender (male, female) (male, female) Computer literacyComputer literacy (programs computers, (programs computers,

frequently uses a computer, routinely uses frequently uses a computer, routinely uses output of a computer, has no interaction with output of a computer, has no interaction with a computer) a computer)

Tendency to join existing HMOsTendency to join existing HMOs (enrolled (enrolled in an HMO, not enrolled in an HMO) in an HMO, not enrolled in an HMO)

Step 5. Test for Step 5. Test for Independence Independence

If you have data:If you have data:– Reduction of sample sizeReduction of sample size– Correlation analysisCorrelation analysis

If you do not have data:If you do not have data:– Draw a causal modelDraw a causal model– Ask experts to group clues if one tells us Ask experts to group clues if one tells us

a lot about another with specific a lot about another with specific populationspopulations

Step 6. Estimate Likelihood Step 6. Estimate Likelihood RatiosRatios

Of 100 people who do join, how many Of 100 people who do join, how many are younger than 30? are younger than 30? Of 100 people who do not join the HMO, Of 100 people who do not join the HMO, how many are younger than 30?how many are younger than 30?

Imagine two employees, one who will Imagine two employees, one who will join the HMO and one who will not. Who join the HMO and one who will not. Who is more likely to be younger than 30?is more likely to be younger than 30?How many times more likely? How many times more likely?

How to Improve Estimates of How to Improve Estimates of Likelihoods?Likelihoods?

Ask experts not novicesAsk experts not novices Ask only in format familiar with the expert.Ask only in format familiar with the expert. Provide access to tools and available data, Provide access to tools and available data,

if relevantif relevant Train experts in probability conceptsTrain experts in probability concepts

– Likelihood ratio of 1, less, or more than 1Likelihood ratio of 1, less, or more than 1– Relationship between odds and probability Relationship between odds and probability

(Odds of 2‑to‑1 mean a probability of 0.67; (Odds of 2‑to‑1 mean a probability of 0.67; odds of 5‑to‑1 mean a probability of 0.83)odds of 5‑to‑1 mean a probability of 0.83)

Rely on more than one expert and discuss Rely on more than one expert and discuss first estimate as well as any estimate with first estimate as well as any estimate with large differences large differences

Step 7. Estimate Prior OddsStep 7. Estimate Prior Odds Out of 100 employees, how many will Out of 100 employees, how many will

join? join? Odds for joining = p(Joining) /  [1 - Odds for joining = p(Joining) /  [1 -

p(Joining)]p(Joining)] Probability of Joining = Odds of Probability of Joining = Odds of

Joining / (1 + odds of joining)Joining / (1 + odds of joining)

Step 8. Develop Scenarios Step 8. Develop Scenarios Select one level for each clueSelect one level for each clue Organize on one piece of paperOrganize on one piece of paper Ask the expert to rate on a scale Ask the expert to rate on a scale

from 0 1o 100 the chances that from 0 1o 100 the chances that target event will happentarget event will happen

Optimistic Scenario for HMO Optimistic Scenario for HMO Example Example

A 29‑year‑old male employee A 29‑year‑old male employee Earns more than Earns more than $60,000. $60,000. He is busy and values his time; He is busy and values his time; He is familiar with computers, using them He is familiar with computers, using them

both at work and at home. both at work and at home. He is currently an HMO member, though He is currently an HMO member, though

not completely satisfied with it.not completely satisfied with it.

On a scale from 0 1o 100, how likely do On a scale from 0 1o 100, how likely do you think is for this person to join the you think is for this person to join the proposed HMO?proposed HMO?

Pessimistic Scenario for HMO Pessimistic Scenario for HMO Example Example

A 55‑year‑old female employee A 55‑year‑old female employee earning less than $85,000. She has earning less than $85,000. She has never used computers and has refused never used computers and has refused to join the firm's existing HMO .to join the firm's existing HMO .

On a scale from 0 1o 100, how likely On a scale from 0 1o 100, how likely do you think is for this person to join do you think is for this person to join the proposed HMO?the proposed HMO?

A More Realistic Scenario for A More Realistic Scenario for HMO Example HMO Example

A 55‑year‑old female employee A 55‑year‑old female employee earning more than earning more than $60,000 $60,000 has used has used computers but did not join the firm's computers but did not join the firm's existing HMO. existing HMO. On a scale from 0 1o 100, how likely On a scale from 0 1o 100, how likely do you think is for this person to join do you think is for this person to join the proposed HMO?the proposed HMO?

Scenario PlanningScenario Planning Helps decision makers understand Helps decision makers understand

possible futurespossible futures Helps decision makers work to Helps decision makers work to

realize alternative futuresrealize alternative futures

Step 9: Validate the ModelStep 9: Validate the Model

Step 10. Make a Forecast Step 10. Make a Forecast Likelihood ratiosLikelihood ratios

– 1.2 for being young, 1.2 for being young, – 1.1 for being male, 1.1 for being male, – 1.2 for having a high hourly rate, 1.2 for having a high hourly rate, – 3.0 for being computer literate, 3.0 for being computer literate, – 0.5 for not being a member of an HMO. 0.5 for not being a member of an HMO.

  Odds of joining = 1.1 x 1.2 x 3 x 0.5 x 1 = Odds of joining = 1.1 x 1.2 x 3 x 0.5 x 1 = 1.981.98

Probability of joining = 1.98 / (1 + 1.98) = Probability of joining = 1.98 / (1 + 1.98) = 0.660.66

Take Home LessonsTake Home Lessons Forecasts of unique events are possibleForecasts of unique events are possible Known clues and relationships can be Known clues and relationships can be

used to forecast an eventused to forecast an event Bayes model needs likelihood ratios Bayes model needs likelihood ratios

and prior oddsand prior odds– Experts can supply these estimatesExperts can supply these estimates

We can validate model against experts’ We can validate model against experts’ judgmentsjudgments