Health Care Manag Sci (2012) 1591ndash102DOI 101007s10729-011-9185-4

A Markov decision model for determining optimaloutpatient scheduling

Jonathan Patrick

Received 7 March 2011 Accepted 27 October 2011 Published online 17 November 2011copy Springer Science+Business Media LLC 2011

Abstract Managing an efficient outpatient clinic canoften be complicated by significant no-show rates andescalating appointment lead times One method thathas been proposed for avoiding the wasted capacitydue to no-shows is called open or advanced access Theessence of open access is ldquodo todayrsquos demand todayrdquoWe develop a Markov Decision Process (MDP) modelthat demonstrates that a short booking window doessignificantly better than open access We analyze anumber of scenarios that explore the trade-off betweenpatient-related measures (lead times) and physician- orsystem-related measures (revenue overtime and idletime) Through simulation we demonstrate that over awide variety of potential scenarios and clinics the MDPpolicy does as well or better than open access in termsof minimizing costs (or maximizing profits) as well asproviding more consistent throughput

Keywords Clinic scheduling middot Open acess middotMarkov decision processes middot Dynamic programming middotSimulation

1 An introduction to outpatient scheduling

In recent years a form of scheduling called advancedaccess or open access (OA) has been touted as thepreferred booking policy for outpatient clinics Themantra is ldquodo todayrsquos demand todayrdquo The implicitassumption is that due to a combination of the cost

J Patrick (B)Telfer School of Management University of Ottawa55 Laurier Avenue Ottawa ON K1N 6N5 Canadae-mail patricktelferuottawaca

of no shows and the value the clinic places on same-day access the benefit associated with being able tosmooth demand over a period of time is insufficient towarrant delaying appointments This begs the questionas to how prevalent no-shows need to be andor howstrongly the clinic needs to value same-day access in or-der to outweigh the benefit of being able to smooth outdemand It is such trade-offs that this research seeks toexplore by developing an MDP model to determine theoptimal booking policy for given wait-time-dependentno-show rates and by exploring potential cost structuresfor various clinics

The detriments of a strict OA policy have begun toappear in the literature as some papers have suggestedthat a short booking window is more appropriate andthat there are certain conditions that need to be sat-isfied in order for OA to succeed This work falls in linewith that literature and proposes a different outpatientscheduling policy that does in fact use a short bookingwindow combined with overbooking to mitigate theimpact of no-shows

We assume that we are dealing with a clinic that hasa fixed capacity to see C clients per day but with ad-ditional overtime available if necessary Clients call forappointments before the start of a service period andcan either be booked into that period or else bookedinto the next available appointment slot in a futureservice period Appointments are subject to a no-showprobability that increases the further in advance theappointment is booked Decisions regarding how manyrequests to serve today and how many to book inadvance have to be made prior to knowing if any of thepreviously booked appointments will fail to show up

The rest of the paper proceeds as follows InSection 2 a review of the literature on open access

92 J Patrick

and clinic scheduling is presented and how our ap-proach differs from what has been done previously isdemonstrated In Section 3 the MDP model for theclinic scheduling problem is described as well as someof the assumptions inherent in the model and twocomplications that seek to overcome some of those as-sumptions In Section 4 the results of testing the MDPpolicy against OA in a simulation are presented and inSection 5 the form of the MDP policy is describedFinally Section 6 presents the conclusions from thisresearch

2 Literature review

Murray and Tantau [10] first proposed the idea ofOA as a solution to the high levels of no-shows oftenpresent in outpatient clinics They provide a case studyof a successful implementation of open access in theUS Kopach et al [6] provide a simulation study todetermine the impact of various clinic characteristics onthe successful implementation of OA Their simulationallows for a multi-doctor clinic and patient specific no-show probabilities They allow for limited overbookingin some of the scenarios analyzed in the simulationRobinson and Chen [12] demonstrate that OA mostoften outperforms a traditional booking system (witha fixed number of patients per day) but is itself out-performed by a ldquosame-or-next-dayrdquo scheduling policythus demonstrating that there is some advantage toflexibility in scheduling They use a weighted sum ofphysician idle time direct patient waiting time andovertime as the basis for their performance measure

More generally Cayirli and Veral [2] as well asGupta and Denton [4] provide overviews of ap-pointment scheduling research Kim and Giachetti[5] present a stochastic model that seeks to addressthe question of how many patients to book in ad-vance given a known distribution for walk-in demandand no-shows The no-show probability is dependentsolely on the advance booking policy (ABP) LaGangaand Lawrence [7] also demonstrate the advantage ofoverbooking in a stochastic model that computes theexpected benefit from overbooking Neither paperhowever provides a dynamic scheduling policy wheredecisions can be dependent on the state of the bookingslate More recently Muthuraman and Lawley [11]model a clinic with a fixed number of ldquoslotsrdquo and whereunfinished demand ldquooverflowsrdquo into the next slot Theyprovide a myopic scheduling policy that maximizesrevenue but that does not take into account futuredemand Patients are added to the booking slate untilthe expected profit ceases to increase at which point

demand is rejected Zeng et al [13] build on the modelof Muthuraman and Lawley by allowing the no-showprobabilities to vary between patients They proposetwo sequential scheduling algorithmsmdashone that is my-opic in a similar fashion to Muthuraman and anotherthat attempts to include future demand in the decisionprocess through a forecasting model

Finally Liu et al [9] provide a dynamic program-ming approach that takes into account future demandand allows for state dependent cancelation and no-show rates Their model tracks the number of bookedappointments that are i days out and that were bookedj days in advance This creates an intractable MDP dueto the size of the state space They therefore resort toa one-step policy iteration to improve the policy anddemonstrate (in line with the work of Robinson andChen) that a two day booking window outperformssame day booking and that through their one-step pol-icy iteration they can improve on any initial policy

This research adds to the above literature by provid-ing a dynamic program that can be solved to optimal-ity allows for wait time dependent no-show rates andprovides clear evidence that a short booking windowwith overbooking can provide greater benefit to a clinicthan a strict adherence to OA The cost structure inthe model is sufficiently flexible to provide a dynamicscheduling policy for a variety of clinics and the sim-ulation results demonstrate the distinct advantage ofthe scheduling policy derived from the MDP for all theclinic types tested

3 MDP model for clinic scheduling

Scheduling decisions are assumed to be made beforeeach service period but after todayrsquos demand has ar-rived Ideally the probability of a given client keep-ing hisher appointment would depend on how far inadvance the client has been scheduled However theMDP model would quickly become intractable if theactual waiting time of each client was incorporated intothe state space Instead the assumption that clients whoare booked in advance are given the first available sloton a first-come-first-served basis allows the size of thequeue to act as a proxy for wait time

The ldquoadvanced booking policyrdquo (ABP) is defined asthe number of clients who can be booked in advanceinto each future day The schedule is not constrainedto book only C clients per day as the potential forno-shows may in fact make overbooking desirable andthe possibility of same day bookings may make under-booking desirable

A Markov decision model for determining optimal outpatient scheduling 93

The state is represented by a vector s = (w x y)with w representing the current ABP x representingthe number of previously booked appointments andy representing new demand The ABP is restricted tothe set 1 M with M gt C in order to allow forlimited overbooking Thus M represents the maximumnumber of patients who can be booked in advanceinto each future day The total number of advancedbookings are also restricted so that x isin 0 N forsome finite N gt 0 Thus N is the maximum size ofthe queue x and y and x or y are used to represent theminimum and maximum of x and y respectively

Each decision epoch two decisions are required ofthe manager She must decide whether to change thecurrent ABP as well as how much of the new demandto service today Let a = (a b) represent the combinedaction where a represents the new ABP and b repre-sents how much new demand to serve today In thisversion of the model any change in the ABP takesplace by the next day

Actions are constrained in four ways First bookingscannot exceed demand so b le y Second the number ofpatients in the advance booking slate cannot exceed theimposed limit so b ge [x minus x and w + y minus N]+ (number oftodayrsquos demand treated today must be at least equalto the current number of bookings minus minus todayrsquosbookings + demand minus limit on the number of advancedbookings) Thirdly the new ABP is no more than 1patient per day different from the previous ABP anddoes not deviate outside the imposed limits so 1 or (w minus1) le a le (w + 1) and M The restriction to only a onepatient per day change in the ABP is simply to limit theproblem to a reasonable size and due to the doubtfulbenefit of making radical changes Finally all actionsare positive and integer so (a b) isin Z + times Z +

The transition to the next state can be described as

s = (w x y) rarr sprime = (a x minus x and w + y minus b D) (1)

where D is a random variable representing new de-mand The new booking slate consists of the previousbooking slate minus todayrsquos slate (x and w) plus any ofyesterdayrsquos demand that was not served immediately(y minus b)

There are a number of potential rewardscosts as-sociated with booking patients to a clinic There maybe some revenue associated with each patient servicedf R a cost to servicing a patient through overtime f OT a cost associated with idle time f IT and a cost asso-ciated with patient appointment lead times f LTmdashthatis the number of days between the request for serviceand the date of service Service times are assumed tobe deterministic so that overtime and idle time costs

are simply a function of the number of patients whoshow up to their appointment and capacity Finally inorder to avoid changes in the ABP that donrsquot resultin a major benefit a cost for switching the ABP f Scan be imposed The expected reward function can bewritten as

r(s a) = f R(ps(w x)(w and x) + psdb)

minus f OTsum

(i j)isinWtimesB

(i+ jminusC)+ Pr(AB= i)Pr(SD= j)

minus f ITsum

(i j)isinWtimesB

(Cminusiminus j)+ Pr(AB= i)Pr(SD= j)

minus f LT x + f S(w minus a)+ (2)

where ps(w x) is the probability that an advance book-ing shows up (henceforth called the show probability)given there are x clients in the queue and w is thecurrent ABP psd is the show probability for a same-daybooking W is the set of possible values for w and x andB is the set of possible values for b AB is a randomvariable representing the number of clients booked inadvance for an appointment today who show for theirappointment and SD is a random variable representingthe number of clients given a same day appointment fortoday who show for their appointment Obviously it ispossible to ldquoshut offrdquo any element of the reward func-tion to better reflect the objectives of the actual clinicThe costsrewards can be viewed as a trade-off betweenphysician- or system-related measures (revenue over-time and idle time) and patient-related measures (leadtime)

The choice for the probability of a client arriving forhisher appointment is clearly crucial Kopach et al [6]provide a logistic regression (with an R2 = 08071) foran outpatient clinic that includes age session (whetherthe client is booked into the morning or afternoon)weather and insurance type as predictors of the showprobability They recognize that appointment lead timeis also crucial but had no data to substantiate the actualimpact They therefore adjust the show probability byan exponential function based on lead times deter-mined by ldquoexpertrdquo opinion Gallucci et al [3] demon-strate for a community mental health center that ap-pointment lead time does in fact have a significantimpact with that impact stabilizing for a lead timegreater than 7 days Of 5901 patients in the data set31 failed to show up for their appointment The showprobability for same-day appointments was 88 anddropped to 77 for next day appointments Appoint-ments booked 7 days out had a show probability of 58while a 13 day delay only dropped the show probabilityto 56 Finally Lee et al [8] use a logistic regression

94 J Patrick

model to demonstrate that age race appointment leadtime previously failed appointments provision of a cellphone number and distance from the hospital wereall significant factors in predicting show probabilitiesHowever their appointment lead time analysis onlycompares lead times less than 7 days to lead timesgreater than 21 days

While it would be ideal to incorporate all the factorsmentioned above such detail would make the modelintractable Thus this research focuses only on ap-pointment lead time as the factor that is most clearlyimpacted by the scheduling policy To that end thefollowing show probability is used

ps(w x) = max(

1 minus β1 + β2 lowast log(LT + 1)

100 β3

)(3)

with LT representing appointment lead time The logfunction easily models the Gallucci results (with β1 =12 β2 = 3654 β3 sim= 5) that impose a diminishing im-pact of an extra dayrsquos wait the further out a clientis booked β3 represents a lower bound on the showprobability of any client The addition of one to LTinside the log function insures that the show probabilityfor a same day booking (LT = 0) is psd = 1 minus β1100For advance bookings

LT = max(1 xw) (4)

where xw is the largest integer smaller than xwThe maximum reflects the fact that even if the num-ber of bookings is less than the ABP xw lt 1 anyclient booked in advance still waits a day Equation 4allows the ABP to impact on the relationship betweenappointment lead times and queue size (if you allowmore clients to be booked per day then the same xreflects less of a wait) and therefore better reflects anequal show probability for patients booked an equalnumber of days out

The above description provides the five elements(decision epochs state space action space costs andtransition probabilities) for a Markov Decision Process(MDP) formulation An infinite horizon setting waschosen as clinics do not have a closing date and thus anyend to the scheduling horizon is necessarily arbitraryThis forces the model to ignore any ldquoday-of-the-weekrdquoeffect but avoids any arbitrary terminal rewards

The discounted version of the MDP is used to reflectthe fact that upfront costs are always more expensivethan future costs However a discount factor of γ =099 is used so as not to unduly bias the model againstthe future The standard optimality equations for a

discounted infinite horizon MDP of this initial modelcan therefore be written as

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)

times v (a x minus x and w + y minus b k)

(5)

where pd(k) is the probability that D = k

31 Assumptions to the model

As with any modeling exercise there are a number ofassumptions that help keep the model tractable Firstclient service times are assumed to be deterministicand we do not allow for advance notice of cancela-tions Second as mentioned in Section 3 the modelassumes that all demand for a given day has arrivedbefore any scheduling decision needs to be made Inreality scheduling decisions have to be made as eachdemand request arrives However batch arrivals allowsdecisions to be made in discrete time and the resultingpolicy as will be demonstrated in Section 5 is easilytranslatable into the more realistic setting where thescheduling decision occurs at the time of each demandrequest Third the utilization of the size of the queue asa proxy for wait time when calculating the probabilitythat a client shows up for an appointment means thatthe no-show rate of a client will depend on the sizeof the queue at the time of service when in reality itdepends on the size of the queue at the time of bookingWhile this may mean that an individualrsquos no-show ratemay not be accurate it will still penalize a system thatbooks well in advance to the same degree as a morerealistic (but intractable) model It is also an issue onlyif the length of the queue varies dramatically whereasthe policy derived from the MDP model shows no suchvariation Fourth for the sake of simplicity the issue ofpatient availability is ignored as each client is assumedto take the offered slot To do otherwise would requirethe model to track actual appointment lead times whichwould again lead to an intractable model Finally inthe model described in Section 3 changes in the ABPoccur before the next decision epoch This may meanthat some appointments need to be postponed (if theABP is lowered) or that some clients may need tohave their appointment day advanced (if the ABP isincreased) A more complex model that includes a lagtime in the enactment of any change in the ABP isdeveloped in Section 33 in order to avoid this problemHowever a small cost for any change to the ABPforestalls this issue as it tends to result in the optimalpolicy settling on a single ABP Such a cost is imposed

A Markov decision model for determining optimal outpatient scheduling 95

in the simulation results presented in Section 4 in orderto insure a simpler policy that is easily implementableIn all simulation runs in which the model was run withand without a cost for changing the ABP the differencein the objective between the more complex policy andthe simpler one was not statistically significant Thenext two subsections provide two complications to themodel that allow for more realism

32 Complication 1 two stream demand

To add an additional element of realism the model caneasily be adapted to allow for a stochastic stream ofclients who must be booked in advance The transitionswould then become

s = (w x y) rarr sprime = (a x minus x and w + y minus b + A D)

(6)

where A is a random variable representing the demandrequiring advanced booking The optimality equationsare now

v(s) = maxaisinA(s)

r(s a) + γ

sum

iisinD jisinA

pd(i)pa( j)

times v (a x minus x and w + y minus b + j i)

foralls isin S

(7)

where A is the set of possible advanced booking re-quests and pa( j) is the probability of getting j advancedbooking requests in a day Since advanced bookingrequests (for a clinic running OA) generally arise as

follow-up visits from todayrsquos appointments the numberof such visits is assumed to be unknown until aftertodayrsquos booking decisions are made

An additional issue that this complication to themodel raises is that it is now impossible to insure thatthe booking slate does not exceed N clients simply byrestricting the action set To avoid this issue a clientrsquosrequest for an appointment is rejected at a high costshould accepting it cause the booking slate to exceedN Making this cost sufficiently high insures that suchextreme measures are never required

33 Complication 2 lag time in ABP changes

As mentioned earlier it is somewhat unrealistic toexpect a change in the ABP to be implemented imme-diately However the model can be adjusted to imposea lag time on any change in the ABP Let the newstate s = (w x y z) where w x and y are as beforeand z represents the lag time remaining until the nexttriggered ABP change The lag can be set to xw or 1to insure that the lag is equal to the number of daysthat are already fully booked and thus no previouslybooked appointments need to be canceled and no newbookings jump the queue If z lt 0 then the decision isto reduce the ABP by one and if z gt 0 then the decisionis to increase the ABP by one More drastic changes arenot considered

The same actions are available to the manager withthe same restrictions The evolution of the system de-pends on the current lag time and can be described bythe following set of transitions

s = (w x y z) rarr

sprime =

⎧⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎩

(w x minus x and w + y minus b D l + 1) If l lt minus1(w x minus x and w + y minus b D l minus 1) If l gt 1(a x minus x and w + y minus b D 0) If l = minus1 xw le 1 or l = 1(w x minus x and w + y minus b D minusxw minus 1) If l = 0 xw gt 1 and a lt w(w x minus x and w + y minus b D xw minus 1) If l = 0 xw gt 1 and a gt w(w x minus x and w + y minus b D 0) If l = 0 and a = w

(8)

Here D is again a random variable representing newdemand Notice that once a decision to change theABP is made no new change can be made until theday when that decision is implemented If tomorrowrsquosslate is not yet full (xw le 1) then the ABP change isimplemented by tomorrow as in the original model

Finally the optimality equations are

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)v(sprime)

(9)

where D represents the set of all possible demand pd

is the probability distribution for new demand and sprime

is determined by the transitions given in Eq 8 withD = k

4 Simulation results

In this section the simulation results are presentedthat compare the MDP policy to OA in a variety of

96 J Patrick

scenarios that are chosen to explore the trade-off be-tween the system-related costsrewards (revenue over-time and idle time) and the patient-related ones (leadtime) We consider three clinic types for the system-related costsrewards The first clinic ignores idle timeand sets revenue to twice the cost of overtime f R =20 f OT = 10 and f IT = 0 The second clinic adds anadditional cost for idle time f R = 20 f OT = 10 andf IT = 5 The third clinic does not receive remunera-tion for each patient seen but seeks to maximize theutilization of the available capacity to meet all demandf R = 0 f OT = 10 and f IT = 5 In all cases the costof changing the ABP is set equal to the cost of oneovertime slot The relative values of the cost parame-ters is somewhat arbitrary We have followed commonpractice of valuing idle time (when it is consideredimportant) at half the cost of overtime and have chosento set revenue at twice the value of overtime Therationale for these choices is to represent both clinicsthat are privately run or where the physician is paid ona fee-for-service basis (so that revenue is important) aswell as clinics that are publicly funded and thereforewhere revenue is not a factor

For each of these clinics lead time costs are set at 01 and 5 leading to a total of 9 clinic types differentiatedby the relative weight they place on each costreward Ifthe cost of a dayrsquos lead time is set equal to the overtimecost then the optimal policy would clearly be OA Thuslead time costs of 1 and 5 seem reasonable optionsfor demonstrating the impact of increasing the valueof providing same-day service For each clinic type weconsider 6 scenarios The base case for each clinic as-sumes a Poisson arrival rate and sets average demandλ equal to capacity C = 10 Such a small clinic sizeis chosen in order to restrict the amount of computa-tion time for running numerous scenarios Larger clinicsizes can obviously be solved The Gallucci show-ratedescribed in Section 3 is used to determine show ratesAdmittedly a mental health clinic may not be the moststandard clinic upon which to base the no-show ratesHowever the intent is to demonstrate the benefit of a

short booking window in spite of the presence of no-shows thus using a show-rate that is perhaps overly pes-simistic is reasonable Any clinic seeking to implementthe MDP policy would have to determine their ownshow-rate first

In addition to the base case five other scenariosare run for each clinic type The average demand rateis varied above and below the capacity a stream ofdemand is introduced that must be booked in advance(arriving with rate μ) the show rate is given a steeperrate of decline and finally same day bookings aregiven a show probability of one The actual parametervalues for each of these scenarios are summarized inTable 1 No results are provided for the lag time versionof the model as the cost (equal to one overtime slot)associated with switching the ABP results in a policythat does not vary the ABP thus making the lag timeirrelevant Running the model with no switching costleads to a much more complicated policy but providesa statistically insignificant improvement over the policywith a fixed ABP

For each scenario 50 replications each 5000 daysin length are run both for the MDP policy and OAThe two policies are compared on the basis of through-put overtime idle time appointment lead times andprofitcost Statistics are collected after the first 500days

One technical challenge was the calculation of theexpected number of clients scheduled today who showfor their appointment when the show rate is dependenton the actual lead time (as in the simulation) Thechallenge is that each client may have been booked adifferent number of days in advance and thus may havedifferent show probabilities This problem is avoidedin the MDP model by using queue size as a proxy forwait times but in the simulation actual wait times foreach client are used Each day the probability distri-bution of the number of clients who show up for theirappointment based on the scheduled clients for that dayneeds to be calculated This is accomplished based onthe algorithm outlined in [1] Once that probability is

Table 1 Scenarios analyzed Base case (λ = C) λ = 10 μ = 0 β1 = 12 β2 = 3654 β3 = 50Reduced demand (λ = C minus 2) λ = 8 μ = 0 β1 = 12 β2 = 3654 β3 = 50Increased demand (λ = C + 2) λ = 12 μ = 0 β1 = 12 β2 = 3654 β3 = 50Advanced bookings λ = 7 μ = 3 β1 = 12 β2 = 3654 β3 = 50Show rate with steep decline λ = 10 μ = 0

β1 = 12 (same day rate unchanged)β2 = 80 (steeper decline)β3 = 20 (levels out at lower rate)

Show rate with same day = 100 Same as base case except show probability is 1for same day appointments

A Markov decision model for determining optimal outpatient scheduling 97

determined it is an easy step to calculate the expectedrewardscosts each day and thus to obtain the optimalaction based on the argmax of Eq 5

Table 2 provides a comparison of OA and the MDPpolicy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 f OT =10 f IT = 0 and appointment lead time costs of 01and 5 Throughput is given as a percentage of demandwhile overtime and idle time are given as a percentageof capacity The results of the simulation demonstratethat the two policies are essentially equivalent in termsof average daily profit with the MDP policy slightlyoutperforming OA in most scenarios It is not surpris-ing that OA performs relatively well for a clinic whererevenue is present since any no-show is detrimental inthat it reduces revenue even if overtime is requiredThus a policy that minimizes the number of no-shows(which is the aim of OA) will clearly have an advantageThe surprise perhaps is that one can do as well byreducing overtime and idle time through a more sophis-ticated scheduling policy even if that policy implies thatthere will be less throughput The crucial additional

advantage of the MDP policy is that the day-to-dayworkload is significantly less variable (see Fig 1) andthe peak load is significantly reduced (from 20 to 15 inthe base case with zero lead time costs for instance)

Table 3 provides the comparison of OA and theMDP policy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 F OT =10 f IT = 5 and appointment lead time costs of 0 1 and5 These clinics demonstrate relatively similar results tothose clinics that ignored idle time as revenue is stillthe primary driver Thus the two policies are essen-tially equivalent in terms of average daily profit withthe MDP policy outperforming OA by slightly highermargins than for the previous clinics Again there isa significant reduction in the peak workload and asmoothing of the variation in daily throughput (seeFig 2)

Finally Table 4 provides the comparison of OAand the MDP policy for each of the scenarios forthe clinics with system-related parameters equal tof R = 0 F OT = 10 f IT = 5 and appointment lead timecosts of 0 1 and 5 For such clinics the MDP policy

Table 2 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 0

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19540demand MDP 0 867 114 74 1 19669 07

1 875 137 87 1 19596 015 880 157 102 0 19540 00

Show rate OA 1000 125 125 0 18750with same MDP 0 982 83 101 1 18806 03day = 100 1 991 101 111 1 18758 00

5 1000 125 125 0 1875 00Base case OA 880 69 189 0 16908

MDP 0 868 31 163 1 17051 081 880 69 189 0 16908 005 880 69 189 0 16908 00

Show rate OA 880 69 189 0 16908with steep MDP 0 872 49 178 1 16939 02decline 1 878 64 186 1 16911 00

5 880 69 189 0 16908 00Advanced OA 846 56 210 1 16066

bookings MDP 0 837 26 189 2 16483 261 840 34 194 2 16116 035 846 56 210 1 16066 00

Decreased OA 880 21 317 0 13873demand MDP 0 875 05 305 1 13945 05

1 876 07 307 1 13912 035 880 19 315 1 13879 00

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

92 J Patrick

and clinic scheduling is presented and how our ap-proach differs from what has been done previously isdemonstrated In Section 3 the MDP model for theclinic scheduling problem is described as well as someof the assumptions inherent in the model and twocomplications that seek to overcome some of those as-sumptions In Section 4 the results of testing the MDPpolicy against OA in a simulation are presented and inSection 5 the form of the MDP policy is describedFinally Section 6 presents the conclusions from thisresearch

2 Literature review

Murray and Tantau [10] first proposed the idea ofOA as a solution to the high levels of no-shows oftenpresent in outpatient clinics They provide a case studyof a successful implementation of open access in theUS Kopach et al [6] provide a simulation study todetermine the impact of various clinic characteristics onthe successful implementation of OA Their simulationallows for a multi-doctor clinic and patient specific no-show probabilities They allow for limited overbookingin some of the scenarios analyzed in the simulationRobinson and Chen [12] demonstrate that OA mostoften outperforms a traditional booking system (witha fixed number of patients per day) but is itself out-performed by a ldquosame-or-next-dayrdquo scheduling policythus demonstrating that there is some advantage toflexibility in scheduling They use a weighted sum ofphysician idle time direct patient waiting time andovertime as the basis for their performance measure

More generally Cayirli and Veral [2] as well asGupta and Denton [4] provide overviews of ap-pointment scheduling research Kim and Giachetti[5] present a stochastic model that seeks to addressthe question of how many patients to book in ad-vance given a known distribution for walk-in demandand no-shows The no-show probability is dependentsolely on the advance booking policy (ABP) LaGangaand Lawrence [7] also demonstrate the advantage ofoverbooking in a stochastic model that computes theexpected benefit from overbooking Neither paperhowever provides a dynamic scheduling policy wheredecisions can be dependent on the state of the bookingslate More recently Muthuraman and Lawley [11]model a clinic with a fixed number of ldquoslotsrdquo and whereunfinished demand ldquooverflowsrdquo into the next slot Theyprovide a myopic scheduling policy that maximizesrevenue but that does not take into account futuredemand Patients are added to the booking slate untilthe expected profit ceases to increase at which point

demand is rejected Zeng et al [13] build on the modelof Muthuraman and Lawley by allowing the no-showprobabilities to vary between patients They proposetwo sequential scheduling algorithmsmdashone that is my-opic in a similar fashion to Muthuraman and anotherthat attempts to include future demand in the decisionprocess through a forecasting model

Finally Liu et al [9] provide a dynamic program-ming approach that takes into account future demandand allows for state dependent cancelation and no-show rates Their model tracks the number of bookedappointments that are i days out and that were bookedj days in advance This creates an intractable MDP dueto the size of the state space They therefore resort toa one-step policy iteration to improve the policy anddemonstrate (in line with the work of Robinson andChen) that a two day booking window outperformssame day booking and that through their one-step pol-icy iteration they can improve on any initial policy

This research adds to the above literature by provid-ing a dynamic program that can be solved to optimal-ity allows for wait time dependent no-show rates andprovides clear evidence that a short booking windowwith overbooking can provide greater benefit to a clinicthan a strict adherence to OA The cost structure inthe model is sufficiently flexible to provide a dynamicscheduling policy for a variety of clinics and the sim-ulation results demonstrate the distinct advantage ofthe scheduling policy derived from the MDP for all theclinic types tested

3 MDP model for clinic scheduling

Scheduling decisions are assumed to be made beforeeach service period but after todayrsquos demand has ar-rived Ideally the probability of a given client keep-ing hisher appointment would depend on how far inadvance the client has been scheduled However theMDP model would quickly become intractable if theactual waiting time of each client was incorporated intothe state space Instead the assumption that clients whoare booked in advance are given the first available sloton a first-come-first-served basis allows the size of thequeue to act as a proxy for wait time

The ldquoadvanced booking policyrdquo (ABP) is defined asthe number of clients who can be booked in advanceinto each future day The schedule is not constrainedto book only C clients per day as the potential forno-shows may in fact make overbooking desirable andthe possibility of same day bookings may make under-booking desirable

A Markov decision model for determining optimal outpatient scheduling 93

The state is represented by a vector s = (w x y)with w representing the current ABP x representingthe number of previously booked appointments andy representing new demand The ABP is restricted tothe set 1 M with M gt C in order to allow forlimited overbooking Thus M represents the maximumnumber of patients who can be booked in advanceinto each future day The total number of advancedbookings are also restricted so that x isin 0 N forsome finite N gt 0 Thus N is the maximum size ofthe queue x and y and x or y are used to represent theminimum and maximum of x and y respectively

Each decision epoch two decisions are required ofthe manager She must decide whether to change thecurrent ABP as well as how much of the new demandto service today Let a = (a b) represent the combinedaction where a represents the new ABP and b repre-sents how much new demand to serve today In thisversion of the model any change in the ABP takesplace by the next day

Actions are constrained in four ways First bookingscannot exceed demand so b le y Second the number ofpatients in the advance booking slate cannot exceed theimposed limit so b ge [x minus x and w + y minus N]+ (number oftodayrsquos demand treated today must be at least equalto the current number of bookings minus minus todayrsquosbookings + demand minus limit on the number of advancedbookings) Thirdly the new ABP is no more than 1patient per day different from the previous ABP anddoes not deviate outside the imposed limits so 1 or (w minus1) le a le (w + 1) and M The restriction to only a onepatient per day change in the ABP is simply to limit theproblem to a reasonable size and due to the doubtfulbenefit of making radical changes Finally all actionsare positive and integer so (a b) isin Z + times Z +

The transition to the next state can be described as

s = (w x y) rarr sprime = (a x minus x and w + y minus b D) (1)

where D is a random variable representing new de-mand The new booking slate consists of the previousbooking slate minus todayrsquos slate (x and w) plus any ofyesterdayrsquos demand that was not served immediately(y minus b)

There are a number of potential rewardscosts as-sociated with booking patients to a clinic There maybe some revenue associated with each patient servicedf R a cost to servicing a patient through overtime f OT a cost associated with idle time f IT and a cost asso-ciated with patient appointment lead times f LTmdashthatis the number of days between the request for serviceand the date of service Service times are assumed tobe deterministic so that overtime and idle time costs

are simply a function of the number of patients whoshow up to their appointment and capacity Finally inorder to avoid changes in the ABP that donrsquot resultin a major benefit a cost for switching the ABP f Scan be imposed The expected reward function can bewritten as

r(s a) = f R(ps(w x)(w and x) + psdb)

minus f OTsum

(i j)isinWtimesB

(i+ jminusC)+ Pr(AB= i)Pr(SD= j)

minus f ITsum

(i j)isinWtimesB

(Cminusiminus j)+ Pr(AB= i)Pr(SD= j)

minus f LT x + f S(w minus a)+ (2)

where ps(w x) is the probability that an advance book-ing shows up (henceforth called the show probability)given there are x clients in the queue and w is thecurrent ABP psd is the show probability for a same-daybooking W is the set of possible values for w and x andB is the set of possible values for b AB is a randomvariable representing the number of clients booked inadvance for an appointment today who show for theirappointment and SD is a random variable representingthe number of clients given a same day appointment fortoday who show for their appointment Obviously it ispossible to ldquoshut offrdquo any element of the reward func-tion to better reflect the objectives of the actual clinicThe costsrewards can be viewed as a trade-off betweenphysician- or system-related measures (revenue over-time and idle time) and patient-related measures (leadtime)

The choice for the probability of a client arriving forhisher appointment is clearly crucial Kopach et al [6]provide a logistic regression (with an R2 = 08071) foran outpatient clinic that includes age session (whetherthe client is booked into the morning or afternoon)weather and insurance type as predictors of the showprobability They recognize that appointment lead timeis also crucial but had no data to substantiate the actualimpact They therefore adjust the show probability byan exponential function based on lead times deter-mined by ldquoexpertrdquo opinion Gallucci et al [3] demon-strate for a community mental health center that ap-pointment lead time does in fact have a significantimpact with that impact stabilizing for a lead timegreater than 7 days Of 5901 patients in the data set31 failed to show up for their appointment The showprobability for same-day appointments was 88 anddropped to 77 for next day appointments Appoint-ments booked 7 days out had a show probability of 58while a 13 day delay only dropped the show probabilityto 56 Finally Lee et al [8] use a logistic regression

94 J Patrick

model to demonstrate that age race appointment leadtime previously failed appointments provision of a cellphone number and distance from the hospital wereall significant factors in predicting show probabilitiesHowever their appointment lead time analysis onlycompares lead times less than 7 days to lead timesgreater than 21 days

While it would be ideal to incorporate all the factorsmentioned above such detail would make the modelintractable Thus this research focuses only on ap-pointment lead time as the factor that is most clearlyimpacted by the scheduling policy To that end thefollowing show probability is used

ps(w x) = max(

1 minus β1 + β2 lowast log(LT + 1)

100 β3

)(3)

with LT representing appointment lead time The logfunction easily models the Gallucci results (with β1 =12 β2 = 3654 β3 sim= 5) that impose a diminishing im-pact of an extra dayrsquos wait the further out a clientis booked β3 represents a lower bound on the showprobability of any client The addition of one to LTinside the log function insures that the show probabilityfor a same day booking (LT = 0) is psd = 1 minus β1100For advance bookings

LT = max(1 xw) (4)

where xw is the largest integer smaller than xwThe maximum reflects the fact that even if the num-ber of bookings is less than the ABP xw lt 1 anyclient booked in advance still waits a day Equation 4allows the ABP to impact on the relationship betweenappointment lead times and queue size (if you allowmore clients to be booked per day then the same xreflects less of a wait) and therefore better reflects anequal show probability for patients booked an equalnumber of days out

The above description provides the five elements(decision epochs state space action space costs andtransition probabilities) for a Markov Decision Process(MDP) formulation An infinite horizon setting waschosen as clinics do not have a closing date and thus anyend to the scheduling horizon is necessarily arbitraryThis forces the model to ignore any ldquoday-of-the-weekrdquoeffect but avoids any arbitrary terminal rewards

The discounted version of the MDP is used to reflectthe fact that upfront costs are always more expensivethan future costs However a discount factor of γ =099 is used so as not to unduly bias the model againstthe future The standard optimality equations for a

discounted infinite horizon MDP of this initial modelcan therefore be written as

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)

times v (a x minus x and w + y minus b k)

(5)

where pd(k) is the probability that D = k

31 Assumptions to the model

As with any modeling exercise there are a number ofassumptions that help keep the model tractable Firstclient service times are assumed to be deterministicand we do not allow for advance notice of cancela-tions Second as mentioned in Section 3 the modelassumes that all demand for a given day has arrivedbefore any scheduling decision needs to be made Inreality scheduling decisions have to be made as eachdemand request arrives However batch arrivals allowsdecisions to be made in discrete time and the resultingpolicy as will be demonstrated in Section 5 is easilytranslatable into the more realistic setting where thescheduling decision occurs at the time of each demandrequest Third the utilization of the size of the queue asa proxy for wait time when calculating the probabilitythat a client shows up for an appointment means thatthe no-show rate of a client will depend on the sizeof the queue at the time of service when in reality itdepends on the size of the queue at the time of bookingWhile this may mean that an individualrsquos no-show ratemay not be accurate it will still penalize a system thatbooks well in advance to the same degree as a morerealistic (but intractable) model It is also an issue onlyif the length of the queue varies dramatically whereasthe policy derived from the MDP model shows no suchvariation Fourth for the sake of simplicity the issue ofpatient availability is ignored as each client is assumedto take the offered slot To do otherwise would requirethe model to track actual appointment lead times whichwould again lead to an intractable model Finally inthe model described in Section 3 changes in the ABPoccur before the next decision epoch This may meanthat some appointments need to be postponed (if theABP is lowered) or that some clients may need tohave their appointment day advanced (if the ABP isincreased) A more complex model that includes a lagtime in the enactment of any change in the ABP isdeveloped in Section 33 in order to avoid this problemHowever a small cost for any change to the ABPforestalls this issue as it tends to result in the optimalpolicy settling on a single ABP Such a cost is imposed

A Markov decision model for determining optimal outpatient scheduling 95

in the simulation results presented in Section 4 in orderto insure a simpler policy that is easily implementableIn all simulation runs in which the model was run withand without a cost for changing the ABP the differencein the objective between the more complex policy andthe simpler one was not statistically significant Thenext two subsections provide two complications to themodel that allow for more realism

32 Complication 1 two stream demand

To add an additional element of realism the model caneasily be adapted to allow for a stochastic stream ofclients who must be booked in advance The transitionswould then become

s = (w x y) rarr sprime = (a x minus x and w + y minus b + A D)

(6)

where A is a random variable representing the demandrequiring advanced booking The optimality equationsare now

v(s) = maxaisinA(s)

r(s a) + γ

sum

iisinD jisinA

pd(i)pa( j)

times v (a x minus x and w + y minus b + j i)

foralls isin S

(7)

where A is the set of possible advanced booking re-quests and pa( j) is the probability of getting j advancedbooking requests in a day Since advanced bookingrequests (for a clinic running OA) generally arise as

follow-up visits from todayrsquos appointments the numberof such visits is assumed to be unknown until aftertodayrsquos booking decisions are made

An additional issue that this complication to themodel raises is that it is now impossible to insure thatthe booking slate does not exceed N clients simply byrestricting the action set To avoid this issue a clientrsquosrequest for an appointment is rejected at a high costshould accepting it cause the booking slate to exceedN Making this cost sufficiently high insures that suchextreme measures are never required

33 Complication 2 lag time in ABP changes

As mentioned earlier it is somewhat unrealistic toexpect a change in the ABP to be implemented imme-diately However the model can be adjusted to imposea lag time on any change in the ABP Let the newstate s = (w x y z) where w x and y are as beforeand z represents the lag time remaining until the nexttriggered ABP change The lag can be set to xw or 1to insure that the lag is equal to the number of daysthat are already fully booked and thus no previouslybooked appointments need to be canceled and no newbookings jump the queue If z lt 0 then the decision isto reduce the ABP by one and if z gt 0 then the decisionis to increase the ABP by one More drastic changes arenot considered

The same actions are available to the manager withthe same restrictions The evolution of the system de-pends on the current lag time and can be described bythe following set of transitions

s = (w x y z) rarr

sprime =

⎧⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎩

(w x minus x and w + y minus b D l + 1) If l lt minus1(w x minus x and w + y minus b D l minus 1) If l gt 1(a x minus x and w + y minus b D 0) If l = minus1 xw le 1 or l = 1(w x minus x and w + y minus b D minusxw minus 1) If l = 0 xw gt 1 and a lt w(w x minus x and w + y minus b D xw minus 1) If l = 0 xw gt 1 and a gt w(w x minus x and w + y minus b D 0) If l = 0 and a = w

(8)

Here D is again a random variable representing newdemand Notice that once a decision to change theABP is made no new change can be made until theday when that decision is implemented If tomorrowrsquosslate is not yet full (xw le 1) then the ABP change isimplemented by tomorrow as in the original model

Finally the optimality equations are

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)v(sprime)

(9)

where D represents the set of all possible demand pd

is the probability distribution for new demand and sprime

is determined by the transitions given in Eq 8 withD = k

4 Simulation results

In this section the simulation results are presentedthat compare the MDP policy to OA in a variety of

96 J Patrick

scenarios that are chosen to explore the trade-off be-tween the system-related costsrewards (revenue over-time and idle time) and the patient-related ones (leadtime) We consider three clinic types for the system-related costsrewards The first clinic ignores idle timeand sets revenue to twice the cost of overtime f R =20 f OT = 10 and f IT = 0 The second clinic adds anadditional cost for idle time f R = 20 f OT = 10 andf IT = 5 The third clinic does not receive remunera-tion for each patient seen but seeks to maximize theutilization of the available capacity to meet all demandf R = 0 f OT = 10 and f IT = 5 In all cases the costof changing the ABP is set equal to the cost of oneovertime slot The relative values of the cost parame-ters is somewhat arbitrary We have followed commonpractice of valuing idle time (when it is consideredimportant) at half the cost of overtime and have chosento set revenue at twice the value of overtime Therationale for these choices is to represent both clinicsthat are privately run or where the physician is paid ona fee-for-service basis (so that revenue is important) aswell as clinics that are publicly funded and thereforewhere revenue is not a factor

For each of these clinics lead time costs are set at 01 and 5 leading to a total of 9 clinic types differentiatedby the relative weight they place on each costreward Ifthe cost of a dayrsquos lead time is set equal to the overtimecost then the optimal policy would clearly be OA Thuslead time costs of 1 and 5 seem reasonable optionsfor demonstrating the impact of increasing the valueof providing same-day service For each clinic type weconsider 6 scenarios The base case for each clinic as-sumes a Poisson arrival rate and sets average demandλ equal to capacity C = 10 Such a small clinic sizeis chosen in order to restrict the amount of computa-tion time for running numerous scenarios Larger clinicsizes can obviously be solved The Gallucci show-ratedescribed in Section 3 is used to determine show ratesAdmittedly a mental health clinic may not be the moststandard clinic upon which to base the no-show ratesHowever the intent is to demonstrate the benefit of a

short booking window in spite of the presence of no-shows thus using a show-rate that is perhaps overly pes-simistic is reasonable Any clinic seeking to implementthe MDP policy would have to determine their ownshow-rate first

In addition to the base case five other scenariosare run for each clinic type The average demand rateis varied above and below the capacity a stream ofdemand is introduced that must be booked in advance(arriving with rate μ) the show rate is given a steeperrate of decline and finally same day bookings aregiven a show probability of one The actual parametervalues for each of these scenarios are summarized inTable 1 No results are provided for the lag time versionof the model as the cost (equal to one overtime slot)associated with switching the ABP results in a policythat does not vary the ABP thus making the lag timeirrelevant Running the model with no switching costleads to a much more complicated policy but providesa statistically insignificant improvement over the policywith a fixed ABP

For each scenario 50 replications each 5000 daysin length are run both for the MDP policy and OAThe two policies are compared on the basis of through-put overtime idle time appointment lead times andprofitcost Statistics are collected after the first 500days

One technical challenge was the calculation of theexpected number of clients scheduled today who showfor their appointment when the show rate is dependenton the actual lead time (as in the simulation) Thechallenge is that each client may have been booked adifferent number of days in advance and thus may havedifferent show probabilities This problem is avoidedin the MDP model by using queue size as a proxy forwait times but in the simulation actual wait times foreach client are used Each day the probability distri-bution of the number of clients who show up for theirappointment based on the scheduled clients for that dayneeds to be calculated This is accomplished based onthe algorithm outlined in [1] Once that probability is

Table 1 Scenarios analyzed Base case (λ = C) λ = 10 μ = 0 β1 = 12 β2 = 3654 β3 = 50Reduced demand (λ = C minus 2) λ = 8 μ = 0 β1 = 12 β2 = 3654 β3 = 50Increased demand (λ = C + 2) λ = 12 μ = 0 β1 = 12 β2 = 3654 β3 = 50Advanced bookings λ = 7 μ = 3 β1 = 12 β2 = 3654 β3 = 50Show rate with steep decline λ = 10 μ = 0

β1 = 12 (same day rate unchanged)β2 = 80 (steeper decline)β3 = 20 (levels out at lower rate)

Show rate with same day = 100 Same as base case except show probability is 1for same day appointments

A Markov decision model for determining optimal outpatient scheduling 97

determined it is an easy step to calculate the expectedrewardscosts each day and thus to obtain the optimalaction based on the argmax of Eq 5

Table 2 provides a comparison of OA and the MDPpolicy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 f OT =10 f IT = 0 and appointment lead time costs of 01and 5 Throughput is given as a percentage of demandwhile overtime and idle time are given as a percentageof capacity The results of the simulation demonstratethat the two policies are essentially equivalent in termsof average daily profit with the MDP policy slightlyoutperforming OA in most scenarios It is not surpris-ing that OA performs relatively well for a clinic whererevenue is present since any no-show is detrimental inthat it reduces revenue even if overtime is requiredThus a policy that minimizes the number of no-shows(which is the aim of OA) will clearly have an advantageThe surprise perhaps is that one can do as well byreducing overtime and idle time through a more sophis-ticated scheduling policy even if that policy implies thatthere will be less throughput The crucial additional

advantage of the MDP policy is that the day-to-dayworkload is significantly less variable (see Fig 1) andthe peak load is significantly reduced (from 20 to 15 inthe base case with zero lead time costs for instance)

Table 3 provides the comparison of OA and theMDP policy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 F OT =10 f IT = 5 and appointment lead time costs of 0 1 and5 These clinics demonstrate relatively similar results tothose clinics that ignored idle time as revenue is stillthe primary driver Thus the two policies are essen-tially equivalent in terms of average daily profit withthe MDP policy outperforming OA by slightly highermargins than for the previous clinics Again there isa significant reduction in the peak workload and asmoothing of the variation in daily throughput (seeFig 2)

Finally Table 4 provides the comparison of OAand the MDP policy for each of the scenarios forthe clinics with system-related parameters equal tof R = 0 F OT = 10 f IT = 5 and appointment lead timecosts of 0 1 and 5 For such clinics the MDP policy

Table 2 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 0

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19540demand MDP 0 867 114 74 1 19669 07

1 875 137 87 1 19596 015 880 157 102 0 19540 00

Show rate OA 1000 125 125 0 18750with same MDP 0 982 83 101 1 18806 03day = 100 1 991 101 111 1 18758 00

5 1000 125 125 0 1875 00Base case OA 880 69 189 0 16908

MDP 0 868 31 163 1 17051 081 880 69 189 0 16908 005 880 69 189 0 16908 00

Show rate OA 880 69 189 0 16908with steep MDP 0 872 49 178 1 16939 02decline 1 878 64 186 1 16911 00

5 880 69 189 0 16908 00Advanced OA 846 56 210 1 16066

bookings MDP 0 837 26 189 2 16483 261 840 34 194 2 16116 035 846 56 210 1 16066 00

Decreased OA 880 21 317 0 13873demand MDP 0 875 05 305 1 13945 05

1 876 07 307 1 13912 035 880 19 315 1 13879 00

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

A Markov decision model for determining optimal outpatient scheduling 93

The state is represented by a vector s = (w x y)with w representing the current ABP x representingthe number of previously booked appointments andy representing new demand The ABP is restricted tothe set 1 M with M gt C in order to allow forlimited overbooking Thus M represents the maximumnumber of patients who can be booked in advanceinto each future day The total number of advancedbookings are also restricted so that x isin 0 N forsome finite N gt 0 Thus N is the maximum size ofthe queue x and y and x or y are used to represent theminimum and maximum of x and y respectively

Each decision epoch two decisions are required ofthe manager She must decide whether to change thecurrent ABP as well as how much of the new demandto service today Let a = (a b) represent the combinedaction where a represents the new ABP and b repre-sents how much new demand to serve today In thisversion of the model any change in the ABP takesplace by the next day

Actions are constrained in four ways First bookingscannot exceed demand so b le y Second the number ofpatients in the advance booking slate cannot exceed theimposed limit so b ge [x minus x and w + y minus N]+ (number oftodayrsquos demand treated today must be at least equalto the current number of bookings minus minus todayrsquosbookings + demand minus limit on the number of advancedbookings) Thirdly the new ABP is no more than 1patient per day different from the previous ABP anddoes not deviate outside the imposed limits so 1 or (w minus1) le a le (w + 1) and M The restriction to only a onepatient per day change in the ABP is simply to limit theproblem to a reasonable size and due to the doubtfulbenefit of making radical changes Finally all actionsare positive and integer so (a b) isin Z + times Z +

The transition to the next state can be described as

s = (w x y) rarr sprime = (a x minus x and w + y minus b D) (1)

where D is a random variable representing new de-mand The new booking slate consists of the previousbooking slate minus todayrsquos slate (x and w) plus any ofyesterdayrsquos demand that was not served immediately(y minus b)

There are a number of potential rewardscosts as-sociated with booking patients to a clinic There maybe some revenue associated with each patient servicedf R a cost to servicing a patient through overtime f OT a cost associated with idle time f IT and a cost asso-ciated with patient appointment lead times f LTmdashthatis the number of days between the request for serviceand the date of service Service times are assumed tobe deterministic so that overtime and idle time costs

are simply a function of the number of patients whoshow up to their appointment and capacity Finally inorder to avoid changes in the ABP that donrsquot resultin a major benefit a cost for switching the ABP f Scan be imposed The expected reward function can bewritten as

r(s a) = f R(ps(w x)(w and x) + psdb)

minus f OTsum

(i j)isinWtimesB

(i+ jminusC)+ Pr(AB= i)Pr(SD= j)

minus f ITsum

(i j)isinWtimesB

(Cminusiminus j)+ Pr(AB= i)Pr(SD= j)

minus f LT x + f S(w minus a)+ (2)

where ps(w x) is the probability that an advance book-ing shows up (henceforth called the show probability)given there are x clients in the queue and w is thecurrent ABP psd is the show probability for a same-daybooking W is the set of possible values for w and x andB is the set of possible values for b AB is a randomvariable representing the number of clients booked inadvance for an appointment today who show for theirappointment and SD is a random variable representingthe number of clients given a same day appointment fortoday who show for their appointment Obviously it ispossible to ldquoshut offrdquo any element of the reward func-tion to better reflect the objectives of the actual clinicThe costsrewards can be viewed as a trade-off betweenphysician- or system-related measures (revenue over-time and idle time) and patient-related measures (leadtime)

The choice for the probability of a client arriving forhisher appointment is clearly crucial Kopach et al [6]provide a logistic regression (with an R2 = 08071) foran outpatient clinic that includes age session (whetherthe client is booked into the morning or afternoon)weather and insurance type as predictors of the showprobability They recognize that appointment lead timeis also crucial but had no data to substantiate the actualimpact They therefore adjust the show probability byan exponential function based on lead times deter-mined by ldquoexpertrdquo opinion Gallucci et al [3] demon-strate for a community mental health center that ap-pointment lead time does in fact have a significantimpact with that impact stabilizing for a lead timegreater than 7 days Of 5901 patients in the data set31 failed to show up for their appointment The showprobability for same-day appointments was 88 anddropped to 77 for next day appointments Appoint-ments booked 7 days out had a show probability of 58while a 13 day delay only dropped the show probabilityto 56 Finally Lee et al [8] use a logistic regression

94 J Patrick

model to demonstrate that age race appointment leadtime previously failed appointments provision of a cellphone number and distance from the hospital wereall significant factors in predicting show probabilitiesHowever their appointment lead time analysis onlycompares lead times less than 7 days to lead timesgreater than 21 days

While it would be ideal to incorporate all the factorsmentioned above such detail would make the modelintractable Thus this research focuses only on ap-pointment lead time as the factor that is most clearlyimpacted by the scheduling policy To that end thefollowing show probability is used

ps(w x) = max(

1 minus β1 + β2 lowast log(LT + 1)

100 β3

)(3)

with LT representing appointment lead time The logfunction easily models the Gallucci results (with β1 =12 β2 = 3654 β3 sim= 5) that impose a diminishing im-pact of an extra dayrsquos wait the further out a clientis booked β3 represents a lower bound on the showprobability of any client The addition of one to LTinside the log function insures that the show probabilityfor a same day booking (LT = 0) is psd = 1 minus β1100For advance bookings

LT = max(1 xw) (4)

where xw is the largest integer smaller than xwThe maximum reflects the fact that even if the num-ber of bookings is less than the ABP xw lt 1 anyclient booked in advance still waits a day Equation 4allows the ABP to impact on the relationship betweenappointment lead times and queue size (if you allowmore clients to be booked per day then the same xreflects less of a wait) and therefore better reflects anequal show probability for patients booked an equalnumber of days out

The above description provides the five elements(decision epochs state space action space costs andtransition probabilities) for a Markov Decision Process(MDP) formulation An infinite horizon setting waschosen as clinics do not have a closing date and thus anyend to the scheduling horizon is necessarily arbitraryThis forces the model to ignore any ldquoday-of-the-weekrdquoeffect but avoids any arbitrary terminal rewards

The discounted version of the MDP is used to reflectthe fact that upfront costs are always more expensivethan future costs However a discount factor of γ =099 is used so as not to unduly bias the model againstthe future The standard optimality equations for a

discounted infinite horizon MDP of this initial modelcan therefore be written as

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)

times v (a x minus x and w + y minus b k)

(5)

where pd(k) is the probability that D = k

31 Assumptions to the model

As with any modeling exercise there are a number ofassumptions that help keep the model tractable Firstclient service times are assumed to be deterministicand we do not allow for advance notice of cancela-tions Second as mentioned in Section 3 the modelassumes that all demand for a given day has arrivedbefore any scheduling decision needs to be made Inreality scheduling decisions have to be made as eachdemand request arrives However batch arrivals allowsdecisions to be made in discrete time and the resultingpolicy as will be demonstrated in Section 5 is easilytranslatable into the more realistic setting where thescheduling decision occurs at the time of each demandrequest Third the utilization of the size of the queue asa proxy for wait time when calculating the probabilitythat a client shows up for an appointment means thatthe no-show rate of a client will depend on the sizeof the queue at the time of service when in reality itdepends on the size of the queue at the time of bookingWhile this may mean that an individualrsquos no-show ratemay not be accurate it will still penalize a system thatbooks well in advance to the same degree as a morerealistic (but intractable) model It is also an issue onlyif the length of the queue varies dramatically whereasthe policy derived from the MDP model shows no suchvariation Fourth for the sake of simplicity the issue ofpatient availability is ignored as each client is assumedto take the offered slot To do otherwise would requirethe model to track actual appointment lead times whichwould again lead to an intractable model Finally inthe model described in Section 3 changes in the ABPoccur before the next decision epoch This may meanthat some appointments need to be postponed (if theABP is lowered) or that some clients may need tohave their appointment day advanced (if the ABP isincreased) A more complex model that includes a lagtime in the enactment of any change in the ABP isdeveloped in Section 33 in order to avoid this problemHowever a small cost for any change to the ABPforestalls this issue as it tends to result in the optimalpolicy settling on a single ABP Such a cost is imposed

A Markov decision model for determining optimal outpatient scheduling 95

in the simulation results presented in Section 4 in orderto insure a simpler policy that is easily implementableIn all simulation runs in which the model was run withand without a cost for changing the ABP the differencein the objective between the more complex policy andthe simpler one was not statistically significant Thenext two subsections provide two complications to themodel that allow for more realism

32 Complication 1 two stream demand

To add an additional element of realism the model caneasily be adapted to allow for a stochastic stream ofclients who must be booked in advance The transitionswould then become

s = (w x y) rarr sprime = (a x minus x and w + y minus b + A D)

(6)

where A is a random variable representing the demandrequiring advanced booking The optimality equationsare now

v(s) = maxaisinA(s)

r(s a) + γ

sum

iisinD jisinA

pd(i)pa( j)

times v (a x minus x and w + y minus b + j i)

foralls isin S

(7)

where A is the set of possible advanced booking re-quests and pa( j) is the probability of getting j advancedbooking requests in a day Since advanced bookingrequests (for a clinic running OA) generally arise as

follow-up visits from todayrsquos appointments the numberof such visits is assumed to be unknown until aftertodayrsquos booking decisions are made

An additional issue that this complication to themodel raises is that it is now impossible to insure thatthe booking slate does not exceed N clients simply byrestricting the action set To avoid this issue a clientrsquosrequest for an appointment is rejected at a high costshould accepting it cause the booking slate to exceedN Making this cost sufficiently high insures that suchextreme measures are never required

33 Complication 2 lag time in ABP changes

As mentioned earlier it is somewhat unrealistic toexpect a change in the ABP to be implemented imme-diately However the model can be adjusted to imposea lag time on any change in the ABP Let the newstate s = (w x y z) where w x and y are as beforeand z represents the lag time remaining until the nexttriggered ABP change The lag can be set to xw or 1to insure that the lag is equal to the number of daysthat are already fully booked and thus no previouslybooked appointments need to be canceled and no newbookings jump the queue If z lt 0 then the decision isto reduce the ABP by one and if z gt 0 then the decisionis to increase the ABP by one More drastic changes arenot considered

The same actions are available to the manager withthe same restrictions The evolution of the system de-pends on the current lag time and can be described bythe following set of transitions

s = (w x y z) rarr

sprime =

⎧⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎩

(w x minus x and w + y minus b D l + 1) If l lt minus1(w x minus x and w + y minus b D l minus 1) If l gt 1(a x minus x and w + y minus b D 0) If l = minus1 xw le 1 or l = 1(w x minus x and w + y minus b D minusxw minus 1) If l = 0 xw gt 1 and a lt w(w x minus x and w + y minus b D xw minus 1) If l = 0 xw gt 1 and a gt w(w x minus x and w + y minus b D 0) If l = 0 and a = w

(8)

Here D is again a random variable representing newdemand Notice that once a decision to change theABP is made no new change can be made until theday when that decision is implemented If tomorrowrsquosslate is not yet full (xw le 1) then the ABP change isimplemented by tomorrow as in the original model

Finally the optimality equations are

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)v(sprime)

(9)

where D represents the set of all possible demand pd

is the probability distribution for new demand and sprime

is determined by the transitions given in Eq 8 withD = k

4 Simulation results

In this section the simulation results are presentedthat compare the MDP policy to OA in a variety of

96 J Patrick

scenarios that are chosen to explore the trade-off be-tween the system-related costsrewards (revenue over-time and idle time) and the patient-related ones (leadtime) We consider three clinic types for the system-related costsrewards The first clinic ignores idle timeand sets revenue to twice the cost of overtime f R =20 f OT = 10 and f IT = 0 The second clinic adds anadditional cost for idle time f R = 20 f OT = 10 andf IT = 5 The third clinic does not receive remunera-tion for each patient seen but seeks to maximize theutilization of the available capacity to meet all demandf R = 0 f OT = 10 and f IT = 5 In all cases the costof changing the ABP is set equal to the cost of oneovertime slot The relative values of the cost parame-ters is somewhat arbitrary We have followed commonpractice of valuing idle time (when it is consideredimportant) at half the cost of overtime and have chosento set revenue at twice the value of overtime Therationale for these choices is to represent both clinicsthat are privately run or where the physician is paid ona fee-for-service basis (so that revenue is important) aswell as clinics that are publicly funded and thereforewhere revenue is not a factor

For each of these clinics lead time costs are set at 01 and 5 leading to a total of 9 clinic types differentiatedby the relative weight they place on each costreward Ifthe cost of a dayrsquos lead time is set equal to the overtimecost then the optimal policy would clearly be OA Thuslead time costs of 1 and 5 seem reasonable optionsfor demonstrating the impact of increasing the valueof providing same-day service For each clinic type weconsider 6 scenarios The base case for each clinic as-sumes a Poisson arrival rate and sets average demandλ equal to capacity C = 10 Such a small clinic sizeis chosen in order to restrict the amount of computa-tion time for running numerous scenarios Larger clinicsizes can obviously be solved The Gallucci show-ratedescribed in Section 3 is used to determine show ratesAdmittedly a mental health clinic may not be the moststandard clinic upon which to base the no-show ratesHowever the intent is to demonstrate the benefit of a

short booking window in spite of the presence of no-shows thus using a show-rate that is perhaps overly pes-simistic is reasonable Any clinic seeking to implementthe MDP policy would have to determine their ownshow-rate first

In addition to the base case five other scenariosare run for each clinic type The average demand rateis varied above and below the capacity a stream ofdemand is introduced that must be booked in advance(arriving with rate μ) the show rate is given a steeperrate of decline and finally same day bookings aregiven a show probability of one The actual parametervalues for each of these scenarios are summarized inTable 1 No results are provided for the lag time versionof the model as the cost (equal to one overtime slot)associated with switching the ABP results in a policythat does not vary the ABP thus making the lag timeirrelevant Running the model with no switching costleads to a much more complicated policy but providesa statistically insignificant improvement over the policywith a fixed ABP

For each scenario 50 replications each 5000 daysin length are run both for the MDP policy and OAThe two policies are compared on the basis of through-put overtime idle time appointment lead times andprofitcost Statistics are collected after the first 500days

One technical challenge was the calculation of theexpected number of clients scheduled today who showfor their appointment when the show rate is dependenton the actual lead time (as in the simulation) Thechallenge is that each client may have been booked adifferent number of days in advance and thus may havedifferent show probabilities This problem is avoidedin the MDP model by using queue size as a proxy forwait times but in the simulation actual wait times foreach client are used Each day the probability distri-bution of the number of clients who show up for theirappointment based on the scheduled clients for that dayneeds to be calculated This is accomplished based onthe algorithm outlined in [1] Once that probability is

Table 1 Scenarios analyzed Base case (λ = C) λ = 10 μ = 0 β1 = 12 β2 = 3654 β3 = 50Reduced demand (λ = C minus 2) λ = 8 μ = 0 β1 = 12 β2 = 3654 β3 = 50Increased demand (λ = C + 2) λ = 12 μ = 0 β1 = 12 β2 = 3654 β3 = 50Advanced bookings λ = 7 μ = 3 β1 = 12 β2 = 3654 β3 = 50Show rate with steep decline λ = 10 μ = 0

β1 = 12 (same day rate unchanged)β2 = 80 (steeper decline)β3 = 20 (levels out at lower rate)

Show rate with same day = 100 Same as base case except show probability is 1for same day appointments

A Markov decision model for determining optimal outpatient scheduling 97

determined it is an easy step to calculate the expectedrewardscosts each day and thus to obtain the optimalaction based on the argmax of Eq 5

Table 2 provides a comparison of OA and the MDPpolicy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 f OT =10 f IT = 0 and appointment lead time costs of 01and 5 Throughput is given as a percentage of demandwhile overtime and idle time are given as a percentageof capacity The results of the simulation demonstratethat the two policies are essentially equivalent in termsof average daily profit with the MDP policy slightlyoutperforming OA in most scenarios It is not surpris-ing that OA performs relatively well for a clinic whererevenue is present since any no-show is detrimental inthat it reduces revenue even if overtime is requiredThus a policy that minimizes the number of no-shows(which is the aim of OA) will clearly have an advantageThe surprise perhaps is that one can do as well byreducing overtime and idle time through a more sophis-ticated scheduling policy even if that policy implies thatthere will be less throughput The crucial additional

advantage of the MDP policy is that the day-to-dayworkload is significantly less variable (see Fig 1) andthe peak load is significantly reduced (from 20 to 15 inthe base case with zero lead time costs for instance)

Table 3 provides the comparison of OA and theMDP policy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 F OT =10 f IT = 5 and appointment lead time costs of 0 1 and5 These clinics demonstrate relatively similar results tothose clinics that ignored idle time as revenue is stillthe primary driver Thus the two policies are essen-tially equivalent in terms of average daily profit withthe MDP policy outperforming OA by slightly highermargins than for the previous clinics Again there isa significant reduction in the peak workload and asmoothing of the variation in daily throughput (seeFig 2)

Finally Table 4 provides the comparison of OAand the MDP policy for each of the scenarios forthe clinics with system-related parameters equal tof R = 0 F OT = 10 f IT = 5 and appointment lead timecosts of 0 1 and 5 For such clinics the MDP policy

Table 2 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 0

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19540demand MDP 0 867 114 74 1 19669 07

1 875 137 87 1 19596 015 880 157 102 0 19540 00

Show rate OA 1000 125 125 0 18750with same MDP 0 982 83 101 1 18806 03day = 100 1 991 101 111 1 18758 00

5 1000 125 125 0 1875 00Base case OA 880 69 189 0 16908

MDP 0 868 31 163 1 17051 081 880 69 189 0 16908 005 880 69 189 0 16908 00

Show rate OA 880 69 189 0 16908with steep MDP 0 872 49 178 1 16939 02decline 1 878 64 186 1 16911 00

5 880 69 189 0 16908 00Advanced OA 846 56 210 1 16066

bookings MDP 0 837 26 189 2 16483 261 840 34 194 2 16116 035 846 56 210 1 16066 00

Decreased OA 880 21 317 0 13873demand MDP 0 875 05 305 1 13945 05

1 876 07 307 1 13912 035 880 19 315 1 13879 00

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

94 J Patrick

model to demonstrate that age race appointment leadtime previously failed appointments provision of a cellphone number and distance from the hospital wereall significant factors in predicting show probabilitiesHowever their appointment lead time analysis onlycompares lead times less than 7 days to lead timesgreater than 21 days

While it would be ideal to incorporate all the factorsmentioned above such detail would make the modelintractable Thus this research focuses only on ap-pointment lead time as the factor that is most clearlyimpacted by the scheduling policy To that end thefollowing show probability is used

ps(w x) = max(

1 minus β1 + β2 lowast log(LT + 1)

100 β3

)(3)

with LT representing appointment lead time The logfunction easily models the Gallucci results (with β1 =12 β2 = 3654 β3 sim= 5) that impose a diminishing im-pact of an extra dayrsquos wait the further out a clientis booked β3 represents a lower bound on the showprobability of any client The addition of one to LTinside the log function insures that the show probabilityfor a same day booking (LT = 0) is psd = 1 minus β1100For advance bookings

LT = max(1 xw) (4)

where xw is the largest integer smaller than xwThe maximum reflects the fact that even if the num-ber of bookings is less than the ABP xw lt 1 anyclient booked in advance still waits a day Equation 4allows the ABP to impact on the relationship betweenappointment lead times and queue size (if you allowmore clients to be booked per day then the same xreflects less of a wait) and therefore better reflects anequal show probability for patients booked an equalnumber of days out

The above description provides the five elements(decision epochs state space action space costs andtransition probabilities) for a Markov Decision Process(MDP) formulation An infinite horizon setting waschosen as clinics do not have a closing date and thus anyend to the scheduling horizon is necessarily arbitraryThis forces the model to ignore any ldquoday-of-the-weekrdquoeffect but avoids any arbitrary terminal rewards

The discounted version of the MDP is used to reflectthe fact that upfront costs are always more expensivethan future costs However a discount factor of γ =099 is used so as not to unduly bias the model againstthe future The standard optimality equations for a

discounted infinite horizon MDP of this initial modelcan therefore be written as

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)

times v (a x minus x and w + y minus b k)

(5)

where pd(k) is the probability that D = k

31 Assumptions to the model

As with any modeling exercise there are a number ofassumptions that help keep the model tractable Firstclient service times are assumed to be deterministicand we do not allow for advance notice of cancela-tions Second as mentioned in Section 3 the modelassumes that all demand for a given day has arrivedbefore any scheduling decision needs to be made Inreality scheduling decisions have to be made as eachdemand request arrives However batch arrivals allowsdecisions to be made in discrete time and the resultingpolicy as will be demonstrated in Section 5 is easilytranslatable into the more realistic setting where thescheduling decision occurs at the time of each demandrequest Third the utilization of the size of the queue asa proxy for wait time when calculating the probabilitythat a client shows up for an appointment means thatthe no-show rate of a client will depend on the sizeof the queue at the time of service when in reality itdepends on the size of the queue at the time of bookingWhile this may mean that an individualrsquos no-show ratemay not be accurate it will still penalize a system thatbooks well in advance to the same degree as a morerealistic (but intractable) model It is also an issue onlyif the length of the queue varies dramatically whereasthe policy derived from the MDP model shows no suchvariation Fourth for the sake of simplicity the issue ofpatient availability is ignored as each client is assumedto take the offered slot To do otherwise would requirethe model to track actual appointment lead times whichwould again lead to an intractable model Finally inthe model described in Section 3 changes in the ABPoccur before the next decision epoch This may meanthat some appointments need to be postponed (if theABP is lowered) or that some clients may need tohave their appointment day advanced (if the ABP isincreased) A more complex model that includes a lagtime in the enactment of any change in the ABP isdeveloped in Section 33 in order to avoid this problemHowever a small cost for any change to the ABPforestalls this issue as it tends to result in the optimalpolicy settling on a single ABP Such a cost is imposed

A Markov decision model for determining optimal outpatient scheduling 95

in the simulation results presented in Section 4 in orderto insure a simpler policy that is easily implementableIn all simulation runs in which the model was run withand without a cost for changing the ABP the differencein the objective between the more complex policy andthe simpler one was not statistically significant Thenext two subsections provide two complications to themodel that allow for more realism

32 Complication 1 two stream demand

To add an additional element of realism the model caneasily be adapted to allow for a stochastic stream ofclients who must be booked in advance The transitionswould then become

s = (w x y) rarr sprime = (a x minus x and w + y minus b + A D)

(6)

where A is a random variable representing the demandrequiring advanced booking The optimality equationsare now

v(s) = maxaisinA(s)

r(s a) + γ

sum

iisinD jisinA

pd(i)pa( j)

times v (a x minus x and w + y minus b + j i)

foralls isin S

(7)

where A is the set of possible advanced booking re-quests and pa( j) is the probability of getting j advancedbooking requests in a day Since advanced bookingrequests (for a clinic running OA) generally arise as

follow-up visits from todayrsquos appointments the numberof such visits is assumed to be unknown until aftertodayrsquos booking decisions are made

An additional issue that this complication to themodel raises is that it is now impossible to insure thatthe booking slate does not exceed N clients simply byrestricting the action set To avoid this issue a clientrsquosrequest for an appointment is rejected at a high costshould accepting it cause the booking slate to exceedN Making this cost sufficiently high insures that suchextreme measures are never required

33 Complication 2 lag time in ABP changes

As mentioned earlier it is somewhat unrealistic toexpect a change in the ABP to be implemented imme-diately However the model can be adjusted to imposea lag time on any change in the ABP Let the newstate s = (w x y z) where w x and y are as beforeand z represents the lag time remaining until the nexttriggered ABP change The lag can be set to xw or 1to insure that the lag is equal to the number of daysthat are already fully booked and thus no previouslybooked appointments need to be canceled and no newbookings jump the queue If z lt 0 then the decision isto reduce the ABP by one and if z gt 0 then the decisionis to increase the ABP by one More drastic changes arenot considered

The same actions are available to the manager withthe same restrictions The evolution of the system de-pends on the current lag time and can be described bythe following set of transitions

s = (w x y z) rarr

sprime =

⎧⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎩

(w x minus x and w + y minus b D l + 1) If l lt minus1(w x minus x and w + y minus b D l minus 1) If l gt 1(a x minus x and w + y minus b D 0) If l = minus1 xw le 1 or l = 1(w x minus x and w + y minus b D minusxw minus 1) If l = 0 xw gt 1 and a lt w(w x minus x and w + y minus b D xw minus 1) If l = 0 xw gt 1 and a gt w(w x minus x and w + y minus b D 0) If l = 0 and a = w

(8)

Here D is again a random variable representing newdemand Notice that once a decision to change theABP is made no new change can be made until theday when that decision is implemented If tomorrowrsquosslate is not yet full (xw le 1) then the ABP change isimplemented by tomorrow as in the original model

Finally the optimality equations are

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)v(sprime)

(9)

where D represents the set of all possible demand pd

is the probability distribution for new demand and sprime

is determined by the transitions given in Eq 8 withD = k

4 Simulation results

In this section the simulation results are presentedthat compare the MDP policy to OA in a variety of

96 J Patrick

scenarios that are chosen to explore the trade-off be-tween the system-related costsrewards (revenue over-time and idle time) and the patient-related ones (leadtime) We consider three clinic types for the system-related costsrewards The first clinic ignores idle timeand sets revenue to twice the cost of overtime f R =20 f OT = 10 and f IT = 0 The second clinic adds anadditional cost for idle time f R = 20 f OT = 10 andf IT = 5 The third clinic does not receive remunera-tion for each patient seen but seeks to maximize theutilization of the available capacity to meet all demandf R = 0 f OT = 10 and f IT = 5 In all cases the costof changing the ABP is set equal to the cost of oneovertime slot The relative values of the cost parame-ters is somewhat arbitrary We have followed commonpractice of valuing idle time (when it is consideredimportant) at half the cost of overtime and have chosento set revenue at twice the value of overtime Therationale for these choices is to represent both clinicsthat are privately run or where the physician is paid ona fee-for-service basis (so that revenue is important) aswell as clinics that are publicly funded and thereforewhere revenue is not a factor

For each of these clinics lead time costs are set at 01 and 5 leading to a total of 9 clinic types differentiatedby the relative weight they place on each costreward Ifthe cost of a dayrsquos lead time is set equal to the overtimecost then the optimal policy would clearly be OA Thuslead time costs of 1 and 5 seem reasonable optionsfor demonstrating the impact of increasing the valueof providing same-day service For each clinic type weconsider 6 scenarios The base case for each clinic as-sumes a Poisson arrival rate and sets average demandλ equal to capacity C = 10 Such a small clinic sizeis chosen in order to restrict the amount of computa-tion time for running numerous scenarios Larger clinicsizes can obviously be solved The Gallucci show-ratedescribed in Section 3 is used to determine show ratesAdmittedly a mental health clinic may not be the moststandard clinic upon which to base the no-show ratesHowever the intent is to demonstrate the benefit of a

short booking window in spite of the presence of no-shows thus using a show-rate that is perhaps overly pes-simistic is reasonable Any clinic seeking to implementthe MDP policy would have to determine their ownshow-rate first

In addition to the base case five other scenariosare run for each clinic type The average demand rateis varied above and below the capacity a stream ofdemand is introduced that must be booked in advance(arriving with rate μ) the show rate is given a steeperrate of decline and finally same day bookings aregiven a show probability of one The actual parametervalues for each of these scenarios are summarized inTable 1 No results are provided for the lag time versionof the model as the cost (equal to one overtime slot)associated with switching the ABP results in a policythat does not vary the ABP thus making the lag timeirrelevant Running the model with no switching costleads to a much more complicated policy but providesa statistically insignificant improvement over the policywith a fixed ABP

For each scenario 50 replications each 5000 daysin length are run both for the MDP policy and OAThe two policies are compared on the basis of through-put overtime idle time appointment lead times andprofitcost Statistics are collected after the first 500days

One technical challenge was the calculation of theexpected number of clients scheduled today who showfor their appointment when the show rate is dependenton the actual lead time (as in the simulation) Thechallenge is that each client may have been booked adifferent number of days in advance and thus may havedifferent show probabilities This problem is avoidedin the MDP model by using queue size as a proxy forwait times but in the simulation actual wait times foreach client are used Each day the probability distri-bution of the number of clients who show up for theirappointment based on the scheduled clients for that dayneeds to be calculated This is accomplished based onthe algorithm outlined in [1] Once that probability is

Table 1 Scenarios analyzed Base case (λ = C) λ = 10 μ = 0 β1 = 12 β2 = 3654 β3 = 50Reduced demand (λ = C minus 2) λ = 8 μ = 0 β1 = 12 β2 = 3654 β3 = 50Increased demand (λ = C + 2) λ = 12 μ = 0 β1 = 12 β2 = 3654 β3 = 50Advanced bookings λ = 7 μ = 3 β1 = 12 β2 = 3654 β3 = 50Show rate with steep decline λ = 10 μ = 0

β1 = 12 (same day rate unchanged)β2 = 80 (steeper decline)β3 = 20 (levels out at lower rate)

Show rate with same day = 100 Same as base case except show probability is 1for same day appointments

A Markov decision model for determining optimal outpatient scheduling 97

determined it is an easy step to calculate the expectedrewardscosts each day and thus to obtain the optimalaction based on the argmax of Eq 5

Table 2 provides a comparison of OA and the MDPpolicy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 f OT =10 f IT = 0 and appointment lead time costs of 01and 5 Throughput is given as a percentage of demandwhile overtime and idle time are given as a percentageof capacity The results of the simulation demonstratethat the two policies are essentially equivalent in termsof average daily profit with the MDP policy slightlyoutperforming OA in most scenarios It is not surpris-ing that OA performs relatively well for a clinic whererevenue is present since any no-show is detrimental inthat it reduces revenue even if overtime is requiredThus a policy that minimizes the number of no-shows(which is the aim of OA) will clearly have an advantageThe surprise perhaps is that one can do as well byreducing overtime and idle time through a more sophis-ticated scheduling policy even if that policy implies thatthere will be less throughput The crucial additional

advantage of the MDP policy is that the day-to-dayworkload is significantly less variable (see Fig 1) andthe peak load is significantly reduced (from 20 to 15 inthe base case with zero lead time costs for instance)

Table 3 provides the comparison of OA and theMDP policy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 F OT =10 f IT = 5 and appointment lead time costs of 0 1 and5 These clinics demonstrate relatively similar results tothose clinics that ignored idle time as revenue is stillthe primary driver Thus the two policies are essen-tially equivalent in terms of average daily profit withthe MDP policy outperforming OA by slightly highermargins than for the previous clinics Again there isa significant reduction in the peak workload and asmoothing of the variation in daily throughput (seeFig 2)

Finally Table 4 provides the comparison of OAand the MDP policy for each of the scenarios forthe clinics with system-related parameters equal tof R = 0 F OT = 10 f IT = 5 and appointment lead timecosts of 0 1 and 5 For such clinics the MDP policy

Table 2 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 0

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19540demand MDP 0 867 114 74 1 19669 07

1 875 137 87 1 19596 015 880 157 102 0 19540 00

Show rate OA 1000 125 125 0 18750with same MDP 0 982 83 101 1 18806 03day = 100 1 991 101 111 1 18758 00

5 1000 125 125 0 1875 00Base case OA 880 69 189 0 16908

MDP 0 868 31 163 1 17051 081 880 69 189 0 16908 005 880 69 189 0 16908 00

Show rate OA 880 69 189 0 16908with steep MDP 0 872 49 178 1 16939 02decline 1 878 64 186 1 16911 00

5 880 69 189 0 16908 00Advanced OA 846 56 210 1 16066

bookings MDP 0 837 26 189 2 16483 261 840 34 194 2 16116 035 846 56 210 1 16066 00

Decreased OA 880 21 317 0 13873demand MDP 0 875 05 305 1 13945 05

1 876 07 307 1 13912 035 880 19 315 1 13879 00

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

A Markov decision model for determining optimal outpatient scheduling 95

in the simulation results presented in Section 4 in orderto insure a simpler policy that is easily implementableIn all simulation runs in which the model was run withand without a cost for changing the ABP the differencein the objective between the more complex policy andthe simpler one was not statistically significant Thenext two subsections provide two complications to themodel that allow for more realism

32 Complication 1 two stream demand

To add an additional element of realism the model caneasily be adapted to allow for a stochastic stream ofclients who must be booked in advance The transitionswould then become

s = (w x y) rarr sprime = (a x minus x and w + y minus b + A D)

(6)

where A is a random variable representing the demandrequiring advanced booking The optimality equationsare now

v(s) = maxaisinA(s)

r(s a) + γ

sum

iisinD jisinA

pd(i)pa( j)

times v (a x minus x and w + y minus b + j i)

foralls isin S

(7)

where A is the set of possible advanced booking re-quests and pa( j) is the probability of getting j advancedbooking requests in a day Since advanced bookingrequests (for a clinic running OA) generally arise as

follow-up visits from todayrsquos appointments the numberof such visits is assumed to be unknown until aftertodayrsquos booking decisions are made

An additional issue that this complication to themodel raises is that it is now impossible to insure thatthe booking slate does not exceed N clients simply byrestricting the action set To avoid this issue a clientrsquosrequest for an appointment is rejected at a high costshould accepting it cause the booking slate to exceedN Making this cost sufficiently high insures that suchextreme measures are never required

33 Complication 2 lag time in ABP changes

As mentioned earlier it is somewhat unrealistic toexpect a change in the ABP to be implemented imme-diately However the model can be adjusted to imposea lag time on any change in the ABP Let the newstate s = (w x y z) where w x and y are as beforeand z represents the lag time remaining until the nexttriggered ABP change The lag can be set to xw or 1to insure that the lag is equal to the number of daysthat are already fully booked and thus no previouslybooked appointments need to be canceled and no newbookings jump the queue If z lt 0 then the decision isto reduce the ABP by one and if z gt 0 then the decisionis to increase the ABP by one More drastic changes arenot considered

The same actions are available to the manager withthe same restrictions The evolution of the system de-pends on the current lag time and can be described bythe following set of transitions

s = (w x y z) rarr

sprime =

⎧⎪⎪⎪⎪⎪⎪⎨

⎪⎪⎪⎪⎪⎪⎩

(w x minus x and w + y minus b D l + 1) If l lt minus1(w x minus x and w + y minus b D l minus 1) If l gt 1(a x minus x and w + y minus b D 0) If l = minus1 xw le 1 or l = 1(w x minus x and w + y minus b D minusxw minus 1) If l = 0 xw gt 1 and a lt w(w x minus x and w + y minus b D xw minus 1) If l = 0 xw gt 1 and a gt w(w x minus x and w + y minus b D 0) If l = 0 and a = w

(8)

Here D is again a random variable representing newdemand Notice that once a decision to change theABP is made no new change can be made until theday when that decision is implemented If tomorrowrsquosslate is not yet full (xw le 1) then the ABP change isimplemented by tomorrow as in the original model

Finally the optimality equations are

v(s) = maxaisinA(s)

r(s a) + γ

sum

kisinD

pd(k)v(sprime)

(9)

where D represents the set of all possible demand pd

is the probability distribution for new demand and sprime

is determined by the transitions given in Eq 8 withD = k

4 Simulation results

In this section the simulation results are presentedthat compare the MDP policy to OA in a variety of

96 J Patrick

scenarios that are chosen to explore the trade-off be-tween the system-related costsrewards (revenue over-time and idle time) and the patient-related ones (leadtime) We consider three clinic types for the system-related costsrewards The first clinic ignores idle timeand sets revenue to twice the cost of overtime f R =20 f OT = 10 and f IT = 0 The second clinic adds anadditional cost for idle time f R = 20 f OT = 10 andf IT = 5 The third clinic does not receive remunera-tion for each patient seen but seeks to maximize theutilization of the available capacity to meet all demandf R = 0 f OT = 10 and f IT = 5 In all cases the costof changing the ABP is set equal to the cost of oneovertime slot The relative values of the cost parame-ters is somewhat arbitrary We have followed commonpractice of valuing idle time (when it is consideredimportant) at half the cost of overtime and have chosento set revenue at twice the value of overtime Therationale for these choices is to represent both clinicsthat are privately run or where the physician is paid ona fee-for-service basis (so that revenue is important) aswell as clinics that are publicly funded and thereforewhere revenue is not a factor

For each of these clinics lead time costs are set at 01 and 5 leading to a total of 9 clinic types differentiatedby the relative weight they place on each costreward Ifthe cost of a dayrsquos lead time is set equal to the overtimecost then the optimal policy would clearly be OA Thuslead time costs of 1 and 5 seem reasonable optionsfor demonstrating the impact of increasing the valueof providing same-day service For each clinic type weconsider 6 scenarios The base case for each clinic as-sumes a Poisson arrival rate and sets average demandλ equal to capacity C = 10 Such a small clinic sizeis chosen in order to restrict the amount of computa-tion time for running numerous scenarios Larger clinicsizes can obviously be solved The Gallucci show-ratedescribed in Section 3 is used to determine show ratesAdmittedly a mental health clinic may not be the moststandard clinic upon which to base the no-show ratesHowever the intent is to demonstrate the benefit of a

short booking window in spite of the presence of no-shows thus using a show-rate that is perhaps overly pes-simistic is reasonable Any clinic seeking to implementthe MDP policy would have to determine their ownshow-rate first

In addition to the base case five other scenariosare run for each clinic type The average demand rateis varied above and below the capacity a stream ofdemand is introduced that must be booked in advance(arriving with rate μ) the show rate is given a steeperrate of decline and finally same day bookings aregiven a show probability of one The actual parametervalues for each of these scenarios are summarized inTable 1 No results are provided for the lag time versionof the model as the cost (equal to one overtime slot)associated with switching the ABP results in a policythat does not vary the ABP thus making the lag timeirrelevant Running the model with no switching costleads to a much more complicated policy but providesa statistically insignificant improvement over the policywith a fixed ABP

For each scenario 50 replications each 5000 daysin length are run both for the MDP policy and OAThe two policies are compared on the basis of through-put overtime idle time appointment lead times andprofitcost Statistics are collected after the first 500days

One technical challenge was the calculation of theexpected number of clients scheduled today who showfor their appointment when the show rate is dependenton the actual lead time (as in the simulation) Thechallenge is that each client may have been booked adifferent number of days in advance and thus may havedifferent show probabilities This problem is avoidedin the MDP model by using queue size as a proxy forwait times but in the simulation actual wait times foreach client are used Each day the probability distri-bution of the number of clients who show up for theirappointment based on the scheduled clients for that dayneeds to be calculated This is accomplished based onthe algorithm outlined in [1] Once that probability is

Table 1 Scenarios analyzed Base case (λ = C) λ = 10 μ = 0 β1 = 12 β2 = 3654 β3 = 50Reduced demand (λ = C minus 2) λ = 8 μ = 0 β1 = 12 β2 = 3654 β3 = 50Increased demand (λ = C + 2) λ = 12 μ = 0 β1 = 12 β2 = 3654 β3 = 50Advanced bookings λ = 7 μ = 3 β1 = 12 β2 = 3654 β3 = 50Show rate with steep decline λ = 10 μ = 0

β1 = 12 (same day rate unchanged)β2 = 80 (steeper decline)β3 = 20 (levels out at lower rate)

Show rate with same day = 100 Same as base case except show probability is 1for same day appointments

A Markov decision model for determining optimal outpatient scheduling 97

determined it is an easy step to calculate the expectedrewardscosts each day and thus to obtain the optimalaction based on the argmax of Eq 5

Table 2 provides a comparison of OA and the MDPpolicy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 f OT =10 f IT = 0 and appointment lead time costs of 01and 5 Throughput is given as a percentage of demandwhile overtime and idle time are given as a percentageof capacity The results of the simulation demonstratethat the two policies are essentially equivalent in termsof average daily profit with the MDP policy slightlyoutperforming OA in most scenarios It is not surpris-ing that OA performs relatively well for a clinic whererevenue is present since any no-show is detrimental inthat it reduces revenue even if overtime is requiredThus a policy that minimizes the number of no-shows(which is the aim of OA) will clearly have an advantageThe surprise perhaps is that one can do as well byreducing overtime and idle time through a more sophis-ticated scheduling policy even if that policy implies thatthere will be less throughput The crucial additional

advantage of the MDP policy is that the day-to-dayworkload is significantly less variable (see Fig 1) andthe peak load is significantly reduced (from 20 to 15 inthe base case with zero lead time costs for instance)

Table 3 provides the comparison of OA and theMDP policy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 F OT =10 f IT = 5 and appointment lead time costs of 0 1 and5 These clinics demonstrate relatively similar results tothose clinics that ignored idle time as revenue is stillthe primary driver Thus the two policies are essen-tially equivalent in terms of average daily profit withthe MDP policy outperforming OA by slightly highermargins than for the previous clinics Again there isa significant reduction in the peak workload and asmoothing of the variation in daily throughput (seeFig 2)

Finally Table 4 provides the comparison of OAand the MDP policy for each of the scenarios forthe clinics with system-related parameters equal tof R = 0 F OT = 10 f IT = 5 and appointment lead timecosts of 0 1 and 5 For such clinics the MDP policy

Table 2 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 0

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19540demand MDP 0 867 114 74 1 19669 07

1 875 137 87 1 19596 015 880 157 102 0 19540 00

Show rate OA 1000 125 125 0 18750with same MDP 0 982 83 101 1 18806 03day = 100 1 991 101 111 1 18758 00

5 1000 125 125 0 1875 00Base case OA 880 69 189 0 16908

MDP 0 868 31 163 1 17051 081 880 69 189 0 16908 005 880 69 189 0 16908 00

Show rate OA 880 69 189 0 16908with steep MDP 0 872 49 178 1 16939 02decline 1 878 64 186 1 16911 00

5 880 69 189 0 16908 00Advanced OA 846 56 210 1 16066

bookings MDP 0 837 26 189 2 16483 261 840 34 194 2 16116 035 846 56 210 1 16066 00

Decreased OA 880 21 317 0 13873demand MDP 0 875 05 305 1 13945 05

1 876 07 307 1 13912 035 880 19 315 1 13879 00

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

96 J Patrick

scenarios that are chosen to explore the trade-off be-tween the system-related costsrewards (revenue over-time and idle time) and the patient-related ones (leadtime) We consider three clinic types for the system-related costsrewards The first clinic ignores idle timeand sets revenue to twice the cost of overtime f R =20 f OT = 10 and f IT = 0 The second clinic adds anadditional cost for idle time f R = 20 f OT = 10 andf IT = 5 The third clinic does not receive remunera-tion for each patient seen but seeks to maximize theutilization of the available capacity to meet all demandf R = 0 f OT = 10 and f IT = 5 In all cases the costof changing the ABP is set equal to the cost of oneovertime slot The relative values of the cost parame-ters is somewhat arbitrary We have followed commonpractice of valuing idle time (when it is consideredimportant) at half the cost of overtime and have chosento set revenue at twice the value of overtime Therationale for these choices is to represent both clinicsthat are privately run or where the physician is paid ona fee-for-service basis (so that revenue is important) aswell as clinics that are publicly funded and thereforewhere revenue is not a factor

For each of these clinics lead time costs are set at 01 and 5 leading to a total of 9 clinic types differentiatedby the relative weight they place on each costreward Ifthe cost of a dayrsquos lead time is set equal to the overtimecost then the optimal policy would clearly be OA Thuslead time costs of 1 and 5 seem reasonable optionsfor demonstrating the impact of increasing the valueof providing same-day service For each clinic type weconsider 6 scenarios The base case for each clinic as-sumes a Poisson arrival rate and sets average demandλ equal to capacity C = 10 Such a small clinic sizeis chosen in order to restrict the amount of computa-tion time for running numerous scenarios Larger clinicsizes can obviously be solved The Gallucci show-ratedescribed in Section 3 is used to determine show ratesAdmittedly a mental health clinic may not be the moststandard clinic upon which to base the no-show ratesHowever the intent is to demonstrate the benefit of a

short booking window in spite of the presence of no-shows thus using a show-rate that is perhaps overly pes-simistic is reasonable Any clinic seeking to implementthe MDP policy would have to determine their ownshow-rate first

In addition to the base case five other scenariosare run for each clinic type The average demand rateis varied above and below the capacity a stream ofdemand is introduced that must be booked in advance(arriving with rate μ) the show rate is given a steeperrate of decline and finally same day bookings aregiven a show probability of one The actual parametervalues for each of these scenarios are summarized inTable 1 No results are provided for the lag time versionof the model as the cost (equal to one overtime slot)associated with switching the ABP results in a policythat does not vary the ABP thus making the lag timeirrelevant Running the model with no switching costleads to a much more complicated policy but providesa statistically insignificant improvement over the policywith a fixed ABP

For each scenario 50 replications each 5000 daysin length are run both for the MDP policy and OAThe two policies are compared on the basis of through-put overtime idle time appointment lead times andprofitcost Statistics are collected after the first 500days

One technical challenge was the calculation of theexpected number of clients scheduled today who showfor their appointment when the show rate is dependenton the actual lead time (as in the simulation) Thechallenge is that each client may have been booked adifferent number of days in advance and thus may havedifferent show probabilities This problem is avoidedin the MDP model by using queue size as a proxy forwait times but in the simulation actual wait times foreach client are used Each day the probability distri-bution of the number of clients who show up for theirappointment based on the scheduled clients for that dayneeds to be calculated This is accomplished based onthe algorithm outlined in [1] Once that probability is

Table 1 Scenarios analyzed Base case (λ = C) λ = 10 μ = 0 β1 = 12 β2 = 3654 β3 = 50Reduced demand (λ = C minus 2) λ = 8 μ = 0 β1 = 12 β2 = 3654 β3 = 50Increased demand (λ = C + 2) λ = 12 μ = 0 β1 = 12 β2 = 3654 β3 = 50Advanced bookings λ = 7 μ = 3 β1 = 12 β2 = 3654 β3 = 50Show rate with steep decline λ = 10 μ = 0

β1 = 12 (same day rate unchanged)β2 = 80 (steeper decline)β3 = 20 (levels out at lower rate)

Show rate with same day = 100 Same as base case except show probability is 1for same day appointments

A Markov decision model for determining optimal outpatient scheduling 97

determined it is an easy step to calculate the expectedrewardscosts each day and thus to obtain the optimalaction based on the argmax of Eq 5

Table 2 provides a comparison of OA and the MDPpolicy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 f OT =10 f IT = 0 and appointment lead time costs of 01and 5 Throughput is given as a percentage of demandwhile overtime and idle time are given as a percentageof capacity The results of the simulation demonstratethat the two policies are essentially equivalent in termsof average daily profit with the MDP policy slightlyoutperforming OA in most scenarios It is not surpris-ing that OA performs relatively well for a clinic whererevenue is present since any no-show is detrimental inthat it reduces revenue even if overtime is requiredThus a policy that minimizes the number of no-shows(which is the aim of OA) will clearly have an advantageThe surprise perhaps is that one can do as well byreducing overtime and idle time through a more sophis-ticated scheduling policy even if that policy implies thatthere will be less throughput The crucial additional

advantage of the MDP policy is that the day-to-dayworkload is significantly less variable (see Fig 1) andthe peak load is significantly reduced (from 20 to 15 inthe base case with zero lead time costs for instance)

Table 3 provides the comparison of OA and theMDP policy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 F OT =10 f IT = 5 and appointment lead time costs of 0 1 and5 These clinics demonstrate relatively similar results tothose clinics that ignored idle time as revenue is stillthe primary driver Thus the two policies are essen-tially equivalent in terms of average daily profit withthe MDP policy outperforming OA by slightly highermargins than for the previous clinics Again there isa significant reduction in the peak workload and asmoothing of the variation in daily throughput (seeFig 2)

Finally Table 4 provides the comparison of OAand the MDP policy for each of the scenarios forthe clinics with system-related parameters equal tof R = 0 F OT = 10 f IT = 5 and appointment lead timecosts of 0 1 and 5 For such clinics the MDP policy

Table 2 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 0

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19540demand MDP 0 867 114 74 1 19669 07

1 875 137 87 1 19596 015 880 157 102 0 19540 00

Show rate OA 1000 125 125 0 18750with same MDP 0 982 83 101 1 18806 03day = 100 1 991 101 111 1 18758 00

5 1000 125 125 0 1875 00Base case OA 880 69 189 0 16908

MDP 0 868 31 163 1 17051 081 880 69 189 0 16908 005 880 69 189 0 16908 00

Show rate OA 880 69 189 0 16908with steep MDP 0 872 49 178 1 16939 02decline 1 878 64 186 1 16911 00

5 880 69 189 0 16908 00Advanced OA 846 56 210 1 16066

bookings MDP 0 837 26 189 2 16483 261 840 34 194 2 16116 035 846 56 210 1 16066 00

Decreased OA 880 21 317 0 13873demand MDP 0 875 05 305 1 13945 05

1 876 07 307 1 13912 035 880 19 315 1 13879 00

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

A Markov decision model for determining optimal outpatient scheduling 97

determined it is an easy step to calculate the expectedrewardscosts each day and thus to obtain the optimalaction based on the argmax of Eq 5

Table 2 provides a comparison of OA and the MDPpolicy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 f OT =10 f IT = 0 and appointment lead time costs of 01and 5 Throughput is given as a percentage of demandwhile overtime and idle time are given as a percentageof capacity The results of the simulation demonstratethat the two policies are essentially equivalent in termsof average daily profit with the MDP policy slightlyoutperforming OA in most scenarios It is not surpris-ing that OA performs relatively well for a clinic whererevenue is present since any no-show is detrimental inthat it reduces revenue even if overtime is requiredThus a policy that minimizes the number of no-shows(which is the aim of OA) will clearly have an advantageThe surprise perhaps is that one can do as well byreducing overtime and idle time through a more sophis-ticated scheduling policy even if that policy implies thatthere will be less throughput The crucial additional

advantage of the MDP policy is that the day-to-dayworkload is significantly less variable (see Fig 1) andthe peak load is significantly reduced (from 20 to 15 inthe base case with zero lead time costs for instance)

Table 3 provides the comparison of OA and theMDP policy for each of the scenarios for the clinics withsystem-related parameters equal to f R = 20 F OT =10 f IT = 5 and appointment lead time costs of 0 1 and5 These clinics demonstrate relatively similar results tothose clinics that ignored idle time as revenue is stillthe primary driver Thus the two policies are essen-tially equivalent in terms of average daily profit withthe MDP policy outperforming OA by slightly highermargins than for the previous clinics Again there isa significant reduction in the peak workload and asmoothing of the variation in daily throughput (seeFig 2)

Finally Table 4 provides the comparison of OAand the MDP policy for each of the scenarios forthe clinics with system-related parameters equal tof R = 0 F OT = 10 f IT = 5 and appointment lead timecosts of 0 1 and 5 For such clinics the MDP policy

Table 2 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 0

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19540demand MDP 0 867 114 74 1 19669 07

1 875 137 87 1 19596 015 880 157 102 0 19540 00

Show rate OA 1000 125 125 0 18750with same MDP 0 982 83 101 1 18806 03day = 100 1 991 101 111 1 18758 00

5 1000 125 125 0 1875 00Base case OA 880 69 189 0 16908

MDP 0 868 31 163 1 17051 081 880 69 189 0 16908 005 880 69 189 0 16908 00

Show rate OA 880 69 189 0 16908with steep MDP 0 872 49 178 1 16939 02decline 1 878 64 186 1 16911 00

5 880 69 189 0 16908 00Advanced OA 846 56 210 1 16066

bookings MDP 0 837 26 189 2 16483 261 840 34 194 2 16116 035 846 56 210 1 16066 00

Decreased OA 880 21 317 0 13873demand MDP 0 875 05 305 1 13945 05

1 876 07 307 1 13912 035 880 19 315 1 13879 00

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

98 J Patrick

Fig 1 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 0 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 3 Simulation results for a clinic with f R = 20 f OT = 10 f IT = 5

Scenario Policy Lead time Average daily Max lead Profit Increase in profit

cost TH OT IT time (days) for MDP

Increased OA 880 157 102 0 19033demand MDP 0 862 103 68 1 19321 15

1 871 123 78 1 19170 075 880 157 102 0 19033 00

Show rate OA 1000 125 125 0 18125with same MDP 0 970 60 90 1 18344 12day = 100 1 981 81 100 1 18238 06

5 1000 125 125 0 18125 00Base case OA 880 69 189 0 15963

MDP 0 866 26 160 2 16249 181 869 35 165 1 16129 105 878 62 183 1 15961 00

Show rate OA 880 69 189 0 15963with steep MDP 0 866 40 174 1 16046 05decline 1 870 47 177 1 15963 03

5 880 69 189 0 15963 00Advanced OA 846 56 210 0 15303

bookings MDP 0 833 19 186 3 15545 161 838 28 190 2 15469 115 844 48 204 1 15305 00

Decreased OA 880 21 317 0 12290demand MDP 0 875 05 305 1 12421 11

1 876 06 305 1 12395 095 878 13 310 1 12316 02

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

A Markov decision model for determining optimal outpatient scheduling 99

Fig 2 Daily throughputfor a clinic with f R = 20

f OT = 10 f IT = 5 f LT = 0

0

100

200

300

400

500

600

700

800

900

1000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Fre

qu

ency

Daily Throughput

OA MDP

Table 4 Simulation results for a clinic with f R = 0 f OT = 10 f IT = 5

Scenario Lead time Policy Average daily Max lead Cost Decrease in cost

cost TH OT IT time (days) for MDP

Show rate OA 1000 125 125 0 1875with same MDP 0 916 10 94 3 567 698day = 100 1 934 18 84 2 892 524

5 971 61 90 1 1693 97Increased OA 880 157 101 0 2081

demand MDP 0 780 29 92 4 750 6401 839 69 62 2 1429 3135 878 147 94 2 2067 07

Base case OA 880 69 189 0 1634MDP 0 841 07 165 3 892 454

1 859 17 158 2 1145 2995 874 45 171 1 1564 42

Show rate OA 880 69 189 0 1634with steep MDP 0 827 08 181 2 981 400decline 1 843 15 172 2 1160 290

5 866 40 174 1 1551 51Advanced OA 846 56 210 0 1616

bookings MDP 0 815 06 191 3 1011 3741 829 14 185 2 1203 2555 842 37 196 1 1389 140

Decreased OA 880 21 317 0 1789demand MDP 0 869 00 305 2 1528 146

1 872 02 304 1 1593 1105 876 08 307 1 1732 32

Throughput (TH) is given as a percentage of demand and overtime (OT) and idle time (IT) as a percentage of regular hour capacityStarred lines represent scenarios where the MDP policy improved significantly (α = 005) upon the OA policy in terms of cost basedon a matched pairs t-test

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

100 J Patrick

Fig 3 Daily throughputfor a clinic with f R = 0

f OT = 10 f IT = 5 f LT = 0

0

200

400

600

800

1000

1200

1400

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Fre

qu

ency

Daily Throughput

OA MDP

significantly reduces average daily cost compared toOA in all instances Figure 3 again demonstrates thesignificant reduction in variation in daily throughputand in the peak workload achieved by the MDP policyas opposed to OA

The longest appointment lead times in any of thescenarios examined was four days demonstrating thatwhile the MDP does increase patient lead times it doesnot do so excessively It is also worth noting that for thethree clinic types differentiated by the system-relatedparameter values the scenarios where the MDP policyperformed the best were the scenario with same dayshow probability equal to 100 (a scenario that onemight have thought would benefit OA the most) andthe scenario where demand exceeded capacity Thismight seem counter-intuitive as one would expect ascenario with higher demand to lead to higher levelsof overtime However the presence of a significant no-show rate means that the ef fective demand can to someextent be controlled by the scheduling policy that isimplemented as a larger booking window means higherno-show rates Finally for all clinic types the MDPpolicy approaches an OA policy the more capacityexceeds demand or not surprisingly as the lead timecost increases

In the simulation and in the MDP a linear functionfor overtime costs was used A more realistic piecewiselinear function that incorporates increasing overtimecosts with a larger overtime load would increase the

benefit of the MDP policy over OA Additionally themaximum number of arrivals in a day was restrictedto twice the average demand in ordered to keep thestate space relatively small If higher daily demandlevels were permitted an even greater improvement inperformance would have been achieved by the MDPpolicy compared to OA

5 The form of the MDP policy

The MDP policy without a cost associated with chang-ing the ABP is not easily described for any of the clinicsas it depends on a number of factorsmdashthe currentABP queue size and demand as well as todayrsquos slateof patients and todayrsquos expected throughput Howeveronce a cost is imposed for changing the ABP the policybecomes quite simple and depends on two factorsmdashthe size of the queue (those already booked plus thosewaiting to be booked) and todayrsquos current slate

For all of the clinics and scenarios analyzed theMDP policy is defined by a series of thresholds Thepolicy will act initially like OA booking any new re-quest into todayrsquos slate However once todayrsquos slatereaches a certain threshold (the value being depen-dent on the scenario and particularly on the lead timecost) it will begin to defer any further requests to thenext available appointment slot The policy will revertback to booking a new request into todayrsquos slate if

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

A Markov decision model for determining optimal outpatient scheduling 101

Fig 4 Example of the formof the optimal policy fromthe MDP DEMAND 0 1 2 3 4 5 6 7 8 9 10 11

0 1 0 0 0 0 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0 0 0 0 0 0

2 0 0 1 0 0 0 0 0 0 0 0 0

3 0 0 0 1 0 0 0 0 0 0 0 0

4 0 0 0 0 1 0 0 0 0 0 0 0

5 0 0 0 0 0 1 0 0 0 0 0 0

6 0 0 0 0 0 0 1 0 0 0 0 0

7 0 0 0 0 0 0 0 1 0 0 0 0

8 0 0 0 0 0 0 0 0 1 0 0 0

9 0 0 0 0 0 0 0 0 0 1 0 0

10 0 0 0 0 0 0 0 0 0 0 1 0

11 0 0 0 0 0 0 0 0 0 0 1 0

12 0 0 0 0 0 0 0 0 0 0 1 0

13 0 0 0 0 0 0 0 0 0 0 1 0

14 0 0 0 0 0 0 0 0 0 0 1 0

15 0 0 0 0 0 0 0 0 0 0 1 0

16 0 0 0 0 0 0 0 0 0 0 1 0

17 0 0 0 0 0 0 0 0 0 0 1 0

18 0 0 0 0 0 0 0 0 0 0 1 0

19 0 0 0 0 0 0 0 0 0 0 0 1

20 0 0 0 0 0 0 0 0 0 0 0 1

Number of Clients Booked For Today

the queue size reaches its own threshold Dependingon the scenario the policy may then simply revert toOA and book any additional demand into today or itmay book only one additional client today and thendefer again until another threshold triggers a secondoverbook If the lead time cost is set to zero then thefirst threshold that triggers the initial decision to beginbooking requests into future days is equal to capacityAs the lead time cost is increased that first threshold isincreased as well

Figure 4 gives a demonstration of the policy for thebase case for a clinic with cost parameters equal tof R = 0 f OT = 10 f IT = 5 f LT = 0 and for a day withno advance bookings at the outset The MDP policyacts like OA until capacity is reached at which pointit begins to delay demand to the next day Howeverif the number of requests exceeds 18 then it booksan additional client into todayrsquos schedule before onceagain delaying any further demand to the next availableappointment slot If there are advance bookings alreadyon the slate then the policy looks very similar to thatin Fig 4 but with the threshold shifted down by thenumber already in the queue If the lead time cost isincreased to one then the policy acts like OA untiltodayrsquos bookings reach eleven (capacity plus one) andthen defers demand to the next available appointmentslot If the lead time cost is further increased to five thenthe initial decision to stop booking any new requestsinto today is delayed until todayrsquos bookings reach 12

before a series of thresholds on the queue size triggersadditional overbooks

Recall that the MDP formulation assumes that alldemand for the day was collected before any schedulingdecision is made However the form of the policymakes it a simple matter to translate the MDP policyinto the more realistic setting where demand is sched-uled as it arrives As a new request for an appointmentarrives the booking clerk checks the MDP policy todetermine whether that request should be booked to-day or delayed based on the look-up tables from theMDP as if this were the last request of the day Thusin the example described in the previous paragraph ifthe 18th request of the day arrives it will be bookedinto the next available appointment slot but if a 19threquest arrives it will be booked into todayrsquos slate eventhough todayrsquos slate is already full The negative aspectof the MDP policy is that now a client who called latermay in fact receive an earlier appointment

6 Conclusion

The argument behind OA is two-fold Either (1) thepresence of no-shows has such a debilitating effect onclinic performance that it is worthwhile to minimizesuch occurrences as much as possible by ldquodoing todayrsquosdemand todayrdquo or else (2) the advantage of providingsame-day access is worth the cost What this paper

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144

102 J Patrick

demonstrates is that the impact of no-shows can beeasily mitigated without resorting to OA A policy thatinstead uses a short booking window to smooth out de-mand can reduce both idle time and overtime substan-tially and thus improve resource utilization and reducecosts The robustness with which the MDP policy main-tains its advantage over OA across all the scenariosdemonstrates that this superiority is not a function ofthe particular parameters used Even for those clinicswhere revenue is dominant and thus higher throughputis most advantageous the MDP demonstrates distinctadvantages over OA in that the variation in the day-to-day workload and the peak work load are significantlyreduced without sacrificing profit

If a high enough cost is associated with appointmentlead times then clearly OA will eventually become opti-mal However the cost (in resource efficiency) incurredby the clinic for insuring same-day access as opposed tousing a short booking window can be quite high Thisis not to say that there are not clinics where OA canbe readily implemented but that the trade-offs needto be clearly understood The popularity of OA withinthe medical community suggests that perhaps they havenot This research contributes to the growing evidenceprovided by the operations research community thatdepending on a clinicrsquos priorities open access may infact be sub-optimal The benefit of this research isto provide an alternative scheduling policy that cansignificantly improve resource utilization while onlymarginally increasing client appointment lead times

Acknowledgements This work was funded in part by a researchgrant from National Science and Engineering Research Council(NSERC) I would like to thank Martin Puterman from theUniversity of British Columbia his help in revising this paper

References

1 Butler K Stephens M (1993) Distribution of a sum ofbinomial random variables Technical report No 457 for theOffice of Naval Research Available at httpwwwdticmilcgi-binGetTRDocAD=ADA266969ampLocation=U2ampdoc=GetTRDocpdf

2 Cayirli T Veral E (2003) Outpatient scheduling in healthcare a review of literature Prod Oper Manag 12519ndash549

3 Gallucci G Swartz W Hackerman F (2005) Impact of thewait for an initial appointment on the rate of kept appoint-ments at a mental health center Psychiatr Serv 56344ndash346

4 Gupta D Wang L (2008) Revenue management for aprimary-care clinic in the presence of patient choice OperRes 56576ndash592

5 Kim S Giachetti R (2006) A stochastic mathematical ap-pointment overbooking model for healthcare providers toimprove profits IEEE Trans Syst Man Cybern Part A SystHumans 361211ndash1219

6 Kopach R et al (2007) Effects of clinical characteristics onsuccessfull open access scheduling Health Care Manage Sci10111ndash124

7 LaGanga L Lawrence S (2007) Clinic overbooking to im-prove patient access and increase provider productivityDecis Sci 38251ndash276

8 Lee V Earnest A Chen M Krishnan B (2005) Predictors offailed attendances in a multi-specialty outpatient centre usingelectronic databases BMC Health Serv Res 51ndash8

9 Liu N Ziya S Kulkarni V (2009) Dynamic scheduling ofoutpatient appointments under patient no-shows and cancel-lations Manuf Serv Oper Manag 12347ndash364

10 Murray M Tantau C (1999) Redefining open access to pri-mary care Manag Care Q 745ndash51

11 Muthuraman K Lawley M (2008) A stochastic overbookingmodel for outpatient clnical scheduling with no-shows IIETrans 40820ndash837

12 Robinson L Chen R (2009) Traditional and open-accessappointment scheduling policies The effects of patient no-shows Manuf Serv Oper Manag 12330ndash346

13 Zeng B Turkcan A Lin J Lawley M (2009) Clinic schedulingmodels with overbooking for patients with heterogeneous no-show probabilities Ann Oper Res 178121ndash144