MANUAL FOR SCIENTIFIC MONITORING AND EVALUATIONDrug Treatment Courtsin the Americas
Man
ual f
or S
cien
tific
Mon
itor
ing
and
Eval
uati
on D
rug
Trea
tmen
t Cou
rts
in th
e A
mer
icas
ISBN 978-0-8270-6830-8
Organization of American StatesSecretariat for Multidimensional Security
Inter-American Drug Abuse Control Commission
This publication was prepared by the Executive Secretariat of the Inter-American Drug Abuse
Control Commission (ES-CICAD), Secretariat for Multidimensional Security of the Organization of
American States (SMS/OAS). It was developed under the aegis of the Drug Treatment Court
Program for the Americas, an initiative coordinated by ES-CICAD/SMS/OAS. This publication was
made possible through the financial contribution of the Government of the United States of
America, through the Bureau of International Narcotics and Law Enforcement Affairs (INL) of the
U.S. Department of State, and of the Government of Canada, through the Anti-Crime Capacity
Building Program (ACCBP) of Global Affairs Canada. The contents expressed in this document are
presented exclusively for informational purposes and do not necessarily represent the opinion
or official position of the Organization of American States, its General Secretariat, or its Member
States.
OAS Cataloging-in-Publication Data
Inter-American Drug Abuse Control Commission. Manualforscientificmonitoringandevaluation[of]DrugTreatmentCourtsintheAmericas.v.;cm(OAS.Officialrecords;OEA/Ser.L/XIV.6.61)ISBN978-0-8270-6830-81.Drugaddiction--Treatment.2.Substanceabuse--Treatment.3.Drugaddiction--America.4.Drugcourts.I.Title.II.Title:DrugTreatmentCourtsintheAmericas.III.Marlowe,DouglasB.IV.DrugTreatmentCourtProgramfortheAmericas.V.OrganizationofAmericanStates.SecretariatforMultidimensionalSecurity.VI.Series.OEA/Ser.L/XIV.6.61
From the Organization of American States
LuisAlmagro,SecretaryGeneral
AmbassadorNestorMendez,AssistantSecretaryGeneral
Dr.FarahUrrutia,SecretaryforMultidimensionalSecurity
AmbassadorAdamE.Namm,ExecutiveSecretary,Inter-AmericanDrugAbuseControlCommission(CICAD)
Principal AuthorDr.DouglasB.Marlowe
Chief EditorsDr.DouglasB.MarloweAntonioLombaMaurandi
Co-AuthorsDr.ShannonM.CareyDr.MichaelW.Finigan
Project DirectorAntonioLombaMaurandi
ContributorsCaroline CooperDr.CharlotteColmanYacsire CutlerAnnmarie DookieFranklinGonzalezAlfredoGonzalezPortilloHon.JusticeStephaneJackson-HaisleyJavieraIbacacheDr.NohemiLiraDr. Mariano Montenegro ConradoQuinteroMichaelRempelJeffreyE.Zinsmeister
Editorial CommitteeJavieraIbacacheDr.SofiaCoboDr. Francisco CumsilleTaniaGordilloLuisaFernandaNeiraLuisSuarez-IsazaKarellyVillanuevaPamelaI.WestfallJeffreyE.Zinsmeister
Graphic DesignAlejandroAhumadaMaria Montas
ISBN978-0-8270-6830-8
COPYRIGHT©(2019)OrganizationofAmericanStates.AllrightsreservedunderInternationalandPanamericanConventions.Thispublicationmaybereproducedprovidedcreditisgiventothesource.
ThispublicationwaspreparedbytheExecutiveSecretariatoftheInter-AmericanDrugAbuseControlCommission(ES-CICAD),SecretariatforMultidimensionalSecurityoftheOrganizationofAmericanStates(SMS/OAS).ItwasdevelopedundertheaegisoftheDrugTreatmentCourtProgramfortheAmericas,aninitiativecoordinatedbyES-CICAD/SMS/OAS.ThispublicationwasmadepossiblethroughthefinancialcontributionoftheGovernmentoftheUnitedStatesofAmerica,throughtheBureauofInternationalNarcoticsandLawEnforcementAffairs(INL)of theU.S.DepartmentofState,andof theGovernmentofCanada, through theAnti-CrimeCapacityBuildingProgram(ACCBP)ofGlobalAffairsCanada.ThecontentsexpressedinthisdocumentarepresentedexclusivelyforinformationalpurposesanddonotnecessarilyrepresenttheopinionorofficialpositionoftheOrganizationofAmericanStates,itsGeneralSecretariat,oritsMemberStates.
MANUAL FOR SCIENTIFIC MONITORING AND EVALUATIONDrug Treatment Courtsin the Americas
ACKNOWLEDGEMENTS
SpecialthankstoCICADAssistant
ExecutiveSecretaryAngelaCrowdy,
forhertirelesssupportofCICAD’s
missionandOASmemberstates;to
AmbassadorPaulSimons,forinitiating
thedialoguethatledtothedrafting
ofthisdocumentduringhisserviceas
CICADExecutiveSecretary;toDouglas
Boyd,forhisresearchassistancein
thedevelopmentofthismanual;to
theNationalAssociationofDrugCourt
Professionals,theCanadianAssociation
ofDrugTreatmentCourtProfessionals,
andFundaciónPazCiudadana,for
theircontinuedsupport;andtothe
participantsinourworkshopsinThe
Bahamas,Chile,andMexico,whose
contributionswereinvaluableforthe
creationofthismanual.
TABLE OF CONTENTS
A Welcome from the Organization of American States ......................9
Foreword ....................................................................................... 11
Background ..................................................................................... 15WhyWereDrugTreatmentCourtsCreated?.............................15AboutThisEvaluationManual..................................................18
Logic Model for DTC Program Evaluations ..................................... 21
Types of Program Evaluations ........................................................ 25ProcessEvaluations ................................................................ 25Short-TermOutcomeEvaluations.............................................28Long-TermOutcomeEvaluations.............................................29Cost Evaluations ..................................................................... 30
Sources of Information ................................................................. 32ProgramDocumentation..........................................................33StaffSurveys.............................................................................34ParticipantFocusGroups........................................................35ParticipantSurveys .................................................................36ParticipantRecords..................................................................36SiteObservations....................................................................37CollateralSources ...................................................................37
Operationalizing Variables ............................................................. 39Measurement Precision ......................................................... 42
What to Measure ............................................................................ 45ParticipantCharacteristics.......................................................45PerformanceIndicators............................................................48PerformanceBenchmarks........................................................50
Calculating Performance Indicators .............................................. 52CourtSupervision....................................................................59
SubstanceUseTreatment.........................................................60DrugandAlcoholTesting..........................................................64Abstinence ............................................................................. 65RetentionandGraduation........................................................67Recidivism...............................................................................70Incarceration .......................................................................... 73TimelinessofServices...............................................................75ProbationSupervision..............................................................76RewardsandSanctions.............................................................78RestorativeJusticeInterventions..............................................80Employment ............................................................................81Education ............................................................................... 83Housing .................................................................................84EmotionalHealth....................................................................85MedicalandDentalHealth.......................................................87Family Relationships .............................................................. 88SocialRelationships.................................................................89BirthofDrug-FreeBabies........................................................90
Attitudinal Variables ....................................................................... 91ParticipantAttitudes................................................................91StaffAttitudes......................................................................... 94
Observer Rating Scales ................................................................... 95
Comparison Groups ....................................................................... 96RandomAssignment...............................................................96Quasi-ExperimentalComparisonGroups.................................98Matched Comparison Groups ............................................... 99PropensityScoreAnalyses.....................................................100InvalidComparisonGroups....................................................101
Intent-to-Treat Analyses .............................................................. 102Cohorts..................................................................................103
Follow-Up Windows .................................................................... 104StartingtheClock...................................................................105TimeatRisk............................................................................106TimeatLiberty........................................................................107
Avoiding Common Analytic Mistakes .......................................... 109TargetPopulation...................................................................109DisadvantagedGroups..........................................................111
AdjustingforBaselineDifferences..........................................112ImpactofSpecificServicesor Interventions .......................114InfrequentEvents..................................................................116
Cost Analyses .............................................................................. 118CostAssessmentTechniques.................................................122Conclusion-CostAnalyses......................................................137
Selecting Statistical Consultants ................................................. 138
Management Information Systems ............................................ 140RecommendedMISFeatures..................................................141
Accessing Administrative Databases ........................................... 146ReachingConsensus..............................................................147Linking Records .................................................................... 148TransferringData...................................................................150EthicsandLegalTraining........................................................150
Closing Comments ........................................................................ 151
References ................................................................................ 153
Glossary of Technical Terms ........................................................ 163
Appendix: Resources for Drug Treatment Courts ........................ 173PerformanceIndicatorsandPerformanceMeasurement....173ManagementInformationSystems(MIS)..............................173AccessingAdministrativeDatabases ....................................174StaffSurveysofDTCPoliciesandPractices.............................174RatingScalesofCorrectionalProgramQuality......................174Risk-Assessment Instruments .............................................. 174Clinical-AssessmentInstruments..........................................175ProceduralJustice/ProceduralFairnessScales......................176PerceivedDeterrenceScales...................................................176Motivation/ReadinessforChangeScales...............................176SatisfactionwiththeProgram/TreatmentScales...................177TherapeuticAllianceScales...................................................177ObserverRatingScalesforCourtSessions..............................177On-SiteTrainingandTechnicalAssistance...............................177On-LineWebinarsandDistanceLearningPrograms............178Evidence-BasedTreatmentsforSubstanceAbuse.................178TrainingonResearchEthics.....................................................178
“Scientificevidenceshouldunderpin
thedevelopmentofdrugpolicy:Inthe
past,toomanypublicpolicydecisions
ondrugsweretakenbasedon‘good
intentions,’ratherthanonscientific
evidence,andsetunrealisticcompliance
goals.Nowourmemberstatesseek
toestablishandimplementevidence-
baseddrugpoliciesandactionswith
attainablegoals.Theyarealsoaware
thattoachievethisstatus,theyneedto
buildinstitutionscapableofdeveloping
objective,reliable,andcomparative
information.”
—CICADExecutiveSecretariat,
IntroductiontotheHemisphericDrug
Strategy,2010
9
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
DearReader,
Thank you for your interest in this guide on drug treatment court (DTC)
monitoring andevaluation.Wecommissioned itspublicationbecause theSecre-
tariatforMultidimensionalSecurity(SMS)seekstoguaranteegreatersecurityfor
communitiesintheWesternHemisphere.Inordertodoso,wemustaddressthe
effectsoftheworld’sdrugprobleminourhemisphere.
Ourmissiontoassess,prevent,confront,andrespondeffectivelytosecurity
threatsincludesamandatetoseeknewsolutionsforreducingandpreventingcrime,
andforprotectinghumanrights.IntheWesternHemisphere,aclearrelationship
existsbetweencrime,insufficientprotectionofhumanrights,andcorruption.In
theareaofdrugpolicy,theseissueshaveallowedcriminalorganizationsthattraffic
drugsandconductotherillicitactivitiestoprosper.
Consequently, the OAS, through the Inter-American Drug Abuse Control
Commission(knownbyitsSpanishlanguageacronym,CICAD)ofSMSsupportsand
promotesevidence-baseddrugpoliciesthatreducecrime,protecthumanrights,
focusonthewell-beingoftheperson,anddonotcharacterizethedruguserasan
objectofthecriminaljusticesystem.Insodoing,CICADrecognizestheneedto
promotedrugpoliciesanddialoguethatfocusonpublichealthandhumanrights,
while taking into considerationevidence-basedpolicies, regulatory frameworks,
civilsocietyparticipation,andgender.
TheDTCmodelisanexcellentexampleofthistypeofevidence-baseddrug
policy.OvertwodecadesofrobustacademicinquirydemonstratesthatwhenDTC
programsfollowevidence-basedpractices,theyareeffectiveinreducingcriminal
recidivism,savingpublicfunds,protectinghumanrights,andhelpingparticipants
A WELCOME FROM THE ORGANIZATION OF AMERICAN STATES
10
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
recover fromdebilitating substance use disorders. Itmay thus not come as a
surprisethatfifteenOASmemberstateshaveeitherimplementedtheDTCmodel
orareexploringitsimplementation.Moreover,DTCsareconsistentwiththeOAS
HemisphericDrugStrategyandcorrespondingPlanofActiononDrugs2016-2020,
servingasanalternativetocriminalprosecutionorimprisonment,andasasafe-
guardofhumanrights.
Nonetheless, as the term “evidence-based” suggests, the continued
success of theDTCmodel dependson sustained, rigorous scientificmonitoring
and evaluation by practitioners and the research community. This guide exists
to support these efforts, and tomake the fruits of over 20 years of academic
experience readily available to new researchers—especially those in countries
thatare intheprocessofadaptingthemodeltotheirownnationalcontexts. It
iswrittenby someof thebest and thebrightest in thefield, and covers every
majoraspectofDTCprocessesandoutcomes.Ihopeyouwillfinditasusefulasits
author,contributors,editors,andfundershavedesignedittobe.
IextendmysinceregratitudetoCanadaandtheUnitedStatesofAmerica
for providing the resources to write and publish this guide. Their citizens’
commitment to, and successwith, theDTCmodelhas servedasan inspiration
acrosstheHemisphere.
Dr. Farah UrrutiaSecretary for Multidimensional Security
11
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
FrommyvantagepointastheExecutiveSecretaryofCICAD,drugtreatment
courts (DTCs) are one of the most important programs we have, in strict
accordance with the OAS Hemispheric Drug Strategy and corresponding Plan
ofActiononDrugs2016-2020.Thesetwofundamentaldocumentscall forthe
means of offering treatment, rehabilitation, and recovery support services to
drug-dependentcriminaloffendersasanalternativetocriminalprosecutionor
imprisonment,andinfullrespectofhumanrights.
Asitstitleindicates,themanualyouarereadingexiststoteachresearchers
howtomonitorandevaluateDTCprogramsmoreeffectively.Tothelayreader,a
needtomonitorandevaluatemightsuggestacriticalproblemwiththemodelthat
needsfixing.InthecaseofDTCs,however,therealityisverymuchthecontrary.
TheneedtomonitorandevaluateDTCsstemsfromtheirsuccess.Asyouwill
seeinthefollowingpages,theDTCmodelrestsuponfoundationsofempiricism
andresearch.Lookingbacktothemodel’sinceptionin1989,wedonotfindlofty
theoreticiansdesigningprogramsperchedatopanivorytower.Instead,thein-
trepidprofessionalswhocreatedthefirstDTCsdidsofromthefield,responding
toarealproblem:theywerepractitionersmiredinthecrackcocainecrisisofthat
era.Confrontedbyawaveofcrime—fueledbysubstanceabuse—thatthreatened
tooverwhelmthecriminaljusticesystem,thesemenandwomenweredesperate
foranalternativeresponsetoaseriousdilemma.
Theysawthesameoffenders—whosesubstanceusedisorderswerethepri-
mary factor driving their criminal activity—pass through their courts again and
again.Jailwasclearlynotresolvingthatunderlyingissue.Thesedefendantswould
serve their time, but their substance abuse problem always followed them,
FOREWORD
12
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
shadow-like, back into civilian life. Predictably, many of them committed new
crimes,werere-arrested,andsentbacktojail.Meanwhile,theirsubstanceabuse
problemremaineduntreatedandunabated.
Touseaconcreteexample,considersomeonewhostealsbicyclestopayfor
adrughabit.Sendingthispersonto jail leavesthesubstanceabuseproblem
untreated—andsometimesevenundetected.Heorsheservesthesentence,ends
upbackonthestreetwiththesameneedtobuydrugsasbefore,andthevicious
cyclecontinues.
Faced with this seemingly intractable problem, these local practitioners
forged a novel solution using tools they had on hand: the criminal justice and
healthsectorswouldcometogethertooffervoluntary,court-supervisedsubstance
abusetreatmenttosubstance-dependentdefendantsinterestedinparticipating.
The concept behind this new, more collaborative approach was straight-
forward:providetreatmentandrehabilitationinsteadofexactingretribution.Of
course, putting the model into practice was more challenging. There was
noprecedentorguidelinestofollow.Thetaskofcoordinatingpractitionersfrom
thedifferentworldsofcriminaljusticeandpublichealthwasnotsimple.Judges,
prosecutors,defenseattorneys,doctors,andsocialworkershadtolearntowork
togetherasateamtohelptheDTCparticipants,insteadofseeingoneanotheras
adversaries.
Assuch,at itsoutsetthemodelranonimprovisationandgoodwill.Practi-
tionersprogressively improved themodel, keepingwhatworkedandmodifying
whatcouldbeimproved,buttheyobservedthatitseemedtofunctionoverall.DTC
graduatesappearedtodobetterthanthosewhowenttojail.
Such initial success drove an ever-greater interest in this new alternative
to incarceration. By the mid-1990s, the research community had taken note.
ResearchersappliedtheirtalentstomorerigorousevaluationoftheDTCmodel,
13
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
tosee ifpractitioners’anecdotalobservationshelduptoscientificscrutiny.This
applicationofthescientificmethodprovedpowerful.Moreformalevaluationof
themodelhasteneditsrefinementandsolidifieditsevidentiarysupport.
By2006,thescientificcommunityhadconcluded,fromadvancedresearch
techniques called meta-analyses, that the DTCmodel indeed reduced criminal
recidivism. Other studies also showed that the model also saved money over
traditionalcriminaljusticeapproaches.Moreover,researchfromothernationslike
CanadaandChile,whichhadadaptedtheDTCmodeltotheirowncircumstances,
begantoconfirmitsusefulnessinotherjurisdictionsandnationalcontexts.Bythe
closeofthatdecade,moreresearchhadbeenpublishedontheeffectsofadult
DTCsthanonvirtuallyallothercriminaljusticeprogramscombined.
Sincethen,thescientificcommunityhasbegunconsolidatingtheavailable
research into guides for practitioners, such as theNationalAssociationofDrug
CourtProfessionals’two-volumeAdultDrugCourtBestPracticeStandards.Such
documents serve a key role in ensuring that the results of very sophisticated
researchdonotendupforgotteninadustyacademicarchive,butinsteaddirectly
improvehowDTCpractitionersworkwithprogramparticipants.
Thatbringsustothedocumentyouarereadingrightnow.Forthefirsttime,
thescientificcommunityispresentingacomprehensiveguideabouttheeffective
studyofDTCsforusebypractitioners,academics,andevaluators.Thisservestwo
importantpurposes. First, continued research isessential tokeeping theDTC
modelrobustandfunctional.ThesuccesstheDTCmodelenjoysrestsonem-
piricism.Thisvolumerepresentsa“how-to”guidetospurmoreempiricalresearch.
Second,asmorecountriesadapttheDTCmodel totheirparticularneeds, their
ownacademiccommunitieswillneedtomonitorandevaluateprocessesandout-
comes.Thisguidewillacceleratethelearningcurveforresearchersthatarenew
tothefield.
14
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Inotherwords, thisdocumentdistillsmore than twodecadesof scholarly
effortintoonepubliclyavailable,accessiblevolume.Weaskedsomeofthefield’s
mostaccomplishedfigurestoserveasitsauthorandcontributors.Theseexperts
havecoveredeffectivemonitoringandevaluationpracticesfromstarttofinish.
Assuch,onbehalfofCICAD,Ihumblypositthatifyouwishtolearnhowto
monitorandevaluateDTCs,youhavecometo the rightplace—and Ihope that
whenyoufinish,youwillagreethatthisguidewasworththetimeyouinvestedin
readingit.
Ambassador Adam E. NammExecutive Secretary
Inter-American Drug Abuse Control Commission (CICAD)
15
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Why Were Drug Treatment Courts Created?
Drugtreatmentcourts(DTCs)werecreatedtoaddresstheproblemofcrimethat is
drivenbyunderlyingsubstanceusedisorders.Substanceusedisordersareamajorcon-
tributortocrimeintheWesternHemisphereandaroundtheglobe(OAS,2010).Reduce
drugandalcoholconsumption,andcrimedeclines; increaseit,andcrimerises(Chandler
etal.,2009;Hollowayetal.,2006).Yetdespiterisingincarcerationratesfordrug-related
crimesintheAmericas,consumptionanddistributionofillegaldrugshascontinuedtorise
precipitously(OAS,2011,2012a).
DTCsemergedasonealternativetoincarcerationdesignedtoaddressthisissue.DTCs
emphasize intensive community-based substance use treatment and supervision of
addictedindividualschargedwithdrug-relatedoffenses,underjudicialoversight.TheDTC
judgeleadsamultidisciplinaryteamofprofessionalsthatcommonlyincludesrepresenta-
tives from the prosecutor’s office, defense bar, treatment agencies, police, andproba-
tiondepartment.Theteammembersmeetfrequentlytoreviewparticipants’progressin
treatmentandmakerecommendationstothe judgeaboutsuitableresponsesto impose
forprogressorlackthereofintreatment.Theseresponsesmayinclude:(A)rewardssuch
asverbalpraise,reducedsupervisionrequirementsorsmallprizesforproductiveachieve-
ments;(B)sanctionssuchasverbalreprimands,communityserviceorbriefintervalsofjail
detentionforinfractions;or(C)adjustmentstoparticipants’treatmentregimensinlightof
insufficientclinicalprogress.
Additionally, participants are tested for drug and alcohol use on a random basis,
are required to complete a substance use treatment program, and must often satisfy
restorativejusticerequirements,suchasmakingrestitutiontovictims.Inpre-adjudication
DTCs,successfulgraduateshavetheircriminalcharge(s)droppedorwithdrawn,andinpost-
BACKGROUND
16
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
adjudicationDTCs,graduatesmayavoidincarcerationorreducethelengthorconditionsof
probation.
Thisapproachhasproventoworkempirically.Extensiveresearchonthemodeland
itseffects indicates thatmostDTCshavegenerated significant reductions in recidivism
anddruguse,savingsfortaxpayers,andreducedvictimizationfromavoidedcrimes.For
example,rigorousstudiesinAustralia(Jones,2011);Canada(Latimeretal.,2006;Somers
et al., 2011); and theUnited States (Mitchell et al., 2012; Rossman et al., 2011) have
demonstratedthatDTCscansignificantlyreducecrime,promoterecoveryfromaddiction,
andproducesignificantcostbenefitsmoreeffectivelythantraditionalcriminaljustice
approaches. Therefore, althoughDTCsarenot theonly solution to crime fueledby sub-
stanceusedisorders, theydo represent apromising, evidence-basedmodel that canbe
adopted,adapted,andevaluatedinotherlocalandnationalsettings.
Thisstrongempirical foundationhasdrivenasignificantexpansionofDTCs inthe
United States since thefirst suchprogramwas founded in 1989 inMiami-DadeCounty,
Florida.Asof2015,thereweremorethan3,100DTCsintheUnitedStates(NationalInsti-
tuteofJustice,2018),andthatgrowthwilllikelycontinue.A2017reportbythePresident’s
CommissiononCombatingDrugAddictionandtheOpioidCrisisrecommendedfurtherex-
pansionandfundingforDTCsatthefederallevel,andsupportforDTCsatthestatelevel
remainsstrong(President’sCommissiononCombatingDrugAddictionandtheOpioidCri-
sis,2018).
The success of DTCs in the United States has also created demand for themodel
internationally.Sincetheturnofthecentury,othernationshaveadaptedtheDTCmodel
forimplementationintheirownparticularlegal,social,andpoliticalcontexts.Startingin
the late 1990s and into the following decade, Canada, Bermuda, Cayman Islands, Chile,
Jamaica,andMexicostartedDTCpilotprojects.By2010,DTCshadexpandedtothepoint
where theOrganizationofAmerican States, through the Inter-AmericanCommission for
DrugAbuseControl(CICAD),launchedtheDrug Treatment Courts Program in the Americas,
17
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
withtheaimofsupportingtheexpansionofthemodeltoothermemberstatesthatrequest
assistance.
This CICAD programhas supported other nations’ adoption and adaptation of the
model.By2019,atleastfifteennationsandtwoterritoriesintheAmericashadexplored,
developed,orimplementedsometypeofDTCmodel:Argentina,Barbados,Belize,Bermuda,
Canada,CaymanIslands,Chile,Colombia,CostaRica,DominicanRepublic,Guyana,Jamaica,
Mexico,Panama,Peru,UnitedStates,andTrinidadandTobago. TheDTCmodelhasalso
spreadacrosstheoceantonationsinothercontinents.
InmostnationsoutsidetheUnitedStates,DTCprogramsarestill intheirformative
stages,andeffortstoevaluatetheiroutcomeshaveonlyrecentlybeeninitiated.Nonethe-
less,initialresultsarepromising.AsurveyconductedbyAmericanUniversityonbehalfof
theCICAD/OASanalyzedresponsesfromofficialsinseveralOASmemberstates,including
Canada,Mexico,Chile,Suriname,Bermuda,andJamaica(Cooperetal.,2010).Themajority
ofrespondentsreportedthatDTCsintheircountryappearedtoreducecrimebetterthan
traditionalcorrectionaldispositions,andapproximatelyhalfof therespondentsreported
achievingnotablecostsavings.A2018impactevaluationfromChile,forexample,showed
thatDTCstherereducedrecidivismbyalmost30percentcomparedtoacontrolgroup,and
a2014studyreportedsavingsofalmost75percent (US$500perparticipantpermonth)
(FundaciónPazCiudadana,2018;FundaciónPazCiudadana2014).Thesefigures suggest
thatimplementingDTCsinSouthAmericanandCaribbeannationsisfeasibleandpotentially
desirable.Inaddition,arecentprocessevaluationofthetreatmentcourtinNuevoLeón,
Mexico,concludedthattheprogramdelivershighquality,evidence-basedservices,andwas
perceivedpositivelybyparticipants(Rempeletal.,2014).
Thisexpansionandinteresthascontinuedthroughthepresentday.Consequently,asmore
andmorecountrieshaveexpressedinterestincollaboratingwiththeOrganizationofAmerican
Statesanditspartners,ithasbecomeincreasinglymoreimportanttopromotethedevelopment
ofmonitoringandevaluationcapabilitiesnecessarytosustainevidence-basedDTCprograms
18
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
(Rempel,et.al.2014).Thatneedistheprimarymotivationforthismanual,whichaimstoassist
professionalsacrosstheworldwithunderstandinghowtomonitorandevaluateDTCprograms
effectively.
About This Evaluation Manual
Inthesummerof2010,theInter-AmericanDrugAbuseControlCommission(CICAD)
oftheOrganizationofAmericanStates(OAS)adoptedtheHemisphericDrugStrategywhich,
amongotherprovisions,encouragesmemberstatestodevelopalternativestoincarceration
forindividualsthatsufferfromasubstanceabusedisorderandwhohavebeenchargedwith
crimesrelatedtothatdisorder(OAS,2010).Sincethen,andasoneofthosealternatives,by
therequestofagrowingnumberofOASmemberstates,ES/CICAD/OAShasbeenoffering
trainingandtechnicalassistancetohelpplan,implement,andevaluatenewDrugTreatment
Court(DTC)programs.
Subsequently,representativesfromseveralOASmemberstatesconvenedtoestablish
apreliminary framework forevaluatingDTCs (OAS,2012b).A consensuswas reachedat
thismeetingthateachcountryshouldevaluate itsDTCprogramstoensuretransparency
andaccountability,identifyeffectiveandineffectivepractices,attractexternalfunding,and
promotethemeritsof theDTCmodel. Itwas furtheragreedthatmemberstatesshould
consider collectinga commondatasetof coreperformance indicatorswhichcould serve
asthebasisformonitoringDTCactivitiesandimpactsintheAmericas.Theseperformance
indicators should be SMART (i.e., Specific,Measurable, Achievable, Relevant, and Time-
bound).Theyshouldalsobevalue-neutralbecausebenchmarksforsuccesshavenotyet
beenestablishedformanycountries;therefore,thereisnobasisforknowingwhatlevelsof
performanceshouldbeexpectedfromprogramsinthosecountries.Collectingperformance
datawillenableOASmemberstatestodevelopandvalidateperformancebenchmarksfor
theirprograms.
19
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Intheyearsthatfollowed,CICAD/OASledanefforttocreateandconsolidateguid-
anceforDTCmonitoringandevaluation.Throughdialoguewithexpertsandmemberstates,
andcarryingoutprocessevaluationsinsevennations,aconsensuswasdevelopedoverthe
stateofexistingresearchandcertaincoreperformanceindicators,andwrittendowninthis
manual. Themanualwasthensubjectedtoextensivepeerreviewbyexpertsfromboth
governmentandcivilsocietyorganizations.
AlthoughnobenchmarksforsuccesshaveyetbeenestablishedformanyOASmember
states,itwasagreedthatthismanualshouldpresentlessonslearnedinAustralia,Canada,
andtheUnitedStates.Performancebenchmarksfromthesethreenations,whilenotsome
sort of “gold standard” for other OAS member states to copy blindly, may still serve
as useful startingpoints for evaluators to considerwhen creating and testing their own
scientifichypotheses.Theseempiricalfoundationstookdecadesofpainstakingobservation
andresearch,asisthecasewithmanynewconcepts.Forexample,althoughthefirstDTC
wasfoundedin1989,itwasnotuntil2005(sixteenyearslater)thattheU.S.government
agencyresponsibleforevaluatingthesuccessoffederally-fundedprogramsconcludedthere
wassufficientscientificevidencetoprovethatDTCsreducedcrime(U.S.Government
AccountabilityOffice,2005).Itwasnotuntil2011,twenty-twoyearsafterthefirstDTCwas
created,thatresearchersintheUnitedStatesconcludedthatDTCsproducedotherbenefits
beyondreducingcrime,suchasreducingsubstanceuseandfamilyconflict(Rossmanetal.,
2011).Anditwasonlyin2012thatscientistsintheUnitedStatescompletedthebodyof
studiesthatwererequiredtoidentifythebestcandidatesforDTCsandsomeofthebest
practicesthatproducebetteroutcomesinDTCprograms(Marlowe,2012a).
Accordingly, this manual is designed to allow other OAS member states to take
advantageofthesedecadesofexperience. Hopefully, itwillallowthemtoestablishthe
effectivenessoftheirprogramsandidentifybestpracticesfortheirnationalcontextmore
quicklyandefficiently.
21
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Figure1depictsalogicmodelformonitoringandevaluatingDTCs.Detaileddescrip-
tionsoftheactivitiesdepictedinthelogicmodelareprovidedinthenextseveralchapters
ofthisevaluationmanual.Thelogicmodelalsoincludesanapproximatetimelineforper-
formingvarioustypesofmonitoringandevaluationactivities.
Asthediagramindicates,whenaDTCisfirstestablished,theinitialtaskforevaluators
is to perform a process evaluation aimed at ensuring theDTC is following the intended
model, treating the appropriate participants, and delivering effective services.Many of
theseactivitieswillbeconductedinthefirstyearoftheprogram,andthenrepeatedover
subsequentyearstoensureprogramoperationsarenotdepartingfrombestpractices.Part
oftheprocessevaluationmayalsoincludeestimatingthecostsofadministeringtheDTC
(cost analyses)andunderstandinghowthosecostsareincurredbydifferentagencies,such
asthecourt,probationdepartment,andtreatmentprograms(cost allocation analyses).
AftertheDTChasbeeninoperationforatleastayear(andoftenlongerthanthat),
evaluatorsmayconductanoutcome evaluationtoassesshowparticipantsareperforming.
Outcomeevaluationsmaybeconductedovertheshorttermwhileparticipantsarestillen-
rolledintheprogram(short-term outcome evaluations),and/orafterparticipantshavebeen
dischargedfromtheprogram(long-term outcome evaluations).Finally,evaluatorsmayde-
termineoverthelonger-termwhetherthecostsofadministeringtheDTCarejustifiedby
resulting improvements inparticipants’adaptivefunctioning (cost-effectiveness analyses)
andfinancial savings toparticipantsandsociety, suchasavoidingcosts related tonew
arrestsornewincarcerationsorgeneratingfinancial incomefromimprovedemployment
andeducation(cost-benefit analyses).
Animportantpointtoemphasizeisthatevaluators,DTCpersonnel,andpolicymakers
should not become discouraged or overwhelmed by the potential scope of evaluation
LOGIC MODEL FOR DTC PROGRAM EVALUATIONS
22
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
activities. Theadage“first thingsfirst” isparticularlyapt in this regard. EveryDTCcan
beginatrelativelymodestcostandefforttomeasurethequalityandquantityofitsservices.
As discussed in greater detail infra, information garnered from this process
evaluation is likely to be highly predictive of long-termoutcomes. If a DTC is delivering
high-qualityservicesandtreatingtherightparticipants,itisverylikelytoreducerecidivism,
enhance recovery from addiction, and generate cost benefits for society. Consequently,
DTCs that lack sufficient resources to conduct high-quality outcome evaluations and
cost-effectivenessevaluationscannonethelessbegintheevaluationprocessbygenerating
importantandusefulinformation.Later,ifandwhenresourcesbecomeavailable,theycan
increase the sophisticationof its evaluation activities. Additionally, tracking participants’
progress and recording/organizing data appropriately from the outset of the DTC
program—acomparativelysimpleandlow-costpropositionifdonecorrectly—willgreatly
facilitatefuturemonitoringandevaluationactivities.
23
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
FIG
UR
E 1:
DR
UG
TR
EATM
ENT
COU
RTS
EVA
LUAT
ION
LO
GIC
MO
DEL
24
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Finally,therearemanywaystoobtainusefulinformationforDTCevaluations.Forexample,
ifevaluatorswant tomeasurehowmuchtreatmentparticipants receive in theprogram,
they can review the DTC’s policies and proceduresmanual to determine howmuch
treatmentisintendedtobeprovided,interviewstaffmembersorparticipantsabouthow
much treatment is commonlyprovided,ormeasure theactualnumberof sessionseach
participantreceives.Whilethecostandeffortrequiredtoobtaintheinformationincreases
withtheprecisionofthemeasurement,inmanyinstanceslessprecisemeasurementswill
stillprovesufficientforausefulevaluation. The lessonforDTCswith limitedbudgetsor
evaluationexpertiseisthatitisoftenpossibletoobtainadequateinformationatreasonable
costandeffort.
25
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Programevaluationsgenerallyfallintooneoffourbroadcategories,dependingonthe
scopeandaimsoftheevaluation:
(1) Processevaluations,
(2) Short-termoutcomeevaluations,
(3) Long-termoutcomeevaluations,and
(4) Costevaluations.
In addition, the term impact evaluation is commonly reserved for short-term and
long-termoutcomeevaluationsthatincludeacomparisongroup.Aswillbediscussedlater,
it is necessary to include a comparison group in order to determinewhether outcomes
producedbytheDTCrepresentedasignificantimprovementoverwhatwouldhavebeen
achievedwithouttheDTC.Forexample,ifaDTChasa60percentre-arrestrate,thismight
seemhigh;however,ifthecomparisongrouphadare-arrestrateof80percent,thenthe
DTCproducedasignificantpositiveimpact.
Process EvaluationsProcess evaluations indicatewhether a program functions as planned, treats the
intendedtargetpopulationofparticipants,anddeliversthetypesanddosagesofservices
thatarelikelytoproducefavorableoutcomes.Questionsthatarecommonlyaddressedin
processevaluationsinclude:
�Whatproblemsandbarrierswereencounteredinimplementingthe
program,andhowweretheyresolved?
�HowdoDTCstaffmembers,participantsandstakeholdersperceivethe
program’saimsandeffectiveness?
�Whatarethecharacteristicsandneedsoftheparticipants,andarethe
servicesofferedintheprogramadequatetomeetthoseneeds?
TYPES OF PROGRAM EVALUATIONS
26
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
�Istheprogramservingitsintendedtargetpopulationofparticipants?
�Doestheprogramprovideasufficientdosageandqualityofservicesthatit
islikelytoproducefavorableoutcomes?
�Whattypesofrewardsandsanctionsaretypicallydelivered?
�Whatrestorativejusticerequirements(e.g.,victimrestitutionorcommunity
service)arebeingimplemented,anddoparticipantscomplywiththose
requirements?
�ArecasesresolvedmorerapidlyorlessrapidlyforDTCparticipantsthanfor
comparableindividualsinthetraditionalcriminaljusticesystem?
Aswasnotedearlier,severalCICAD-sponsoredstakeholdermeetingswereheld in
thecourseofdevelopingthisevaluationmanual.Audienceparticipantsraisednumerous
additionalquestionsofinteresttotheircountries.RepresentativesfromChile,forexample,
statedthatthefollowingissuesarecriticalforsuccessfullyinstallingaDTCinthatcountry,
anduntilsuchissuesareresolveditisverydifficulttoengageinshort-termandlong-term
planningforDTCprograms:
� IstheresufficientpoliticalwilltodesignandimplementtheDTCmodelin
thiscountry?
� Doesthephilosophyofthecriminaljusticesysteminthiscountryfocus
predominantlyonincreasingsocialcontrolofoffenders,ordoesitalsoincludean
emphasisonprovidingadequaterehabilitation,treatmentandsocialservices?
� Isthereanadequatebudgetavailableforpurposesofdevelopingand
maintainingtheDTC?
� WhichstaffmembersandstakeholdersinvolvedintheDTCareproneto
implementtheDTCmodel,andwhichonesmaybereluctantorambivalent
aboutcomponentsofthemodel?
� Whatgovernmentsector(s)isinchargeofadministeringthedaily
operationsoftheDTC?
27
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
� Isthereacoordinatingunitoragencywiththepoliticalandtechnicalsupport
neededtoleaddevelopmentofthemodelduringtheimplementationstage?
� Doesthiscoordinationunithavethepowertodevelopandfundcomponents
oftheDTC,conveneothersponsorsandstakeholderswhenneeded,and
intervenetoresolvedisagreementsandbarrierstoeffectiveimplementation?
� ArethedefenseattorneysinthecountryreluctanttoaccepttheDTCmodel
(because,forexample,theybelievetreatmentandrehabilitationimposea
higherpenalburdenthanotherapproaches)?
� DoprosecutorsbelievetheDTCmodelistoolenientcomparedwithother
approaches?
� DojudgesconsidertheDTCmodeltobeafunctionofthetreatmentor
socialservicesystemsratherthanthecourtsorjusticesystem?
� Isthereanadequatenetworkoftreatmentandrehabilitationservicesfor
peoplesufferingfromsubstanceabuseandmentalhealthissues?
� Isthereanadequatenetworkoftreatmentandrehabilitationservices
willingtoacceptandtreatpersonschargedwithcriminaloffenses?
� DoesthemultidisciplinaryteamofprofessionalsinvolvedintheDTC
possesssufficientknowledgeandexpertiserelatedtobothtreatmentand
criminaljusticeinterventions?
� DotreatmentprofessionalsinvolvedwiththeDTCagreetocoordinate
theireffortswithjudicialauthoritiesandsharecarefullydefinedtreatment-
relatedinformation?
� IstheDTCachievingthegoalsandmilestonesspecifiedinthecountry’s
legislationorrulesauthorizingandestablishingtheprogram?
Asstatedearlier,processevaluationsareusuallythefirstorderofbusinessforDTCevalua-
tors.Theyshouldideallybeperformedearlyinthecourseofdevelopingtheprogram.Until
itisestablishedthatDTCprocessesarebeingimplementedcorrectlyandthattheprogramis
servingtherightpeople,thereislittlepurposeinexaminingshort-andlong-termoutcomes.
28
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Fortunately,processevaluationscanusuallybeperformedatmodestcosttotheprogram
becausetheyoftenconsistofdescriptionsofprogramoperationsandattitudesandskillsof
staffmembers—informationthatisrelativelyinexpensivetoobtain.
Short-Term Outcome Evaluations
Short-term outcome evaluations (also referred to as proximal or intermediate
outcomeevaluations)measure participants’ performancewhile they are still enrolled in
the program. The evaluation typically focuses on outcomes during treatment that are
likelytopredictpost-programperformance.Examplesoftheseso-calledduring-treatment
outcomesthatarecommonlyexaminedinclude:
�Whatpercentageofparticipantssuccessfullygraduatedfromtheprogram?
�Whatistheaveragelengthofstayintheprogram?
�Whatistheaverageattendancerateattreatmentsessions?
�Whatproportionofurinedrugtestswasnegativeforallsubstancesofabuse?
�Whatproportionofparticipantscommittedanewoffenseortechnical
violationwhileenrolledintheprogram?
Theseshort-termoutcomesareoftenmeaningfulintheirownrighttopolicymakers,
thepublic,andotherstakeholdersofDTCprograms.Forexample,oneofthecentralaims
ofaDTCistorehabilitatepersonssufferingfromaddiction.Therefore,helpingparticipants
successfullystayinandgraduatefromtreatmentisanimportantindicatorofsuccess.
Ultimately,however,thepublicandotherstakeholdersarelikelytojudgethemerits
ofaDTCbyhowwellitreducescrime,incarcerationrates,andpublicexpenditures.There-
fore, it is important forat leastsomeshort-termoutcomemeasures tobesignificant
predictorsofcriminalreoffendingandotherlong-termoutcomes.
Fortunately,studiesconsistentlyfindthatlongerretentionintreatmentandsuccess-
fulgraduationfromDTCsdo,infact,predictgreaterreductionsincriminalrecidivism(Carey
etal.,2012;Gottfredsonetal.,2007,2008;Petersetal.,2002).Therefore,demonstrating
29
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
thataDTCincreasestreatmentretentionandgraduationratessuggeststhatitwillresultin
futurereductionsincrime.
Long-Term Outcome Evaluations
Long-term outcome evaluations (also referred to as distal outcome evaluations)
measureparticipants’performanceaftertheyarenolongerenrolledintheDTCprogram.
Often, evaluators will report post-program outcomes for participants who successfully
completedtheDTCandthosewhowereunsuccessfullydischargedorvoluntarilywithdrew
fromtheprogram.Commonquestionsthatmightbeaddressedinclude:
�Whatpercentagesofparticipantswerearrestedforanewoffenseor
convictedofanewoffenseafterbeingdischargedfromtheprogram?
�Whatpercentageofparticipantswasincarceratedforanewcrime,andfor
howlong?
�Whatpercentageofparticipantswaslivinginsafeanddrug-freehousingat
follow-up?
�Whatpercentagesofparticipantsweregainfullyemployedorenrolledin
educationalprograms?
�Whatpercentagesofparticipantswereexperiencingseriousmedical,
psychiatric,orfamilyproblemsatfollow-up?
�WhatpercentageofbabiesborntoDTCparticipantswasdeliveredhealthy
anddrug-free?
�Whatpercentageofparticipantswassufferingfromorengagedindomestic
violence?
�Whatpercentageofparticipantswasstillcommittingcrimeorabusingillicit
drugs,buthasnotbeenincontactwiththecriminaljusticesystem?
Long-termoutcomeevaluationsareoftendifficultforDTCstoconductontheirown,
because staffmust locateparticipantsafterdischarge todeterminehow theyaredoing.
FewDTCshavesufficientresourcestofollowupwithparticipantsafterdischarge,andpar-
30
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
ticipantsareoftenreluctanttospeakhonestlywithDTCstaffabouttheirperformance.
Participantsmay,forexample,feargettingintotroubleagainwiththecriminaljustice
systemiftheyadmittosubstanceuseorcrime.Similarly,theymaynotwanttodisappoint
their former therapists or other staff by acknowledging relapses or other treatment
setbacks.Inmostinstances,thereforeDTCswillneedtohireindependentevaluatorsto
conductlong-termoutcomeevaluations.
Cost Evaluations
Costevaluationsattachmonetaryvaluestotheresultsoftheaboveevaluationsto
estimatethenetfinancialimpactsofaDTC.Incostevaluations,distinctionsaremadebe-
tweenthreeconcepts:(1)investmentcosts,(2)outcomecosts,and(3)outcomesavings.
Investment costs represent theadditionalexpenditures thatwererequiredtoadminister
theDTCprogram,suchastheaddedcostsoftreatmentandfrequentcourthearings.Out-
come costsaretheexpendituresthatwereincurredbytaxpayersorthegovernmenttodeal
withtheparticipants’subsequentbehaviors,suchasthecostsofprosecutingnewoffenses
orincarceratingparticipantsfornewcrimes.Outcome savingsaremoniesrepaidtoorre-
claimedbysocietyasaresultoftheimprovedfunctioningofDTCparticipants.Forexample,
participantsmayfindworkandpayincometaxes,contributetothefinancialsupportoftheir
children,orvolunteertoworkincharitiesorsocialserviceagencies.InPanama,forexam-
ple,theCriminalProcedureCodespecifiescharitableworkasaconditionofrehabilitation,
includingvolunteeringtoworkininstitutionsofsocialwelfare,education,publicinfrastruc-
ture,orlocalgovernmentwithnopaymentfromthestate.
Bycomparingthefinancialexpendituresandsavings,economistsestimatethenet
financialbenefitsoftheDTCprogram.Questionsthatmightbeaddressedinacostevalua-
tioninclude:
�Whatweretheadditionalcostsofprovidingtreatmentandsupervision
servicesfortheDTCparticipants?
31
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
�WhatweretheadditionalcostsofholdingfrequentcourthearingsforDTC
participants?
�HowdothecostsoftheDTCcomparetothecostsoftraditional
approachestocasedisposition?
�WhatweretheoutcomecostsincurredbytheDTCparticipantsresulting
fromcriminalrecidivismortheuseofpublicorgovernmentresources,
comparedtothoseofacontrolgroup?
�Whatoutcomesavingswereproducedastheresultofincreased
employmentorproductivityonthepartoftheDTCparticipants?
�WeretheinvestmentcostsoftheDTCrecoupedbytheoutcomesavings?
ManyDTCsexamineinvestmentcostsaspartofaprocessevaluationorshort-term
outcomeevaluation.Often,DTCpersonnelarecapableofestimatinghowmuchstafftime
andotherresourcesarerequiredtoadministertheprogram.
Itismoredifficult,however,toestimateoutcomecostsandoutcomesavingsbased
onreductionsincrimeandotherlong-termoutcomes.Therefore,independentevaluators
withexpertiseincostevaluationsareoftenneededtoconductcost-effectivenessanalyses
orcost-benefitanalyses,whichcompareinvestmentcostsagainstlong-termoutcomecosts
andsavings.
32
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Bysomeestimates,conductingathoroughprogramevaluationcanrequire10percentto
25percentofaDTC’soperatingbudget(Peters,1996).Thisis,ofcourse,unsustainablefor
countriesthataredevelopingtheirDTCprogramsonapilotbasisoronasmallbudgetasa
proofofconceptbeforecommittingsubstantialresourcestotheendeavor.
Fortunately,however,therearevariouswaystoobtaininformationforaDTCprogram
evaluation,andsomemethodswill requireconsiderably lesseffortandexpensethan
others. Generally speaking, the more precise the source of the information, the more
difficultandcostly itwillbefortheevaluatortocollectthedata.Evaluatorsmustdecide
howmucheffortandresourcestheyarecapableofdevotingtomeasuringagivenvariable,
andwhethertheaddedbenefitsofmoreprecisemeasurementjustifytheadditionaltime
andexpensethatmayberequired.
Figure2depictsahierarchyofdata sources thatmaybeused inaDTCevaluation,
rangingfrominformationthatisrelativelysimpleandinexpensivetocollectatthebottom
ofthefiguretoinformationthatismorecostlyandcomplextocollectatthetopofthe
figure.Thehigherinthefigurethesourceoftheinformation,themoreprecisethedatawill
be,butalsothemoredifficultandcostlyitwillbetocollectthatdata.
SOURCES OF INFORMATION
FIGURE 2: SOURCES OF INFORMATION
33
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Consider,forexample,thevariouswaysinwhichanevaluatorcouldmeasurethefre-
quencyandimpactofcourthearingsinaDTC.Theevaluatorcouldreviewtheprogram’s
policies andproceduresmanual todeterminehowoften courthearings are intended to
beheld,interviewstaffmembersorparticipantsabouthowfrequentlycourthearingsare
typicallyheld,orextractdatafromprogramchartsorrecordsonactualattendanceratesat
courthearings.Theevaluatorcouldalsoobservecourthearingsdirectlyorinterviewpartici-
pants’friendsandfamilymemberstodeterminehowimpactfulthehearingstendtobe.
Eachoftheseapproachescarrieswithitbenefitsandburdensthatmustbetakeninto
accountbyanevaluatorwhenplanninganevaluationdesign.Bycarefullycomparingthe
advantagesanddisadvantagesofeachinadvance,theevaluatorcandesignanevaluation
thatwillbothmeetthenecessaryobjectivesandstaywithinitsallocatedbudget.
Program Documentation
PerhapsthesimplestandleastcostlymethodistoreviewtheDTC’spoliciesandpro-
ceduresmanualorotherdocumentationtodeterminehowfrequentlycourthearingsare
plannedtobescheduled.Thepoliciesandproceduresmanualmightspecify,forexample,
thatcourthearingsshouldbescheduledeverytwoweeksduringthefirstphaseofthepro-
gramandmonthlythereafter,unlessthereisagoodreasonforthejudgetoschedule
hearingsmoreor less frequently. Although thismight seem like a blunt and imprecise
methodofmeasurement,studieshavefoundthatitcan,infact,significantlypredictout-
comesinDTCsanddifferentiateeffectivefromineffectiveprograms(Careyetal.,2012;
Shaffer,2010;Rossmanetal.,2011).
Thereareobviousadvantagestothisapproach.First,theinformationcanbecollected
quicklyandwithminimaleffortbyresearchstaff. Second, it isunnecessarytowaitfora
substantialnumberofparticipantstoenterandcompletetheprogrambeforebeingableto
collectattendanceinformationatcourthearings.Areviewofprogramdocumentationcan
beaccomplishedbeforetheDTCopensitsdoorsorinthefirstyearofoperations.
34
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Ofcourse,programsdonotalwaysadheretowhatiswrittenintheirpoliciesandproce-
duresmanual,andsomeprogramsmaynothaveamanual.Practicesalsohaveatendencyto
changeor“drift”overtime.ADTCmightstartoutwiththeintentofholdingfrequentcourt
hearings,butthenfinditselfschedulinghearings lessfrequentlyduetotimeorresource
constraints.Greaterprecisionislikelytobeachievedbymeasuringwhatactuallyhappensin
theDTCratherthanwhatpeoplethoughtwouldhappenwhentheprogramwasfirstde-
velopedorthepoliciesandproceduresmanualwaslastupdated.
Staff Surveys
The next simplest and least costly approach is to survey staffmembers about
howoftencourthearingsaretypicallyheldintheprogram.MostDTCshaveacoreteam
ofapproximatelysixtotenstaffmembers,includingthejudge,programcoordinator,and
representativesfromtheprosecution,defense,treatmentprograms,probationdepartment,
andlawenforcement.Itisusuallypossibletosurveythissmallgroupofprofessionalswith
minimalexpenseandinconvenience.
Inseverallarge-scalestudiesintheUnitedStates,evaluatorsusedweb-basedsurvey
instruments (Carey et al., 2012; Shaffer, 2010) or computer-assisted personal interviews
(CAPI)(Rossmanetal.,2011)tocollectinformationfromstaffmembersindozensofgeo-
graphicallydispersedDTCprograms.Theweb-basedsurveysgatheredself-reportinforma-
tionfromstaffmemberswhologgedintoencrypteddatabasesusingsecuredusernames
andpasswords. In theCAPIstudies, the interviewersusedcomputerswithsoftwarethat
guidedthemthroughastructuredinterviewprocessandallowedthemtorecordresponses
directlyintoanelectronicdatabase.
In both types of studies, the staff members’ responses were entered into a
databasethatautomaticallytabulatedandsummarizedtheresults.TheAppendixtothis
manualprovidesinformationonhowtoobtainstaffmembersurveysthathavebeenused
successfullyinlarge-scaleDTCprogramevaluations.
35
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Thereareseverallimitationstothismethod,however.Staffmembersmaynotrecall
accuratelyhowoftencourthearingswereheld,ortheymaybemotivatedtopresentthe
DTCinanundeservedlyfavorablelight.Evaluatorsmayalsofinditnecessarytoreconcile
discrepancies instaffmembers’responses.Forexample, if the judgereportedthatcourt
hearingstypicallyoccureverytwoweeks,buttheprosecutorreportedtheyoccureveryfour
weeks,theevaluatorwouldneedtocontactthejudgeandprosecutoragaintoresolvethe
discrepancy.
Participant Focus Groups
Anothermethodistoorganizefocusgroupswithasubsetofparticipantstogauge
theirperceptionsabouttheprogram.Participantsmaymorelikelythanstafftoofferneutral
ornegativeobservationsabouttheDTC,andtheymayalsobemorelikelytorecallaccurate-
lyhowoftentheywererequiredtoappearbeforethejudgeincourt.Anotheradvantage
tofocusgroups isthattheevaluatordoesnotneedto intervieweveryparticipant inthe
program. Informationcanbecollected fromamanageablenumberofparticipantsat
reasonablecostandeffort.
Ordinarily,participantsshouldberandomlyselectedtoparticipateinfocusgroups
toensuretheyrepresenttheviewsandexperiencesofmostparticipantsintheprogram.
Itisoftennotappropriate,forexample,toselectonlysuccessfulgraduatestoparticipate
infocusgroups,becausegraduatesmayhaveverydifferentperceptionsabouttheDTCor
experiencesintheprogramthanotherparticipants.Graduatesmayhavebeensuccessful
precisely because they attendedmore frequent court hearings than other personswho
wereunsuccessfullydischargedfromtheprogram.Ifthisisthecase,thenrelyingsolelyon
graduates’responsesmightleadtheevaluatortoconclude,erroneously,thattheDTCholds
morefrequenthearingsthanitactuallydoesformanyparticipants.
36
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Participant Surveys
Amoreprecise,butalsomorecostly,methodistosurveyallormostofthepar-
ticipantsintheprogram.Thisapproachensurestheevaluatorwillassessthefullrangeof
participants’experiences in theprogram,and therefore theresultsaremore likely tobe
representativeofhowtheDTCoperatesinpractice.
Becauseeachparticipantwillreportonhisorherownattendanceincourt,theeval-
uatorwillobtainarangeofscoresfortheentiresampleofparticipants.Havingarangeof
scoreswill allow theevaluator to calculate suchmeasuresas theaveragenumberof
hearings thatwere attended, aswell as variability in thenumberof hearings thatwere
attended,suchastherangeorstandarddeviation.Aswillbediscussed later,collectinga
rangeofscoresallowstheevaluatortoemploymoresophisticatedstatisticaltechniquesand
answermoresophisticatedquestions.
Participant Records
Anothermoreprecisemethod istoexaminedatafromcourtortreatmentrecords
concerningparticipants’attendanceatcourthearings.Assumingtheinformationisentered
accurately, thismethodmayyieldthemostprecisemeasurementofattendanceatcourt
hearings,andtheevaluatorwouldobtainarangeofscoresontheentiresampleofpar-
ticipants.Measuringactualattendanceratesavoidsconcernsthatstaffmembersor
participantsmaynotrecallaccuratelyhowoftenhearingswereheld. Italsoavoidscon-
cernsthatsomestafforparticipantsmay,consciouslyorunconsciously,presenttheDTCin
anundulyfavorableorunfavorablelight.
Amajor limitationof thisapproach is that it relieson informationbeingentered
accuratelyintoparticipantrecords.Unfortunately,itisnotuncommonforevaluatorstoen-
countermissingorincompleteinformationinDTCrecords.StaffmembersinDTCsareoften
verybusy,andmaynotgivetherelativelylessexcitingtaskofdatacollectionandentrythe
attentionandcareitrequires.Ifinformationisnotenteredaccuratelyintoparticipantre-
37
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
cords,evaluatorsmayhavenooptionbuttoconductsurveysorfocusgroupsinanattempt
tomakeupformissingorincompleteinformation.
Site Observations
Evaluatorsoftenfind ithighlyenlightening toobserveDTCoperationsdirectly.For
example,ifanevaluatorwishestoexaminethequalityandimpactofcourthearings,thereis
oftennosubstituteforconductingon-siteobservationsofcourtproceedings.Theevaluator
canassess,forexample,howlongthejudgeinteractswithparticipantsincourt,whetherthe
judgegivesparticipantsachancetoexplaintheirviews,andwhetherotherteammembers,
suchasthedefenseattorneyorprosecutor,contributevaluableinput.Thislevelofqualitative
informationisdifficultorimpossibletoobtainfromprogramrecordsorparticipantsurveys.
Needlesstosay,collectingthislevelofinformationcanbecostlyandresource-
intensiveforevaluators.Notonlymustevaluatorsattendasufficientnumberofhearings
togetareliablepictureofhowtheprogramoperates,butevaluatorsmustalsobetrained
carefullyonhowtorecordreliableandvaliddataconcerningtheirobservations.Itisoften
necessarytousestandardizedratingscalestoensureinformationisrecordedinanunbiased
mannerandusefulformat.Additionalinformationaboutobserverratingscalesisprovided
inlaterchaptersofthismanual.
DTCstaffshouldnotconductsiteobservationsthemselves,astheycannotbeexpected
toprovideaccurateandunbiasedappraisalsofhoweffectivelythey interactwithpartici-
pantsorperformtheirroles.Instead,well-trained,independentobserversarerequiredfor
thistask.Siteobservationsarethereforegenerallyviewedasanadvancedevaluationpro-
cedure,andarerarelyperformedoutsideofwell-fundedevaluationprojects.
Collateral Sources
Finally, evaluators may find it useful to collect information from participants’
friends,familymembers,orotherindividualswhoareknowledgeableaboutparticipants’
38
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
experiencesintheprogram.Forexample,participantsmayviewfrequentcourthearings
asanunnecessaryburdenontheirtime,especiallyduringtheearlyphasesoftreatment.
Familymembers,however,mayhaveaverydifferentviewabouttheimportanceofcourt
hearings—seeingthem,forexample,asindispensabletokeepingtheirlovedonelaw-abiding
andabstinentfromdrugsandalcohol.Ifanevaluatorreliedsolelyonparticipantperceptions
toassessthevalueofcourthearings,heorshemightreachincorrectconclusions(e.g.,that
courthearingsshouldbescheduledlessoften,wheninfactfrequenthearingswereacritical
elementforsuccess).
Interviewingcollateral sourcescanbedifficult formanyDTCs.Participantsmaybe
reluctanttohavestaffmembersspeakwiththeirfriendsorfamilymembers,anditmaybe
difficulttoconvincefriendsorfamilytocometocourtandbeinterviewedbycriminal
justice authorities.Often it is necessary tohave trained and independent evaluators
conductcollateralinterviews,whichcanbecostlyanddifficultformanyDTCs.
39
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
In1997,theNationalAssociationofDrugCourtProfessionals intheUnitedStates
publishedwhatarecommonlyreferredtoasthe10KeyComponentsofDTCs(NADCP,1997).
Severalyearslater,theInternationalAssociationofDrugTreatmentCourts(IADTC)adopted
the10KeyComponents,andaddedthreeadditionalcomponents:
(1) Focusingoncasemanagementtoaddressancillaryneedsofparticipants
andpromotetheirsocialreintegration;
(2) Ensuringindividualizedtreatmenttoaddresstheneedsofspecial
populations,suchaswomen,participantswithco-occurringdisorders,
indigenouspopulations,andethnicminorities;and
(3) Provisionofaftercarerecoveryservices.
TheIADTCprinciplesareembodiedinadocumententitledthe13KeyPrinciplesfor
Court-directedTreatmentandRehabilitationPrograms(“13KeyPrinciples”).
OPERATIONALIZING VARIABLES
40
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
INTERNATIONAL ASSOCIATION OF DRUG TREATMENT COURTS (IADTC)13 Key Principles for Court-Directed Treatment and Rehabilitation Programmes
(1)Theprogrammesintegratesubstancedependencytreatmentserviceswithjusticesystemcaseprocessing.
(2) Using a non-adversarial approach, prosecution and defense lawyers promotepublicsafetywhileprotectingoffenders’dueprocessrights.
(3)Eligibleoffendersareidentifiedearlyandpromptlyintegratedintotheprogramme.
(4) The programmes ensure access to a continuum of substance dependencytreatmentandotherrehabilitationservices.
(5)Complianceisobjectivelymonitoredbyfrequentsubstanceabusetesting.
(6) A coordinated strategy governs responses of the court to programme non-compliance(andcompliance)byoffenders.
(7)Ongoingjudicialinteractionwitheachoffenderinaprogrammeisessential.
(8)Monitoringandevaluationmeasuretheachievementofprogrammegoalsandgaugeeffectiveness.
(9)Continuinginterdisciplinaryeducationpromoteseffectiveplanning,implementation,andoperationsofthesecourt-directedprogrammes.
(10)Forgingpartnershipsamongcourtsdirectingtreatmentprogrammes,publicagencies,andcommunity-basedorganizationsgenerateslocalsupportandenhancesprogrammeeffectiveness.
(11)Ongoingcasemanagementincludesthesocialsupportnecessarytoachievesocialreintegration.
(12) There is appropriate flexibility in adjusting programme content, includingincentivesandsanctions,tobetterachieveprogrammeresultswithparticulargroups,suchaswomen,indigenouspeopleandminorityethnicgroups.
(13)Posttreatmentandafter-careservicesshouldbeestablishedinordertoenhancelongtermprogrammeeffects.
41
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
The 13 Key Principles represent a revolutionary change in thinking about how to
successfully rehabilitate individuals suffering fromaddictionwhoare chargedwithdrug-
relatedcrimes.Itshouldbeimmediatelyapparent,however,thatmanyofthesePrinciples
aredescribedinverygeneralterms.
Consider,forexample,Principle7,whichstatesthatDTCsshouldprovide“ongoingju-
dicialinteractionwitheachoffender.”Howdoesonedefine“ongoing?”Isthefrequencyof
face-to-facecontactssufficienttomeasurethisvariable?Ifitis,howfrequentlyshouldthe
judgemeetwithparticipants?Shouldthejudgeholdhearingsonadaily,weeklyormonthly
basis,or should the frequencyofhearingsbebasedon theneedsofeachparticipant?
Similarly,whatdoes“interaction”mean?Mustthejudgespeakdirectlytoparticipants,give
themachancetovoicetheirownviews,orisitsufficienttosimplyrevieweachparticipant’s
progressandimposeconsequences?
EvaluatorsarefacedwiththedifficulttaskofdefiningthesePrinciplesinamannerthat
permitsthemtobemeasuredandexaminedobjectively.Theprocessofdefiningvariables
inmeasurableandobjectivetermsisreferredtoasoperationalizingthevariables.Different
evaluatorsarenotrequiredtoagreeonthesamedefinitionofavariable,buteachmust
make itpainstakinglyclearhowheorshe isdefiningthatvariable. If thedefinitionofa
variableisnotclearlydescribed,itwillnotbepossibleforotherevaluatorstoreplicatethe
studyorinterpretthefindings.
Forexample, it ispossibletomeasure“ongoingjudicial interaction” inanumberof
ways. Anevaluatorcouldmeasurehowoftenparticipantsappearedbefore the judge in
court. Alternatively, the he or she couldmeasure how long the judge spokewith each
participant during court hearings, or could rate how attentive or encouraging the judge
appeared to beduring his or her interactionswith participants. There is noone correct
waytooperationalize“judicialinteraction”oranyoftheother13KeyPrinciplesofaDTC
program.Instead,decidinghowtooperationalizeavariableisessentiallyavaluejudgment
thatisoftenbasedonthefollowingconsiderations:
�Burden of Collection: Some variables are considerably simpler and less costly to
42
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
measurethanothers.Forexample,itismucheasiertomeasurethenumberofcourt
hearingsthatparticipantsattendedthantouseastopwatchtotimehowlongthey
appearedbeforethejudgeincourt.Evengreatereffortwouldberequiredtoratethe
qualityofthejudge’sinteractionswiththeparticipants.
�Reliability of Measurement: Any evaluator should be able to reliably measure
thenumberof court hearings that participants attended. However, twodifferent
evaluatorscouldreachdifferentconclusionsabouthow“attentive”or“encouraging”
thejudgeappearedtobeduringhisorherinteractionswithparticipants.Toavoidthis
problem,substantialeffortandexpensemayberequiredtouseratinginstruments
thatyieldcomparablefindingsacrossdifferentevaluators.Thistime-intensiveprocess
isreferredtoasensuringinter-rater reliability.
�Cultural Significance: Variablesmayhavegreaterorlessersignificanceindifferent
contextsorcultures.Forexample,someculturesmayplacegreatvalueonshowing
respectforauthorityfigures.Citizensinthoseculturesmightbesocializedfroman
early age to avoid looking authority figures in theeye, disagreeingwith authority
figures,orspeakingfranklytoauthorityfigures. Ratherthanimprovingoutcomes,
longerandmorepersonalizedinteractionswithajudgemightcauseanxietyordis-
comfortforsuchindividuals.Therefore,itisnecessarytovalidatetheimpactofany
variable in different countries or cultures. The fact that personalized interactions
withajudgemightimproveoutcomesintheUnitedStatesdoesnotmeannecessarily
theywillhavethesameeffectinothercultures.
Measurement Precision
Variables may also be operationalized differently depending on the precision of
measurementthattheevaluatordesires.Generallyspeaking,themorepreciselyavariable
ismeasured,themoresophisticatedthequestions itcanaddress.Consider,forexample,
howanevaluatormightmeasuretheprovisionofsubstanceusetreatment inaDTC.Put
aside for themoment themorecomplicatedquestionofhow tomeasure thequalityof
treatment,andconsiderhowtomeasuretheamountoftreatmentthatwasprovided.This
43
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
variablecouldbemeasuredinatleastthreeways:
�Dosage:Thenumberoftreatmentsessionsthatwereattended.
�Attendance rate:Theratioofthenumberoftreatmentsessionsthatwereattended,
dividedbythenumberoftreatmentsessionsthatwereoriginallyscheduled.
�Density:Thenumberoftreatmentsessionsthatwerescheduledorattendedper
unitoftime,suchaspermonthorperphaseoftheprogram.
Measuringthedosageoftreatmentisrelativelystraightforwardandrequiresonlya
simpletally.Measuringtheattendancerateisabitmorecomplicatedbecauseitrequires
theevaluator tocollect twopiecesof information: thenumberofsessions thatwere
attended,andthenumberofsessionsthatwereoriginallyscheduled.Althoughthismight
seemlikeaminorburden,experienceindicatesthatmanyDTCprogramsdonotdoagood
jobofenteringmissedappointmentsintoadatabaseorparticipantrecords.Thisissome-
timesreferredtoasthe“problemofthemissingdenominator”becausethedenominator
intheattendanceratio(thenumberofsessionsthatwereoriginallyscheduled)isunknown.
To avoid this problem, it is essential for staffmembers inDTCs to recordwhether each
scheduledappointmentwaskept,notkept,rescheduled,orexcused.Asaruleofthumb,
sessionsarecountedashavingbeenrescheduledorexcusedonlyif(1)astaffmembergave
permissioninadvanceforthesessionnottooccur,or(2)theparticipantprovidedobjective
documentationthatheorshewasunabletoattendthesession,suchasprovidingadoctor’s
noteconfirmingthattheparticipantwasillorinjured.
Measuring the density of treatment is evenmore complicated because it requires
whatiscalleddate-stamping.Thismeansthateachentrymustbelinkedtothedateonwhich
theeventoccurredorshouldhaveoccurred. Forexample,attendanceateachcoun-
selingsessionmustbelinkedtothedateonwhichthesessionwasheld. Date-stamping
canbecriticalforpredictingoutcomesinDTCs.Forexample,aparticipantwhohasirregular
attendance in treatmentandtakesayear tocomplete twenty-six sessionsofcounseling is
likelytohavepooreroutcomesthananotherparticipantwhocompletesthefullsequenceof
twenty-sixsessionsontimewithinthreemonths.Withoutdate-stamping,theremightbeno
44
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
waytodifferentiatethesetwocasesandcriticallyimportantinformationcouldbelost.
Although thesemight seem like trivialmatters,manyDTCprogramevaluationshave
beenunabletoreportappreciablymorethandosage informationbecausedataonmissed
appointmentsorthedatesonwhichappointmentsoccurredwereneverrecorded.Perhapsthe
evaluatorsdidnotanticipatethisproblemorperhapstheywereunabletoconvinceprogram
staffmemberstorecordtheinformationmoreprecisely.Eitherway,greatereffortsmaybe
requiredtoperformmoresophisticatedanalysesinDTCevaluations.
Evaluatorsmustdecideforthemselveswhethertheadditionaleffortthatwillbere-
quiredtomeasureavariablemorepreciselyisjustifiedbytheabilitytoanswermoresophis-
ticatedquestionsthattheadditionalinformationprovides.Oneimportantissuetoconsider
inthisregardistheredundancyoftheinformation.Forexample,thenumberoftreatment
sessionsthatparticipantsattended ispartof thecalculationforall threevariablesof
dosage,attendancerate,anddensity.Asaresult,thesethreevariablesmaybesignificantly,
althoughonlypartially,correlatedwitheachother.Inotherwords,theymaybepartiallyre-
dundant.Variablesthatarepartiallyredundantmightpredictoutcomessimilarly,although
notequally.Densitymight,forexample,predictoutcomesbetterthandosage,butperhaps
thedegreeofpredictionthatisprovidedbydosagealonewouldbegoodenoughformany
evaluators’purposes.
Ideally,evaluatorsshouldlooktoempiricalfindingstodeterminewhethermorepre-
cisemeasurementisneededfortheirevaluations.Whereinformationisavailableonthe
redundancyofalternativemeasures,thatinformationisprovidedinthisevaluationmanual.
ThissectionalsounderscoreshowimportantandvaluableitisforDTCstafftomake
robustdatacollection,entry,andmanagementapriority.Enteringparticipantdatacorrectly
intoDTCrecordsasitisgeneratedisrelativelystraightforwardiftheDTChasthepropersys-
temsinplacetoidentify,collect,enter,andmanagerelevantdata,trainsstaffappropriately,
andperiodicallyverifiesthatthesesystemsandproceduresarebeingproperlyimplemented.
Doingsoiscertainlylessexpensiveandchallengingthanhavingtodesignsubsequentmoni-
toringandevaluationeffortsaroundincompleteormissingdata.
45
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Participant Characteristics
Participantcharacteristicsrefertoattributesofparticipantsthatpredatetheirentryinto
theDTC.Thesetypicallyinclude:
�Socio-demographicvariables,suchasage,gender,race,andemploymentstatus;
�Clinicalvariables,suchasprimarysubstance(s)used,psychiatricorsubstance
usediagnoses,andhistoryofsubstanceuseormentalhealthtreatment;and
�Criminalhistoryvariables,suchaspriorarrests,convictionsandincarcerations.
ManyDTCsandprobationdepartmentsadministerriskandneedassessmenttoolsto
matchparticipantstoappropriatelevelsoftreatmentandsupervision,andscoresonthese
toolsarealsoimportantvariablesforassessingparticipantcharacteristics.Furtherinforma-
tionaboutriskandneedassessmenttoolsisdiscussedinlaterchaptersofthismanual.
Participantcharacteristicsarereferredtoaspredictorvariablesiftheycorrelatesignifi-
cantlywithoutcomesinDTCprograms.Theyarereferredtoasriskfactorsiftheycorrelate
withpooreroutcomes,suchashigherrecidivismrates. Theymayalsobereferredtoas
moderator variables,butonlyiftheydifferentiallypredictoutcomesinDTCsascompared
toalternativeprograms,suchasprobation.Forexample,iffemalesperformedsignificant-
lybetterinaDTCthanprobation,butmalesperformedbetteronprobation,thengender
wouldbeamoderatorvariablefortheeffectsoftheDTC.Aswillbediscussedlater,deter-
miningwhetheraparticipantcharacteristicisamoderatorvariableasopposedtomerelya
predictorvariableorriskfactorrequirestheuseofmoresophisticatedstatisticaltechniques
thansimplecorrelations.
Inthiscontext,theDTCmodelasembodiedinthe13KeyPrinciplesmakestwofunda-
mentalassumptionsaboutparticipantsintheprogram.Itassumesthatparticipantsare(1)
WHAT TO MEASURE
46
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
addictedtoillicit1drugsoralcohol,and(2)unlikelytoimprovetheirbehaviorinaprogram
thatislessintensivethanaDTC,suchasprobationorareferraltotreatment.Participants
whosatisfybothconditionsarereferredtoashigh needandhigh risk.Thismeanstheyhave
ahighneedforsubstanceusetreatmentandarealsoathighriskforfailingtocomplete
treatmentunlessajudgeandteamofotherprofessionalsholdthemaccountableforcom-
plyingwithtreatment.
IfaDTCservesparticipantswhodonothavethesecharacteristics, itmightfinditself
providingtreatmenttopeoplewhodonotneedtreatmentorprovidingjudicialsupervision
topeoplewhodonotneedjudicialsupervision.StudiesfromCanadaandtheUnitedStates
have revealed that treating low-needor low-risk individuals inDTCshas thepotential to
wastescarcetreatmentand judicial resources (DeMatteoetal.,2006). It can also make
outcomesworseforlow-riskorlow-needparticipantsbyexposingthemtoantisocialpeers
orinterferingwiththeirinvolvementinproductiveactivities,suchaswork,school,home-
makingorchildcare(Lowenkamp&Latessa,2004;Taxman&Marlowe,2006).
Theparticipantcharacteristicslistedbelowhaveconsistentlybeenfoundtobepredictor
variablesor risk factors inDTCprogramevaluations inseveralcountries (Marloweetal.,
2003;Newton-Tayloretal.,2009;Somersetal.,2012).
�Current age
�Gender
�Education(numberofacademicgradescompleted)
�Employment(fulltime,parttime,seasonalortemporary,recentlyunemployed,chronicallyunemployed)
�Maritalstatus(married,nevermarried,divorced,widowed,cohabitating)
�Race
�Ethnicity
1 Illicitdrugsincludeprescriptionmedicationsthatareusedforanon-prescribedornon-medically-indicatedpurpose.
47
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
�Nationality
�Numberofpriorcriminalconvictions
�Monthsofpriorincarcerations
�Priorsubstanceusetreatmentepisodes
�Age of onset of substance use
�Ageofonsetofdelinquentorcriminalactivity
�Stableanddrug-freehousing(yesorno)
�Proportionoftimespentinteractingwithotherindividualsengagedincrimeorsubstanceuse(from“none”to“most”)
�Diagnosisofsubstancedependence/addictionvs.substanceabuse
�Primarysubstance(s)used(e.g.,opiates,cocaine,amphetamines,sedatives,hallucinogens,marijuana,alcohol)
�Co-occurringdiagnosisofmajorpsychiatricdisorder(i.e.majordepression,bipolardisorder,psychoticdisorder,post-traumaticstressdisorder)
Mostofthesevariablesaresimpleandinexpensivetocollectandshouldbeexaminedin
anyDTCprogramevaluation.Includingthesevariablesinmoderatoranalysescanassistthe
evaluatortodeterminewhichparticipantswerehelpedbytheDTC,andwhichparticipants
mightnothavebeenhelpedorperhapsevenharmed.Performingmoderatoranalysescan
prevent anevaluator from reaching theunfounded conclusion that aDTCdidnotwork,
wheninfacttherealproblemmighthavebeenthatittreatedthewrongpeople.
Better prediction is often achieved by using structured questionnaires or interviews
that combine several of these predictor variables into scales or summative scores.Risk
assessment toolsmeasurethelikelihoodthatparticipantswillcommitanewoffenseorfailin
treatmentunlesstheyreceiveintensivesupervisionsuchasthatprovidedinaDTC.Clinical
assessment toolsevaluatethediagnosticcriteriaforsubstancedependenceoraddiction,as
wellasthesymptomsofothermajorpsychiatricdisorders.TheAppendixprovideslinksto
riskassessmenttoolsandclinicalassessmenttoolsthathavebeenusedsuccessfullyinDTC
48
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
programevaluations.SeveralofthesetoolshavebeentranslatedintoSpanishorvalidated
foruseinSouthAmericancountries.
Notethattheabovelistincludesonlyvariablesthathavebeenshowntocorrelatewith
outcomesinDTCs.Aswasnotedearlier,mostDTCstudieshavebeenconductedinthe
UnitedStates,CanadaandAustralia,anditispossiblethatothervariablesmayemergeas
significantpredictorsofDTCoutcomes in SouthAmericanandCaribbeannations. These
variablesmayalsobedefinedormeasureddifferentlyinothercountries.
Performance Indicators
Performance indicators differ from participant characteristics in that they reflect
eventsoccurringafterparticipantsenteredtheDTCprogram.Therearetwogeneraltypesof
performanceindicators.Program-level performanceindicators(alsoreferredtoasinputs)
representtheservicesorinterventionsthatweredeliveredintheDTCprogram.Participant-
level performance indicators (also referred to as outputs) represent the impacts of the
DTContheparticipants.Examplesofprogram-levelperformanceindicatorsmightinclude
howoftencourthearingsortreatmentsessionswereheld. Examplesofparticipant-level
performance indicatorsmight includehowoftenparticipants testednegative for alcohol
andillicitdrugs,orthepercentageofparticipantswhograduatedfromtheDTCprogram.
Aspreviouslymentioned, itcanbechallengingforevaluatorstomeasurethewide
rangeofservicesthatDTCsprovideandthediverseimpactstheseprogramsmayhaveon
theirparticipants. Theprimary functionofaperformance indicator is to reduce this
complexinformationintoamanageableandanalyzablesetofnumericalindexes,suchas
ratios,sumsorpercentages.Thereisnoonecorrectwaytooperationalizeaperformance
indicator,andOASmemberstatesare freetodefineandmeasure thesevariables in the
mannertheydeemmostappropriateandinformative.
At thesametime,however, substantialbenefitswillbegainedbyhavingdifferent
49
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
countriescollectacommondatasetofcoreperformanceindicators.Thiswillallowmember
statestocomparetheperformanceoftheirDTCstothatofprogramsinothercountries.It
willalsobepossibletoaggregateperformanceinformationandexaminetheregionalimpact
ofDTCsintheWesternHemisphere.
Core Performance Indicators:Asnoted inaprevious section, representatives from
severalOASmemberstatesconvenedontwooccasionstoprovideguidanceonthecontent
andstructureofthisevaluationmanual.Theyagreedthatitwouldbedesirableformember
states to collect a coredatasetofperformance indicators. Collectionof the coreperfor-
manceindicatorswillbevoluntaryandeachcountrywilldecideforitselfwhethertoshare
itsfindingswithothernations.Itwasdecidedthatthefollowingcriteriashouldguidethe
selectionofcoreperformanceindicators:
�Thecoredatasetshouldcontainnomorethantenperformanceindicators
toavoidplacinganonerousburdenonmemberstates.
�Theperformanceindicatorsmustbeeasyandinexpensivetocollect,and
objectivelyandreliablymeasurable.
�Program-levelperformanceindicatorsshouldreflecttheessential
componentsofaDTC,withoutwhichaprogramwouldnotbeconsidered
aDTC.Theseinclude,ataminimum,courthearings,substanceuse
treatment,anddrugandalcoholtesting.
�Participant-levelperformanceindicatorsshouldreflectoutcomesof
primaryinteresttomanystakeholdersofDTCprograms.Theseinclude,at
aminimum,retentioninsubstanceusetreatment,useofillicitdrugsand
alcohol,andcriminalrecidivism.
�Theperformanceindicatorsmusthavebeendemonstratedinpriorresearch
topredictpost-programoutcomesinDTCsordifferentiateeffectivefrom
ineffectiveDTCprograms.
Recommended Performance Indicators:Itwasagreedfurtherthatalternativelistsof
50
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
recommendedanddiscretionaryperformanceindicatorsshouldalsobeprovided.Countries
interestedinconductingmorein-depthevaluationscanreviewtheselistsforfurtherideas
aboutpossiblewaystomeasureperformanceintheirprograms.Performanceindicatorsare
listedasrecommendediftheyhaveshownpromiseforpredictingoutcomesinDTCs,butare
difficulttomeasureorarepartiallyredundantwithotherperformanceindicators.
Discretionary Performance Indicators:Finally,performanceindicatorsarelistedas
discretionaryiftheyaretheoreticallyrelevanttoDTCprogramsorhavebeenrecommended
byleadingresearchorganizations,buthavenotyetbeenthesubjectofsubstantialresearch.
Performance Benchmarks
Performancebenchmarks refer to specific thresholds for successonperformance
indicatorsthatDTCsshouldstrivetoachieve.Forexample,aperformanceindicatormight
revealthatparticipantsinaDTCattendedanaverageoftwocourthearingspermonth.But
whetherthisisadesirablelevelofattendanceremainsanunansweredquestion.Perhaps
fourcourthearingspermonthwouldbemoreeffective,oronehearingpermonthwould
bemorecost-effective.
Whenprogramsarenew,andresearchhasnotyetidentifiedwhichpracticesproduce
betteroutcomes,performancebenchmarksarebasedoneducatedhypothesesoranecdotal
observations.Forexample,whentheearliestDTCsweredevelopedinthelate1980sand
early1990s,evaluatorshadlittlebasisforknowinghowtheprogramsshouldoperateand
whatservicestheyshouldprovide.Theevaluatorsmadeeducatedhypothesesaboutthe
bestwaystostructureanddeliverDTCservices,operationalizedvariablestomeasurethose
services,andexaminedtheeffectsoftheservicesonoutcomes.
Over time, evaluators determined empirically which practices produced better
outcomes inDTCprograms. Forexample, researchers inAustraliaandtheUnitedStates
found that scheduling court hearings every two weeks during the first phase of a DTC
51
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
produced significantly better outcomes than scheduling court hearings less frequently
(Careyetal.,2012;Jones,2011;Festingeretal.,2002;Marloweetal.,2006,2007;Zweiget
al.,2012).Basedonthatbodyofresearch,onebenchmarkforsuccessintheUnitedStatesand
AustraliaistoschedulecourthearingseverytwoweeksduringthefirstphaseofaDTCprogram.
Thesamefindingsmightnotgeneralizetonewculturesornewcountries.Research
might reveal that court hearings have a greater or lesser impact on outcomes in South
AmericanorCaribbeanNationsthantheydointheUnitedStatesorAustralia. Thisdoes
not,however,entitleevaluatorsinSouthAmericanorCaribbeancountriestoignoreprior
research.Especiallywhenfindingshavebeenreplicatedbydifferentresearchersindifferent
countries,evaluatorsinnewnationshavearesponsibilitytotakenoteofthosefindingsand
determineempiricallywhetherthesamelessonsmightapplyintheirownprograms.
ThisisnottosuggestthatevaluatorsinOASmemberstatesshouldlimittheirinquiries
toreplicatingwhatwasfoundpreviouslyinothercountries.Allevaluatorsareencouraged
toasknewquestions, developnewbenchmarks, and add to the international bodyof
knowledgeonDTCprograms.
The point here is that science proceeds, in part, through careful replication of prior
studiesandassessingthegeneralizabilityofpreviousfindings.Evaluatorshaveanobligation
to take into considerationfindings that have comebeforewhenplanning their research
agenda and conducting program evaluations. Therefore, if a benchmark for success has
beenidentifiedforagivenperformanceindicatorinatleastonecountry,thatinformation
isprovidedinthismanualforevaluatorsinOASmemberstatestoconsiderwhenexamining
theperformanceoftheirprograms.
52
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
This chapterprovidesdetailed informationabout calculatingperformance indica-
torsforDTCprogramevaluations.Table1summarizesthedataelementsthatarerequired
tocalculateeachperformanceindicator,theformulasforthecalculations,andpossible
interpretationsof theresults. The last twocolumnsofTable1 indicatewhetherstudies
have validated those performance indicators andwhether benchmarks for success have
beenidentifiedinothercountries.Foreaseofidentification,coreperformanceindicators
aredesignatedassuchinred font,recommendedperformanceindicatorsaredesignatedin
blue font,anddiscretionaryperformanceindicatorsaredesignatedinblack font.
CALCULATING PERFORMANCE INDICATORS
53
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
TABL
E 1:
Rec
omm
ende
d Pe
rfor
man
ce In
dica
tors
& P
oten
tial P
erfo
rman
ce B
ench
mar
ks fo
r DTC
Eva
luat
ions
in th
e A
mer
icas
*C
orepe
rforman
cein
dicatorsre
flectth
ede
finingfeaturesofD
TCs,arere
ason
ablyin
expe
nsivetocollect,and
sign
ificantlypredictlo
ng-te
rmoutcomes.Re
commen
dedpe
rforman
ce
indicatorssignifican
tlypredictoutcomesbutm
aybedifficulttom
easureorredu
ndan
twith
otherperform
ancein
dicators.Discretio
naryperform
ancein
dicatorsarethe
oreti
cally
impo
rtan
ttoDT
Csbuthavebe
enin
sufficien
tlystud
ied.
**Pe
rforman
ceben
chmarksarederived
fromstudiescon
ducted
inAustralia,C
anad
a,and
/ortheUnitedStates.Th
esamepe
rforman
ceben
chmarksm
aynotbevalid
inSou
th
Amer
ican
or C
arib
bean
cou
ntrie
s.
Vari
able
Data
Ele
men
tsPe
rfor
man
ce In
dica
tors
*Su
b-an
alys
esIn
terp
reta
tion
Pred
ictio
n of
Po
st-P
rogr
am
Reci
divi
sm o
r Co
sts
Pote
ntia
l Pe
rfor
man
ce
Benc
hmar
ks**
Cour
t su
perv
isio
n# h
earin
gs sc
hedu
led (H
rgS)
# hea
rings
atten
ded (
HrgA
)# h
earin
gs ca
ncell
ed or
resch
edule
d (H
rgC)
Date
stam
ped
Dose
of h
earin
gs =
Hrg
A
Cour
t app
eara
nce
rate
= H
rgA
(H
rgS
– Hrg
C)
By m
onth
or
phas
e of p
rogram
(d
ensit
y)
Inten
sity o
f judic
ial
supe
rvisio
n
Com
plian
ce w
ith
cour
t hea
rings
Mod
eratel
y pr
edict
iveEv
ery 2
week
s dur
ing
first
phas
e
Every
4 we
eks
there
after
Subs
tanc
e us
e tr
eatm
ent
# trea
tmen
t ses
sions
sche
duled
(TxS
)# t
reatm
ent s
essio
ns at
tende
d (Tx
A)# t
reatm
ent s
essio
ns ca
ncell
ed, e
xcuse
d or
resch
edule
d (Tx
C)Da
te sta
mpe
d
Dose
of t
reat
men
t = Tx
A
Trea
tmen
t atte
ndan
ce ra
te =
Tx
A (T
xS –
TxC)
By m
onth
or
phas
e of p
rogram
(d
ensit
y)
By le
vel o
f care
(e.
g., ou
tpati
ent
vs. re
siden
tial)
By m
odali
ty (e.
g., co
gnitiv
e-be
havio
ral vs
. ins
ight-o
rient
ed)
Inten
sity o
f tre
atmen
t prov
ided
Com
plian
ce w
ith
treatm
ent c
ondit
ions
Mod
eratel
y pr
edict
ive6 t
o 10 h
ours
per w
eek
durin
g first
phas
e
100 t
o 200
hour
s tota
l de
pend
ing on
the
seve
rity o
f the c
ase
54
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
TABL
E 1:
Rec
omm
ende
d Pe
rfor
man
ce In
dica
tors
& P
oten
tial P
erfo
rman
ce B
ench
mar
ks fo
r DTC
Eva
luat
ions
in th
e A
mer
icas
*C
orepe
rforman
cein
dicatorsre
flectth
ede
finingfeaturesofD
TCs,arere
ason
ablyin
expe
nsivetocollect,and
sign
ificantlypredictlo
ng-te
rmoutcomes.Re
commen
dedpe
rforman
ce
indicatorssignifican
tlypredictoutcomesbutm
aybedifficulttom
easureorredu
ndan
twith
otherperform
ancein
dicators.Discretio
naryperform
ancein
dicatorsarethe
oreti
cally
impo
rtan
ttoDT
Csbuthavebe
enin
sufficien
tlystud
ied.
**Pe
rforman
ceben
chmarksarederived
fromstudiescon
ducted
inAustralia,C
anad
a,and
/ortheUnitedStates.Th
esamepe
rforman
ceben
chmarksm
aynotbevalid
inSou
th
Amer
ican
or C
arib
bean
cou
ntrie
s.
Vari
able
Data
Ele
men
tsPe
rfor
man
ce In
dica
tors
*Su
b-an
alys
esIn
terp
reta
tion
Pred
ictio
n of
Po
st-P
rogr
am
Reci
divi
sm o
r Co
sts
Pote
ntia
l Pe
rfor
man
ce
Benc
hmar
ks**
Drug
& a
lcoh
ol
test
ing
-and
-
Abst
inen
ce fr
om
drug
s & a
lcoh
ol
# urin
e, sa
liva o
r brea
th an
alyse
s sch
ed-
uled (
UaS)
# urin
e, sa
liva o
r brea
th sa
mple
s prov
id-ed
(UaP
)# s
ample
s ne
gativ
e for
all il
licit d
rugs
an
d alco
hol (U
aN)
# sam
ples i
nvali
d or a
dulte
rated
(UaI)
# sam
ples e
xcuse
d (Ua
E)Da
te sta
mpe
d
Alte
rnat
ive:
# day
s on c
ontin
uous
drug
surve
illanc
e; e.g
., ank
let, s
weat
patch
(Day
CDS)
# day
s with
no po
sitive
read
ing fo
r illic
it dr
ugs o
r alco
hol (D
ayCD
Sneg
)
Test
ing
dose
= U
aP
- or -
DayC
DS
Test
ing
com
plia
nce
rate
= (U
aP
– UaI
) (U
aS –
UaE)
Abst
inen
ce ra
te =
(UaN
– Ua
I)
(UaS
– Ua
E)
- or -
DayC
DSne
g
By m
onth
or
phas
e of p
rogram
(d
ensit
y)
By su
bstan
ces
detec
ted
Inten
sity o
f dru
g &
alcoh
ol tes
ting;
Com
plian
ce w
ith
drug
& alc
ohol
testin
g;
Absti
nenc
e from
dr
ugs &
alco
hol
Mod
eratel
y pre-
dictiv
e Tw
ice pe
r wee
k for
ur
ine or
saliv
a tes
ting
Mini
mum
of 90
days
on
cont
inuou
s mon
i-tor
ing
Mini
mum
of 90
co
nsec
utive
days
of
absti
nenc
e; ide
ally 1
80
cons
ecut
ive da
ys of
ab
stine
nce
Grad
uatio
n or
re
tent
ion
# part
icipa
nts e
ntere
d prog
ram (E
nt)
# part
icipa
nts g
radua
ted (G
rad)
# neu
tral d
ischa
rges
(Neu
)# s
till en
rolled
(Enr
)
Alte
rnat
ive:
# day
s from
entry
to di
scharg
e or la
st in-
perso
n con
tact (D
ayED
)
Grad
uatio
n ra
te =
Gra
d (E
nt
– Neu
)- o
r –
Rete
ntio
n ra
te =
(Gra
d +
Enr
) (E
nt –
Neu)
Leng
th o
f sta
y =
DayE
D
Reas
ons f
or te
rmi-
natio
nAb
ility o
f DTC
to
retain
parti
cipan
ts lon
g eno
ugh t
o mee
t reh
abilit
ative
goals
High
ly pr
edict
iveAv
g. gr
adua
tion r
ate
~ 60
%
18 –
24 m
onth
s of
enro
llmen
t in th
e DTC
Mini
mum
of 90
days
in
treatm
ent, a
nd
ideall
y 9- 1
2 mon
ths i
n tre
atmen
t
55
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
TABL
E 1:
Rec
omm
ende
d Pe
rfor
man
ce In
dica
tors
& P
oten
tial P
erfo
rman
ce B
ench
mar
ks fo
r DTC
Eva
luat
ions
in th
e A
mer
icas
*C
orepe
rforman
cein
dicatorsre
flectth
ede
finingfeaturesofD
TCs,arere
ason
ablyin
expe
nsivetocollect,and
sign
ificantlypredictlo
ng-te
rmoutcomes.Re
commen
dedpe
rforman
ce
indicatorssignifican
tlypredictoutcomesbutm
aybedifficulttom
easureorredu
ndan
twith
otherperform
ancein
dicators.Discretio
naryperform
ancein
dicatorsarethe
oreti
cally
impo
rtan
ttoDT
Csbuthavebe
enin
sufficien
tlystud
ied.
**Pe
rforman
ceben
chmarksarederived
fromstudiescon
ducted
inAustralia,C
anad
a,and
/ortheUnitedStates.Th
esamepe
rforman
ceben
chmarksm
aynotbevalid
inSou
th
Amer
ican
or C
arib
bean
cou
ntrie
s.
Vari
able
Data
Ele
men
tsPe
rfor
man
ce In
dica
tors
*Su
b-an
alys
esIn
terp
reta
tion
Pred
ictio
n of
Po
st-P
rogr
am
Reci
divi
sm o
r Co
sts
Pote
ntia
l Pe
rfor
man
ce
Benc
hmar
ks**
Reci
divi
sm# p
artici
pant
s ent
ered p
rogram
(Ent
)# n
eutra
l disc
harg
es (N
eu)
# arre
sted f
or an
y new
offen
se (A
rr)# c
onvic
ted of
any n
ew of
fense
(Con
v)Da
te sta
mpe
dAlt
ernati
ves:
# of n
ew ar
rests
per p
artici
pant
(NoA
rr)
# of n
ew co
nvict
ions p
er pa
rticip
ant
(NoC
onv)
# of d
ays f
rom en
try to
first
new
arres
t (D
ayAr
r)
Re-a
rres
t rat
e =
Arr
(En
t –
Neu)
- and
/or -
Re-co
nvic
tion
rate
= C
onv
(Ent
– Ne
u)
Tim
e to
re-a
rres
t = D
ayAr
r
Leve
l of o
ffens
e (fe
lony v
s. m
is-de
mea
nor v
s. su
mm
ary)
Type o
f offe
nse
(e.g.,
drug
-relat
ed,
theft
, prop
erty,
or
violen
t)
In-pr
ogram
vs.
post-
prog
ram
recidi
vism
Resu
mpt
ion of
cri
mina
l acti
vity
High
ly pr
edict
iveNo
infor
mati
on
Inca
rcer
atio
n# p
artici
pant
s ent
ered p
rogram
(Ent
)# n
eutra
l disc
harg
es (N
eu)
# inc
arcera
ted fo
r any
new
offen
se or
tec
hnica
l viol
ation
(Inc)
# day
s inc
arcera
ted (D
ayInc
)
New
inca
rcer
atio
n ra
te =
Inc
(E
nt –
Neu)
Days
inca
rcer
ated
= D
ayIn
c
In-pr
ogram
vs.
post-
prog
ram
incarc
eratio
n
Priso
n vs.
jail
Exten
t of
incarc
eratio
nHi
ghly
pred
ictive
No in
form
ation
Tim
elin
ess o
f se
rvic
es
# day
s from
arres
t or v
iolati
on to
entry
(D
ayAE
)
# day
s from
entry
to fir
st tre
atmen
t con
-tac
t (Day
ET)
Inta
ke e
ffici
ency
= D
ayAE
Trea
tmen
t ent
ry e
ffici
ency
=
DayE
T
Abilit
y of p
rogr
am
to red
uce d
elays
in
case
reso
lution
an
d cap
italiz
e on
moti
vatio
n from
the
arres
t eve
nt
Mod
eratel
y pr
edict
iveM
axim
um of
50 da
ys
from
arres
t to en
try;
ideall
y with
in 20
days
56
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
TABL
E 1:
Rec
omm
ende
d Pe
rfor
man
ce In
dica
tors
& P
oten
tial P
erfo
rman
ce B
ench
mar
ks fo
r DTC
Eva
luat
ions
in th
e A
mer
icas
*C
orepe
rforman
cein
dicatorsre
flectth
ede
finingfeaturesofD
TCs,arere
ason
ablyin
expe
nsivetocollect,and
sign
ificantlypredictlo
ng-te
rmoutcomes.Re
commen
dedpe
rforman
ce
indicatorssignifican
tlypredictoutcomesbutm
aybedifficulttom
easureorredu
ndan
twith
otherperform
ancein
dicators.Discretio
naryperform
ancein
dicatorsarethe
oreti
cally
impo
rtan
ttoDT
Csbuthavebe
enin
sufficien
tlystud
ied.
**Pe
rforman
ceben
chmarksarederived
fromstudiescon
ducted
inAustralia,C
anad
a,and
/ortheUnitedStates.Th
esamepe
rforman
ceben
chmarksm
aynotbevalid
inSou
th
Amer
ican
or C
arib
bean
cou
ntrie
s.
Vari
able
Data
Ele
men
tsPe
rfor
man
ce In
dica
tors
*Su
b-an
alys
esIn
terp
reta
tion
Pred
ictio
n of
Po
st-P
rogr
am
Reci
divi
sm o
r Co
sts
Pote
ntia
l Pe
rfor
man
ce
Benc
hmar
ks**
Prob
atio
n su
perv
isio
n
# prob
ation
sessi
ons s
ched
uled (
ProbS
)# p
robati
on se
ssion
s atte
nded
(Prob
A)# p
robati
on se
ssion
s can
celle
d or r
e-sch
edule
d (Pro
bC)
# prob
ation
field
visits
; e.g.
, hom
e, wo
rk,
schoo
l (Prob
FV)
Date
stam
ped
Prob
atio
n do
se =
Pro
bA +
Pr
obFV
Prob
atio
n at
tend
ance
rate
=
Prob
A (P
robS
– Pr
obC)
By m
onth
or
phas
e of p
rogram
(d
ensit
y)
Field
vs. o
ffice
visits
Inten
sity o
f pro
ba-
tion s
uperv
ision
Ins
ufficie
ntly
studie
dNo
infor
mati
on
Rew
ards
&
sanc
tions
# rew
ards a
dmini
stered
per p
artici
pant
(R
ew)
# san
ction
s im
pose
d per
parti
cipan
t (Sa
nc)
# infr
actio
ns pe
r part
icipa
nt (In
fr)# a
chiev
emen
ts pe
r part
icipa
nt (A
ch)
Bala
nce
of re
info
rcem
ent =
Rew
Sa
nc
Cert
aint
y of
rew
ards
= R
ew A
ch
Cert
aint
y of
sanc
tions
= S
anc
Infr
By m
agnit
ude o
f rew
ards o
r sev
erity
of sa
nctio
ns
Cons
eque
nces
im
pose
d for
parti
ci-pa
nts’
cond
uct
Incon
sisten
tly
pred
ictive
4 : 1
ratio
of rew
ards t
o sa
nctio
ns
Rest
orat
ive
just
ice
inte
rven
tions
# hou
rs co
mm
unity
servi
ce pe
rform
ed
Paym
ent o
f cou
rt fin
es &
fees (
none
, pa
rtial,
com
plete)
Resti
tutio
n to v
ictim
s (no
ne, p
artial
, co
mple
te)
Satis
factio
n of o
ther
finan
cial o
bliga
-tio
ns; e
.g., ch
ild su
ppor
t, alim
ony (
none
, pa
rtial,
com
plete)
# hou
rs in
victim
impa
ct pa
nels
Not d
efine
d in t
he lit
eratu
reDe
gree
to w
hich t
he
prog
ram ap
plies
res
torati
ve ju
stice
pr
incipl
es an
d int
er-ve
ntion
s
Incon
sisten
tly
pred
ictive
No in
form
ation
57
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
TABL
E 1:
Rec
omm
ende
d Pe
rfor
man
ce In
dica
tors
& P
oten
tial P
erfo
rman
ce B
ench
mar
ks fo
r DTC
Eva
luat
ions
in th
e A
mer
icas
*C
orepe
rforman
cein
dicatorsre
flectth
ede
finingfeaturesofD
TCs,arere
ason
ablyin
expe
nsivetocollect,and
sign
ificantlypredictlo
ng-te
rmoutcomes.Re
commen
dedpe
rforman
ce
indicatorssignifican
tlypredictoutcomesbutm
aybedifficulttom
easureorredu
ndan
twith
otherperform
ancein
dicators.Discretio
naryperform
ancein
dicatorsarethe
oreti
cally
impo
rtan
ttoDT
Csbuthavebe
enin
sufficien
tlystud
ied.
**Pe
rforman
ceben
chmarksarederived
fromstudiescon
ducted
inAustralia,C
anad
a,and
/ortheUnitedStates.Th
esamepe
rforman
ceben
chmarksm
aynotbevalid
inSou
th
Amer
ican
or C
arib
bean
cou
ntrie
s.
Vari
able
Data
Ele
men
tsPe
rfor
man
ce In
dica
tors
*Su
b-an
alys
esIn
terp
reta
tion
Pred
ictio
n of
Po
st-P
rogr
am
Reci
divi
sm o
r Co
sts
Pote
ntia
l Pe
rfor
man
ce
Benc
hmar
ks**
Empl
oym
ent
# part
icipa
nts e
ntere
d prog
ram (E
nt)
# neu
tral d
ischa
rges
(Neu
)# e
mplo
yed a
t ent
ry (Em
pE)
# em
ploye
d at d
ischa
rge (
EmpD
)
Disc
harg
e em
ploy
men
t rat
e =
Em
pD (
Ent –
Neu
)
Empl
oym
ent i
mpr
ovem
ent r
ate
= (E
mpD
– Em
pE)
(Ent
– Ne
u)
Full t
ime v
s. pa
rt tim
eEx
cludin
g tem
-po
rary,
seas
onal
or of
f the b
ooks
em
ploym
ent
Type o
f wor
k (e.g
., cle
rical,
man
ual
labor,
supe
rviso
ry)
Degr
ee to
whic
h the
pr
ogram
impr
oves
th
e em
ploym
ent o
f pa
rticip
ants
High
ly pr
edict
iveDT
C offe
rs vo
catio
nal
servi
ces a
s nee
ded
Emplo
ymen
t (or
educ
ation
al en
rollm
ent) i
s a c
ondit
ion of
gr
adua
tion
Educ
atio
n# p
artici
pant
s ent
ered p
rogram
(Ent
)# n
eutra
l disc
harg
es (N
eu)
# with
high
scho
ol dip
loma o
r equ
ivalen
-cy
at en
try (D
ipE)
# with
high
scho
ol dip
loma o
r equ
ivalen
-cy
at di
scharg
e (Di
pD)
# enr
olled
in ed
ucati
onal
prog
ram at
en
try (E
dE)
# enr
olled
in ed
ucati
onal
prog
ram at
dis
charg
e (Ed
D)
Educ
atio
nal i
mpr
ovem
ent r
ate
= (D
ipD
+ E
dD) –
(Dip
E +
EdE
)
(Ent
– Ne
u)
Degr
ee to
whic
h the
pr
ogram
impr
oves
th
e edu
catio
n of
parti
cipan
ts
Incon
sisten
tly
pred
ictive
Educ
ation
al de
gree
or
enrol
lmen
t (if
unem
ploye
d) is
a c
ondit
ion of
gr
adua
tion
Hou
sing
# part
icipa
nts e
ntere
d prog
ram (E
nt)
# neu
tral d
ischa
rges
(Neu
)# i
n stab
le ho
using
at en
try (H
ouE)
# in s
table
hous
ing at
disch
arge (
HouD
)
Disc
harg
e ho
usin
g ra
te =
Hou
D (E
nt –
Neu)
Hous
ing
impr
ovem
ent r
ate
=
(Hou
D – H
ouE)
(En
t – N
eu)
By ty
pe of
hous
-ing
; e.g.
, livin
g wi
th fa
mily
or
frien
ds, re
cove
ry ho
use
Degr
ee to
whic
h the
pr
ogram
impr
oves
th
e hou
sing s
tabilit
y of
parti
cipan
ts
High
ly pr
edict
iveSo
ber h
ousin
g is
a con
dition
of
grad
uatio
n
58
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
TABL
E 1:
Rec
omm
ende
d Pe
rfor
man
ce In
dica
tors
& P
oten
tial P
erfo
rman
ce B
ench
mar
ks fo
r DTC
Eva
luat
ions
in th
e A
mer
icas
*C
orepe
rforman
cein
dicatorsre
flectth
ede
finingfeaturesofD
TCs,arere
ason
ablyin
expe
nsivetocollect,and
sign
ificantlypredictlo
ng-te
rmoutcomes.Re
commen
dedpe
rforman
ce
indicatorssignifican
tlypredictoutcomesbutm
aybedifficulttom
easureorredu
ndan
twith
otherperform
ancein
dicators.Discretio
naryperform
ancein
dicatorsarethe
oreti
cally
impo
rtan
ttoDT
Csbuthavebe
enin
sufficien
tlystud
ied.
**Pe
rforman
ceben
chmarksarederived
fromstudiescon
ducted
inAustralia,C
anad
a,and
/ortheUnitedStates.Th
esamepe
rforman
ceben
chmarksm
aynotbevalid
inSou
th
Amer
ican
or C
arib
bean
cou
ntrie
s.
Vari
able
Data
Ele
men
tsPe
rfor
man
ce In
dica
tors
*Su
b-an
alys
esIn
terp
reta
tion
Pred
ictio
n of
Po
st-P
rogr
am
Reci
divi
sm o
r Co
sts
Pote
ntia
l Pe
rfor
man
ce
Benc
hmar
ks**
Emot
iona
lH
ealth
# part
icipa
nts e
ntere
d prog
ram (E
nt)
# neu
tral d
ischa
rges
(Neu
)# r
epor
ted se
rious
emoti
onal
prob
lems a
t ent
ry (Em
otE)
# rep
orted
serio
us em
otion
al pr
oblem
s at
disch
arge (
EmotD
)
Disc
harg
e em
otio
nal p
robl
ems
= E
mot
D (E
nt –
Neu)
Emot
iona
l im
prov
emen
t rat
e =
(E
mot
E – E
mot
D) (
Ent –
Neu
)
By ty
pe of
sym
p-tom
s; e.g
., de-
pres
sion,
hosti
lity,
paran
oia
Degr
ee to
whic
h the
pr
ogram
impr
oves
th
e em
otion
al he
alth
of pa
rticip
ants
Mod
eratel
y pre-
dictiv
eDT
C offe
rs m
ental
he
alth c
ouns
eling
as
need
ed
Emoti
onal
prob
lems
have
subs
tantia
lly
impr
oved
prior
to
disch
arge
Med
ical
and
de
ntal
Hea
lth
# part
icipa
nts e
ntere
d prog
ram (E
nt)
# neu
tral d
ischa
rges
(Neu
)# r
epor
ted se
rious
med
ical o
r den
tal
prob
lems a
t ent
ry (M
edE)
# rep
orted
serio
us m
edica
l or d
ental
pr
oblem
s at d
ischa
rge (
Med
D)
Disc
harg
e m
edic
al o
r den
tal
prob
lem
s = M
edD
(Ent
– Ne
u)M
edic
al a
nd d
enta
l im
prov
e-m
ent r
ate
= (M
edE
– Med
D)
(Ent
– Ne
u)
Degr
ee to
whic
h the
pr
ogram
impr
oves
th
e med
ical a
nd
dent
al he
alth o
f pa
rticip
ants
Insuffi
cient
ly stu
died
DTC o
ffers
med
ical
and d
ental
servi
ces a
s ne
eded
Fam
ily
Rela
tions
hips
# part
icipa
nts e
ntere
d prog
ram (E
nt)
# neu
tral d
ischa
rges
(Neu
)# r
epor
ted se
rious
fam
ily pr
oblem
s at
entry
(Fam
E)# r
epor
ted se
rious
fam
ily pr
oblem
s at
disch
arge (
Fam
D)
Disc
harg
e fa
mily
pro
blem
s =
Fam
D (E
nt –
Neu)
Fam
ily im
prov
emen
t rat
e =
(F
amE
– Fam
D) (
Ent –
Neu
)
Degr
ee to
whic
h the
pr
ogram
redu
ces t
he
family
prob
lems o
f pa
rticip
ants
Insuffi
cient
ly stu
died
DTC o
ffers
family
coun
-se
ling a
s nee
ded
Fam
ily pr
oblem
s hav
e su
bstan
tially
impr
oved
pr
ior to
disch
arge
Soci
al
Rela
tions
hips
# part
icipa
nts e
ntere
d prog
ram (E
nt)
# neu
tral d
ischa
rges
(Neu
)# r
epor
ted se
rious
socia
l prob
lems a
t en
try (S
ocE)
# rep
orted
serio
us so
cial p
roblem
s at
disch
arge (
SocD
)
Disc
harg
e so
cial
pro
blem
s =
SocD
(En
t – N
eu)
Soci
al im
prov
emen
t rat
e =
(Soc
E – S
ocD)
(En
t– N
eu)
Degr
ee to
whic
h the
pr
ogram
redu
ces
the s
ocial
or in
ter-
perso
nal c
onflic
ts of
parti
cipan
ts
Insuffi
cient
ly stu
died
No in
form
ation
Birt
h of
Drug
-free
bab
ies
# dru
g- an
d alco
hol-fr
ee ba
bies d
elive
red
or fa
there
d by D
TC pa
rticip
ants
(Bab
)
Date
stam
ped
Drug
-free
bab
ies =
Bab
In-pr
ogram
vs.
post-
prog
ram
birth
s
Degr
ee to
whic
h the
pr
ogram
cont
ribut
es
to th
e hea
lth of
ne
wbor
ns
Insuffi
cient
ly stu
died
No in
form
ation
59
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Court Supervision
Inmanyways,courtsupervisionisthedefiningingredientofaDTC.Manycorrectional
rehabilitationprogramsprovidesubstanceusetreatment,probationsupervision,ordrug
andalcoholtestingfordrug-addictedpersonschargedwithcrimes;however,onlyDTCsare
supervisedbyajudgeandrequireparticipantstoappearfrequentlyincourtforstatusre-
viewhearings.NumerousstudiesinAustraliaandtheUnitedStateshavefoundthatout-
comesinDTCsweresignificantlyinfluencedbyhowoftenparticipantsappearedbeforethe
judgeincourt(Careyetal.,2012;Festingeretal.,2002;Jones,2011;Marloweetal.,2006,
2007;Mitchelletal.,2012;Zweigetal.,2012).
Courtsupervisioniscommonlymeasuredintermsofthe“dosage”ornumberofcourt
hearingsthatparticipantsattended.Dosageisgenerallyconsideredtobeaprogram-level
performance indicator because it indicates how frequently the judge supervised partici-
pantsfromthebench.Itisproposedasacoreperformanceindicator(indicatedbyred font
belowandinTable1becauseit:(1)reflectsadefiningingredientofDTCprograms,(2) is
simpleandinexpensivetocollect,(3)isobjectivelyandreliablymeasuredand(4)hasbeen
foundconsistentlytopredictoutcomesinDTCevaluations.
Dose of hearings = # of hearings attended
Evaluatorsmayalsowishtoexamineparticipants’appearancerateatscheduledcourt
hearings.Theappearancerateisaparticipant-levelperformanceindicatorbecauseitindi-
catesthedegreetowhichparticipantswerecompliantwiththeirrequirementstoattend
courthearings.Thisvariableisslightlymorecomplicatedtomeasurethandosagebecause
itrequirestheevaluatortoexamineboththenumberofhearingsthatwereattendedand
thenumberofhearingsthatwereoriginallyscheduled.Itiscalculatedbydividingthenumber
ofhearingsthatwereattendedbythenumberthatwerescheduled,minusanyhearings
thatmayhavebeencancelledorrescheduledinadvanceorwiththecourt’sapproval.
60
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Court appearance rate =# of hearings attended
# of hearings scheduled – # of hearings cancelled or rescheduled
Inmanystudies,courtappearancerateshavenotbeensignificantlymorepredictiveof
outcomesthanthedosageofcourthearings.Perhapsbecausetherepercussionsformissing
courtsessionscanbesevereandmayincludeincarceration,appearancerateshavetypically
exceeded85percent.Formathematicalreasons,variablestendtobelesspredictivewhen
thereisarestrictedrangeofscoresonthosevariables.Therefore,thecourtappearancerate
islistedasadiscretionaryperformanceindicator(representedbyblackfontintheformula
aboveandinTable1).OASmemberstatesmightfindthisperformanceindicatortobemore
predictiveofoutcomesifthereisgreatervariabilityonthemeasureintheirDTCs.
Finally,bydate-stampingthe informationrelatedtoparticipants’attendanceatcourt
hearings, evaluators canmeasure the density of court hearings thatwere scheduled or
attendedduringeachmonthorphaseoftheprogram.Intermsofperformancebenchmarks,
evidencefromtheUnitedStatessuggestscourthearingsshouldbeheldeverytwoweeks
during thefirstphaseof theprogram,butmaysubsequentlybe reduced tomonthly for
participantswhoarecompliantwiththeirtreatmentobligations(Careyetal.,2012;Marlowe
etal.,2007).IfOASmemberstateswishtosimilarlydeterminewhenitissafeandeffective
toreducethe frequencyofcourthearings in theirprograms, itwillbenecessaryto
date-stamptheattendanceinformationatcourthearings.
Substance Use Treatment
Substance use treatment is another defining ingredient of a DTC. The basic
assumptionof theDTCmodel is thataddiction is causingorexacerbatingparticipants’
criminalactivity.Therefore,substanceusetreatmentisbelievedtobeessentialtoachieve
long-termdesistance from crime. In fact, if an individual can desist from crimewithout
61
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
substanceusetreatment,heorsheisgenerallynotconsideredtobeasuitablecandidate
foratraditionalDTCprogram(Marlowe,2012b).
Similartomeasuringcourtsupervision,theprovisionofsubstanceusetreatment is
commonlymeasuredintermsofdosage(i.e.,thenumberofsessionsthatwereattended)
orparticipants’attendancerateatscheduledtreatmentsessions.Thedosageoftreatment
is considered a program-level performance indicator because it reflects the amount
of treatment services thatwere delivered. The attendance rate is a participant-level
performanceindicatorbecauseitreflectsthedegreetowhichparticipantscompliedwith
theirtreatmentobligations.Finally,bydate-stampingtreatmentinformation,evaluatorscan
measurethedensityoftreatmentthatwasscheduledordeliveredduringeachmonthor
phaseoftheprogram.
Thedosageofsubstanceusetreatment isproposedhereasacoreperformancein-
dicatorbecause it isadefining ingredientofDTCprograms, issimpleand inexpensiveto
collect,canbeobjectivelyandreliablymeasured,andsignificantlypredictsoutcomesinDTC
programs.StudiesinDTCshaveconsistentlyfoundthatthelongerparticipantsremainedin
substanceusetreatmentandthemoresessionstheyattended,thebettertheiroutcomes
(Banks&Gottfredson,2003;Gottfredsonetal.,2007;Gottfredsonetal.,2008;Petersetal.,
2002;Shaffer,2010;Taxman&Bouffard,2005).
Dose of treatment = # of treatment sessions attended
Participants’attendancerateintreatmentislistedasarecommended performance in-
dicator forOASmemberstates (reflectedbyblue font belowand inTable1). Similar to
treatment dosage, this indicator significantly predicts post-programoutcomes.However,
theattendancerateisabitmoredifficulttomeasurethandosagebecauseitrequiresthe
evaluator to examineboth thenumberof sessions thatwere attendedand thenumber
ofsessions thatwereoriginallyscheduled. Inaddition,becausethedosageof treatment
andtheattendanceratearepartiallyredundant,itisunclearwhethertheattendancerate
62
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
predictsoutcomessignificantlybetterthandosagealone.EvaluatorsinOASmemberstates
mightfindthatthedosageoftreatmentisadequatetocharacterizetheprovisionoftreat-
mentandpredictpost-programoutcomesintheirDTCprograms.
The treatmentattendance rate is calculatedbydividing thenumberof treatment
sessionsthatwereattendedbythenumberoriginallyscheduled,minusanysessionsthat
werecancelled,excusedorrescheduledinadvanceorwiththeapprovaloftreatmentstaff.
Treatment attendance rate =# of sessions attended
# of sessions scheduled- # of sessions cancelled or rescheduled
In termsofperformancebenchmarks, studies inCanadaand theUnited Stateshave
reportedthatthebestoutcomeswereachievedwhenparticipantsattendedapproximately
sixtotenhoursoftreatmentper weekduringthefirstphaseoftheprogram,foratotalof
100to200hoursoftreatmentoverninetotwelvemonths(Bourgon&Armstrong,2005;
Huebner&Cobbina,2007;Landenberger&Lipsey,2005;Petersetal.,2002;Sperberetal.,
2013).Thestudiesfoundthat100hoursoftreatmentwasgenerallysufficientfortheless
seriouslyaddictedorantisocialparticipants,whereas200hourswasrequiredforhigh-risk
participantswhohadmoreseriousaddictionsorcriminalhistories.Itisunknownwhether
thesesameperformancebenchmarkswillbeapplicable inSouthAmericanorCaribbean
countries.LocalevaluatorsinOASmemberstatesareencouragedtoexaminetheoptimum
dosageofsubstanceusetreatmentintheirownDTCprograms.
Itisalsoimportanttocategorizesubstanceusetreatmentbythemodalityorlevel
ofcare. Forexample,residentialor inpatienttreatment isusuallyconsiderablymoreex-
pensivethanoutpatienttreatmentandmaybevalueddifferentiallyincostevaluations.In
addition,thetimeparticipantsspendinresidentialorinpatienttreatmentislikelytoreduce
theirtime at libertysignificantly.Thismeansthatparticipantsarelikelytohavereduced
63
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
opportunitiestoengageinsubstanceuse,crime,andotherbehaviorsofinteresttoevalua-
torswhiletheyarebeingtreatedinaresidentialsetting.Aswillbediscussedlater,theeval-
uatormayneedtostatisticallyadjustforthenumberofdaysparticipantswereinresidential
treatmentwhenconductingcertaintypesofoutcomeanalyses.Evaluatorsarestrongly
encouraged,therefore,todate-stampthedaysonwhichparticipantsenterandaresub-
sequentlydischargedfromresidentialandinpatienttreatmentprograms.
Itmayalsobeusefultocharacterizetheorientationorphilosophyoftreatment
beingdelivered.StudiesinCanadaandtheUnitedStateshavedeterminedthatcognitive-
behavioraltherapies(CBT)producedsignificantlybetteroutcomesamongcriminaljustice
populationsthaninsight-orientedorpsychodynamictherapies(Andrews&Bonta,2010;
Gendreau,1996).Anumberofratingtoolshavebeendevelopedtomeasurethequality
andorientationoftreatmentandotherservicesdelivered incorrectionalrehabilitation
programs. These include the Correctional ProgramAssessment Inventory (CPAI)which
wasdevelopedinCanada,andtheCorrectionalProgramChecklist(CPC)developedinthe
UnitedStates.Higherratingsoftreatmentqualityontheseinstrumentshavebeenshown
to predict higher graduation rates and lower recidivism rates among DTC participants
(Gutierrez& Bourgon, 2012) and participants in other correctional treatment programs
(Lowenkampetal.,2006,2010).Importantly,ratersmustbetrainedcarefullyonhowto
administertheseinstruments.Informationontrainingcurriculafortheseinstrumentsis
providedintheAppendix.
Finally, thequalityof therelationshipbetweentheparticipantandcounselorhas
alsobeenshowntosignificantlyinfluenceoutcomesinsubstanceusetreatment.Aswillbe
discussedlater,severalinstrumentshavebeendevelopedtoassessparticipants’satisfaction
with substance use treatment and the quality of the therapeutic alliance between the
participantandtreatmentstaff.Evaluatorsmaywishtoconsiderusingthesetools,which
arelistedintheAppendix,intheirDTCprogramevaluations.
64
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Drug and Alcohol Testing
DrugandalcoholtestingisanotherkeycomponentofDTCprograms.Drugandal-
coholtestresultsoftenserveastheprincipalbasisforassessingparticipants’response
to treatment, adjusting the conditionsof treatmentwhere indicated, and administering
rewardsforabstinenceandsanctionsforcontinuedsubstanceuse.Participants inDTCs
commonly attribute their success, in part, to the frequency of drug and alcohol testing
(Goldkampetal.,2002;Kleinpeteretal.,2010;Wolfer,2006).
Somedrugtests,suchasurineorsalivatests,mustbeadministeredrepeatedlybecause
theycanonlydetectdrugoralcoholuseovershortperiodsofapproximatelytwotofour
days.Forthesetypesoftests,thecoreprogram-levelperformanceindicatoristhedosage
ornumberofteststhatwereadministered. Othertests,suchassweatpatchesorankle
monitors,canprovidecontinuoussurveillanceofdrugoralcoholuseoverextendedperiods
ofseveralweeks. Forthesetests,thecoreperformance indicator isthenumberofdays
participantsweresubjectedtocontinuoussurveillance.
Testing dose = # of urine, breath, or saliva tests administered
-and/or–
Testing dose = # of days on continuous surveillance
Evaluatorsmaywishtoexamineparticipants’complianceratewithscheduleddrugtests.
Thecompliancerateiscalculatedbydividingthenumberofteststhatwereprovided,minus
anyteststhatwere invalidoradulterated,bythenumberofteststhatwerescheduled
minusteststhatwereexcusedinadvanceorwiththecourt’sapproval.
Testing compliance rate =# of drug tests provided-# of drug tests invalid or adulterated
# of drug tests scheduled-# of drug tests excused
Fewstudieshaveexaminedtheimpactofcompliancewithdrugtestingonoutcomesin
DTCs.Compliancewithdrugtestingisoftenhighlycorrelatedorredundantwiththedosage
65
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
ofdrugtesting (discussedabove)andwith theresultsofdrugtesting (discussedbelow);
therefore, the compliance rate is unlikely to provide significant predictionbeyondother
performanceindicators.Forthisreason,itislistedasadiscretionaryperformanceindicator.
StudiesintheUnitedStateshaverevealedthattoachievethemosteffectiveoutcomes,
drugtestswithshortdetectionwindowsshouldbeadministeredno less frequentlythan
twiceperweekforatleastthefirstseveralmonthsoftheprogram(Careyetal.,2012;Gibbs
&Wakefield,20142014;Kleinpeteretal.,2010)andcontinuoussurveillancetestingshould
beconducted foraminimumofninetyconsecutivedays (Flango&Cheesman,2009). It
remainstobedeterminedwhethersimilarperformancebenchmarkswillemergeinother
OASmemberstates.
Abstinence
Achievingabstinencefromillicit2drugsandalcoholisoneoftheprimarygoalsofa
DTC.Asdiscussedearlier,theDTCmodelassumessubstanceuseiscausingorexacerbating
criminal activity. Therefore, abstinence from addictive and intoxicating substances is
believedessentialforreducingcriminalrecidivism.
Fordrugtestswithshortdetectionwindows,thecoreparticipant-levelperformance
indicatorforabstinenceisthepercentageofteststhatwerenegativeforallillicitdrugsand
alcohol.Samplesthatareinvalidoradulteratedarenotcountedasdrug-negative.Fortests
involving continuous surveillance, the coreperformance indicator is thenumberofdays
participantswereoncontinuoussurveillancewithoutapositivetestreading.
Abstinence rate=# of drug tests negative for all illicit substances – # of drug tests invalid or adulterated
# of drug tests scheduled – # of drug tests excused
-and/or–
Abstinence = # of days with no positive reading on continuous monitoring
2 Illicitdrugsincludeprescriptionmedicationsthatareusedforanon-prescribedornon-medically-indicatedpurpose.
66
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
In termsofperformancebenchmarks, studies from theUnitedStates suggestninety
consecutivedaysofabstinenceisaminimumthresholdforachievingpositiveoutcomes
aftergraduation (Careyetal.,2012). Ideally,programsshould strive forat leastone
hundredandeightydaysofconsecutiveabstinencepriortograduationinordertoincrease
theoddsofstable,long-termsobriety(McLellanetal.,2000).
The failuretoprovideadrugoralcohol testhasanuancedsignificancethatmerits
particularattention.BecauseparticipantsinDTCsaresometimesrewardedforproviding
negativedrugtestsandsanctionedforprovidingpositivedrugtests,theyareconsiderably
morelikelytoprovideaspecimeniftheyexpectittobedrug-negativethaniftheyexpectit
tobedrug-positive.
Forthisreason,thefailuretoprovideaspecimencannotbeassumedtobearandom
or ignorable event, but rather is likely to reflect an effort to conceal substance use. In
fact,manyDTCsview the failure toprovideavalid specimenasa separate infraction
from substanceuse, andone thatmay receive amore severe sanction thanproviding
adrug-positivespecimen,whichoften receivesa light sanctionoradjustment to the
participant’streatmentplan(Marlowe,2008,2011).Thisisbecauseparticipantswhofail
toprovideaspecimenarereasonablyassumedbothtohaveengagedinsubstanceuseand
tohaveconcealedtheusage.Thus,theDTCmightgiveparticipanttworesponsesoramore
severesanctionforthiscompoundissue.
Thegenerallyrecommendedcourseofactionistoassumethatmissingspecimensare
drug-positive.Ofcourse,ifaparticipantwasexcusedbystafffromprovidingaspecimen
foranacceptablereason(e.g.,becauseofillnessoranemploymentobligation),thenitis
appropriatetotreatthemissingspecimenasmissingdata.Evaluatorsmayelecttoexamine
thedatabothways,firstbytreatingmissingspecimensasdrug-positiveandsecondlyby
treatingthemasmissingdata.However,iftheresultsaresignificantlydiscrepantinthetwo
analyses,theformerresultsaremorelikelytoreflectthetruelevelsofdruguse.
Thereareadvancedstatistical imputationproceduresthatcansometimesbeusedto
67
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
compensateformissingdrug-testdata.Someimputationproceduresexaminethepattern
ofdrugtestresults immediatelybeforeandafteramissingspecimen.Forexample, ifa
participanthadseveralnegativespecimensjustbeforeandjustafteramissedspecimen,
themissingspecimenmightbepresumedtobedrug-negative.Alternatively,somepro-
ceduresmayimputetheaverageormostprevalentresultfortheparticipantorforthe
populationasthemostlikelyresultinplaceofamissingspecimen.Selectinganappropriate
imputationprocedureiscomplicatedandexpertstatisticalconsultationisusuallyrequired
toapplytheseprocedurescorrectly.
Retention and Graduation
Retention in substanceuse treatment isoneof themost significantpredictorsof
long-termoutcomes inDTCprograms. The longerparticipants remain in treatment, the
better theiroutcomes (Gottfredsonetal., 2008;Peterset al., 2002;Taxman&Bouffard,
2005).Similarly,byfarthebestoutcomesareachievedbythosewhosuccessfullycomplete
orgraduatefromaDTCprogram(Careyetal.,2012;Gottfredsonetal.,2007;Mitchelletal.,
2012;U.S.GovernmentAccountabilityOffice,2005).
MostDTCs have similar requirements for graduation and hold formal graduation
ceremoniesthatmarksuccessfulcompletionoftheprogram.DTCsthatdonothaveformal
criteriaforgraduationorgraduationceremonieshavesignificantlypooreroutcomesthan
thosethatdo(Careyetal.,2012).Graduatesaretypicallyrequiredtocompleteasubstance
usetreatmentprogram,maintainabstinencefromall illicitdrugsandalcohol forat least
ninetytoonehundredandeightydays,findemploymentorenrollinaneducationalpro-
gram,behoused insober livingquarters,payapplicablefinesandfees,andavoidnew
criminalchargesorserioustechnicalviolations.Participantswhocanachievethesegoals
arefarmorelikelythanotherstoremaindrug-freeandcrime-freeaftergraduation.
SomeDTCsmayassignaneutral dischargetoparticipantswhowerewithdrawnfrom
68
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
theprogramforreasonsbeyondthecontroloftheparticipantandtheprogram.Aneutral
dischargeisassignedmostcommonlytoparticipantswhoarewithdrawnfromtheprogram
becausetheymovedoutofthejurisdiction(withthecourt’spermission),enlistedinthe
military, or became toomedically or psychiatrically unstable to provide continuing
competent consent to participation.Aneutral dischargemay also be assigned if it is
discoveredafterthefactthattheparticipanthadbeenadmittedtotheprogramerro-
neously;forexample,iftheparticipanthadapriorconvictionthatprecludedeligibilityor
residedinajudicialdistrictthatwasnotwithinthejurisdictionalboundariesoftheDTC.A
neutraldischargeshouldneverbeappliedtocasesinwhichterminationwasrelatedinany
waytoaparticipant’sperformanceintheDTC.
Participantswho receive aneutral discharge are removed from thedenominator
whencalculatingthegraduationrate.Theformulaisthereforethenumberofparticipants
whograduatedfromtheprogram,dividedbythenumberwhoenteredtheprogramminus
thenumberwhoreceivedaneutraldischarge.ParticipantswhoarestillenrolledintheDTC
arenot included in this calculation. In termsofperformancebenchmarks, theaverage
graduationrateisapproximatelysixtypercentintheUnitedStates,withmostDTCsgraduating
betweenfiftyandseventy-fivepercentoftheirparticipants(Marloweetal.,2016).
Abstinence rate =# of participants who graduated
# of participants who entered DTC program = # of neutral discharges
Aswillbediscussedlater,evaluatorssometimesmeasureoutcomesbasedonco-
hortsofDTCparticipants.AcohortisdefinedasagroupofindividualswhoenteredtheDTC
duringthesamespecifiedtimeperiod,usuallyanintervaloftwelvemonths(Heck,2006;
Rubioetal.,2008).Forexample,allparticipantswhoenteredtheDTCbetweenJanuary1
andDecember31ofagivenyearmightbedefinedasacohort.Outcomeanalysesmaythen
beconductedseparatelyforcohortsdefined,forexampleashavingenteredtheprogram
betweenJanuary1andDecember31,2012,betweenJanuary1andDecember31,2013,
69
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
andsoforth.Thisprovidesaseriesof“snapshots”indicatinghowwelltheDTCperformed
oversuccessiveyearsofoperation.
Whenevaluatingcohorts,itisoftennecessarytoexamineparticipantswhoarestill
enrolledintheDTC.Inthiscase,theretentionrateiscalculatedbydividingthenumberof
participantswhograduatedfromtheprogramplusthenumberthatarestillenrolled,bythe
numberwhoenteredtheprogramminustheneutraldischarges:
Abstinence rate =# of participants graduated + # of participants still enrolled
# of participants that entered program - # of neutral discharges
Analternativewaytomeasureretention is the length of stay in theprogram.This is
definedasthenumberofdaysfromaparticipant’sentryintotheDTCtohisorherdischarge
or last in-person contactwith DTC staff. The length of stay is listed as a recommended
performance indicatorbecause it ishighlypredictiveofoutcomesbut isalso likely tobe
redundantwiththeretentionrateorgraduationrate.Intermsofperformancebenchmarks,
studiesintheUnitedStatessuggestDTCshavebetteroutcomeswhentheplannedlength
of stay for the program is between approximately eighteen and twenty-four months
(Careyetal.,2012;Shaffer,2010).
Length of stay = date of discharge or last personal contact with staff - date of entry
ItisnotalwaysobvioushowtodefinetheentrydateordischargedateforaDTCpartici-
pant.MostDTCshaveaformalentryhearingatwhichparticipantsenteraplea,voluntarily
waivecertainlegalrights,andareofficiallyenrolledintheprogram.Thedateofthishearing
istypicallycountedastheentrydatebecauseitmarksthespecificpointintimewhenthe
programgainedauthorityovertheindividualtoimposeconditionsoftreatmentandsuper-
vision.Someparticipantsmaystopattendingtreatmentforseveralweeksormonthsbefore
beingformallydischargedfromaDTC.Undersuchcircumstances,thelastin-personcontact
withstaffisthebestindicatorofwhentheparticipantlefttheprogram.
70
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Recidivism
Formanypolicymakersandmembersofthepublic,reducingcriminalrecidivismis
theprimarygoalofaDTC.Recidivismisdefinedhereasanyreturntocriminalactivityafter
theparticipantenteredtheDTCprogram.Ifaparticipantisarrestedfororchargedwitha
crimeforallegedactivitythatoccurredbeforeheorsheenteredtheDTC,thisisnotcounted
asrecidivismbecausetheeventoccurredpriortoentry.
Recidivismistypicallymeasuredbynew(1)arrests,(2)convictions,(3)incarcerations
or(4)self-reportedcriminalactivity.Eachmeasurehasitsownadvantagesanddisadvantages
thattheevaluatorwillneedtotakeintoconsideration.Incarcerationisdiscussedseparately
belowbecauseithasdistinctpublicsafetyandcostimplications.
There are several potential benefits to analyzingnewarrests as ameasureof
criminal recidivism. First, arrests are usually substantially closer in time to the alleged
criminal activity than convictions. In some countries, it may take months or years to
conductpleanegotiations,holdacriminaltrial,anddetermineguiltorinnocence.Evaluators
canusuallyreportarrestoutcomes inamuchshortertimethanwaitingfor lengthy legal
proceedingstoberesolved. Second,criminalcasesmaybedismissedorpleddowntoa
lesserchargeforreasonshavinglittletodowithfactualguilt,suchasinsufficientevidence
orpleabargains.Asaresult,theabsenceofaconvictionorconvictiononalessercharge
maynotreflecttheseverityoftheoffensethatactuallyoccurred.
On the other hand, individuals may be arrested for crimes they did not in fact
commit.Thismeasurecouldseriouslyoverestimatethelevelofcriminalrecidivism.Relying
onconvictiondataratherthanarrestdatamayprovidegreaterassurancesthatthecrimes
didinfactoccur.
Evaluatorsmust also consider the timeliness and accuracy of information that is
availableincriminaljusticedatabases.InsomecountriessuchastheUnitedStates,arrest
datamaybeenteredinamoretimelyandfaithfulmannerthanconvictiondata.Inother
71
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
countriessuchCanadaorJamaica,convictiondatamaybeequallyormoreaccuratethan
arrestdata. Evaluatorsmust familiarizethemselveswithhowandwhen information is
enteredintonational,state,andlocaladministrativedatabases.
Self-reportedinformationcanpotentiallyprovidethemostaccurateassessmentof
criminal recidivismbecause itdoesnotrequiredetectionorprosecutionby lawenforce-
ment.Becausemostcrimesgounreportedbyvictimsandundetectedbytheauthorities,
arrestandconvictiondataoftenunderestimatetruelevelsofcriminalactivity.Forobvious
reasons, however, individuals cannot be reliedupon to acknowledge crimesunless they
receivestrictassurancesthattheinformationwillbekeptconfidentialandwillnotbeused
againsttheminacriminalproceeding.ThisordinarilyrequiresaDTCtohireanindependent
interviewerwhohasnoconnectiontothecourtorcriminaljusticesystemtoconfidentially
surveytheparticipants.FewDTCsarelikelytohaveadequateresourcestohiresuchinde-
pendentinterviewers.
MostDTC evaluations in theUnited States have relied on arrests as the primary
measureof recidivism,whereasstudies inCanadaandAustraliahavereliedprimarilyon
convictions.With thenotableexceptionofonewell-fundednational study in theUnited
States(Rossmanetal.,2011),fewDTCevaluationshaveexaminedself-reportdataasthe
measureofrecidivism.
Arrest rates and conviction rates are calculated by dividing the number of
participantswhowerearrestedorconvictedofanewoffensebythenumberwhoentered
the program minus neutral discharges. Technical violations, defined as violations of a
courtorderthatdonotconstituteacrimeper se,areusuallycountedseparatelyfromnew
offenses.Forexample,drinkingalcoholislegalformostadultsinmostcountries,butmay
beatechnicalviolationforaDTCparticipantandcouldleadtoanewarrestorconviction
foraprobationviolation.Sanctionsfortechnicalviolationsareoftencountedasinvestment
costsfortheDTCprogram,whereassentencesordispositionsfornewoffensesareoften
countedasoutcomecosts.
72
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Re-arrest rate =# arrested
# entered program - # of neutral discharges
- and/or –
Re-arrest rate =# convicted
# entered program - # of neutral discharges
Evaluatorsmayalsowishtoexaminerecidivismintermsoftheaveragenumberof
newarrestsornewconvictionsperparticipant.Thisisusuallynotveryinformativewithin
thefirsttwelvetotwenty-fourmonthsofaparticipant’sentry.Itusuallytakesasubstantial
timeforparticipantstore-engageincriminalconduct,bedetectedbylawenforcement,and
havecriminalchargesfiledagainstthem.Forparticipantswhoreturnquicklytocriminal
behavior,mostwillhaveonlyoneortwonewarrestswithinthefirsttwoyears.Aswillbe
discussed later, thiscancreatewhat iscalledaskewed or non-normal distribution in the
data.Formathematicalreasons,oneisunlikelytodetectstatisticallysignificantdifferences
ifthedistributionisskewed.Therefore,recidivismduringthefirsttwoyearsisusuallymost
informativewhenanalyzed intermsofpercentagesofparticipantswhohadat leastone
newarrestorconviction.
Date-stamping arrest and conviction data permits the evaluator to analyze in-
programversuspost-programrecidivismseparately,andtocalculatetheaveragelengthof
timeuntilthefirstnewarrestorconvictionhasoccurred.Usingastatisticaltechniquecalled
a survival analysis,theevaluatorcandeterminethedegreetowhichtheDTCsignificantly
delayedtheonsetofnewcriminalactivity.Thetimetore-arrestislistedasarecommended
performanceindicatorbecauseithasimportantimplicationsforpublicsafetyandcostbut
isalsooftenhighlycorrelatedorredundantwithnewarrestandconvictionrates.
Itisusuallyadvisabletocategorizenewoffensesbythelevelorseverityofthe
offense(e.g.,felonyvs.misdemeanorvs.summaryoffenses)andbythetypeofcrimethat
was committed (e.g., drug offenses, property or theft offenses, violent offenses,
73
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
technicalviolations,prostitution,andtrafficoffenses).Differentcategoriesofoffensesoften
haveverydifferent implicationsforpublicsafetyandcost. Forexample,violentoffenses
mayhaveseriousvictimizationcostsandmayresultinsubstantialjailorprisonsentences,
whereasdrug-possessionoffensesmaynotdirectlyinvolvevictimsandaremorelikelyto
receivealesscostlyprobationsentence.
Aswill bediscussed later,most studieshaveexamined recidivism foraperiodof
between twoandfive years after entry into theDTC. BecausemanyDTCprogramsare
approximatelytwelvetoeighteenmonthsinlength,measuringoutcomesaftertwoyears
allowsadequatetimetoelapseformostparticipantstohavecompletedorbeendischarged
fromtheprogram.Afterthreetofiveyears,recidivismratesforpersonschargedwith
drug-relatedcrimesarelikelytoreachaplateau(Gossopetal.,2005;Inciardietal.,2004;
Martinetal.,1999).Thismeansthatmostparticipantswhowillrecidivatearelikelytohave
donesobythattime.Therefore,estimatesofrecidivismarelikelytobereasonablystable
afteraperiodofapproximatelythreetofiveyears.
Incarceration
Incarcerationtypicallyaccountsforthegreatestcosttosocietyfrompersonscharged
with crimes related to substanceusedisorders.Dependingon the country and the
conditionsofsupervision,adayinjailorprisonmaycostbetweenfiveandtwentytimes
morethanadayonprobationorincommunity-basedsubstanceusetreatment(Belenkoet
al.,2005;Zarkinetal.,2012).
Newincarcerationratesarecalculatedinasimilarmannertonewarrestratesandnew
convictionrates.Thenumberofparticipantswhowereincarceratedforanewoffenseis
dividedbythenumberwhoenteredtheprogramminustheneutraldischarges:
New incarceration rate =# incercarated
# entered program-# of neutral discharges
74
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
New incarceration ratesareusuallynot very informative,however,because theyare
redundantwithnewarrestandconvictionrates.Inmostinstances,aparticipantwillnot
beincarceratedunlessheorshewasfirstarrestedforandconvictedofanewoffenseorfor
repeatedlyviolatingthetermsofprobation.Therefore,newincarcerationratesgenerally
donotprovidemuchusefulinformationthatnewarrestandconvictionratesdonot.
Nonetheless,thelengthoftimethatparticipantswereincarceratedisfarmoreinforma-
tivethantheincarcerationrate,especiallywhenevaluatingthecostimpactsofaDTC.The
coreperformanceindicatoristhetotalnumberofdaysthatparticipantswereincarcerated.
Date-stampingisthereforecriticalforexaminingthelengthofincarceration.Thisrequires
theevaluatortorecordthedatesthatparticipantsenteredandweresubsequentlyreleased
fromcustody. Ifaparticipantwas incarceratedonmorethanoneoccasion,thedaysfor
eachepisodearesummedtodeterminethetotallengthofincarceration.
Days of incarceration = date of release from custody = date of entry into custody
Evaluatorstypicallydistinguishbetweenincarcerationthatoccurredwhileparticipants
wereenrolled in theDTC,and incarceration thatoccurredafterdischarge fromtheDTC.
In-programincarcerationoftenreflectsbriefjailsanctionsthatmaybeimposedbythejudge
asasanctionformisconduct intheprogram,whereaspost-programincarcerationtypically
reflectspre-trialdetentionor criminal sentences fornewcharges. In costevaluations,
in-programjailsanctionsareusuallycountedas investmentcosts fortheDTC,whereas
post-programsentencesordetentionarecountedasoutcomecosts.
Insomecountries,theremaybeanimportantdistinctionbetweenjailsandprisons.In
thosenationalcontexts,jailsaretypicallyusedforpre-trialdetentionorshortsentencesof
lessthanoneyear,whereasprisonsaretypicallyreservedforsentencesoflongerthanone
year.Jailsareoftenlocatedinandadministeredbythesamecountyorprecinctwherethe
offenderresidesorwheretheoffensewascommitted,whereasprisonsmaybelocateda
75
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
distanceawayandmaybeadministeredbystate,provincial,orfederalgovernmentagen-
cies.Fromafiscalstandpoint,adayinprisonmaybesubstantiallymoreexpensivethana
dayinjail,andmaythusbevalueddifferentlyinacostanalysis.
Timeliness of Services
WhentheearliestDTCsweredevelopedinthelate1980sandearly1990s,twooftheir
goalsweretoclearcloggedcaseloadsinthecourtsandresolvedrug-relatedcasesmore
efficiently.Havingaspecializedteamwithrepresentativesfromthedefenseandprosecu-
tionwas intended, inpart, toachievea speedier resolutionof factual controversiesand
imposerehabilitativedispositionsmorequickly.
Likemanyotherperformanceindicatorsmentionedinthismanual,measuringthetime-
linessofservicesinaDTCrequirescarefulattentiontodate-stamping.Onemeasure,intake
efficiency,isdefinedasthenumberofdaysfromarresttoentryintotheDTCprogram.Some
DTCsserveindividualswhoarechargedwithviolationsofprobation,inwhichcasethedateof
theprobationviolationiscountedinlieuofanarrestdate.StudiesintheUnitedStateshave
foundthatoutcomesweresignificantlybetterforDTCsthatreducedthedelayfromarrestto
entrytolessthanfiftydays,andideallytolessthantwentydays(Careyetal.,2012).Evalua-
torsfromotherOASmemberstateswillneedtoestablishtheirownperformancebenchmarks
fortheirprograms,butitishighlylikelythatspeedierentryintotheDTCprogramwillproduce
significantlybetteroutcomes.Intakeefficiencyisthereforecalculatedas:
Intake efficiency = Date of entry into DTC-Date of arrest or probation violation
EvaluatorsmayalsowishtocalculatethetimedelayfromentryintotheDTCtothefirst
treatmentsession,calledtreatment intake efficiency:
Inmoststudiestodate,onceparticipantsenteredtheDTCs,theprogramswereusually
abletoinitiatetreatmentwithinonlyaboutonetotwoweeks.Therefore,theshorttime
delaybetweenentryandthefirsttreatmentsessionwasnotfoundtopredictoutcomesin
76
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
manyinstances.Ifothercountriesexperiencelongerdelaysininitiatingtreatmentfortheir
DTCparticipants,thisperformanceindicatormightprovidebetterpredictionofoutcomes.
Becauseithasgenerallynotbeenfoundtobepredictiveofoutcomesthusfar,itislistedasa
discretionaryperformanceindicatorforevaluatorstoconsider,andiscalculatedasfollows:
Treatment intake efficiency = Date of first treatment session-Date of entry into DTC
Probation Supervision
DTCscombinesubstanceusetreatmentwithstrictbehavioralaccountability.Insome
countriessuchastheUnitedStates,Canada,andtheUnitedKingdom,probationofficersor
othercommunitysupervisionofficersplayacriticalroleinsupervisingparticipantperfor-
manceintheprogram.Inthosecountries,theprobationofficermaymonitorparticipants’
compliancewithtreatment,conducthomeoremploymentvisits,performdrugandalcohol
testing,andreportcomplianceinformationtothejudge.Ifaparticipantcommitsaserious
infractionorabscondsfromtheprogram,theprobationofficermaybeempoweredtotake
theindividualintocustodyandbringhimorherbeforethecourtforaresponse.
Othercountries,suchasPanama,donothaveprobationofficers;rather,thejudgeand
psychosocialteamareprimarilyresponsibleformonitoringcomplianceintheprogram.For
countriesthatdonothaveprobationofficers,performanceindicatorsforprobationsuper-
visionmaynotbearelevantconsiderationforDTCevaluations.
Fewstudieshaveexamined thecontributionsofprobation toDTCoutcomes,andno
performancebenchmarkshaveyetbeen identified forprobationsupervision inDTCpro-
grams.Forthisreason,theperformanceindicatorsforprobationsupervisionarelistedas
discretionary.ItishopedthatevaluatorsinOASmemberstateswillfocusgreaterattention
onthisimportantissueandcontributetotheinternationalbodyofresearchonbestpractices
forprobationofficers,whererelevant,inDTCprograms.
Performanceindicatorsforprobationsupervisionaremeasuredinasimilarmannerto
77
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
those forcourthearingsandsubstanceusetreatment.Thedosageofprobationsuper-
visioniscalculatedbythenumberofin-officeprobationsessionsthatwereattendedplus
thenumberof field visits thatwere conducted.A field visitmight include inspecting
aparticipant’shomeorverifyingthattheparticipantwaspresentathisorherschoolor
placeofemploymentattheappropriatetimes.Thisvariableisconsideredaprogram-level
performance indicatorbecause it reflects theamountofprobationsupervision thatwas
providedtoparticipants.
Dose of probation = # of probation sessions attended + # of field visits
Theprobationattendanceraterepresentstheproportionofprobationsessionsthatwere
attended,dividedbythenumberofsessionsthatwerescheduledminusanysessionsthat
werecancelledorrescheduledinadvanceorwiththepermissionoftheprobationofficer.
Fieldvisitsarenotincludedinthiscalculationbecausetheyaretypicallyunannounced,and
participantsusuallydonotchoosewhethertocomplywiththem.Thisvariableisapartici-
pant-levelperformanceindicatorbecauseitreflectsthedegreetowhichparticipantswere
compliantwithprobationrequirements.Also,bydate-stampingprobationinformation,the
evaluatorcanmeasurethedensityofprobationservicesthatwerescheduledordelivered
duringeachmonthorphaseoftheprogram.
InmanyDTCs, probation supervision, like court hearings andother supervision re-
quirements,aregraduallyreducedinintensityorfrequencyasparticipantsmakeprogress
intreatment.Byexaminingthedensityofsupervisionpermonthorphaseoftheprogram,
evaluatorscanmeasurethedegreetowhichprobationsupervisionandotherserviceswere
decreasedinintensityovertime.Theprobationattendancerateiscalculatedasfollows:
Treatment intake efficiency =# of probation sessions attended
# of sessions scheduled - # of sessions cancelled or rescheduled
78
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Rewards and Sanctions
PerformanceindicatorsforrewardsandsanctionshavemetwithlimitedsuccessinDTC
evaluationstodate(Bonomo,2012;Linquistetal.,2006).Moststudieshavereportedthe
averagenumbersofrewardsandsanctionsthatwereadministered,andperhapscatego-
rizedtherewardsandsanctionsbymagnitudeorseverity.
Thisapproachhasnotbeeninformativeforatleasttworeasons.First,theimposition
of rewardsandsanctions ishighlycorrelatedwithotherperformance indicators,suchas
treatmentattendanceratesanddrugtestresults.Bydesign,participantsinDTCsreceive
rewards for suchbehaviorsasattending treatment sessionsandprovidingdrug-negative
urinespecimensandreceivesanctionsformissingtreatmentsessionsandcommittingother
violations.Therefore,thenumbersofrewardsandsanctionsthatareimposedaretypically
redundantwithotherperformanceindicators.
Second,behavioralresearchrevealsthatthecriticalfactorforsuccessisnothowmany
rewardsorsanctionswereimposed,butratherwhethertheywereappliedwithcertainty
andimmediacy(Harrell&Roman,2001;Hawkin&Kleiman,2009).Certaintyreferstothe
ratioofrewardstoachievements,ortheratioofsanctionstoinfractions(Marlowe&Kirby,
1999).Forexample,ifparticipantsreceivedarewardforeveryachievementintheprogram,
orasanctionforeveryinfraction,thenthecertaintyratiowouldbe1.0(or100percent)and
theprogramwouldbemorelikelytoproducefavorableoutcomes.
Certainty of rewards =# of rewards administered
# of achievements
Certainty of sanctions =# of sanctions imposed
# of infractions committed
79
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Thecertaintyratiosforrewardsandsanctionsarelistedasdiscretionaryperformance
indicatorsbecausetheyhavenotbeenvalidatedinDTCprogramevaluations,andbecause
noperformancebenchmarkshavebeen identified foruse inDTCsorother criminal jus-
ticeprograms.Inaddition,theyarelikelytobeconsiderablymoredifficultandexpensive
tomeasurethanotherperformanceindicatorsdiscussedthusfar.Evaluatorswillneedto
keeptracknotonlyofthenumbersofrewardsandsanctionsthatareimposed,butalsoof
thenumbersofachievementsandinfractionsthatoccurred.Becausemanydifferent
behaviorsmaybecountedasachievementsor infractionsinaDTC,thisprocessrequires
carefulmeasurementofnumerousvariables.Formanyevaluators,thislevelofeffortmay
bemorecomplicatedorcostlythanisfeasibleordesirable.
Effectiveness is also substantially influencedby the immediacywithwhich rewards
andsanctionsareimposed(Harrell&Roman,2001;Hawkin&Kleiman,2009;Marlowe&
Kirby,1999).Thesoonerrewardsareadministeredafteranachievement,andthesooner
sanctionsareimposedafteraninfraction,thebettertheeffectsarelikelytobe.Examining
immediacywouldrequireevaluatorstomeasurethetimedelaybetweeneachachievement
andthedeliveryofareward,andbetweeneachinfractionandthedeliveryofasanction.
Thisprocesswouldadddate-stampingtothealreadycomplicatedmeasurementthatisre-
quiredtoexaminecertainty.Becausethisprocessislikelytobeprohibitivelycostlyand
difficultformanyevaluators,immediacyisnotincludedinthelistofperformanceindicators.
Someevidencesuggestsoutcomesmaybebetterforprogramsthatprovidea4:1ratio
ofrewardstosanctionsforparticipants(Gendreau,1996;Wodahletal.,2011).ManyDTC
participantshavelonghistoriesofreceivingpunishmentforwrongdoings,buttheyareof-
tenunaccustomedtoreceivingpositivereinforcementforbehavingappropriately.Offering
agreaterproportionofrewardsforproductivebehaviorsthansanctionsforinfractionsis
hypothesizedtoincreasetheeffectivenessofcorrectionalrehabilitationprograms.
Support for the 4:1 ratiomust be viewed as preliminary, however, because it was
80
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
derivedfrompost hoc(afterthefact)correlationsratherthancontrolledresearchstudies.
Bydesign,sanctionsareimposedforpoorperformanceandrewardsareprovidedforgood
performance. Therefore, a greaterproportionof rewardsover sanctionsmightnothave
caused better outcomes, but rather better outcomesmight have elicited a greater
proportionof rewards. For this reason, thebalanceof reinforcement (i.e., the ratioof
rewardstosanctions)islistedasadiscretionaryperformanceindicatorforevaluatorsto
considerexaminingintheirprograms.
Balance of reinforcement =# of rewards administered
# of sanctions imposed
Restorative Justice Interventions
ManyDTCsincluderestorativejusticeinterventionsintheirprograms.Participantsmay
berequired,forexample,topayrestitutiontovictims,participateinvictimimpactpanels,or
performcommunityservicesuchasworkinginasoupkitchenorhomelessshelter.Panama,
forexample,requirescompensationorrestitutionforcrimevictims.
Themostcommonlyreportedperformance indicatorsforrestorativejustice interven-
tions include the number of hours of community service thatwere performed, and the
degreetowhichparticipantssatisfiedtheirfinancialobligations(fromnopaymenttopartial
paymenttofullpayment).Thefinancialobligationsaretypicallycategorizedaccordingto
paymentoffinesandfeestothecourt,paymentofrestitutiontovictims,andpaymentof
otherpersonalfinancialobligationssuchaschildsupportoralimony.
Although restorative justice interventions have been shown to improve outcomes in
othercriminaljusticecontexts(Latimeretal.,2005;Sullivan&Tifft,2008),nostudyhas
examinedtheirimpactinDTCsandnoperformancebenchmarkshavebeenidentifiedfor
theirusewithindividualschargedwithdrug-relatedcrimes.EvaluatorsfromOASmember
81
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
statesareinvitedtoexaminetheroleofrestorativejusticeinterventionsintheirDTCpro-
gramsandsharetheinformationtheylearnwiththeglobalresearchcommunity.
Employment
Outcomes in DTC programs are significantly better for participants who were em-
ployedatentryorwhoobtainedemploymentduringtheirenrollmentintheprogram
(Deschenesetal.,2009;Leukefeldetal.,2007).BetteroutcomesarealsoproducedbyDTCs
thatofferemploymentcounselingandrequireparticipantstobeemployedorenrolledin
aneducationalprogramasaconditionofgraduationfromtheprogram(Careyetal.,2012).
AcommoncomplaintamongDTCparticipants,especiallyracialandethnicminoritypartici-
pants,isthattheprogramspaidinsufficientattentiontotheiremploymentneeds(Cresswell
etal.,2001;Gallagher,2012).
EmploymentoutcomesarecommonlyreportedinoneoftwowaysinDTCevaluations.
Thedischarge employment rate representstheproportionofparticipantswhowereem-
ployedatdischarge,dividedbythenumberwhoenteredtheprogramminustheneutral
discharges.Beingemployed(orenrolledinaneducationalprogram)atdischargepredicts
significantlybetterpost-programoutcomes.(Beingemployedatentryalsopredictsbetter
post-programoutcomes;however,thisvariableisaparticipantcharacteristicorpredictor
variableratherthanaperformanceindicatorbecauseit isnotinfluencedbyparticipants’
enrollmentintheDTC.)
Discharge employment rate =# employed at discharge
# entered program-# of neutral discharges
Althoughemploymentatdischargeissignificantlypredictiveofoutcomes,itdoesnot
indicatewhether theDTCwas responsible for improvingparticipants’employment.High
employmentratesatdischargecouldsimply indicatethattheDTCtargeted low-risk
82
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
individualswhowerealreadymorelikelytohaveajob.Thiswouldsayverylittleabouthow
theDTCperformed.Forthisreason,thedischargeemploymentrateislistedasadiscre-
tionaryperformanceindicator.
Theemployment improvement rateiscalculatedbysubtractingthenumberofpartici-
pantswhowerealreadyemployedatentryfromthosewhowereemployedatdischarge,
andthendividingbythenumberwhoenteredtheprogramminustheneutraldischarges.
Ifmoreparticipantsareemployedatdischargethanatentry,thevalueofthisratiowillbe
positiveandwillrepresentthepercentageofindividualswhobecameemployedwhilethey
wereenrolled in theDTCprogram.Thisperformance indicator is listedasrecommended
because itsignificantlypredictsoutcomes,but itoftenrequirestheuseofself-report
assessmentsofparticipants’employmentstatus.
Employment improvement rate =# employed at discharge - # employed at entry
# entered program - # of neutral discharges
Nonetheless,neitheroftheaboveperformanceindicatorsmeasuresthequality of
participants’employment. Forexample,theydonot indicatewhethersomeparticipants
mayhavetransitionedfrompart-timetofull-timework,fromseasonaltostableemployment,
and/orreceivedaraiseorpromotionattheirjob.
Evaluatorstypicallyassessthenatureorqualityofparticipants’employmentusing
self-reportassessmenttools.Examplesofcommonlyusedtoolsforthispurposeinclude,
butarenotlimitedto,theAddictionSeverityIndex(ASI)andtheGlobalAppraisalofIndi-
vidualNeeds(GAIN).TheAppendixprovideslinkstowebsiteswherethereadercanobtain
furtherinformationabouttheseandotherassessmenttools.Itemsinthesetoolsinquire
aboutsuchissuesasthenumberofdaystheparticipantwasemployedduringthepreceding
month,thetypeofworkthatwasperformed(e.g.,clerical,manuallabor,orsupervisory),
andwhethertheemploymentwasfull-time,part-time,temporary,orseasonal.Although
83
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
thesetoolsweredevelopedintheUnitedStates,theyhavebeentranslatedintoSpanish
andotherlanguagesandhavebeenvalidatedinseveralcountries,includingSouthAmeri-
can countries.
Education
Many DTCs require or encourage participants to earn a high school diploma or
equivalencydegreeiftheydonothaveonealready.Becauseitusuallytakesseveralmonths
oryearstoaccomplishthisgoal,mostparticipantswillhavecompletedtheDTCprogram
beforetheycouldhaveearnedaneducationalorequivalencydegree.Forthisreason,itis
usuallynotinformativetoreportthepercentageofparticipantswhoearnedadegreebythe
timeofdischarge.
Theperformanceindicatorfortheeducational improvement rate ismeasuredeitheras
achangeinparticipants’educationalstatusorenrollmentinaneducationalprogram.The
differencebetweenthenumberofparticipantswhometeitherofthesecriteriaatdischarge
andthenumberwhometeithercriterionatentryisdividedbythenumberwhoenteredthe
programminustheneutraldischarges.Ifmoreparticipantshaveaneducationaldegreeor
areenrolledinaneducationalprogramatdischargethanatentry,thisratiowillbepositive
andwillrepresentthepercentageofindividualswhoimprovedorareintheprocessof
improvingtheireducation.
Educational improvement rate =(# with diploma or equivalent at discharge + # enrolled in educational program at discharge) - (# with diploma or equivalent at entry + # enrolled in educational program at entry)
# entered program – # of neutral discharges
Studiesareinconclusiveconcerningtheinfluenceofeducationalimprovementsonout-
comesinDTCprograms.Becausesupportfortheeffectsofeducationalefforts isunclear
atpresent,educationalimprovementislistedasadiscretionaryperformanceindicatorfor
evaluatorstoconsider.
84
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Housing
SignificantlybetteroutcomesareachievedbyDTCsthatrequireparticipantstobeliving
insafeanddrug-freehousingasaconditionofgraduationfromtheprogram(Careyetal.,
2012). Outcomesarealsosubstantiallybetter forDTCsthatprovidetransitionalhousing
whenitisneeded(McKee,2010).Withoutsafeanddrug-freehousing,itishighlyunlike-
lythatparticipantswillmaintainsobrietyordesistfromcriminalactivityafterleavingthe
program.Becausedrug-freehousingisstronglyandconsistentlyassociatedwithsuccessful
outcomes,improvementsinhousingarelistedasacoreperformanceindicator.
Theperformanceindicatorsforhousingarecalculatedinasimilarfashiontothosefor
employmentandeducation.Thedischarge housing raterepresentsthenumberofpar-
ticipantswhowere living insafeandsoberhousingatdischarge,dividedbythenumber
whoenteredtheprogramminustheneutraldischarges.
Discharge housing rate =# in stable housing at discharge
# entered program-# of neutral discharges
Thehousing improvement raterepresentsthenumberofparticipantswhowereliving
insafeanddrug-freehousingatdischargeminusthosewhowerealreadylivinginsafeand
soberhousingatentry,dividedbythenumberwhoenteredtheprogramminustheneutral
discharges.Thisperformanceindicatorreflectschangesinhousingthatoccurredwhilepar-
ticipantswereenrolledintheDTCprogram.Ifmoreparticipantsarelivinginstablehousing
atdischargethanatentry,thevalueofthisratiowillbepositiveandwillrepresenttheper-
centageofparticipantswhosehousingimprovedwhiletheywereenrolledintheDTC.
Housing improvement rate =# in stable housing at discharge - # in stable housing at entry
# entered program - # of neutral discharges
85
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Theseperformanceindicatorsdonotmeasurethequalityofthehousingarrangements.
Evaluatorsareencouraged tocategorize thequalityofhousing in termsof, forexample,
whether theparticipantsownor rent their residence,are livingwith familymembersor
friends,orarelivinginarecoveryhouseorresidentialtreatmentfacility.Instrumentssuch
astheASIandGAIN(listedintheAppendix)evaluatethequalityofparticipants’housingin
thecontextofsubstanceabusetreatment.
Emotional Health
Outcomesare significantlybetter forDTCs thatofferpsychiatricormentalhealth
treatmentforparticipantswhoneedtheseservices(Careyetal.,2012).Continuedemo-
tional problems at discharge predict significantly poorer post-programoutcomes in DTC
evaluations(Mendozaetal.,2013;Petersetal.,2012).
Inmoststudies,evaluatorshaveusedself-reportquestionnairesorstructuredinterviews
todeterminewhichparticipantsneededmentalhealthservicesandwhethertheirsymp-
tomsimprovedduringtheirenrollmentintheDTC.Manyoftheseinstrumentsassess
problems inmultiple areas of participants’ lives, includingmedical, psychiatric, family,
andsocialproblems.Therefore,thesametoolscanoftenbeusedtodevelopperformance
indicatorsforseveralofthedomainsdiscussedhereandbelow.TheAppendixlistsseveral
clinicalassessmentinstrumentsthathavebeenusedtoassessemotionalproblemsandother
psychosocialproblemsinDTCevaluationsandhavebeentranslatedintoSpanishorvali-
datedinmultiplecountries.
Discharge emotional problems are calculated by dividing the number of participants
whowereexperiencing significantemotionalproblemsatdischargeby thenumberwho
enteredtheprogramminustheneutraldischarges.Thisratiorepresentstheproportionof
participantswhodidnotresolvetheiremotionalproblemspriortodischarge,andtherefore
maybeatincreasedriskforrelapsetosubstanceuseorcriminalrecidivism.
86
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Discharge emotional problems =# with emotional problems at discharge
# entered program - # of neutral discharges
Theemotional improvement rateindicatesthedegreetowhichtheDTCwasresponsible
forreducingparticipants’emotionalproblems.Itiscalculatedbydividing(A)thenumberof
participants who were experiencing significant emotional problems at entryminus those
experiencingsignificantemotionalproblemsatdischarge,by (B) thenumberwhoentered
theprogramminustheneutraldischarges.Iffewerparticipantsareexperiencingemotional
problemsatdischargethanatentry,thevalueoftheratiowillbepositiveandwillrepresent
thepercentageofparticipantswhoseemotionalproblemsresolvedwhiletheywereintheDTC
program.Thisperformanceindicatorislistedasrecommendedbecauseitsignificantlypredicts
outcomesbutoftenrequirestheuseofrepeatedself-reportassessmentsofparticipants.
Emotional improvement rate =# with emotional problems at entry - # with emotional problems at discharge
# entered program - # of neutral discharges
AsubstantialproportionofDTCparticipantsmay leavetheprogramprematurelyand
without warning to staff members. This can complicate efforts to assess participants’
emotionalstatus(ortheirstatusonotherpsychosocialvariables)atthepointofdischarge.
CounselorsinDTCprogramsarethereforeencouragedtoperiodicallyreassessparticipants’
emotional health and status on other psychosocial variables. Many assessment tools,
includingtheASI,GAINandotherslistedintheAppendix,canbeusedtoreassessparticipants’
functioningat intervalsofevery three tosixmonths.Performingperiodic reassessments
helpstoensurethatinformationaboutparticipants’emotionalfunctioningwillbeavailable
atornearthetimeofdischargeandisalsoacriticalcomponentofgoodclinicalpractice.
Effectivecliniciansperiodicallyreevaluatetheirclientstodeterminewhethertreatmentis
workingorwhetheradjustmentsmaybeneededtothetreatmentplan.
87
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Medical and Dental Health
Preliminary evidence suggests outcomesmay be better for DTCs that offer medical
ordentaltreatmentforparticipantswhoneedtheseservices(Careyetal.,2012).Aswith
emotionalproblems,inmoststudiesevaluatorshavereliedonself-reportquestionnairesor
interviewstodetermineparticipants’needformedicalordentalservices.Nostudieshave
examinedthedegreetowhichcontinuedmedicalordentalproblemsatdischargepredict
long-term outcomes. Therefore, performance indicators related to medical and dental
servicesarelistedasdiscretionaryvariablesforevaluatorstoconsider.
Theperformance indicator fordischarge medical or dental problems is calculatedby
dividing thenumberofparticipantswhowereexperiencingsignificantmedicalordental
problems at discharge by the number who entered the program minus the neutral
discharges.Thisratiorepresentstheproportionofparticipantswhodidnotresolvetheir
medicalordentalproblemspriortodischarge,andthereforemaybeatriskforrelapseto
substanceabuseorcriminalrecidivisminthefuture.
Discharge medical or dental problems =# with medical/dental problems at discharge
# entered program-# of neutral discharges
Theperformanceindicatorformedical or dental improvementiscalculatedbydividing
thenumberofparticipantswhowereexperiencingmedical ordental problemsat entry
minus thoseexperiencingmedical ordental problemsatdischarge,by thenumberwho
enteredtheprogramminustheneutraldischarges.Iffewerparticipantsareexperiencing
medicalordentalproblemsatdischargethanatentry,thevalueofthisratiowillbepositive
andwillrepresentthepercentageofparticipantswhosemedicalordentalproblemswere
resolvedwhiletheywereenrolledintheDTCprogram.
Medical/dental improvement =# with medical/dental problems at entry - # with medical/dental problems at discharge
# entered program - # of neutral discharges
88
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Family Relationships
Outcomesaresignificantlybetter inDTCsthatofferfamilycounselingforparticipants
whoneedtheseservices (Careyetal.,2012). Anational studyof twenty-threeDTCs in
theUnitedStatesfoundthatreducingfamilyconflictsandimprovingfamilysupportwere
commonlyderivedbenefitsofDTCprograms (Rossmanet al., 2011). Evaluators typically
administerself-reportquestionnairesorinterviewssuchastheASIorGAINtodetermine
whetherparticipantsneedfamily-basedservicesandtogaugewhetherfamilyfunctioning
hasimproved.
Discharge family problemsarecalculatedbydividingthenumberofparticipantswho
wereexperiencingsignificantfamilyproblemsatdischargebythenumberwhoenteredthe
programminustheneutraldischarges.Thisratiorepresentstheproportionofparticipants
whodidnot resolve their familyproblemsprior todischarge,and thereforemaybeat
increasedriskforrelapsetosubstanceabuseorcriminalrecidivism.
Discharge family problems =# with family problems at discharge
# entered program - # of neutral discharges
Thefamily improvement rateiscalculatedbydividingthenumberofparticipantswho
wereexperiencingfamilyproblemsatentryminusthoseexperiencingfamilyproblemsat
discharge,bythenumberwhoenteredtheprogramminustheneutraldischarges.Iffewer
participantsareexperiencingfamilyproblemsatdischargethanatentry,thevalueofthe
ratiowill be positive andwill represent the percentage of participantswhose family
problems resolvedwhile theywere in theDTCprogram. This performance indicator is
listedasrecommendedbecauseitsignificantlypredictsoutcomesbutrequirestheuseof
self-report assessments.
Family improvement rate =# with family problems at entry - # with family problems at discharge
# entered program - # of neutral discharges
89
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Aswas notedpreviously, a substantial proportionofDTCparticipantsmay leave the
programprematurelyandwithoutwarning.Therefore,counselorsareencouragedtope-
riodicallyreassessparticipants’familyfunctioningatintervalsofapproximatelyeverythree
to sixmonths. Performing periodic reassessmentswill ensure that information about
participants’familyfunctioningisavailableatorneardischargeandisalsoafeatureofgood
clinicalpractice.
Social Relationships
Socialproblemsrefertoseriousorrepeatedinterpersonalconflictswithpersonsother
than familymembers,aswell as social isolationoralienation fromotherpeople. Many
DTCprogramsprovideinterventionsdesignedtoreduceinterpersonalconflictsandincrease
participants’involvementinhealthysocialinteractions.However,nostudieshaveexamined
theimpactofcontinuedsocialorinterpersonalproblemsonlong-termoutcomesfromDTC
programs.Therefore,performanceindicatorsrelatedtosocialproblemsareincludedasdis-
cretionaryvariablesforevaluatorstoconsider.
Theperformance indicator fordischarge social problems iscalculatedbydividingthe
numberofparticipantswhowereexperiencingsignificantsocialproblemsatdischarge,bythe
numberwhoenteredtheprogramminustheneutraldischarges.Thisratiorepresentsthepro-
portionofparticipantswhodidnotresolvetheirsocialproblemspriortodischarge,andthere-
foremaybeatincreasedriskforrelapsetosubstanceabuseorcriminalrecidivisminthefuture.
Discharge social problems =# with social problems at discharge
# entered program - # of neutral discharges
Theperformanceindicatorforthesocial improvement rateisthenumberofparticipants
whowereexperiencingsocialproblemsatentryminusthoseexperiencingsocialproblems
atdischarge,dividedbythenumberwhoenteredtheprogramminustheneutraldischarges.
90
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Iffewerparticipantsareexperiencingsocialproblemsatdischargethanatentry,theratio
willbepositiveandwill represent thepercentageofparticipantswhosesocialproblems
resolvedwhiletheywereintheprogram.
Social improvement rate =# with social problems at entry - # with social problems at discharge
# entered program - # of neutral discharges
Birth of Drug-Free Babies
Consumption of alcohol or other drugs during pregnancy, especially cocaine
or stimulants, is associated with serious birth complications and physiological and
developmentaldeficitsforthenewborn(Cooper,2004).Theaddedcoststosocietyofcaring
foradrug-exposedbabycanbeveryhigh.
BecausefewDTCparticipantsortheirmatesdeliverbabiesduringtheirenrollmentin
theprogram, it is difficult tomeasure statistically significant effectson this variable.
Nevertheless, the sentiments of policymakers, members of the public, and other
stakeholdersareunderstandablyswayedbythisoutcomemeasure. Savingoneinnocent
newbornfromalifetimeofmiserymightbeworththetotalcostsofaDTCtomanypeople.
Because births are infrequent events in DTC programs, it is usually not informative to
calculatedischargeratesorimprovementrates.Mostevaluationssimplyreportthenumber
ofdrug-freenewbornsthatweredeliveredtoorfatheredbyDTCparticipants.ForDTCsthat
havebeeninexistenceformanyyears,theresultsmaybedozensofinnocentlivessaved.
Treatment intake efficiency = Date of first treatment session-Date of entry into DTC
91
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Theperformanceindicatorsdiscussedthusfarhavefocusedonobjectiveeventsorbe-
haviors,suchasdrugtestresultsorschedulesofcourthearings.Ithasalsobeenhypothe-
sizedthatDTCsmayimproveoutcomesbyalteringthesubjectiveattitudesofparticipants
orstaffmembers.Forexample,DTCsmightenhanceparticipants’intrinsicmotivationfor
changeortheirrespectforthejusticesystem.Inturn,thesepositiveattitudinalchanges
mightinfluencethemtoreducetheirdruguseorcriminalactivity.Similarly,staffmembers
inDTCsmightdevelopmoreoptimistic attitudes towards rehabilitation, and this in turn
mightinfluencethemtointeractmoreproductivelywithparticipants.
Aswillbediscussedlater,testingsuchhypothesesrequiresevaluatorstoperformwhat
are calledmediation analyses. Changes in participants’ or staff members’ attitudes are
sometimes referred toasmediator variablesbecause theyoccur in themiddlebetween
thedeliveryofDTCservicesandsubsequentchangesinparticipants’behaviorsandare
hypothesizedtobeanecessaryconditionforthosebehavioralchangestooccur.Unlike
moderator analyses, which as discussed earlier indicatewho DTCs benefit, mediator
analysesareintendedtoindicatehowtheyfunction.
Participant Attitudes
Researchinthisareaisrelativelynew;however,afewstudieshavefoundthatpar-
ticipants’scoresonattitudinalmeasuresdidappeartoinfluencelonger-termimprovements
insobrietyordesistence fromcrime.Onestudy, forexample, foundthatalthoughmany
participants initially entered aDTC to avoid going to prison, thosewho gradually devel-
opedintrinsicmotivationtoimprovetheirliveshadsignificantlybetterlong-termoutcomes
(Kirk,2012).Inotherstudies,significantlybetteroutcomeswereachievedbyparticipants
whoperceivedtheDTCasapplyingfairproceduresandhavingbenevolentaims,character-
ATTITUDINAL VARIABLES
92
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
isticsreferredtoasprocedural justice or procedural fairness (Dane,2012;Frazer,2006;
Gottfredsonet al., 2007; Zweiget al., 2012). Betteroutcomeshavealsobeenachieved
byparticipantswhoperceivedarationalconnectionbetweentheirownbehaviorsandthe
impositionofsanctionsandincentivesintheDTC,acharacteristicreferredtoasperceived
deterrence(Marloweetal.,2005).
Thefollowingattitudinalvariablesonthepartofparticipantshavebeenhypothesizedto
mediateoutcomesinDTCsorhavebeenexaminedinDTCprogramevaluations.Itisprema-
turetoconcludewhetherthesevariablesreliablyimpactlong-termoutcomesinDTCs,but
evaluatorsinOASmemberstatesmaywishtoexaminethesevariablesintheirownresearch.
TheAppendixprovidesinformationonhowtoobtainassessmenttoolsthatmeasurethese
variablesandthathavebeentranslatedintoSpanishorvalidatedinmultiplecountries.
Procedural Justice or Procedural Fairness: Asubstantialbodyofresearchhasfound
thatcriminaldefendantsandotherlitigantsweremorelikelytoreactfavorablytoanad-
versejudgmentorpunitivesanctioniftheybelievedfairprocedureswerefollowedinreach-
ingthedecision.Thebestoutcomeswereachievedwhenthedefendantswere(1)given
areasonableopportunitytoexplaintheirsideofthedispute;(2)treatedinanequivalent
mannertosimilarpeopleinsimilarcircumstances;and(3)treatedwithrespectanddignity
throughouttheprocess(Burke&Leben,2007;Tyler,2007).SomecommentatorsinAustra-
lia(King,2009),Scotland(McIvor,2009)andtheUnitedStates(Burke,2012;Wieneretal.,
2010)havehypothesizedthatenhancingparticipants’perceptionsofproceduraljusticemay
betheprimarymechanismbywhichDTCsreducecrimeandsubstanceabuse.
Motivation for Change:DTCsweredesignedtotreatpersonssufferingfromaddiction
whohavelowmotivationtochangetheirsubstanceuseorcriminalactivity.Thepowerof
thecourtisusedtomaintainparticipants’adherencetotreatmentdespiteapotentialdesire
toleavetreatmentprematurely.Eventually,however,participantsmustgraduatefromthe
DTC. If theyhavenotdeveloped intrinsicmotivationbythattime,theoddsof long-term
successarelikelytobeslimtonone.Therefore,evaluatorsshouldperiodicallyassesspar-
93
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
ticipants’motivationforchangeatapproximatelythree-tosix-monthintervalstodetermine
whetheritisimprovingovertime.Aswasnotedabove,atleastonestudyreportedbetter
outcomesforDTCparticipantswhoseinternalmotivationimprovedovertime(Kirk,2012).
Perceived deterrence:Thecriminaljusticesystemusuallydoesnotapplybehavioral
contingenciessuchassanctionsandincentiveseffectively.Punishmentmaybeadministered
afterlongdelaysormultipleinfractions,andmaybetoosevereortoolenientinmagnitude
toachieveeffectiveresults.OnegoalofaDTCistoadministerrewardsandsanctionsina
systematic, rapid,andeven-temperedmannerthat isconsistentwithscientificprinciples
ofeffectivebehaviormodification.StudiessuggestparticipantsinDTCsmayrecognizethe
rationalitybywhichrewardsandsanctionsareadministeredandmayrespondfavorablyas
aresult(Goldkampetal.,2002;Harrell&Roman,2001;Marloweetal.,2005).
Satisfaction with the Program or Treatment:Participants’satisfactionwithsubstance
usetreatment, includingtheculturalsensitivityoftreatmentstaff, isareliablepredictorof
outcomes incommunity-basedtreatmentprograms inseveralcountries (Richardsonetal.,
2011). Studieshavealmostuniformlyreportedhighlevelsofsatisfactionwiththeservices
provided inDTCs. For statistical reasons,variables tend tobe lesspredictiveofoutcomes
whenmostparticipantsscorethesameornearlythesameontheassessmenttools.Perhaps
forthisreason,satisfactionwiththeprogramhasfrequentlynotbeenfoundtobepredictive
ofoutcomes inDTCevaluations. Ifparticipants inotherOASmemberstateshaveawider
rangeofreactionstoDTCs,thisvariablemighthavebetterutilityforpredictingoutcomes.
Therapeutic Alliance:Thequalityofthetherapist-patientalliancehasbeenfoundto
beareliablepredictorofpsychotherapyoutcomesinseveralcountries includingCanada,
Italy,AustraliaandtheUnitedStates(Ardito&Rabellino,2011;Summers&Barber,2003).
NostudyhasexaminedtheimpactofthetherapeuticallianceonoutcomesinDTCs,butthis
variable(particularlythealliancebetweentheparticipantandjudge)hasbeenhypothesized
toinfluenceoutcomesinDTCprogramsaswell.Researchisneededtotestthishypothesis
inarangeofDTCprogramsandindifferentcountries.
94
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Staff Attitudes
Considerably less researchhasexamined staffmembers’ attitudesas compared to
participants’attitudesinDTCprograms.Evidencesuggeststhemoretrainingstaffmembers
receiveandthelongertheyworkinDTCprograms,themorelikelytheyaretoendorse
optimistic attitudes aboutoffender rehabilitationand to favor rehabilitative goals over
punitivegoals(VanWormer,2010).Changesinstaffmemberattitudeshavenot,however,
beencorrelatedwithoutcomesinDTCprograms.Additionalresearchisneededtoun-
derstandthedegreetowhichstaffmemberattitudesmayimpactoutcomesinDTC’sand
howtoinfluencethoseattitudestobringaboutmoreeffectivebehavioralchange.
95
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Observer rating scalesmayalso contributevaluable information toDTCprogram
evaluations.ObserverratingscalesmaybeusedtoassessobjectiveeventsoccurringinDTCs,
suchasthenumberoftimesthejudgespokedirectlytoparticipantsduringcourtsessions.
Morecommonly,however,theyareusedtoassesssubjectivetraitsorcharacteristicsofstaff
membersorparticipants,suchasthedegreetowhichthejudgeappearedtobe“competent”
or“knowledgeable,”orparticipantsappearedtobe“attentive”or“compliant.”
Inafewstudies,independentratersobservedcourthearingsinDTCsandratedthebe-
haviorsofthejudgeorparticipants.Inonestudyoftwenty-threeDTCs,significantlygreater
reductionsincrimeandsubstanceusewereproducedbyjudgeswhowereratedasbeing
morerespectful,fair,attentive,enthusiastic,consistent,andcaringintheirinteractionswith
participants(Zweigetal.,2012).AnotherstudyfoundthatparticipantsinaDTCweresignifi-
cantlylesslikelytobearrestedforanewcrimeiftheywereratedasbeingmoreprepared
andbehavingappropriatelyduringcourthearings(Reingleetal.,2012).Stillotherstudies
havefoundthatparticipantsandstaffmemberstendedtoratecourtinteractionssimilarly
(Farole&Cissner,2007;Satel,1998).Forexample,theycommonlyagreedonwhetherin-
teractionsappearedtobeproductiveorunproductive.TheAppendixprovidesinformation
aboutobserverratingscalesthathavebeenusedsuccessfullyinDTCprogramevaluations.
Observer rating scales typically require substantially more effort on the part of
evaluatorsthanself-reportquestionnaires.Ratersmustbetrainedcarefullyonhowtoscore
the instrumentsandmustdemonstratehigh levelsof inter-rater reliabilityoragreement
with other raters. If two raters reach different conclusions about the same behavior or
characteristicsofindividualstheyareobserving,thentheinformationwillnotbeusefultoan
evaluator.Itcanalsobeverycostlytopayindependentobserverstoattendcourthearings
orothersessionsandscoreratinginstruments.Asaresult,observerratingscalesaremost
likely to be used inwell-funded studies that are conductedby professional scientists or
university-basedresearchers.
OBSERVER RATING SCALES
96
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Performanceindicatorscannotbeinterpretedinavacuum.Toanswermostresearch
questions, the performance of a DTCmust be compared to that of an equivalent and
unbiasedcomparisongroup.ComparingwhathappenedintheDTCtowhatwouldmost
likelyhavehappenediftheDTCdidnotexistisreferredtoasexaminingthecounterfactual
hypothesis orthepossibilitythattheDTCwasineffective.
Somecomparisonconditionsarereasonablyunbiasedandcanyieldafairandaccurate
assessmentofwhatwouldmostlikelyhaveoccurredwithouttheDTC.Others,however,may
besystematicallybiasedorskewedinsuchamannerastomaketheDTClookbetterorworse
thanitdeserves.Thismayleadtoanunwarrantedrejectionofthecounterfactualhypothesis
andtheerroneousconclusionthattheDTCwaseffectivewhen,infact,itwasnot.
Random Assignment
Fromascientificstandpoint,thebeststrategyistorandomlyassigneligibleindividuals
either to the DTC program or to an alternative disposition, such as probation. Ran-
domassignmentprovidesthegreatestassurancethatthegroupsstartedoutwithanequal
chanceofsuccess.Therefore,ifoutcomesarebetterfortheDTC,thesecanbeconfidently
attributedtotheeffectsoftheDTCprogramasopposedtodifferences intheseverityof
participants’problemsbeforetheyenteredtheprograms.
Evenwhenanevaluatoremploysrandomassignment,thereisstillapossibility(although
agreatlydiminishedpossibility)thatthegroupsdifferedonimportantdimensionsfromthe
outset.Thisrequirestheevaluatortoperformaconfirmationoftherandomizationproce-
dure.Aswillbediscussedlater,theevaluatorwillneedtocheckforpossiblepre-existing
differencesbetweenthegroupsthatcouldhaveaffectedtheresults.Ifsuchdifferencesare
foundtoexist, therearestatisticalprocedurestheevaluatormightbeable toemployto
COMPARISON GROUPS
97
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
adjust for thosedifferencesandobtainscientificallydefensibleresults.Forexample,DTC
participantsmighthavehad less severedrugproblems tobeginwith thancomparison
participants.Ifso,thenbetteroutcomesfortheDTCmightsimplyhavebeentheresultof
treatinghigherfunctioningindividuals. Ifrandomassignmentfailstoavoidthisproblem,
theevaluatorwillneedtoadjuststatisticallyfordifferencesindrugproblems.
Inmany instances, randomassignment isnot feasible for legal,ethical,orpractical
reasons.Somestaffmembersmayhaveethicalconcernsaboutdenyingservicestoother-
wiseeligible individuals. Inaddition,someDTCsmayhavedifficultyfillingtheirslotsand
maynotwanttoturnawayeligibleindividuals.
Nonetheless,randomassignmentisgenerallynotconsideredtobeunethicalifappro-
priatesafeguardsareinstituted(NationalResearchCouncil,2001).Randomassignmenthas
beenusedsuccessfullyinseveralresearchstudiesofDTCsinAustralia(Jonesetal.,2011)
and theUnitedStates (Breckenridgeetal., 2000;Gottfredsonetal., 2003;Harrell etal.,
1999;MacDonaldetal.,2007;Turneretal.,1999).Safeguardsmayincluderequiringpar-
ticipantstoprovideinformedconsenttoberandomlyassigned,orhavinganindependent
ethicsreviewboardoverseethesafetyandfairnessofthestudy.Insomecountries,local
collegesoruniversitieshaveethicsreviewboardswithnamessuchasInstitutionalReview
Board (IRB)orDataandSafetyMonitoringBoard (DSMB),whichhave theauthorityand
expertisetoprovidesuchethicaloversightforresearchstudies.
Ethical scholars generally agree that random assignment is acceptable, and indeed
shouldberequired,ifatreatmentprogramhasnotyetbeenproveneffective(Edwardset
al.,1998;NationalResearchCouncil,2001).Inmanycountries,forexample,newmedica-
tionsmustbeevaluatedthroughrandomassignmentbeforebeingapprovedforpublicuse.
Itmaybeunethicalorunlawfulnot touserandomassignment infinal-phasemedication
trials.Randomassignmentbecomesobjectionableonlyafteratreatmenthasbeenproven
tobeeffective,andthenitmaynotbeethicaltodenythateffectivetreatmenttodeserving
individuals.Insomecountries,DTCshavebeenproventobeeffective,andthereforeitmay
98
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
raiseethicalconcernstouserandomassignmenttodecidewhogetsintothoseprograms.
IncountriesthathavenotyetestablishedtheeffectivenessoftheirDTCsfortheircitizens,it
isthereforenotnecessarilyunethicaltouserandomassignment
Additionally,randomassignmentposesfarfewerethicalchallengesifaDTChasin-
sufficientcapacitytotreatmanyindividualswhowouldotherwisebeeligibleforitsservices.
Ifmanydeservingpeoplemustbeturnedawayanyway,thenitwouldperhapsbefairestto
selectparticipantsrandomlyratherthanallowstaffmemberstopickandchoosewhogets
intotheprogram.Undersuchcircumstances,randomassignmentmightprovidethebest
protectionagainstunfairdiscriminationorunconsciousbias.
RandomassignmentmayalsobelessobjectionableifitisconductedwithinaDTCafter
participants have alreadybeen accepted. For example, studies have randomly assigned
participantswithinDTCstoreceivedifferentschedulesofcourthearings(Festingeretal.,
2002;Marloweetal.,2006,2007),differentrewardsforaccomplishments(Marloweetal.,
2008a),orspecializedtreatmentservices(Messinaetal.,2012). Becauseallparticipants
werealreadyadmittedtotheDTCs,eachparticipantreceivedatleastsomeoftheadditional
servicesandopportunitiestheprogramshadtooffer.Anydifferencesintheservicesthey
receivedwereamatterofdegreeratherthanbeingallornothing.Thisarrangementmaybe
alsomoreethicallyacceptabletostaffmembersandotherstakeholdersofDTCprogramsif
participantsprovidevoluntaryandinformedconsenttobeingrandomlyassigned.
Despite these safeguards, itmightneverthelessbe impractical toengage in random
assignment.Ifrandomassignmentisnotpossible,thenseveralalternativestrategiesare
availabletoselectacceptablecomparisonconditions.
Quasi-Experimental Comparison GroupsThenextbestapproachafterrandomassignmentistouseaquasi-experimentalcom-
parisongroup.ThistermreferstoindividualswhowouldhavebeeneligiblefortheDTCbut
didnotentertheprogramfor reasons that are unlikely to have affected their outcomes. An
99
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
excellentexampleisindividualswhowereeligibleforandwillingtoentertheDTCbutwere
deniedaccesssimplybecausetherewerenoemptyslotsavailable.Thisisreferredtoasa
wait-list comparison group.Themerehappenstancethattheprogramwasfullisunlikelyto
haveledtothesystematicexclusionofindividualswhohadmoresevereproblemsorpoorer
prognosesthantheaverageparticipant,andthereforeisunlikelytohavebiasedtheresults.
Lessoptimal,butstillpotentiallyacceptable,quasi-experimentalcomparisongroups
includeindividualswhowouldhavebeeneligiblefortheDTCbutwerearrestedintheyear
orsobeforetheDTCwasestablished,orwhowerearrested inan immediatelyadjacent
county thatdoesnothaveaDTC. Because these individualswerearrestedatanearlier
pointintimeorinadifferentgeographicregionthantheDTCparticipants,theremightstill
besystematicdifferencesbetweenthegroupsthatcouldhavebiasedtheoutcomes.For
example,socioeconomicconditionsorthetypesofdrugsthatarecommonlyusedmight
differbetweenneighboring communities. However, the likelihood of this occurring is
usuallynotsubstantial,andthesemaybetheonlypracticalcomparisonconditionsthat
canbeusedforpurposesofmanyDTCprogramevaluations.
Whenusingaquasi-experimentalcomparisongroup,itisessentialfortheevaluatorto
checkforpre-existingdifferencesbetweenthegroupsthatcouldhaveaffectedtheresults.
Forexample,thecomparisonindividualsmighthavehadmoreseriouscriminalhistoriesto
beginwith.This,inturn,mighthaveputthematgreaterriskforcriminalrecidivism.Ifso,
thensuperioroutcomesfortheDTCgroupmightnotbeattributabletotheeffectsofthe
DTCprogram,butrathertothefactthatittreatedalessseverepopulation.Aswillbe
discussedlater,therearevariousstatisticalproceduresanevaluatorcanemploytoadjust
forsuchdifferencesandobtainscientificallydefensibleresults.
Matched Comparison Groups
Evaluatorsdonotalwayshaveaquasi-experimentalcomparisongroupattheirdisposal.
Undersuchcircumstances,theymayberequiredtoconstructacomparisongroupoutofa
100
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
largepopulationofindividualsinvolvedinthejusticesystem.Forexample,anevaluatormight
needtoselectcomparisonsubjectsfromastatewideprobationdatabase.Manyofthose
probationerswouldnothavebeeneligibleforaDTCoraredissimilartoDTCparticipants
on characteristics that are likely to have influenced their outcomes. For example, some
probationersmightnothavehaddrugproblems,ormighthavebeenchargedwithoffenses
thatwouldexcludethemfromparticipationinaDTC.Theevaluatormust,therefore,select
asubsetofindividualsfromtheprobationpoolthatarecomparabletotheDTCparticipants
oncharacteristicsthatareknowntoaffectoutcomes.Forexample,theevaluatormightpair
eachDTCparticipantwithaprobationerwhohasthesamecriminalhistory,demographic
characteristicsandsubstanceusediagnosis.Becausetheevaluatorwillchooseonlythose
probationers who are comparable to DTC participants on multiple characteristics, it is
necessary to startwith a large sample of potential candidates fromwhich to select the
comparison sample.
Thesuccessofanymatchingstrategywilldependonwhethertheevaluatorhasade-
quateinformationaboutthecomparisoncandidatestomakevalidmatches.Ifdataarenot
availableonsuchimportantvariablesastheprobationers’criminalhistoriesorsubstance
useproblems,itwillnotbepossibletoplaceconfidenceinthevalidityofthematches.Itis
notsufficienttosimplymatchthegroupsonvariablesthatareeasytomeasureandreadily
available,suchasgender,ethnicityorrace,becausethegroupsmighthavedifferedonother
importantdimensionsthatwerenotexamined.
Propensity Score Analyses
An evaluatormay also use a statistical procedure called a propensity score analysis
tomathematically adjust for differences between the groups. This advanced procedure
calculatestheprobabilitythatanindividualwithagivensetofcharacteristicswouldbein
theDTCgroupasopposedtothecomparisongroup—inotherwords,therelativesimilarity
101
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
oftheindividualtoonegroupasopposedtotheother.Theanalysisthenstatisticallyadjusts
for this relative similarity when comparing outcomes between the groups. Advanced
statisticalexpertiseisordinarilyrequiredtoimplementandinterpretthisprocedure.
Aswithanystatisticaladjustment,thesuccessofapropensityscoreanalysiswilldepend
onwhethertheevaluatorhasadequateinformationaboutthecomparisonsubjectstomake
validadjustments.Ifdataarenotavailableonsuchimportantvariablesasthecomparison
subjects’criminalhistoriesorsubstanceuseproblems,itwillnotbepossibletoplace
confidenceintheadjustments.Again,itisnotsufficienttomerelyadjustthescoresbased
oneasilymeasuredvariablessuchasgenderor race,becausethegroupsmighthave
differedonotherimportantcharacteristicsthatwereneverexamined.
Invalid Comparison Groups
ThereareseveralcomparisongroupsthathavebeenusedinDTCprogramevaluations
whicharelikelyyieldedbiasedresults.Inmostinstances,itisnotjustifiedtocompareout-
comesfromaDTCtothoseofpersonswhorefusedtoentertheDTC,weredeniedaccessto
theDTCduetotheirclinicalorcriminalhistories,droppedoutoftheDTC,orweredischarged
unsuccessfullyfromtheDTC.Theprobabilityisunacceptablyhighthatsuchindividualshad
relativelypoorerprognosesormoresevereproblemstobeginwith.Forexample,thereis
ahighlikelihoodthattheymayhavehadmoreseriouscriminalorsubstanceusehistories,
lowermotivationforchange,orlessersocialsupports.Asaresult,comparingthemtothe
DTCsampleisapttounfairlyfavortheDTCprogram.Giventhehighlikelihoodthatthese
groupswereseriouslydisadvantagedfromtheoutset,statisticaladjustmentscannotbe
reliedupontoovercomethebiaseddifferences.
102
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
AcommonerrorinDTCevaluationsistoexamineoutcomesonlyforparticipantswho
successfullygraduatedfromtheprogram.Thelogicforperformingsuchananalysisisunder-
standable.Evaluatorsareoftenmostinterestedinlearningaboutoutcomesofparticipants
whoreceivedallservicestheprogramhastooffer.Ifindividualswhodroppedoutorwere
terminatedprematurelyareincludedintheanalyses,thentheoutcomeswillbeinfluenced
bythosewhodidnotactuallyreceiveallintendedservices.
Althoughthisreasoningmightseemlogical,itisscientificallyflawed.Outcomesshould
beexaminedforallindividualswhoparticipatedintheDTC,regardlessofwhetherthey
successfullygraduatedorwereunsuccessfullyterminatedfromtheprogram.Thisisreferred
to as an intent-to-treat analysis becauseitexaminesoutcomesonallindividualswhomthe
programinitiallysetouttotreat.Itisnotappropriatetoreportoutcomesonlyforgraduates
becausethisunfairlyandfalselyinflatestheapparentsuccessoftheprogram.Individuals
whograduatedfromtheDTCarelikely,forexample,tohavehadlessseveredrugoralcohol
problems tobeginwith, less severe criminal propensities, highermotivation for change,
orbettersocialsupportsystemsthantheaverageDTCparticipant.Asaresult,theymight
havebeenlesslikelytocommitfutureoffensesorrelapsetosubstanceuse,regardlessof
theservicestheyreceivedintheDTC.Thecriticalquestionishowtheprogramfaredforall
participantswhomettheprogram’seligibilitycriteriaandwereacceptedintotheprogram.
Thisissueisparticularlyimportantifoutcomesarecontrastedagainstthoseofacompari-
songroup,suchasprobationers.SelectingthemostsuccessfulDTCcasesandcomparingtheir
outcomestoallprobationersisunfair.ItisakintoselectingonlytheA+studentsfromoneclass-
room,comparingtheirtestscorestoallstudentsinasecondclassroom,andthenconcluding
thatthefirstclasshadabetterteacher.Thiswouldclearlybeabiasedandunfaircomparison.
This isnot tosuggest thatoutcomes forgraduatesareofno interest.Programsmay,
indeed,wanttoknowwhathappenstoindividualswhoreceivedallservicesofferedbythe
INTENT-TO-TREAT ANALYSES
103
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
DTC.However,thisshouldbeasecondary analysisthatisconductedaftertheintent-to-treat
analysishasbeencompleted.If it isfirstdeterminedthattheprogramachievedpositive
outcomesonan intent-to-treatbasis, itmaythenbeappropriatetoproceedfurtherand
determinewhetheroutcomeswereevenbetterforthegraduates.However,iftheintent-
to-treat analysis is not significant, it is generally not acceptable tomoveon to evaluate
outcomesforgraduatesalone.
Importantly,ifsecondaryanalysesareperformedonDTCgraduates,thenthecomparison
samplemustalsobecomprisedofsuccessfulcompleters.Forexample,outcomesforDTC
graduatesshouldbecomparedagainstthoseofprobationerswhosatisfiedtheconditions
ofprobation.ComparingoutcomesforDTCgraduatestoallprobationers, includingthe
probationfailures,wouldunfairlyfavortheDTCprogram.
Cohorts
Itisusuallynotjustifiabletoevaluateaprogramatasinglepointintimeandassume
theresultsarerepresentativeofwhatcanbeexpectedfromtheprograminthefuture.Con-
ditionsoftenchangeovertimeandmaysignificantlyinfluencetheeffectivenessofaDTC
program.Forexample,aDTCmightexperiencesubstantialstaffturnoveroradecreasein
fundingwhichcouldseriouslydetractfromthequalityoftheservicesitcanoffer.Converse-
ly, staffmembersmay improve their skills overtime throughexperience and continuing
education,whichcouldsignificantlyimprovetheprogram’sperformance.
Itisimportantforevaluationstobeconductedperiodicallytomeasurechangesinper-
formanceovertime.Commonly,thisisaccomplishedbyevaluatingcohortsofDTCpartici-
pants.Aswasmentionedearlier,acohortisdefinedasagroupofindividualswhoentered
theprogramduringthesamespecifiedtimeperiod,typicallyanintervalof12months(Heck,
2006;Rubioetal.,2008).Forexample,participantswhoenteredtheDTCduringthecalen-
daryears2011,2012,and2013mightbedefinedasthreecohorts,andoutcomeanalyses
couldbeconductedseparatelyforeachcohort.Thiswouldprovideaseriesof“snapshots”
indicatinghowwelltheprogramperformedoverconsecutiveyears.
104
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
There isnoonecorrect follow-upwindowortimeperiod forevaluatingtheperfor-
manceofaDTC.Themostappropriate lengthof thefollow-upwindowwilldependon
whatperformanceindicatorisbeingassessed.Forexample,itmaybeinformativetomea-
sureparticipants’attendanceintreatmentduringthefirstseveralmonthsofaDTCpro-
gram.ItisimportanttoknowwhethertheDTCissuccessfulatretainingparticipantsin
treatmentandprovidinganadequatedosageoftreatment.However,mostDTCprograms
graduallydecreasethedosageoftreatmentasparticipantsmovethroughthephasesof
theprogram.Asaresult,treatmentattendancemaybecomearelativelylessimportant
performance indicatorafterseveralmonthsorayear. Itmaybecomemore important
afterseveralmonthstoknowwhetherparticipants’substanceusedeclinedortheirem-
ploymentrateincreased.
Itisusuallynotveryinformativetomeasurenewarrestratesornewconvictionrates
duringthefirstfewmonthsofaDTCprogram.Ittypicallytakesseveralmonthsforapar-
ticipanttore-engageincriminalbehavior,bedetectedbytheauthorities,andbeformally
chargedorconvicted ina legalproceeding. Asa result,newarrestandconviction rates
are likelytobe lowduringthefirstseveralmonthsof treatment. (Thisdoesnot include
technicalviolations,whichmaybemorecommon.) This issuecanbeproblematicifnew
arrestsorconvictionsarecontrastedagainstthoseofacomparisongroup,suchasproba-
tioners.Ifrecidivismislowinbothgroups,itwillbedifficulttodetectstatisticallysignificant
differencesbetweenthegroups.ThisistrueeveniftheDTCis,infact,asuperiorprogram.
Forexample,iftenpercentoftheprobationersreoffendedduringthefirstsixmonthsand
fivepercentoftheDTCparticipantsreoffended,formathematicalreasonsthisdifference
wouldprobablynotbestatisticallysignificantunlesstherewerenumerousparticipantsin
thestudy.However,itmightbequitemeaningfulfromaclinicalorpublicpolicyperspective.
FOLLOW-UP WINDOWS
105
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Reducingrecidivismbyfivepercentagepointscanprovideimportantpublicsafetyandcost
benefitsforlocalcommunities.
Inmanystudies,thebestcourseofactionmaybetowaitatleasttwelvemonths,and
preferably twenty-four months, from entry before reporting recidivism outcomes. This
shouldallowadequatetimetoelapseforreoffendingratestodivergesufficientlybetween
thegroupstodetectstatisticalsignificance.
Aswasmentionedearlier,ifitisfeasible,recidivismshouldbetrackedforatleastthree
yearspost-entryandideallyuptofiveyearspost-entry.Researchsuggestsmostnewoffens-
esfordrugandalcohol-involvedpersonsoccurwithinthreetofiveyearsaftertreatment
(Gossopetal.,2005;Inciardietal.,2004;Martinetal.,1999).Therefore,followingpartici-
pantsforthreetofiveyearsshouldensurethatmostrecidivismeventsareaccountedforin
theevaluationresults.Recidivismanalysesmaystillbeinformativeafterthefirstorsecond
year,andthoseinterimanalysesshouldcertainlybereportedasthedatabecomeavailable.
However,followingparticipantsforthreetofiveyearsislikelytoelicitstableestimatesof
criminalrecidivism.
Starting the Clock
Inmanyevaluations, the follow-upwindow isstarted fromthetimeparticipantsfirst
enteredtheDTCprogramorcomparisoncondition.Forexample,outcomesforeachpar-
ticipantmightbemeasuredoveraperiodoftwenty-fourmonthsstartingfromthedateof
entryintotheDTCorprobation.Onereasonforthispracticeisthatitgivesallparticipants
anequivalentfollow-upwindow.Eachparticipantwouldhavetwenty-fourmonthsinwhich
toengage inbehaviors thatarerelevanttoevaluationoutcomes,suchascommittinga
newoffenseorfindingajob.Althoughit isnotalwaysclearwhen“entry”hasoccurred,
mostDTCsandprobationprogramshaveaformalentryhearingorcomparableproceeding
atwhichtheparticipantentersapleaorisadjudicatedandisofficiallyenrolledinthepro-
gram.Thedateofthishearingorproceedingistypicallycountedastheentrydatebecause
106
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
itmarksthepointintimewhentheprogramsgainedauthorityoverparticipantstoimpose
conditionsoftreatmentandsupervision.
Consider,instead,whatwouldhappenifoutcomesweremeasuredfortwelvemonths
from the date of discharge from the program. Some participantsmight have been dis-
chargedaftertwelvemonthsoftreatmentwhereasothersmighthavebeendischargedaf-
tereighteenmonthsoftreatment.Asaresult,outcomesfortheformerparticipantswould
bemeasuredfortwenty-fourmonthsfromentryandoutcomesforthelatterparticipants
wouldbemeasuredfor thirtymonths fromentry.The latter individualswouldhavesix
additionalmonthsinwhichtocommitnewoffensesorengageinotheroutcomebehaviors.
Time at Risk
Regardlessofthelengthofthefollow-upwindow,itisimportantforthefollow-upwin-
dowtobecomparableforallparticipants.Thisensuresthatallparticipantshavethesame
time at risk,meaningtheyhavethesameopportunitytoengageinsubstanceuse,crime,
andotherbehaviorsof interesttotheevaluator. If, forexample,anevaluatormeasured
criminalrecidivismovertwelvemonthsfortheDTCparticipantsbutmeasuredrecidivism
overtwenty-fourmonthsforthecomparisongroup,thiswouldgiveanunfairadvantageto
theDTCparticipants.Thecomparisongroupwouldhavetwelveadditionalmonthsinwhich
tocommitnewcrimes.
Ifthetimeatriskdifferssignificantlybetweenthegroups,theevaluatormightbeable
tocompensateforthisproblembystatisticallyadjustingforthetimeatriskintheoutcome
analyses.Forexample,theevaluatormightenterthetimeatriskasacovariateinthesta-
tisticalanalyses.Acovariateisavariablethatisenteredfirstintoastatisticalanalysis,and
thentheindependenteffectofthevariableofinterest(inthiscase,treatmentinaDTC)is
evaluatedafterfirsttakingtheeffectofthecovariate intoconsideration. Thisprocedure
would indicate whether DTC participants had significantly better outcomes than proba-
tionersafterfirstaccountingfortheinfluenceoftheirtimeatrisk.Theuseofcovariatesis
107
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
notalwayssuccessful,however,andthebestcourseofactionistobeginwithequivalent
follow-upwindowswheneverpossible.
Time at Liberty
Theissueoftime at libertyiscomparabletotimeatriskinthatbothrelatetopartici-
pants’opportunitiestoreoffendorengageinotheroutcomebehaviors.Thedifferenceis
thattimeatlibertyreferstowhetherrestrictiveconditionswereplacedontheparticipant.
Themostobviousrestrictiveconditionsinvolvephysicalbarrierstofreedom,suchasincar-
cerationorplacementinaresidentialtreatmentfacility.Thesephysicalbarriersmaysevere-
lycurtailparticipants’abilitytousedrugs,commitnewoffenses,obtainajob,orengagein
otherbehaviorsofinteresttotheevaluator.
AcommonerrorinDTCevaluationsistoneglecttheissueoftimeatliberty(andtimeat
risk)whencontrastingoutcomestoacomparisongroup.Insomejurisdictions,forexample,
individualswhodonotenterDTCmaybemorelikelytoreceiveajailsentence.Iftheywere
jailed foraportionof the follow-upperiod, theymighthavehad feweropportunities to
reoffendorusedrugsthantheDTCparticipantswhoweretreatedinthecommunity.The
evaluatormightwrongfullyconcludethattheDTCcausedparticipantstoreoffendoruse
drugsmoreoftenthanthecomparisonindividuals,wheninfacttheysimplyhadmoretime
atlibertytodoso.
Undersuchcircumstances,theevaluatormightneedtostatisticallyadjustforthetime
atlibertyintheoutcomeanalyses.Forexample,theevaluatormightneedtoenterthetime
atlibertyasacovariateinthestatisticalmodels.ThiswouldrevealwhethertheDTCpartici-
pantshadsignificantlybetteroutcomesafterfirsttakingintoaccounttheinfluenceoftheir
timeatliberty.
Evaluatorsarenotalwaysadvised,however,toadjustfortimeatliberty.Incostanalyses,
forexample,thetimethatparticipantsspentinjailoraresidentialtreatmentfacilitymight
beanimportantoutcomemeasureinitsownright,andmightbevalueddifferentlythan
108
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
othervariablesfromafiscalstandpoint.Decidingonwhethertostatisticallyadjustforpar-
ticipants’timeatlibertyrequirescarefulconsiderationoftheaimsofeachanalysis.
Adjustingforparticipants’timeatriskandtimeatlibertyobviouslyrequiresevaluators
topaycarefulattentiontotheissueofdate-stamping.Itisessentialtorecordthedateson
whichparticipants entered andwere subsequently released from jail, prison, residential
treatmentfacilities,homedetention,andotherrestrictivesettings.Ifthesedatesarenot
recorded,thenitwillnotbepossibletocontrolforparticipants’timeatlibertyortimeatrisk
ascovariatesintheoutcomeanalyses,andtheresultsoftheevaluationcouldbeseriously
compromised.
Additionally,thisonceagainhighlightshowvaluableitisforDTCprogramstotakethe
timetosetupqualitydatacollection,entry,andmanagementsystemsattheiroutset.Proper,
consistentdatapractices in theseareaswillhelpprovide futureevaluatorswith reliable,
usefuldata—andinturnmakefutureevaluationsbothlessexpensiveandmorereliable.
109
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
This chapter discusses common analytical errors that have beenmade inmanyDTC
programevaluations,andwhichhavelaidwastetohundredsofhoursofworkandthousands
ofdollarsinevaluationfees.Guardingagainstthesecommonerrorscanavoidsubstantial
distressstemmingfromindefensibleevaluationresults.
Target Population
It is commonpractice for evaluators to correlateparticipant characteristics—such as
age,gender,race,nationality,ornumberofpriorconvictions—withoutcomesinDTCpro-
grams.Determiningwhichtypesofindividualsperformedbestintheprogrammayhelpto
identifytheoptimaltargetpopulationfortheDTCandmayindicatewhichparticipantswill
requireenhancedservicesinordertosucceed.Aswasdiscussedearlier,participantcharac-
teristicsthatcorrelatereliablywithoutcomesarecalledpredictorvariablesorriskfactors
becausetheypredictresultsorputparticipantsatriskforrelativelypooreroutcomes.
Aswasalsonotedearlier,however,thereisacriticaldistinctionbetweenpredictorvariables
andmoderatorvariables.Predictorvariablessimplycorrelatewithoutcomes.Forexample,
first-timeoffendersoftenhavebetteroutcomesinDTCsthanindividualswithpriorcriminal
convictions.Therefore,criminalhistoryisapredictorvariableforDTCprograms.Moderator
variables,incontrast,indicatewhichtypesofparticipantsperformedbetterintheDTCascom-
paredtoanalternativeprogram,suchasprobation.Individualswithpriorconvictionshave
poorer outcomes in allcorrectionalrehabilitationprograms,notjustinDTCs.Ifeveryprogram
chosetotargetfirst-timeoffendersbecausetheytendtohavebetteroutcomes,therewould
benoprogramsavailablefortheindividualswhoneedtreatmentthemost.
To determine whether participants’ criminal history is amoderator variable, the
evaluatormustdeterminewhetherdifferences inoutcomesbetween theDTCand com-
AVOIDING COMMON ANALYTIC MISTAKES
110
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
parisonprogramvariedinmagnitudebyparticipants’criminalhistory.Forexample,assume
thattenpercentoffirst-timeoffenderscommittedanewcrimeafterbeingtreatedinaDTC,
andfifteenpercentoffirst-timeoffenderscommittedanewcrimeafterbeingsentencedto
probation.Assumefurtherthattwentypercentofoffenderswithpriorcriminalconvictions
committedanewcrimeafterbeingtreatedinaDTCandthirty-fivepercentofoffenderswith
priorconvictionscommittedanewcrimeafterbeingsentencedtoprobation.Thedifference
inrecidivismforthefirst-timeoffendersisonlyfivepercentagepoints(10percentvs.15per-
cent)butthedifferenceinrecidivismforoffenderswithpriorconvictionsisfifteenpercentage
points(20percentvs.35percent).Inotherwords,themagnitudeoftheeffectoftheDTCis
smallerforthefirst-timeoffendersthanfortheindividualswithpriorconvictions.Thisistrue
eventhoughrecidivismwaslowerforthefirst-timeoffendersintheDTCthanforoffenders
withpriorconvictionsintheDTC(10percentvs.20percent).Underthesecircumstances,par-
ticipants’criminalhistorywouldbeamoderatorvariablefortheeffectsoftheDTC.
Failuretoappreciatethedistinctionbetweenpredictorvariablesandmoderatorvari-
ableshasledmanyDTCstotreatthewrongpopulation.Forexample,studiesintheUnited
StateshaverevealedthatDTCsreducedcrimeapproximatelytwiceasmuchandwere
approximatelyfiftypercentmorecost-effectivewhentheytreatedpersonswithpriorcon-
victionsorotherriskfactorsforreoffending(Bhatietal.,2008;Careyetal.,2012;Lowen-
kampetal.,2005).YetmanyDTCsintheUnitedStatescontinuetoservefirst-timeandlow-
riskoffendersbecausetheseindividualsaremorelikelytograduateandlesslikelytocommit
newcrimesthanrepeatoffenders.Becausefirst-timeandlow-riskoffenderstypicallyper-
formjustaswellonprobationorpre-trialdiversionastheydoinaDTC,thecontributions
oftheseDTCstopublicsafetyandpublichealthtendtobeminimal(Sevignyetal.,2013).
Toavoidmakingthissamemistake,evaluatorsinOASmemberstatesshouldperform
whatarecalledmoderator analyses or interaction analysesratherthansimplycorrelating
participantcharacteristicswithoutcomes.Theseanalysesrevealthreefindingsthatmust
beinterpretedtogether. Theyreveal(1)whichprogramproducedbetteroutcomes(e.g.,
111
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
DTCvs.probation);(2)whichtypesofparticipantshadbetteroutcomes(e.g.,first-timevs.
repeatoffenders)and;mostimportantly,(3)whichtypesofparticipantsperformedbetter
inwhichtypeofprogram.Thisthirdpieceofinformationindicatesthebestwaytomatch
individualstothemosteffectiveprogramstomeettheirneedsandindicateshowacountry
orjurisdictionshouldmarshalitsresourcestobemostefficientandcost-effective.Ifthere-
sultsofmoderatoranalysesarethesameinotherOASmemberstatesastheyhavebeenin
CanadaandtheUnitedStates,theyarelikelytoindicatethat,allelsebeingequal,first-time
addictedoffendersshouldtypicallybeplacedonprobationorpretrialdiversion,andrepeat
addictedoffendersshouldbeplacedinaDTC.
Disadvantaged Groups
DTCs commonly serve persons who have historically experienced discrimination or
reduced social opportunities due to their race, ethnicity, nationality, gender, physical or
mentaldisability,religion,orsocioeconomicstatus.Itisimportanttodeterminewhether
membersofthesedisadvantagedgroupsreceivethesameservicesandsucceedatthesame
rateasotherparticipantsinDTCprograms.
Manyevaluatorshaveattemptedtoaddressthisissuebycorrelatingvariablessuchas
raceorethnicitywithoutcomes.Forexample,studiesinAustralia(Rysavyetal.,2011)and
theUnitedStates (Finigan,2009;Marlowe,2013) found that racialminoritiesandmem-
bersofindigenouspopulationshadsignificantlypooreroutcomesinmanyDTCsthanother
participants.Thesefindingshaveledsomecommentatorstoconclude—prematurely—that
DTCsdiscriminateagainstpersonsofcolor(O’Hear,2009).
Itisthereforenotsufficienttosimplycorrelateoutcomeswithparticipantcharacteris-
ticssuchasraceorethnicity.Othervariablesmightbecorrelatedwithraceorethnicityand
mightbetrulyresponsibleforthedifferencesinoutcomes.
Infact,evidencesuggestsdisparitiesforminorities inDTCsarenotafunctionofrace
orethnicityper se,butratherareexplainedbybroadersocietalburdenswhichareoften
112
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
bornedisproportionatelybyracialandethnicminorities,suchaslessereducationalorem-
ploymentopportunitiesoragreaterinfiltrationofcrackcocaineandotherseriouslyaddic-
tivedrugsintosomeminoritycommunities.Whenevaluatorshaveadjustedstatisticallyfor
theseconfoundingvariablesintheanalyses,theimpactofraceandethnicitydisappeared
altogether(Dannerbecketal.,2006).
Thisfindingrequiresevaluatorstoaccountstatisticallyfortheinfluenceofvariablesthat
maybecorrelatedwithrace,ethnicity,orothercharacteristicsofdisadvantagedgroups—
especiallyparticipants’ socioeconomic statusandprimarydrug(s)used—and thendeter-
minewhetherraceorethnicitycontinuestopredictpooreroutcomesafterthesecovariates
havebeenfactoredout.Thisprocesscanbeaccomplishedinseveralways,buttheessen-
tialprocedureistofollowthreebasicsteps:First,theevaluatormustdeterminewhether
variablessuchassocioeconomicstatusordrugsusedaresignificantlycorrelatedwithrace,
ethnicityorothercharacteristicsofdisadvantagedgroups.Iftheanswertothisfirstques-
tionisyes,thentheevaluatormustdeterminewhetherthesevariablesarealsocorrelated
withoutcomes,suchasgraduationratesornewarrestrates.Iftheanswertothesecond
questionisalsoyes,thenthethirdstepistoenterthesevariablesascovariatesintoanalyses
comparingoutcomesbetweenmembersofdisadvantagedgroupsandotherparticipants.
Ifthedisadvantagedgroupscontinuetohavesignificantlypooreroutcomesafterthisthird
stephasbeenfollowed,thenandonlythenwoulditbejustifiedtoconcludethatthereare
disparateimpactsforthosedisadvantagedgroupsinDTCs.
Adjusting for Baseline Differences
Aswasdiscussedpreviously,mostevaluationscompareoutcomesbetweenDTCpar-
ticipants and those of a comparison group such as probationers. Prior tomaking these
comparisons, it isnecessarytoruleoutcompetingexplanationsthatmightaccountfor
differencesinoutcomesotherthantheeffectsoftheDTCprogram.Forexample,thepro-
bationersmighthavehadmoreseverecriminalhistoriestobeginwith.Ifso,thensuperior
113
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
outcomesfortheDTCparticipantsmighthavehadnothingwhatsoevertodowiththeeffects
oftheDTCprogram,butrathermightsimplyhavereflectedthefactthattheDTCtreateda
populationthatwasalreadymorelikelytoarriveatmorefavorableoutcomes.Indeed,even
whenanevaluatoremploysrandomassignment,thereisstillapossibility(althoughagreat-
lydiminishedpossibility)thatthegroupsmaydifferonimportant/materialdimensions.
Ifthegroupsdidmateriallydifferatthetimeofentry,allisnotnecessarilylost.There
areproceduresanevaluator canemploy toadjust for thedifferencesandobtainpoten-
tiallydefensibleresults.Thistypicallyinvolvesathree-stepprocess.First,thegroupsare
comparedonriskfactorsthatareknowntoinfluenceoutcomesamongdrug-involvedindi-
vidualschargedwithcriminaloffenses.Theseriskfactors,suchasageandpriorcriminal
convictions,arelistedearlierinthismanual.Ifthegroupsarefoundtodifferonanyofthese
riskfactors,theevaluatormustnextdeterminewhether, in fact, theseriskfactorspre-
dictedpooreroutcomesinthecurrentsample.Forexample,althoughayoungerageoften
predictspooreroutcomesamongdrug-involvedoffenders,itmightnothavedonesointhe
presentstudy.Ifitdoesnotpredictoutcomesinthecurrentstudy,theevaluatordoesnot
needtobeconcernedaboutthisvariablegoingforward.
Third,anyvariablesthatboth(A)differbetweenthegroupsand(B)predictoutcomes
mustbecontrolledforstatisticallyintheoutcomeanalyses.Forexample,ifthegroups
differbyage,andtheageofaparticipantpredictsoutcomes,thenagemustbeenteredasa
covariateintheoutcomeanalyses.Howthisisaccomplishedwilldependonwhatstatistical
test isbeingperformed. Forexample,theevaluatormightuseananalysisofcovariance
(ANCOVA)orahierarchicalregressionanalysis,whichcanadjustforcovariatesthatmight
beconfoundingtheresults.
Unfortunately,inmanyevaluationstheDTCmightnothavecollecteddataconcerningall
riskfactorsthatshouldbeconsidered.Forexample,theevaluatormightnothaveanyway
ofknowingwhethertheparticipantshadpriorcriminalconvictionsorhadbeeninsubstance
usetreatmentpreviously.Withoutthisimportantinformation,thereisnowayofknowing
114
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
whetherthegroupsdifferedonthesecharacteristicswhentheyenteredtheprogramsand
whethersuchdifferencesmighthaveinfluencedtheoutcomesthatwereproduced. Ifthis
criticalinformationislostorunavailable,theresultsoftheevaluationcannotbereliedupon
withareasonabledegreeofscientificconfidence.(Oncemore,thevalueofDTCs’initialdesign
andimplementationofrobustdatacollection,entry,andmanagementmethodsismanifest.)
Impact of Specific Services or Interventions
ManyDTCswillwanttoknowwhetherspecificservicestheyprovideorconsequences
theyimposeimprove,hinder,orhavenoeffectonoutcomes.Forexample,aDTCpracti-
tionermightwanttoknowwhetheradministeringpunitivesanctionsorreferringpartici-
pantstoanewtypeofcounselinggroupimprovedtheiroutcomesandwasworththeadded
effortandexpense.
Ideally,suchanalysesshouldnotbeconductedpost hoc,orafterthefact.Assume,for
example,thatanevaluatorcorrelatedthenumberofsanctionsthatparticipantsreceived
with theiroutcomes inaDTC. It isdetermined from this analysis thatparticipantswho
receivedmoresanctionswere less likelytograduatesuccessfully fromtheprogram.This
couldleadtheevaluatortotheerroneousconclusionthatsanctionsmadeoutcomesworse.
Butsuchaconclusionislikelytohaveconfusedcausewitheffect.Itismorelikelythatpoor
performanceledthestafftoapplymoresanctions.Ifso,thenmoresanctionsdidnotcause
worseoutcomes,butratherworseoutcomeselicitedmoresanctions.
Similarly, participantswhohavemore severe addictionproblemsorwho arenot re-
spondingtostandardtreatmentaremorelikelytobereferredformoreintensiveorspe-
cializedtreatment.Simplycorrelatingtreatmentwithoutcomescouldleadtotheerroneous
conclusionthatmoretreatmentledtopooreroutcomes,wheninfactpoorperformance
oftenleadstoareferralformoretreatment.Forexample,participantswithmorese-
vere addictions aremore likely to be referred for residential treatment than outpatient
treatment.Iftheirlong-termoutcomesareworsethanforparticipantswhowerereferred
115
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
tooutpatienttreatment,theevaluatormightreachthewrongconclusionthatresidential
treatmentisineffectiveorharmful,wheninfactresidentialprogramssimplytreatedamore
seriouspopulationthanoutpatientprograms.
Thebestmethod istoaddressthesequestionsa priori,orbeforehand. Ifpossible,
participantsshouldberandomlyassignedinadvancetodifferenttypesordosagesofser-
vices.Forexample,onarandombasissomeparticipantsmightberequiredtoattendfifty
hoursoftreatmentwhereasothersmightberequiredtoattendtwentyhoursoftreatment.
Usingthisresearchdesign,anydifferencesinoutcomesmaybereasonablyattributedtothe
dosageoftreatmentandnotmerelytosystematicreferralsbystaff.
Analternativeapproachistoconductamediation analysis.Asmentionedearlier,ame-
diatorvariableisathirdvariablethataffectstherelationshipbetweenapredictorvariable
(suchasenrollmentinaDTC)andanoutcomevariable(suchasrecidivismorgraduation
rates).Themediatorvariablecanbethoughtofasaninterveningvariablethatexplainsthe
relationshipbetweenthepredictorvariableandtheoutcomevariable.Inordertoconfirm
thatthereisamediationeffect,severalstatisticalrelationshipsmustbeestablished(Baron
&Kenny,1986;MacKinnon,2008):(1)thepredictorvariablemustbesignificantlycorrelated
withtheoutcomevariable,(2)thepredictorvariablemustbesignificantlycorrelatedwith
themediatorvariable,and(3)themediatorvariablemustbesignificantlycorrelatedwith
theoutcomevariable. If these three conditions are allmet, then itmustbe further
establishedthat(4)therelationshipbetweenthepredictorvariableandtheoutcome
variableiseliminatedorsignificantlyreducedinmagnitudeaftercontrollingfortheme-
diatorvariableasacovariate.
Toillustrate,assumethatanevaluatorwantedtoknowwhetherDTCswereresponsible
forbetteroutcomesthanprobation,becausetheformerprovidemorehoursoftreatment
toparticipants.Todoso,theevaluatorwouldneedtoestablishthat(1)beingassignedto
theDTCwascorrelatedwithbetteroutcomesthanbeingassignedtoprobation,(2)being
assignedtotheDTCwascorrelatedwithreceivingmorehoursoftreatmentthanprobation,
116
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
and(3)receivingmorehoursoftreatmentwascorrelatedwithbetteroutcomes.Then,the
evaluatorwouldneedtoestablishthattheeffectoftheDTCwassignificantlyreducedafter
accountingfortheamountoftreatmentparticipantsreceived. Ifallthreeconditionsare
met,thentheevaluatorcanreasonablyconcludetheDTCworkedbyincreasingparticipants’
exposuretotreatment,whichinturnledtobetteroutcomes.Inotherwords,increasingex-
posuretotreatmentpartiallymediatedorpartiallyexplainedthepositiveeffectsoftheDTC.
Commonstatisticalproceduresthatmaybeusedtotestformediationinclude,butare
notlimitedto,ordinaryleastsquares(OLS)multipleregressionanalyses,structuralequation
modeling(SEM),logisticregressions(forbinarymediatororoutcomevariables),andmulti-
levelmodeling(forclusteredornesteddata).Inmanyinstances,expertstatisticalconsulta-
tionwillberequiredtoconductthesesophisticatedmediationanalysescorrectly.
Infrequent Events
SomeeventsoccurinfrequentlyinDTCprograms.Forexample,ifoutcomesareexamined
duringthefirstsixmonths,arrestsfornewoffensesmightnotoccurathighrates.Itusually
takesa longertimethansixmonthsforparticipantstore-engageincriminalconduct,be
detectedbylawenforcement,andhavecriminalchargesfiledagainstthem.Amongthose
participantswhodoquicklyreturntocriminalbehavior,onewouldnotexpecttoseemore
thanaboutoneortwonewarrestsduringthefirstsixmonths.
Acommonmistake inDTCevaluations is toemploystatistical tests suchas t-testsor
analyses of variance (ANOVA)which examine the averagenumberof infrequent events,
suchastheaveragenumberofnewarrests.Becausethenumberofnewarrestsislikelyto
bezeroformanyparticipants,theaveragenumberofarrestsislikelytobelessthanoneor
two.Thiscreateswhatiscalledaskewed distribution or non-normal distribution of scores.
Formathematicalreasons,oneisunlikelytodetectstatisticallysignificantdifferencesifthe
distributionisskewedinthismanner.Althoughtheremightbeaclinicallymeaningfuldiffer-
encebetweenthegroups,itisunlikelytobedetectedbythestatisticaltest.
117
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Forinfrequentevents,itmaybeadvisabletotransformthedataintoabinaryor
dichotomous variable(e.g.,anynewarrestvs.nonewarrest)andtousewhatarecalled
nonparametricstatisticaltests.Forexample,ratherthananalyzingtheaveragenumberof
newarrests,theevaluatormightanalyzethepercentageofparticipantswhohadatleastone
newarrest.Nonparametrictestscanhelptoavoidtheproblemsassociatedwithskewed
distributionsandcanleadtomoresensitiveandpowerfulfindings.Examplesofcommonly
used nonparametric tests include, but are not limited to, chi-square tests, Fishers Exact
ProbabilityTests,MannUtests,Wilcoxontests,andlogisticregressionanalyses.
Alternatively,evaluatorsmayperformwhatiscalledalog-linear transformationofthe
data to compensate for the skewedornon-normaldistribution. Thisprocedurehas the
mathematicaleffectofsmoothingouttheskeweddistributionormakingitfunctionasifit
wereanormaldistribution.Expertstatisticalconsultationwilloftenberequiredtoknow
whenandhowtoemployanadvancedcompensatoryproceduresuchasalog-lineartrans-
formation.
118
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
DTCprogramsuseresourcesfrommultiplesystemsandagencies(e.g., thecourt,the
criminal justicesystem,andthesubstanceabusetreatmentsystem).Suchprogramsthus
needtobedesignedandimplementedinamannerthatmaximizesbenefitsandminimizes
coststoparticipants,thevariouspublicsystems/agenciesinvolved,andtosocietyingeneral.
Policymakers and thepublic thereforehave an interest in calculating and comparing
thesebenefitsandcosts.Onewaytoprovideafamiliarandusefulstandardforsuchcal-
culationsandcomparisons ismonetizing(puttingdollaramountsto)thevariousbenefits
andcoststhroughso-calledcost-efficiencyanalyses.ComprehensiveDTCcost-efficiency
analysesrequirethefollowingelements:
1) Acarefulprocessevaluationtoidentifytimesandplaceswherecostsandpotential
benefitsareincurred;
2) Collectionofquantitativedataonprogramactivities(e.g.,fromadministrative
databasessuchascriminaljusticedatabasesthatcontaindatesofarrestsandcharges);
3) AnanalysisoftheDTCprogram’sbenefitsandcosts;and
4) AnanalysisofanappropriatecomparisongroupinanalternativetotheDTC,to
juxtaposewiththeanalysisoftheDTCprogram.
Naturally,puttingtheoryintopracticeisoftenmorecomplexthanitmayfirstappear.
Potentialchallengesinherentindesigningandcarryingoutcost-efficiencyanalysesinclude:
�AssigningdollaramountstothevalueofcertainDTCbenefitscanbedifficult;
�Cost-efficiencyanalysisinvolvesmakingcertainassumptions(e.g.,thatthegoalsand
outcomessoughtbydifferentprogramsarethesame;thatthedollarvalueofacost
orbenefitinonelocationisthesameasitsvalueinanother)thatmustbetestedto
ensuretheintegrityoftheanalysis;and
�MonetarybenefitsthataDTCmaygenerateoveranalternativeapproacharesome-
timeaccruedbyagenciesotherthantheDTCitself(e.g.,statecorrectionsagencies).
COST ANALYSESMike Finigan and Shannon Carey
119
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
TherearefourmajortypesofcostanalysisthathavebeenappliedtoDTCsaswellas
substanceabusetreatmentprogramsingeneral.Theyrangefromsimple/inexpensivebut
lessaccurateapproachestothosethatarecomplex/expensivebutmoreaccurate.Assuch,
each implies certain trade-offs thatpolicymakers,evaluators, andDTCprogramstaffwill
needtotakeintoconsideration.
Type 1: Program-cost analysis. Program-costanalysesinvolvecalculatingalloftheex-
pensesrequiredtoimplementandoperateagivenprogram.Theyarethemostbasicform
ofcostanalysis,andallothertypesbuilduponthistype.
Abasic program-cost analysis is used to showwhat resources areneeded to fund a
particularprogram.Theprogram’sbudget(s)andanaccountingofallotherresourcesput
intotheprogram(in-kindresources,etc.)arerequiredtoproperlyassessaprogram’sfull
cost.Consequently,theprogramcoststhatabasicprogram-costanalysismeasuresshould
includeboththemorevisibleandeasilycalculateddirectcosts,suchassalariesandbenefits
forDTCstaff,butalsotheprogram’sindirectcosts,suchasthoseforsupportservices,train-
ing,officespace,andadministrativeoverheadgenerally.
Duetotheirrelativesimplicityandinexpensivenature,program-costanalysescanbeuseful
forprogramsthathavefewevaluationresourcesand/orneedonlytocarryouta limited
assessment of costs.
Type 2: Cost-allocation analysis.Costallocationanalysesdeterminetheunitcost(or
costperunit)ofaparticularserviceintheprogram(e.g.,thecostpercourtsession,orthe
cost per screening). This provides thebasic information required formore sophisticated
cost-effectivenessanalysisorcost-benefitanalysis,thethirdandfourthtypesofanalyses
explainedhere.Moreover,cost-allocationanalysesarevitaltostudyDTCprogramsinwhich
multipleagenciescontributeresourcesandservices.
Accordinglycost-allocationanalysesrequirenotonlysettingupaccountingorbudgeting
systemsinawaythatallowsprogramstafftoallocatecoststothevariousservicesprovided,
120
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
butalsocollectinginformationaboutprogramparticipantsandservicesdelivered.
Whilecost-allocationanalysespermitcalculationofprogramandunitcosts,theycannot
bythemselvesdeterminewhethertheprogramisanefficientuseofresources.Suchade-
terminationrequiresacomparisonoftheDTCprogramtootheralternatives.Forexample,
asimilarprogrammayproduceequivalentresultstotheDTCprogramatalowercost,thus
makingitmoreefficientintermsofcosts.Inordertoreachconclusionsaboutefficiency,a
cost-effectivenessanalysisisrequired,whichisthenexttypeofanalysisconsidered.
Type 3: Cost-effectiveness analysis. Cost-effectivenessanalysesevaluatealternative
programs’costsandeffectswithregardtoproducingadesiredoutcome.Inacost-effective-
nessanalysis,thecostsofdifferentprogramsareassessedandthencomparedintermsof
similar outcomes.
Nonetheless, keep in mind that program outcomes are not expressed in monetary
terms,butratherinsomeotherimpact,suchasareductioninrecidivism.Inthecontextof
acost-effectivenessanalysis,areductioninrecidivismisnotexpressinmonetaryterms,but
isinsteadexpressedinanon-monetaryvalue,suchasthepercentreductioninrearrests.
Inthatcontext,thecost-effectivenessanalysiswouldexpressresultsintermsthepercent
reductioninrearrestsperacertainamountofmoneyspent.
Thefollowing(seeFigure3)isanexampleofacost-effectivenessanalysiswhereinaDTC
iscomparedtoastandardprobationprogramcost,intermsofeachprogram’seffectiveness
inproducingthedesiredoutcome(reductioninrearrests)pereveryUS$10,000incosts:
121
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Inthisexample,Figure3showsthatDTCfordrugoffendersreducesthepercentageof
rearrestssubstantiallymorepereachUS$10,000spentthanthestandardprobationpro-
gram. Accordingly,whilecost-effectivenessanalysesaremorecomplexand require a full
outcomeevaluation foreachprogrambeingcompared, theyarealsoavaluable tool for
policymakersthatwishtodeterminethecostefficiencyofalternativeapproaches.
Type 4: Cost-benefit analysis. Lastly,acost-benefitanalysis (CBA)calculates the
monetarycostofprogramsaswellasthemonetarycostoftheoutcomestothesystem.This
resultsinacost-to-benefitratiothatmonetizesbothcostsandoutcomes—thatis,aratio
thatindicatesthatthereareXdollarsinbenefitsforeverydollarinvestedintheprogram.A
programwithbenefitsthatoutweighitscostsissaidtohavenet benefits.
CBAsprovidethemostcompletepictureofthevalueofacomplexprogramsuchasDTC,
and represent the only approach that allows an assessment of cost-benefit ratio.While
this is themostcomprehensivetypeofcostanalysis, italsotakesmoretime,effort,and
Percent reduction in rearrests per
US$10,000 spent
DTC Probation
20%
15%
10%
5%
0%
20%
7%
Figure 3. Compares the Impact of Drug Treatment Court and Standard Probation per US$10,000 spent
122
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
resourcestoconduct.CBAsrequireafulloutcomeevaluation,dataonoutcomesforboth
programparticipantsandacomparisongroup,andcostdatathatcanbeassociatedwith
thoseoutcomedata. Nonetheless, given that policymakers are interested in seeing the
monetarybenefitsofanewprogrambeforedecidingtoprovidefinancing,CBAscanbeex-
tremelyuseful.
Toprovideareal-worldexample,oneoftheearliestCBAsofU.S.DTCsfoundthe
following:AnestimatedUS$1,002,979wasspentpercohortofclientswhoparticipatedin
theDTCprogrameachyear.Ontheotherhand,theDTCprogramresultedinUS$2,476,795
ofavoidedcosts (or savings). Therefore, the studycalculated thatevery taxpayerdollar
spentonthosecohortsproducedUS$2.50ofsavingstothetaxpayer.Furthermore,ifthe
broadercosts(includingvictimizationandtheftcosts)wereestimatedandincludedinthe
analysis,theratioofbenefittothetaxpayerwasten-to-one—thatis,theprogramsavedten
dollarsforeverydollaritspent.(Carey&Finigan,2004).
Cost Assessment Techniques
Thewaycostsareassessedalsovariesdependingonthepolicyquestionsdesiredand
thebudget forcostevaluation. Anumberofassessmenttechniquesexists,asexplained
below.
Funded budgets
SomecostanalysesassociatedwiththeimplementationofDTCshaveassumedthatarea-
sonablewayofassessingthecostofDTCistofocussolelyonthefundsspentcreatingtheDTC
program.(IntheUnitedStates,theseareoftenfederalgrantfundsspentontheproject.)This
approach,however,failstoaccountforresourcesthattheexistingcriminaljusticeandtreat-
mentsystems,whicharenotfundedbyDTCgrantsorotherdirectDTCappropriations,must
contributetotheDTCprogram.Thus,theapproachofusingfederalgrantmoneyordirect
federalDTCappropriationstoassess investmentcostsseriouslyunderestimatesthe invest-
mentofstateandlocaljurisdictionalandagencyresourcesforDTCs.
123
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Marginal costs
Themarginalcostrepresentstheamountthatthetotalprogramcostchangeswhen
aunitofoutput(alsoreferredtoas“workload”)changes.Inacost-benefitanalysis,“mar-
ginal”thusdoesnotmeansmallorinsignificant,butratherthecoststhatareatthemargin
ofanexistinglevelofoperations.Inotherwords,thetermdescribesthecostorbenefitthat
willberealizedbecauseofchangesinunitsofactivity.
Inthecontextofthecriminaljusticesystem,themarginalcostistheamountofchange
inanagency’stotaloperatingcostswhenoutput(suchasarrests,courtfilings,orjaildays)
changesbecauseofchangestopoliciesorprograms.Forexample,costsforaprison—such
asbuildingmaintenance,utilities,andoverheadexpense—arefixedandgenerallydonot
changewhetherornotaprisonbedisfilled.Variablecostssuchasfoodsupplies,however,
wouldchangeiftherewerefewerormoreoffendersintheprison.
Themarginal cost approach argues thatmarginal costs should only include variable
costs(e.g.,overtime,supplies,food)andnotfixedcosts(e.g.,rents,utilities),becausefixed
costsdonotmateriallychangewiththeadditionofonemoreunitofworkload(Henrichson
&Galgano,2013). Therefore,whenusingamarginal costapproach,only thesevariable
costsshouldbecountedinmeasuringincreasesordecreasesincost(ormeasuringmone-
tarysavings).
Thereare,however,severalreasonsthatthismarginalcostapproachmaynotworkin
complexinter-organizationalsettingssuchastheDTCoperatingenvironment.First,itdoes
notaccountfororganizationalorotherstructuralchangesthatoccurinacriminal justice
systemwhenaDTCisimplemented.Forinstance,becausethemarginalcostapproach
assumes that the standard operation of the court remains the same, it fails to account
forthefactthatcourtcalendarsmaychangeasjudgesincreaseordecreasecaseloadsto
accommodatetheDTCjudge’scaseload. Duetothesamekindofassumptionaboutthe
standardprobationapproachremainingthesamewhenaDTCisaddedtothesystem,the
marginal costapproachalso fails toassess the systemiceffect involved in caseload re-
124
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
assignmentsforprobationofficersresultingfromtheassignmentofDTCcaseloads.These
caseloadstypicallyhavemoreintensivesupervisionrequirementsassociatedwiththeDTC
program,apracticethataffectsbothfixedandmarginalcosts.Inaddition,thisapproachof-
tenunderestimatesthecostsofday-to-dayoperations,so-called“business-as-usual”costs
thatDTCsoftenoffsetorreduce.Inotherwords,themarginalcostapproachassumesthat
theDTCenvironmentwouldnotimprovethecostofprocessing“business-as-usual”cases,
andthusoffsetsomeofthenewcostsand/oreliminatingsomeofthese“business-as-usual”
costs. Itisclearthatinsomecases,theimplementationofDTCsactuallyreducesthecosts
ofprocessingeligibleclientsascomparedto“business-as-usual.”InastudyofnineDTCsin
California,onecourtfoundthatprocessingclientsthroughDTCactuallysavedthecriminal
justicesystemUS$1,500perclientwhenbothfixedandmarginalcostssuchasrentandutil-
itieswereincludedintheanalysis(Carey,Finigan,Crumpton,&Waller,2006). Marginal cost
analysismayhaveidentifiedthe“new”costsfromtheintroductionoftheDTCprogramand
failedtoassessthesavingsinthetraditionalsystemthataccruedfromthisnewmethodof
processingcases.ThesameresultwasfoundinaDTCinOregon(Carey&Finigan,2004).It
isalsoclearfromseveralothersitesthatsimplyassessingthevariableorsemi-fixedcostsof
allDTCcomponentsthatarenewtothesystemwouldinaccuratelyrepresentthenetcost
oftheDTC.
Broader societal outcomes
SomecostapproachescanmeasuretheimpactofDTCsbeyondtheirimmediateimpact
onthecriminaljustice,courtandsubstanceabusesystems—somethingthatsomepolicy-
makersdesire.AnotableexampleofthisapproachistherecentanalysisbytheNationalIn-
stituteofJustice(theMultisiteAdultDrugCourtEvaluation,MADCE),whichincludedsocial
productivityandbroaderserviceuseinitsanalysis(Roman,2013)(seeTable2).
125
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Table 2. Outcomes Measured by MADCE
Outcome Category Outcome Sub-Category Impacts
1. Social productivity
A.Employment B.Education C.Servicesandsupportprovided
• Earnings• Schooling• Childsupportpayments,com-
munityservice
2. Criminal justice system
A. Monitoring B.Police C. Courts D.Corrections E.DrugTreatmentCourt
• Probationofficermeetings,drugtests,electronicmonitoring
• Arrests• Hearings• Jailandprison(sanctionsor
otherwise)• Casemanagement,
administrativecosts
3. Crime and victimization • Crimescommitted
4. Service use A. Drug treatment B.Medicaltreatment C.Mentalhealthtreatment D.Other
• Emergencyroom,detoxification,residentialcare,outpatient,methadone
• Hospitalstaysunrelatedtodrugs
• Staysinmentalhealthfacilitiesunrelatedtodrugs
• Halfwayhouses,publichousing,homelessshelters
TheMADCEresearchersuseda“bottom-up”approach,whichcomparingthecostsand
avoidedcosts foreachparticipant in termsof socialproductivity, criminal justice system
activities,crimeandvictimization,andserviceuse.Theresultingmeasurecouldbeapositive
ornegativevalueforeachpersoninthestudy(Roman,2013).TheMADCEfoundthatinsome
broaderareassuchasserviceuseandfinancialsupportthedrugcourtparticipantscostmore,
yetthesavingswithinthecriminaljusticesystemmorethanoffsetthesecosts.Theoverall
resultwasacostsavings(althoughthesavingsinthisparticularstudywerenotsignificant).
126
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Thisbroadapproachproducesexcellentsociety-wideresultsbutisdifficulttoimplement
becausegettinggoodcostdataonimpactedexpensesrequiresextremelydetailedmeasure-
ment,asdescribedearlierinthesectionon“SourcesofInformation.”Theseincludefinding
dataonsuchquestionsasuseofdetoxificationservices,emergencyroomvisits,lostwages
orwageincreases,taxpayments,welfarepayments,anddisabilitypayments(asdescribed
inTable2).Datasourcesforthesedataelementsareoftenquitedifficulttofind,ordon’t
existatall.
Transactional and Institutional Cost Analysis (TICA)
Adifferentcost-benefitapproachthatusesbothmarginalcostsandfixedcostscomes
from transactional economics, and is called Transactional and Institutional Cost Analysis
(TICA)(Crumpton,Carey,&Finigan,2004).TheTICAapproachviewsanindividual’sinter-
actionwithpubliclyfundedagenciesasasetoftransactionsinwhichtheindividualutilizes
resourcescontributedfrommultipleagencies.Transactionsarethosepointswithinasystem
whereresourcesareconsumedand/orchangehands.
In the caseofDTCs,whenaDTCparticipantappears in courtorhasadrug test, re-
sourcessuchasjudgetime,defenseattorneytime,courtfacilities,andurinecupsarecon-
sumed.Courtappearancesanddrugtestscan,inthiscontext,beanalyzedastransactions.
Inaddition,theTICAapproachrecognizesthatthesetransactionstakeplacewithinmultiple
organizationsandinstitutionsthatworktogethertocreatetheprogramofinterest.These
organizationsandinstitutionscontributetothecostofeachtransactionthatoccursforpro-
gramparticipants.BecauseTICAmeasuresthecostsofbothtransactionsandthespecific
organizationsthatarecontributingresources,TICAisaparticularlyappropriateapproach
toconductingcostsassessmentsinanenvironmentsuchasaDTC,whichinvolvescomplex
interactionsamongthemultipleorganizationsinvolvedintheDTCprocess.
127
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Cost to the taxpayer
Inordertoproduceinformationusefultopolicymakers,intheUnitedStatesa“cost-to-tax-
payer”approachisused.Thisfocusonthetaxpayer(orintheaggregate,thegovernment)
helpsdefinewhichcostdatashouldbecollected(costsandavoidedcostsinvolvingpublic
funds)andwhichcostdatashouldbeomittedfromtheanalyses(e.g.,coststotheindividual
participatingintheprogram).Althoughthis“taxpayer”approachmaynotbeapplicablein
somecountries,asimilarapproachcouldbetakenbyfocusingonaspecificfundingsource
(orasubsetoffundingsources)thatareofparticularrelevanceorimportintheoperation
oftheprogram.Focusingonasmallersubsetoffundingsourcescanhelpnarrowdownand
betterdefinethebreadthofdatathatmustbecollected,whichmaymakeacostanalysis
morefeasibletoaccomplishwithalimitedbudget,oralimitedtime.
Opportunity resources
Finally, theTICAapproach looksat costs as “opportunity resources.” The concept is
similartothatofopportunitycostfromtheeconomicliterature,whichsuggeststhatsystem
resourcesareavailabletobeusedinothercontexts iftheyarenotspentonaparticular
transaction. The termopportunity resource describes these resources that are now
availablefordifferentuse.
Forexample,ifsubstanceabusetreatmentreducesthenumberoftimesthataclientis
subsequentlyincarcerated,thelocalsheriffmayseenochangeinhisorherbudget.How-
ever,anopportunityresourcewillbeavailabletothesheriffintheformofajailbedthat
cannowbefilledbyanotherpersonwho,perhaps,possessesamoreseriouscriminaljustice
record than does the individualwho has received treatment and successfully avoided
subsequentincarceration.
Notably,theTICAapproachdoesnotuseamarginalcostapproach.Allcostsreported
are fully loaded,3totalcosts.Thus,theTICAapproachdoesnotassumethatfixedcosts,such
3 Fully-loadedcostsarethosethatincludedirectcosts(suchassalaries),aswellasindirectcostssuchasbene-
128
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
asforbuildingmaintenance,utilities,salaries,orotheroverheadexpenses,areoff-limits.
Thosecostsarestillbornebyfundersandcanchangeovertimeorbedeployedfordifferent
purposes.
TheTICAapproachalsodoesnotmakeassumptionsastowhetherornotagencieswill
changetheirbudgetsinwaysthatenablesavingstobeliterallyrealized.All“savings”
calculatedusingtheTICAapproachrepresent“opportunityresources”thatarenewlyavail-
abletogovernmentoragencyactorstodeployastheychoose.Ratherthanassumethat
directdollarswill be immediately saved (e.g., through reductions in the followingyear’s
budget),theTICAapproachfullyquantifiesallresources(marginalorfixed)—suchasajail
bed,treatmentslot,ortimespentinacourthearing—thatarenewlymadeavailablefora
differentoffenderoradifferentpurpose.
Table3providesthesixkeystepsintheTICAapproach.
fitsandoverhead.
129
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Table 3. The Six Steps of TICA
Description Tasks
Step 1:Determineflow/process(i.e.,howprogramparticipantsmovethroughthesystem)
Sitevisits/directobservationsofprogrampractice
Interviewswithkeyinformants(agencyandprogramstaff)usingadrugcourttypologyandcostguide(Seeguideonwww.npcresearch.com)
Step 2:
Identifythetransactionsthatoccurwithinthisflow(i.e.,whereclientsinteractwiththesystem).SeeTables4and5foralistofprogramtransactions.
AnalysisofprocessinformationgainedinStep1
Step 3:
Identifytheagenciesinvolvedineachtransaction(e.g.,court,treatment,police)
AnalysisofprocessinformationgainedinStep1
Directobservationofprogramtransactions
Step 4:
Determinetheresourcesusedbyeachagencyforeachtransaction(e.g.,amountofjudgetimepertransaction,amountofattorneytimepertransaction,numberoftransactionevents)
Interviewswithkeyprograminformantsusingprogramtypologyandcostguide
Directobservationofprogramtransactions
Administrativedatacollectionofnumberoftransactions(e.g.,numberofcourtappearances,numberoftreatmentsessions,numberofdrugtests)
Step 5:
Determinethecostoftheresourcesusedbyeachagencyforeachtransaction(e.g.,staffsalaries,overheadrates,supplycosts)
Interviewswithbudgetandfinanceofficers
Documentreviewofagencybudgetsandotherfinancialpaperwork
Step 6:
Calculatecostresults(e.g.,costpertransaction,totalcostoftheprogramperparticipant)
Indirectsupportandoverheadcosts(asapercentageofdirectcosts)areaddedtothedirectcostsofeachtransactiontodeterminethecostpertransaction
Thetransactioncostismultipliedbytheaveragenumberoftransactionstodeterminethetotalaveragecostpertransactiontype
Thesetotalaveragecostspertransactiontypeareaddedtodeterminetheprogramandoutcomecosts.
130
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Aspartofatypicalprocessevaluation,Step1(understandingtheflow,orprocess,of
theprogram),Step2(identifyingthetransactionsintheprocess)andStep3(identifyingthe
agenciesinvolved)areperformed(seethesectionofthismanualonprocessevaluation).
Athoroughprocessevaluationshouldprovideyouwithanunderstandingoftheprogram
process,includingwheretransactionsareoccurring(e.g.,placeswhereprogramstafftime
oragencyresourcesarebeingused)andwhatagenciesare involvedineachtransaction.
Forexample,whenperformingasitevisittotheprogram,anevaluatorcanobserveaDTC
hearing(i.e.,atransaction),theevaluatorcanobservehowthestaffandparticipantsin-
teract(theprogramflow),andwhichagenciesareinattendanceatthehearing.
Step 4 (determiningtheresourcescontributedbyeachagency)occursthrough inter-
viewsandobservation,aswellasgatheringquantitativedataonnumbersoftransactions
foreachparticipant.Continuingwiththeexampleofthecourthearing,theevaluatorcan
usea stopwatch (oraclock) totimehow long thestaff fromeachagencyspends in the
courtroom.Inaddition,eachofthestaffmembersshouldbeinterviewedtoaskthemhow
longtheyspendpreparingforthecourthearingandhowmuchtimetheyspendonother
DTC-relatedtransactions.Finally,dataonthenumberofcourthearingsattendedbyeach
participantintheprogramshouldbecollected.Iftheprogramiscollectingdatesofcourt
hearingsinaprogramdatabase,thenumberofcourthearingsattendediseasilycounted.
Iftheprogramisnotcollectingthisinformation,theevaluatorcanestimatethenumberof
courthearingsbasedonprogrampolicyon the frequencyofcourthearings required for
participants(althoughthisestimatewilllikelybelessaccurate).
Step 5(determiningthecostoftheresources)iscalculatedbydeterminingstaffsalaries
(howmuchiseachstaffmemberpaidforhisorhertime)andreviewingagencybudgetsfor
expendituresrelatedtoDTCactivities,aswellastodeterminestaffbenefitratesandany
overheadratestheagencymayhave.Theseratesaregenerallyexpressedasapercentage
ofstaffsalaries.
131
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Step 6(calculatingthecostresults)mostlyinvolvessimplemultiplicationandaddition.
Foracourthearing(oneof themostcomplicatedtransactions,as it involveseveryteam
memberandagencythatcontributesresources),thesalaryofeachstaffmemberismulti-
pliedbythepercentageoftimespentbothpreparingforandattendingthecourtsession.
Thecostforeachstaffmemberisthenmultipliedbytheoverheadrateandbenefitratefor
thatagency.The“fullyloaded”costsforeachstaffmemberarethenaddedtogetherfor
thesumtotalcostofthecourtappearancetransaction.Thistotalcostcanbedividedby
thenumberofparticipantsthatattendacourtsessiontodeterminethecostofaDTC
appearanceperparticipant.Followingisasamplecalculation.
Ifajudgeworks40hoursperweek,andspends10hoursperweekofthattimepreparing
forDTCstatusreviewhearings(e.g.,gatheringinformationfromtheDTCteam,participating
inpre-courtstaffingmeetings)andparticipatingintheDTChearingsthemselves,thenthe
judgespends25percentofhisorhertimeonpreparingfororparticipatinginDTChearings.
IfthejudgeearnsUS$100,000peryear,thenUS$25,000(US$100,000X.25)goestoward
statusreviewhearingseachyear.DividingUS$25,000bythe52weeksinayearresultsina
costofUS$480.77perweek.IfaDTCprogramsees50participantsinstatusreviewhearings
eachweek,thendividingUS$480.77by50resultsinacostofUS$9.62perparticipantper
hearingforthejudge’stime.
Toachievethe“fullyloaded”cost,itisnecessarytoaddthecostofanyfringebenefits
earnedbythejudge(e.g.,thejudgemayearnanadditional20percentofhisorhersalary
invacationtime,retirementpayments,andhealthcarebenefits)andaddthecostofany
overheadpaidbythecourt(e.g.,afivepercentoverheadratetoheatandlightthecourt-
housewherethejudgeworks,etc.).So,thefullyloadedcostforthejudgetoparticipatein
asingleparticipant’scourthearingwouldaddthisadditional20percentandfivepercentto
theUS$9.62,resultinginafullyloadedunitcostofacourthearingforthejudgeofUS$12.03
perDTCparticipant.
132
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
ThisexerciseisrepeatedforeveryDTCteammemberthatparticipatesinstatusreview
hearings,andthefully loadedcostsforallteammembersaresummedtodeterminethe
totalunitcostpercourthearingperparticipant.Finally,tofindthecostofstatusreview
hearingsperparticipantthroughoutthelengthoftheprogram,thisunitcostismultipliedby
thenumberofhearingsattendedbyeachparticipant(seeTable4).
Amajoradvantageofthisapproachisthatitoffersathoroughassessmentofboththe
investmentcostsofDTCforeachtransactionbyeachparticipatingagencyororganizationin
thesystemaswellastheavoidedcostsduetothebenefitoftheoutcomestoeachagency
ororganization.Forexample,Tables4through6belowdemonstratethekindsofcostsand
savingsDTCshaveshownusingtheTICAapproach.
Investment costs
Table4providestheunitcostpertransaction(e.g.,eachcourtappearanceforaDTC
participantcostsUS$83.62)andtheaveragenumberofeachtypeoftransactioneventper
DTCparticipant(e.g.,onaverage,participantsinthisprogramhadalittleover23appear-
ancesbeforethejudge).Thetotalcostpertransactionistheunitcost(US$83.62)timesthe
numberoftransactionevents(23courtappearances),whichequalsatotaltransactioncost
forcourtappearancesofUS$1,945perparticipant.ThecostoftheDTCprogramisthenthe
sumofthetotalcostsofalltransactions.
133
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Table 4. Program Costs per Participant
Transaction Unit costAvg. # of transactions/events
per DC participants
Perperson
Avg. Cost per DC participant
Perperson
Drug Treatment Court Appearances US$83.62 US$23.26 US$1,945
Case Management Days 5.83 494.37 2,882
Outpatient and Residential Treatment Months
746.74 12.00 8,961
UA Drug Tests 7.00 53.28 373
Jail Sanction Days 46.85 12.48 585
Jail Days While Awaiting Residential Treatment4 46.85 20.42 957
TOTAL US$15,703
Theunitcostmultipliedbythenumberofeventsperpersonresultsinthecostforeach
transaction during the course of the program. In this example,when the costs of the
transactionsaresummed,theresultisatotalprogramcostperparticipantofUS$15,703.
Similarly,thesesamesteps(TICASteps1-6)canbeperformedforacomparisongroup.
(Thisisacomparisongroupthatwouldbeusedintheoutcomeevaluationandthisiswhy
acost-benefitanalysisrequiresavalidoutcomestudy).Forexample,usingprocessevalua-
tionmethods,anevaluatorcandeterminetheprocessthatoccursforaDTC-eligiblecourt
caseforindividualswhodidnotparticipateinDTC,determinethetransactionsthatoccur
throughthatprocess,whatagenciesareinvolved,etc.Then,thedifferencebetweenthe
costoftraditionalcaseprocessingandthecostofparticipatinginDTC(e.g.,thecostofDTC
minusthecostoftraditionalcaseprocessing)resultsinthecostofrunningaDTCoverand
4 Whenaresidentialtreatmentbedisnotavailable,thisprogram’sparticipantsaresenttojailuntilabedopensup.
134
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
above the cost of traditional case processing (i.e., the cost the systemwould havepaid
forthatcaseanyway,iftherehadbeennoDTC).Itisimportanttonotethatthisrequires
goodadministrativedatacollectiononthecomparisongroupaswellasonDTCparticipants,
whichdoesnotalwaysoccur.
Outcome costs
Todeterminetheoutcomecostsandsavings,Steps1through6oftheTICAapproach
areperformedforeachoutcometransactionof interest(e.g.,thecostofanewarrest,a
newcourtcase,daysspent incarcerated). Thecostperoutcometransaction is summed
acrossalltransactionsforboththeDTCparticipantsandforamatchedcomparisongroup.
Forexample,Table5showstheoutcometransactionsmeasuredinthesameDTCasTable
4.Theoutcometransactionsincludedinthestudywerearrests,courtcases,daysonpro-
bation,daysinjail,anddaysinprison.Theseoutcomesweremeasuredduringathree-year
periodfromthetimeofDTCentryforeachparticipantinthestudy.Table5providesthe
unitcostpertransaction(e.g.,thecostofasinglearrestisUS$129.47),andthetotalcostof
eachtransactionperDTCparticipantsandpercomparisongroupmember(e.g.,theaverage
numberofarrestsperpersonin3yearswasmultipliedbytheunitcostofUS$129.47for
atotalcostofUS$66perparticipant,revealingthattherewasanaverageoflessthanone
rearrestperparticipantduringthatperiod).Theright-handcolumnprovidesthedifference
incostbetweenthetwogroups(thecostofthecomparisongroupminusthecostofthe
DTCgroup).Whenthisdifferenceispositive,thisisconsideredacost“savings.”Thesumof
thesavingsacrosstransactionsisthetotalsavingsperparticipantduetoparticipationinthe
DTCprogram.InTable5,thetotalsavingsisshowntobeUS$23,957perDTCparticipant.
135
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Table 5. Recidivism (Outcome)Costs per Participant Over 3 Years
Transaction Unit cost
Drug court participants
Per person
(n=103)
Comparison group
Per person
(n=358)
Cost difference
Comparison minus drug court
Rearrests US$129.47 US$66 US$388 US$322
Criminal District Court Cases 1,448.02 738 4,344 3,606
Probation Days 3.15 559 1,053 494
Jail Days 46.85 1,849 5,100 3,251
Prison Days 49.93 4,038 20,322 16,284
TOTAL US$7,250 US431,207 US$23,957
Cost-benefit ratio
This table indicates that the cost for those processed through the DTC per person
wasUS$23,957 less than standard court processing. It also allows calculation of a
cost-to-benefitratio.InthecaseofthisDTC,whenthecostoftheprograminvestmentper
participant(US$15,703)isassessedagainsttheoutcomesavingsperparticipant(US$23,957),
(i.e.US$23,957dividedbyUS$15,703)thecost-benefitratiocomesto1:1.5.Thatis,for
everydollarspentontheprogram,thereisareturnofUS$1.50.
Inthisexample,thecost-benefitratiomayactuallybeanunderestimate,sincethecost
oftraditionalcourtforthecomparisongroupwasnotincludedinthisexample.Whenthein-
vestmentinatraditionalcourtcaseisincluded,thetotalinvestmentcostusedforthisratio
wouldthenbeforeverydollarspentover and above thecostoftraditionalcourt.So,ifthe
costoftraditionalcourtwasUS$10,000percase,the“net”investmentwouldbeUS$15,703
minusUS$10,000foraninvestmentofUS$5,703perparticipantintheprogramoverand
136
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
abovethecostoftraditionalcourt.Thecost-benefitratiowouldthenbe1:4.2—thatis,for
everydollarspentover and above thecostoftraditionalcourt,thereisareturnofUS$4.20.
Inaddition,becausetheTICAapproachmeasurestheresources(e.g.,eachstaffmem-
ber)contributedbyeachagency involved intheprogram, it ispossibletodeterminethe
investmentandsavingsforeachagency.Table6showsanexampleofthecostorsavings
incurredbyeachagencyduetothecostofoutcomesforDTCparticipantscomparedtoa
traditionalcourtsystem.
Table 6. Recidivism (Outcome) Costs per Participant by Agency over three years
AgencyDrug Court
outcome costs per participant
Comparison group outcome costs per individual
Difference/ savings
per individual
Criminal District Court US$161 US$375 US$214
District Attorney 194 453 259
Appointed Defense Attorney 383 895 512
Department of Criminal Justice 4,000 4,240 240
Law Enforcement 1,915 5,249 3,334
Adult Probation 559 1,037 478
TOTAL US$7,212 US$12,249 US$5,037
Table6demonstratesthateveryagencyinvolvedintheprogramrealizessomesavings
duetoanoffender’sparticipationintheDTCprogram.Inthisexample,lawenforcement
realizesthemostsavingsduetodecreasedincarcerationcosts.
137
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Conclusion – Cost Analyses
Thetypeofcostanalysisapproachdependstoalargeextentonthedataavailableand
thepolicyquestionsbeingaddressed,aswellasthebudgetavailabletoperformthecost
analysis.
�Program-cost analysiscanbeusefulwhenaprogramhasfewevaluation
resourcesandneedsonlyaminimalassessmentofprogramcosts.
�Cost-allocation analysis isvitalinsituationssuchasDTCswheremultiple
agenciescontributeresourcesandservices.Nonetheless,whilecost-allocation
analysisallowsyoutocalculateprogramandunitcosts,itcannotdetermine
whethertheprogramisanefficientuseofresources.
�Cost-effectiveness analysis isusefulwhenonehasclearpolicygoalsto
producethesameoutcomes,andoutcomedataforalltheoptions,butdoesnot
needtoplaceamonetaryvalueontheoutcomes.
�Cost-benefit analysis (suchastheTICAapproach)isusefulwhenoneneeds
todeterminewhetheraprogramactuallysavesmoneyandwantstocompare
theinvestmentcoststothesavings(oravoidedcosts)thatresultfrompositive
outcomes.
138
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Apart fromrelativelysimpletypesofanalyses, it isoftenadvisabletorelyonexpe-
riencedevaluators toperformappropriatestatistical tests,avoidcommonanalyticmis-
takes, and interpret thefindings. If thedatahavebeen recordedproperly and stored
intherightformat, itshouldnottakelongforanevaluatortoanalyzetheresults. The
bigquestion,however, ishowDTCstaffmembers—whohave typicallybeen trainedas
lawyers,clinicians,orcriminal justiceprofessionals—can identifycompetentevaluation
consultants.
Thefirststep inselectingtherightteamofevaluators istoreviewtheirpriorevalua-
tionreports,especiallyanyevaluationsrelatedtoDTCsorotherproblem-solvingcourtpro-
grams.FamiliaritywithhowDTCsoperateisoftenessentialforknowingwhatperformance
indicators,mediator variables,moderator variables andoutcomes toexamine. It is also
importantforknowinghowtointerprettheresultsanddescribethefindingsinapractical
andunderstandablewaytoDTCstaffandtreatmentprofessionals.
When reviewingpriorevaluation reports, it isuseful to consider thequestions listed
below. Thesearenotpresented inorderof importance,and somequestionsmightnot
berelevantforallevaluations. Forexample, if theevaluatorsdidnotcontrastoutcomes
againstthoseofacomparisonsample,thentheremightbenoneedtocontrolforbaseline
differencesbetweenthegroups.However,undersuchcircumstancestheevaluatorswould
beextremelylimitedintermsoftheconclusionstheycoulddrawfromthestudy.Lacking
acomparisonsample,itwouldnotbepossibletoconcludehowwelltheDTCperformedin
comparisontootherprograms.Iftheevaluatorswentbeyondthedatainmakingsuchan
interpretation,thatmightbodepoorlyfortheircompetenceasresearchers.
�Didtheevaluationteamanalyzeimportantandrelevantperformanceindicators?
�Didtheyproperlyaccountformissingdata?
SELECTING STATISTICAL CONSULTANTS
139
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
�Didtheyexamineoutcomesaccordingtoparticipants’riskfactorsormoderator
variables?
�Didtheyidentifythebesttargetpopulationfortheprogrambyexamininginteraction
effectsratherthanmerelyexaminingsignificantpredictorsofoutcomes?
�Didtheyruleoutpotentialconfoundingvariablesbeforeidentifyingdisparate
impactsbygender,race,ethnicorculturalgroups?
�Weretheanalysesconductedonanintent-to-treatbasis?
�Didtheyselectanunbiasedcomparisonsample?
�Didtheycheckforbaselinedifferencesbetweenthegroupsatentryand
statisticallycontrolforthemwherenecessary?
�Didtheyuseareasonablefollow-upwindowgiventheoutcomevariablebeing
measured?Forexample,didtheyexaminerecidivismoveralongenoughtime
periodfornewarrestsorconvictionstobeexpectedtooccur?
�Didtheystarttheclockrunningatanequivalenttimeforallparticipants?For
example,didtheystartitrunningatthepointofentryintoeithertheDTCoran
alternativedispositionsuchasprobation?
�Didtheystatisticallycontrol,whererelevant,fordaysatriskanddaysatlibertyin
thecommunity?
�Didtheytransformthedataorusenonparametrictestsforinfrequentevents?
�Weretheinterpretationsofthefindingsdefensiblegiventhelimitationsofthe
researchdesign?
Finally,themostimportantquestionofallis:Did the evaluators’ interpretations of the
findings make sense, and did they point to concrete actions the DTC could take to improve
its performance and enhance outcomes? Theprimarygoalofaprogramevaluation isto
improveoutcomesandcontinuouslymoveforward.Ifanevaluationreportsimplystates
whetherornottheprogramworkedinthepastbutdoessuggestshowitcouldworkbetter
inthefuture,itisofverylimitedutility.
140
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Thereaderhasundoubtedlyfoundthepreviousdiscussionsconcerningsuchtopicsas
participantcharacteristics,performance indicators,attitudinalvariables,mediationanaly-
ses, and cost analyses extremelydaunting, especiallywhen issuesof date-stamping and
theproblemofthemissingdenominatorareconsidered.Theunfortunaterealityisthatfew
DTCprogramscollectthenecessaryinformationforavalidevaluationinareliableorusable
manner.Whenprogramsdocapturetheappropriatedataelements,theyrarelyaccountfor
missedappointments,connecteventstothedatesonwhichtheyoccurredorweresup-
posedtohaveoccurred,orenterthedataintoanelectronicformatthatpermitsstatistical
analyses. Inmany instances,evaluatorsare required toextract information fromwritten
recordsorunwieldystatisticalspreadsheetswithlittlerecourseforreconcilinginconsisten-
ciesinthedataoraccountingformissingentries.Theresultisthatmanyevaluationsare
completedmonthsoryearsafterthefact—whentheresultsmaynolongerreflectwhatis
occurringintheprogram—andtheremaybesomanygapsorcaveatsinthedatathatthe
conclusionswhichcanbedrawnaretentativeatbest.
Date-stampingcanbeparticularlyunwieldyifevaluatorsusetraditionalformsofsta-
tisticalspreadsheets,suchasExcel.Mostspreadsheetsaretwo-dimensional,meaningthey
arecomprisedofrowsandcolumns.If,forexample,participants’namesarelistedbyrows
andtheservicesthatareprovidedarelistedbycolumns,howdoesoneaccountforthedates
onwhichtheservicesweredelivered?Itisusuallynecessarytocreateseparatecolumns
for each serviceon eachdate, resulting in hundreds of columns. Fortunately, new-
ergenerationsofdata-entrysystemsmayautomaticallydate-stampentries.Forexample,
data-entryscreensmightappear likeaprofessional’sappointmentcalendar. Information
canbeenteredontheappropriatedayinthecalendarandisautomaticallydate-stamped
foranalysis.DTCprogramsareencouragedtousemanagementinformationsystems(MISs)
MANAGEMENT INFORMATION SYSTEMS
141
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
thatautomaticallydate-stampentries.
Itcannotbestressedenoughthatthesecrettovalid,timely,andcost-efficientprogram
evaluationsdependsontheselectionofasuitableMIS.ThecostofpurchasingausefulMIS
willbeoffsetmanytimesoverbyprovidinggreaterefficienciesinoperationsandyielding
thetypeofperformancefeedbackthatisnecessarytocontinuouslyimproveandfine-tune
one’sDTCprogram.
SomeoftheolderandlesssophisticatedMISscanbeobtainedfreeofcharge.Forex-
ample,the“BuffaloSystem”—sonamedbecauseitwasdevelopedinaDTCinBuffalo,New
York,intheUnitedStates—canbeobtainedforfreebycontactingtheNADCPintheUnited
States. NewerMISsmustbepurchased,but theyare alsomore likely tobeweb-based
andcan,therefore,beaccessedsimultaneouslybymultipleprograms.Havingmultiplepro-
gramsusethesameMIS,eachwithitsownsecuredandencryptedaccess,canspreadthe
costsoftheMISacrossmultipleusersorcountries.TheAppendixprovidesinformationon
howtoaccesstheBuffaloSystemandotherproprietaryMISproducts.
Recommended MIS Features
NewergenerationsofMISsarecapableofstreamliningprogramevaluationsandre-
ducingtheburdenonstaffmembersandparticipants.Whereitisfeasible,itisrecommended
thatDTCprogramsselectMISswiththefollowingcharacteristics:
Web-Enabled. Staffmembers inDTCsareusuallyemployedbydifferentagencies
such as the court, probation department or treatment program, and theymay have
officesinseverallocations.Theymayalsoberequiredtovisitparticipantsinjail,residential
treatmentfacilitiesorattheirhomestoconductassessmentsordeliverservices.Thisre-
quiresstaffmemberstohaveaccesstotheMISwhiletheyaretravelingandfrommultiple
locations.IftheMISisweb-enabled,itcanbeaccessedfromanylocationthathasinternet
access,includingpersonalpalmdevicesorlaptopcomputers.
Security Protected. The data should be stored and transferred using indus-
142
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
try-standard128-bitSSLencryptionorbetter.Itisalsonecessarytoregulatestaffmember
accesstotheinformationbasedontheirjoblevelsanddescriptions.Onlyundercarefully
specifiedcircumstancesshouldonestaffmemberbeabletoalterdatathatwasenteredby
anotherstaffmember.Forexample,thejudgeshouldnotbeabletoalterinformationthat
wasenteredbythetreatmentprovider.Thejudgemight,however,haveread-onlyaccessto
certaininformationenteredbythetreatmentprovider,suchasaparticipant’sattendance
rateatcounselingsessions. Theauthorizedlevelofaccessforeachstaffmembershould
bespecifiedbyanMISAdministratorandcorrespondtothatstaffmember’spasswordand
username.
Less is More.Staffmembersshouldonlyberequiredtoviewdata-entryscreensthat
arerelevanttotheirjobs.Forexample,atreatmentproviderordinarilyshouldnotbefaced
withdata-entryscreensrelatingtoprobationcontactsorcourthearings.Thetreatment
providermight be permitted to view summary reports on probation contacts or court
hearingsbutshouldnotberequiredtoscrollthroughthatmaterial if it isnotdirectly
relevanttothetreatmentprovider’sduties.
Need to Know.DTCprofessionalshavearighttoknowwhytheyarebeingasked
tocollect informationandshouldavoidduplicationofeffort. Ifthere isnoobviousand
empiricallydefensiblereasonwhyparticular informationisbeingcollected,thenperhaps
itisunnecessarytocollectthatinformation.Redundanciesshouldalsobeeliminated.For
example,onceaparticipant’sageisenteredinonedata-entryscreen,itshouldautomatically
beenteredor“cross-walked”intotherespectivefieldsinotherscreens.
Minimal Burden. Itshouldordinarilyrequirenomorethantwotothreeminutes
toenteralldataelements thatare required foragivenparticipantduringagivenweek.
The data-entry screens should also be intuitive and easy to use.Most professionals are
accustomedtousingtheInternettopaytheirbills,purchasegoodsandservices,orgather
information.Agooddealofefforthasgoneintodevelopingcommercialwebsitesthatare
intuitiveandsimpletouse.TheMISfortheDTCshouldhaveasimilarlyintuitivedesign.The
143
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
morecomplex,confusing,and/ortime-consumingtheeffortrequiredtoenterdataproves
tobe,thelesslikelystaffwillenterdatacorrectlyandappropriately.
Automatic Date Stamping. Aswasmentionedpreviously,somedata-entryscreens
appearlikeaprofessional’sappointmentcalendar.Informationisenteredontheappro-
priatedateinthecalendarandisautomaticallydate-stampedforanalysis.
Reminders/Ticklers for Missing Data. ModernMISsmayroutinelyremindor“tick-
le”userstoentermissingorincompletedata.Forexample,ifaprobationofficeraccesses
theMIStoenterattendanceinformationfortheweekofOctober15,buthasnotentered
theattendanceinformationforthepreviousweekofOctober8,thesystemshouldalert
theprobationofficeraboutthemissingdata.Suchremindersorticklersmakeitlesslikely
thatsomuchtimewillpassthatitbecomesdifficulttoreliablyreconstructtheeventsfrom
memoryorhandwrittenrecords.
Flexible Input Screens.Itisoftennecessarytoaddnewitems,deleteor“grayout”
unwanteditems,orchangethewordingofitems.Forexample,aDTCmightcontractwith
anewtreatmentprogramtoprovidementalhealth services.Thismight requireanew
data-entry screen to be addedwith items pertaining to participants’ attendance at
mentalhealthcounselingsessions.Itshouldbepossibletoaddnewitems,deleteitems,
andchangethewordingofitemsinnomorethananhour.
Hidden Database. DTC professionals should not be required to wade through
columnsofnumericdata. Thedatashouldbestoredbehindthescenesinananalyzable
formatthatpermitsimmediatestatisticalanalysisandqueries,andallowseasyandquick
visualizationthroughaninterfacethatintuitivelymakessensetotheuser.Thedatashould
alsobestoredinspecifiedfieldsthatcanbeselectedreadilyandenteredinstatisticalmodels.
Longitudinal Database.Undernocircumstanceshouldthesystemoverwriteprevi-
ousdata.Forexample,ifaparticipantwasunemployedwhenheorsheenteredtheDTC,
andthenobtainedajobafewmonthslater,theparticipant’semploymentstatusshouldnot
beoverwritten.Doingsowouldmakeitdifficultorimpossibletodeterminelaterwhether
144
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
theparticipantobtainedajobwhileheorshewasintheDTC,orwhethertheparticipant
enteredtheprogramwithajobtobeginwith.Moreover,itwouldbedifficulttodetermine
howlongittooktheparticipanttofindajob.Thedatabaseshouldbearrangedlongitudi-
nally,meaningthatneweventsareappendedalongsideolderevents. If thedatabase is
arrangedlongitudinallyandtheentriesaredate-stamped,itwillbepossibletodetermine
whetherparticipants’employmentstatuschangedovertimeandhowlongittookforthe
changestooccur.
Descriptive Reports. ThereisnolimittotheresearchquestionsthatDTCstaffmem-
bersorstakeholdersmightask.However,therearealimitednumberofbasicdescriptive
analyses thatmostprogramswillwant to conduct. Forexample, virtuallyeveryDTCwill
want to know its graduation rate, re-arrest rate, attendance rate in counseling sessions,
andproportionofalcohol-anddrug-positivetests.Inaddition,manyprogramswillwant
toknowwhethertheseoutcomesdifferbetweenmalesandfemalesorbetweenracialor
ethnicgroups.Itispossibletowritestatisticalsyntaxsothatthesecommonanalysescan
beperformedatthepushofabutton.Formoresophisticatedresearchquestions,itmaybe
necessarytosendthedatatoastatisticianforanalyses;however,formanyroutineques-
tionsitshouldbepossibletogeneratereportsnearlyinstantly.Inlinewiththeadagethat
“apicture iswortha thousandwords,” the reports shouldalso includeeasy-to-interpret
graphs,tablesandpiecharts.
Continuous Performance Feedback.ThemostimportantreasonforevaluatingaDTC
istoimproveitsoperationsandoutcomes.Ifaninterventionisnotworking,itisimportant
tolearnthisfactquicklywhilethereisstilltimetoadjusttheparticipant’streatmentplan.
ThisrequirestheMIStocontinuouslymonitorthedataandissueautomatedalertstostaff
memberswheneveraparticularactionmightbecalledfor.Forexample,thesystemmight
automaticallyalertstaffwheneveraparticipantmissedtwocounselingsessionsinarowor
providedtwodrug-positiveurinespecimens.Thiswouldpromptthestafftodiscussthepar-
ticipantatthenextstaffmeetinganddecideonasuitableresponse.Researchindicatesthat
145
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
generatingautomatedfeedbackinthismannercansubstantiallyimproveaDTC’soperations
andparticipantoutcomes(Marloweetal.,2008b;Rempel,2007).
User Accountability Reports. Oneofthebiggestthreatstoavalidprogramevalua-
tionisthefailureofmanystaffmemberstoenterdatainatimelymanner.TheMISshould
generateuseraccountabilityreportswhichindicatehowlongittookforthedatatobeen-
teredaftertherelevanteventsoccurred.Forexample,supervisorsshouldbemadeaware
thatacounselorhasbeenenteringattendanceinformationanaverageoffourweeksafter
thesessionswereheld,orthatdrugtestresultsarebeinglistedas“pending”bythelabfor
anaverageofmorethantwoweeks.Thiswouldenablethesupervisortointervenerapidly
toimprovethequalityoftheevaluationandtheeffectivenessoftheDTC.
Attheriskofrepetition, if an MIS is easy to use, collects the essential performance
indicators, and stores the data in an analyzable format, the likelihood of completing a
successful and valid evaluation is high.AndthecostofpurchasingsuchanMISwillbe
offsetmanytimesoverbythefactthatthereisnolongeraneedforevaluatorstospend
hundredsofhoursattempting(oftenwithlimitedsuccess)toextractusableinformation
from written charts or spreadsheets. If the necessary information can be handed to
researchersinaproperformat,usefulfindingsshouldbeobtainableinashortperiodof
timeandataminimalcost.
Ofcourse,ifaDTCdoesnothaveadequateresourcestopurchaseanMISwiththeabove
characteristics,anyeffortatevaluation isbetter thannoeffort.TheDTCcoulduse the
BuffaloSystemdescribedearlier,whichisfreeofcharge,ordevelopwrittenchecklistsor
dataspreadsheetstocollectinformation.EvenanExcelspreadsheet,ifconfiguredandused
correctly,canhandleavarietyofdatacollectiontasks.Gettingstartedevaluatingone’sDTC
isthehardeststep.Onceevaluationactivitiesarebegun,DTCstaffmembersareapttoiden-
tifybetterwaysovertimetocollectusefulinformationandexaminetheservicesprovided
andimpactsproducedbytheirprogram.
146
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Much of the information that is required to evaluate a DTC is likely to be in the
possessionoftheprogram.Forexample,dataonparticipants’drugtestsresults,treatment
attendance informationandgraduationratesshouldbereadilyavailable incourtrecords
orparticipants’treatmentrecords. However,toconstructasuitablecomparisonsample,
anevaluatorwillalsoneedinformationaboutsimilarindividualswhowerenotenrolledin
theDTC,suchasprobationers.Forexample,theevaluatorwillneedinformationaboutrisk
factorsoftheprobationers,includingtheircriminalrecordsortreatmenthistories.Without
thisinformation,itwillnotbepossibletoconstructavalidmatched-comparisonsampleor
toperformavalidpropensity-scoreanalysis.
Evaluatorswillalsoneeddataonnewarrests,newconvictionsand incarcerationsfor
both theDTC and comparison samples. This recidivismdatawill often not be in the
possessionof theDTCprogram,but rathermaybe collectedand storedbyanother
governmentagency,suchasthepolice,departmentofcorrections,oradministrative
officeofthecourtsystem.Thiswillrequireevaluatorstoobtaininformationfromdata-
basesthatmaynotbewithintheircontrol,whichtheymaynotbeauthorizedtoaccess,
andwhichmaybeunfamiliartothem.
Oftentheremaybetechnologicalorattitudinalbarrierstoaccessingotherdatabases.
Lawsandregulationsconcerningthesharingofcriminalrecordsmaybecontradictoryor
difficultto interpret insomecountries.This can lead toparalysison thepartof some
agencies for fear of committing a breach of privacy or other legal protection. For
technologicalreasons,informationoftencannotbetransferredreadilyfromonedatabase
to another due to inconsistent data definitions, incompatible hardware or software,
or proprietary technologies that cannot interface with one another. Important terms
such as risk assessment may also have different meanings for agencies with different
ACCESSING ADMINISTRATIVE DATABASES
147
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
cultures. Toaprobationofficer,ariskassessmentmayrefertothelikelihoodofcriminal
reoffending,whereastoatreatmentprofessionalitmightrefertothelikelihoodofleaving
treatment prematurely or relapsing to substance use or mental illness. Agencies may
beunderstandably reluctant to share this informationoutofa concern that itmightbe
misinterpreted.
Thesebarriers,althoughdaunting,arenotinsurmountable.Evaluatorshavefoundeffec-
tiveandethicalwaystoaccesscriticallyimportantinformationfromgovernmentdatabases
withoutviolatinglegalrequirements,ethicalmandatesoreffectiveclinicalpractice(Treat-
mentResearchInstitute,2012).
Reaching Consensus
Thefirststeptosharingsensitiveinformationisreachingaconsensusaboutwhatin-
formationisavailable,whichdataelementsmaybeshared,andunderwhatconditionsthe
datamaybeshared.Representativesfromtheinterestedagenciesmustcometoamutual
understandingaboutthenatureofthedatathatiscollected,thedefinitionsandlimitations
ofthedata,andhowthedataisstoredelectronically.Fromthere,theagenciesmustex-
ecutedata-sharingagreementsandotherstandardsthatsatisfythelegalandethicalcon-
cernsofallinterestedparties.
Oneresourcethatmayassistagencies toreachthisconsensus is theNational Infor-
mationExchangeModel(NIEM).5TheNIEMisconsideredthestandardforinformationex-
changeamongcriminaljusticeagenciesintheUnitedStates.Ithasalsobeenappliedinter-
nationallyandisavailabletoanypublic,private,ornon-profitagency.TheNIEMoffersbasic
data-sharingtools,standardsandprocedurestohelpagenciessharesensitiveinformation.
Forexample, theNIEMcanprovidesamplesofdatadictionariescontainingagreed-upon
definitionsandrecommendedformatsforcriminaljusticedata.Thedatadictionariescan
help toensure that informationcarriesa consistentmeaningacrossagencies.TheNIEM
5 https://www.niem.gov/aboutniem/Pages/niem.aspx
148
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
mayalsoprovidesampledata-sharingagreementsandstandardsforthetransferofdata.
EvaluatorsandpolicymakersfromOASmemberstatescanusethesampledatadictio-
nariesanddata-sharingagreementsasstartingpointsforreachingagreementabouthow
datamightbedefined,formattedorsharedintheirownDTCprogramevaluations.Legal
andethicalmandateswill,ofcourse,differbetweennations,buttheessentialprinciplesand
practicesofeffectivedata-sharingarrangementshavecommonelementsthatarelikelyto
applytoawiderangeofevaluationcontexts.
After agencies have reached an agreement about how data should be defined and
shared,theNIEMcan,ifdesired,developcomputersyntaxthatallowsforthestandardiza-
tionandexchangeofthedatabetweenMISs.Thisdoesnotrequiretheagenciestochange
theirownMISordata-storagemethods. Rather,usingwhat issometimesreferredtoas
middlewaretheNIEMcandevelopelectronicconduitsforthetransferofinformationbe-
tweendatabases.Thisprocessislikelytobemorecomplicatedorcostlythaniswarranted
formanyDTCpilotprogramsinOASmemberstates,butitmaybecomemoreusefuland
relevantasdozensofDTCsaredevelopedinmultiplenationsovertime.
Linking Records
ItisessentialforrecordstocorrespondtothesameindividualacrossdifferentMISsor
databases. Ideally, unique identifying information shouldbeused to link records across
agenciesorsystems.Forexample,somecountriessuchasBarbadoshaveanationalregistry
numberthatcorrespondstoeachcitizen.
Alternatively,identifyingnumbersmaybeassignedtoindividualswhentheycontactthe
criminaljusticesystem.Someofthesenumbersmaycorrespondtotheperson,andthus
thesamenumbermaybeassignedtodifferentcasesoroffensesinvolvingthesameindivid-
ual.Othernumbersmaycorrespondtothecase,andthusthesamenumberwouldfollow
149
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
thepersonfromarrestthroughsentencingonthesameoffensebutwouldnotbeassigned
tosubsequentoffenses. Dependingonthecountry, these identifyingnumbersmightbe
usefulfor linkingcasesacrossagencieswithinthecriminal justicesystem,suchas linking
casesbetweenthecourtandprobationdepartment.Often,however,theycannotbeused
tolinkcriminaljusticerecordswithrecordsfromothertreatmentorsocialserviceagencies.
Ifuniqueidentifyinginformationisnotavailable,therearealternativeproceduresthat
evaluatorscanusetolinkrecordswithvaryingdegreesofcertaintythattherecordsrepre-
sentthesameindividual.Thereareseveralprobabilistic record linkagemethodswhichare
availablefreelyinthepublicdomain,suchas“LinkPlus”or“TheLinkKing”(Campbelletal.,
2008).Theseprobabilisticmethodslinkcasesfromdifferentdatasetsusingacombination
ofavailabledataelements,suchasbirthdatesandfirstandlastnames. Thelinkscanbe
accomplishedtoaspecifieddegreeofconfidence;forexample,with80percentconfidence
thatthecasesrepresentthesameperson.Thelevelofconfidencethatcanbeplacedinthe
linksisbasedonsuchfactorsasthenumberofpersonsinthedatabaseandthenumberof
variablesthatareavailabletocreatethematches.
Ofcourse,therewillalwaysbesomedegreeoferrorwhenusingprobabilisticrecord
linkages.However,themagnitudeoftheerrorshouldbedistributedevenlybetweenthe
DTCandcomparisongroups.Therefore,itshouldnotseriouslybiasthequestionofwhich
grouphadlowerrecidivismrates.However,estimationsofthemagnitudeoftheeffectof
theDTConrecidivismcouldbeoffbyseveralpercentagepoints.
Ifallelsefails,bearinmindthateventhemostrestrictivelawsandethicalmandateswill
usuallypermitde-identifieddatatobesharedortransferred.Manyethicsreviewcommittees
permitevaluatorstoassignanonymousresearchidentificationnumberstocases.Afterthecas-
eshavebeenlinkedtorecordsfromadministrativedatabases,theevaluatoragreestodestroy
thelistsconnectingthesubjects’namestotheiridentificationnumbers.Alternatively,theagen-
cythatisstoringtheadministrativedata,suchasthepoliceorprobationdepartment,mightbe
responsiblefordestroyingthelinkstosubjects’namesafterthedatahasbeentransferred.The
150
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
datamaythenbeanalyzedatthegrouplevelintermsofwhetheraparticularcasebelongstothe
DTCorcomparisongroupbutcannotbetracedbacktoanindividualparticipant.
Transferring DataGiventheseriousresourcechallengesthatconfrontmostcountries,data-sharingproce-
duresmuststartwithrelativelysimpleandlow-costmethodswhicharereadilyavailablefor
mostprograms.Forexample,Direct Secure Messaging(DSM)allowsprogramstotransmit
encryptedinformationtoknownandtrustedrecipientsovertheInternet.Similarly,“cloud”
technologymaybeattractivetosmallandunder-resourcedprogramsbecauseitdoesnot
requireMISdepartmentstohavesophisticatedhardware,networkingcapabilities,ormain-
tenancestaff.Finally,portableexternaldrivescanholdsubstantialamountsofinformation
andcanbeusedtophysicallytransferinformationbetweenlocations.
Ethics and Legal Training
AllprofessionalswhoareinvolvedintheevaluationofaDTCorwhohandlesensitive
datashouldfirstcompleteeducationalcoursesonethicalandlegalprotectionsinresearch.
Becauseapplicablelawsmaybedifferentindifferentnations,evaluatorsmightneedto
identifytrainingprogramswithintheirowncountries.Web-basedcoursesonresearchethics
areavailableinseveralcountries.Forexample,theU.S.NationalInstitutesofHealthprovide
web-basedtrainingsonlegalandethicalprotectionsforresearchparticipants.6 Comparable
web-basedprogramsorwritteneducationalmaterialsmaybeavailableinothercountries
aswell.
6http://phrp.nihtraining.com/users/login.php
151
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
MoreresearchhasbeenpublishedgloballyontheeffectsofDTCsthanonvirtually
allothercriminaljusticeandsubstanceusetreatmentprogramscombined.Theresultsof
thisintenselevelofscrutinyhavebeenalmostuniversallypositive.Inaneraofglobaleco-
nomicuncertaintyandshrinkingpublicdollars,DTCsarebeingfundedandexpandedata
steadilyincreasingrate.Therearenowmorethan3,000DTCsandothertreatmentcourts
intheUnitedStates,andnewDTCprogramsarebeingdevelopedorplannedinmorethan
30othercountries.
Theonlyway toexplain this success is tonote that the researchhaspaidoff.By
provingthatDTCsreducecrime,savelives,andsavemoney,evaluatorshavejustifiedthe
programs’worthtoaskepticalaudienceofpolicymakersandthepublicatlarge.Political
andsocialbeliefsmaychangewiththetimesandpetprojectsmayfadeinandoutoffavor,
butscientificproofisageless.Convincinglywordedargumentsandstronglyfeltemotions
mayswaypublicopiniontemporarily,buttheyhaveadifficulttimestandinguptoscientifi-
callyreliableandvalidcontradictoryevidence.Inthefinalanalysis,thefactshaveawayof
winningout.
Itwould be nice to say thatDTCs planned these events from thebeginning, but
theydidnot.ManyoftheearlieststudiesinDTCsviolatedfundamentalprinciplesofgood
science.Forexample,theyexaminedoutcomesonlyforDTCgraduatesorselectedunfairly
disadvantagedcomparisonsamples.IttooknearlytwodecadesfortheDTCfieldtogetit
rightandconductthepropertypesofstudiesthatwererequiredtoprovethatDTCswork.
Thereisnoneedforothercountriestorelivethesetime-consuminglessonsthatDTC
pioneershadto learnthroughpainstakingeffort. OtherOASmemberstatescanbenefit
frommorethanaquartercenturyofexperienceinAustralia,Canada,theUnitedStatesand
afewothernationstoconductproperresearchfromtheoutset.
CLOSING COMMENTS
152
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
ThisisnottosuggestthatothercountriesshoulddevelopandadministertheirDTCs
inthesamemannerasAustralia,CanadaortheUnitedStates.Differentcultureswillfind
differentwaystomeettheneedsoftheirowncitizens.Butthelawsofscienceareuniver-
sal.Therearecorrectwaysandincorrectwaystoevaluateaprogram.Performingflawed
evaluationsdoesnotmerelywastetime,resources,andopportunities.Itcanalsoresultin
lostcredibility,bymakingDTCsseemasiftheyareattemptingtocutcornersormanipulate
thetruth.Itisnotenoughtohaveaworthyconceptifonecannotproveitsworthinafair
andimpartialtest.
ToprofessionalsworkinginDTCprograms,rememberthis:Itiseasytobeagood
salespersonifyouhaveasuperiorproduct.AndDTCsare,indeed,asuperiorproduct.Do
nothesitatetostudythemcarefullyandfollowtheresultswherevertheymayleadyou.The
factsareonyoursideandgoodscientificmethodologywillenableyoutoproveit.
153
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Andrews,D.A.,&Bonta,J.(2010).The Psychology of Criminal Conduct (5thed.)NewProvidence,NJ:LexisNexis.
Ardito,R.B.,&Rabellino,D.(2011).Therapeuticallianceandoutcomeofpsychotherapy:Historicalexcursus,measurements,andprospectsforresearch.Frontiers in Psychology, 2,11.
Banks,D.,&Gottfredson,D.C.(2003).Theeffectsofdrugtreatmentandsupervisionontimetorearrestamongdrugtreatmentcourtparticipants.Journal of Drug Issues, 33,385-412.
Baron,R.M.,&Kenny,D.A.(1986).Themoderator-mediatorvariabledistinctioninsocialpsychologicalresearch–Conceptual,strategic,andstatisticalconsiderations. Journal of Personality & Social Psychology,51,1173–1182.
Belenko,S.,Patapis,N.,&French,M.T.(2005).Economic benefits of drug treatment: A critical review of the evidence for policy makers. MissouriFoundationforHealth,NationalRuralAlcohol&DrugAbuseNetwork.
Bhati,A.S.,Roman,J.K.,&Chalfin,A.(2008).To treat or not to treat: Evidence on the prospects of expanding treatment to drug-involved offenders.Washington,DC:TheUrbanInstitute.
Bonomo,E.(2012).Sanctionable behavior in a felony level drug court: Categorizing noncompliant behavior through a criminal-thinking lens[Master’sthesis].Atlanta:GeorgiaStateUniversity,Dept.ofCriminalJustice.
Bourgon,G.,&Armstrong,B.(2005).Transferringtheprinciplesofeffectivetreatmentintoa“realworld”prisonsetting.Criminal Justice & Behavior, 32,3-25.
Breckenridge,J.F.,Winfree,L.T.,Maupin,J.R.,&Clason,D.L.(2000).Drunkdrivers,DWI“drugcourt”treatment,andrecidivism:Whofails?Justice Research & Policy, 2,87-105.
Burke,K.S.(2010).Justwhatmadedrugcourtssuccessful?New England Journal on Criminal & Civil Confinement, 36,39-58.
Burke,K.,&Leben,S.(2007).Proceduralfairness:Akeyingredientinpublicsatisfaction.Court Review, 44,4-25.
Campbell,K.,Deck,D.,&Krupski,A.(2008).Recordlinkagesoftwareinthepublicdomain:AcomparisonofLinkPlus,TheLinkKing,anda“basic”deterministicalgorithm.Health Informatics Journal, 14(1),5-15.
Carey,S.M.,&Finigan,M.W.(2004).Adetailedcostanalysisinamaturedrugcourtsetting:acost-benefitevaluationoftheMultnomahCountyDrugCourt.Journalof
REFERENCES
154
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
ContemporaryCriminalJustice,20(3)292-338.
Carey,S.M.,Finigan,M.W.,Crumpton,C.D.,&Waller,M.S.(2006).CaliforniaDrugCourts:Outcomes,CostsandPromisingPractices:AnOverviewofPhaseIIinaStatewideStudy.JournalofPsychoactiveDrugs.SARCsupplement3.
Carey,S.M.,Mackin,J.R.,&Finigan,M.W.(2012).Whatworks?The10keycomponentsofDrugCourt:Research-basedbestpractices.Drug Court Review,8,6-42.
Chandler,R.K.,Fletcher,B.W.,&Volkow,N.D.(2009).Treatingdrugabuseandaddictioninthecriminaljusticesystem:Improvingpublichealthandsafety.Journal of the American Medical Association, 301, 183-190.
Cooper,C.S.(2004).Costs associated with the birth of drug and/or alcohol addicted/exposed infants.WashingtonDC:BureauofJusticeAssistance,DrugCourtClearinghouse&TechnicalAssistance Center at American University.
Cooper,C.S.,Franklin,B.,&Mease,T.(2010,April).Establishing drug treatment courts: Strategies, expe-riences and preliminary outcomes. WashingtonDC:JusticeProgramsOffice,SchoolofPublicAffairs,AmericanUniversity.Available at http://www.cicad.oas.org/fortalec-imiento_institucional/dtca/files/Establishing_DTC_%20Strategies_Ex-periences_Preliminary_Outcomes_volume%201.pdf.
Cresswell,L.S.,&Deschenes,E.P.(2001).Minorityandnon-minoritypercep-tionsofdrugcourtprogramseverityandeffectiveness.Journal of Drug Issues, 31,259-291.
Crumpton,D.,Carey,S.M.,&Finigan,M.W.(2004).EnhancingCostAnalysisofDrugCourts:TheTransactionalandInstitutionalCostAnalysisApproach.NPCResearch:Portland,OR
Dane,C.B.(2012).Examining the relationship between procedural justice and recidivism in a jail-based residential substance abuse treatment program [Master’sthesis].GeorgiaStateUniversity,AndrewYoungSchoolofPolicyStudies.
Dannerbeck,A.,Harris,G.,Sundet,P.,&Lloyd,K.(2006).Understandingandrespondingtoracialdifferencesindrugcourtoutcomes.Journal of Ethnicity in Substance Abuse5,1-22.
DeMatteo,D.S.,Marlowe,D.B.,&Festinger,D.S.(2006).Secondarypreventionservicesforclientswhoarelowriskindrugcourt:Aconceptualmodel.Crime & Delinquency, 52,114-134.
Deschenes.E.P.,Ireland,C.,&Kleinpeter,C.B.(2009).Enhancingdrugcourt success. Journal of Offender Rehabilitation, 48,19-36.
Edwards,S.J.L.,Lilford,R.J.,&Hewison,J.(1998).Theethicsofrandomisedcontrolledtrialsfromtheperspectivesofpatients,thepublic,andhealthcareprofessionals.British Medical Journal, 317,1209-1212.
155
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Farole,D.J.,&Cissner,A.B.(2007).Seeingeyetoeye:Participantandstaffperspectivesondrugcourts.InG.Berman,M.Rempel&R.V.Wolf(Eds.),Documenting Results: Research on Problem-Solving Justice (pp.51-73).NewYork:CenterforCourtInnovation.
Festinger,D.S.,Marlowe,D.B.,Lee,P.A.,Kirby,K.C.,Bovasso,G.,&McLellan,A.T.(2002).Statushearingsindrugcourt:Whenmoreislessandlessismore. Drug & Alcohol Dependence, 68,151-157.
Finigan,M.W.(2009).UnderstandingracialdisparitiesinDrugCourts.Drug Court Review, 6,135-142.
Flango,V.E.,&Cheesman,F.L.(2009).EffectivenessoftheSCRAMalcoholmonitoringdevice:Apreliminarytest. Drug Court Review, 6,109-134.
Frazer,M.S.(2006).The impact of the com-munity court model on defendant perceptions of fairness: A case study at the Red Hook Community Justice Center.NewYork:CenterforCourtInnovation.Availableathttp://www.courtinnovation.org/sites/de-fault/files/Procedural_Fairness.pdf.
FundaciónPazCiudadana,MinisteriodeJusticiadeChileandFiscalíadeChile(2014).Tribunales de Tratamiento de Drogas Compendio estadístico 2010, 2011 y 2012.[ebook]SantiagodeChile.Availableat:http://www.pazciudadana.cl/wp-content/up-loads/2015/03/ttd-analisis-estadisti-co.pdf[Accessed16Apr.2018].
FundaciónPazCiudadana/UldaOmarFigueroaOssa(2018).Tribunales de Tratamiento de Drogas: Evaluacion de su impacto en la disminucion de la reincidencia delictual en Chile.
Gallagher,J.R.(2012).Evaluating Drug Court effectiveness and exploring racial disparities in Drug Court outcomes: A mixed methods study [Doctoraldissertation].Arlington:Univ.ofTexas.
Gendreau,P.(1996).Theprinciplesofeffec-tiveinterventionwithoffenders.InA.Harland(Ed.),Choosing correc-tional options that work(pp.117-130).ThousandOaks,CA:Sage.
Gibbs,B.R.,&Wakefield,W.(2014).Theefficacyofenhancedalcohol-usemonitoring:Anexperimentexam-iningtheeffectsofanincreasedsupervisionprotocolinanadultdrugcourt.Drug Court Review, 9(1),1-22.
Goldkamp,J.S.,White,M.D.,&Robinson,J.B.(2002).Anhonestchance:Per-spectivesondrugcourts.Federal Sen-tencing Reporter, 6, 369–372.
Gossop,M.,Tradaka,K.,Stewart,D.,&Witton,J.(2005).Reductionsincriminalconvictionsafteraddictiontreatment:5-yearfollow-up.Drug&AlcoholDependence,79,295-302.
Gottfredson,D.C.,Kearley,B.W.,&Bush-way,S.D.(2008).Substanceuse,drugtreatment,andcrime:Anexaminationofintra-individualvari-ationinadrugcourtpopulation.JournalofDrugIssues,38,601-630.
156
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Gottfredson,D.C.,Kearley,B.W.,Najaka,S.S.,&Rocha,C.M.(2007).Howdrugtreatmentcourtswork:Ananalysisofmediators.JournalofResearchonCrime&Delinquency,44,3-35.
Gottfredson,D.C.,Najaka,S.S.,&Kearley,B.(2003).Effectivenessofdrugtreatmentcourts:Evidencefromarandomizedtrial.Criminology & Public Policy, 2, 171-196.
Gutierrez,L.,&Bourgon,G.(2012).Drugtreatmentcourts:Aquantitativereviewofstudyandtreatmentqual-ity.Justice Research & Policy, 14,47-77.
Harrell,A.,Cavanagh,S.,&Roman,J.(1999).Final report: Findings from the Evalu-ation of the D.C. Superior Court Drug Intervention Program.Washington,DC:TheUrbanInstitute.
Harrell,A.,&Roman,J.(2001).Reducingdruguseandcrimeamongoffend-ers:Theimpactofgraduatedsanc-tions.Journal of Drug Issues, 31, 207-232.
Hawken,A.,&Kleiman,M.(2009).Manag-ingdruginvolvedprobationerswithswiftandcertainsanctions:Eval-uatingHawaii’sHOPE[NCJRSno.229023]. WashingtonDC:NationalInstituteofJustice.Availableathttp://www.ncjrs.gov/pdffiles1/nij/grants/229023.pdf.
Heck,C.(2006).Local drug court research: Navigating performance measures and process evaluations [Mono-graphSeriesNo.6].Alexandria,VA:NationalDrugCourtInstitute.
Henrichson,C.,&Galgano,S.(2013).AGuidetoCalculatingJustice-SystemMarginalCosts.NewYorkVeraInstituteofJustice.https://www.bja.gov/Publications/Vera_Calculating-Justice-System-Marginal-Costs.pdf
Holloway,K.R.,Bennett,T.H.,&Far-rington,D.P.(2006).Theeffective-nessofdrugtreatmentprogramsinreducingcriminalbehavior.Psico-thema,18,620-629.
Huddleston,W.,&Marlowe,D.B.(2011).Painting the current picture: A national report on drug courts and other problem solving court programs in the United States. Al-exandria,VA:NationalDrugCourtInstitute.
Huebner,B.M.,&Cobbina,J.(2007).Theeffectofdruguse,drugtreatmentparticipation,andtreatmentcom-pletiononprobationerrecidivism.Journal of Drug Issues, 37,619-641.
Inciardi,J.,Martin,S.,&Butzin,C.(2004).Five-yearoutcomesoftherapeuticcommunitytreatmentofdrug-in-volvedoffendersafterreleasefromprison. Crime & Delinquency, 50,88-107.
Jones,C.(2011).Intensivejudicialsuper-visionanddrugcourtoutcomes:Interimfindingsfromarandomizedcontrolledtrial.Crime & Justice Bul-letin, 152,1-16.NewSouthWales,Australia:BureauofCrimeStatistics&Research.Available at http://www.courtwise.nsw.gov.au/lawlink/bocsar/ll_bocsar.nsf/vw-Files/CJB152.pdf/$file/CJB152.pdf.
157
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
King,M.S.(2009).The solution-focused judging bench book.Melbourne,Australia:AustralasianInstituteofJudicialAdministration&LegalSer-vicesBoardofVictoria.
Kirk,J.M.(2012).Assessing self determin-ing attitudes and behaviors in court mandated treatment clients [Doc-toraldissertation].OklahomaStateUniversity.
Kleinpeter,C.B.,Brocatao,J.,&Koob,J.J.(2010).Doesdrugtestingdeterdrugcourtparticipantsfromusingdrugsoralcohol?Journal of Of-fender Rehabilitation, 49,434-444.
Landenberger,N.A.,&Lipsey,N.W.(2005).Thepositiveeffectsofcognitive-be-havioralprogramforoffenders:Ameta-analysisoffactorsassociatedwitheffectivetreatment.Journal of Experimental Criminology,1(4),451-476.
Latimer,J.,Dowden,C.,&Muise,D.(2005).Theeffectivenessofrestorativejus-ticepractices:Ameta-analysis.The Prison Journal, 85,127-144.
Latimer,J.,Morton-Bourgon,K.,&Chre-tien,J.(2006).A meta-analytic ex-amination of drug treatment courts: Do they reduce recidivism?CanadaDept.ofJustice,Research&Statis-ticsDivision.
Leukefeld,C.,Webster,J.M.,Staton-Tin-dall,M.,&Duvall,J.(2007).Em-ploymentandworkamongdrugcourtclients:12-monthoutcomes.Substance Use & Misuse, 42,1109-1126.
Lindquist,C.H.,Krebs,C.P.,&Lattimore,P.K.(2006).Sanctionsandrewardsindrugcourtprograms:Implemen-tation,perceivedefficacy,anddeci-sion making. Journal of Drug Issues, 36,119-146.
Lowenkamp,C.T.,Flores,A.W.,Holsinger,A.M.,Makarios,M.D.,&Latessa,E.J.(2010).Intensivesupervisionpro-grams:Doesprogramphilosophyandtheprinciplesofeffectiveinter-ventionmatter?Journal of Criminal Justice, 38,368-375.
Lowenkamp,C.T.,Holsinger,A.M.,&La-tessa,E.J.(2005).Aredrugcourtseffective:Ameta-analyticreview.Journal of Community Corrections, 15 (1), 5-28.
Lowenkamp,C.T.,&Latessa,E.J.(2004).Understandingtheriskprinciple:Howandwhycorrectionalinter-ventionscanharmlow-riskoffend-ers. Topics in Community Correc-tions-2004, 3–8.
Lowenkamp,C.T.,Latessa,E.J.,&Smith,P.(2006).Doescorrectionalprogramqualityreallymatter?Theimpactofadheringtotheprinciplesofef-fectiveintervention.Criminology&PublicPolicy,5,575-594.
MacDonald,J.M.,Morral,A.R.,Raymond,B.,&Eibner,C.(2007).Theeffica-cyoftheRioHondoDUICourt:A2-yearfieldexperiment.Evaluation Review, 31, 4-23.
MacKinnon,D.P.(2008).Introduction to statistical mediation analysis.NewYork:Erlbaum.
158
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Marlowe,D.B.(2008).Applicationofsanc-tions.InC.Hardin&J.N.Kushner(Eds.),Quality improvement for drug courts: Evidence-based practic-es(pp.107-114)[MonographSeriesNo.9].Alexandria,VA:NationalDrugCourtInstitute.
Marlowe,D.B.(2011).Applyingincentivesandsanctions.InD.B.Marlowe&W.G.Meyer(Eds.),The drug court judicial benchbook(pp.139-157).Alexandria,VA:NationalDrugCourtInstitute.Availableathttp://www.ndci.org/sites/default/files/nad-cp/14146_NDCI_Benchbook_v6.pdf.
Marlowe,D.B.(2012a)(Ed.).SpecialIssueonbestpracticesinDrugCourts.Drug Court Review, 8 (1).
Marlowe,D.B.(2012b).Targeting the right participants for adult drug courts [PractitionerFactSheetVol.VII,No.1].Alexandria,VA:NationalDrugCourtInstitute.Athttp://www.ndci.org/sites/default/files/nadcp/Targeting_Part_I.pdf.
Marlowe,D.B.(2013).Achievingracialandethnicfairnessindrugcourts.Court Review,49(1),40-47.
Marlowe,D.B.,Festinger,D.S.,Arabia,P.L.,Dugosh,K.L.,Benasutti,K.M.,Croft,J.R.,&McKay,J.R.(2008b).Adaptiveinterventionsindrugcourt:Apilotexperiment.Criminal Justice Review, 33,343-360.
Marlowe,D.B.,Festinger,D.S.,Dugosh,K.L.,Arabia,P.L.,&Kirby,K.C.(2008a).Aneffectivenesstrialofcontingencymanagementinafel-onypre-adjudicationdrugcourt.
Journal of Applied Behavior Analy-sis, 41,565-577.
Marlowe,D.B.,Festinger,D.S.,Dugosh,K.L.,Lee,P.A.,&Benasutti,K.M.(2007).Adaptingjudicialsupervi-siontotherisklevelofdrugoffend-ers:Dischargeandsix-monthout-comesfromaprospectivematchingstudy.Drug & Alcohol Dependence, 88S,4-13.
Marlowe,D.B.,Festinger,D.S.,Foltz,C.,Lee,P.A.,&Patapis,N.S.(2005).Perceiveddeterrenceandoutcomesindrugcourt.Behavioral Sciences & the Law, 23,183-198.
Marlowe,D.B.,Festinger,D.S.,Lee,P.A.,Du-gosh,K.L.,&Benasutti,K.M.(2006).Matchingjudicialsupervisiontocli-ents’riskstatusindrugcourt.Crime & Delinquency, 52,52-76.
Marlowe,D.B.,Hardin,C.D.,&Fox,C.L.(2016).Painting the current pic-ture: A national report on drug courts and other problem-solving courts in the United States. Alex-andria,VA:NationalDrugCourtInstitute.Retrievedfromhttp://www.ndci.org/wp-content/up-loads/2016/05/Painting-the-Cur-rent-Picture-2016.pdf
Marlowe,D.B.,&Kirby,K.C.(1999).Effec-tiveuseofsanctionsindrugcourts:Lessonsfrombehavioralresearch.National Drug Court Institute Re-view, 2, 1-31.
Marlowe,D.B.,Patapis,N.S.,&DeMatteo,D.S.(2003).Amenabilitytotreat-mentofdrugoffenders.Federal Probation,67,40-46.
159
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Martin,S.S.,Butzin,C.A.,Saum,S.A.,&Inciardi,J.A.(1999).Three-yearoutcomesoftherapeuticcommu-nitytreatmentfordrug-involvedoffendersinDelaware.Prison Jour-nal, 79, 294-320.
McIvor,G.(2009).Therapeuticjurispru-denceandproceduraljusticeinScottishdrugcourts.Criminology & Criminal Justice, 9,29-49.
McKee,M.(2010).San Francisco drug court transitional housing program outcome study,athttp://www.sfsuperiorcourt.org/sites/default/files/pdfs/2676%20Outcome%20on%20SF%20Drug%20Court%20Transitional%20Hous-ing%20Program.pdf.
McLellan,A.T.,Lewis,D.C.,O’Brien,C.P.,&Kleber,H.D.(2000).Drugdepen-dence,achronicmedicalillness:Im-plicationsfortreatment,insurance,andoutcomesevaluation.JAMA, 284,1689-1695.
Mendoza,N.S.,Trinidad,J.R.,Nochajski,T.H.,&Farrell,M.C.(2013).Symp-tomsofdepressionandsuccessfuldrugcourtcompletion.Commu-nity Mental Health Journal,DOI:10.1007/s10597-013-9595-5.
Messina,N.,Calhoun,S.,&Warda,U.(2012).Gender-responsivedrugcourttreatment:Arandomizedcontrolledtrial.Criminal Justice & Behavior, 39,1539-1558.
Mitchell,O.,Wilson,D.B.,Eggers,A.,&MacKenzie,D.L.(2012).Assessingtheeffectivenessofdrugcourtsonrecidivism:Ameta-analyticreview
oftraditionalandnon-traditionaldrugcourts.Journal of Criminal Justice, 40,60-71.
NationalAssociationofDrugCourtPro-fessionals.(1997).Defining drug courts: The key components.Wash-ington,DC:OfficeofJusticePro-grams,U.S.Dept.ofJustice.
NationalInstituteofJustice.(2018,August23).DrugCourts.RetrievedDecem-ber20,2018,fromhttps://www.nij.gov/topics/courts/drug-courts/Pages/welcome.aspx.
NationalResearchCouncil.(2001).Inform-ing America’s policy on illegal drugs: What we don’t know keeps hurting us.WashingtonDC:NationalAcademyPress.
Newton-Taylor,B.,Patra,J.,&Gliksman,L.(2009).TorontoDrugTreatmentCourt:Participantintakecharacter-isticsaspredictorsof“successful”programcompletion.Journal of Drug Issues, Fall,965-987.
O’Hear,M.M.(2009).Rethinkingdrugcourts:Restorativejusticeasare-sponsetoracialinjustice.Stanford Law & Policy Review, 20,101-137.
OrganizationofAmericanStates,In-ter-American Drug Abuse Control Commission.(2010).Hemispheric drug strategy: Plan of action 2011-2015. Available at http://www.cicad.oas.org/Main/Template.asp?File=/main/about-cicad/basicdocuments/strate-gy_2010_eng.asp.
OrganizationofAmericanStates,In-ter-American Drug Abuse Control
160
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Commission.(2010b).Introduction to the New Hemispheric Drug Strat-egy. Available at http://www.cicad.oas.org/en/basicdocuments/Hemispheric%20Drug%20Strategy100603.pdf..
OrganizationofAmericanStates,In-ter-American Drug Abuse Control Commission.(2012b).Report on the first intensive technical work-shop on the development of meth-odologies and indicators for DTC evaluation. WashingtonDC:Author.
OrganizationofAmericanStates,Obser-vatoryonCitizenSecurity.(2011).Report on citizen security in the Americas. Available at http://www.oas.org/dsp/alertamerica/Aler-tamerica_2011.pdf.
OrganizationofAmericanStates,Obser-vatoryonCitizenSecurity.(2012a).Report on citizen security in the Americas. Available at http://www.oas.org/dsp/alertamerica/Report/Alertamerica2012.pdf.
Peters,R.H.(1996).Evaluating drug court programs: An overview of issues and alternative strategies.Wash-ingtonDC:JusticeProgramsOfficeatAmericanUniversity.Availableat http://www1.spa.american.edu/justice/documents/220.pdf
Peters,R.H.,Haas,A.L.,&Hunt,W.M.(2002).Treatment“dosage”effectsindrugcourtprograms.Journal of Offender Rehabilitation, 33,63-72.
Peters,R.H.,Kremling,J.,Bekman,N.M.,&Caudy,M.S.(2012).Co-occur-ringdisordersintreatment-based
courts:Resultsofanationalsurvey.Behavioral Sciences & the Law,DOI:10.1002/bsl.2024.
President’sCommissiononCombatingDrugAddictionandtheOpioidCrisis.(2017).Final Report to the President[PDFfile].Retrievedfromhttps://www.whitehouse.gov/sites/whitehouse.gov/files/images/Final_Report_Draft_11-1-2017.pdf.
Reingle,J.M.,Striley,C.W.,Small,E.,Cre-celius,R.,O’Leary,C.C.,&Cottler,L.B.(2012).Cancourtroombehaviorpredictrecidivism?AnassessmentoftheCourtroomBehaviorCheckListforwomenpresentingindrugcourt. American Journal of Criminal Justice,DOI:10.1007/s12103-012-9190-2.
Rempel,M.(2007).Actionresearch:Usinginformationtoimproveyourdrugcourt.InG.Berman,M.Rempel&R.V.Wolf(Eds.),Documenting re-sults: Research on problem-solving justice (pp.101-122).NewYork,NY:CenterforCourtInnovation.
Rempel,M.,Raine,V.,&Lambson,SH.(20142014).A diagnostic study of the Addiction Treatment Court in Guadalupe, Nuevo León, Mexico: Findings and recommendations. WashingtonDC:Inter-AmericanDrugAbuseControlCommission,OrganizationofAmericanStates,andCenterforCourtInnovation.
Richardson,M.,Katsakou,C.,Torres-Gon-zalez,F.,Onchev,G.,Kallert,T.,&Priebe,S.(2011).FactorialvalidityandmeasurementequivalenceoftheClientAssessmentofTreatment
161
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Scaleforpsychiatricinpatientcare–AstudyinthreeEuropeancoun-tries. Psychiatry Research, 188,156-160.
Roman,J.(2013).Cost-benefitanalysisofcriminaljusticereforms.NIJJournal,272.https://ncjrs.gov/pdffiles1/nij/241929.pdf
Rossman,S.B.,Rempel,M.,Roman,J.K.,Zweig,J.M.,Lindquist,C.H.,Green,M.,Downey,P.M.,Yahner,J.,Bha-ti,A.S.,&Farole,D.J.(2011).The Multi-Site Adult Drug Court Evalua-tion: The impact of Drug Courts, vol-ume 4. WashingtonDC:UrbanInsti-tuteJusticePolicyCenter.Availableat https://www.ncjrs.gov/pdffiles1/nij/grants/237112.pdf.
Rubio,D.,M.,Cheesman,F.,&Federspiel,W.(2008,July).Performance mea-surement of drug courts: The state of the art [StateTechnicalAssis-tanceBulletinVol.6]. Williams-burg,VA:NationalCenterforStateCourts.
Rysavy,P.,Cunningham,T.,O’Reilly-Mar-tinez,R.(2011).PreliminaryanalysisoftheNorthernTerri-tory’sillicitdrugcourtdiversionprogramhighlightstheneedtoexaminelowerprogramcom-pletionratesforindigenouscli-ents. Drug & Alcohol Review, 30,671-676.DOI:10.1111/j.1465-3362.2010.00275.x
Satel,S.(1998).Observationalstudyofcourtroomdynamicsinselecteddrugcourts.National Drug Court Institute Review, 1,43-72.
Sevigny,E.L.,Pollack,H.A.,&Reuter,P.(2013).Candrugcourtshelptoreduceprisonandjailpopulations?Annals of the American Academy of Political & Social Science, 647,190-212.
Shaffer,D.K.(2010).Lookinginsidetheblackboxofdrugcourts:Ameta-analyticreview.Justice Quarterly, 28, 493-521.
Somers,J.M.,Currie,L.,Moniruzzaman,A.,Eiboff,F.,&Patterson,M.(2011).DrugTreatmentCourtofVancouver(DTCV):Anempiricalevaluationofrecidivism.Addiction Research & Therapy, 2,1-7.Availableathttp://dx.doi.org/10.4172/2155-6105.1000117.
Somers,J.M.,Rezansoff,S.N.,&Moniru-zzaman,A.(2012).Mediators of drug treatment court outcomes in Vancouver Canada.BritishColum-bia,Canada:SimonFraserUniv.
Sperber,K.M.,Latessa,E.,&Makarios,M.D.(2013).Examiningtheinterac-tionbetweenlevelofriskanddos-age of treatment. Criminal Justice & Behavior, 40,338-348.
Sullivan,D.,&Tifft,L.(2008).Handbook of restorative justice.NewYork:Rout-ledge.
Summers,R.E.,&Barber,J.P.(2003).Therapeuticallianceasameasur-ablepsychotherapyskill.Academic Psychiatry, 27,160-165.
Taxman,F.S.,&Bouffard,J.A.(2005).Treatmentasapartofdrugcourt:Theimpactongraduationrates.Journal of Offender Rehabilitation, 42,23-50.
162
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Taxman,F.S.,&Marlowe,D.B.(Eds.)(2006).Risk,needs,responsivity:Inactionorinaction?[SpecialIssue].Crime & Delinquency, 52(1).
TreatmentResearchInstitute.(2012).In-creasing effective communication between criminal justice and treat-ment settings using health informa-tion technology.Philadelphia,PA:Author.
Turner,S.,Greenwood,P.Fain,T.,&De-schenes,E.(1999).Perceptionsofdrugcourt:Howoffendersvieweaseofprogramcompletion,strengthsandweaknesses,andtheimpactontheirlives.National Drug Court Institute Review, 2,61-85.
Tyler,T.R.(2007).Proceduraljusticeandthecourts.Court Review, 44,26.
U.S.GovernmentAccountabilityOffice.(2005).Adult drug courts: Evidence indicates recidivism reductions and mixed results for other outcomes [No.GAO-05-219].Washington,DC:Author.
VanWormer,J.(2010).Understanding op-erational dynamics of drug courts [Doctoraldissertation].Dept.ofCriminalJustice,UniversityofWashington.
Wiener,R.,Winick,B.J.,Georges,L.S.,&Castro,A.(2010).Atestabletheoryofproblem-solvingcourts:Avoidingpastempiricalandlegalfailures.International Journal of Law & Psy-chiatry, 33,417-427.
Wodahl,E.J.,Garland,B.,Culhane,S.E.,&McCarty,W.P.(2011).Utilizingbehav-ioralinterventionstoimprovesuper-visionoutcomesincommunity-basedcorrections.Criminal Justice & Behav-ior, 38,386-405.
Wolfer,L.(2006).Graduatesspeak:Aqual-itativeexplorationofdrugcourtgraduates’viewsofthestrengthsandweaknessesoftheprogram.Contemporary Drug Problems, 33,303-320.
Zarkin,G.A.,Cowell,A.J.,Hicks,K.A.,Mills,M.J.,Belenko,S.,Dunlap,L.J.,&Keyes,V.(2012).Lifetimebenefitsandcostsofdivertingsub-stance-abusingoffendersfromstateprison.Crime&Delinquency.DOI:10.1177/0011128712461904.
Zweig,J.M.,Lindquist,C.,Downey,P.M.,Roman,J.,&Rossman,S.B.(2012).Drugcourtpoliciesandpractices:Howprogramimplementationaf-fectsoffendersubstanceuseandcriminalbehavioroutcomes.Drug Court Review,8,43-79.
163
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
AddictionApsychiatricdiagnosisreflectingcompulsiveuseofalcoholorotherdrugs.Characteris-ticsymptoms includecravings for thesub-stance, uncontrolled usage, or uncomfort-ablewithdrawal symptomswhen levels ofthesubstancedeclineinthebloodstreamorcentralnervoussystem.InthemostrecentversionofDiagnosticandStatisticalManualofMentalDisorders(DSM-5orDSM-V),itisreferredtoas“substanceusedisorder.”Seealso:Substance dependence.
Adulterated test specimensBodilyspecimenssuchasurineorsalivathatshow evidence of having been tamperedwith or of being fraudulent or unreliable.Most laboratories routinely evaluate drugtestspecimensforevidenceofadulterationbyexaminingsuchindicatorsastemperature,pH, creatinine, and specific gravity. ManyDTCs assume adulterated specimens to besubstance-positive or require that a newspecimen be delivered. Evaluators shouldordinarily treat adulterated specimens assubstance-positive or as not having beendeliveredasdirected (i.e., asanunexcusedfailuretoprovideascheduledsample).
Attendance rateTheratioofthenumberofsessionsorser-vicesthatparticipantsreceived,dividedbythe number of sessions or services theywerescheduledtoreceive.
Best practicesSpecific services or interventions in DTCsthathavebeendemonstratedthroughem-piricalresearchtosignificantlyimproveout-comes.
Clinical assessment toolsStandardized and validated questionnairesor interviews that assess the diagnosticcriteria for or symptoms of substance de-pendenceandothermajorpsychiatricdis-orders.
CohortA group of participants who entered theDTCduringthesamespecifiedtimeperiod,oftendefinedasan intervalof12months.Forexample,allparticipantswhoenteredaDTCbetween January1 andDecember31ofagivenyearmightbedefinedasacohort. Community serviceA type of restorative justice interventionrequiringoffenderstoworkonavolunteerbasisinthecommunity.Commonexamplesofcommunityservicemayincludecleaninguplitteronthehighwayorworkinginasoupkitchenforthehomeless.
Comparison groupA sample of individuals who are similarto the DTC participants but did notparticipateintheDTC.InDTCevaluations,the comparison group is often comprisedof drug-involved offenders who weresentencedtoprobationorwhounderwentadjudication as usual. The comparisongroup should be as equivalent as possibletotheDTCgroupwithrespecttovariablesthatwouldbeexpectedtoaffectoutcomes.Without a comparison group, it cannotbe determined whether the outcomeswereaffectedbytheDTCorwhethertheymight have occurred anyway even if theparticipantshadnotenteredtheDTC.
GLOSSARY OF TECHNICAL TERMS
164
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Computer-assisted personal interview (CAPI)Acomputerizedsurveythatallowstheeval-uator to enter items directly into an ana-lyzable database. The computer presentsitemsoneatatime;mayofferhelp-menusindicatinghowtophrase,score,orinterpretan item;andskipsquestions thatare logi-cally inapplicable for a participant (e.g., aquestion concerningpregnancy for amaleinterviewee).
Computer syntax:See:Statistical syntax.
CorrelationThedegree towhich twovariablesareas-sociatedwitheachother.Acorrelationin-dicatesthereisarelationshipbetweenthevariables,butdoesnotprovecausality.Forexample,moretreatmentinaDTCmightbecorrelated with better outcomes, but thisdoesnotprovethattreatmentwasrespon-siblefortheoutcomes.
Cost evaluationAsystematicevaluationthatattachesmon-etaryvaluestotheresultsofanimpacteval-uationtoestimatethenetfinancialimpactsofaDTC.
Counterfactual hypothesisAn alternative hypothesis to the researchhypothesis which predicts that a DTC didnotaffectoutcomes.Inmostinstances,be-ingabletotestthecounterfactualhypoth-esisrequirestheevaluatortouseafairandunbiasedcomparisonsampleagainstwhichto compare the performance of the DTCparticipants. CovariateAvariablethatisenteredfirstintoastatis-tical analysis, andwhose influence is thenfactored out to evaluate the effect of thetrue variable of interest. This procedure
helps to rule out confounds or alternativeexplanationsthatmighthaveaccountedfortheeffectsbeingobserved.
Data & Safety Monitoring Board (DSMB)A multidisciplinary group of professionalswith expertise in research methods, re-search ethics and statistical analyses whoareresponsibleforoverseeingtheintegrityofdatacollectionanddataanalysesduringrandomized studies. Among other duties,theDSMBmayperformoroverseeinterimanalysestodeterminewhetheranypartici-pantsinthestudyarebeingdisadvantagedor suffering adverse events related to theinterventions.
Date-stampingThe practice of connecting services oreventstothedatesonwhichtheyoccurredorweresupposedtohaveoccurred.Forex-ample,indicatingthatacounselingsessionwasattendedonJanuary1,2013,isaformof date-stamping. Date-stamping is criti-cally important for measuring many per-formance indicators in DTC program eval-uations. Some computerized data-entrysystemsautomaticallydate-stampentries.
Density of servicesTheamountoftreatmentorotherservicesthatweredeliveredperunitoftime,suchaspermonthorperphaseoftheprogram.Forexample,attendanceatfivesessionsinonemonth(5÷1=5)reflectsagreaterdensityof services thanattendanceat15sessionsinsixmonths(15÷6=2.5)eventhough15sessionsaremorethanfivesessions.
Descriptive analysesStatisticalinformationthatdoesnotseektoinfercauseandeffectorattributethefind-ings to a larger population. For example,calculating thepercentageofwomenwhograduatedfromaDTCortheaveragenum-
165
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
beroftreatmentsessionsthatwereattend-edaredescriptiveanalyses.
Direct secure messaging (DSM)A method for transmitting encrypted in-formationtoknownandtrustedrecipientsovertheInternet.
Distal outcomesLong-term outcomes often occurring afterparticipants are no longer enrolled in theDTCprogram.
Dosage or doseTheamountofaserviceortreatmentthatparticipants actually received, as opposedto what they were scheduled to receive.Dosage is typicallymeasured by the num-ber of sessions thatwere attendedor thelengthoftimeparticipantswereexposedtotheservice.
EntryThespecificdateonwhichaparticipantoffi-ciallyenteredaDTCorcomparableprogramand the program obtained legal authorityovertheindividualtoordertreatmentandsupervision.ManyDTCsandprobationpro-gramshaveaformalentryhearingincourtatwhichtheparticipantpleadsorisfoundguilty (or in some jurisdictions,where thepre-trial/pre-plea criminal proceedings aresuspended),maywaivecertainlegalrights,andformallyenterstheprogram.
Field visitsSupervision activities conducted by pro-bationofficersoutsideoftheiroffices. Ex-amplesmay include visits to anoffender’shomeorplaceofemployment.
Follow-up windowThetimeperiodduringwhichoutcomesaremeasured. For example, if outcomes aremeasuredfor12monthsaftereachpartic-
ipant’sentryintoaDTC,thenthefollow-upwindow is 12 months in length. In mostanalyses, follow-up windows should beequivalent or comparable in length for allparticipants.
GeneralizabilityThe degree to which the same result isfound in different programsor amongdif-ferentpopulationsofparticipants.
GraduationSuccessful discharge from a DTC markingcompletionoftherequirementsofthepro-gram.OutcomesaresignificantlybetterforDTCs that have formal criteria for gradua-tionandholdaformalgraduationceremonyin court.
High-need offendersOffenderswith relatively severe substanceabuse or mental health disorders. High-need offenders have been found to havebetteroutcomesinDTCsascomparedtoal-ternativeprogramssuchasprobation.
High-risk offendersOffenderswithmoreseverecriminalhisto-riesorwhohaveahistoryoffailingintreat-mentoroncorrectionalsupervision.Theseindividualsareatahigherriskforre-offend-ing,andtendtohaveapoorprognosisforsuccess instandardcorrectionalrehabilita-tionprogramsandrequireintensivesuper-vision to succeed in treatment. High-riskaddictedoffendershavebeenfoundtohavebetteroutcomesinDTCsascomparedtoal-ternativeprogramssuchasprobation.
Illicit drugsDrugs that are legally banned and pre-scription medications that are used for anon-prescribed or non-medically-indicatedpurpose.
166
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Imputation proceduresStatisticalproceduresthatmaybeemployedtocompensateformissingdata.Someim-putationproceduresaccountforthepatternof results immediately before and after amissingresult.Othersassumetheaverageormostprevalentresultfortheparticipantorthesampletobethelikelyoutcomeofamissingscore.Expertconsultationisusuallyrequiredtodecidewhetherandhowtousetheseprocedures.
Institutional Review Board (IRB)A multidisciplinary group of professionalsandcommunityrepresentativeswithknowl-edgeorexpertiseinresearchmethods,re-search ethics, and the subjectmatterof astudy.TheIRBisresponsiblefordetermin-ingwhetherastudyisethicalandsafewithregardtosuchmattersasinformedconsent,confidentiality,andtherisk/benefitratioforparticipants. The IRBmaybeempoweredto stop a study or require changes to themethodsasaconditionofpre-approvalorannualre-approval.
Intent-to-treat analysisAnanalysisthatincludesdataonallindivid-uals who entered the DTC or comparisonprogram regardless of whether they com-pleted, dropped out, or were terminatedfromtheprograms.Intent-to-treatanalysesshouldordinarilybereportedastheprimaryanalysesformostDTCprogramevaluations.
Interaction analysesStatisticalprocedures thatexamine the in-fluence of more than one variable on anoutcomeanddeterminehowthevariablesaffectedtheoutcomealoneandincombina-tion.InDTCevaluations,interactionanaly-sesmightbeusedtodeterminewhichtypesofparticipantshadbetteroutcomes intheDTCasopposedtoacomparisoncondition.
Inter-rater reliabilityThedegreetowhichdifferentratersoreval-uators assign similar valueswhenmeasur-ingavariable.
Investment costsTheadditionalexpendituresrequiredtoad-ministeraDTCprogram,suchastheaddedcostsof treatmentor frequentcourthear-ings.
JailInsomejurisdictions,acorrectionalinstitu-tion that typically detains individuals on apre-trialbasisorforsentencesoflessthanoneyearinlength.
Log-linear transformationAmathematicalprocedurethatmaycom-pensateforaskewedornon-normaldistri-butionof scores. This procedurehas theeffect of smoothing out a skewed distri-bution ormaking it function as if itwerea normal distribution. Expert statisticalconsultation is often required to employthisadvancedstatisticalprocedureappro-priately.
Long-term outcome evaluationAsystematicevaluationofparticipants’per-formanceaftertheyarenolongerenrolledintheDTCprogram.Theevaluationtypical-lyfocusesonpost-programoutcomessuchasnewarrestorconvictionrates, re-incar-ceration rates, and employment rates. Itmayalsobereferredtoasadistaloutcomeevaluation.
Low-need offendersOffenders who do not have serious sub-stance abuse or mental health disorders.Low-needoffendersdonotneedtobetreat-edinprogramssuchasDTCsthatprovideahighleveloftreatmentandrehabilitation.
167
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Low-risk offendersOffenderswith relatively less severecrimi-nalhistorieswhohaveagoodprognosisforsuccess instandardsubstanceabusetreat-mentorcorrectionalrehabilitation.Suchin-dividualsareatalowerriskforre-offendingandoftendonotrequireintensiveprogramssuchasDTCstohavepositiveoutcomes.
Management information system (MIS)An automated computer system that col-lects standardizeddata elements andmayrun statistical analyses and provide out-comereports.InDTCs,anMISmightcollectandanalyzedataontheservicesdeliveredintheDTCandparticipants’performance.
Matched comparison groupIndividualswhoarepairedwithDTCpartic-ipantsbasedonmultiplevariablesthatareknowntoaffectoutcomes.Forexample,anevaluatormightmatcheachDTCparticipantwithaprobationerwho is similar in termsofcriminalhistory,demographiccharacter-istics,andsubstanceabuseproblems. Thegroupsshouldbematchedonvariablesthatpredictoutcomes,andnotmerelyvariablesthatareeasytomeasuresuchasbasicde-mographics.
Mediator variableAvariable thataffects the relationshipbe-tweenapredictorvariable (suchasenroll-ment in a DTC) and an outcome variable(suchasrecidivismorgraduation).Theme-diatorvariablecanbethoughtofasan in-terveningvariablethathelpstoexplaintherelationshipbetweenthepredictorvariableandoutcomevariable. Mediatorvariablesare identified through advanced statisticalanalysescalledmediationanalyses.
Missing denominatorReferstoaproblemcommonlyencounteredinDTCprogramevaluationsinwhichthere
is a failure to faithfully record informationabout events that should have transpiredbutdidnot.Forexample,datamightnotberecordedon treatment sessions thatwerescheduledtooccurbutwerenotattended.This can complicate the interpretation offindings. It is importantforstaffmembersto record information about whether ap-pointmentswerekept,notkept,excusedorrescheduled.Seealso:Problem of the miss-ing denominator.
Moderator variablesCharacteristicsofparticipantsthatdifferen-tiallypredictoutcomesinDTCsascomparedtoalternativeprograms,suchasprobation.Moderator variables may be identifiedthrough the use of statistical techniquescalled moderator analyses or interactionanalyses. Theseanalyses assist theevalu-ator to determine which types of partici-pantswerehelpedbyaDTCandwhichpar-ticipantsmightnothavebeenhelpedbytheDTC.
Motivation for changeReferstoabodyofresearchfindingbetteroutcomes in substance abuse treatment for patients who were intrinsically motivatedtoimprovetheir lives. IntrinsicmotivationforchangedoesnotappeartoberequiredwhenparticipantsfirstenteraDTCbutmaybecome necessary before they are dis-charged.Itmayalsobereferredtoasread-inessforchange.
Neutral dischargeA type of discharge from a DTC that indi-catesneitherfailurefromtheprogramnorsuccessfulcompletionoftheprogram.ItistypicallyreservedforparticipantswhohaveseriousmedicalorpsychiatricillnessesthattheDTCisunabletotreat,orwhomoveoutof the jurisdiction or enlist in themilitarywith the permission of the court. Partici-
168
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
pantswho receive a neutral discharge areremovedfromtheequationwhencalculat-ingmostperformanceindicatorsforaDTC.
Nonparametric statistical testsStatistical tests that do not require thescoresinthesampletobenormallydistrib-utedortohaveawiderange.
Operationalizing variablesThe process of defining variables such asthe services provided in a DTC or partici-pants’outcomesinobjective,concrete,andmeasurable terms.
Outcome costsThe expenditures incurred by taxpayers orthegovernmenttodealwithparticipantsaf-tertheycompletedtheDTCprogram,suchasthecostsofprosecutingnewoffensesorin-carceratingtheparticipantsfornewcrimes.
Outcome evaluationA systematic study of how a DTC affectedparticipants’ performance on outcomessuch as substance use or crime. See also:Short-term outcome evaluation or Long-term outcome evaluation.
Outcome savingsThemoniesrepaidtoorreclaimedbysoci-etyasaresultoftheimprovedsocialfunc-tioning of DTC participants. For example,DTC participants might pay more taxes,contributetothefinancialsupportoftheirchildren,orvolunteertoworkincharitiesorsocialserviceagencies.
Participant characteristicsAttributes of participants that pre-datedtheirentryintotheDTCorcomparisonpro-gram. Thesetypically includesocio-demo-graphicvariablessuchasage,gender,race,and employment status; clinical variablessuchasparticipants’primarysubstancesof
abuse,psychiatric/substanceusediagnoses,and history of substance abuse ormentalhealthtreatment;andcriminalhistoryvari-ablessuchaspriorarrests,convictions,andincarcerations.See also: Predictor variables and Risk factors.
Participant-level performance indicatorsQuantifiablemeasuresofhowparticipantsperformedwhiletheywereenrolledintheDTC and after discharge. Examplesmightincludehowoftenparticipants testedneg-ative for alcohol and other illicit drugs orgraduated from the DTC. They may alsobereferredtoasoutputs.See also: Perfor-mance indicators.
Perceived deterrenceReferstoabodyofresearchindicatingthatoffendershavebetteroutcomeswhentheyperceive a direct and rational connectionbetweentheirownconductandtheimpo-sitionofrewardsandsanctionsbycriminaljustice authorities. Better perceptions ofperceiveddeterrencehavebeenassociatedwithbetteroutcomesinDTCs.
Performance benchmarksSpecificthresholdsorlevelsofperformancethatDTCsshouldstrivetoachieve.Ideally,performancebenchmarksshouldbebasedonempiricalevidenceindicatingwhichprac-ticesproducedbetteroutcomesinDTCs.
Performance indicatorsPerformance indicators are quantifiablemeasuresoftheservicesprovidedinaDTC(called program-level performance indica-tors or inputs) and how participants per-formedintheprogramandafterdischarge(participant-levelperformanceindicatorsoroutputs).Examplesofprogram-levelperfor-mance indicatorsmight includehowoftencourthearingswereheldorhowoftenpar-
169
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
ticipants attended substance abuse treat-ment.Examplesofparticipant-levelperfor-mance indicatorsmight includehowoftenparticipantstestednegativeforalcoholandillicitdrugsorgraduatedfromtheprogram.Seealso:Participant-level performance in-dicators and Program-level performance indicators.
Policies and procedures manualAwrittenmanual documenting the essen-tialpractices,interventionsandpoliciesfortheDTC.ResearchindicatesDTCshavebet-teroutcomeswhenthey(1)haveapoliciesand procedures manual, (2) generally ad-heretowhatiswritteninthemanual,and(3)updatethemanualonaregularbasis.
Post-adjudication DTCADTCinwhichafinalconvictionhasbeenenteredandparticipantscompletethepro-gram in lieuof receivinga longerormoreseveresentence.
Pre-adjudication DTCA DTC in which successful graduates canhave their guilty plea or conviction with-drawnorvacatedandmayhavethearrestexpungedorerasedfromtheircriminalre-cord (or inother jurisdictionswhereapri-orguiltypleaorconviction isnot requiredforentry,aDTC inwhichthepre-trial/pre-plea criminal proceedings are suspended,andinwhichsuccessfulgraduateshavethechargesagainstthemdropped).
Predictor variablesCharacteristicsofparticipantsthatpre-datedtheirentryintotheDTCorcomparisonpro-gramandreliablycorrelatewithoutcomes.
PrisonInsomejurisdictions,acorrectionalinstitu-tion that typically incarcerates individualsforsentencesoflongerthanoneyear.
Probabilistic record linkageAmethod for linkingcases indifferentad-ministrative databases when no uniqueidentifyingnumberorvariable isavailable.Cases are linked using a combination ofavailabledataelementssuchbirthdatesandfirstand lastnames. Thedegreeof confi-dencethatcanbeplacedinthelinksisde-pendentonsuch factorsas thenumberofvariables that are available to create thelinkages,thedegreeofvariabilityonthosevariables,and thenumberofpersons thatareinthedatabase.
ProbationAcriminalsentencerequiringanoffendertobesupervisedinthecommunityinsteadofbeingdetainedinajailorprison.
Probation officerA criminal justice professional who is pri-marilyresponsibleforsupervisingoffenderswhohavebeen sentenced toprobation inthecommunity.
Problem of the missing denominatorSee:Missing denominator.
Procedural justice or procedural fairnessThe phenomenon, demonstrated by abody of research, in which litigants reactmorefavorablytoanadversejudgmentorpunitive sanction if they believe fair pro-cedureswerefollowedinreachingthede-cision. Greaterperceptionsofproceduraljustice have been associated with betteroutcomesinDTCs.
Process evaluationA systematic study indicating whether aDTC is functioning as originally planned,treating the intended target population,and delivering the types and dosages ofservices that are likely to produce favor-able outcomes.
170
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Program-level performance indicatorsQuantifiablemeasuresof theservices thatwereprovidedintheDTC.Examplesmightincludehowoftencourthearingswereheldor how often participants attended sub-stanceabusetreatment.Thesemayalsobereferredtoasinputs.
Propensity score analysisA statistical procedure thatmaybeusedto control for differences in participantcharacteristics between groups. It in-volves mathematically calculating theprobabilitythatanindividualwouldbeintheDTCgroupasopposedtothecompar-isongroup,ortherelativesimilarityoftheindividualtoonegroupasopposedtotheother. The analysis statistically accountsfor this relative probability when com-paring outcomes between the groups.Advancedstatisticalexpertiseisoftenre-quiredtoimplementandinterpretthere-sultsofthisprocedure.
Proximal outcomesShort term outcomes usually occurringwhile participants are still enrolled in theDTCprogram.Examplesmightincludecoun-selingattendanceorgraduationrates.
Quasi-experimental comparison groupA comparison sample of individuals whodidnotentertheDTCforreasonsthatareunlikelytohaveaffectedtheiroutcomes.Agood example might be drug-involved of-fenderswhowereeligibleforandwillingtoentertheDTCbutcouldnotgetinbecausetherewerenoemptyslotsavailable.
Random assignmentAprocedureforassigningparticipantstodif-ferentgroupsinanunsystematicandunbi-asedmanner,suchasbytheflipofthecoin.Random assignment provides the greatestassurancesthatthegroupsstartedoutwith
anequal chanceof success, and thus thatanypositiveoutcomescanbeattributedtotheeffectsof theDTCprogramandnottoextraneousfactors.
RecidivismThe incidenceofnewcriminalactivityoc-curringafterparticipantsenteredtheDTCor comparison program. This includescriminal activity occurring while partici-pants were enrolled in the program, andafter they graduated or were terminat-ed fromtheprogram. Recidivism ismostcommonly measured by the number orpercentageofnewarrestsornewconvic-tionsafterentry.Itmayalsobereferredtoasre-offending.
Redundancy or redundant variablesThe degree to which predictor variablesarecorrelatedwitheachotherandthusdonotprovideindependentpredictionofout-comes.
ReplicationTheprocessof repeatingor reproducingastudy to ensure the same results are ob-tained.Replicationincreasesthelikelihoodthatthefindingsarevalidandreliable.
Restorative justice interventionsRequirementsthatmaybeimposedonpar-ticipants inDTCstocompensatevictimsorsocietyfortheircrimes. Examplesmayin-cludevictimrestitution,communityservice,victimimpactpanels,andpaymentoffinesor fees.
RewardsConsequencesforbehaviorthataredesiredbyoffenderssuchasverbalpraise,applause,smallgifts,andreductionsinsupervisionre-quirements. Theymayalsobereferredtoasincentives.
171
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Risk factorsCharacteristicsofparticipantsthatpre-dat-ed theirentry into theDTCorcomparisonprogramandreliablycorrelatewithpooreroutcomes. Commonexamplesof risk fac-tors include a younger age, prior failuresin treatmentor rehabilitation,andamoreserious criminal or substance abuse histo-ry. Individualswith such risk factors typi-callyrequiremoreintensiveandstructuredinterventions to succeed in treatmentandrefrainfromcriminalactivity.
Risk assessment toolsStandardized and validated questionnairesorinterviewsthatmeasurethelikelihoodofparticipantsfailingintreatmentorcommit-tinganewoffense.Offenderswhoareiden-tifiedasbeinghighriskonthesetoolstendto have better outcomes in DTCs as com-paredtolessintensivedispositionssuchasprobation.
SanctionsConsequencesforbehaviorthataredislikedby offenders such as verbal reprimands,monetary fines, community service, in-creased supervision requirements, or in-carceration. Theuseofsanctions,andthetypes of sanctions used, varies from juris-dictiontojurisdiction.
SCRAM®Secure Continuous Remote Alcohol Moni-tor. An ankletmonitoringdevice that candetect alcohol in sweat vapors and trans-mitsdatawirelesslytoacentralmonitoringstation.
Secondary analysesDataanalysesthatexamineoutcomesonlyfor subgroups of participants (e.g., gradu-ates only). Secondary analyses are morelikely than intent-to-treat analyses to turnupunreliableorchancefindings.Therefore,
theyshouldordinarilybeperformedonlyifthe intent-to-treat analysis first indicatedsignificantresults.
Short-term outcome evaluationAsystematicevaluationofparticipants’per-formancewhiletheyareenrolledintheDTCprogram. The evaluation typically focusesonduring-treatmentoutcomesthatarelike-ly to predict post-program outcomes suchas graduation rates, treatment attendancerates,andratesofdrug-negativeurinetests.Itmayalsobereferredtoasaproximalorintermediateoutcomeevaluation.
Skewed distributionRefers to scores on a variable that have arestrictedrangeorsimilarvaluesforalargeproportionofparticipants.Manystatisticaltestsassumethatdistributionsarenormalorhaveawiderangeofvalues,andthere-forecannotbeusedtoanalyzedatawithaskewed distribution. The evaluator mightneedtouseaNonparametric statistical test or a Log-linear transformation to compen-satefordatathathasaskeweddistribution.
Starting clockThetimepoint fromwhichdata collectiononperformanceindicatorsbegins.ForDTCprogramevaluations,theclockshouldordi-narilybestartedon thedateofentry intotheDTCorcomparisonprogram.
Statistical imputation proceduresSee:Imputation procedures.
Statistical significance or statisticallysignificant differencesDifferences between groups that have ahigh mathematical probability (usually 95percentorhigher)ofbeingreproducibleinthefuture.Statisticallysignificantdifferenc-espermitgreaterconfidenceinthereliabil-ityofone’sfindings.
172
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Statistical syntaxPre-programmed statistical equations in-structinghowthedatashouldbeanalyzed.Forexample,acomputersystemmighthavea pre-programmed statistical formula forcalculating theaverageageofparticipantsinaDTCprogram.
Substance dependenceApsychiatricdiagnosisreflectingcompulsiveuseofalcoholorotherdrugs. Characteris-tic symptoms include cravings for the sub-stance, uncontrolled usage, or uncomfort-able withdrawal symptoms when levels ofthesubstancedeclineinthebloodstreamorcentralnervoussystem.Seealso:Addiction.
Survival analysisAstatisticalprocedurethatcomparesgroupsofparticipantsontheaveragelengthoftimeuntilaspecifiedeventoccurs,suchasanewcriminalarrestorrelapsetodruguse.
Target populationThe sub-group of drug-involved offenderswhoperformsignificantlybetterinDTCsascomparedtoalternativeprograms,andthusshouldbeprioritizedforentryintoDTCs.
Technical violationsViolationsofacourtorderthatdonotcon-stituteacrimeperse.Forexample,drink-ingalcohol is legalformostadults inmostcountries,butmaybeatechnicalviolationforaDTCparticipantandmayleadtoanar-restorconvictionforaprobationviolation.
Therapeutic allianceThephenomenon,demonstratedbyabodyofresearch, inwhichpatients’perceptionsofthequalityofthetherapist-patientrela-tionship reliablypredictsoutcomes inpsy-chotherapy.TheimpactofthetherapeuticalliancehasnotbeenwellstudiedinDTCs,buthasbeenhypothesizedtobeanimport-
antfactorforsuccessinDTCprograms.Itisalsoreferredtoastheworkingalliance.
Time at libertyTheproportionoftimeduringthefollow-upperiod when participants were relativelyfree in the community to engage in drugabuse,crime,orotherbehaviorsofinteresttoevaluators.Restrictionsonparticipants’timeatlibertytypicallyincludephysicalbar-rierssuchasjailorresidentialtreatment.
Time at riskThe length of time in which participantscould have engaged in drug abuse, crime,or other behaviors of interest to the eval-uator. Generally speaking, the longer thefollow-upperiod,thelongerthetimeatrisk.
VarianceThedegreetowhichparticipantsproduceda range of different scores on ameasure.Forexample,ifallparticipantsarebetweenthe ages of 21 and 23 years, the variancein age is low. For mathematical reasons,it iseasier todetect statistically significantdifferencesbetweengroupswhenthevari-anceonameasureishigh.
Victim impact panelA type of restorative justice interventionthatrequiresoffenderstomeetwithcrimevictimsandlearnhowtheywereaffectedbythecrime.Thegoalistohelpoffendersde-velopempathyandgainperspectiveonhowtheiractionsaffectedotherpeople.
Wait-list comparison groupDrug-involvedoffenderswhowere eligiblefor and willing to enter the DTC but whocould not get in because there were noemptyslotsavailable. Await-listcompari-sonsampleisoftenthebestalternativeaf-ter randomassignment foraDTCprogramevaluation.
173
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Performance Indicators and Performance MeasurementNationalCenterforStateCourts(NCSC)&JusticeProgramsOfficeatAmericanUniversityTranslatingDrugCourtResearchintoPractice(R2P)https://nicic.gov/series/r2p-translating-drug-court-research-practice
Performance Measurement of Drug Courts: The State of the ArtNationalCenterforStateCourts(2008)http://cdm16501.contentdm.oclc.org/cdm/ref/collection/spcts/id/171
Local Drug Court Research: Navigating Performance Measures and Process EvaluationsNationalDrugCourtInstitute(2006)http://www.ndci.org/publications/monograph-series/navigating-perfor-mance-measures-and-process-evaluations
Evaluating Drug Court Programs: An Overview of Issues and Alternative Strategies JusticeProgramsOfficeatAmericanUniversityhttps://jpo.wrlc.org/handle/11204/3306
Introductory Handbook for DWI Court Program Evaluations. NationalCenterforDWICourts(2010)http://www.dwicourts.org/sites/default/files/nadcp/DWI%20Ct%20Eval%20Manual%20REVISED-8-10.pdf
Recidivism 101: Evaluating the Impact of Your Drug Court CenterforCourtInnovation(2005)http://www.courtinnovation.org/sites/default/files/Recidivism_101%5B2%5D.pdf
Management Information Systems (MIS) TreatmentResearchInstituteCourtEvaluationProgram(TRI-CEP)http://triweb.tresearch.org//?s=tricep
DrugCourtCaseManagementSystem(DCCM)AdvancedComputerTechnologieshttp://www.actinnovations.com/Products/DCCM
eCourtSystemU.S.NationalInstituteofDrugAbuse
APPENDIX: RESOURCES FOR DRUG TREATMENT COURTS
174
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
CriminalJustice-DrugAbuseTreatmentStudies(CJ-DATS)http://www.gmuace.org/documents/prod-pub/cjdats/cjdats-summary-ecourt.pdf
Buffalo,NY,DrugCourtCaseManagementSystemhttp://www.ndci.org/[email protected]
Accessing Administrative DatabasesNationalInformationExchangeModel(NIEM)https://www.niem.gov/aboutniem/Pages/niem.aspx.
Staff Surveys of DTC Policies and PracticesU.S.NationalInstituteofDrugAbuseCriminalJustice-DrugAbuseTreatmentStudies(CJ-DATS)NationalDrugCourtSurveyhttp://www.gmuace.org/documents/prod-pub/cjdats/cjdats-summa-ry-drug-court.pdf
U.S.NationalInstituteofJustice MultisiteAdultDrugCourtEvaluation(MADCE)
https://www.ncjrs.gov/pdffiles1/nij/grants/237110.pdf(AppendixA)
NPCResearch,Inc.AdultDrugCourtTypologyInterviewhttp://npcresearch.com/wp-content/uploads/Drug-Court-typology-guide-NPC-Research-01-26-04-copyrighted.pdf
Rating Scales of Correctional Program QualityCorrectionalProgramAssessmentInventory(CPAI)http://www.state.in.us/idoc/files/CPAI_overview.ppt
CorrectionalProgramChecklist(CPC)https://www.uc.edu/content/dam/uc/gencounsel/docs/CPC%20Training%20MOU%2011.12.14.pdf
Risk-Assessment InstrumentsLevelofServiceInventory-Revised(LSI-R)* † ‡https://ecom.mhs.com/(S(zhkd5d55qlwc3lr2gzqq5w55))/product.aspx-?gr=saf&prod=lsi-r&id=overview
175
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
RiskandNeedsTriage(RANT)http://www.trirant.org/
CorrectionalOffenderManagementProfilingforAlternativeSanctions(COMPAS)https://www.cdcr.ca.gov/rehabilitation/docs/FS_COMPAS_Final_4-15-09.pdf
OhioRiskAssessmentSystem(ORAS)https://www.uscourts.gov/sites/default/files/74_1_2_0.pdf
FederalPostConvictionRiskAssessment(PCRA)http://www.uscourts.gov/FederalCourts/ProbationPretrialServices/Supervi-sion/PCRA.aspx RiskPredictionIndex(RPI)https://www.fjc.gov/content/328578/fjc-50th-improving-criminal-justice-ad-ministration-and-sentencing-federal-courts WisconsinRiskandNeedAssessmentScale(WRN)https://csgjusticecenter.org/corrections/projects/wisconsin-state-risk-assess-ment-validation/
Clinical-Assessment InstrumentsAddictionSeverityIndex(ASI) * † ‡TreatmentResearchInstitutehttp://triweb.tresearch.org/index.php/tools/download-asi-instruments-manu-als/
GlobalAppraisalofIndividualNeeds(GAIN) * GAINCoordinatingCenterhttp://gaincc.org/instruments/
AlcoholUseandAssociatedDisabilitiesInterviewSchedule(AUDADIS)* †U.S.NationalInstituteofAlcoholismandAlcoholAbusehttp://www.ncbi.nlm.nih.gov/pubmed/10606491
DrugAbuseScreeningTest(DAST) * †http://www.camh.ca/en/education/about/camh_publications/Pages/drug_abuse_screening_test.aspx
MultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)
176
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
StructuredClinicalInterviewfortheDSM-IV(SCID) * http://www.scid4.org/
TexasChristianUniversity(TCU)DrugDependenceScreen(availableinEnglish,Spanish,Russian,Chinese(simplifiedandtraditional),andVietnamese)* https://ibr.tcu.edu/forms/tcu-drug-screen/
Procedural Justice / Procedural Fairness ScalesMultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)
ProceduralJusticeQuestionnaire(PJQ)TreatmentResearchInstitutehttp://triweb.tresearch.org/index.php/about-us/contact-us/
Perceived Deterrence ScalesMultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)
PerceivedDeterrenceQuestionnaire(PDQ)TreatmentResearchInstitutehttp://triweb.tresearch.org/index.php/about-us/contact-us/
Motivation / Readiness for Change ScalesInventariodeProcesosdeCambio* †EuropeanMonitoringCentreforDrugsandDrugAddictionhttp://www.emcdda.europa.eu/html.cfm/index3662EN.html
PersonalConcernsInventory(PCI)http://pubs.niaaa.nih.gov/publications/AssessingAlcohol/InstrumentPDFs/47_PCI.pdf
UniversityofRhodeIslandChangeAssessment(URICA)https://elcentro.sonhs.miami.edu/research/measures-library/urica/index.htmlhttps://www.ncbi.nlm.nih.gov/books/NBK64976/table/A62309/
MultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)
177
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
Circumstances,Motivation,Readiness,andSuitabilityScales(CMRS)http://www.ncbi.nlm.nih.gov/pubmed/7832182
Satisfaction with the Program / Treatment ScalesTreatmentPerceptionQuestionnaire(TPQ)EuropeanMonitoringCentreforDrugsandDrugAddictionhttp://www.emcdda.europa.eu/html.cfm/index4322EN.html
ClientSatisfactionQuestionnaire(CSQ)http://www.csqscales.com
ClientAssessmentofTreatmentScale(CATS) *http://www.ncbi.nlm.nih.gov/pubmed/21342706
Therapeutic Alliance ScalesHelpingAllianceQuestionnaire-II(HAQ-II)http://www.med.upenn.edu/cpr/instruments.html
WorkingAllianceInventoryhttp://wai.profhorvath.com/
Observer Rating Scales for Court SessionsMultisiteAdultDrugCourtEvaluation(MADCE)https://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf
On-Site Training and Technical Assistance InstitutionalStrengtheningandIntegralProgramsSectionInter-AmericanDrugAbuseControlCommission(CICAD)OrganizationofAmericanStateshttp://www.cicad.oas.org/Main/Template.asp?File=/fortalecimiento_insti-tucional/default_eng.asp
U.S.NationalDrugCourtInstitute(NDCI)http://www.ndci.org/training/advanced-training
DrugCourtClearinghouseandTechnicalAssistanceProjectatAmericanUni-versityhttps://www.american.edu/spa/jpo/initiatives/drug-court/
178
MAN
UAL
FO
R SC
IEN
TIFI
C M
ON
ITO
RIN
G A
ND
EVA
LUAT
ION
On-Line Webinars and Distance Learning Programs U.S.NationalDrugCourtInstitute(NDCI)http://www.ndci.org/training/online-trainings-webinars
NationalDrugCourtResourceCenter(NDCRC)https://www.ndci.org/resources/training/
CenterforCourtInnovationhttp://drugcourtonline.org/
NationalInstituteofJustice,NationalCenterforStateCourts(NCSC)&JusticeProgramsOfficeatAmericanUniversityTranslatingDrugCourtResearchintoPractice(R2P)https://www.nij.gov/topics/courts/drug-courts/pages/research2practice.aspx
Evidence-Based Treatments for Substance AbuseNationalRegistryofEvidence-BasedProgramsandPractices(NREPP)U.S.SubstanceAbuseandMentalHealthServicesAdministrationhttps://www.samhsa.gov/nrepp
Training on Research EthicsU.S.NationalInstitutesofHealthOfficeofResearchIntegrityhttp://phrp.nihtraining.com/users/login.php
*Spanishtranslationavailable.
†ValidatedamongindividualsofHispanicorLatinoethnicity.
‡ValidatedinatleastoneSouthAmericanorCaribbeanNation.
MANUAL FOR SCIENTIFIC MONITORING AND EVALUATIONDrug Treatment Courtsin the Americas
Man
ual f
or S
cien
tific
Mon
itor
ing
and
Eval
uati
on D
rug
Trea
tmen
t Cou
rts
in th
e A
mer
icas
ISBN 978-0-8270-6830-8
Organization of American StatesSecretariat for Multidimensional Security
Inter-American Drug Abuse Control Commission
This publication was prepared by the Executive Secretariat of the Inter-American Drug Abuse
Control Commission (ES-CICAD), Secretariat for Multidimensional Security of the Organization of
American States (SMS/OAS). It was developed under the aegis of the Drug Treatment Court
Program for the Americas, an initiative coordinated by ES-CICAD/SMS/OAS. This publication was
made possible through the financial contribution of the Government of the United States of
America, through the Bureau of International Narcotics and Law Enforcement Affairs (INL) of the
U.S. Department of State, and of the Government of Canada, through the Anti-Crime Capacity
Building Program (ACCBP) of Global Affairs Canada. The contents expressed in this document are
presented exclusively for informational purposes and do not necessarily represent the opinion
or official position of the Organization of American States, its General Secretariat, or its Member
States.