manual for scientific monitoring and evaluation … · oas cataloging-in-publication data...

MANUAL FOR SCIENTIFIC MONITORING AND EVALUATIONDrug Treatment Courtsin the Americas

Man

ual f

or S

cien

tific

Mon

itor

ing

and

Eval

uati

on D

rug

Trea

tmen

t Cou

rts

in th

e A

mer

icas

ISBN 978-0-8270-6830-8

Organization of American StatesSecretariat for Multidimensional Security

Inter-American Drug Abuse Control Commission

This publication was prepared by the Executive Secretariat of the Inter-American Drug Abuse

Control Commission (ES-CICAD), Secretariat for Multidimensional Security of the Organization of

American States (SMS/OAS). It was developed under the aegis of the Drug Treatment Court

Program for the Americas, an initiative coordinated by ES-CICAD/SMS/OAS. This publication was

made possible through the financial contribution of the Government of the United States of

America, through the Bureau of International Narcotics and Law Enforcement Affairs (INL) of the

U.S. Department of State, and of the Government of Canada, through the Anti-Crime Capacity

Building Program (ACCBP) of Global Affairs Canada. The contents expressed in this document are

presented exclusively for informational purposes and do not necessarily represent the opinion

or official position of the Organization of American States, its General Secretariat, or its Member

States.

OAS Cataloging-in-Publication Data

Inter-American Drug Abuse Control Commission. Manualforscientificmonitoringandevaluation[of]DrugTreatmentCourtsintheAmericas.v.;cm(OAS.Officialrecords;OEA/Ser.L/XIV.6.61)ISBN978-0-8270-6830-81.Drugaddiction--Treatment.2.Substanceabuse--Treatment.3.Drugaddiction--America.4.Drugcourts.I.Title.II.Title:DrugTreatmentCourtsintheAmericas.III.Marlowe,DouglasB.IV.DrugTreatmentCourtProgramfortheAmericas.V.OrganizationofAmericanStates.SecretariatforMultidimensionalSecurity.VI.Series.OEA/Ser.L/XIV.6.61

From the Organization of American States

LuisAlmagro,SecretaryGeneral

AmbassadorNestorMendez,AssistantSecretaryGeneral

Dr.FarahUrrutia,SecretaryforMultidimensionalSecurity

AmbassadorAdamE.Namm,ExecutiveSecretary,Inter-AmericanDrugAbuseControlCommission(CICAD)

Principal AuthorDr.DouglasB.Marlowe

Chief EditorsDr.DouglasB.MarloweAntonioLombaMaurandi

Co-AuthorsDr.ShannonM.CareyDr.MichaelW.Finigan

Project DirectorAntonioLombaMaurandi

ContributorsCaroline CooperDr.CharlotteColmanYacsire CutlerAnnmarie DookieFranklinGonzalezAlfredoGonzalezPortilloHon.JusticeStephaneJackson-HaisleyJavieraIbacacheDr.NohemiLiraDr. Mariano Montenegro ConradoQuinteroMichaelRempelJeffreyE.Zinsmeister

Editorial CommitteeJavieraIbacacheDr.SofiaCoboDr. Francisco CumsilleTaniaGordilloLuisaFernandaNeiraLuisSuarez-IsazaKarellyVillanuevaPamelaI.WestfallJeffreyE.Zinsmeister

Graphic DesignAlejandroAhumadaMaria Montas

ISBN978-0-8270-6830-8

COPYRIGHT©(2019)OrganizationofAmericanStates.AllrightsreservedunderInternationalandPanamericanConventions.Thispublicationmaybereproducedprovidedcreditisgiventothesource.

ThispublicationwaspreparedbytheExecutiveSecretariatoftheInter-AmericanDrugAbuseControlCommission(ES-CICAD),SecretariatforMultidimensionalSecurityoftheOrganizationofAmericanStates(SMS/OAS).ItwasdevelopedundertheaegisoftheDrugTreatmentCourtProgramfortheAmericas,aninitiativecoordinatedbyES-CICAD/SMS/OAS.ThispublicationwasmadepossiblethroughthefinancialcontributionoftheGovernmentoftheUnitedStatesofAmerica,throughtheBureauofInternationalNarcoticsandLawEnforcementAffairs(INL)of theU.S.DepartmentofState,andof theGovernmentofCanada, through theAnti-CrimeCapacityBuildingProgram(ACCBP)ofGlobalAffairsCanada.ThecontentsexpressedinthisdocumentarepresentedexclusivelyforinformationalpurposesanddonotnecessarilyrepresenttheopinionorofficialpositionoftheOrganizationofAmericanStates,itsGeneralSecretariat,oritsMemberStates.

ACKNOWLEDGEMENTS

SpecialthankstoCICADAssistant

ExecutiveSecretaryAngelaCrowdy,

forhertirelesssupportofCICAD’s

missionandOASmemberstates;to

AmbassadorPaulSimons,forinitiating

thedialoguethatledtothedrafting

ofthisdocumentduringhisserviceas

CICADExecutiveSecretary;toDouglas

Boyd,forhisresearchassistancein

thedevelopmentofthismanual;to

theNationalAssociationofDrugCourt

Professionals,theCanadianAssociation

ofDrugTreatmentCourtProfessionals,

andFundaciónPazCiudadana,for

theircontinuedsupport;andtothe

participantsinourworkshopsinThe

Bahamas,Chile,andMexico,whose

contributionswereinvaluableforthe

creationofthismanual.

TABLE OF CONTENTS

A Welcome from the Organization of American States ......................9

Foreword ....................................................................................... 11

Background ..................................................................................... 15WhyWereDrugTreatmentCourtsCreated?.............................15AboutThisEvaluationManual..................................................18

Logic Model for DTC Program Evaluations ..................................... 21

Types of Program Evaluations ........................................................ 25ProcessEvaluations ................................................................ 25Short-TermOutcomeEvaluations.............................................28Long-TermOutcomeEvaluations.............................................29Cost Evaluations ..................................................................... 30

Sources of Information ................................................................. 32ProgramDocumentation..........................................................33StaffSurveys.............................................................................34ParticipantFocusGroups........................................................35ParticipantSurveys .................................................................36ParticipantRecords..................................................................36SiteObservations....................................................................37CollateralSources ...................................................................37

Operationalizing Variables ............................................................. 39Measurement Precision ......................................................... 42

What to Measure ............................................................................ 45ParticipantCharacteristics.......................................................45PerformanceIndicators............................................................48PerformanceBenchmarks........................................................50

Calculating Performance Indicators .............................................. 52CourtSupervision....................................................................59

SubstanceUseTreatment.........................................................60DrugandAlcoholTesting..........................................................64Abstinence ............................................................................. 65RetentionandGraduation........................................................67Recidivism...............................................................................70Incarceration .......................................................................... 73TimelinessofServices...............................................................75ProbationSupervision..............................................................76RewardsandSanctions.............................................................78RestorativeJusticeInterventions..............................................80Employment ............................................................................81Education ............................................................................... 83Housing .................................................................................84EmotionalHealth....................................................................85MedicalandDentalHealth.......................................................87Family Relationships .............................................................. 88SocialRelationships.................................................................89BirthofDrug-FreeBabies........................................................90

Attitudinal Variables ....................................................................... 91ParticipantAttitudes................................................................91StaffAttitudes......................................................................... 94

Observer Rating Scales ................................................................... 95

Comparison Groups ....................................................................... 96RandomAssignment...............................................................96Quasi-ExperimentalComparisonGroups.................................98Matched Comparison Groups ............................................... 99PropensityScoreAnalyses.....................................................100InvalidComparisonGroups....................................................101

Intent-to-Treat Analyses .............................................................. 102Cohorts..................................................................................103

Follow-Up Windows .................................................................... 104StartingtheClock...................................................................105TimeatRisk............................................................................106TimeatLiberty........................................................................107

Avoiding Common Analytic Mistakes .......................................... 109TargetPopulation...................................................................109DisadvantagedGroups..........................................................111

AdjustingforBaselineDifferences..........................................112ImpactofSpecificServicesor Interventions .......................114InfrequentEvents..................................................................116

Cost Analyses .............................................................................. 118CostAssessmentTechniques.................................................122Conclusion-CostAnalyses......................................................137

Selecting Statistical Consultants ................................................. 138

Management Information Systems ............................................ 140RecommendedMISFeatures..................................................141

Accessing Administrative Databases ........................................... 146ReachingConsensus..............................................................147Linking Records .................................................................... 148TransferringData...................................................................150EthicsandLegalTraining........................................................150

Closing Comments ........................................................................ 151

References ................................................................................ 153

Glossary of Technical Terms ........................................................ 163

Appendix: Resources for Drug Treatment Courts ........................ 173PerformanceIndicatorsandPerformanceMeasurement....173ManagementInformationSystems(MIS)..............................173AccessingAdministrativeDatabases ....................................174StaffSurveysofDTCPoliciesandPractices.............................174RatingScalesofCorrectionalProgramQuality......................174Risk-Assessment Instruments .............................................. 174Clinical-AssessmentInstruments..........................................175ProceduralJustice/ProceduralFairnessScales......................176PerceivedDeterrenceScales...................................................176Motivation/ReadinessforChangeScales...............................176SatisfactionwiththeProgram/TreatmentScales...................177TherapeuticAllianceScales...................................................177ObserverRatingScalesforCourtSessions..............................177On-SiteTrainingandTechnicalAssistance...............................177On-LineWebinarsandDistanceLearningPrograms............178Evidence-BasedTreatmentsforSubstanceAbuse.................178TrainingonResearchEthics.....................................................178

“Scientificevidenceshouldunderpin

thedevelopmentofdrugpolicy:Inthe

past,toomanypublicpolicydecisions

ondrugsweretakenbasedon‘good

intentions,’ratherthanonscientific

evidence,andsetunrealisticcompliance

goals.Nowourmemberstatesseek

toestablishandimplementevidence-

baseddrugpoliciesandactionswith

attainablegoals.Theyarealsoaware

thattoachievethisstatus,theyneedto

buildinstitutionscapableofdeveloping

objective,reliable,andcomparative

information.”

—CICADExecutiveSecretariat,

IntroductiontotheHemisphericDrug

Strategy,2010

9

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

DearReader,

Thank you for your interest in this guide on drug treatment court (DTC)

monitoring andevaluation.Wecommissioned itspublicationbecause theSecre-

tariatforMultidimensionalSecurity(SMS)seekstoguaranteegreatersecurityfor

communitiesintheWesternHemisphere.Inordertodoso,wemustaddressthe

effectsoftheworld’sdrugprobleminourhemisphere.

Ourmissiontoassess,prevent,confront,andrespondeffectivelytosecurity

threatsincludesamandatetoseeknewsolutionsforreducingandpreventingcrime,

andforprotectinghumanrights.IntheWesternHemisphere,aclearrelationship

existsbetweencrime,insufficientprotectionofhumanrights,andcorruption.In

theareaofdrugpolicy,theseissueshaveallowedcriminalorganizationsthattraffic

drugsandconductotherillicitactivitiestoprosper.

Consequently, the OAS, through the Inter-American Drug Abuse Control

Commission(knownbyitsSpanishlanguageacronym,CICAD)ofSMSsupportsand

promotesevidence-baseddrugpoliciesthatreducecrime,protecthumanrights,

focusonthewell-beingoftheperson,anddonotcharacterizethedruguserasan

objectofthecriminaljusticesystem.Insodoing,CICADrecognizestheneedto

promotedrugpoliciesanddialoguethatfocusonpublichealthandhumanrights,

while taking into considerationevidence-basedpolicies, regulatory frameworks,

civilsocietyparticipation,andgender.

TheDTCmodelisanexcellentexampleofthistypeofevidence-baseddrug

policy.OvertwodecadesofrobustacademicinquirydemonstratesthatwhenDTC

programsfollowevidence-basedpractices,theyareeffectiveinreducingcriminal

recidivism,savingpublicfunds,protectinghumanrights,andhelpingparticipants

A WELCOME FROM THE ORGANIZATION OF AMERICAN STATES

10

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

recover fromdebilitating substance use disorders. Itmay thus not come as a

surprisethatfifteenOASmemberstateshaveeitherimplementedtheDTCmodel

orareexploringitsimplementation.Moreover,DTCsareconsistentwiththeOAS

HemisphericDrugStrategyandcorrespondingPlanofActiononDrugs2016-2020,

servingasanalternativetocriminalprosecutionorimprisonment,andasasafe-

guardofhumanrights.

Nonetheless, as the term “evidence-based” suggests, the continued

success of theDTCmodel dependson sustained, rigorous scientificmonitoring

and evaluation by practitioners and the research community. This guide exists

to support these efforts, and tomake the fruits of over 20 years of academic

experience readily available to new researchers—especially those in countries

thatare intheprocessofadaptingthemodeltotheirownnationalcontexts. It

iswrittenby someof thebest and thebrightest in thefield, and covers every

majoraspectofDTCprocessesandoutcomes.Ihopeyouwillfinditasusefulasits

author,contributors,editors,andfundershavedesignedittobe.

IextendmysinceregratitudetoCanadaandtheUnitedStatesofAmerica

for providing the resources to write and publish this guide. Their citizens’

commitment to, and successwith, theDTCmodelhas servedasan inspiration

acrosstheHemisphere.

Dr. Farah UrrutiaSecretary for Multidimensional Security

11

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

FrommyvantagepointastheExecutiveSecretaryofCICAD,drugtreatment

courts (DTCs) are one of the most important programs we have, in strict

accordance with the OAS Hemispheric Drug Strategy and corresponding Plan

ofActiononDrugs2016-2020.Thesetwofundamentaldocumentscall forthe

means of offering treatment, rehabilitation, and recovery support services to

drug-dependentcriminaloffendersasanalternativetocriminalprosecutionor

imprisonment,andinfullrespectofhumanrights.

Asitstitleindicates,themanualyouarereadingexiststoteachresearchers

howtomonitorandevaluateDTCprogramsmoreeffectively.Tothelayreader,a

needtomonitorandevaluatemightsuggestacriticalproblemwiththemodelthat

needsfixing.InthecaseofDTCs,however,therealityisverymuchthecontrary.

TheneedtomonitorandevaluateDTCsstemsfromtheirsuccess.Asyouwill

seeinthefollowingpages,theDTCmodelrestsuponfoundationsofempiricism

andresearch.Lookingbacktothemodel’sinceptionin1989,wedonotfindlofty

theoreticiansdesigningprogramsperchedatopanivorytower.Instead,thein-

trepidprofessionalswhocreatedthefirstDTCsdidsofromthefield,responding

toarealproblem:theywerepractitionersmiredinthecrackcocainecrisisofthat

era.Confrontedbyawaveofcrime—fueledbysubstanceabuse—thatthreatened

tooverwhelmthecriminaljusticesystem,thesemenandwomenweredesperate

foranalternativeresponsetoaseriousdilemma.

Theysawthesameoffenders—whosesubstanceusedisorderswerethepri-

mary factor driving their criminal activity—pass through their courts again and

again.Jailwasclearlynotresolvingthatunderlyingissue.Thesedefendantswould

serve their time, but their substance abuse problem always followed them,

FOREWORD

12

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

shadow-like, back into civilian life. Predictably, many of them committed new

crimes,werere-arrested,andsentbacktojail.Meanwhile,theirsubstanceabuse

problemremaineduntreatedandunabated.

Touseaconcreteexample,considersomeonewhostealsbicyclestopayfor

adrughabit.Sendingthispersonto jail leavesthesubstanceabuseproblem

untreated—andsometimesevenundetected.Heorsheservesthesentence,ends

upbackonthestreetwiththesameneedtobuydrugsasbefore,andthevicious

cyclecontinues.

Faced with this seemingly intractable problem, these local practitioners

forged a novel solution using tools they had on hand: the criminal justice and

healthsectorswouldcometogethertooffervoluntary,court-supervisedsubstance

abusetreatmenttosubstance-dependentdefendantsinterestedinparticipating.

The concept behind this new, more collaborative approach was straight-

forward:providetreatmentandrehabilitationinsteadofexactingretribution.Of

course, putting the model into practice was more challenging. There was

noprecedentorguidelinestofollow.Thetaskofcoordinatingpractitionersfrom

thedifferentworldsofcriminaljusticeandpublichealthwasnotsimple.Judges,

prosecutors,defenseattorneys,doctors,andsocialworkershadtolearntowork

togetherasateamtohelptheDTCparticipants,insteadofseeingoneanotheras

adversaries.

Assuch,at itsoutsetthemodelranonimprovisationandgoodwill.Practi-

tionersprogressively improved themodel, keepingwhatworkedandmodifying

whatcouldbeimproved,buttheyobservedthatitseemedtofunctionoverall.DTC

graduatesappearedtodobetterthanthosewhowenttojail.

Such initial success drove an ever-greater interest in this new alternative

to incarceration. By the mid-1990s, the research community had taken note.

ResearchersappliedtheirtalentstomorerigorousevaluationoftheDTCmodel,

13

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

tosee ifpractitioners’anecdotalobservationshelduptoscientificscrutiny.This

applicationofthescientificmethodprovedpowerful.Moreformalevaluationof

themodelhasteneditsrefinementandsolidifieditsevidentiarysupport.

By2006,thescientificcommunityhadconcluded,fromadvancedresearch

techniques called meta-analyses, that the DTCmodel indeed reduced criminal

recidivism. Other studies also showed that the model also saved money over

traditionalcriminaljusticeapproaches.Moreover,researchfromothernationslike

CanadaandChile,whichhadadaptedtheDTCmodeltotheirowncircumstances,

begantoconfirmitsusefulnessinotherjurisdictionsandnationalcontexts.Bythe

closeofthatdecade,moreresearchhadbeenpublishedontheeffectsofadult

DTCsthanonvirtuallyallothercriminaljusticeprogramscombined.

Sincethen,thescientificcommunityhasbegunconsolidatingtheavailable

research into guides for practitioners, such as theNationalAssociationofDrug

CourtProfessionals’two-volumeAdultDrugCourtBestPracticeStandards.Such

documents serve a key role in ensuring that the results of very sophisticated

researchdonotendupforgotteninadustyacademicarchive,butinsteaddirectly

improvehowDTCpractitionersworkwithprogramparticipants.

Thatbringsustothedocumentyouarereadingrightnow.Forthefirsttime,

thescientificcommunityispresentingacomprehensiveguideabouttheeffective

studyofDTCsforusebypractitioners,academics,andevaluators.Thisservestwo

importantpurposes. First, continued research isessential tokeeping theDTC

modelrobustandfunctional.ThesuccesstheDTCmodelenjoysrestsonem-

piricism.Thisvolumerepresentsa“how-to”guidetospurmoreempiricalresearch.

Second,asmorecountriesadapttheDTCmodel totheirparticularneeds, their

ownacademiccommunitieswillneedtomonitorandevaluateprocessesandout-

comes.Thisguidewillacceleratethelearningcurveforresearchersthatarenew

tothefield.

14

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Inotherwords, thisdocumentdistillsmore than twodecadesof scholarly

effortintoonepubliclyavailable,accessiblevolume.Weaskedsomeofthefield’s

mostaccomplishedfigurestoserveasitsauthorandcontributors.Theseexperts

havecoveredeffectivemonitoringandevaluationpracticesfromstarttofinish.

Assuch,onbehalfofCICAD,Ihumblypositthatifyouwishtolearnhowto

monitorandevaluateDTCs,youhavecometo the rightplace—and Ihope that

whenyoufinish,youwillagreethatthisguidewasworththetimeyouinvestedin

readingit.

Ambassador Adam E. NammExecutive Secretary

Inter-American Drug Abuse Control Commission (CICAD)

15

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Why Were Drug Treatment Courts Created?

Drugtreatmentcourts(DTCs)werecreatedtoaddresstheproblemofcrimethat is

drivenbyunderlyingsubstanceusedisorders.Substanceusedisordersareamajorcon-

tributortocrimeintheWesternHemisphereandaroundtheglobe(OAS,2010).Reduce

drugandalcoholconsumption,andcrimedeclines; increaseit,andcrimerises(Chandler

etal.,2009;Hollowayetal.,2006).Yetdespiterisingincarcerationratesfordrug-related

crimesintheAmericas,consumptionanddistributionofillegaldrugshascontinuedtorise

precipitously(OAS,2011,2012a).

DTCsemergedasonealternativetoincarcerationdesignedtoaddressthisissue.DTCs

emphasize intensive community-based substance use treatment and supervision of

addictedindividualschargedwithdrug-relatedoffenses,underjudicialoversight.TheDTC

judgeleadsamultidisciplinaryteamofprofessionalsthatcommonlyincludesrepresenta-

tives from the prosecutor’s office, defense bar, treatment agencies, police, andproba-

tiondepartment.Theteammembersmeetfrequentlytoreviewparticipants’progressin

treatmentandmakerecommendationstothe judgeaboutsuitableresponsesto impose

forprogressorlackthereofintreatment.Theseresponsesmayinclude:(A)rewardssuch

asverbalpraise,reducedsupervisionrequirementsorsmallprizesforproductiveachieve-

ments;(B)sanctionssuchasverbalreprimands,communityserviceorbriefintervalsofjail

detentionforinfractions;or(C)adjustmentstoparticipants’treatmentregimensinlightof

insufficientclinicalprogress.

Additionally, participants are tested for drug and alcohol use on a random basis,

are required to complete a substance use treatment program, and must often satisfy

restorativejusticerequirements,suchasmakingrestitutiontovictims.Inpre-adjudication

DTCs,successfulgraduateshavetheircriminalcharge(s)droppedorwithdrawn,andinpost-

BACKGROUND

16

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

adjudicationDTCs,graduatesmayavoidincarcerationorreducethelengthorconditionsof

probation.

Thisapproachhasproventoworkempirically.Extensiveresearchonthemodeland

itseffects indicates thatmostDTCshavegenerated significant reductions in recidivism

anddruguse,savingsfortaxpayers,andreducedvictimizationfromavoidedcrimes.For

example,rigorousstudiesinAustralia(Jones,2011);Canada(Latimeretal.,2006;Somers

et al., 2011); and theUnited States (Mitchell et al., 2012; Rossman et al., 2011) have

demonstratedthatDTCscansignificantlyreducecrime,promoterecoveryfromaddiction,

andproducesignificantcostbenefitsmoreeffectivelythantraditionalcriminaljustice

approaches. Therefore, althoughDTCsarenot theonly solution to crime fueledby sub-

stanceusedisorders, theydo represent apromising, evidence-basedmodel that canbe

adopted,adapted,andevaluatedinotherlocalandnationalsettings.

Thisstrongempirical foundationhasdrivenasignificantexpansionofDTCs inthe

United States since thefirst suchprogramwas founded in 1989 inMiami-DadeCounty,

Florida.Asof2015,thereweremorethan3,100DTCsintheUnitedStates(NationalInsti-

tuteofJustice,2018),andthatgrowthwilllikelycontinue.A2017reportbythePresident’s

CommissiononCombatingDrugAddictionandtheOpioidCrisisrecommendedfurtherex-

pansionandfundingforDTCsatthefederallevel,andsupportforDTCsatthestatelevel

remainsstrong(President’sCommissiononCombatingDrugAddictionandtheOpioidCri-

sis,2018).

The success of DTCs in the United States has also created demand for themodel

internationally.Sincetheturnofthecentury,othernationshaveadaptedtheDTCmodel

forimplementationintheirownparticularlegal,social,andpoliticalcontexts.Startingin

the late 1990s and into the following decade, Canada, Bermuda, Cayman Islands, Chile,

Jamaica,andMexicostartedDTCpilotprojects.By2010,DTCshadexpandedtothepoint

where theOrganizationofAmerican States, through the Inter-AmericanCommission for

DrugAbuseControl(CICAD),launchedtheDrug Treatment Courts Program in the Americas,

17

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

withtheaimofsupportingtheexpansionofthemodeltoothermemberstatesthatrequest

assistance.

This CICAD programhas supported other nations’ adoption and adaptation of the

model.By2019,atleastfifteennationsandtwoterritoriesintheAmericashadexplored,

developed,orimplementedsometypeofDTCmodel:Argentina,Barbados,Belize,Bermuda,

Canada,CaymanIslands,Chile,Colombia,CostaRica,DominicanRepublic,Guyana,Jamaica,

Mexico,Panama,Peru,UnitedStates,andTrinidadandTobago. TheDTCmodelhasalso

spreadacrosstheoceantonationsinothercontinents.

InmostnationsoutsidetheUnitedStates,DTCprogramsarestill intheirformative

stages,andeffortstoevaluatetheiroutcomeshaveonlyrecentlybeeninitiated.Nonethe-

less,initialresultsarepromising.AsurveyconductedbyAmericanUniversityonbehalfof

theCICAD/OASanalyzedresponsesfromofficialsinseveralOASmemberstates,including

Canada,Mexico,Chile,Suriname,Bermuda,andJamaica(Cooperetal.,2010).Themajority

ofrespondentsreportedthatDTCsintheircountryappearedtoreducecrimebetterthan

traditionalcorrectionaldispositions,andapproximatelyhalfof therespondentsreported

achievingnotablecostsavings.A2018impactevaluationfromChile,forexample,showed

thatDTCstherereducedrecidivismbyalmost30percentcomparedtoacontrolgroup,and

a2014studyreportedsavingsofalmost75percent (US$500perparticipantpermonth)

(FundaciónPazCiudadana,2018;FundaciónPazCiudadana2014).Thesefigures suggest

thatimplementingDTCsinSouthAmericanandCaribbeannationsisfeasibleandpotentially

desirable.Inaddition,arecentprocessevaluationofthetreatmentcourtinNuevoLeón,

Mexico,concludedthattheprogramdelivershighquality,evidence-basedservices,andwas

perceivedpositivelybyparticipants(Rempeletal.,2014).

Thisexpansionandinteresthascontinuedthroughthepresentday.Consequently,asmore

andmorecountrieshaveexpressedinterestincollaboratingwiththeOrganizationofAmerican

Statesanditspartners,ithasbecomeincreasinglymoreimportanttopromotethedevelopment

ofmonitoringandevaluationcapabilitiesnecessarytosustainevidence-basedDTCprograms

18

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

(Rempel,et.al.2014).Thatneedistheprimarymotivationforthismanual,whichaimstoassist

professionalsacrosstheworldwithunderstandinghowtomonitorandevaluateDTCprograms

effectively.

About This Evaluation Manual

Inthesummerof2010,theInter-AmericanDrugAbuseControlCommission(CICAD)

oftheOrganizationofAmericanStates(OAS)adoptedtheHemisphericDrugStrategywhich,

amongotherprovisions,encouragesmemberstatestodevelopalternativestoincarceration

forindividualsthatsufferfromasubstanceabusedisorderandwhohavebeenchargedwith

crimesrelatedtothatdisorder(OAS,2010).Sincethen,andasoneofthosealternatives,by

therequestofagrowingnumberofOASmemberstates,ES/CICAD/OAShasbeenoffering

trainingandtechnicalassistancetohelpplan,implement,andevaluatenewDrugTreatment

Court(DTC)programs.

Subsequently,representativesfromseveralOASmemberstatesconvenedtoestablish

apreliminary framework forevaluatingDTCs (OAS,2012b).A consensuswas reachedat

thismeetingthateachcountryshouldevaluate itsDTCprogramstoensuretransparency

andaccountability,identifyeffectiveandineffectivepractices,attractexternalfunding,and

promotethemeritsof theDTCmodel. Itwas furtheragreedthatmemberstatesshould

consider collectinga commondatasetof coreperformance indicatorswhichcould serve

asthebasisformonitoringDTCactivitiesandimpactsintheAmericas.Theseperformance

indicators should be SMART (i.e., Specific,Measurable, Achievable, Relevant, and Time-

bound).Theyshouldalsobevalue-neutralbecausebenchmarksforsuccesshavenotyet

beenestablishedformanycountries;therefore,thereisnobasisforknowingwhatlevelsof

performanceshouldbeexpectedfromprogramsinthosecountries.Collectingperformance

datawillenableOASmemberstatestodevelopandvalidateperformancebenchmarksfor

theirprograms.

19

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Intheyearsthatfollowed,CICAD/OASledanefforttocreateandconsolidateguid-

anceforDTCmonitoringandevaluation.Throughdialoguewithexpertsandmemberstates,

andcarryingoutprocessevaluationsinsevennations,aconsensuswasdevelopedoverthe

stateofexistingresearchandcertaincoreperformanceindicators,andwrittendowninthis

manual. Themanualwasthensubjectedtoextensivepeerreviewbyexpertsfromboth

governmentandcivilsocietyorganizations.

AlthoughnobenchmarksforsuccesshaveyetbeenestablishedformanyOASmember

states,itwasagreedthatthismanualshouldpresentlessonslearnedinAustralia,Canada,

andtheUnitedStates.Performancebenchmarksfromthesethreenations,whilenotsome

sort of “gold standard” for other OAS member states to copy blindly, may still serve

as useful startingpoints for evaluators to considerwhen creating and testing their own

scientifichypotheses.Theseempiricalfoundationstookdecadesofpainstakingobservation

andresearch,asisthecasewithmanynewconcepts.Forexample,althoughthefirstDTC

wasfoundedin1989,itwasnotuntil2005(sixteenyearslater)thattheU.S.government

agencyresponsibleforevaluatingthesuccessoffederally-fundedprogramsconcludedthere

wassufficientscientificevidencetoprovethatDTCsreducedcrime(U.S.Government

AccountabilityOffice,2005).Itwasnotuntil2011,twenty-twoyearsafterthefirstDTCwas

created,thatresearchersintheUnitedStatesconcludedthatDTCsproducedotherbenefits

beyondreducingcrime,suchasreducingsubstanceuseandfamilyconflict(Rossmanetal.,

2011).Anditwasonlyin2012thatscientistsintheUnitedStatescompletedthebodyof

studiesthatwererequiredtoidentifythebestcandidatesforDTCsandsomeofthebest

practicesthatproducebetteroutcomesinDTCprograms(Marlowe,2012a).

Accordingly, this manual is designed to allow other OAS member states to take

advantageofthesedecadesofexperience. Hopefully, itwillallowthemtoestablishthe

effectivenessoftheirprogramsandidentifybestpracticesfortheirnationalcontextmore

quicklyandefficiently.

21

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Figure1depictsalogicmodelformonitoringandevaluatingDTCs.Detaileddescrip-

tionsoftheactivitiesdepictedinthelogicmodelareprovidedinthenextseveralchapters

ofthisevaluationmanual.Thelogicmodelalsoincludesanapproximatetimelineforper-

formingvarioustypesofmonitoringandevaluationactivities.

Asthediagramindicates,whenaDTCisfirstestablished,theinitialtaskforevaluators

is to perform a process evaluation aimed at ensuring theDTC is following the intended

model, treating the appropriate participants, and delivering effective services.Many of

theseactivitieswillbeconductedinthefirstyearoftheprogram,andthenrepeatedover

subsequentyearstoensureprogramoperationsarenotdepartingfrombestpractices.Part

oftheprocessevaluationmayalsoincludeestimatingthecostsofadministeringtheDTC

(cost analyses)andunderstandinghowthosecostsareincurredbydifferentagencies,such

asthecourt,probationdepartment,andtreatmentprograms(cost allocation analyses).

AftertheDTChasbeeninoperationforatleastayear(andoftenlongerthanthat),

evaluatorsmayconductanoutcome evaluationtoassesshowparticipantsareperforming.

Outcomeevaluationsmaybeconductedovertheshorttermwhileparticipantsarestillen-

rolledintheprogram(short-term outcome evaluations),and/orafterparticipantshavebeen

dischargedfromtheprogram(long-term outcome evaluations).Finally,evaluatorsmayde-

termineoverthelonger-termwhetherthecostsofadministeringtheDTCarejustifiedby

resulting improvements inparticipants’adaptivefunctioning (cost-effectiveness analyses)

andfinancial savings toparticipantsandsociety, suchasavoidingcosts related tonew

arrestsornewincarcerationsorgeneratingfinancial incomefromimprovedemployment

andeducation(cost-benefit analyses).

Animportantpointtoemphasizeisthatevaluators,DTCpersonnel,andpolicymakers

should not become discouraged or overwhelmed by the potential scope of evaluation

LOGIC MODEL FOR DTC PROGRAM EVALUATIONS

22

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

activities. Theadage“first thingsfirst” isparticularlyapt in this regard. EveryDTCcan

beginatrelativelymodestcostandefforttomeasurethequalityandquantityofitsservices.

As discussed in greater detail infra, information garnered from this process

evaluation is likely to be highly predictive of long-termoutcomes. If a DTC is delivering

high-qualityservicesandtreatingtherightparticipants,itisverylikelytoreducerecidivism,

enhance recovery from addiction, and generate cost benefits for society. Consequently,

DTCs that lack sufficient resources to conduct high-quality outcome evaluations and

cost-effectivenessevaluationscannonethelessbegintheevaluationprocessbygenerating

importantandusefulinformation.Later,ifandwhenresourcesbecomeavailable,theycan

increase the sophisticationof its evaluation activities. Additionally, tracking participants’

progress and recording/organizing data appropriately from the outset of the DTC

program—acomparativelysimpleandlow-costpropositionifdonecorrectly—willgreatly

facilitatefuturemonitoringandevaluationactivities.

23

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

FIG

UR

E 1:

DR

UG

TR

EATM

ENT

COU

RTS

EVA

LUAT

ION

LO

GIC

MO

DEL

24

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Finally,therearemanywaystoobtainusefulinformationforDTCevaluations.Forexample,

ifevaluatorswant tomeasurehowmuchtreatmentparticipants receive in theprogram,

they can review the DTC’s policies and proceduresmanual to determine howmuch

treatmentisintendedtobeprovided,interviewstaffmembersorparticipantsabouthow

much treatment is commonlyprovided,ormeasure theactualnumberof sessionseach

participantreceives.Whilethecostandeffortrequiredtoobtaintheinformationincreases

withtheprecisionofthemeasurement,inmanyinstanceslessprecisemeasurementswill

stillprovesufficientforausefulevaluation. The lessonforDTCswith limitedbudgetsor

evaluationexpertiseisthatitisoftenpossibletoobtainadequateinformationatreasonable

costandeffort.

25

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Programevaluationsgenerallyfallintooneoffourbroadcategories,dependingonthe

scopeandaimsoftheevaluation:

(1) Processevaluations,

(2) Short-termoutcomeevaluations,

(3) Long-termoutcomeevaluations,and

(4) Costevaluations.

In addition, the term impact evaluation is commonly reserved for short-term and

long-termoutcomeevaluationsthatincludeacomparisongroup.Aswillbediscussedlater,

it is necessary to include a comparison group in order to determinewhether outcomes

producedbytheDTCrepresentedasignificantimprovementoverwhatwouldhavebeen

achievedwithouttheDTC.Forexample,ifaDTChasa60percentre-arrestrate,thismight

seemhigh;however,ifthecomparisongrouphadare-arrestrateof80percent,thenthe

DTCproducedasignificantpositiveimpact.

Process EvaluationsProcess evaluations indicatewhether a program functions as planned, treats the

intendedtargetpopulationofparticipants,anddeliversthetypesanddosagesofservices

thatarelikelytoproducefavorableoutcomes.Questionsthatarecommonlyaddressedin

processevaluationsinclude:

�Whatproblemsandbarrierswereencounteredinimplementingthe

program,andhowweretheyresolved?

�HowdoDTCstaffmembers,participantsandstakeholdersperceivethe

program’saimsandeffectiveness?

�Whatarethecharacteristicsandneedsoftheparticipants,andarethe

servicesofferedintheprogramadequatetomeetthoseneeds?

TYPES OF PROGRAM EVALUATIONS

26

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

�Istheprogramservingitsintendedtargetpopulationofparticipants?

�Doestheprogramprovideasufficientdosageandqualityofservicesthatit

islikelytoproducefavorableoutcomes?

�Whattypesofrewardsandsanctionsaretypicallydelivered?

�Whatrestorativejusticerequirements(e.g.,victimrestitutionorcommunity

service)arebeingimplemented,anddoparticipantscomplywiththose

requirements?

�ArecasesresolvedmorerapidlyorlessrapidlyforDTCparticipantsthanfor

comparableindividualsinthetraditionalcriminaljusticesystem?

Aswasnotedearlier,severalCICAD-sponsoredstakeholdermeetingswereheld in

thecourseofdevelopingthisevaluationmanual.Audienceparticipantsraisednumerous

additionalquestionsofinteresttotheircountries.RepresentativesfromChile,forexample,

statedthatthefollowingissuesarecriticalforsuccessfullyinstallingaDTCinthatcountry,

anduntilsuchissuesareresolveditisverydifficulttoengageinshort-termandlong-term

planningforDTCprograms:

� IstheresufficientpoliticalwilltodesignandimplementtheDTCmodelin

thiscountry?

� Doesthephilosophyofthecriminaljusticesysteminthiscountryfocus

predominantlyonincreasingsocialcontrolofoffenders,ordoesitalsoincludean

emphasisonprovidingadequaterehabilitation,treatmentandsocialservices?

� Isthereanadequatebudgetavailableforpurposesofdevelopingand

maintainingtheDTC?

� WhichstaffmembersandstakeholdersinvolvedintheDTCareproneto

implementtheDTCmodel,andwhichonesmaybereluctantorambivalent

aboutcomponentsofthemodel?

� Whatgovernmentsector(s)isinchargeofadministeringthedaily

operationsoftheDTC?

27

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

� Isthereacoordinatingunitoragencywiththepoliticalandtechnicalsupport

neededtoleaddevelopmentofthemodelduringtheimplementationstage?

� Doesthiscoordinationunithavethepowertodevelopandfundcomponents

oftheDTC,conveneothersponsorsandstakeholderswhenneeded,and

intervenetoresolvedisagreementsandbarrierstoeffectiveimplementation?

� ArethedefenseattorneysinthecountryreluctanttoaccepttheDTCmodel

(because,forexample,theybelievetreatmentandrehabilitationimposea

higherpenalburdenthanotherapproaches)?

� DoprosecutorsbelievetheDTCmodelistoolenientcomparedwithother

approaches?

� DojudgesconsidertheDTCmodeltobeafunctionofthetreatmentor

socialservicesystemsratherthanthecourtsorjusticesystem?

� Isthereanadequatenetworkoftreatmentandrehabilitationservicesfor

peoplesufferingfromsubstanceabuseandmentalhealthissues?

� Isthereanadequatenetworkoftreatmentandrehabilitationservices

willingtoacceptandtreatpersonschargedwithcriminaloffenses?

� DoesthemultidisciplinaryteamofprofessionalsinvolvedintheDTC

possesssufficientknowledgeandexpertiserelatedtobothtreatmentand

criminaljusticeinterventions?

� DotreatmentprofessionalsinvolvedwiththeDTCagreetocoordinate

theireffortswithjudicialauthoritiesandsharecarefullydefinedtreatment-

relatedinformation?

� IstheDTCachievingthegoalsandmilestonesspecifiedinthecountry’s

legislationorrulesauthorizingandestablishingtheprogram?

Asstatedearlier,processevaluationsareusuallythefirstorderofbusinessforDTCevalua-

tors.Theyshouldideallybeperformedearlyinthecourseofdevelopingtheprogram.Until

itisestablishedthatDTCprocessesarebeingimplementedcorrectlyandthattheprogramis

servingtherightpeople,thereislittlepurposeinexaminingshort-andlong-termoutcomes.

28

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Fortunately,processevaluationscanusuallybeperformedatmodestcosttotheprogram

becausetheyoftenconsistofdescriptionsofprogramoperationsandattitudesandskillsof

staffmembers—informationthatisrelativelyinexpensivetoobtain.

Short-Term Outcome Evaluations

Short-term outcome evaluations (also referred to as proximal or intermediate

outcomeevaluations)measure participants’ performancewhile they are still enrolled in

the program. The evaluation typically focuses on outcomes during treatment that are

likelytopredictpost-programperformance.Examplesoftheseso-calledduring-treatment

outcomesthatarecommonlyexaminedinclude:

�Whatpercentageofparticipantssuccessfullygraduatedfromtheprogram?

�Whatistheaveragelengthofstayintheprogram?

�Whatistheaverageattendancerateattreatmentsessions?

�Whatproportionofurinedrugtestswasnegativeforallsubstancesofabuse?

�Whatproportionofparticipantscommittedanewoffenseortechnical

violationwhileenrolledintheprogram?

Theseshort-termoutcomesareoftenmeaningfulintheirownrighttopolicymakers,

thepublic,andotherstakeholdersofDTCprograms.Forexample,oneofthecentralaims

ofaDTCistorehabilitatepersonssufferingfromaddiction.Therefore,helpingparticipants

successfullystayinandgraduatefromtreatmentisanimportantindicatorofsuccess.

Ultimately,however,thepublicandotherstakeholdersarelikelytojudgethemerits

ofaDTCbyhowwellitreducescrime,incarcerationrates,andpublicexpenditures.There-

fore, it is important forat leastsomeshort-termoutcomemeasures tobesignificant

predictorsofcriminalreoffendingandotherlong-termoutcomes.

Fortunately,studiesconsistentlyfindthatlongerretentionintreatmentandsuccess-

fulgraduationfromDTCsdo,infact,predictgreaterreductionsincriminalrecidivism(Carey

etal.,2012;Gottfredsonetal.,2007,2008;Petersetal.,2002).Therefore,demonstrating

29

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

thataDTCincreasestreatmentretentionandgraduationratessuggeststhatitwillresultin

futurereductionsincrime.

Long-Term Outcome Evaluations

Long-term outcome evaluations (also referred to as distal outcome evaluations)

measureparticipants’performanceaftertheyarenolongerenrolledintheDTCprogram.

Often, evaluators will report post-program outcomes for participants who successfully

completedtheDTCandthosewhowereunsuccessfullydischargedorvoluntarilywithdrew

fromtheprogram.Commonquestionsthatmightbeaddressedinclude:

�Whatpercentagesofparticipantswerearrestedforanewoffenseor

convictedofanewoffenseafterbeingdischargedfromtheprogram?

�Whatpercentageofparticipantswasincarceratedforanewcrime,andfor

howlong?

�Whatpercentageofparticipantswaslivinginsafeanddrug-freehousingat

follow-up?

�Whatpercentagesofparticipantsweregainfullyemployedorenrolledin

educationalprograms?

�Whatpercentagesofparticipantswereexperiencingseriousmedical,

psychiatric,orfamilyproblemsatfollow-up?

�WhatpercentageofbabiesborntoDTCparticipantswasdeliveredhealthy

anddrug-free?

�Whatpercentageofparticipantswassufferingfromorengagedindomestic

violence?

�Whatpercentageofparticipantswasstillcommittingcrimeorabusingillicit

drugs,buthasnotbeenincontactwiththecriminaljusticesystem?

Long-termoutcomeevaluationsareoftendifficultforDTCstoconductontheirown,

because staffmust locateparticipantsafterdischarge todeterminehow theyaredoing.

FewDTCshavesufficientresourcestofollowupwithparticipantsafterdischarge,andpar-

30

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

ticipantsareoftenreluctanttospeakhonestlywithDTCstaffabouttheirperformance.

Participantsmay,forexample,feargettingintotroubleagainwiththecriminaljustice

systemiftheyadmittosubstanceuseorcrime.Similarly,theymaynotwanttodisappoint

their former therapists or other staff by acknowledging relapses or other treatment

setbacks.Inmostinstances,thereforeDTCswillneedtohireindependentevaluatorsto

conductlong-termoutcomeevaluations.

Cost Evaluations

Costevaluationsattachmonetaryvaluestotheresultsoftheaboveevaluationsto

estimatethenetfinancialimpactsofaDTC.Incostevaluations,distinctionsaremadebe-

tweenthreeconcepts:(1)investmentcosts,(2)outcomecosts,and(3)outcomesavings.

Investment costs represent theadditionalexpenditures thatwererequiredtoadminister

theDTCprogram,suchastheaddedcostsoftreatmentandfrequentcourthearings.Out-

come costsaretheexpendituresthatwereincurredbytaxpayersorthegovernmenttodeal

withtheparticipants’subsequentbehaviors,suchasthecostsofprosecutingnewoffenses

orincarceratingparticipantsfornewcrimes.Outcome savingsaremoniesrepaidtoorre-

claimedbysocietyasaresultoftheimprovedfunctioningofDTCparticipants.Forexample,

participantsmayfindworkandpayincometaxes,contributetothefinancialsupportoftheir

children,orvolunteertoworkincharitiesorsocialserviceagencies.InPanama,forexam-

ple,theCriminalProcedureCodespecifiescharitableworkasaconditionofrehabilitation,

includingvolunteeringtoworkininstitutionsofsocialwelfare,education,publicinfrastruc-

ture,orlocalgovernmentwithnopaymentfromthestate.

Bycomparingthefinancialexpendituresandsavings,economistsestimatethenet

financialbenefitsoftheDTCprogram.Questionsthatmightbeaddressedinacostevalua-

tioninclude:

�Whatweretheadditionalcostsofprovidingtreatmentandsupervision

servicesfortheDTCparticipants?

31

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

�WhatweretheadditionalcostsofholdingfrequentcourthearingsforDTC

participants?

�HowdothecostsoftheDTCcomparetothecostsoftraditional

approachestocasedisposition?

�WhatweretheoutcomecostsincurredbytheDTCparticipantsresulting

fromcriminalrecidivismortheuseofpublicorgovernmentresources,

comparedtothoseofacontrolgroup?

�Whatoutcomesavingswereproducedastheresultofincreased

employmentorproductivityonthepartoftheDTCparticipants?

�WeretheinvestmentcostsoftheDTCrecoupedbytheoutcomesavings?

ManyDTCsexamineinvestmentcostsaspartofaprocessevaluationorshort-term

outcomeevaluation.Often,DTCpersonnelarecapableofestimatinghowmuchstafftime

andotherresourcesarerequiredtoadministertheprogram.

Itismoredifficult,however,toestimateoutcomecostsandoutcomesavingsbased

onreductionsincrimeandotherlong-termoutcomes.Therefore,independentevaluators

withexpertiseincostevaluationsareoftenneededtoconductcost-effectivenessanalyses

orcost-benefitanalyses,whichcompareinvestmentcostsagainstlong-termoutcomecosts

andsavings.

32

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Bysomeestimates,conductingathoroughprogramevaluationcanrequire10percentto

25percentofaDTC’soperatingbudget(Peters,1996).Thisis,ofcourse,unsustainablefor

countriesthataredevelopingtheirDTCprogramsonapilotbasisoronasmallbudgetasa

proofofconceptbeforecommittingsubstantialresourcestotheendeavor.

Fortunately,however,therearevariouswaystoobtaininformationforaDTCprogram

evaluation,andsomemethodswill requireconsiderably lesseffortandexpensethan

others. Generally speaking, the more precise the source of the information, the more

difficultandcostly itwillbefortheevaluatortocollectthedata.Evaluatorsmustdecide

howmucheffortandresourcestheyarecapableofdevotingtomeasuringagivenvariable,

andwhethertheaddedbenefitsofmoreprecisemeasurementjustifytheadditionaltime

andexpensethatmayberequired.

Figure2depictsahierarchyofdata sources thatmaybeused inaDTCevaluation,

rangingfrominformationthatisrelativelysimpleandinexpensivetocollectatthebottom

ofthefiguretoinformationthatismorecostlyandcomplextocollectatthetopofthe

figure.Thehigherinthefigurethesourceoftheinformation,themoreprecisethedatawill

be,butalsothemoredifficultandcostlyitwillbetocollectthatdata.

SOURCES OF INFORMATION

FIGURE 2: SOURCES OF INFORMATION

33

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Consider,forexample,thevariouswaysinwhichanevaluatorcouldmeasurethefre-

quencyandimpactofcourthearingsinaDTC.Theevaluatorcouldreviewtheprogram’s

policies andproceduresmanual todeterminehowoften courthearings are intended to

beheld,interviewstaffmembersorparticipantsabouthowfrequentlycourthearingsare

typicallyheld,orextractdatafromprogramchartsorrecordsonactualattendanceratesat

courthearings.Theevaluatorcouldalsoobservecourthearingsdirectlyorinterviewpartici-

pants’friendsandfamilymemberstodeterminehowimpactfulthehearingstendtobe.

Eachoftheseapproachescarrieswithitbenefitsandburdensthatmustbetakeninto

accountbyanevaluatorwhenplanninganevaluationdesign.Bycarefullycomparingthe

advantagesanddisadvantagesofeachinadvance,theevaluatorcandesignanevaluation

thatwillbothmeetthenecessaryobjectivesandstaywithinitsallocatedbudget.

Program Documentation

PerhapsthesimplestandleastcostlymethodistoreviewtheDTC’spoliciesandpro-

ceduresmanualorotherdocumentationtodeterminehowfrequentlycourthearingsare

plannedtobescheduled.Thepoliciesandproceduresmanualmightspecify,forexample,

thatcourthearingsshouldbescheduledeverytwoweeksduringthefirstphaseofthepro-

gramandmonthlythereafter,unlessthereisagoodreasonforthejudgetoschedule

hearingsmoreor less frequently. Although thismight seem like a blunt and imprecise

methodofmeasurement,studieshavefoundthatitcan,infact,significantlypredictout-

comesinDTCsanddifferentiateeffectivefromineffectiveprograms(Careyetal.,2012;

Shaffer,2010;Rossmanetal.,2011).

Thereareobviousadvantagestothisapproach.First,theinformationcanbecollected

quicklyandwithminimaleffortbyresearchstaff. Second, it isunnecessarytowaitfora

substantialnumberofparticipantstoenterandcompletetheprogrambeforebeingableto

collectattendanceinformationatcourthearings.Areviewofprogramdocumentationcan

beaccomplishedbeforetheDTCopensitsdoorsorinthefirstyearofoperations.

34

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Ofcourse,programsdonotalwaysadheretowhatiswrittenintheirpoliciesandproce-

duresmanual,andsomeprogramsmaynothaveamanual.Practicesalsohaveatendencyto

changeor“drift”overtime.ADTCmightstartoutwiththeintentofholdingfrequentcourt

hearings,butthenfinditselfschedulinghearings lessfrequentlyduetotimeorresource

constraints.Greaterprecisionislikelytobeachievedbymeasuringwhatactuallyhappensin

theDTCratherthanwhatpeoplethoughtwouldhappenwhentheprogramwasfirstde-

velopedorthepoliciesandproceduresmanualwaslastupdated.

Staff Surveys

The next simplest and least costly approach is to survey staffmembers about

howoftencourthearingsaretypicallyheldintheprogram.MostDTCshaveacoreteam

ofapproximatelysixtotenstaffmembers,includingthejudge,programcoordinator,and

representativesfromtheprosecution,defense,treatmentprograms,probationdepartment,

andlawenforcement.Itisusuallypossibletosurveythissmallgroupofprofessionalswith

minimalexpenseandinconvenience.

Inseverallarge-scalestudiesintheUnitedStates,evaluatorsusedweb-basedsurvey

instruments (Carey et al., 2012; Shaffer, 2010) or computer-assisted personal interviews

(CAPI)(Rossmanetal.,2011)tocollectinformationfromstaffmembersindozensofgeo-

graphicallydispersedDTCprograms.Theweb-basedsurveysgatheredself-reportinforma-

tionfromstaffmemberswhologgedintoencrypteddatabasesusingsecuredusernames

andpasswords. In theCAPIstudies, the interviewersusedcomputerswithsoftwarethat

guidedthemthroughastructuredinterviewprocessandallowedthemtorecordresponses

directlyintoanelectronicdatabase.

In both types of studies, the staff members’ responses were entered into a

databasethatautomaticallytabulatedandsummarizedtheresults.TheAppendixtothis

manualprovidesinformationonhowtoobtainstaffmembersurveysthathavebeenused

successfullyinlarge-scaleDTCprogramevaluations.

35

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Thereareseverallimitationstothismethod,however.Staffmembersmaynotrecall

accuratelyhowoftencourthearingswereheld,ortheymaybemotivatedtopresentthe

DTCinanundeservedlyfavorablelight.Evaluatorsmayalsofinditnecessarytoreconcile

discrepancies instaffmembers’responses.Forexample, if the judgereportedthatcourt

hearingstypicallyoccureverytwoweeks,buttheprosecutorreportedtheyoccureveryfour

weeks,theevaluatorwouldneedtocontactthejudgeandprosecutoragaintoresolvethe

discrepancy.

Participant Focus Groups

Anothermethodistoorganizefocusgroupswithasubsetofparticipantstogauge

theirperceptionsabouttheprogram.Participantsmaymorelikelythanstafftoofferneutral

ornegativeobservationsabouttheDTC,andtheymayalsobemorelikelytorecallaccurate-

lyhowoftentheywererequiredtoappearbeforethejudgeincourt.Anotheradvantage

tofocusgroups isthattheevaluatordoesnotneedto intervieweveryparticipant inthe

program. Informationcanbecollected fromamanageablenumberofparticipantsat

reasonablecostandeffort.

Ordinarily,participantsshouldberandomlyselectedtoparticipateinfocusgroups

toensuretheyrepresenttheviewsandexperiencesofmostparticipantsintheprogram.

Itisoftennotappropriate,forexample,toselectonlysuccessfulgraduatestoparticipate

infocusgroups,becausegraduatesmayhaveverydifferentperceptionsabouttheDTCor

experiencesintheprogramthanotherparticipants.Graduatesmayhavebeensuccessful

precisely because they attendedmore frequent court hearings than other personswho

wereunsuccessfullydischargedfromtheprogram.Ifthisisthecase,thenrelyingsolelyon

graduates’responsesmightleadtheevaluatortoconclude,erroneously,thattheDTCholds

morefrequenthearingsthanitactuallydoesformanyparticipants.

36

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Participant Surveys

Amoreprecise,butalsomorecostly,methodistosurveyallormostofthepar-

ticipantsintheprogram.Thisapproachensurestheevaluatorwillassessthefullrangeof

participants’experiences in theprogram,and therefore theresultsaremore likely tobe

representativeofhowtheDTCoperatesinpractice.

Becauseeachparticipantwillreportonhisorherownattendanceincourt,theeval-

uatorwillobtainarangeofscoresfortheentiresampleofparticipants.Havingarangeof

scoreswill allow theevaluator to calculate suchmeasuresas theaveragenumberof

hearings thatwere attended, aswell as variability in thenumberof hearings thatwere

attended,suchastherangeorstandarddeviation.Aswillbediscussed later,collectinga

rangeofscoresallowstheevaluatortoemploymoresophisticatedstatisticaltechniquesand

answermoresophisticatedquestions.

Participant Records

Anothermoreprecisemethod istoexaminedatafromcourtortreatmentrecords

concerningparticipants’attendanceatcourthearings.Assumingtheinformationisentered

accurately, thismethodmayyieldthemostprecisemeasurementofattendanceatcourt

hearings,andtheevaluatorwouldobtainarangeofscoresontheentiresampleofpar-

ticipants.Measuringactualattendanceratesavoidsconcernsthatstaffmembersor

participantsmaynotrecallaccuratelyhowoftenhearingswereheld. Italsoavoidscon-

cernsthatsomestafforparticipantsmay,consciouslyorunconsciously,presenttheDTCin

anundulyfavorableorunfavorablelight.

Amajor limitationof thisapproach is that it relieson informationbeingentered

accuratelyintoparticipantrecords.Unfortunately,itisnotuncommonforevaluatorstoen-

countermissingorincompleteinformationinDTCrecords.StaffmembersinDTCsareoften

verybusy,andmaynotgivetherelativelylessexcitingtaskofdatacollectionandentrythe

attentionandcareitrequires.Ifinformationisnotenteredaccuratelyintoparticipantre-

37

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

cords,evaluatorsmayhavenooptionbuttoconductsurveysorfocusgroupsinanattempt

tomakeupformissingorincompleteinformation.

Site Observations

Evaluatorsoftenfind ithighlyenlightening toobserveDTCoperationsdirectly.For

example,ifanevaluatorwishestoexaminethequalityandimpactofcourthearings,thereis

oftennosubstituteforconductingon-siteobservationsofcourtproceedings.Theevaluator

canassess,forexample,howlongthejudgeinteractswithparticipantsincourt,whetherthe

judgegivesparticipantsachancetoexplaintheirviews,andwhetherotherteammembers,

suchasthedefenseattorneyorprosecutor,contributevaluableinput.Thislevelofqualitative

informationisdifficultorimpossibletoobtainfromprogramrecordsorparticipantsurveys.

Needlesstosay,collectingthislevelofinformationcanbecostlyandresource-

intensiveforevaluators.Notonlymustevaluatorsattendasufficientnumberofhearings

togetareliablepictureofhowtheprogramoperates,butevaluatorsmustalsobetrained

carefullyonhowtorecordreliableandvaliddataconcerningtheirobservations.Itisoften

necessarytousestandardizedratingscalestoensureinformationisrecordedinanunbiased

mannerandusefulformat.Additionalinformationaboutobserverratingscalesisprovided

inlaterchaptersofthismanual.

DTCstaffshouldnotconductsiteobservationsthemselves,astheycannotbeexpected

toprovideaccurateandunbiasedappraisalsofhoweffectivelythey interactwithpartici-

pantsorperformtheirroles.Instead,well-trained,independentobserversarerequiredfor

thistask.Siteobservationsarethereforegenerallyviewedasanadvancedevaluationpro-

cedure,andarerarelyperformedoutsideofwell-fundedevaluationprojects.

Collateral Sources

Finally, evaluators may find it useful to collect information from participants’

friends,familymembers,orotherindividualswhoareknowledgeableaboutparticipants’

38

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

experiencesintheprogram.Forexample,participantsmayviewfrequentcourthearings

asanunnecessaryburdenontheirtime,especiallyduringtheearlyphasesoftreatment.

Familymembers,however,mayhaveaverydifferentviewabouttheimportanceofcourt

hearings—seeingthem,forexample,asindispensabletokeepingtheirlovedonelaw-abiding

andabstinentfromdrugsandalcohol.Ifanevaluatorreliedsolelyonparticipantperceptions

toassessthevalueofcourthearings,heorshemightreachincorrectconclusions(e.g.,that

courthearingsshouldbescheduledlessoften,wheninfactfrequenthearingswereacritical

elementforsuccess).

Interviewingcollateral sourcescanbedifficult formanyDTCs.Participantsmaybe

reluctanttohavestaffmembersspeakwiththeirfriendsorfamilymembers,anditmaybe

difficulttoconvincefriendsorfamilytocometocourtandbeinterviewedbycriminal

justice authorities.Often it is necessary tohave trained and independent evaluators

conductcollateralinterviews,whichcanbecostlyanddifficultformanyDTCs.

39

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

In1997,theNationalAssociationofDrugCourtProfessionals intheUnitedStates

publishedwhatarecommonlyreferredtoasthe10KeyComponentsofDTCs(NADCP,1997).

Severalyearslater,theInternationalAssociationofDrugTreatmentCourts(IADTC)adopted

the10KeyComponents,andaddedthreeadditionalcomponents:

(1) Focusingoncasemanagementtoaddressancillaryneedsofparticipants

andpromotetheirsocialreintegration;

(2) Ensuringindividualizedtreatmenttoaddresstheneedsofspecial

populations,suchaswomen,participantswithco-occurringdisorders,

indigenouspopulations,andethnicminorities;and

(3) Provisionofaftercarerecoveryservices.

TheIADTCprinciplesareembodiedinadocumententitledthe13KeyPrinciplesfor

Court-directedTreatmentandRehabilitationPrograms(“13KeyPrinciples”).

OPERATIONALIZING VARIABLES

40

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

INTERNATIONAL ASSOCIATION OF DRUG TREATMENT COURTS (IADTC)13 Key Principles for Court-Directed Treatment and Rehabilitation Programmes

(1)Theprogrammesintegratesubstancedependencytreatmentserviceswithjusticesystemcaseprocessing.

(2) Using a non-adversarial approach, prosecution and defense lawyers promotepublicsafetywhileprotectingoffenders’dueprocessrights.

(3)Eligibleoffendersareidentifiedearlyandpromptlyintegratedintotheprogramme.

(4) The programmes ensure access to a continuum of substance dependencytreatmentandotherrehabilitationservices.

(5)Complianceisobjectivelymonitoredbyfrequentsubstanceabusetesting.

(6) A coordinated strategy governs responses of the court to programme non-compliance(andcompliance)byoffenders.

(7)Ongoingjudicialinteractionwitheachoffenderinaprogrammeisessential.

(8)Monitoringandevaluationmeasuretheachievementofprogrammegoalsandgaugeeffectiveness.

(9)Continuinginterdisciplinaryeducationpromoteseffectiveplanning,implementation,andoperationsofthesecourt-directedprogrammes.

(10)Forgingpartnershipsamongcourtsdirectingtreatmentprogrammes,publicagencies,andcommunity-basedorganizationsgenerateslocalsupportandenhancesprogrammeeffectiveness.

(11)Ongoingcasemanagementincludesthesocialsupportnecessarytoachievesocialreintegration.

(12) There is appropriate flexibility in adjusting programme content, includingincentivesandsanctions,tobetterachieveprogrammeresultswithparticulargroups,suchaswomen,indigenouspeopleandminorityethnicgroups.

(13)Posttreatmentandafter-careservicesshouldbeestablishedinordertoenhancelongtermprogrammeeffects.

41

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

The 13 Key Principles represent a revolutionary change in thinking about how to

successfully rehabilitate individuals suffering fromaddictionwhoare chargedwithdrug-

relatedcrimes.Itshouldbeimmediatelyapparent,however,thatmanyofthesePrinciples

aredescribedinverygeneralterms.

Consider,forexample,Principle7,whichstatesthatDTCsshouldprovide“ongoingju-

dicialinteractionwitheachoffender.”Howdoesonedefine“ongoing?”Isthefrequencyof

face-to-facecontactssufficienttomeasurethisvariable?Ifitis,howfrequentlyshouldthe

judgemeetwithparticipants?Shouldthejudgeholdhearingsonadaily,weeklyormonthly

basis,or should the frequencyofhearingsbebasedon theneedsofeachparticipant?

Similarly,whatdoes“interaction”mean?Mustthejudgespeakdirectlytoparticipants,give

themachancetovoicetheirownviews,orisitsufficienttosimplyrevieweachparticipant’s

progressandimposeconsequences?

EvaluatorsarefacedwiththedifficulttaskofdefiningthesePrinciplesinamannerthat

permitsthemtobemeasuredandexaminedobjectively.Theprocessofdefiningvariables

inmeasurableandobjectivetermsisreferredtoasoperationalizingthevariables.Different

evaluatorsarenotrequiredtoagreeonthesamedefinitionofavariable,buteachmust

make itpainstakinglyclearhowheorshe isdefiningthatvariable. If thedefinitionofa

variableisnotclearlydescribed,itwillnotbepossibleforotherevaluatorstoreplicatethe

studyorinterpretthefindings.

Forexample, it ispossibletomeasure“ongoingjudicial interaction” inanumberof

ways. Anevaluatorcouldmeasurehowoftenparticipantsappearedbefore the judge in

court. Alternatively, the he or she couldmeasure how long the judge spokewith each

participant during court hearings, or could rate how attentive or encouraging the judge

appeared to beduring his or her interactionswith participants. There is noone correct

waytooperationalize“judicialinteraction”oranyoftheother13KeyPrinciplesofaDTC

program.Instead,decidinghowtooperationalizeavariableisessentiallyavaluejudgment

thatisoftenbasedonthefollowingconsiderations:

�Burden of Collection: Some variables are considerably simpler and less costly to

42

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

measurethanothers.Forexample,itismucheasiertomeasurethenumberofcourt

hearingsthatparticipantsattendedthantouseastopwatchtotimehowlongthey

appearedbeforethejudgeincourt.Evengreatereffortwouldberequiredtoratethe

qualityofthejudge’sinteractionswiththeparticipants.

�Reliability of Measurement: Any evaluator should be able to reliably measure

thenumberof court hearings that participants attended. However, twodifferent

evaluatorscouldreachdifferentconclusionsabouthow“attentive”or“encouraging”

thejudgeappearedtobeduringhisorherinteractionswithparticipants.Toavoidthis

problem,substantialeffortandexpensemayberequiredtouseratinginstruments

thatyieldcomparablefindingsacrossdifferentevaluators.Thistime-intensiveprocess

isreferredtoasensuringinter-rater reliability.

�Cultural Significance: Variablesmayhavegreaterorlessersignificanceindifferent

contextsorcultures.Forexample,someculturesmayplacegreatvalueonshowing

respectforauthorityfigures.Citizensinthoseculturesmightbesocializedfroman

early age to avoid looking authority figures in theeye, disagreeingwith authority

figures,orspeakingfranklytoauthorityfigures. Ratherthanimprovingoutcomes,

longerandmorepersonalizedinteractionswithajudgemightcauseanxietyordis-

comfortforsuchindividuals.Therefore,itisnecessarytovalidatetheimpactofany

variable in different countries or cultures. The fact that personalized interactions

withajudgemightimproveoutcomesintheUnitedStatesdoesnotmeannecessarily

theywillhavethesameeffectinothercultures.

Measurement Precision

Variables may also be operationalized differently depending on the precision of

measurementthattheevaluatordesires.Generallyspeaking,themorepreciselyavariable

ismeasured,themoresophisticatedthequestions itcanaddress.Consider,forexample,

howanevaluatormightmeasuretheprovisionofsubstanceusetreatment inaDTC.Put

aside for themoment themorecomplicatedquestionofhow tomeasure thequalityof

treatment,andconsiderhowtomeasuretheamountoftreatmentthatwasprovided.This

43

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

variablecouldbemeasuredinatleastthreeways:

�Dosage:Thenumberoftreatmentsessionsthatwereattended.

�Attendance rate:Theratioofthenumberoftreatmentsessionsthatwereattended,

dividedbythenumberoftreatmentsessionsthatwereoriginallyscheduled.

�Density:Thenumberoftreatmentsessionsthatwerescheduledorattendedper

unitoftime,suchaspermonthorperphaseoftheprogram.

Measuringthedosageoftreatmentisrelativelystraightforwardandrequiresonlya

simpletally.Measuringtheattendancerateisabitmorecomplicatedbecauseitrequires

theevaluator tocollect twopiecesof information: thenumberofsessions thatwere

attended,andthenumberofsessionsthatwereoriginallyscheduled.Althoughthismight

seemlikeaminorburden,experienceindicatesthatmanyDTCprogramsdonotdoagood

jobofenteringmissedappointmentsintoadatabaseorparticipantrecords.Thisissome-

timesreferredtoasthe“problemofthemissingdenominator”becausethedenominator

intheattendanceratio(thenumberofsessionsthatwereoriginallyscheduled)isunknown.

To avoid this problem, it is essential for staffmembers inDTCs to recordwhether each

scheduledappointmentwaskept,notkept,rescheduled,orexcused.Asaruleofthumb,

sessionsarecountedashavingbeenrescheduledorexcusedonlyif(1)astaffmembergave

permissioninadvanceforthesessionnottooccur,or(2)theparticipantprovidedobjective

documentationthatheorshewasunabletoattendthesession,suchasprovidingadoctor’s

noteconfirmingthattheparticipantwasillorinjured.

Measuring the density of treatment is evenmore complicated because it requires

whatiscalleddate-stamping.Thismeansthateachentrymustbelinkedtothedateonwhich

theeventoccurredorshouldhaveoccurred. Forexample,attendanceateachcoun-

selingsessionmustbelinkedtothedateonwhichthesessionwasheld. Date-stamping

canbecriticalforpredictingoutcomesinDTCs.Forexample,aparticipantwhohasirregular

attendance in treatmentandtakesayear tocomplete twenty-six sessionsofcounseling is

likelytohavepooreroutcomesthananotherparticipantwhocompletesthefullsequenceof

twenty-sixsessionsontimewithinthreemonths.Withoutdate-stamping,theremightbeno

44

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

waytodifferentiatethesetwocasesandcriticallyimportantinformationcouldbelost.

Although thesemight seem like trivialmatters,manyDTCprogramevaluationshave

beenunabletoreportappreciablymorethandosage informationbecausedataonmissed

appointmentsorthedatesonwhichappointmentsoccurredwereneverrecorded.Perhapsthe

evaluatorsdidnotanticipatethisproblemorperhapstheywereunabletoconvinceprogram

staffmemberstorecordtheinformationmoreprecisely.Eitherway,greatereffortsmaybe

requiredtoperformmoresophisticatedanalysesinDTCevaluations.

Evaluatorsmustdecideforthemselveswhethertheadditionaleffortthatwillbere-

quiredtomeasureavariablemorepreciselyisjustifiedbytheabilitytoanswermoresophis-

ticatedquestionsthattheadditionalinformationprovides.Oneimportantissuetoconsider

inthisregardistheredundancyoftheinformation.Forexample,thenumberoftreatment

sessionsthatparticipantsattended ispartof thecalculationforall threevariablesof

dosage,attendancerate,anddensity.Asaresult,thesethreevariablesmaybesignificantly,

althoughonlypartially,correlatedwitheachother.Inotherwords,theymaybepartiallyre-

dundant.Variablesthatarepartiallyredundantmightpredictoutcomessimilarly,although

notequally.Densitymight,forexample,predictoutcomesbetterthandosage,butperhaps

thedegreeofpredictionthatisprovidedbydosagealonewouldbegoodenoughformany

evaluators’purposes.

Ideally,evaluatorsshouldlooktoempiricalfindingstodeterminewhethermorepre-

cisemeasurementisneededfortheirevaluations.Whereinformationisavailableonthe

redundancyofalternativemeasures,thatinformationisprovidedinthisevaluationmanual.

ThissectionalsounderscoreshowimportantandvaluableitisforDTCstafftomake

robustdatacollection,entry,andmanagementapriority.Enteringparticipantdatacorrectly

intoDTCrecordsasitisgeneratedisrelativelystraightforwardiftheDTChasthepropersys-

temsinplacetoidentify,collect,enter,andmanagerelevantdata,trainsstaffappropriately,

andperiodicallyverifiesthatthesesystemsandproceduresarebeingproperlyimplemented.

Doingsoiscertainlylessexpensiveandchallengingthanhavingtodesignsubsequentmoni-

toringandevaluationeffortsaroundincompleteormissingdata.

45

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Participant Characteristics

Participantcharacteristicsrefertoattributesofparticipantsthatpredatetheirentryinto

theDTC.Thesetypicallyinclude:

�Socio-demographicvariables,suchasage,gender,race,andemploymentstatus;

�Clinicalvariables,suchasprimarysubstance(s)used,psychiatricorsubstance

usediagnoses,andhistoryofsubstanceuseormentalhealthtreatment;and

�Criminalhistoryvariables,suchaspriorarrests,convictionsandincarcerations.

ManyDTCsandprobationdepartmentsadministerriskandneedassessmenttoolsto

matchparticipantstoappropriatelevelsoftreatmentandsupervision,andscoresonthese

toolsarealsoimportantvariablesforassessingparticipantcharacteristics.Furtherinforma-

tionaboutriskandneedassessmenttoolsisdiscussedinlaterchaptersofthismanual.

Participantcharacteristicsarereferredtoaspredictorvariablesiftheycorrelatesignifi-

cantlywithoutcomesinDTCprograms.Theyarereferredtoasriskfactorsiftheycorrelate

withpooreroutcomes,suchashigherrecidivismrates. Theymayalsobereferredtoas

moderator variables,butonlyiftheydifferentiallypredictoutcomesinDTCsascompared

toalternativeprograms,suchasprobation.Forexample,iffemalesperformedsignificant-

lybetterinaDTCthanprobation,butmalesperformedbetteronprobation,thengender

wouldbeamoderatorvariablefortheeffectsoftheDTC.Aswillbediscussedlater,deter-

miningwhetheraparticipantcharacteristicisamoderatorvariableasopposedtomerelya

predictorvariableorriskfactorrequirestheuseofmoresophisticatedstatisticaltechniques

thansimplecorrelations.

Inthiscontext,theDTCmodelasembodiedinthe13KeyPrinciplesmakestwofunda-

mentalassumptionsaboutparticipantsintheprogram.Itassumesthatparticipantsare(1)

WHAT TO MEASURE

46

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

addictedtoillicit1drugsoralcohol,and(2)unlikelytoimprovetheirbehaviorinaprogram

thatislessintensivethanaDTC,suchasprobationorareferraltotreatment.Participants

whosatisfybothconditionsarereferredtoashigh needandhigh risk.Thismeanstheyhave

ahighneedforsubstanceusetreatmentandarealsoathighriskforfailingtocomplete

treatmentunlessajudgeandteamofotherprofessionalsholdthemaccountableforcom-

plyingwithtreatment.

IfaDTCservesparticipantswhodonothavethesecharacteristics, itmightfinditself

providingtreatmenttopeoplewhodonotneedtreatmentorprovidingjudicialsupervision

topeoplewhodonotneedjudicialsupervision.StudiesfromCanadaandtheUnitedStates

have revealed that treating low-needor low-risk individuals inDTCshas thepotential to

wastescarcetreatmentand judicial resources (DeMatteoetal.,2006). It can also make

outcomesworseforlow-riskorlow-needparticipantsbyexposingthemtoantisocialpeers

orinterferingwiththeirinvolvementinproductiveactivities,suchaswork,school,home-

makingorchildcare(Lowenkamp&Latessa,2004;Taxman&Marlowe,2006).

Theparticipantcharacteristicslistedbelowhaveconsistentlybeenfoundtobepredictor

variablesor risk factors inDTCprogramevaluations inseveralcountries (Marloweetal.,

2003;Newton-Tayloretal.,2009;Somersetal.,2012).

�Current age

�Gender

�Education(numberofacademicgradescompleted)

�Employment(fulltime,parttime,seasonalortemporary,recentlyunemployed,chronicallyunemployed)

�Maritalstatus(married,nevermarried,divorced,widowed,cohabitating)

�Race

�Ethnicity

1 Illicitdrugsincludeprescriptionmedicationsthatareusedforanon-prescribedornon-medically-indicatedpurpose.

47

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

�Nationality

�Numberofpriorcriminalconvictions

�Monthsofpriorincarcerations

�Priorsubstanceusetreatmentepisodes

�Age of onset of substance use

�Ageofonsetofdelinquentorcriminalactivity

�Stableanddrug-freehousing(yesorno)

�Proportionoftimespentinteractingwithotherindividualsengagedincrimeorsubstanceuse(from“none”to“most”)

�Diagnosisofsubstancedependence/addictionvs.substanceabuse

�Primarysubstance(s)used(e.g.,opiates,cocaine,amphetamines,sedatives,hallucinogens,marijuana,alcohol)

�Co-occurringdiagnosisofmajorpsychiatricdisorder(i.e.majordepression,bipolardisorder,psychoticdisorder,post-traumaticstressdisorder)

Mostofthesevariablesaresimpleandinexpensivetocollectandshouldbeexaminedin

anyDTCprogramevaluation.Includingthesevariablesinmoderatoranalysescanassistthe

evaluatortodeterminewhichparticipantswerehelpedbytheDTC,andwhichparticipants

mightnothavebeenhelpedorperhapsevenharmed.Performingmoderatoranalysescan

prevent anevaluator from reaching theunfounded conclusion that aDTCdidnotwork,

wheninfacttherealproblemmighthavebeenthatittreatedthewrongpeople.

Better prediction is often achieved by using structured questionnaires or interviews

that combine several of these predictor variables into scales or summative scores.Risk

assessment toolsmeasurethelikelihoodthatparticipantswillcommitanewoffenseorfailin

treatmentunlesstheyreceiveintensivesupervisionsuchasthatprovidedinaDTC.Clinical

assessment toolsevaluatethediagnosticcriteriaforsubstancedependenceoraddiction,as

wellasthesymptomsofothermajorpsychiatricdisorders.TheAppendixprovideslinksto

riskassessmenttoolsandclinicalassessmenttoolsthathavebeenusedsuccessfullyinDTC

48

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

programevaluations.SeveralofthesetoolshavebeentranslatedintoSpanishorvalidated

foruseinSouthAmericancountries.

Notethattheabovelistincludesonlyvariablesthathavebeenshowntocorrelatewith

outcomesinDTCs.Aswasnotedearlier,mostDTCstudieshavebeenconductedinthe

UnitedStates,CanadaandAustralia,anditispossiblethatothervariablesmayemergeas

significantpredictorsofDTCoutcomes in SouthAmericanandCaribbeannations. These

variablesmayalsobedefinedormeasureddifferentlyinothercountries.

Performance Indicators

Performance indicators differ from participant characteristics in that they reflect

eventsoccurringafterparticipantsenteredtheDTCprogram.Therearetwogeneraltypesof

performanceindicators.Program-level performanceindicators(alsoreferredtoasinputs)

representtheservicesorinterventionsthatweredeliveredintheDTCprogram.Participant-

level performance indicators (also referred to as outputs) represent the impacts of the

DTContheparticipants.Examplesofprogram-levelperformanceindicatorsmightinclude

howoftencourthearingsortreatmentsessionswereheld. Examplesofparticipant-level

performance indicatorsmight includehowoftenparticipants testednegative for alcohol

andillicitdrugs,orthepercentageofparticipantswhograduatedfromtheDTCprogram.

Aspreviouslymentioned, itcanbechallengingforevaluatorstomeasurethewide

rangeofservicesthatDTCsprovideandthediverseimpactstheseprogramsmayhaveon

theirparticipants. Theprimary functionofaperformance indicator is to reduce this

complexinformationintoamanageableandanalyzablesetofnumericalindexes,suchas

ratios,sumsorpercentages.Thereisnoonecorrectwaytooperationalizeaperformance

indicator,andOASmemberstatesare freetodefineandmeasure thesevariables in the

mannertheydeemmostappropriateandinformative.

At thesametime,however, substantialbenefitswillbegainedbyhavingdifferent

49

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

countriescollectacommondatasetofcoreperformanceindicators.Thiswillallowmember

statestocomparetheperformanceoftheirDTCstothatofprogramsinothercountries.It

willalsobepossibletoaggregateperformanceinformationandexaminetheregionalimpact

ofDTCsintheWesternHemisphere.

Core Performance Indicators:Asnoted inaprevious section, representatives from

severalOASmemberstatesconvenedontwooccasionstoprovideguidanceonthecontent

andstructureofthisevaluationmanual.Theyagreedthatitwouldbedesirableformember

states to collect a coredatasetofperformance indicators. Collectionof the coreperfor-

manceindicatorswillbevoluntaryandeachcountrywilldecideforitselfwhethertoshare

itsfindingswithothernations.Itwasdecidedthatthefollowingcriteriashouldguidethe

selectionofcoreperformanceindicators:

�Thecoredatasetshouldcontainnomorethantenperformanceindicators

toavoidplacinganonerousburdenonmemberstates.

�Theperformanceindicatorsmustbeeasyandinexpensivetocollect,and

objectivelyandreliablymeasurable.

�Program-levelperformanceindicatorsshouldreflecttheessential

componentsofaDTC,withoutwhichaprogramwouldnotbeconsidered

aDTC.Theseinclude,ataminimum,courthearings,substanceuse

treatment,anddrugandalcoholtesting.

�Participant-levelperformanceindicatorsshouldreflectoutcomesof

primaryinteresttomanystakeholdersofDTCprograms.Theseinclude,at

aminimum,retentioninsubstanceusetreatment,useofillicitdrugsand

alcohol,andcriminalrecidivism.

�Theperformanceindicatorsmusthavebeendemonstratedinpriorresearch

topredictpost-programoutcomesinDTCsordifferentiateeffectivefrom

ineffectiveDTCprograms.

Recommended Performance Indicators:Itwasagreedfurtherthatalternativelistsof

50

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

recommendedanddiscretionaryperformanceindicatorsshouldalsobeprovided.Countries

interestedinconductingmorein-depthevaluationscanreviewtheselistsforfurtherideas

aboutpossiblewaystomeasureperformanceintheirprograms.Performanceindicatorsare

listedasrecommendediftheyhaveshownpromiseforpredictingoutcomesinDTCs,butare

difficulttomeasureorarepartiallyredundantwithotherperformanceindicators.

Discretionary Performance Indicators:Finally,performanceindicatorsarelistedas

discretionaryiftheyaretheoreticallyrelevanttoDTCprogramsorhavebeenrecommended

byleadingresearchorganizations,buthavenotyetbeenthesubjectofsubstantialresearch.

Performance Benchmarks

Performancebenchmarks refer to specific thresholds for successonperformance

indicatorsthatDTCsshouldstrivetoachieve.Forexample,aperformanceindicatormight

revealthatparticipantsinaDTCattendedanaverageoftwocourthearingspermonth.But

whetherthisisadesirablelevelofattendanceremainsanunansweredquestion.Perhaps

fourcourthearingspermonthwouldbemoreeffective,oronehearingpermonthwould

bemorecost-effective.

Whenprogramsarenew,andresearchhasnotyetidentifiedwhichpracticesproduce

betteroutcomes,performancebenchmarksarebasedoneducatedhypothesesoranecdotal

observations.Forexample,whentheearliestDTCsweredevelopedinthelate1980sand

early1990s,evaluatorshadlittlebasisforknowinghowtheprogramsshouldoperateand

whatservicestheyshouldprovide.Theevaluatorsmadeeducatedhypothesesaboutthe

bestwaystostructureanddeliverDTCservices,operationalizedvariablestomeasurethose

services,andexaminedtheeffectsoftheservicesonoutcomes.

Over time, evaluators determined empirically which practices produced better

outcomes inDTCprograms. Forexample, researchers inAustraliaandtheUnitedStates

found that scheduling court hearings every two weeks during the first phase of a DTC

51

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

produced significantly better outcomes than scheduling court hearings less frequently

(Careyetal.,2012;Jones,2011;Festingeretal.,2002;Marloweetal.,2006,2007;Zweiget

al.,2012).Basedonthatbodyofresearch,onebenchmarkforsuccessintheUnitedStatesand

AustraliaistoschedulecourthearingseverytwoweeksduringthefirstphaseofaDTCprogram.

Thesamefindingsmightnotgeneralizetonewculturesornewcountries.Research

might reveal that court hearings have a greater or lesser impact on outcomes in South

AmericanorCaribbeanNationsthantheydointheUnitedStatesorAustralia. Thisdoes

not,however,entitleevaluatorsinSouthAmericanorCaribbeancountriestoignoreprior

research.Especiallywhenfindingshavebeenreplicatedbydifferentresearchersindifferent

countries,evaluatorsinnewnationshavearesponsibilitytotakenoteofthosefindingsand

determineempiricallywhetherthesamelessonsmightapplyintheirownprograms.

ThisisnottosuggestthatevaluatorsinOASmemberstatesshouldlimittheirinquiries

toreplicatingwhatwasfoundpreviouslyinothercountries.Allevaluatorsareencouraged

toasknewquestions, developnewbenchmarks, and add to the international bodyof

knowledgeonDTCprograms.

The point here is that science proceeds, in part, through careful replication of prior

studiesandassessingthegeneralizabilityofpreviousfindings.Evaluatorshaveanobligation

to take into considerationfindings that have comebeforewhenplanning their research

agenda and conducting program evaluations. Therefore, if a benchmark for success has

beenidentifiedforagivenperformanceindicatorinatleastonecountry,thatinformation

isprovidedinthismanualforevaluatorsinOASmemberstatestoconsiderwhenexamining

theperformanceoftheirprograms.

52

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

This chapterprovidesdetailed informationabout calculatingperformance indica-

torsforDTCprogramevaluations.Table1summarizesthedataelementsthatarerequired

tocalculateeachperformanceindicator,theformulasforthecalculations,andpossible

interpretationsof theresults. The last twocolumnsofTable1 indicatewhetherstudies

have validated those performance indicators andwhether benchmarks for success have

beenidentifiedinothercountries.Foreaseofidentification,coreperformanceindicators

aredesignatedassuchinred font,recommendedperformanceindicatorsaredesignatedin

blue font,anddiscretionaryperformanceindicatorsaredesignatedinblack font.

CALCULATING PERFORMANCE INDICATORS

53

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

TABL

E 1:

Rec

omm

ende

d Pe

rfor

man

ce In

dica

tors

& P

oten

tial P

erfo

rman

ce B

ench

mar

ks fo

r DTC

Eva

luat

ions

in th

e A

mer

icas

*C

orepe

rforman

cein

dicatorsre

flectth

ede

finingfeaturesofD

TCs,arere

ason

ablyin

expe

nsivetocollect,and

sign

ificantlypredictlo

ng-te

rmoutcomes.Re

commen

dedpe

rforman

ce

indicatorssignifican

tlypredictoutcomesbutm

aybedifficulttom

easureorredu

ndan

twith

otherperform

ancein

dicators.Discretio

naryperform

ancein

dicatorsarethe

oreti

cally

impo

rtan

ttoDT

Csbuthavebe

enin

sufficien

tlystud

ied.

**Pe

rforman

ceben

chmarksarederived

fromstudiescon

ducted

inAustralia,C

anad

a,and

/ortheUnitedStates.Th

esamepe

rforman

ceben

chmarksm

aynotbevalid

inSou

th

Amer

ican

or C

arib

bean

cou

ntrie

s.

Vari

able

Data

Ele

men

tsPe

rfor

man

ce In

dica

tors

*Su

b-an

alys

esIn

terp

reta

tion

Pred

ictio

n of

Po

st-P

rogr

am

Reci

divi

sm o

r Co

sts

Pote

ntia

l Pe

rfor

man

ce

Benc

hmar

ks**

Cour

t su

perv

isio

n# h

earin

gs sc

hedu

led (H

rgS)

# hea

rings

atten

ded (

HrgA

)# h

earin

gs ca

ncell

ed or

resch

edule

d (H

rgC)

Date

stam

ped

Dose

of h

earin

gs =

Hrg

A

Cour

t app

eara

nce

rate

= H

rgA

(H

rgS

– Hrg

C)

By m

onth

or

phas

e of p

rogram

(d

ensit

y)

Inten

sity o

f judic

ial

supe

rvisio

n

Com

plian

ce w

ith

cour

t hea

rings

Mod

eratel

y pr

edict

iveEv

ery 2

week

s dur

ing

first

phas

e

Every

4 we

eks

there

after

Subs

tanc

e us

e tr

eatm

ent

# trea

tmen

t ses

sions

sche

duled

(TxS

)# t

reatm

ent s

essio

ns at

tende

d (Tx

A)# t

reatm

ent s

essio

ns ca

ncell

ed, e

xcuse

d or

resch

edule

d (Tx

C)Da

te sta

mpe

d

Dose

of t

reat

men

t = Tx

A

Trea

tmen

t atte

ndan

ce ra

te =

Tx

A (T

xS –

TxC)

By m

onth

or

phas

e of p

rogram

(d

ensit

y)

By le

vel o

f care

(e.

g., ou

tpati

ent

vs. re

siden

tial)

By m

odali

ty (e.

g., co

gnitiv

e-be

havio

ral vs

. ins

ight-o

rient

ed)

Inten

sity o

f tre

atmen

t prov

ided

Com

plian

ce w

ith

treatm

ent c

ondit

ions

Mod

eratel

y pr

edict

ive6 t

o 10 h

ours

per w

eek

durin

g first

phas

e

100 t

o 200

hour

s tota

l de

pend

ing on

the

seve

rity o

f the c

ase

54

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

TABL

E 1:

Rec

omm

ende

d Pe

rfor

man

ce In

dica

tors

& P

oten

tial P

erfo

rman

ce B

ench

mar

ks fo

r DTC

Eva

luat

ions

in th

e A

mer

icas

*C

orepe

rforman

cein

dicatorsre

flectth

ede

finingfeaturesofD

TCs,arere

ason

ablyin

expe

nsivetocollect,and

sign

ificantlypredictlo

ng-te

rmoutcomes.Re

commen

dedpe

rforman

ce



aybedifficulttom

easureorredu

ndan

twith

otherperform

ancein

dicators.Discretio

naryperform

ancein

dicatorsarethe

oreti

cally

impo

rtan

ttoDT

Csbuthavebe

enin

sufficien

tlystud

ied.

**Pe

rforman

ceben

chmarksarederived

fromstudiescon

ducted

inAustralia,C

anad

a,and


esamepe

rforman

ceben

chmarksm

aynotbevalid

inSou

th

Amer

ican

or C

arib

bean

cou

ntrie

s.

Vari

able

Data

Ele

men

tsPe

rfor

man

ce In

dica

tors

*Su

b-an

alys

esIn

terp

reta

tion

Pred

ictio

n of

Po

st-P

rogr

am

Reci

divi

sm o

r Co

sts

Pote

ntia

l Pe

rfor

man

ce

Benc

hmar

ks**

Drug

& a

lcoh

ol

test

ing

-and

-

Abst

inen

ce fr

om

drug

s & a

lcoh

ol

# urin

e, sa

liva o

r brea

th an

alyse

s sch

ed-

uled (

UaS)

# urin

e, sa

liva o

r brea

th sa

mple

s prov

id-ed

(UaP

)# s

ample

s ne

gativ

e for

all il

licit d

rugs

an

d alco

hol (U

aN)

# sam

ples i

nvali

d or a

dulte

rated

(UaI)

# sam

ples e

xcuse

d (Ua

E)Da

te sta

mpe

d

Alte

rnat

ive:

# day

s on c

ontin

uous

drug

surve

illanc

e; e.g

., ank

let, s

weat

patch

(Day

CDS)

# day

s with

no po

sitive

read

ing fo

r illic

it dr

ugs o

r alco

hol (D

ayCD

Sneg

)

Test

ing

dose

= U

aP

- or -

DayC

DS

Test

ing

com

plia

nce

rate

= (U

aP

– UaI

) (U

aS –

UaE)

Abst

inen

ce ra

te =

(UaN

– Ua

I)

(UaS

– Ua

E)

- or -

DayC

DSne

g

By m

onth

or

phas

e of p

rogram

(d

ensit

y)

By su

bstan

ces

detec

ted

Inten

sity o

f dru

g &

alcoh

ol tes

ting;

Com

plian

ce w

ith

drug

& alc

ohol

testin

g;

Absti

nenc

e from

dr

ugs &

alco

hol

Mod

eratel

y pre-

dictiv

e Tw

ice pe

r wee

k for

ur

ine or

saliv

a tes

ting

Mini

mum

of 90

days

on

cont

inuou

s mon

i-tor

ing

Mini

mum

of 90

co

nsec

utive

days

of

absti

nenc

e; ide

ally 1

80

cons

ecut

ive da

ys of

ab

stine

nce

Grad

uatio

n or

re

tent

ion

# part

icipa

nts e

ntere

d prog

ram (E

nt)

# part

icipa

nts g

radua

ted (G

rad)

# neu

tral d

ischa

rges

(Neu

)# s

till en

rolled

(Enr

)

Alte

rnat

ive:

# day

s from

entry

to di

scharg

e or la

st in-

perso

n con

tact (D

ayED

)

Grad

uatio

n ra

te =

Gra

d (E

nt

– Neu

)- o

r –

Rete

ntio

n ra

te =

(Gra

d +

Enr

) (E

nt –

Neu)

Leng

th o

f sta

y =

DayE

D

Reas

ons f

or te

rmi-

natio

nAb

ility o

f DTC

to

retain

parti

cipan

ts lon

g eno

ugh t

o mee

t reh

abilit

ative

goals

High

ly pr

edict

iveAv

g. gr

adua

tion r

ate

~ 60

%

18 –

24 m

onth

s of

enro

llmen

t in th

e DTC

Mini

mum

of 90

days

in

treatm

ent, a

nd

ideall

y 9- 1

2 mon

ths i

n tre

atmen

t

55

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

TABL

E 1:

Rec

omm

ende

d Pe

rfor

man

ce In

dica

tors

& P

oten

tial P

erfo

rman

ce B

ench

mar

ks fo

r DTC

Eva

luat

ions

in th

e A

mer

icas

*C

orepe

rforman

cein

dicatorsre

flectth

ede

finingfeaturesofD

TCs,arere

ason

ablyin

expe

nsivetocollect,and

sign

ificantlypredictlo

ng-te

rmoutcomes.Re

commen

dedpe

rforman

ce



aybedifficulttom

easureorredu

ndan

twith

otherperform

ancein

dicators.Discretio

naryperform

ancein

dicatorsarethe

oreti

cally

impo

rtan

ttoDT

Csbuthavebe

enin

sufficien

tlystud

ied.

**Pe

rforman

ceben

chmarksarederived

fromstudiescon

ducted

inAustralia,C

anad

a,and


esamepe

rforman

ceben

chmarksm

aynotbevalid

inSou

th

Amer

ican

or C

arib

bean

cou

ntrie

s.

Vari

able

Data

Ele

men

tsPe

rfor

man

ce In

dica

tors

*Su

b-an

alys

esIn

terp

reta

tion

Pred

ictio

n of

Po

st-P

rogr

am

Reci

divi

sm o

r Co

sts

Pote

ntia

l Pe

rfor

man

ce

Benc

hmar

ks**

Reci

divi

sm# p

artici

pant

s ent

ered p

rogram

(Ent

)# n

eutra

l disc

harg

es (N

eu)

# arre

sted f

or an

y new

offen

se (A

rr)# c

onvic

ted of

any n

ew of

fense

(Con

v)Da

te sta

mpe

dAlt

ernati

ves:

# of n

ew ar

rests

per p

artici

pant

(NoA

rr)

# of n

ew co

nvict

ions p

er pa

rticip

ant

(NoC

onv)

# of d

ays f

rom en

try to

first

new

arres

t (D

ayAr

r)

Re-a

rres

t rat

e =

Arr

(En

t –

Neu)

- and

/or -

Re-co

nvic

tion

rate

= C

onv

(Ent

– Ne

u)

Tim

e to

re-a

rres

t = D

ayAr

r

Leve

l of o

ffens

e (fe

lony v

s. m

is-de

mea

nor v

s. su

mm

ary)

Type o

f offe

nse

(e.g.,

drug

-relat

ed,

theft

, prop

erty,

or

violen

t)

In-pr

ogram

vs.

post-

prog

ram

recidi

vism

Resu

mpt

ion of

cri

mina

l acti

vity

High

ly pr

edict

iveNo

infor

mati

on

Inca

rcer

atio

n# p

artici

pant

s ent

ered p

rogram

(Ent

)# n

eutra

l disc

harg

es (N

eu)

# inc

arcera

ted fo

r any

new

offen

se or

tec

hnica

l viol

ation

(Inc)

# day

s inc

arcera

ted (D

ayInc

)

New

inca

rcer

atio

n ra

te =

Inc

(E

nt –

Neu)

Days

inca

rcer

ated

= D

ayIn

c

In-pr

ogram

vs.

post-

prog

ram

incarc

eratio

n

Priso

n vs.

jail

Exten

t of

incarc

eratio

nHi

ghly

pred

ictive

No in

form

ation

Tim

elin

ess o

f se

rvic

es

# day

s from

arres

t or v

iolati

on to

entry

(D

ayAE

)

# day

s from

entry

to fir

st tre

atmen

t con

-tac

t (Day

ET)

Inta

ke e

ffici

ency

= D

ayAE

Trea

tmen

t ent

ry e

ffici

ency

=

DayE

T

Abilit

y of p

rogr

am

to red

uce d

elays

in

case

reso

lution

an

d cap

italiz

e on

moti

vatio

n from

the

arres

t eve

nt

Mod

eratel

y pr

edict

iveM

axim

um of

50 da

ys

from

arres

t to en

try;

ideall

y with

in 20

days

56

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

TABL

E 1:

Rec

omm

ende

d Pe

rfor

man

ce In

dica

tors

& P

oten

tial P

erfo

rman

ce B

ench

mar

ks fo

r DTC

Eva

luat

ions

in th

e A

mer

icas

*C

orepe

rforman

cein

dicatorsre

flectth

ede

finingfeaturesofD

TCs,arere

ason

ablyin

expe

nsivetocollect,and

sign

ificantlypredictlo

ng-te

rmoutcomes.Re

commen

dedpe

rforman

ce



aybedifficulttom

easureorredu

ndan

twith

otherperform

ancein

dicators.Discretio

naryperform

ancein

dicatorsarethe

oreti

cally

impo

rtan

ttoDT

Csbuthavebe

enin

sufficien

tlystud

ied.

**Pe

rforman

ceben

chmarksarederived

fromstudiescon

ducted

inAustralia,C

anad

a,and


esamepe

rforman

ceben

chmarksm

aynotbevalid

inSou

th

Amer

ican

or C

arib

bean

cou

ntrie

s.

Vari

able

Data

Ele

men

tsPe

rfor

man

ce In

dica

tors

*Su

b-an

alys

esIn

terp

reta

tion

Pred

ictio

n of

Po

st-P

rogr

am

Reci

divi

sm o

r Co

sts

Pote

ntia

l Pe

rfor

man

ce

Benc

hmar

ks**

Prob

atio

n su

perv

isio

n

# prob

ation

sessi

ons s

ched

uled (

ProbS

)# p

robati

on se

ssion

s atte

nded

(Prob

A)# p

robati

on se

ssion

s can

celle

d or r

e-sch

edule

d (Pro

bC)

# prob

ation

field

visits

; e.g.

, hom

e, wo

rk,

schoo

l (Prob

FV)

Date

stam

ped

Prob

atio

n do

se =

Pro

bA +

Pr

obFV

Prob

atio

n at

tend

ance

rate

=

Prob

A (P

robS

– Pr

obC)

By m

onth

or

phas

e of p

rogram

(d

ensit

y)

Field

vs. o

ffice

visits

Inten

sity o

f pro

ba-

tion s

uperv

ision

Ins

ufficie

ntly

studie

dNo

infor

mati

on

Rew

ards

&

sanc

tions

# rew

ards a

dmini

stered

per p

artici

pant

(R

ew)

# san

ction

s im

pose

d per

parti

cipan

t (Sa

nc)

# infr

actio

ns pe

r part

icipa

nt (In

fr)# a

chiev

emen

ts pe

r part

icipa

nt (A

ch)

Bala

nce

of re

info

rcem

ent =

Rew

Sa

nc

Cert

aint

y of

rew

ards

= R

ew A

ch

Cert

aint

y of

sanc

tions

= S

anc

Infr

By m

agnit

ude o

f rew

ards o

r sev

erity

of sa

nctio

ns

Cons

eque

nces

im

pose

d for

parti

ci-pa

nts’

cond

uct

Incon

sisten

tly

pred

ictive

4 : 1

ratio

of rew

ards t

o sa

nctio

ns

Rest

orat

ive

just

ice

inte

rven

tions

# hou

rs co

mm

unity

servi

ce pe

rform

ed

Paym

ent o

f cou

rt fin

es &

fees (

none

, pa

rtial,

com

plete)

Resti

tutio

n to v

ictim

s (no

ne, p

artial

, co

mple

te)

Satis

factio

n of o

ther

finan

cial o

bliga

-tio

ns; e

.g., ch

ild su

ppor

t, alim

ony (

none

, pa

rtial,

com

plete)

# hou

rs in

victim

impa

ct pa

nels

Not d

efine

d in t

he lit

eratu

reDe

gree

to w

hich t

he

prog

ram ap

plies

res

torati

ve ju

stice

pr

incipl

es an

d int

er-ve

ntion

s

Incon

sisten

tly

pred

ictive

No in

form

ation

57

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

TABL

E 1:

Rec

omm

ende

d Pe

rfor

man

ce In

dica

tors

& P

oten

tial P

erfo

rman

ce B

ench

mar

ks fo

r DTC

Eva

luat

ions

in th

e A

mer

icas

*C

orepe

rforman

cein

dicatorsre

flectth

ede

finingfeaturesofD

TCs,arere

ason

ablyin

expe

nsivetocollect,and

sign

ificantlypredictlo

ng-te

rmoutcomes.Re

commen

dedpe

rforman

ce



aybedifficulttom

easureorredu

ndan

twith

otherperform

ancein

dicators.Discretio

naryperform

ancein

dicatorsarethe

oreti

cally

impo

rtan

ttoDT

Csbuthavebe

enin

sufficien

tlystud

ied.

**Pe

rforman

ceben

chmarksarederived

fromstudiescon

ducted

inAustralia,C

anad

a,and


esamepe

rforman

ceben

chmarksm

aynotbevalid

inSou

th

Amer

ican

or C

arib

bean

cou

ntrie

s.

Vari

able

Data

Ele

men

tsPe

rfor

man

ce In

dica

tors

*Su

b-an

alys

esIn

terp

reta

tion

Pred

ictio

n of

Po

st-P

rogr

am

Reci

divi

sm o

r Co

sts

Pote

ntia

l Pe

rfor

man

ce

Benc

hmar

ks**

Empl

oym

ent

# part

icipa

nts e

ntere

d prog

ram (E

nt)

# neu

tral d

ischa

rges

(Neu

)# e

mplo

yed a

t ent

ry (Em

pE)

# em

ploye

d at d

ischa

rge (

EmpD

)

Disc

harg

e em

ploy

men

t rat

e =

Em

pD (

Ent –

Neu

)

Empl

oym

ent i

mpr

ovem

ent r

ate

= (E

mpD

– Em

pE)

(Ent

– Ne

u)

Full t

ime v

s. pa

rt tim

eEx

cludin

g tem

-po

rary,

seas

onal

or of

f the b

ooks

em

ploym

ent

Type o

f wor

k (e.g

., cle

rical,

man

ual

labor,

supe

rviso

ry)

Degr

ee to

whic

h the

pr

ogram

impr

oves

th

e em

ploym

ent o

f pa

rticip

ants

High

ly pr

edict

iveDT

C offe

rs vo

catio

nal

servi

ces a

s nee

ded

Emplo

ymen

t (or

educ

ation

al en

rollm

ent) i

s a c

ondit

ion of

gr

adua

tion

Educ

atio

n# p

artici

pant

s ent

ered p

rogram

(Ent

)# n

eutra

l disc

harg

es (N

eu)

# with

high

scho

ol dip

loma o

r equ

ivalen

-cy

at en

try (D

ipE)

# with

high

scho

ol dip

loma o

r equ

ivalen

-cy

at di

scharg

e (Di

pD)

# enr

olled

in ed

ucati

onal

prog

ram at

en

try (E

dE)

# enr

olled

in ed

ucati

onal

prog

ram at

dis

charg

e (Ed

D)

Educ

atio

nal i

mpr

ovem

ent r

ate

= (D

ipD

+ E

dD) –

(Dip

E +

EdE

)

(Ent

– Ne

u)

Degr

ee to

whic

h the

pr

ogram

impr

oves

th

e edu

catio

n of

parti

cipan

ts

Incon

sisten

tly

pred

ictive

Educ

ation

al de

gree

or

enrol

lmen

t (if

unem

ploye

d) is

a c

ondit

ion of

gr

adua

tion

Hou

sing

# part

icipa

nts e

ntere

d prog

ram (E

nt)

# neu

tral d

ischa

rges

(Neu

)# i

n stab

le ho

using

at en

try (H

ouE)

# in s

table

hous

ing at

disch

arge (

HouD

)

Disc

harg

e ho

usin

g ra

te =

Hou

D (E

nt –

Neu)

Hous

ing

impr

ovem

ent r

ate

=

(Hou

D – H

ouE)

(En

t – N

eu)

By ty

pe of

hous

-ing

; e.g.

, livin

g wi

th fa

mily

or

frien

ds, re

cove

ry ho

use

Degr

ee to

whic

h the

pr

ogram

impr

oves

th

e hou

sing s

tabilit

y of

parti

cipan

ts

High

ly pr

edict

iveSo

ber h

ousin

g is

a con

dition

of

grad

uatio

n

58

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

TABL

E 1:

Rec

omm

ende

d Pe

rfor

man

ce In

dica

tors

& P

oten

tial P

erfo

rman

ce B

ench

mar

ks fo

r DTC

Eva

luat

ions

in th

e A

mer

icas

*C

orepe

rforman

cein

dicatorsre

flectth

ede

finingfeaturesofD

TCs,arere

ason

ablyin

expe

nsivetocollect,and

sign

ificantlypredictlo

ng-te

rmoutcomes.Re

commen

dedpe

rforman

ce



aybedifficulttom

easureorredu

ndan

twith

otherperform

ancein

dicators.Discretio

naryperform

ancein

dicatorsarethe

oreti

cally

impo

rtan

ttoDT

Csbuthavebe

enin

sufficien

tlystud

ied.

**Pe

rforman

ceben

chmarksarederived

fromstudiescon

ducted

inAustralia,C

anad

a,and


esamepe

rforman

ceben

chmarksm

aynotbevalid

inSou

th

Amer

ican

or C

arib

bean

cou

ntrie

s.

Vari

able

Data

Ele

men

tsPe

rfor

man

ce In

dica

tors

*Su

b-an

alys

esIn

terp

reta

tion

Pred

ictio

n of

Po

st-P

rogr

am

Reci

divi

sm o

r Co

sts

Pote

ntia

l Pe

rfor

man

ce

Benc

hmar

ks**

Emot

iona

lH

ealth

# part

icipa

nts e

ntere

d prog

ram (E

nt)

# neu

tral d

ischa

rges

(Neu

)# r

epor

ted se

rious

emoti

onal

prob

lems a

t ent

ry (Em

otE)

# rep

orted

serio

us em

otion

al pr

oblem

s at

disch

arge (

EmotD

)

Disc

harg

e em

otio

nal p

robl

ems

= E

mot

D (E

nt –

Neu)

Emot

iona

l im

prov

emen

t rat

e =

(E

mot

E – E

mot

D) (

Ent –

Neu

)

By ty

pe of

sym

p-tom

s; e.g

., de-

pres

sion,

hosti

lity,

paran

oia

Degr

ee to

whic

h the

pr

ogram

impr

oves

th

e em

otion

al he

alth

of pa

rticip

ants

Mod

eratel

y pre-

dictiv

eDT

C offe

rs m

ental

he

alth c

ouns

eling

as

need

ed

Emoti

onal

prob

lems

have

subs

tantia

lly

impr

oved

prior

to

disch

arge

Med

ical

and

de

ntal

Hea

lth

# part

icipa

nts e

ntere

d prog

ram (E

nt)

# neu

tral d

ischa

rges

(Neu

)# r

epor

ted se

rious

med

ical o

r den

tal

prob

lems a

t ent

ry (M

edE)

# rep

orted

serio

us m

edica

l or d

ental

pr

oblem

s at d

ischa

rge (

Med

D)

Disc

harg

e m

edic

al o

r den

tal

prob

lem

s = M

edD

(Ent

– Ne

u)M

edic

al a

nd d

enta

l im

prov

e-m

ent r

ate

= (M

edE

– Med

D)

(Ent

– Ne

u)

Degr

ee to

whic

h the

pr

ogram

impr

oves

th

e med

ical a

nd

dent

al he

alth o

f pa

rticip

ants

Insuffi

cient

ly stu

died

DTC o

ffers

med

ical

and d

ental

servi

ces a

s ne

eded

Fam

ily

Rela

tions

hips

# part

icipa

nts e

ntere

d prog

ram (E

nt)

# neu

tral d

ischa

rges

(Neu

)# r

epor

ted se

rious

fam

ily pr

oblem

s at

entry

(Fam

E)# r

epor

ted se

rious

fam

ily pr

oblem

s at

disch

arge (

Fam

D)

Disc

harg

e fa

mily

pro

blem

s =

Fam

D (E

nt –

Neu)

Fam

ily im

prov

emen

t rat

e =

(F

amE

– Fam

D) (

Ent –

Neu

)

Degr

ee to

whic

h the

pr

ogram

redu

ces t

he

family

prob

lems o

f pa

rticip

ants

Insuffi

cient

ly stu

died

DTC o

ffers

family

coun

-se

ling a

s nee

ded

Fam

ily pr

oblem

s hav

e su

bstan

tially

impr

oved

pr

ior to

disch

arge

Soci

al

Rela

tions

hips

# part

icipa

nts e

ntere

d prog

ram (E

nt)

# neu

tral d

ischa

rges

(Neu

)# r

epor

ted se

rious

socia

l prob

lems a

t en

try (S

ocE)

# rep

orted

serio

us so

cial p

roblem

s at

disch

arge (

SocD

)

Disc

harg

e so

cial

pro

blem

s =

SocD

(En

t – N

eu)

Soci

al im

prov

emen

t rat

e =

(Soc

E – S

ocD)

(En

t– N

eu)

Degr

ee to

whic

h the

pr

ogram

redu

ces

the s

ocial

or in

ter-

perso

nal c

onflic

ts of

parti

cipan

ts

Insuffi

cient

ly stu

died

No in

form

ation

Birt

h of

Drug

-free

bab

ies

# dru

g- an

d alco

hol-fr

ee ba

bies d

elive

red

or fa

there

d by D

TC pa

rticip

ants

(Bab

)

Date

stam

ped

Drug

-free

bab

ies =

Bab

In-pr

ogram

vs.

post-

prog

ram

birth

s

Degr

ee to

whic

h the

pr

ogram

cont

ribut

es

to th

e hea

lth of

ne

wbor

ns

Insuffi

cient

ly stu

died

No in

form

ation

59

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Court Supervision

Inmanyways,courtsupervisionisthedefiningingredientofaDTC.Manycorrectional

rehabilitationprogramsprovidesubstanceusetreatment,probationsupervision,ordrug

andalcoholtestingfordrug-addictedpersonschargedwithcrimes;however,onlyDTCsare

supervisedbyajudgeandrequireparticipantstoappearfrequentlyincourtforstatusre-

viewhearings.NumerousstudiesinAustraliaandtheUnitedStateshavefoundthatout-

comesinDTCsweresignificantlyinfluencedbyhowoftenparticipantsappearedbeforethe

judgeincourt(Careyetal.,2012;Festingeretal.,2002;Jones,2011;Marloweetal.,2006,

2007;Mitchelletal.,2012;Zweigetal.,2012).

Courtsupervisioniscommonlymeasuredintermsofthe“dosage”ornumberofcourt

hearingsthatparticipantsattended.Dosageisgenerallyconsideredtobeaprogram-level

performance indicator because it indicates how frequently the judge supervised partici-

pantsfromthebench.Itisproposedasacoreperformanceindicator(indicatedbyred font

belowandinTable1becauseit:(1)reflectsadefiningingredientofDTCprograms,(2) is

simpleandinexpensivetocollect,(3)isobjectivelyandreliablymeasuredand(4)hasbeen

foundconsistentlytopredictoutcomesinDTCevaluations.

Dose of hearings = # of hearings attended

Evaluatorsmayalsowishtoexamineparticipants’appearancerateatscheduledcourt

hearings.Theappearancerateisaparticipant-levelperformanceindicatorbecauseitindi-

catesthedegreetowhichparticipantswerecompliantwiththeirrequirementstoattend

courthearings.Thisvariableisslightlymorecomplicatedtomeasurethandosagebecause

itrequirestheevaluatortoexamineboththenumberofhearingsthatwereattendedand

thenumberofhearingsthatwereoriginallyscheduled.Itiscalculatedbydividingthenumber

ofhearingsthatwereattendedbythenumberthatwerescheduled,minusanyhearings

thatmayhavebeencancelledorrescheduledinadvanceorwiththecourt’sapproval.

60

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Court appearance rate =# of hearings attended

# of hearings scheduled – # of hearings cancelled or rescheduled

Inmanystudies,courtappearancerateshavenotbeensignificantlymorepredictiveof

outcomesthanthedosageofcourthearings.Perhapsbecausetherepercussionsformissing

courtsessionscanbesevereandmayincludeincarceration,appearancerateshavetypically

exceeded85percent.Formathematicalreasons,variablestendtobelesspredictivewhen

thereisarestrictedrangeofscoresonthosevariables.Therefore,thecourtappearancerate

islistedasadiscretionaryperformanceindicator(representedbyblackfontintheformula

aboveandinTable1).OASmemberstatesmightfindthisperformanceindicatortobemore

predictiveofoutcomesifthereisgreatervariabilityonthemeasureintheirDTCs.

Finally,bydate-stampingthe informationrelatedtoparticipants’attendanceatcourt

hearings, evaluators canmeasure the density of court hearings thatwere scheduled or

attendedduringeachmonthorphaseoftheprogram.Intermsofperformancebenchmarks,

evidencefromtheUnitedStatessuggestscourthearingsshouldbeheldeverytwoweeks

during thefirstphaseof theprogram,butmaysubsequentlybe reduced tomonthly for

participantswhoarecompliantwiththeirtreatmentobligations(Careyetal.,2012;Marlowe

etal.,2007).IfOASmemberstateswishtosimilarlydeterminewhenitissafeandeffective

toreducethe frequencyofcourthearings in theirprograms, itwillbenecessaryto

date-stamptheattendanceinformationatcourthearings.

Substance Use Treatment

Substance use treatment is another defining ingredient of a DTC. The basic

assumptionof theDTCmodel is thataddiction is causingorexacerbatingparticipants’

criminalactivity.Therefore,substanceusetreatmentisbelievedtobeessentialtoachieve

long-termdesistance from crime. In fact, if an individual can desist from crimewithout

61

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

substanceusetreatment,heorsheisgenerallynotconsideredtobeasuitablecandidate

foratraditionalDTCprogram(Marlowe,2012b).

Similartomeasuringcourtsupervision,theprovisionofsubstanceusetreatment is

commonlymeasuredintermsofdosage(i.e.,thenumberofsessionsthatwereattended)

orparticipants’attendancerateatscheduledtreatmentsessions.Thedosageoftreatment

is considered a program-level performance indicator because it reflects the amount

of treatment services thatwere delivered. The attendance rate is a participant-level

performanceindicatorbecauseitreflectsthedegreetowhichparticipantscompliedwith

theirtreatmentobligations.Finally,bydate-stampingtreatmentinformation,evaluatorscan

measurethedensityoftreatmentthatwasscheduledordeliveredduringeachmonthor

phaseoftheprogram.

Thedosageofsubstanceusetreatment isproposedhereasacoreperformancein-

dicatorbecause it isadefining ingredientofDTCprograms, issimpleand inexpensiveto

collect,canbeobjectivelyandreliablymeasured,andsignificantlypredictsoutcomesinDTC

programs.StudiesinDTCshaveconsistentlyfoundthatthelongerparticipantsremainedin

substanceusetreatmentandthemoresessionstheyattended,thebettertheiroutcomes

(Banks&Gottfredson,2003;Gottfredsonetal.,2007;Gottfredsonetal.,2008;Petersetal.,

2002;Shaffer,2010;Taxman&Bouffard,2005).

Dose of treatment = # of treatment sessions attended

Participants’attendancerateintreatmentislistedasarecommended performance in-

dicator forOASmemberstates (reflectedbyblue font belowand inTable1). Similar to

treatment dosage, this indicator significantly predicts post-programoutcomes.However,

theattendancerateisabitmoredifficulttomeasurethandosagebecauseitrequiresthe

evaluator to examineboth thenumberof sessions thatwere attendedand thenumber

ofsessions thatwereoriginallyscheduled. Inaddition,becausethedosageof treatment

andtheattendanceratearepartiallyredundant,itisunclearwhethertheattendancerate

62

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

predictsoutcomessignificantlybetterthandosagealone.EvaluatorsinOASmemberstates

mightfindthatthedosageoftreatmentisadequatetocharacterizetheprovisionoftreat-

mentandpredictpost-programoutcomesintheirDTCprograms.

The treatmentattendance rate is calculatedbydividing thenumberof treatment

sessionsthatwereattendedbythenumberoriginallyscheduled,minusanysessionsthat

werecancelled,excusedorrescheduledinadvanceorwiththeapprovaloftreatmentstaff.

Treatment attendance rate =# of sessions attended

# of sessions scheduled- # of sessions cancelled or rescheduled

In termsofperformancebenchmarks, studies inCanadaand theUnited Stateshave

reportedthatthebestoutcomeswereachievedwhenparticipantsattendedapproximately

sixtotenhoursoftreatmentper weekduringthefirstphaseoftheprogram,foratotalof

100to200hoursoftreatmentoverninetotwelvemonths(Bourgon&Armstrong,2005;

Huebner&Cobbina,2007;Landenberger&Lipsey,2005;Petersetal.,2002;Sperberetal.,

2013).Thestudiesfoundthat100hoursoftreatmentwasgenerallysufficientfortheless

seriouslyaddictedorantisocialparticipants,whereas200hourswasrequiredforhigh-risk

participantswhohadmoreseriousaddictionsorcriminalhistories.Itisunknownwhether

thesesameperformancebenchmarkswillbeapplicable inSouthAmericanorCaribbean

countries.LocalevaluatorsinOASmemberstatesareencouragedtoexaminetheoptimum

dosageofsubstanceusetreatmentintheirownDTCprograms.

Itisalsoimportanttocategorizesubstanceusetreatmentbythemodalityorlevel

ofcare. Forexample,residentialor inpatienttreatment isusuallyconsiderablymoreex-

pensivethanoutpatienttreatmentandmaybevalueddifferentiallyincostevaluations.In

addition,thetimeparticipantsspendinresidentialorinpatienttreatmentislikelytoreduce

theirtime at libertysignificantly.Thismeansthatparticipantsarelikelytohavereduced

63

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

opportunitiestoengageinsubstanceuse,crime,andotherbehaviorsofinteresttoevalua-

torswhiletheyarebeingtreatedinaresidentialsetting.Aswillbediscussedlater,theeval-

uatormayneedtostatisticallyadjustforthenumberofdaysparticipantswereinresidential

treatmentwhenconductingcertaintypesofoutcomeanalyses.Evaluatorsarestrongly

encouraged,therefore,todate-stampthedaysonwhichparticipantsenterandaresub-

sequentlydischargedfromresidentialandinpatienttreatmentprograms.

Itmayalsobeusefultocharacterizetheorientationorphilosophyoftreatment

beingdelivered.StudiesinCanadaandtheUnitedStateshavedeterminedthatcognitive-

behavioraltherapies(CBT)producedsignificantlybetteroutcomesamongcriminaljustice

populationsthaninsight-orientedorpsychodynamictherapies(Andrews&Bonta,2010;

Gendreau,1996).Anumberofratingtoolshavebeendevelopedtomeasurethequality

andorientationoftreatmentandotherservicesdelivered incorrectionalrehabilitation

programs. These include the Correctional ProgramAssessment Inventory (CPAI)which

wasdevelopedinCanada,andtheCorrectionalProgramChecklist(CPC)developedinthe

UnitedStates.Higherratingsoftreatmentqualityontheseinstrumentshavebeenshown

to predict higher graduation rates and lower recidivism rates among DTC participants

(Gutierrez& Bourgon, 2012) and participants in other correctional treatment programs

(Lowenkampetal.,2006,2010).Importantly,ratersmustbetrainedcarefullyonhowto

administertheseinstruments.Informationontrainingcurriculafortheseinstrumentsis

providedintheAppendix.

Finally, thequalityof therelationshipbetweentheparticipantandcounselorhas

alsobeenshowntosignificantlyinfluenceoutcomesinsubstanceusetreatment.Aswillbe

discussedlater,severalinstrumentshavebeendevelopedtoassessparticipants’satisfaction

with substance use treatment and the quality of the therapeutic alliance between the

participantandtreatmentstaff.Evaluatorsmaywishtoconsiderusingthesetools,which

arelistedintheAppendix,intheirDTCprogramevaluations.

64

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Drug and Alcohol Testing

DrugandalcoholtestingisanotherkeycomponentofDTCprograms.Drugandal-

coholtestresultsoftenserveastheprincipalbasisforassessingparticipants’response

to treatment, adjusting the conditionsof treatmentwhere indicated, and administering

rewardsforabstinenceandsanctionsforcontinuedsubstanceuse.Participants inDTCs

commonly attribute their success, in part, to the frequency of drug and alcohol testing

(Goldkampetal.,2002;Kleinpeteretal.,2010;Wolfer,2006).

Somedrugtests,suchasurineorsalivatests,mustbeadministeredrepeatedlybecause

theycanonlydetectdrugoralcoholuseovershortperiodsofapproximatelytwotofour

days.Forthesetypesoftests,thecoreprogram-levelperformanceindicatoristhedosage

ornumberofteststhatwereadministered. Othertests,suchassweatpatchesorankle

monitors,canprovidecontinuoussurveillanceofdrugoralcoholuseoverextendedperiods

ofseveralweeks. Forthesetests,thecoreperformance indicator isthenumberofdays

participantsweresubjectedtocontinuoussurveillance.

Testing dose = # of urine, breath, or saliva tests administered

-and/or–

Testing dose = # of days on continuous surveillance

Evaluatorsmaywishtoexamineparticipants’complianceratewithscheduleddrugtests.

Thecompliancerateiscalculatedbydividingthenumberofteststhatwereprovided,minus

anyteststhatwere invalidoradulterated,bythenumberofteststhatwerescheduled

minusteststhatwereexcusedinadvanceorwiththecourt’sapproval.

Testing compliance rate =# of drug tests provided-# of drug tests invalid or adulterated

# of drug tests scheduled-# of drug tests excused

Fewstudieshaveexaminedtheimpactofcompliancewithdrugtestingonoutcomesin

DTCs.Compliancewithdrugtestingisoftenhighlycorrelatedorredundantwiththedosage

65

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

ofdrugtesting (discussedabove)andwith theresultsofdrugtesting (discussedbelow);

therefore, the compliance rate is unlikely to provide significant predictionbeyondother

performanceindicators.Forthisreason,itislistedasadiscretionaryperformanceindicator.

StudiesintheUnitedStateshaverevealedthattoachievethemosteffectiveoutcomes,

drugtestswithshortdetectionwindowsshouldbeadministeredno less frequentlythan

twiceperweekforatleastthefirstseveralmonthsoftheprogram(Careyetal.,2012;Gibbs

&Wakefield,20142014;Kleinpeteretal.,2010)andcontinuoussurveillancetestingshould

beconducted foraminimumofninetyconsecutivedays (Flango&Cheesman,2009). It

remainstobedeterminedwhethersimilarperformancebenchmarkswillemergeinother

OASmemberstates.

Abstinence

Achievingabstinencefromillicit2drugsandalcoholisoneoftheprimarygoalsofa

DTC.Asdiscussedearlier,theDTCmodelassumessubstanceuseiscausingorexacerbating

criminal activity. Therefore, abstinence from addictive and intoxicating substances is

believedessentialforreducingcriminalrecidivism.

Fordrugtestswithshortdetectionwindows,thecoreparticipant-levelperformance

indicatorforabstinenceisthepercentageofteststhatwerenegativeforallillicitdrugsand

alcohol.Samplesthatareinvalidoradulteratedarenotcountedasdrug-negative.Fortests

involving continuous surveillance, the coreperformance indicator is thenumberofdays

participantswereoncontinuoussurveillancewithoutapositivetestreading.

Abstinence rate=# of drug tests negative for all illicit substances – # of drug tests invalid or adulterated

# of drug tests scheduled – # of drug tests excused

-and/or–

Abstinence = # of days with no positive reading on continuous monitoring

2 Illicitdrugsincludeprescriptionmedicationsthatareusedforanon-prescribedornon-medically-indicatedpurpose.

66

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

In termsofperformancebenchmarks, studies from theUnitedStates suggestninety

consecutivedaysofabstinenceisaminimumthresholdforachievingpositiveoutcomes

aftergraduation (Careyetal.,2012). Ideally,programsshould strive forat leastone

hundredandeightydaysofconsecutiveabstinencepriortograduationinordertoincrease

theoddsofstable,long-termsobriety(McLellanetal.,2000).

The failuretoprovideadrugoralcohol testhasanuancedsignificancethatmerits

particularattention.BecauseparticipantsinDTCsaresometimesrewardedforproviding

negativedrugtestsandsanctionedforprovidingpositivedrugtests,theyareconsiderably

morelikelytoprovideaspecimeniftheyexpectittobedrug-negativethaniftheyexpectit

tobedrug-positive.

Forthisreason,thefailuretoprovideaspecimencannotbeassumedtobearandom

or ignorable event, but rather is likely to reflect an effort to conceal substance use. In

fact,manyDTCsview the failure toprovideavalid specimenasa separate infraction

from substanceuse, andone thatmay receive amore severe sanction thanproviding

adrug-positivespecimen,whichoften receivesa light sanctionoradjustment to the

participant’streatmentplan(Marlowe,2008,2011).Thisisbecauseparticipantswhofail

toprovideaspecimenarereasonablyassumedbothtohaveengagedinsubstanceuseand

tohaveconcealedtheusage.Thus,theDTCmightgiveparticipanttworesponsesoramore

severesanctionforthiscompoundissue.

Thegenerallyrecommendedcourseofactionistoassumethatmissingspecimensare

drug-positive.Ofcourse,ifaparticipantwasexcusedbystafffromprovidingaspecimen

foranacceptablereason(e.g.,becauseofillnessoranemploymentobligation),thenitis

appropriatetotreatthemissingspecimenasmissingdata.Evaluatorsmayelecttoexamine

thedatabothways,firstbytreatingmissingspecimensasdrug-positiveandsecondlyby

treatingthemasmissingdata.However,iftheresultsaresignificantlydiscrepantinthetwo

analyses,theformerresultsaremorelikelytoreflectthetruelevelsofdruguse.

Thereareadvancedstatistical imputationproceduresthatcansometimesbeusedto

67

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

compensateformissingdrug-testdata.Someimputationproceduresexaminethepattern

ofdrugtestresults immediatelybeforeandafteramissingspecimen.Forexample, ifa

participanthadseveralnegativespecimensjustbeforeandjustafteramissedspecimen,

themissingspecimenmightbepresumedtobedrug-negative.Alternatively,somepro-

ceduresmayimputetheaverageormostprevalentresultfortheparticipantorforthe

populationasthemostlikelyresultinplaceofamissingspecimen.Selectinganappropriate

imputationprocedureiscomplicatedandexpertstatisticalconsultationisusuallyrequired

toapplytheseprocedurescorrectly.

Retention and Graduation

Retention in substanceuse treatment isoneof themost significantpredictorsof

long-termoutcomes inDTCprograms. The longerparticipants remain in treatment, the

better theiroutcomes (Gottfredsonetal., 2008;Peterset al., 2002;Taxman&Bouffard,

2005).Similarly,byfarthebestoutcomesareachievedbythosewhosuccessfullycomplete

orgraduatefromaDTCprogram(Careyetal.,2012;Gottfredsonetal.,2007;Mitchelletal.,

2012;U.S.GovernmentAccountabilityOffice,2005).

MostDTCs have similar requirements for graduation and hold formal graduation

ceremoniesthatmarksuccessfulcompletionoftheprogram.DTCsthatdonothaveformal

criteriaforgraduationorgraduationceremonieshavesignificantlypooreroutcomesthan

thosethatdo(Careyetal.,2012).Graduatesaretypicallyrequiredtocompleteasubstance

usetreatmentprogram,maintainabstinencefromall illicitdrugsandalcohol forat least

ninetytoonehundredandeightydays,findemploymentorenrollinaneducationalpro-

gram,behoused insober livingquarters,payapplicablefinesandfees,andavoidnew

criminalchargesorserioustechnicalviolations.Participantswhocanachievethesegoals

arefarmorelikelythanotherstoremaindrug-freeandcrime-freeaftergraduation.

SomeDTCsmayassignaneutral dischargetoparticipantswhowerewithdrawnfrom

68

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

theprogramforreasonsbeyondthecontroloftheparticipantandtheprogram.Aneutral

dischargeisassignedmostcommonlytoparticipantswhoarewithdrawnfromtheprogram

becausetheymovedoutofthejurisdiction(withthecourt’spermission),enlistedinthe

military, or became toomedically or psychiatrically unstable to provide continuing

competent consent to participation.Aneutral dischargemay also be assigned if it is

discoveredafterthefactthattheparticipanthadbeenadmittedtotheprogramerro-

neously;forexample,iftheparticipanthadapriorconvictionthatprecludedeligibilityor

residedinajudicialdistrictthatwasnotwithinthejurisdictionalboundariesoftheDTC.A

neutraldischargeshouldneverbeappliedtocasesinwhichterminationwasrelatedinany

waytoaparticipant’sperformanceintheDTC.

Participantswho receive aneutral discharge are removed from thedenominator

whencalculatingthegraduationrate.Theformulaisthereforethenumberofparticipants

whograduatedfromtheprogram,dividedbythenumberwhoenteredtheprogramminus

thenumberwhoreceivedaneutraldischarge.ParticipantswhoarestillenrolledintheDTC

arenot included in this calculation. In termsofperformancebenchmarks, theaverage

graduationrateisapproximatelysixtypercentintheUnitedStates,withmostDTCsgraduating

betweenfiftyandseventy-fivepercentoftheirparticipants(Marloweetal.,2016).

Abstinence rate =# of participants who graduated

# of participants who entered DTC program = # of neutral discharges

Aswillbediscussedlater,evaluatorssometimesmeasureoutcomesbasedonco-

hortsofDTCparticipants.AcohortisdefinedasagroupofindividualswhoenteredtheDTC

duringthesamespecifiedtimeperiod,usuallyanintervaloftwelvemonths(Heck,2006;

Rubioetal.,2008).Forexample,allparticipantswhoenteredtheDTCbetweenJanuary1

andDecember31ofagivenyearmightbedefinedasacohort.Outcomeanalysesmaythen

beconductedseparatelyforcohortsdefined,forexampleashavingenteredtheprogram

betweenJanuary1andDecember31,2012,betweenJanuary1andDecember31,2013,

69

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

andsoforth.Thisprovidesaseriesof“snapshots”indicatinghowwelltheDTCperformed

oversuccessiveyearsofoperation.

Whenevaluatingcohorts,itisoftennecessarytoexamineparticipantswhoarestill

enrolledintheDTC.Inthiscase,theretentionrateiscalculatedbydividingthenumberof

participantswhograduatedfromtheprogramplusthenumberthatarestillenrolled,bythe

numberwhoenteredtheprogramminustheneutraldischarges:

Abstinence rate =# of participants graduated + # of participants still enrolled

# of participants that entered program - # of neutral discharges

Analternativewaytomeasureretention is the length of stay in theprogram.This is

definedasthenumberofdaysfromaparticipant’sentryintotheDTCtohisorherdischarge

or last in-person contactwith DTC staff. The length of stay is listed as a recommended

performance indicatorbecause it ishighlypredictiveofoutcomesbut isalso likely tobe

redundantwiththeretentionrateorgraduationrate.Intermsofperformancebenchmarks,

studiesintheUnitedStatessuggestDTCshavebetteroutcomeswhentheplannedlength

of stay for the program is between approximately eighteen and twenty-four months

(Careyetal.,2012;Shaffer,2010).

Length of stay = date of discharge or last personal contact with staff - date of entry

ItisnotalwaysobvioushowtodefinetheentrydateordischargedateforaDTCpartici-

pant.MostDTCshaveaformalentryhearingatwhichparticipantsenteraplea,voluntarily

waivecertainlegalrights,andareofficiallyenrolledintheprogram.Thedateofthishearing

istypicallycountedastheentrydatebecauseitmarksthespecificpointintimewhenthe

programgainedauthorityovertheindividualtoimposeconditionsoftreatmentandsuper-

vision.Someparticipantsmaystopattendingtreatmentforseveralweeksormonthsbefore

beingformallydischargedfromaDTC.Undersuchcircumstances,thelastin-personcontact

withstaffisthebestindicatorofwhentheparticipantlefttheprogram.

70

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Recidivism

Formanypolicymakersandmembersofthepublic,reducingcriminalrecidivismis

theprimarygoalofaDTC.Recidivismisdefinedhereasanyreturntocriminalactivityafter

theparticipantenteredtheDTCprogram.Ifaparticipantisarrestedfororchargedwitha

crimeforallegedactivitythatoccurredbeforeheorsheenteredtheDTC,thisisnotcounted

asrecidivismbecausetheeventoccurredpriortoentry.

Recidivismistypicallymeasuredbynew(1)arrests,(2)convictions,(3)incarcerations

or(4)self-reportedcriminalactivity.Eachmeasurehasitsownadvantagesanddisadvantages

thattheevaluatorwillneedtotakeintoconsideration.Incarcerationisdiscussedseparately

belowbecauseithasdistinctpublicsafetyandcostimplications.

There are several potential benefits to analyzingnewarrests as ameasureof

criminal recidivism. First, arrests are usually substantially closer in time to the alleged

criminal activity than convictions. In some countries, it may take months or years to

conductpleanegotiations,holdacriminaltrial,anddetermineguiltorinnocence.Evaluators

canusuallyreportarrestoutcomes inamuchshortertimethanwaitingfor lengthy legal

proceedingstoberesolved. Second,criminalcasesmaybedismissedorpleddowntoa

lesserchargeforreasonshavinglittletodowithfactualguilt,suchasinsufficientevidence

orpleabargains.Asaresult,theabsenceofaconvictionorconvictiononalessercharge

maynotreflecttheseverityoftheoffensethatactuallyoccurred.

On the other hand, individuals may be arrested for crimes they did not in fact

commit.Thismeasurecouldseriouslyoverestimatethelevelofcriminalrecidivism.Relying

onconvictiondataratherthanarrestdatamayprovidegreaterassurancesthatthecrimes

didinfactoccur.

Evaluatorsmust also consider the timeliness and accuracy of information that is

availableincriminaljusticedatabases.InsomecountriessuchastheUnitedStates,arrest

datamaybeenteredinamoretimelyandfaithfulmannerthanconvictiondata.Inother

71

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

countriessuchCanadaorJamaica,convictiondatamaybeequallyormoreaccuratethan

arrestdata. Evaluatorsmust familiarizethemselveswithhowandwhen information is

enteredintonational,state,andlocaladministrativedatabases.

Self-reportedinformationcanpotentiallyprovidethemostaccurateassessmentof

criminal recidivismbecause itdoesnotrequiredetectionorprosecutionby lawenforce-

ment.Becausemostcrimesgounreportedbyvictimsandundetectedbytheauthorities,

arrestandconvictiondataoftenunderestimatetruelevelsofcriminalactivity.Forobvious

reasons, however, individuals cannot be reliedupon to acknowledge crimesunless they

receivestrictassurancesthattheinformationwillbekeptconfidentialandwillnotbeused

againsttheminacriminalproceeding.ThisordinarilyrequiresaDTCtohireanindependent

interviewerwhohasnoconnectiontothecourtorcriminaljusticesystemtoconfidentially

surveytheparticipants.FewDTCsarelikelytohaveadequateresourcestohiresuchinde-

pendentinterviewers.

MostDTC evaluations in theUnited States have relied on arrests as the primary

measureof recidivism,whereasstudies inCanadaandAustraliahavereliedprimarilyon

convictions.With thenotableexceptionofonewell-fundednational study in theUnited

States(Rossmanetal.,2011),fewDTCevaluationshaveexaminedself-reportdataasthe

measureofrecidivism.

Arrest rates and conviction rates are calculated by dividing the number of

participantswhowerearrestedorconvictedofanewoffensebythenumberwhoentered

the program minus neutral discharges. Technical violations, defined as violations of a

courtorderthatdonotconstituteacrimeper se,areusuallycountedseparatelyfromnew

offenses.Forexample,drinkingalcoholislegalformostadultsinmostcountries,butmay

beatechnicalviolationforaDTCparticipantandcouldleadtoanewarrestorconviction

foraprobationviolation.Sanctionsfortechnicalviolationsareoftencountedasinvestment

costsfortheDTCprogram,whereassentencesordispositionsfornewoffensesareoften

countedasoutcomecosts.

72

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Re-arrest rate =# arrested

# entered program - # of neutral discharges

- and/or –

Re-arrest rate =# convicted


Evaluatorsmayalsowishtoexaminerecidivismintermsoftheaveragenumberof

newarrestsornewconvictionsperparticipant.Thisisusuallynotveryinformativewithin

thefirsttwelvetotwenty-fourmonthsofaparticipant’sentry.Itusuallytakesasubstantial

timeforparticipantstore-engageincriminalconduct,bedetectedbylawenforcement,and

havecriminalchargesfiledagainstthem.Forparticipantswhoreturnquicklytocriminal

behavior,mostwillhaveonlyoneortwonewarrestswithinthefirsttwoyears.Aswillbe

discussed later, thiscancreatewhat iscalledaskewed or non-normal distribution in the

data.Formathematicalreasons,oneisunlikelytodetectstatisticallysignificantdifferences

ifthedistributionisskewed.Therefore,recidivismduringthefirsttwoyearsisusuallymost

informativewhenanalyzed intermsofpercentagesofparticipantswhohadat leastone

newarrestorconviction.

Date-stamping arrest and conviction data permits the evaluator to analyze in-

programversuspost-programrecidivismseparately,andtocalculatetheaveragelengthof

timeuntilthefirstnewarrestorconvictionhasoccurred.Usingastatisticaltechniquecalled

a survival analysis,theevaluatorcandeterminethedegreetowhichtheDTCsignificantly

delayedtheonsetofnewcriminalactivity.Thetimetore-arrestislistedasarecommended

performanceindicatorbecauseithasimportantimplicationsforpublicsafetyandcostbut

isalsooftenhighlycorrelatedorredundantwithnewarrestandconvictionrates.

Itisusuallyadvisabletocategorizenewoffensesbythelevelorseverityofthe

offense(e.g.,felonyvs.misdemeanorvs.summaryoffenses)andbythetypeofcrimethat

was committed (e.g., drug offenses, property or theft offenses, violent offenses,

73

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

technicalviolations,prostitution,andtrafficoffenses).Differentcategoriesofoffensesoften

haveverydifferent implicationsforpublicsafetyandcost. Forexample,violentoffenses

mayhaveseriousvictimizationcostsandmayresultinsubstantialjailorprisonsentences,

whereasdrug-possessionoffensesmaynotdirectlyinvolvevictimsandaremorelikelyto

receivealesscostlyprobationsentence.

Aswill bediscussed later,most studieshaveexamined recidivism foraperiodof

between twoandfive years after entry into theDTC. BecausemanyDTCprogramsare

approximatelytwelvetoeighteenmonthsinlength,measuringoutcomesaftertwoyears

allowsadequatetimetoelapseformostparticipantstohavecompletedorbeendischarged

fromtheprogram.Afterthreetofiveyears,recidivismratesforpersonschargedwith

drug-relatedcrimesarelikelytoreachaplateau(Gossopetal.,2005;Inciardietal.,2004;

Martinetal.,1999).Thismeansthatmostparticipantswhowillrecidivatearelikelytohave

donesobythattime.Therefore,estimatesofrecidivismarelikelytobereasonablystable

afteraperiodofapproximatelythreetofiveyears.

Incarceration

Incarcerationtypicallyaccountsforthegreatestcosttosocietyfrompersonscharged

with crimes related to substanceusedisorders.Dependingon the country and the

conditionsofsupervision,adayinjailorprisonmaycostbetweenfiveandtwentytimes

morethanadayonprobationorincommunity-basedsubstanceusetreatment(Belenkoet

al.,2005;Zarkinetal.,2012).

Newincarcerationratesarecalculatedinasimilarmannertonewarrestratesandnew

convictionrates.Thenumberofparticipantswhowereincarceratedforanewoffenseis

dividedbythenumberwhoenteredtheprogramminustheneutraldischarges:

New incarceration rate =# incercarated

# entered program-# of neutral discharges

74

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

New incarceration ratesareusuallynot very informative,however,because theyare

redundantwithnewarrestandconvictionrates.Inmostinstances,aparticipantwillnot

beincarceratedunlessheorshewasfirstarrestedforandconvictedofanewoffenseorfor

repeatedlyviolatingthetermsofprobation.Therefore,newincarcerationratesgenerally

donotprovidemuchusefulinformationthatnewarrestandconvictionratesdonot.

Nonetheless,thelengthoftimethatparticipantswereincarceratedisfarmoreinforma-

tivethantheincarcerationrate,especiallywhenevaluatingthecostimpactsofaDTC.The

coreperformanceindicatoristhetotalnumberofdaysthatparticipantswereincarcerated.

Date-stampingisthereforecriticalforexaminingthelengthofincarceration.Thisrequires

theevaluatortorecordthedatesthatparticipantsenteredandweresubsequentlyreleased

fromcustody. Ifaparticipantwas incarceratedonmorethanoneoccasion,thedaysfor

eachepisodearesummedtodeterminethetotallengthofincarceration.

Days of incarceration = date of release from custody = date of entry into custody

Evaluatorstypicallydistinguishbetweenincarcerationthatoccurredwhileparticipants

wereenrolled in theDTC,and incarceration thatoccurredafterdischarge fromtheDTC.

In-programincarcerationoftenreflectsbriefjailsanctionsthatmaybeimposedbythejudge

asasanctionformisconduct intheprogram,whereaspost-programincarcerationtypically

reflectspre-trialdetentionor criminal sentences fornewcharges. In costevaluations,

in-programjailsanctionsareusuallycountedas investmentcosts fortheDTC,whereas

post-programsentencesordetentionarecountedasoutcomecosts.

Insomecountries,theremaybeanimportantdistinctionbetweenjailsandprisons.In

thosenationalcontexts,jailsaretypicallyusedforpre-trialdetentionorshortsentencesof

lessthanoneyear,whereasprisonsaretypicallyreservedforsentencesoflongerthanone

year.Jailsareoftenlocatedinandadministeredbythesamecountyorprecinctwherethe

offenderresidesorwheretheoffensewascommitted,whereasprisonsmaybelocateda

75

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

distanceawayandmaybeadministeredbystate,provincial,orfederalgovernmentagen-

cies.Fromafiscalstandpoint,adayinprisonmaybesubstantiallymoreexpensivethana

dayinjail,andmaythusbevalueddifferentlyinacostanalysis.

Timeliness of Services

WhentheearliestDTCsweredevelopedinthelate1980sandearly1990s,twooftheir

goalsweretoclearcloggedcaseloadsinthecourtsandresolvedrug-relatedcasesmore

efficiently.Havingaspecializedteamwithrepresentativesfromthedefenseandprosecu-

tionwas intended, inpart, toachievea speedier resolutionof factual controversiesand

imposerehabilitativedispositionsmorequickly.

Likemanyotherperformanceindicatorsmentionedinthismanual,measuringthetime-

linessofservicesinaDTCrequirescarefulattentiontodate-stamping.Onemeasure,intake

efficiency,isdefinedasthenumberofdaysfromarresttoentryintotheDTCprogram.Some

DTCsserveindividualswhoarechargedwithviolationsofprobation,inwhichcasethedateof

theprobationviolationiscountedinlieuofanarrestdate.StudiesintheUnitedStateshave

foundthatoutcomesweresignificantlybetterforDTCsthatreducedthedelayfromarrestto

entrytolessthanfiftydays,andideallytolessthantwentydays(Careyetal.,2012).Evalua-

torsfromotherOASmemberstateswillneedtoestablishtheirownperformancebenchmarks

fortheirprograms,butitishighlylikelythatspeedierentryintotheDTCprogramwillproduce

significantlybetteroutcomes.Intakeefficiencyisthereforecalculatedas:

Intake efficiency = Date of entry into DTC-Date of arrest or probation violation

EvaluatorsmayalsowishtocalculatethetimedelayfromentryintotheDTCtothefirst

treatmentsession,calledtreatment intake efficiency:

Inmoststudiestodate,onceparticipantsenteredtheDTCs,theprogramswereusually

abletoinitiatetreatmentwithinonlyaboutonetotwoweeks.Therefore,theshorttime

delaybetweenentryandthefirsttreatmentsessionwasnotfoundtopredictoutcomesin

76

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

manyinstances.Ifothercountriesexperiencelongerdelaysininitiatingtreatmentfortheir

DTCparticipants,thisperformanceindicatormightprovidebetterpredictionofoutcomes.

Becauseithasgenerallynotbeenfoundtobepredictiveofoutcomesthusfar,itislistedasa

discretionaryperformanceindicatorforevaluatorstoconsider,andiscalculatedasfollows:

Treatment intake efficiency = Date of first treatment session-Date of entry into DTC

Probation Supervision

DTCscombinesubstanceusetreatmentwithstrictbehavioralaccountability.Insome

countriessuchastheUnitedStates,Canada,andtheUnitedKingdom,probationofficersor

othercommunitysupervisionofficersplayacriticalroleinsupervisingparticipantperfor-

manceintheprogram.Inthosecountries,theprobationofficermaymonitorparticipants’

compliancewithtreatment,conducthomeoremploymentvisits,performdrugandalcohol

testing,andreportcomplianceinformationtothejudge.Ifaparticipantcommitsaserious

infractionorabscondsfromtheprogram,theprobationofficermaybeempoweredtotake

theindividualintocustodyandbringhimorherbeforethecourtforaresponse.

Othercountries,suchasPanama,donothaveprobationofficers;rather,thejudgeand

psychosocialteamareprimarilyresponsibleformonitoringcomplianceintheprogram.For

countriesthatdonothaveprobationofficers,performanceindicatorsforprobationsuper-

visionmaynotbearelevantconsiderationforDTCevaluations.

Fewstudieshaveexamined thecontributionsofprobation toDTCoutcomes,andno

performancebenchmarkshaveyetbeen identified forprobationsupervision inDTCpro-

grams.Forthisreason,theperformanceindicatorsforprobationsupervisionarelistedas

discretionary.ItishopedthatevaluatorsinOASmemberstateswillfocusgreaterattention

onthisimportantissueandcontributetotheinternationalbodyofresearchonbestpractices

forprobationofficers,whererelevant,inDTCprograms.

Performanceindicatorsforprobationsupervisionaremeasuredinasimilarmannerto

77

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

those forcourthearingsandsubstanceusetreatment.Thedosageofprobationsuper-

visioniscalculatedbythenumberofin-officeprobationsessionsthatwereattendedplus

thenumberof field visits thatwere conducted.A field visitmight include inspecting

aparticipant’shomeorverifyingthattheparticipantwaspresentathisorherschoolor

placeofemploymentattheappropriatetimes.Thisvariableisconsideredaprogram-level

performance indicatorbecause it reflects theamountofprobationsupervision thatwas

providedtoparticipants.

Dose of probation = # of probation sessions attended + # of field visits

Theprobationattendanceraterepresentstheproportionofprobationsessionsthatwere

attended,dividedbythenumberofsessionsthatwerescheduledminusanysessionsthat

werecancelledorrescheduledinadvanceorwiththepermissionoftheprobationofficer.

Fieldvisitsarenotincludedinthiscalculationbecausetheyaretypicallyunannounced,and

participantsusuallydonotchoosewhethertocomplywiththem.Thisvariableisapartici-

pant-levelperformanceindicatorbecauseitreflectsthedegreetowhichparticipantswere

compliantwithprobationrequirements.Also,bydate-stampingprobationinformation,the

evaluatorcanmeasurethedensityofprobationservicesthatwerescheduledordelivered

duringeachmonthorphaseoftheprogram.

InmanyDTCs, probation supervision, like court hearings andother supervision re-

quirements,aregraduallyreducedinintensityorfrequencyasparticipantsmakeprogress

intreatment.Byexaminingthedensityofsupervisionpermonthorphaseoftheprogram,

evaluatorscanmeasurethedegreetowhichprobationsupervisionandotherserviceswere

decreasedinintensityovertime.Theprobationattendancerateiscalculatedasfollows:

Treatment intake efficiency =# of probation sessions attended

# of sessions scheduled - # of sessions cancelled or rescheduled

78

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Rewards and Sanctions

PerformanceindicatorsforrewardsandsanctionshavemetwithlimitedsuccessinDTC

evaluationstodate(Bonomo,2012;Linquistetal.,2006).Moststudieshavereportedthe

averagenumbersofrewardsandsanctionsthatwereadministered,andperhapscatego-

rizedtherewardsandsanctionsbymagnitudeorseverity.

Thisapproachhasnotbeeninformativeforatleasttworeasons.First,theimposition

of rewardsandsanctions ishighlycorrelatedwithotherperformance indicators,suchas

treatmentattendanceratesanddrugtestresults.Bydesign,participantsinDTCsreceive

rewards for suchbehaviorsasattending treatment sessionsandprovidingdrug-negative

urinespecimensandreceivesanctionsformissingtreatmentsessionsandcommittingother

violations.Therefore,thenumbersofrewardsandsanctionsthatareimposedaretypically

redundantwithotherperformanceindicators.

Second,behavioralresearchrevealsthatthecriticalfactorforsuccessisnothowmany

rewardsorsanctionswereimposed,butratherwhethertheywereappliedwithcertainty

andimmediacy(Harrell&Roman,2001;Hawkin&Kleiman,2009).Certaintyreferstothe

ratioofrewardstoachievements,ortheratioofsanctionstoinfractions(Marlowe&Kirby,

1999).Forexample,ifparticipantsreceivedarewardforeveryachievementintheprogram,

orasanctionforeveryinfraction,thenthecertaintyratiowouldbe1.0(or100percent)and

theprogramwouldbemorelikelytoproducefavorableoutcomes.

Certainty of rewards =# of rewards administered

# of achievements

Certainty of sanctions =# of sanctions imposed

# of infractions committed

79

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Thecertaintyratiosforrewardsandsanctionsarelistedasdiscretionaryperformance

indicatorsbecausetheyhavenotbeenvalidatedinDTCprogramevaluations,andbecause

noperformancebenchmarkshavebeen identified foruse inDTCsorother criminal jus-

ticeprograms.Inaddition,theyarelikelytobeconsiderablymoredifficultandexpensive

tomeasurethanotherperformanceindicatorsdiscussedthusfar.Evaluatorswillneedto

keeptracknotonlyofthenumbersofrewardsandsanctionsthatareimposed,butalsoof

thenumbersofachievementsandinfractionsthatoccurred.Becausemanydifferent

behaviorsmaybecountedasachievementsor infractionsinaDTC,thisprocessrequires

carefulmeasurementofnumerousvariables.Formanyevaluators,thislevelofeffortmay

bemorecomplicatedorcostlythanisfeasibleordesirable.

Effectiveness is also substantially influencedby the immediacywithwhich rewards

andsanctionsareimposed(Harrell&Roman,2001;Hawkin&Kleiman,2009;Marlowe&

Kirby,1999).Thesoonerrewardsareadministeredafteranachievement,andthesooner

sanctionsareimposedafteraninfraction,thebettertheeffectsarelikelytobe.Examining

immediacywouldrequireevaluatorstomeasurethetimedelaybetweeneachachievement

andthedeliveryofareward,andbetweeneachinfractionandthedeliveryofasanction.

Thisprocesswouldadddate-stampingtothealreadycomplicatedmeasurementthatisre-

quiredtoexaminecertainty.Becausethisprocessislikelytobeprohibitivelycostlyand

difficultformanyevaluators,immediacyisnotincludedinthelistofperformanceindicators.

Someevidencesuggestsoutcomesmaybebetterforprogramsthatprovidea4:1ratio

ofrewardstosanctionsforparticipants(Gendreau,1996;Wodahletal.,2011).ManyDTC

participantshavelonghistoriesofreceivingpunishmentforwrongdoings,buttheyareof-

tenunaccustomedtoreceivingpositivereinforcementforbehavingappropriately.Offering

agreaterproportionofrewardsforproductivebehaviorsthansanctionsforinfractionsis

hypothesizedtoincreasetheeffectivenessofcorrectionalrehabilitationprograms.

Support for the 4:1 ratiomust be viewed as preliminary, however, because it was

80

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

derivedfrompost hoc(afterthefact)correlationsratherthancontrolledresearchstudies.

Bydesign,sanctionsareimposedforpoorperformanceandrewardsareprovidedforgood

performance. Therefore, a greaterproportionof rewardsover sanctionsmightnothave

caused better outcomes, but rather better outcomesmight have elicited a greater

proportionof rewards. For this reason, thebalanceof reinforcement (i.e., the ratioof

rewardstosanctions)islistedasadiscretionaryperformanceindicatorforevaluatorsto

considerexaminingintheirprograms.

Balance of reinforcement =# of rewards administered

# of sanctions imposed

Restorative Justice Interventions

ManyDTCsincluderestorativejusticeinterventionsintheirprograms.Participantsmay

berequired,forexample,topayrestitutiontovictims,participateinvictimimpactpanels,or

performcommunityservicesuchasworkinginasoupkitchenorhomelessshelter.Panama,

forexample,requirescompensationorrestitutionforcrimevictims.

Themostcommonlyreportedperformance indicatorsforrestorativejustice interven-

tions include the number of hours of community service thatwere performed, and the

degreetowhichparticipantssatisfiedtheirfinancialobligations(fromnopaymenttopartial

paymenttofullpayment).Thefinancialobligationsaretypicallycategorizedaccordingto

paymentoffinesandfeestothecourt,paymentofrestitutiontovictims,andpaymentof

otherpersonalfinancialobligationssuchaschildsupportoralimony.

Although restorative justice interventions have been shown to improve outcomes in

othercriminaljusticecontexts(Latimeretal.,2005;Sullivan&Tifft,2008),nostudyhas

examinedtheirimpactinDTCsandnoperformancebenchmarkshavebeenidentifiedfor

theirusewithindividualschargedwithdrug-relatedcrimes.EvaluatorsfromOASmember

81

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

statesareinvitedtoexaminetheroleofrestorativejusticeinterventionsintheirDTCpro-

gramsandsharetheinformationtheylearnwiththeglobalresearchcommunity.

Employment

Outcomes in DTC programs are significantly better for participants who were em-

ployedatentryorwhoobtainedemploymentduringtheirenrollmentintheprogram

(Deschenesetal.,2009;Leukefeldetal.,2007).BetteroutcomesarealsoproducedbyDTCs

thatofferemploymentcounselingandrequireparticipantstobeemployedorenrolledin

aneducationalprogramasaconditionofgraduationfromtheprogram(Careyetal.,2012).

AcommoncomplaintamongDTCparticipants,especiallyracialandethnicminoritypartici-

pants,isthattheprogramspaidinsufficientattentiontotheiremploymentneeds(Cresswell

etal.,2001;Gallagher,2012).

EmploymentoutcomesarecommonlyreportedinoneoftwowaysinDTCevaluations.

Thedischarge employment rate representstheproportionofparticipantswhowereem-

ployedatdischarge,dividedbythenumberwhoenteredtheprogramminustheneutral

discharges.Beingemployed(orenrolledinaneducationalprogram)atdischargepredicts

significantlybetterpost-programoutcomes.(Beingemployedatentryalsopredictsbetter

post-programoutcomes;however,thisvariableisaparticipantcharacteristicorpredictor

variableratherthanaperformanceindicatorbecauseit isnotinfluencedbyparticipants’

enrollmentintheDTC.)

Discharge employment rate =# employed at discharge


Althoughemploymentatdischargeissignificantlypredictiveofoutcomes,itdoesnot

indicatewhether theDTCwas responsible for improvingparticipants’employment.High

employmentratesatdischargecouldsimply indicatethattheDTCtargeted low-risk

82

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

individualswhowerealreadymorelikelytohaveajob.Thiswouldsayverylittleabouthow

theDTCperformed.Forthisreason,thedischargeemploymentrateislistedasadiscre-

tionaryperformanceindicator.

Theemployment improvement rateiscalculatedbysubtractingthenumberofpartici-

pantswhowerealreadyemployedatentryfromthosewhowereemployedatdischarge,

andthendividingbythenumberwhoenteredtheprogramminustheneutraldischarges.

Ifmoreparticipantsareemployedatdischargethanatentry,thevalueofthisratiowillbe

positiveandwillrepresentthepercentageofindividualswhobecameemployedwhilethey

wereenrolled in theDTCprogram.Thisperformance indicator is listedasrecommended

because itsignificantlypredictsoutcomes,but itoftenrequirestheuseofself-report

assessmentsofparticipants’employmentstatus.

Employment improvement rate =# employed at discharge - # employed at entry


Nonetheless,neitheroftheaboveperformanceindicatorsmeasuresthequality of

participants’employment. Forexample,theydonot indicatewhethersomeparticipants

mayhavetransitionedfrompart-timetofull-timework,fromseasonaltostableemployment,

and/orreceivedaraiseorpromotionattheirjob.

Evaluatorstypicallyassessthenatureorqualityofparticipants’employmentusing

self-reportassessmenttools.Examplesofcommonlyusedtoolsforthispurposeinclude,

butarenotlimitedto,theAddictionSeverityIndex(ASI)andtheGlobalAppraisalofIndi-

vidualNeeds(GAIN).TheAppendixprovideslinkstowebsiteswherethereadercanobtain

furtherinformationabouttheseandotherassessmenttools.Itemsinthesetoolsinquire

aboutsuchissuesasthenumberofdaystheparticipantwasemployedduringthepreceding

month,thetypeofworkthatwasperformed(e.g.,clerical,manuallabor,orsupervisory),

andwhethertheemploymentwasfull-time,part-time,temporary,orseasonal.Although

83

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

thesetoolsweredevelopedintheUnitedStates,theyhavebeentranslatedintoSpanish

andotherlanguagesandhavebeenvalidatedinseveralcountries,includingSouthAmeri-

can countries.

Education

Many DTCs require or encourage participants to earn a high school diploma or

equivalencydegreeiftheydonothaveonealready.Becauseitusuallytakesseveralmonths

oryearstoaccomplishthisgoal,mostparticipantswillhavecompletedtheDTCprogram

beforetheycouldhaveearnedaneducationalorequivalencydegree.Forthisreason,itis

usuallynotinformativetoreportthepercentageofparticipantswhoearnedadegreebythe

timeofdischarge.

Theperformanceindicatorfortheeducational improvement rate ismeasuredeitheras

achangeinparticipants’educationalstatusorenrollmentinaneducationalprogram.The

differencebetweenthenumberofparticipantswhometeitherofthesecriteriaatdischarge

andthenumberwhometeithercriterionatentryisdividedbythenumberwhoenteredthe

programminustheneutraldischarges.Ifmoreparticipantshaveaneducationaldegreeor

areenrolledinaneducationalprogramatdischargethanatentry,thisratiowillbepositive

andwillrepresentthepercentageofindividualswhoimprovedorareintheprocessof

improvingtheireducation.

Educational improvement rate =(# with diploma or equivalent at discharge + # enrolled in educational program at discharge) - (# with diploma or equivalent at entry + # enrolled in educational program at entry)

# entered program – # of neutral discharges

Studiesareinconclusiveconcerningtheinfluenceofeducationalimprovementsonout-

comesinDTCprograms.Becausesupportfortheeffectsofeducationalefforts isunclear

atpresent,educationalimprovementislistedasadiscretionaryperformanceindicatorfor

evaluatorstoconsider.

84

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Housing

SignificantlybetteroutcomesareachievedbyDTCsthatrequireparticipantstobeliving

insafeanddrug-freehousingasaconditionofgraduationfromtheprogram(Careyetal.,

2012). Outcomesarealsosubstantiallybetter forDTCsthatprovidetransitionalhousing

whenitisneeded(McKee,2010).Withoutsafeanddrug-freehousing,itishighlyunlike-

lythatparticipantswillmaintainsobrietyordesistfromcriminalactivityafterleavingthe

program.Becausedrug-freehousingisstronglyandconsistentlyassociatedwithsuccessful

outcomes,improvementsinhousingarelistedasacoreperformanceindicator.

Theperformanceindicatorsforhousingarecalculatedinasimilarfashiontothosefor

employmentandeducation.Thedischarge housing raterepresentsthenumberofpar-

ticipantswhowere living insafeandsoberhousingatdischarge,dividedbythenumber

whoenteredtheprogramminustheneutraldischarges.

Discharge housing rate =# in stable housing at discharge


Thehousing improvement raterepresentsthenumberofparticipantswhowereliving

insafeanddrug-freehousingatdischargeminusthosewhowerealreadylivinginsafeand

soberhousingatentry,dividedbythenumberwhoenteredtheprogramminustheneutral

discharges.Thisperformanceindicatorreflectschangesinhousingthatoccurredwhilepar-

ticipantswereenrolledintheDTCprogram.Ifmoreparticipantsarelivinginstablehousing

atdischargethanatentry,thevalueofthisratiowillbepositiveandwillrepresenttheper-

centageofparticipantswhosehousingimprovedwhiletheywereenrolledintheDTC.

Housing improvement rate =# in stable housing at discharge - # in stable housing at entry


85

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Theseperformanceindicatorsdonotmeasurethequalityofthehousingarrangements.

Evaluatorsareencouraged tocategorize thequalityofhousing in termsof, forexample,

whether theparticipantsownor rent their residence,are livingwith familymembersor

friends,orarelivinginarecoveryhouseorresidentialtreatmentfacility.Instrumentssuch

astheASIandGAIN(listedintheAppendix)evaluatethequalityofparticipants’housingin

thecontextofsubstanceabusetreatment.

Emotional Health

Outcomesare significantlybetter forDTCs thatofferpsychiatricormentalhealth

treatmentforparticipantswhoneedtheseservices(Careyetal.,2012).Continuedemo-

tional problems at discharge predict significantly poorer post-programoutcomes in DTC

evaluations(Mendozaetal.,2013;Petersetal.,2012).

Inmoststudies,evaluatorshaveusedself-reportquestionnairesorstructuredinterviews

todeterminewhichparticipantsneededmentalhealthservicesandwhethertheirsymp-

tomsimprovedduringtheirenrollmentintheDTC.Manyoftheseinstrumentsassess

problems inmultiple areas of participants’ lives, includingmedical, psychiatric, family,

andsocialproblems.Therefore,thesametoolscanoftenbeusedtodevelopperformance

indicatorsforseveralofthedomainsdiscussedhereandbelow.TheAppendixlistsseveral

clinicalassessmentinstrumentsthathavebeenusedtoassessemotionalproblemsandother

psychosocialproblemsinDTCevaluationsandhavebeentranslatedintoSpanishorvali-

datedinmultiplecountries.

Discharge emotional problems are calculated by dividing the number of participants

whowereexperiencing significantemotionalproblemsatdischargeby thenumberwho

enteredtheprogramminustheneutraldischarges.Thisratiorepresentstheproportionof

participantswhodidnotresolvetheiremotionalproblemspriortodischarge,andtherefore

maybeatincreasedriskforrelapsetosubstanceuseorcriminalrecidivism.

86

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Discharge emotional problems =# with emotional problems at discharge


Theemotional improvement rateindicatesthedegreetowhichtheDTCwasresponsible

forreducingparticipants’emotionalproblems.Itiscalculatedbydividing(A)thenumberof

participants who were experiencing significant emotional problems at entryminus those

experiencingsignificantemotionalproblemsatdischarge,by (B) thenumberwhoentered

theprogramminustheneutraldischarges.Iffewerparticipantsareexperiencingemotional

problemsatdischargethanatentry,thevalueoftheratiowillbepositiveandwillrepresent

thepercentageofparticipantswhoseemotionalproblemsresolvedwhiletheywereintheDTC

program.Thisperformanceindicatorislistedasrecommendedbecauseitsignificantlypredicts

outcomesbutoftenrequirestheuseofrepeatedself-reportassessmentsofparticipants.

Emotional improvement rate =# with emotional problems at entry - # with emotional problems at discharge


AsubstantialproportionofDTCparticipantsmay leavetheprogramprematurelyand

without warning to staff members. This can complicate efforts to assess participants’

emotionalstatus(ortheirstatusonotherpsychosocialvariables)atthepointofdischarge.

CounselorsinDTCprogramsarethereforeencouragedtoperiodicallyreassessparticipants’

emotional health and status on other psychosocial variables. Many assessment tools,

includingtheASI,GAINandotherslistedintheAppendix,canbeusedtoreassessparticipants’

functioningat intervalsofevery three tosixmonths.Performingperiodic reassessments

helpstoensurethatinformationaboutparticipants’emotionalfunctioningwillbeavailable

atornearthetimeofdischargeandisalsoacriticalcomponentofgoodclinicalpractice.

Effectivecliniciansperiodicallyreevaluatetheirclientstodeterminewhethertreatmentis

workingorwhetheradjustmentsmaybeneededtothetreatmentplan.

87

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Medical and Dental Health

Preliminary evidence suggests outcomesmay be better for DTCs that offer medical

ordentaltreatmentforparticipantswhoneedtheseservices(Careyetal.,2012).Aswith

emotionalproblems,inmoststudiesevaluatorshavereliedonself-reportquestionnairesor

interviewstodetermineparticipants’needformedicalordentalservices.Nostudieshave

examinedthedegreetowhichcontinuedmedicalordentalproblemsatdischargepredict

long-term outcomes. Therefore, performance indicators related to medical and dental

servicesarelistedasdiscretionaryvariablesforevaluatorstoconsider.

Theperformance indicator fordischarge medical or dental problems is calculatedby

dividing thenumberofparticipantswhowereexperiencingsignificantmedicalordental

problems at discharge by the number who entered the program minus the neutral

discharges.Thisratiorepresentstheproportionofparticipantswhodidnotresolvetheir

medicalordentalproblemspriortodischarge,andthereforemaybeatriskforrelapseto

substanceabuseorcriminalrecidivisminthefuture.

Discharge medical or dental problems =# with medical/dental problems at discharge


Theperformanceindicatorformedical or dental improvementiscalculatedbydividing

thenumberofparticipantswhowereexperiencingmedical ordental problemsat entry

minus thoseexperiencingmedical ordental problemsatdischarge,by thenumberwho

enteredtheprogramminustheneutraldischarges.Iffewerparticipantsareexperiencing

medicalordentalproblemsatdischargethanatentry,thevalueofthisratiowillbepositive

andwillrepresentthepercentageofparticipantswhosemedicalordentalproblemswere

resolvedwhiletheywereenrolledintheDTCprogram.

Medical/dental improvement =# with medical/dental problems at entry - # with medical/dental problems at discharge


88

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Family Relationships

Outcomesaresignificantlybetter inDTCsthatofferfamilycounselingforparticipants

whoneedtheseservices (Careyetal.,2012). Anational studyof twenty-threeDTCs in

theUnitedStatesfoundthatreducingfamilyconflictsandimprovingfamilysupportwere

commonlyderivedbenefitsofDTCprograms (Rossmanet al., 2011). Evaluators typically

administerself-reportquestionnairesorinterviewssuchastheASIorGAINtodetermine

whetherparticipantsneedfamily-basedservicesandtogaugewhetherfamilyfunctioning

hasimproved.

Discharge family problemsarecalculatedbydividingthenumberofparticipantswho

wereexperiencingsignificantfamilyproblemsatdischargebythenumberwhoenteredthe

programminustheneutraldischarges.Thisratiorepresentstheproportionofparticipants

whodidnot resolve their familyproblemsprior todischarge,and thereforemaybeat

increasedriskforrelapsetosubstanceabuseorcriminalrecidivism.

Discharge family problems =# with family problems at discharge


Thefamily improvement rateiscalculatedbydividingthenumberofparticipantswho

wereexperiencingfamilyproblemsatentryminusthoseexperiencingfamilyproblemsat

discharge,bythenumberwhoenteredtheprogramminustheneutraldischarges.Iffewer

participantsareexperiencingfamilyproblemsatdischargethanatentry,thevalueofthe

ratiowill be positive andwill represent the percentage of participantswhose family

problems resolvedwhile theywere in theDTCprogram. This performance indicator is

listedasrecommendedbecauseitsignificantlypredictsoutcomesbutrequirestheuseof

self-report assessments.

Family improvement rate =# with family problems at entry - # with family problems at discharge


89

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Aswas notedpreviously, a substantial proportionofDTCparticipantsmay leave the

programprematurelyandwithoutwarning.Therefore,counselorsareencouragedtope-

riodicallyreassessparticipants’familyfunctioningatintervalsofapproximatelyeverythree

to sixmonths. Performing periodic reassessmentswill ensure that information about

participants’familyfunctioningisavailableatorneardischargeandisalsoafeatureofgood

clinicalpractice.

Social Relationships

Socialproblemsrefertoseriousorrepeatedinterpersonalconflictswithpersonsother

than familymembers,aswell as social isolationoralienation fromotherpeople. Many

DTCprogramsprovideinterventionsdesignedtoreduceinterpersonalconflictsandincrease

participants’involvementinhealthysocialinteractions.However,nostudieshaveexamined

theimpactofcontinuedsocialorinterpersonalproblemsonlong-termoutcomesfromDTC

programs.Therefore,performanceindicatorsrelatedtosocialproblemsareincludedasdis-

cretionaryvariablesforevaluatorstoconsider.

Theperformance indicator fordischarge social problems iscalculatedbydividingthe

numberofparticipantswhowereexperiencingsignificantsocialproblemsatdischarge,bythe

numberwhoenteredtheprogramminustheneutraldischarges.Thisratiorepresentsthepro-

portionofparticipantswhodidnotresolvetheirsocialproblemspriortodischarge,andthere-

foremaybeatincreasedriskforrelapsetosubstanceabuseorcriminalrecidivisminthefuture.

Discharge social problems =# with social problems at discharge


Theperformanceindicatorforthesocial improvement rateisthenumberofparticipants

whowereexperiencingsocialproblemsatentryminusthoseexperiencingsocialproblems

atdischarge,dividedbythenumberwhoenteredtheprogramminustheneutraldischarges.

90

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Iffewerparticipantsareexperiencingsocialproblemsatdischargethanatentry,theratio

willbepositiveandwill represent thepercentageofparticipantswhosesocialproblems

resolvedwhiletheywereintheprogram.

Social improvement rate =# with social problems at entry - # with social problems at discharge


Birth of Drug-Free Babies

Consumption of alcohol or other drugs during pregnancy, especially cocaine

or stimulants, is associated with serious birth complications and physiological and

developmentaldeficitsforthenewborn(Cooper,2004).Theaddedcoststosocietyofcaring

foradrug-exposedbabycanbeveryhigh.

BecausefewDTCparticipantsortheirmatesdeliverbabiesduringtheirenrollmentin

theprogram, it is difficult tomeasure statistically significant effectson this variable.

Nevertheless, the sentiments of policymakers, members of the public, and other

stakeholdersareunderstandablyswayedbythisoutcomemeasure. Savingoneinnocent

newbornfromalifetimeofmiserymightbeworththetotalcostsofaDTCtomanypeople.

Because births are infrequent events in DTC programs, it is usually not informative to

calculatedischargeratesorimprovementrates.Mostevaluationssimplyreportthenumber

ofdrug-freenewbornsthatweredeliveredtoorfatheredbyDTCparticipants.ForDTCsthat

havebeeninexistenceformanyyears,theresultsmaybedozensofinnocentlivessaved.

Treatment intake efficiency = Date of first treatment session-Date of entry into DTC

91

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Theperformanceindicatorsdiscussedthusfarhavefocusedonobjectiveeventsorbe-

haviors,suchasdrugtestresultsorschedulesofcourthearings.Ithasalsobeenhypothe-

sizedthatDTCsmayimproveoutcomesbyalteringthesubjectiveattitudesofparticipants

orstaffmembers.Forexample,DTCsmightenhanceparticipants’intrinsicmotivationfor

changeortheirrespectforthejusticesystem.Inturn,thesepositiveattitudinalchanges

mightinfluencethemtoreducetheirdruguseorcriminalactivity.Similarly,staffmembers

inDTCsmightdevelopmoreoptimistic attitudes towards rehabilitation, and this in turn

mightinfluencethemtointeractmoreproductivelywithparticipants.

Aswillbediscussedlater,testingsuchhypothesesrequiresevaluatorstoperformwhat

are calledmediation analyses. Changes in participants’ or staff members’ attitudes are

sometimes referred toasmediator variablesbecause theyoccur in themiddlebetween

thedeliveryofDTCservicesandsubsequentchangesinparticipants’behaviorsandare

hypothesizedtobeanecessaryconditionforthosebehavioralchangestooccur.Unlike

moderator analyses, which as discussed earlier indicatewho DTCs benefit, mediator

analysesareintendedtoindicatehowtheyfunction.

Participant Attitudes

Researchinthisareaisrelativelynew;however,afewstudieshavefoundthatpar-

ticipants’scoresonattitudinalmeasuresdidappeartoinfluencelonger-termimprovements

insobrietyordesistence fromcrime.Onestudy, forexample, foundthatalthoughmany

participants initially entered aDTC to avoid going to prison, thosewho gradually devel-

opedintrinsicmotivationtoimprovetheirliveshadsignificantlybetterlong-termoutcomes

(Kirk,2012).Inotherstudies,significantlybetteroutcomeswereachievedbyparticipants

whoperceivedtheDTCasapplyingfairproceduresandhavingbenevolentaims,character-

ATTITUDINAL VARIABLES

92

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

isticsreferredtoasprocedural justice or procedural fairness (Dane,2012;Frazer,2006;

Gottfredsonet al., 2007; Zweiget al., 2012). Betteroutcomeshavealsobeenachieved

byparticipantswhoperceivedarationalconnectionbetweentheirownbehaviorsandthe

impositionofsanctionsandincentivesintheDTC,acharacteristicreferredtoasperceived

deterrence(Marloweetal.,2005).

Thefollowingattitudinalvariablesonthepartofparticipantshavebeenhypothesizedto

mediateoutcomesinDTCsorhavebeenexaminedinDTCprogramevaluations.Itisprema-

turetoconcludewhetherthesevariablesreliablyimpactlong-termoutcomesinDTCs,but

evaluatorsinOASmemberstatesmaywishtoexaminethesevariablesintheirownresearch.

TheAppendixprovidesinformationonhowtoobtainassessmenttoolsthatmeasurethese

variablesandthathavebeentranslatedintoSpanishorvalidatedinmultiplecountries.

Procedural Justice or Procedural Fairness: Asubstantialbodyofresearchhasfound

thatcriminaldefendantsandotherlitigantsweremorelikelytoreactfavorablytoanad-

versejudgmentorpunitivesanctioniftheybelievedfairprocedureswerefollowedinreach-

ingthedecision.Thebestoutcomeswereachievedwhenthedefendantswere(1)given

areasonableopportunitytoexplaintheirsideofthedispute;(2)treatedinanequivalent

mannertosimilarpeopleinsimilarcircumstances;and(3)treatedwithrespectanddignity

throughouttheprocess(Burke&Leben,2007;Tyler,2007).SomecommentatorsinAustra-

lia(King,2009),Scotland(McIvor,2009)andtheUnitedStates(Burke,2012;Wieneretal.,

2010)havehypothesizedthatenhancingparticipants’perceptionsofproceduraljusticemay

betheprimarymechanismbywhichDTCsreducecrimeandsubstanceabuse.

Motivation for Change:DTCsweredesignedtotreatpersonssufferingfromaddiction

whohavelowmotivationtochangetheirsubstanceuseorcriminalactivity.Thepowerof

thecourtisusedtomaintainparticipants’adherencetotreatmentdespiteapotentialdesire

toleavetreatmentprematurely.Eventually,however,participantsmustgraduatefromthe

DTC. If theyhavenotdeveloped intrinsicmotivationbythattime,theoddsof long-term

successarelikelytobeslimtonone.Therefore,evaluatorsshouldperiodicallyassesspar-

93

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

ticipants’motivationforchangeatapproximatelythree-tosix-monthintervalstodetermine

whetheritisimprovingovertime.Aswasnotedabove,atleastonestudyreportedbetter

outcomesforDTCparticipantswhoseinternalmotivationimprovedovertime(Kirk,2012).

Perceived deterrence:Thecriminaljusticesystemusuallydoesnotapplybehavioral

contingenciessuchassanctionsandincentiveseffectively.Punishmentmaybeadministered

afterlongdelaysormultipleinfractions,andmaybetoosevereortoolenientinmagnitude

toachieveeffectiveresults.OnegoalofaDTCistoadministerrewardsandsanctionsina

systematic, rapid,andeven-temperedmannerthat isconsistentwithscientificprinciples

ofeffectivebehaviormodification.StudiessuggestparticipantsinDTCsmayrecognizethe

rationalitybywhichrewardsandsanctionsareadministeredandmayrespondfavorablyas

aresult(Goldkampetal.,2002;Harrell&Roman,2001;Marloweetal.,2005).

Satisfaction with the Program or Treatment:Participants’satisfactionwithsubstance

usetreatment, includingtheculturalsensitivityoftreatmentstaff, isareliablepredictorof

outcomes incommunity-basedtreatmentprograms inseveralcountries (Richardsonetal.,

2011). Studieshavealmostuniformlyreportedhighlevelsofsatisfactionwiththeservices

provided inDTCs. For statistical reasons,variables tend tobe lesspredictiveofoutcomes

whenmostparticipantsscorethesameornearlythesameontheassessmenttools.Perhaps

forthisreason,satisfactionwiththeprogramhasfrequentlynotbeenfoundtobepredictive

ofoutcomes inDTCevaluations. Ifparticipants inotherOASmemberstateshaveawider

rangeofreactionstoDTCs,thisvariablemighthavebetterutilityforpredictingoutcomes.

Therapeutic Alliance:Thequalityofthetherapist-patientalliancehasbeenfoundto

beareliablepredictorofpsychotherapyoutcomesinseveralcountries includingCanada,

Italy,AustraliaandtheUnitedStates(Ardito&Rabellino,2011;Summers&Barber,2003).

NostudyhasexaminedtheimpactofthetherapeuticallianceonoutcomesinDTCs,butthis

variable(particularlythealliancebetweentheparticipantandjudge)hasbeenhypothesized

toinfluenceoutcomesinDTCprogramsaswell.Researchisneededtotestthishypothesis

inarangeofDTCprogramsandindifferentcountries.

94

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Staff Attitudes

Considerably less researchhasexamined staffmembers’ attitudesas compared to

participants’attitudesinDTCprograms.Evidencesuggeststhemoretrainingstaffmembers

receiveandthelongertheyworkinDTCprograms,themorelikelytheyaretoendorse

optimistic attitudes aboutoffender rehabilitationand to favor rehabilitative goals over

punitivegoals(VanWormer,2010).Changesinstaffmemberattitudeshavenot,however,

beencorrelatedwithoutcomesinDTCprograms.Additionalresearchisneededtoun-

derstandthedegreetowhichstaffmemberattitudesmayimpactoutcomesinDTC’sand

howtoinfluencethoseattitudestobringaboutmoreeffectivebehavioralchange.

95

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Observer rating scalesmayalso contributevaluable information toDTCprogram

evaluations.ObserverratingscalesmaybeusedtoassessobjectiveeventsoccurringinDTCs,

suchasthenumberoftimesthejudgespokedirectlytoparticipantsduringcourtsessions.

Morecommonly,however,theyareusedtoassesssubjectivetraitsorcharacteristicsofstaff

membersorparticipants,suchasthedegreetowhichthejudgeappearedtobe“competent”

or“knowledgeable,”orparticipantsappearedtobe“attentive”or“compliant.”

Inafewstudies,independentratersobservedcourthearingsinDTCsandratedthebe-

haviorsofthejudgeorparticipants.Inonestudyoftwenty-threeDTCs,significantlygreater

reductionsincrimeandsubstanceusewereproducedbyjudgeswhowereratedasbeing

morerespectful,fair,attentive,enthusiastic,consistent,andcaringintheirinteractionswith

participants(Zweigetal.,2012).AnotherstudyfoundthatparticipantsinaDTCweresignifi-

cantlylesslikelytobearrestedforanewcrimeiftheywereratedasbeingmoreprepared

andbehavingappropriatelyduringcourthearings(Reingleetal.,2012).Stillotherstudies

havefoundthatparticipantsandstaffmemberstendedtoratecourtinteractionssimilarly

(Farole&Cissner,2007;Satel,1998).Forexample,theycommonlyagreedonwhetherin-

teractionsappearedtobeproductiveorunproductive.TheAppendixprovidesinformation

aboutobserverratingscalesthathavebeenusedsuccessfullyinDTCprogramevaluations.

Observer rating scales typically require substantially more effort on the part of

evaluatorsthanself-reportquestionnaires.Ratersmustbetrainedcarefullyonhowtoscore

the instrumentsandmustdemonstratehigh levelsof inter-rater reliabilityoragreement

with other raters. If two raters reach different conclusions about the same behavior or

characteristicsofindividualstheyareobserving,thentheinformationwillnotbeusefultoan

evaluator.Itcanalsobeverycostlytopayindependentobserverstoattendcourthearings

orothersessionsandscoreratinginstruments.Asaresult,observerratingscalesaremost

likely to be used inwell-funded studies that are conductedby professional scientists or

university-basedresearchers.

OBSERVER RATING SCALES

96

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Performanceindicatorscannotbeinterpretedinavacuum.Toanswermostresearch

questions, the performance of a DTCmust be compared to that of an equivalent and

unbiasedcomparisongroup.ComparingwhathappenedintheDTCtowhatwouldmost

likelyhavehappenediftheDTCdidnotexistisreferredtoasexaminingthecounterfactual

hypothesis orthepossibilitythattheDTCwasineffective.

Somecomparisonconditionsarereasonablyunbiasedandcanyieldafairandaccurate

assessmentofwhatwouldmostlikelyhaveoccurredwithouttheDTC.Others,however,may

besystematicallybiasedorskewedinsuchamannerastomaketheDTClookbetterorworse

thanitdeserves.Thismayleadtoanunwarrantedrejectionofthecounterfactualhypothesis

andtheerroneousconclusionthattheDTCwaseffectivewhen,infact,itwasnot.

Random Assignment

Fromascientificstandpoint,thebeststrategyistorandomlyassigneligibleindividuals

either to the DTC program or to an alternative disposition, such as probation. Ran-

domassignmentprovidesthegreatestassurancethatthegroupsstartedoutwithanequal

chanceofsuccess.Therefore,ifoutcomesarebetterfortheDTC,thesecanbeconfidently

attributedtotheeffectsoftheDTCprogramasopposedtodifferences intheseverityof

participants’problemsbeforetheyenteredtheprograms.

Evenwhenanevaluatoremploysrandomassignment,thereisstillapossibility(although

agreatlydiminishedpossibility)thatthegroupsdifferedonimportantdimensionsfromthe

outset.Thisrequirestheevaluatortoperformaconfirmationoftherandomizationproce-

dure.Aswillbediscussedlater,theevaluatorwillneedtocheckforpossiblepre-existing

differencesbetweenthegroupsthatcouldhaveaffectedtheresults.Ifsuchdifferencesare

foundtoexist, therearestatisticalprocedurestheevaluatormightbeable toemployto

COMPARISON GROUPS

97

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

adjust for thosedifferencesandobtainscientificallydefensibleresults.Forexample,DTC

participantsmighthavehad less severedrugproblems tobeginwith thancomparison

participants.Ifso,thenbetteroutcomesfortheDTCmightsimplyhavebeentheresultof

treatinghigherfunctioningindividuals. Ifrandomassignmentfailstoavoidthisproblem,

theevaluatorwillneedtoadjuststatisticallyfordifferencesindrugproblems.

Inmany instances, randomassignment isnot feasible for legal,ethical,orpractical

reasons.Somestaffmembersmayhaveethicalconcernsaboutdenyingservicestoother-

wiseeligible individuals. Inaddition,someDTCsmayhavedifficultyfillingtheirslotsand

maynotwanttoturnawayeligibleindividuals.

Nonetheless,randomassignmentisgenerallynotconsideredtobeunethicalifappro-

priatesafeguardsareinstituted(NationalResearchCouncil,2001).Randomassignmenthas

beenusedsuccessfullyinseveralresearchstudiesofDTCsinAustralia(Jonesetal.,2011)

and theUnitedStates (Breckenridgeetal., 2000;Gottfredsonetal., 2003;Harrell etal.,

1999;MacDonaldetal.,2007;Turneretal.,1999).Safeguardsmayincluderequiringpar-

ticipantstoprovideinformedconsenttoberandomlyassigned,orhavinganindependent

ethicsreviewboardoverseethesafetyandfairnessofthestudy.Insomecountries,local

collegesoruniversitieshaveethicsreviewboardswithnamessuchasInstitutionalReview

Board (IRB)orDataandSafetyMonitoringBoard (DSMB),whichhave theauthorityand

expertisetoprovidesuchethicaloversightforresearchstudies.

Ethical scholars generally agree that random assignment is acceptable, and indeed

shouldberequired,ifatreatmentprogramhasnotyetbeenproveneffective(Edwardset

al.,1998;NationalResearchCouncil,2001).Inmanycountries,forexample,newmedica-

tionsmustbeevaluatedthroughrandomassignmentbeforebeingapprovedforpublicuse.

Itmaybeunethicalorunlawfulnot touserandomassignment infinal-phasemedication

trials.Randomassignmentbecomesobjectionableonlyafteratreatmenthasbeenproven

tobeeffective,andthenitmaynotbeethicaltodenythateffectivetreatmenttodeserving

individuals.Insomecountries,DTCshavebeenproventobeeffective,andthereforeitmay

98

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

raiseethicalconcernstouserandomassignmenttodecidewhogetsintothoseprograms.

IncountriesthathavenotyetestablishedtheeffectivenessoftheirDTCsfortheircitizens,it

isthereforenotnecessarilyunethicaltouserandomassignment

Additionally,randomassignmentposesfarfewerethicalchallengesifaDTChasin-

sufficientcapacitytotreatmanyindividualswhowouldotherwisebeeligibleforitsservices.

Ifmanydeservingpeoplemustbeturnedawayanyway,thenitwouldperhapsbefairestto

selectparticipantsrandomlyratherthanallowstaffmemberstopickandchoosewhogets

intotheprogram.Undersuchcircumstances,randomassignmentmightprovidethebest

protectionagainstunfairdiscriminationorunconsciousbias.

RandomassignmentmayalsobelessobjectionableifitisconductedwithinaDTCafter

participants have alreadybeen accepted. For example, studies have randomly assigned

participantswithinDTCstoreceivedifferentschedulesofcourthearings(Festingeretal.,

2002;Marloweetal.,2006,2007),differentrewardsforaccomplishments(Marloweetal.,

2008a),orspecializedtreatmentservices(Messinaetal.,2012). Becauseallparticipants

werealreadyadmittedtotheDTCs,eachparticipantreceivedatleastsomeoftheadditional

servicesandopportunitiestheprogramshadtooffer.Anydifferencesintheservicesthey

receivedwereamatterofdegreeratherthanbeingallornothing.Thisarrangementmaybe

alsomoreethicallyacceptabletostaffmembersandotherstakeholdersofDTCprogramsif

participantsprovidevoluntaryandinformedconsenttobeingrandomlyassigned.

Despite these safeguards, itmightneverthelessbe impractical toengage in random

assignment.Ifrandomassignmentisnotpossible,thenseveralalternativestrategiesare

availabletoselectacceptablecomparisonconditions.

Quasi-Experimental Comparison GroupsThenextbestapproachafterrandomassignmentistouseaquasi-experimentalcom-

parisongroup.ThistermreferstoindividualswhowouldhavebeeneligiblefortheDTCbut

didnotentertheprogramfor reasons that are unlikely to have affected their outcomes. An

99

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

excellentexampleisindividualswhowereeligibleforandwillingtoentertheDTCbutwere

deniedaccesssimplybecausetherewerenoemptyslotsavailable.Thisisreferredtoasa

wait-list comparison group.Themerehappenstancethattheprogramwasfullisunlikelyto

haveledtothesystematicexclusionofindividualswhohadmoresevereproblemsorpoorer

prognosesthantheaverageparticipant,andthereforeisunlikelytohavebiasedtheresults.

Lessoptimal,butstillpotentiallyacceptable,quasi-experimentalcomparisongroups

includeindividualswhowouldhavebeeneligiblefortheDTCbutwerearrestedintheyear

orsobeforetheDTCwasestablished,orwhowerearrested inan immediatelyadjacent

county thatdoesnothaveaDTC. Because these individualswerearrestedatanearlier

pointintimeorinadifferentgeographicregionthantheDTCparticipants,theremightstill

besystematicdifferencesbetweenthegroupsthatcouldhavebiasedtheoutcomes.For

example,socioeconomicconditionsorthetypesofdrugsthatarecommonlyusedmight

differbetweenneighboring communities. However, the likelihood of this occurring is

usuallynotsubstantial,andthesemaybetheonlypracticalcomparisonconditionsthat

canbeusedforpurposesofmanyDTCprogramevaluations.

Whenusingaquasi-experimentalcomparisongroup,itisessentialfortheevaluatorto

checkforpre-existingdifferencesbetweenthegroupsthatcouldhaveaffectedtheresults.

Forexample,thecomparisonindividualsmighthavehadmoreseriouscriminalhistoriesto

beginwith.This,inturn,mighthaveputthematgreaterriskforcriminalrecidivism.Ifso,

thensuperioroutcomesfortheDTCgroupmightnotbeattributabletotheeffectsofthe

DTCprogram,butrathertothefactthatittreatedalessseverepopulation.Aswillbe

discussedlater,therearevariousstatisticalproceduresanevaluatorcanemploytoadjust

forsuchdifferencesandobtainscientificallydefensibleresults.

Matched Comparison Groups

Evaluatorsdonotalwayshaveaquasi-experimentalcomparisongroupattheirdisposal.

Undersuchcircumstances,theymayberequiredtoconstructacomparisongroupoutofa

100

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

largepopulationofindividualsinvolvedinthejusticesystem.Forexample,anevaluatormight

needtoselectcomparisonsubjectsfromastatewideprobationdatabase.Manyofthose

probationerswouldnothavebeeneligibleforaDTCoraredissimilartoDTCparticipants

on characteristics that are likely to have influenced their outcomes. For example, some

probationersmightnothavehaddrugproblems,ormighthavebeenchargedwithoffenses

thatwouldexcludethemfromparticipationinaDTC.Theevaluatormust,therefore,select

asubsetofindividualsfromtheprobationpoolthatarecomparabletotheDTCparticipants

oncharacteristicsthatareknowntoaffectoutcomes.Forexample,theevaluatormightpair

eachDTCparticipantwithaprobationerwhohasthesamecriminalhistory,demographic

characteristicsandsubstanceusediagnosis.Becausetheevaluatorwillchooseonlythose

probationers who are comparable to DTC participants on multiple characteristics, it is

necessary to startwith a large sample of potential candidates fromwhich to select the

comparison sample.

Thesuccessofanymatchingstrategywilldependonwhethertheevaluatorhasade-

quateinformationaboutthecomparisoncandidatestomakevalidmatches.Ifdataarenot

availableonsuchimportantvariablesastheprobationers’criminalhistoriesorsubstance

useproblems,itwillnotbepossibletoplaceconfidenceinthevalidityofthematches.Itis

notsufficienttosimplymatchthegroupsonvariablesthatareeasytomeasureandreadily

available,suchasgender,ethnicityorrace,becausethegroupsmighthavedifferedonother

importantdimensionsthatwerenotexamined.

Propensity Score Analyses

An evaluatormay also use a statistical procedure called a propensity score analysis

tomathematically adjust for differences between the groups. This advanced procedure

calculatestheprobabilitythatanindividualwithagivensetofcharacteristicswouldbein

theDTCgroupasopposedtothecomparisongroup—inotherwords,therelativesimilarity

101

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

oftheindividualtoonegroupasopposedtotheother.Theanalysisthenstatisticallyadjusts

for this relative similarity when comparing outcomes between the groups. Advanced

statisticalexpertiseisordinarilyrequiredtoimplementandinterpretthisprocedure.

Aswithanystatisticaladjustment,thesuccessofapropensityscoreanalysiswilldepend

onwhethertheevaluatorhasadequateinformationaboutthecomparisonsubjectstomake

validadjustments.Ifdataarenotavailableonsuchimportantvariablesasthecomparison

subjects’criminalhistoriesorsubstanceuseproblems,itwillnotbepossibletoplace

confidenceintheadjustments.Again,itisnotsufficienttomerelyadjustthescoresbased

oneasilymeasuredvariablessuchasgenderor race,becausethegroupsmighthave

differedonotherimportantcharacteristicsthatwereneverexamined.

Invalid Comparison Groups

ThereareseveralcomparisongroupsthathavebeenusedinDTCprogramevaluations

whicharelikelyyieldedbiasedresults.Inmostinstances,itisnotjustifiedtocompareout-

comesfromaDTCtothoseofpersonswhorefusedtoentertheDTC,weredeniedaccessto

theDTCduetotheirclinicalorcriminalhistories,droppedoutoftheDTC,orweredischarged

unsuccessfullyfromtheDTC.Theprobabilityisunacceptablyhighthatsuchindividualshad

relativelypoorerprognosesormoresevereproblemstobeginwith.Forexample,thereis

ahighlikelihoodthattheymayhavehadmoreseriouscriminalorsubstanceusehistories,

lowermotivationforchange,orlessersocialsupports.Asaresult,comparingthemtothe

DTCsampleisapttounfairlyfavortheDTCprogram.Giventhehighlikelihoodthatthese

groupswereseriouslydisadvantagedfromtheoutset,statisticaladjustmentscannotbe

reliedupontoovercomethebiaseddifferences.

102

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

AcommonerrorinDTCevaluationsistoexamineoutcomesonlyforparticipantswho

successfullygraduatedfromtheprogram.Thelogicforperformingsuchananalysisisunder-

standable.Evaluatorsareoftenmostinterestedinlearningaboutoutcomesofparticipants

whoreceivedallservicestheprogramhastooffer.Ifindividualswhodroppedoutorwere

terminatedprematurelyareincludedintheanalyses,thentheoutcomeswillbeinfluenced

bythosewhodidnotactuallyreceiveallintendedservices.

Althoughthisreasoningmightseemlogical,itisscientificallyflawed.Outcomesshould

beexaminedforallindividualswhoparticipatedintheDTC,regardlessofwhetherthey

successfullygraduatedorwereunsuccessfullyterminatedfromtheprogram.Thisisreferred

to as an intent-to-treat analysis becauseitexaminesoutcomesonallindividualswhomthe

programinitiallysetouttotreat.Itisnotappropriatetoreportoutcomesonlyforgraduates

becausethisunfairlyandfalselyinflatestheapparentsuccessoftheprogram.Individuals

whograduatedfromtheDTCarelikely,forexample,tohavehadlessseveredrugoralcohol

problems tobeginwith, less severe criminal propensities, highermotivation for change,

orbettersocialsupportsystemsthantheaverageDTCparticipant.Asaresult,theymight

havebeenlesslikelytocommitfutureoffensesorrelapsetosubstanceuse,regardlessof

theservicestheyreceivedintheDTC.Thecriticalquestionishowtheprogramfaredforall

participantswhomettheprogram’seligibilitycriteriaandwereacceptedintotheprogram.

Thisissueisparticularlyimportantifoutcomesarecontrastedagainstthoseofacompari-

songroup,suchasprobationers.SelectingthemostsuccessfulDTCcasesandcomparingtheir

outcomestoallprobationersisunfair.ItisakintoselectingonlytheA+studentsfromoneclass-

room,comparingtheirtestscorestoallstudentsinasecondclassroom,andthenconcluding

thatthefirstclasshadabetterteacher.Thiswouldclearlybeabiasedandunfaircomparison.

This isnot tosuggest thatoutcomes forgraduatesareofno interest.Programsmay,

indeed,wanttoknowwhathappenstoindividualswhoreceivedallservicesofferedbythe

INTENT-TO-TREAT ANALYSES

103

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

DTC.However,thisshouldbeasecondary analysisthatisconductedaftertheintent-to-treat

analysishasbeencompleted.If it isfirstdeterminedthattheprogramachievedpositive

outcomesonan intent-to-treatbasis, itmaythenbeappropriatetoproceedfurtherand

determinewhetheroutcomeswereevenbetterforthegraduates.However,iftheintent-

to-treat analysis is not significant, it is generally not acceptable tomoveon to evaluate

outcomesforgraduatesalone.

Importantly,ifsecondaryanalysesareperformedonDTCgraduates,thenthecomparison

samplemustalsobecomprisedofsuccessfulcompleters.Forexample,outcomesforDTC

graduatesshouldbecomparedagainstthoseofprobationerswhosatisfiedtheconditions

ofprobation.ComparingoutcomesforDTCgraduatestoallprobationers, includingthe

probationfailures,wouldunfairlyfavortheDTCprogram.

Cohorts

Itisusuallynotjustifiabletoevaluateaprogramatasinglepointintimeandassume

theresultsarerepresentativeofwhatcanbeexpectedfromtheprograminthefuture.Con-

ditionsoftenchangeovertimeandmaysignificantlyinfluencetheeffectivenessofaDTC

program.Forexample,aDTCmightexperiencesubstantialstaffturnoveroradecreasein

fundingwhichcouldseriouslydetractfromthequalityoftheservicesitcanoffer.Converse-

ly, staffmembersmay improve their skills overtime throughexperience and continuing

education,whichcouldsignificantlyimprovetheprogram’sperformance.

Itisimportantforevaluationstobeconductedperiodicallytomeasurechangesinper-

formanceovertime.Commonly,thisisaccomplishedbyevaluatingcohortsofDTCpartici-

pants.Aswasmentionedearlier,acohortisdefinedasagroupofindividualswhoentered

theprogramduringthesamespecifiedtimeperiod,typicallyanintervalof12months(Heck,

2006;Rubioetal.,2008).Forexample,participantswhoenteredtheDTCduringthecalen-

daryears2011,2012,and2013mightbedefinedasthreecohorts,andoutcomeanalyses

couldbeconductedseparatelyforeachcohort.Thiswouldprovideaseriesof“snapshots”

indicatinghowwelltheprogramperformedoverconsecutiveyears.

104

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

There isnoonecorrect follow-upwindowortimeperiod forevaluatingtheperfor-

manceofaDTC.Themostappropriate lengthof thefollow-upwindowwilldependon

whatperformanceindicatorisbeingassessed.Forexample,itmaybeinformativetomea-

sureparticipants’attendanceintreatmentduringthefirstseveralmonthsofaDTCpro-

gram.ItisimportanttoknowwhethertheDTCissuccessfulatretainingparticipantsin

treatmentandprovidinganadequatedosageoftreatment.However,mostDTCprograms

graduallydecreasethedosageoftreatmentasparticipantsmovethroughthephasesof

theprogram.Asaresult,treatmentattendancemaybecomearelativelylessimportant

performance indicatorafterseveralmonthsorayear. Itmaybecomemore important

afterseveralmonthstoknowwhetherparticipants’substanceusedeclinedortheirem-

ploymentrateincreased.

Itisusuallynotveryinformativetomeasurenewarrestratesornewconvictionrates

duringthefirstfewmonthsofaDTCprogram.Ittypicallytakesseveralmonthsforapar-

ticipanttore-engageincriminalbehavior,bedetectedbytheauthorities,andbeformally

chargedorconvicted ina legalproceeding. Asa result,newarrestandconviction rates

are likelytobe lowduringthefirstseveralmonthsof treatment. (Thisdoesnot include

technicalviolations,whichmaybemorecommon.) This issuecanbeproblematicifnew

arrestsorconvictionsarecontrastedagainstthoseofacomparisongroup,suchasproba-

tioners.Ifrecidivismislowinbothgroups,itwillbedifficulttodetectstatisticallysignificant

differencesbetweenthegroups.ThisistrueeveniftheDTCis,infact,asuperiorprogram.

Forexample,iftenpercentoftheprobationersreoffendedduringthefirstsixmonthsand

fivepercentoftheDTCparticipantsreoffended,formathematicalreasonsthisdifference

wouldprobablynotbestatisticallysignificantunlesstherewerenumerousparticipantsin

thestudy.However,itmightbequitemeaningfulfromaclinicalorpublicpolicyperspective.

FOLLOW-UP WINDOWS

105

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Reducingrecidivismbyfivepercentagepointscanprovideimportantpublicsafetyandcost

benefitsforlocalcommunities.

Inmanystudies,thebestcourseofactionmaybetowaitatleasttwelvemonths,and

preferably twenty-four months, from entry before reporting recidivism outcomes. This

shouldallowadequatetimetoelapseforreoffendingratestodivergesufficientlybetween

thegroupstodetectstatisticalsignificance.

Aswasmentionedearlier,ifitisfeasible,recidivismshouldbetrackedforatleastthree

yearspost-entryandideallyuptofiveyearspost-entry.Researchsuggestsmostnewoffens-

esfordrugandalcohol-involvedpersonsoccurwithinthreetofiveyearsaftertreatment

(Gossopetal.,2005;Inciardietal.,2004;Martinetal.,1999).Therefore,followingpartici-

pantsforthreetofiveyearsshouldensurethatmostrecidivismeventsareaccountedforin

theevaluationresults.Recidivismanalysesmaystillbeinformativeafterthefirstorsecond

year,andthoseinterimanalysesshouldcertainlybereportedasthedatabecomeavailable.

However,followingparticipantsforthreetofiveyearsislikelytoelicitstableestimatesof

criminalrecidivism.

Starting the Clock

Inmanyevaluations, the follow-upwindow isstarted fromthetimeparticipantsfirst

enteredtheDTCprogramorcomparisoncondition.Forexample,outcomesforeachpar-

ticipantmightbemeasuredoveraperiodoftwenty-fourmonthsstartingfromthedateof

entryintotheDTCorprobation.Onereasonforthispracticeisthatitgivesallparticipants

anequivalentfollow-upwindow.Eachparticipantwouldhavetwenty-fourmonthsinwhich

toengage inbehaviors thatarerelevanttoevaluationoutcomes,suchascommittinga

newoffenseorfindingajob.Althoughit isnotalwaysclearwhen“entry”hasoccurred,

mostDTCsandprobationprogramshaveaformalentryhearingorcomparableproceeding

atwhichtheparticipantentersapleaorisadjudicatedandisofficiallyenrolledinthepro-

gram.Thedateofthishearingorproceedingistypicallycountedastheentrydatebecause

106

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

itmarksthepointintimewhentheprogramsgainedauthorityoverparticipantstoimpose

conditionsoftreatmentandsupervision.

Consider,instead,whatwouldhappenifoutcomesweremeasuredfortwelvemonths

from the date of discharge from the program. Some participantsmight have been dis-

chargedaftertwelvemonthsoftreatmentwhereasothersmighthavebeendischargedaf-

tereighteenmonthsoftreatment.Asaresult,outcomesfortheformerparticipantswould

bemeasuredfortwenty-fourmonthsfromentryandoutcomesforthelatterparticipants

wouldbemeasuredfor thirtymonths fromentry.The latter individualswouldhavesix

additionalmonthsinwhichtocommitnewoffensesorengageinotheroutcomebehaviors.

Time at Risk

Regardlessofthelengthofthefollow-upwindow,itisimportantforthefollow-upwin-

dowtobecomparableforallparticipants.Thisensuresthatallparticipantshavethesame

time at risk,meaningtheyhavethesameopportunitytoengageinsubstanceuse,crime,

andotherbehaviorsof interesttotheevaluator. If, forexample,anevaluatormeasured

criminalrecidivismovertwelvemonthsfortheDTCparticipantsbutmeasuredrecidivism

overtwenty-fourmonthsforthecomparisongroup,thiswouldgiveanunfairadvantageto

theDTCparticipants.Thecomparisongroupwouldhavetwelveadditionalmonthsinwhich

tocommitnewcrimes.

Ifthetimeatriskdifferssignificantlybetweenthegroups,theevaluatormightbeable

tocompensateforthisproblembystatisticallyadjustingforthetimeatriskintheoutcome

analyses.Forexample,theevaluatormightenterthetimeatriskasacovariateinthesta-

tisticalanalyses.Acovariateisavariablethatisenteredfirstintoastatisticalanalysis,and

thentheindependenteffectofthevariableofinterest(inthiscase,treatmentinaDTC)is

evaluatedafterfirsttakingtheeffectofthecovariate intoconsideration. Thisprocedure

would indicate whether DTC participants had significantly better outcomes than proba-

tionersafterfirstaccountingfortheinfluenceoftheirtimeatrisk.Theuseofcovariatesis

107

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

notalwayssuccessful,however,andthebestcourseofactionistobeginwithequivalent

follow-upwindowswheneverpossible.

Time at Liberty

Theissueoftime at libertyiscomparabletotimeatriskinthatbothrelatetopartici-

pants’opportunitiestoreoffendorengageinotheroutcomebehaviors.Thedifferenceis

thattimeatlibertyreferstowhetherrestrictiveconditionswereplacedontheparticipant.

Themostobviousrestrictiveconditionsinvolvephysicalbarrierstofreedom,suchasincar-

cerationorplacementinaresidentialtreatmentfacility.Thesephysicalbarriersmaysevere-

lycurtailparticipants’abilitytousedrugs,commitnewoffenses,obtainajob,orengagein

otherbehaviorsofinteresttotheevaluator.

AcommonerrorinDTCevaluationsistoneglecttheissueoftimeatliberty(andtimeat

risk)whencontrastingoutcomestoacomparisongroup.Insomejurisdictions,forexample,

individualswhodonotenterDTCmaybemorelikelytoreceiveajailsentence.Iftheywere

jailed foraportionof the follow-upperiod, theymighthavehad feweropportunities to

reoffendorusedrugsthantheDTCparticipantswhoweretreatedinthecommunity.The

evaluatormightwrongfullyconcludethattheDTCcausedparticipantstoreoffendoruse

drugsmoreoftenthanthecomparisonindividuals,wheninfacttheysimplyhadmoretime

atlibertytodoso.

Undersuchcircumstances,theevaluatormightneedtostatisticallyadjustforthetime

atlibertyintheoutcomeanalyses.Forexample,theevaluatormightneedtoenterthetime

atlibertyasacovariateinthestatisticalmodels.ThiswouldrevealwhethertheDTCpartici-

pantshadsignificantlybetteroutcomesafterfirsttakingintoaccounttheinfluenceoftheir

timeatliberty.

Evaluatorsarenotalwaysadvised,however,toadjustfortimeatliberty.Incostanalyses,

forexample,thetimethatparticipantsspentinjailoraresidentialtreatmentfacilitymight

beanimportantoutcomemeasureinitsownright,andmightbevalueddifferentlythan

108

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

othervariablesfromafiscalstandpoint.Decidingonwhethertostatisticallyadjustforpar-

ticipants’timeatlibertyrequirescarefulconsiderationoftheaimsofeachanalysis.

Adjustingforparticipants’timeatriskandtimeatlibertyobviouslyrequiresevaluators

topaycarefulattentiontotheissueofdate-stamping.Itisessentialtorecordthedateson

whichparticipants entered andwere subsequently released from jail, prison, residential

treatmentfacilities,homedetention,andotherrestrictivesettings.Ifthesedatesarenot

recorded,thenitwillnotbepossibletocontrolforparticipants’timeatlibertyortimeatrisk

ascovariatesintheoutcomeanalyses,andtheresultsoftheevaluationcouldbeseriously

compromised.

Additionally,thisonceagainhighlightshowvaluableitisforDTCprogramstotakethe

timetosetupqualitydatacollection,entry,andmanagementsystemsattheiroutset.Proper,

consistentdatapractices in theseareaswillhelpprovide futureevaluatorswith reliable,

usefuldata—andinturnmakefutureevaluationsbothlessexpensiveandmorereliable.

109

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

This chapter discusses common analytical errors that have beenmade inmanyDTC

programevaluations,andwhichhavelaidwastetohundredsofhoursofworkandthousands

ofdollarsinevaluationfees.Guardingagainstthesecommonerrorscanavoidsubstantial

distressstemmingfromindefensibleevaluationresults.

Target Population

It is commonpractice for evaluators to correlateparticipant characteristics—such as

age,gender,race,nationality,ornumberofpriorconvictions—withoutcomesinDTCpro-

grams.Determiningwhichtypesofindividualsperformedbestintheprogrammayhelpto

identifytheoptimaltargetpopulationfortheDTCandmayindicatewhichparticipantswill

requireenhancedservicesinordertosucceed.Aswasdiscussedearlier,participantcharac-

teristicsthatcorrelatereliablywithoutcomesarecalledpredictorvariablesorriskfactors

becausetheypredictresultsorputparticipantsatriskforrelativelypooreroutcomes.

Aswasalsonotedearlier,however,thereisacriticaldistinctionbetweenpredictorvariables

andmoderatorvariables.Predictorvariablessimplycorrelatewithoutcomes.Forexample,

first-timeoffendersoftenhavebetteroutcomesinDTCsthanindividualswithpriorcriminal

convictions.Therefore,criminalhistoryisapredictorvariableforDTCprograms.Moderator

variables,incontrast,indicatewhichtypesofparticipantsperformedbetterintheDTCascom-

paredtoanalternativeprogram,suchasprobation.Individualswithpriorconvictionshave

poorer outcomes in allcorrectionalrehabilitationprograms,notjustinDTCs.Ifeveryprogram

chosetotargetfirst-timeoffendersbecausetheytendtohavebetteroutcomes,therewould

benoprogramsavailablefortheindividualswhoneedtreatmentthemost.

To determine whether participants’ criminal history is amoderator variable, the

evaluatormustdeterminewhetherdifferences inoutcomesbetween theDTCand com-

AVOIDING COMMON ANALYTIC MISTAKES

110

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

parisonprogramvariedinmagnitudebyparticipants’criminalhistory.Forexample,assume

thattenpercentoffirst-timeoffenderscommittedanewcrimeafterbeingtreatedinaDTC,

andfifteenpercentoffirst-timeoffenderscommittedanewcrimeafterbeingsentencedto

probation.Assumefurtherthattwentypercentofoffenderswithpriorcriminalconvictions

committedanewcrimeafterbeingtreatedinaDTCandthirty-fivepercentofoffenderswith

priorconvictionscommittedanewcrimeafterbeingsentencedtoprobation.Thedifference

inrecidivismforthefirst-timeoffendersisonlyfivepercentagepoints(10percentvs.15per-

cent)butthedifferenceinrecidivismforoffenderswithpriorconvictionsisfifteenpercentage

points(20percentvs.35percent).Inotherwords,themagnitudeoftheeffectoftheDTCis

smallerforthefirst-timeoffendersthanfortheindividualswithpriorconvictions.Thisistrue

eventhoughrecidivismwaslowerforthefirst-timeoffendersintheDTCthanforoffenders

withpriorconvictionsintheDTC(10percentvs.20percent).Underthesecircumstances,par-

ticipants’criminalhistorywouldbeamoderatorvariablefortheeffectsoftheDTC.

Failuretoappreciatethedistinctionbetweenpredictorvariablesandmoderatorvari-

ableshasledmanyDTCstotreatthewrongpopulation.Forexample,studiesintheUnited

StateshaverevealedthatDTCsreducedcrimeapproximatelytwiceasmuchandwere

approximatelyfiftypercentmorecost-effectivewhentheytreatedpersonswithpriorcon-

victionsorotherriskfactorsforreoffending(Bhatietal.,2008;Careyetal.,2012;Lowen-

kampetal.,2005).YetmanyDTCsintheUnitedStatescontinuetoservefirst-timeandlow-

riskoffendersbecausetheseindividualsaremorelikelytograduateandlesslikelytocommit

newcrimesthanrepeatoffenders.Becausefirst-timeandlow-riskoffenderstypicallyper-

formjustaswellonprobationorpre-trialdiversionastheydoinaDTC,thecontributions

oftheseDTCstopublicsafetyandpublichealthtendtobeminimal(Sevignyetal.,2013).

Toavoidmakingthissamemistake,evaluatorsinOASmemberstatesshouldperform

whatarecalledmoderator analyses or interaction analysesratherthansimplycorrelating

participantcharacteristicswithoutcomes.Theseanalysesrevealthreefindingsthatmust

beinterpretedtogether. Theyreveal(1)whichprogramproducedbetteroutcomes(e.g.,

111

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

DTCvs.probation);(2)whichtypesofparticipantshadbetteroutcomes(e.g.,first-timevs.

repeatoffenders)and;mostimportantly,(3)whichtypesofparticipantsperformedbetter

inwhichtypeofprogram.Thisthirdpieceofinformationindicatesthebestwaytomatch

individualstothemosteffectiveprogramstomeettheirneedsandindicateshowacountry

orjurisdictionshouldmarshalitsresourcestobemostefficientandcost-effective.Ifthere-

sultsofmoderatoranalysesarethesameinotherOASmemberstatesastheyhavebeenin

CanadaandtheUnitedStates,theyarelikelytoindicatethat,allelsebeingequal,first-time

addictedoffendersshouldtypicallybeplacedonprobationorpretrialdiversion,andrepeat

addictedoffendersshouldbeplacedinaDTC.

Disadvantaged Groups

DTCs commonly serve persons who have historically experienced discrimination or

reduced social opportunities due to their race, ethnicity, nationality, gender, physical or

mentaldisability,religion,orsocioeconomicstatus.Itisimportanttodeterminewhether

membersofthesedisadvantagedgroupsreceivethesameservicesandsucceedatthesame

rateasotherparticipantsinDTCprograms.

Manyevaluatorshaveattemptedtoaddressthisissuebycorrelatingvariablessuchas

raceorethnicitywithoutcomes.Forexample,studiesinAustralia(Rysavyetal.,2011)and

theUnitedStates (Finigan,2009;Marlowe,2013) found that racialminoritiesandmem-

bersofindigenouspopulationshadsignificantlypooreroutcomesinmanyDTCsthanother

participants.Thesefindingshaveledsomecommentatorstoconclude—prematurely—that

DTCsdiscriminateagainstpersonsofcolor(O’Hear,2009).

Itisthereforenotsufficienttosimplycorrelateoutcomeswithparticipantcharacteris-

ticssuchasraceorethnicity.Othervariablesmightbecorrelatedwithraceorethnicityand

mightbetrulyresponsibleforthedifferencesinoutcomes.

Infact,evidencesuggestsdisparitiesforminorities inDTCsarenotafunctionofrace

orethnicityper se,butratherareexplainedbybroadersocietalburdenswhichareoften

112

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

bornedisproportionatelybyracialandethnicminorities,suchaslessereducationalorem-

ploymentopportunitiesoragreaterinfiltrationofcrackcocaineandotherseriouslyaddic-

tivedrugsintosomeminoritycommunities.Whenevaluatorshaveadjustedstatisticallyfor

theseconfoundingvariablesintheanalyses,theimpactofraceandethnicitydisappeared

altogether(Dannerbecketal.,2006).

Thisfindingrequiresevaluatorstoaccountstatisticallyfortheinfluenceofvariablesthat

maybecorrelatedwithrace,ethnicity,orothercharacteristicsofdisadvantagedgroups—

especiallyparticipants’ socioeconomic statusandprimarydrug(s)used—and thendeter-

minewhetherraceorethnicitycontinuestopredictpooreroutcomesafterthesecovariates

havebeenfactoredout.Thisprocesscanbeaccomplishedinseveralways,buttheessen-

tialprocedureistofollowthreebasicsteps:First,theevaluatormustdeterminewhether

variablessuchassocioeconomicstatusordrugsusedaresignificantlycorrelatedwithrace,

ethnicityorothercharacteristicsofdisadvantagedgroups.Iftheanswertothisfirstques-

tionisyes,thentheevaluatormustdeterminewhetherthesevariablesarealsocorrelated

withoutcomes,suchasgraduationratesornewarrestrates.Iftheanswertothesecond

questionisalsoyes,thenthethirdstepistoenterthesevariablesascovariatesintoanalyses

comparingoutcomesbetweenmembersofdisadvantagedgroupsandotherparticipants.

Ifthedisadvantagedgroupscontinuetohavesignificantlypooreroutcomesafterthisthird

stephasbeenfollowed,thenandonlythenwoulditbejustifiedtoconcludethatthereare

disparateimpactsforthosedisadvantagedgroupsinDTCs.

Adjusting for Baseline Differences

Aswasdiscussedpreviously,mostevaluationscompareoutcomesbetweenDTCpar-

ticipants and those of a comparison group such as probationers. Prior tomaking these

comparisons, it isnecessarytoruleoutcompetingexplanationsthatmightaccountfor

differencesinoutcomesotherthantheeffectsoftheDTCprogram.Forexample,thepro-

bationersmighthavehadmoreseverecriminalhistoriestobeginwith.Ifso,thensuperior

113

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

outcomesfortheDTCparticipantsmighthavehadnothingwhatsoevertodowiththeeffects

oftheDTCprogram,butrathermightsimplyhavereflectedthefactthattheDTCtreateda

populationthatwasalreadymorelikelytoarriveatmorefavorableoutcomes.Indeed,even

whenanevaluatoremploysrandomassignment,thereisstillapossibility(althoughagreat-

lydiminishedpossibility)thatthegroupsmaydifferonimportant/materialdimensions.

Ifthegroupsdidmateriallydifferatthetimeofentry,allisnotnecessarilylost.There

areproceduresanevaluator canemploy toadjust for thedifferencesandobtainpoten-

tiallydefensibleresults.Thistypicallyinvolvesathree-stepprocess.First,thegroupsare

comparedonriskfactorsthatareknowntoinfluenceoutcomesamongdrug-involvedindi-

vidualschargedwithcriminaloffenses.Theseriskfactors,suchasageandpriorcriminal

convictions,arelistedearlierinthismanual.Ifthegroupsarefoundtodifferonanyofthese

riskfactors,theevaluatormustnextdeterminewhether, in fact, theseriskfactorspre-

dictedpooreroutcomesinthecurrentsample.Forexample,althoughayoungerageoften

predictspooreroutcomesamongdrug-involvedoffenders,itmightnothavedonesointhe

presentstudy.Ifitdoesnotpredictoutcomesinthecurrentstudy,theevaluatordoesnot

needtobeconcernedaboutthisvariablegoingforward.

Third,anyvariablesthatboth(A)differbetweenthegroupsand(B)predictoutcomes

mustbecontrolledforstatisticallyintheoutcomeanalyses.Forexample,ifthegroups

differbyage,andtheageofaparticipantpredictsoutcomes,thenagemustbeenteredasa

covariateintheoutcomeanalyses.Howthisisaccomplishedwilldependonwhatstatistical

test isbeingperformed. Forexample,theevaluatormightuseananalysisofcovariance

(ANCOVA)orahierarchicalregressionanalysis,whichcanadjustforcovariatesthatmight

beconfoundingtheresults.

Unfortunately,inmanyevaluationstheDTCmightnothavecollecteddataconcerningall

riskfactorsthatshouldbeconsidered.Forexample,theevaluatormightnothaveanyway

ofknowingwhethertheparticipantshadpriorcriminalconvictionsorhadbeeninsubstance

usetreatmentpreviously.Withoutthisimportantinformation,thereisnowayofknowing

114

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

whetherthegroupsdifferedonthesecharacteristicswhentheyenteredtheprogramsand

whethersuchdifferencesmighthaveinfluencedtheoutcomesthatwereproduced. Ifthis

criticalinformationislostorunavailable,theresultsoftheevaluationcannotbereliedupon

withareasonabledegreeofscientificconfidence.(Oncemore,thevalueofDTCs’initialdesign

andimplementationofrobustdatacollection,entry,andmanagementmethodsismanifest.)

Impact of Specific Services or Interventions

ManyDTCswillwanttoknowwhetherspecificservicestheyprovideorconsequences

theyimposeimprove,hinder,orhavenoeffectonoutcomes.Forexample,aDTCpracti-

tionermightwanttoknowwhetheradministeringpunitivesanctionsorreferringpartici-

pantstoanewtypeofcounselinggroupimprovedtheiroutcomesandwasworththeadded

effortandexpense.

Ideally,suchanalysesshouldnotbeconductedpost hoc,orafterthefact.Assume,for

example,thatanevaluatorcorrelatedthenumberofsanctionsthatparticipantsreceived

with theiroutcomes inaDTC. It isdetermined from this analysis thatparticipantswho

receivedmoresanctionswere less likelytograduatesuccessfully fromtheprogram.This

couldleadtheevaluatortotheerroneousconclusionthatsanctionsmadeoutcomesworse.

Butsuchaconclusionislikelytohaveconfusedcausewitheffect.Itismorelikelythatpoor

performanceledthestafftoapplymoresanctions.Ifso,thenmoresanctionsdidnotcause

worseoutcomes,butratherworseoutcomeselicitedmoresanctions.

Similarly, participantswhohavemore severe addictionproblemsorwho arenot re-

spondingtostandardtreatmentaremorelikelytobereferredformoreintensiveorspe-

cializedtreatment.Simplycorrelatingtreatmentwithoutcomescouldleadtotheerroneous

conclusionthatmoretreatmentledtopooreroutcomes,wheninfactpoorperformance

oftenleadstoareferralformoretreatment.Forexample,participantswithmorese-

vere addictions aremore likely to be referred for residential treatment than outpatient

treatment.Iftheirlong-termoutcomesareworsethanforparticipantswhowerereferred

115

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

tooutpatienttreatment,theevaluatormightreachthewrongconclusionthatresidential

treatmentisineffectiveorharmful,wheninfactresidentialprogramssimplytreatedamore

seriouspopulationthanoutpatientprograms.

Thebestmethod istoaddressthesequestionsa priori,orbeforehand. Ifpossible,

participantsshouldberandomlyassignedinadvancetodifferenttypesordosagesofser-

vices.Forexample,onarandombasissomeparticipantsmightberequiredtoattendfifty

hoursoftreatmentwhereasothersmightberequiredtoattendtwentyhoursoftreatment.

Usingthisresearchdesign,anydifferencesinoutcomesmaybereasonablyattributedtothe

dosageoftreatmentandnotmerelytosystematicreferralsbystaff.

Analternativeapproachistoconductamediation analysis.Asmentionedearlier,ame-

diatorvariableisathirdvariablethataffectstherelationshipbetweenapredictorvariable

(suchasenrollmentinaDTC)andanoutcomevariable(suchasrecidivismorgraduation

rates).Themediatorvariablecanbethoughtofasaninterveningvariablethatexplainsthe

relationshipbetweenthepredictorvariableandtheoutcomevariable.Inordertoconfirm

thatthereisamediationeffect,severalstatisticalrelationshipsmustbeestablished(Baron

&Kenny,1986;MacKinnon,2008):(1)thepredictorvariablemustbesignificantlycorrelated

withtheoutcomevariable,(2)thepredictorvariablemustbesignificantlycorrelatedwith

themediatorvariable,and(3)themediatorvariablemustbesignificantlycorrelatedwith

theoutcomevariable. If these three conditions are allmet, then itmustbe further

establishedthat(4)therelationshipbetweenthepredictorvariableandtheoutcome

variableiseliminatedorsignificantlyreducedinmagnitudeaftercontrollingfortheme-

diatorvariableasacovariate.

Toillustrate,assumethatanevaluatorwantedtoknowwhetherDTCswereresponsible

forbetteroutcomesthanprobation,becausetheformerprovidemorehoursoftreatment

toparticipants.Todoso,theevaluatorwouldneedtoestablishthat(1)beingassignedto

theDTCwascorrelatedwithbetteroutcomesthanbeingassignedtoprobation,(2)being

assignedtotheDTCwascorrelatedwithreceivingmorehoursoftreatmentthanprobation,

116

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

and(3)receivingmorehoursoftreatmentwascorrelatedwithbetteroutcomes.Then,the

evaluatorwouldneedtoestablishthattheeffectoftheDTCwassignificantlyreducedafter

accountingfortheamountoftreatmentparticipantsreceived. Ifallthreeconditionsare

met,thentheevaluatorcanreasonablyconcludetheDTCworkedbyincreasingparticipants’

exposuretotreatment,whichinturnledtobetteroutcomes.Inotherwords,increasingex-

posuretotreatmentpartiallymediatedorpartiallyexplainedthepositiveeffectsoftheDTC.

Commonstatisticalproceduresthatmaybeusedtotestformediationinclude,butare

notlimitedto,ordinaryleastsquares(OLS)multipleregressionanalyses,structuralequation

modeling(SEM),logisticregressions(forbinarymediatororoutcomevariables),andmulti-

levelmodeling(forclusteredornesteddata).Inmanyinstances,expertstatisticalconsulta-

tionwillberequiredtoconductthesesophisticatedmediationanalysescorrectly.

Infrequent Events

SomeeventsoccurinfrequentlyinDTCprograms.Forexample,ifoutcomesareexamined

duringthefirstsixmonths,arrestsfornewoffensesmightnotoccurathighrates.Itusually

takesa longertimethansixmonthsforparticipantstore-engageincriminalconduct,be

detectedbylawenforcement,andhavecriminalchargesfiledagainstthem.Amongthose

participantswhodoquicklyreturntocriminalbehavior,onewouldnotexpecttoseemore

thanaboutoneortwonewarrestsduringthefirstsixmonths.

Acommonmistake inDTCevaluations is toemploystatistical tests suchas t-testsor

analyses of variance (ANOVA)which examine the averagenumberof infrequent events,

suchastheaveragenumberofnewarrests.Becausethenumberofnewarrestsislikelyto

bezeroformanyparticipants,theaveragenumberofarrestsislikelytobelessthanoneor

two.Thiscreateswhatiscalledaskewed distribution or non-normal distribution of scores.

Formathematicalreasons,oneisunlikelytodetectstatisticallysignificantdifferencesifthe

distributionisskewedinthismanner.Althoughtheremightbeaclinicallymeaningfuldiffer-

encebetweenthegroups,itisunlikelytobedetectedbythestatisticaltest.

117

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Forinfrequentevents,itmaybeadvisabletotransformthedataintoabinaryor

dichotomous variable(e.g.,anynewarrestvs.nonewarrest)andtousewhatarecalled

nonparametricstatisticaltests.Forexample,ratherthananalyzingtheaveragenumberof

newarrests,theevaluatormightanalyzethepercentageofparticipantswhohadatleastone

newarrest.Nonparametrictestscanhelptoavoidtheproblemsassociatedwithskewed

distributionsandcanleadtomoresensitiveandpowerfulfindings.Examplesofcommonly

used nonparametric tests include, but are not limited to, chi-square tests, Fishers Exact

ProbabilityTests,MannUtests,Wilcoxontests,andlogisticregressionanalyses.

Alternatively,evaluatorsmayperformwhatiscalledalog-linear transformationofthe

data to compensate for the skewedornon-normaldistribution. Thisprocedurehas the

mathematicaleffectofsmoothingouttheskeweddistributionormakingitfunctionasifit

wereanormaldistribution.Expertstatisticalconsultationwilloftenberequiredtoknow

whenandhowtoemployanadvancedcompensatoryproceduresuchasalog-lineartrans-

formation.

118

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

DTCprogramsuseresourcesfrommultiplesystemsandagencies(e.g., thecourt,the

criminal justicesystem,andthesubstanceabusetreatmentsystem).Suchprogramsthus

needtobedesignedandimplementedinamannerthatmaximizesbenefitsandminimizes

coststoparticipants,thevariouspublicsystems/agenciesinvolved,andtosocietyingeneral.

Policymakers and thepublic thereforehave an interest in calculating and comparing

thesebenefitsandcosts.Onewaytoprovideafamiliarandusefulstandardforsuchcal-

culationsandcomparisons ismonetizing(puttingdollaramountsto)thevariousbenefits

andcoststhroughso-calledcost-efficiencyanalyses.ComprehensiveDTCcost-efficiency

analysesrequirethefollowingelements:

1) Acarefulprocessevaluationtoidentifytimesandplaceswherecostsandpotential

benefitsareincurred;

2) Collectionofquantitativedataonprogramactivities(e.g.,fromadministrative

databasessuchascriminaljusticedatabasesthatcontaindatesofarrestsandcharges);

3) AnanalysisoftheDTCprogram’sbenefitsandcosts;and

4) AnanalysisofanappropriatecomparisongroupinanalternativetotheDTC,to

juxtaposewiththeanalysisoftheDTCprogram.

Naturally,puttingtheoryintopracticeisoftenmorecomplexthanitmayfirstappear.

Potentialchallengesinherentindesigningandcarryingoutcost-efficiencyanalysesinclude:

�AssigningdollaramountstothevalueofcertainDTCbenefitscanbedifficult;

�Cost-efficiencyanalysisinvolvesmakingcertainassumptions(e.g.,thatthegoalsand

outcomessoughtbydifferentprogramsarethesame;thatthedollarvalueofacost

orbenefitinonelocationisthesameasitsvalueinanother)thatmustbetestedto

ensuretheintegrityoftheanalysis;and

�MonetarybenefitsthataDTCmaygenerateoveranalternativeapproacharesome-

timeaccruedbyagenciesotherthantheDTCitself(e.g.,statecorrectionsagencies).

COST ANALYSESMike Finigan and Shannon Carey

119

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

TherearefourmajortypesofcostanalysisthathavebeenappliedtoDTCsaswellas

substanceabusetreatmentprogramsingeneral.Theyrangefromsimple/inexpensivebut

lessaccurateapproachestothosethatarecomplex/expensivebutmoreaccurate.Assuch,

each implies certain trade-offs thatpolicymakers,evaluators, andDTCprogramstaffwill

needtotakeintoconsideration.

Type 1: Program-cost analysis. Program-costanalysesinvolvecalculatingalloftheex-

pensesrequiredtoimplementandoperateagivenprogram.Theyarethemostbasicform

ofcostanalysis,andallothertypesbuilduponthistype.

Abasic program-cost analysis is used to showwhat resources areneeded to fund a

particularprogram.Theprogram’sbudget(s)andanaccountingofallotherresourcesput

intotheprogram(in-kindresources,etc.)arerequiredtoproperlyassessaprogram’sfull

cost.Consequently,theprogramcoststhatabasicprogram-costanalysismeasuresshould

includeboththemorevisibleandeasilycalculateddirectcosts,suchassalariesandbenefits

forDTCstaff,butalsotheprogram’sindirectcosts,suchasthoseforsupportservices,train-

ing,officespace,andadministrativeoverheadgenerally.

Duetotheirrelativesimplicityandinexpensivenature,program-costanalysescanbeuseful

forprogramsthathavefewevaluationresourcesand/orneedonlytocarryouta limited

assessment of costs.

Type 2: Cost-allocation analysis.Costallocationanalysesdeterminetheunitcost(or

costperunit)ofaparticularserviceintheprogram(e.g.,thecostpercourtsession,orthe

cost per screening). This provides thebasic information required formore sophisticated

cost-effectivenessanalysisorcost-benefitanalysis,thethirdandfourthtypesofanalyses

explainedhere.Moreover,cost-allocationanalysesarevitaltostudyDTCprogramsinwhich

multipleagenciescontributeresourcesandservices.

Accordinglycost-allocationanalysesrequirenotonlysettingupaccountingorbudgeting

systemsinawaythatallowsprogramstafftoallocatecoststothevariousservicesprovided,

120

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

butalsocollectinginformationaboutprogramparticipantsandservicesdelivered.

Whilecost-allocationanalysespermitcalculationofprogramandunitcosts,theycannot

bythemselvesdeterminewhethertheprogramisanefficientuseofresources.Suchade-

terminationrequiresacomparisonoftheDTCprogramtootheralternatives.Forexample,

asimilarprogrammayproduceequivalentresultstotheDTCprogramatalowercost,thus

makingitmoreefficientintermsofcosts.Inordertoreachconclusionsaboutefficiency,a

cost-effectivenessanalysisisrequired,whichisthenexttypeofanalysisconsidered.

Type 3: Cost-effectiveness analysis. Cost-effectivenessanalysesevaluatealternative

programs’costsandeffectswithregardtoproducingadesiredoutcome.Inacost-effective-

nessanalysis,thecostsofdifferentprogramsareassessedandthencomparedintermsof

similar outcomes.

Nonetheless, keep in mind that program outcomes are not expressed in monetary

terms,butratherinsomeotherimpact,suchasareductioninrecidivism.Inthecontextof

acost-effectivenessanalysis,areductioninrecidivismisnotexpressinmonetaryterms,but

isinsteadexpressedinanon-monetaryvalue,suchasthepercentreductioninrearrests.

Inthatcontext,thecost-effectivenessanalysiswouldexpressresultsintermsthepercent

reductioninrearrestsperacertainamountofmoneyspent.

Thefollowing(seeFigure3)isanexampleofacost-effectivenessanalysiswhereinaDTC

iscomparedtoastandardprobationprogramcost,intermsofeachprogram’seffectiveness

inproducingthedesiredoutcome(reductioninrearrests)pereveryUS$10,000incosts:

121

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Inthisexample,Figure3showsthatDTCfordrugoffendersreducesthepercentageof

rearrestssubstantiallymorepereachUS$10,000spentthanthestandardprobationpro-

gram. Accordingly,whilecost-effectivenessanalysesaremorecomplexand require a full

outcomeevaluation foreachprogrambeingcompared, theyarealsoavaluable tool for

policymakersthatwishtodeterminethecostefficiencyofalternativeapproaches.

Type 4: Cost-benefit analysis. Lastly,acost-benefitanalysis (CBA)calculates the

monetarycostofprogramsaswellasthemonetarycostoftheoutcomestothesystem.This

resultsinacost-to-benefitratiothatmonetizesbothcostsandoutcomes—thatis,aratio

thatindicatesthatthereareXdollarsinbenefitsforeverydollarinvestedintheprogram.A

programwithbenefitsthatoutweighitscostsissaidtohavenet benefits.

CBAsprovidethemostcompletepictureofthevalueofacomplexprogramsuchasDTC,

and represent the only approach that allows an assessment of cost-benefit ratio.While

this is themostcomprehensivetypeofcostanalysis, italsotakesmoretime,effort,and

Percent reduction in rearrests per

US$10,000 spent

DTC Probation

20%

15%

10%

5%

0%

20%

7%

Figure 3. Compares the Impact of Drug Treatment Court and Standard Probation per US$10,000 spent

122

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

resourcestoconduct.CBAsrequireafulloutcomeevaluation,dataonoutcomesforboth

programparticipantsandacomparisongroup,andcostdatathatcanbeassociatedwith

thoseoutcomedata. Nonetheless, given that policymakers are interested in seeing the

monetarybenefitsofanewprogrambeforedecidingtoprovidefinancing,CBAscanbeex-

tremelyuseful.

Toprovideareal-worldexample,oneoftheearliestCBAsofU.S.DTCsfoundthe

following:AnestimatedUS$1,002,979wasspentpercohortofclientswhoparticipatedin

theDTCprogrameachyear.Ontheotherhand,theDTCprogramresultedinUS$2,476,795

ofavoidedcosts (or savings). Therefore, the studycalculated thatevery taxpayerdollar

spentonthosecohortsproducedUS$2.50ofsavingstothetaxpayer.Furthermore,ifthe

broadercosts(includingvictimizationandtheftcosts)wereestimatedandincludedinthe

analysis,theratioofbenefittothetaxpayerwasten-to-one—thatis,theprogramsavedten

dollarsforeverydollaritspent.(Carey&Finigan,2004).

Cost Assessment Techniques

Thewaycostsareassessedalsovariesdependingonthepolicyquestionsdesiredand

thebudget forcostevaluation. Anumberofassessmenttechniquesexists,asexplained

below.

Funded budgets

SomecostanalysesassociatedwiththeimplementationofDTCshaveassumedthatarea-

sonablewayofassessingthecostofDTCistofocussolelyonthefundsspentcreatingtheDTC

program.(IntheUnitedStates,theseareoftenfederalgrantfundsspentontheproject.)This

approach,however,failstoaccountforresourcesthattheexistingcriminaljusticeandtreat-

mentsystems,whicharenotfundedbyDTCgrantsorotherdirectDTCappropriations,must

contributetotheDTCprogram.Thus,theapproachofusingfederalgrantmoneyordirect

federalDTCappropriationstoassess investmentcostsseriouslyunderestimatesthe invest-

mentofstateandlocaljurisdictionalandagencyresourcesforDTCs.

123

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Marginal costs

Themarginalcostrepresentstheamountthatthetotalprogramcostchangeswhen

aunitofoutput(alsoreferredtoas“workload”)changes.Inacost-benefitanalysis,“mar-

ginal”thusdoesnotmeansmallorinsignificant,butratherthecoststhatareatthemargin

ofanexistinglevelofoperations.Inotherwords,thetermdescribesthecostorbenefitthat

willberealizedbecauseofchangesinunitsofactivity.

Inthecontextofthecriminaljusticesystem,themarginalcostistheamountofchange

inanagency’stotaloperatingcostswhenoutput(suchasarrests,courtfilings,orjaildays)

changesbecauseofchangestopoliciesorprograms.Forexample,costsforaprison—such

asbuildingmaintenance,utilities,andoverheadexpense—arefixedandgenerallydonot

changewhetherornotaprisonbedisfilled.Variablecostssuchasfoodsupplies,however,

wouldchangeiftherewerefewerormoreoffendersintheprison.

Themarginal cost approach argues thatmarginal costs should only include variable

costs(e.g.,overtime,supplies,food)andnotfixedcosts(e.g.,rents,utilities),becausefixed

costsdonotmateriallychangewiththeadditionofonemoreunitofworkload(Henrichson

&Galgano,2013). Therefore,whenusingamarginal costapproach,only thesevariable

costsshouldbecountedinmeasuringincreasesordecreasesincost(ormeasuringmone-

tarysavings).

Thereare,however,severalreasonsthatthismarginalcostapproachmaynotworkin

complexinter-organizationalsettingssuchastheDTCoperatingenvironment.First,itdoes

notaccountfororganizationalorotherstructuralchangesthatoccurinacriminal justice

systemwhenaDTCisimplemented.Forinstance,becausethemarginalcostapproach

assumes that the standard operation of the court remains the same, it fails to account

forthefactthatcourtcalendarsmaychangeasjudgesincreaseordecreasecaseloadsto

accommodatetheDTCjudge’scaseload. Duetothesamekindofassumptionaboutthe

standardprobationapproachremainingthesamewhenaDTCisaddedtothesystem,the

marginal costapproachalso fails toassess the systemiceffect involved in caseload re-

124

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

assignmentsforprobationofficersresultingfromtheassignmentofDTCcaseloads.These

caseloadstypicallyhavemoreintensivesupervisionrequirementsassociatedwiththeDTC

program,apracticethataffectsbothfixedandmarginalcosts.Inaddition,thisapproachof-

tenunderestimatesthecostsofday-to-dayoperations,so-called“business-as-usual”costs

thatDTCsoftenoffsetorreduce.Inotherwords,themarginalcostapproachassumesthat

theDTCenvironmentwouldnotimprovethecostofprocessing“business-as-usual”cases,

andthusoffsetsomeofthenewcostsand/oreliminatingsomeofthese“business-as-usual”

costs. Itisclearthatinsomecases,theimplementationofDTCsactuallyreducesthecosts

ofprocessingeligibleclientsascomparedto“business-as-usual.”InastudyofnineDTCsin

California,onecourtfoundthatprocessingclientsthroughDTCactuallysavedthecriminal

justicesystemUS$1,500perclientwhenbothfixedandmarginalcostssuchasrentandutil-

itieswereincludedintheanalysis(Carey,Finigan,Crumpton,&Waller,2006). Marginal cost

analysismayhaveidentifiedthe“new”costsfromtheintroductionoftheDTCprogramand

failedtoassessthesavingsinthetraditionalsystemthataccruedfromthisnewmethodof

processingcases.ThesameresultwasfoundinaDTCinOregon(Carey&Finigan,2004).It

isalsoclearfromseveralothersitesthatsimplyassessingthevariableorsemi-fixedcostsof

allDTCcomponentsthatarenewtothesystemwouldinaccuratelyrepresentthenetcost

oftheDTC.

Broader societal outcomes

SomecostapproachescanmeasuretheimpactofDTCsbeyondtheirimmediateimpact

onthecriminaljustice,courtandsubstanceabusesystems—somethingthatsomepolicy-

makersdesire.AnotableexampleofthisapproachistherecentanalysisbytheNationalIn-

stituteofJustice(theMultisiteAdultDrugCourtEvaluation,MADCE),whichincludedsocial

productivityandbroaderserviceuseinitsanalysis(Roman,2013)(seeTable2).

125

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Table 2. Outcomes Measured by MADCE

Outcome Category Outcome Sub-Category Impacts

1. Social productivity

A.Employment B.Education C.Servicesandsupportprovided

• Earnings• Schooling• Childsupportpayments,com-

munityservice

2. Criminal justice system

A. Monitoring B.Police C. Courts D.Corrections E.DrugTreatmentCourt

• Probationofficermeetings,drugtests,electronicmonitoring

• Arrests• Hearings• Jailandprison(sanctionsor

otherwise)• Casemanagement,

administrativecosts

3. Crime and victimization • Crimescommitted

4. Service use A. Drug treatment B.Medicaltreatment C.Mentalhealthtreatment D.Other

• Emergencyroom,detoxification,residentialcare,outpatient,methadone

• Hospitalstaysunrelatedtodrugs

• Staysinmentalhealthfacilitiesunrelatedtodrugs

• Halfwayhouses,publichousing,homelessshelters

TheMADCEresearchersuseda“bottom-up”approach,whichcomparingthecostsand

avoidedcosts foreachparticipant in termsof socialproductivity, criminal justice system

activities,crimeandvictimization,andserviceuse.Theresultingmeasurecouldbeapositive

ornegativevalueforeachpersoninthestudy(Roman,2013).TheMADCEfoundthatinsome

broaderareassuchasserviceuseandfinancialsupportthedrugcourtparticipantscostmore,

yetthesavingswithinthecriminaljusticesystemmorethanoffsetthesecosts.Theoverall

resultwasacostsavings(althoughthesavingsinthisparticularstudywerenotsignificant).

126

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Thisbroadapproachproducesexcellentsociety-wideresultsbutisdifficulttoimplement

becausegettinggoodcostdataonimpactedexpensesrequiresextremelydetailedmeasure-

ment,asdescribedearlierinthesectionon“SourcesofInformation.”Theseincludefinding

dataonsuchquestionsasuseofdetoxificationservices,emergencyroomvisits,lostwages

orwageincreases,taxpayments,welfarepayments,anddisabilitypayments(asdescribed

inTable2).Datasourcesforthesedataelementsareoftenquitedifficulttofind,ordon’t

existatall.

Transactional and Institutional Cost Analysis (TICA)

Adifferentcost-benefitapproachthatusesbothmarginalcostsandfixedcostscomes

from transactional economics, and is called Transactional and Institutional Cost Analysis

(TICA)(Crumpton,Carey,&Finigan,2004).TheTICAapproachviewsanindividual’sinter-

actionwithpubliclyfundedagenciesasasetoftransactionsinwhichtheindividualutilizes

resourcescontributedfrommultipleagencies.Transactionsarethosepointswithinasystem

whereresourcesareconsumedand/orchangehands.

In the caseofDTCs,whenaDTCparticipantappears in courtorhasadrug test, re-

sourcessuchasjudgetime,defenseattorneytime,courtfacilities,andurinecupsarecon-

sumed.Courtappearancesanddrugtestscan,inthiscontext,beanalyzedastransactions.

Inaddition,theTICAapproachrecognizesthatthesetransactionstakeplacewithinmultiple

organizationsandinstitutionsthatworktogethertocreatetheprogramofinterest.These

organizationsandinstitutionscontributetothecostofeachtransactionthatoccursforpro-

gramparticipants.BecauseTICAmeasuresthecostsofbothtransactionsandthespecific

organizationsthatarecontributingresources,TICAisaparticularlyappropriateapproach

toconductingcostsassessmentsinanenvironmentsuchasaDTC,whichinvolvescomplex

interactionsamongthemultipleorganizationsinvolvedintheDTCprocess.

127

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Cost to the taxpayer

Inordertoproduceinformationusefultopolicymakers,intheUnitedStatesa“cost-to-tax-

payer”approachisused.Thisfocusonthetaxpayer(orintheaggregate,thegovernment)

helpsdefinewhichcostdatashouldbecollected(costsandavoidedcostsinvolvingpublic

funds)andwhichcostdatashouldbeomittedfromtheanalyses(e.g.,coststotheindividual

participatingintheprogram).Althoughthis“taxpayer”approachmaynotbeapplicablein

somecountries,asimilarapproachcouldbetakenbyfocusingonaspecificfundingsource

(orasubsetoffundingsources)thatareofparticularrelevanceorimportintheoperation

oftheprogram.Focusingonasmallersubsetoffundingsourcescanhelpnarrowdownand

betterdefinethebreadthofdatathatmustbecollected,whichmaymakeacostanalysis

morefeasibletoaccomplishwithalimitedbudget,oralimitedtime.

Opportunity resources

Finally, theTICAapproach looksat costs as “opportunity resources.” The concept is

similartothatofopportunitycostfromtheeconomicliterature,whichsuggeststhatsystem

resourcesareavailabletobeusedinothercontexts iftheyarenotspentonaparticular

transaction. The termopportunity resource describes these resources that are now

availablefordifferentuse.

Forexample,ifsubstanceabusetreatmentreducesthenumberoftimesthataclientis

subsequentlyincarcerated,thelocalsheriffmayseenochangeinhisorherbudget.How-

ever,anopportunityresourcewillbeavailabletothesheriffintheformofajailbedthat

cannowbefilledbyanotherpersonwho,perhaps,possessesamoreseriouscriminaljustice

record than does the individualwho has received treatment and successfully avoided

subsequentincarceration.

Notably,theTICAapproachdoesnotuseamarginalcostapproach.Allcostsreported

are fully loaded,3totalcosts.Thus,theTICAapproachdoesnotassumethatfixedcosts,such

3 Fully-loadedcostsarethosethatincludedirectcosts(suchassalaries),aswellasindirectcostssuchasbene-

128

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

asforbuildingmaintenance,utilities,salaries,orotheroverheadexpenses,areoff-limits.

Thosecostsarestillbornebyfundersandcanchangeovertimeorbedeployedfordifferent

purposes.

TheTICAapproachalsodoesnotmakeassumptionsastowhetherornotagencieswill

changetheirbudgetsinwaysthatenablesavingstobeliterallyrealized.All“savings”

calculatedusingtheTICAapproachrepresent“opportunityresources”thatarenewlyavail-

abletogovernmentoragencyactorstodeployastheychoose.Ratherthanassumethat

directdollarswill be immediately saved (e.g., through reductions in the followingyear’s

budget),theTICAapproachfullyquantifiesallresources(marginalorfixed)—suchasajail

bed,treatmentslot,ortimespentinacourthearing—thatarenewlymadeavailablefora

differentoffenderoradifferentpurpose.

Table3providesthesixkeystepsintheTICAapproach.

fitsandoverhead.

129

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Table 3. The Six Steps of TICA

Description Tasks

Step 1:Determineflow/process(i.e.,howprogramparticipantsmovethroughthesystem)

Sitevisits/directobservationsofprogrampractice

Interviewswithkeyinformants(agencyandprogramstaff)usingadrugcourttypologyandcostguide(Seeguideonwww.npcresearch.com)

Step 2:

Identifythetransactionsthatoccurwithinthisflow(i.e.,whereclientsinteractwiththesystem).SeeTables4and5foralistofprogramtransactions.

AnalysisofprocessinformationgainedinStep1

Step 3:

Identifytheagenciesinvolvedineachtransaction(e.g.,court,treatment,police)

AnalysisofprocessinformationgainedinStep1

Directobservationofprogramtransactions

Step 4:

Determinetheresourcesusedbyeachagencyforeachtransaction(e.g.,amountofjudgetimepertransaction,amountofattorneytimepertransaction,numberoftransactionevents)

Interviewswithkeyprograminformantsusingprogramtypologyandcostguide

Directobservationofprogramtransactions

Administrativedatacollectionofnumberoftransactions(e.g.,numberofcourtappearances,numberoftreatmentsessions,numberofdrugtests)

Step 5:

Determinethecostoftheresourcesusedbyeachagencyforeachtransaction(e.g.,staffsalaries,overheadrates,supplycosts)

Interviewswithbudgetandfinanceofficers

Documentreviewofagencybudgetsandotherfinancialpaperwork

Step 6:

Calculatecostresults(e.g.,costpertransaction,totalcostoftheprogramperparticipant)

Indirectsupportandoverheadcosts(asapercentageofdirectcosts)areaddedtothedirectcostsofeachtransactiontodeterminethecostpertransaction

Thetransactioncostismultipliedbytheaveragenumberoftransactionstodeterminethetotalaveragecostpertransactiontype

Thesetotalaveragecostspertransactiontypeareaddedtodeterminetheprogramandoutcomecosts.

130

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Aspartofatypicalprocessevaluation,Step1(understandingtheflow,orprocess,of

theprogram),Step2(identifyingthetransactionsintheprocess)andStep3(identifyingthe

agenciesinvolved)areperformed(seethesectionofthismanualonprocessevaluation).

Athoroughprocessevaluationshouldprovideyouwithanunderstandingoftheprogram

process,includingwheretransactionsareoccurring(e.g.,placeswhereprogramstafftime

oragencyresourcesarebeingused)andwhatagenciesare involvedineachtransaction.

Forexample,whenperformingasitevisittotheprogram,anevaluatorcanobserveaDTC

hearing(i.e.,atransaction),theevaluatorcanobservehowthestaffandparticipantsin-

teract(theprogramflow),andwhichagenciesareinattendanceatthehearing.

Step 4 (determiningtheresourcescontributedbyeachagency)occursthrough inter-

viewsandobservation,aswellasgatheringquantitativedataonnumbersoftransactions

foreachparticipant.Continuingwiththeexampleofthecourthearing,theevaluatorcan

usea stopwatch (oraclock) totimehow long thestaff fromeachagencyspends in the

courtroom.Inaddition,eachofthestaffmembersshouldbeinterviewedtoaskthemhow

longtheyspendpreparingforthecourthearingandhowmuchtimetheyspendonother

DTC-relatedtransactions.Finally,dataonthenumberofcourthearingsattendedbyeach

participantintheprogramshouldbecollected.Iftheprogramiscollectingdatesofcourt

hearingsinaprogramdatabase,thenumberofcourthearingsattendediseasilycounted.

Iftheprogramisnotcollectingthisinformation,theevaluatorcanestimatethenumberof

courthearingsbasedonprogrampolicyon the frequencyofcourthearings required for

participants(althoughthisestimatewilllikelybelessaccurate).

Step 5(determiningthecostoftheresources)iscalculatedbydeterminingstaffsalaries

(howmuchiseachstaffmemberpaidforhisorhertime)andreviewingagencybudgetsfor

expendituresrelatedtoDTCactivities,aswellastodeterminestaffbenefitratesandany

overheadratestheagencymayhave.Theseratesaregenerallyexpressedasapercentage

ofstaffsalaries.

131

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Step 6(calculatingthecostresults)mostlyinvolvessimplemultiplicationandaddition.

Foracourthearing(oneof themostcomplicatedtransactions,as it involveseveryteam

memberandagencythatcontributesresources),thesalaryofeachstaffmemberismulti-

pliedbythepercentageoftimespentbothpreparingforandattendingthecourtsession.

Thecostforeachstaffmemberisthenmultipliedbytheoverheadrateandbenefitratefor

thatagency.The“fullyloaded”costsforeachstaffmemberarethenaddedtogetherfor

thesumtotalcostofthecourtappearancetransaction.Thistotalcostcanbedividedby

thenumberofparticipantsthatattendacourtsessiontodeterminethecostofaDTC

appearanceperparticipant.Followingisasamplecalculation.

Ifajudgeworks40hoursperweek,andspends10hoursperweekofthattimepreparing

forDTCstatusreviewhearings(e.g.,gatheringinformationfromtheDTCteam,participating

inpre-courtstaffingmeetings)andparticipatingintheDTChearingsthemselves,thenthe

judgespends25percentofhisorhertimeonpreparingfororparticipatinginDTChearings.

IfthejudgeearnsUS$100,000peryear,thenUS$25,000(US$100,000X.25)goestoward

statusreviewhearingseachyear.DividingUS$25,000bythe52weeksinayearresultsina

costofUS$480.77perweek.IfaDTCprogramsees50participantsinstatusreviewhearings

eachweek,thendividingUS$480.77by50resultsinacostofUS$9.62perparticipantper

hearingforthejudge’stime.

Toachievethe“fullyloaded”cost,itisnecessarytoaddthecostofanyfringebenefits

earnedbythejudge(e.g.,thejudgemayearnanadditional20percentofhisorhersalary

invacationtime,retirementpayments,andhealthcarebenefits)andaddthecostofany

overheadpaidbythecourt(e.g.,afivepercentoverheadratetoheatandlightthecourt-

housewherethejudgeworks,etc.).So,thefullyloadedcostforthejudgetoparticipatein

asingleparticipant’scourthearingwouldaddthisadditional20percentandfivepercentto

theUS$9.62,resultinginafullyloadedunitcostofacourthearingforthejudgeofUS$12.03

perDTCparticipant.

132

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

ThisexerciseisrepeatedforeveryDTCteammemberthatparticipatesinstatusreview

hearings,andthefully loadedcostsforallteammembersaresummedtodeterminethe

totalunitcostpercourthearingperparticipant.Finally,tofindthecostofstatusreview

hearingsperparticipantthroughoutthelengthoftheprogram,thisunitcostismultipliedby

thenumberofhearingsattendedbyeachparticipant(seeTable4).

Amajoradvantageofthisapproachisthatitoffersathoroughassessmentofboththe

investmentcostsofDTCforeachtransactionbyeachparticipatingagencyororganizationin

thesystemaswellastheavoidedcostsduetothebenefitoftheoutcomestoeachagency

ororganization.Forexample,Tables4through6belowdemonstratethekindsofcostsand

savingsDTCshaveshownusingtheTICAapproach.

Investment costs

Table4providestheunitcostpertransaction(e.g.,eachcourtappearanceforaDTC

participantcostsUS$83.62)andtheaveragenumberofeachtypeoftransactioneventper

DTCparticipant(e.g.,onaverage,participantsinthisprogramhadalittleover23appear-

ancesbeforethejudge).Thetotalcostpertransactionistheunitcost(US$83.62)timesthe

numberoftransactionevents(23courtappearances),whichequalsatotaltransactioncost

forcourtappearancesofUS$1,945perparticipant.ThecostoftheDTCprogramisthenthe

sumofthetotalcostsofalltransactions.

133

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Table 4. Program Costs per Participant

Transaction Unit costAvg. # of transactions/events

per DC participants

Perperson

Avg. Cost per DC participant

Perperson

Drug Treatment Court Appearances US$83.62 US$23.26 US$1,945

Case Management Days 5.83 494.37 2,882

Outpatient and Residential Treatment Months

746.74 12.00 8,961

UA Drug Tests 7.00 53.28 373

Jail Sanction Days 46.85 12.48 585

Jail Days While Awaiting Residential Treatment4 46.85 20.42 957

TOTAL US$15,703

Theunitcostmultipliedbythenumberofeventsperpersonresultsinthecostforeach

transaction during the course of the program. In this example,when the costs of the

transactionsaresummed,theresultisatotalprogramcostperparticipantofUS$15,703.

Similarly,thesesamesteps(TICASteps1-6)canbeperformedforacomparisongroup.

(Thisisacomparisongroupthatwouldbeusedintheoutcomeevaluationandthisiswhy

acost-benefitanalysisrequiresavalidoutcomestudy).Forexample,usingprocessevalua-

tionmethods,anevaluatorcandeterminetheprocessthatoccursforaDTC-eligiblecourt

caseforindividualswhodidnotparticipateinDTC,determinethetransactionsthatoccur

throughthatprocess,whatagenciesareinvolved,etc.Then,thedifferencebetweenthe

costoftraditionalcaseprocessingandthecostofparticipatinginDTC(e.g.,thecostofDTC

minusthecostoftraditionalcaseprocessing)resultsinthecostofrunningaDTCoverand

4 Whenaresidentialtreatmentbedisnotavailable,thisprogram’sparticipantsaresenttojailuntilabedopensup.

134

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

above the cost of traditional case processing (i.e., the cost the systemwould havepaid

forthatcaseanyway,iftherehadbeennoDTC).Itisimportanttonotethatthisrequires

goodadministrativedatacollectiononthecomparisongroupaswellasonDTCparticipants,

whichdoesnotalwaysoccur.

Outcome costs

Todeterminetheoutcomecostsandsavings,Steps1through6oftheTICAapproach

areperformedforeachoutcometransactionof interest(e.g.,thecostofanewarrest,a

newcourtcase,daysspent incarcerated). Thecostperoutcometransaction is summed

acrossalltransactionsforboththeDTCparticipantsandforamatchedcomparisongroup.

Forexample,Table5showstheoutcometransactionsmeasuredinthesameDTCasTable

4.Theoutcometransactionsincludedinthestudywerearrests,courtcases,daysonpro-

bation,daysinjail,anddaysinprison.Theseoutcomesweremeasuredduringathree-year

periodfromthetimeofDTCentryforeachparticipantinthestudy.Table5providesthe

unitcostpertransaction(e.g.,thecostofasinglearrestisUS$129.47),andthetotalcostof

eachtransactionperDTCparticipantsandpercomparisongroupmember(e.g.,theaverage

numberofarrestsperpersonin3yearswasmultipliedbytheunitcostofUS$129.47for

atotalcostofUS$66perparticipant,revealingthattherewasanaverageoflessthanone

rearrestperparticipantduringthatperiod).Theright-handcolumnprovidesthedifference

incostbetweenthetwogroups(thecostofthecomparisongroupminusthecostofthe

DTCgroup).Whenthisdifferenceispositive,thisisconsideredacost“savings.”Thesumof

thesavingsacrosstransactionsisthetotalsavingsperparticipantduetoparticipationinthe

DTCprogram.InTable5,thetotalsavingsisshowntobeUS$23,957perDTCparticipant.

135

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Table 5. Recidivism (Outcome)Costs per Participant Over 3 Years

Transaction Unit cost

Drug court participants

Per person

(n=103)

Comparison group

Per person

(n=358)

Cost difference

Comparison minus drug court

Rearrests US$129.47 US$66 US$388 US$322

Criminal District Court Cases 1,448.02 738 4,344 3,606

Probation Days 3.15 559 1,053 494

Jail Days 46.85 1,849 5,100 3,251

Prison Days 49.93 4,038 20,322 16,284

TOTAL US$7,250 US431,207 US$23,957

Cost-benefit ratio

This table indicates that the cost for those processed through the DTC per person

wasUS$23,957 less than standard court processing. It also allows calculation of a

cost-to-benefitratio.InthecaseofthisDTC,whenthecostoftheprograminvestmentper

participant(US$15,703)isassessedagainsttheoutcomesavingsperparticipant(US$23,957),

(i.e.US$23,957dividedbyUS$15,703)thecost-benefitratiocomesto1:1.5.Thatis,for

everydollarspentontheprogram,thereisareturnofUS$1.50.

Inthisexample,thecost-benefitratiomayactuallybeanunderestimate,sincethecost

oftraditionalcourtforthecomparisongroupwasnotincludedinthisexample.Whenthein-

vestmentinatraditionalcourtcaseisincluded,thetotalinvestmentcostusedforthisratio

wouldthenbeforeverydollarspentover and above thecostoftraditionalcourt.So,ifthe

costoftraditionalcourtwasUS$10,000percase,the“net”investmentwouldbeUS$15,703

minusUS$10,000foraninvestmentofUS$5,703perparticipantintheprogramoverand

136

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

abovethecostoftraditionalcourt.Thecost-benefitratiowouldthenbe1:4.2—thatis,for

everydollarspentover and above thecostoftraditionalcourt,thereisareturnofUS$4.20.

Inaddition,becausetheTICAapproachmeasurestheresources(e.g.,eachstaffmem-

ber)contributedbyeachagency involved intheprogram, it ispossibletodeterminethe

investmentandsavingsforeachagency.Table6showsanexampleofthecostorsavings

incurredbyeachagencyduetothecostofoutcomesforDTCparticipantscomparedtoa

traditionalcourtsystem.

Table 6. Recidivism (Outcome) Costs per Participant by Agency over three years

AgencyDrug Court

outcome costs per participant

Comparison group outcome costs per individual

Difference/ savings

per individual

Criminal District Court US$161 US$375 US$214

District Attorney 194 453 259

Appointed Defense Attorney 383 895 512

Department of Criminal Justice 4,000 4,240 240

Law Enforcement 1,915 5,249 3,334

Adult Probation 559 1,037 478

TOTAL US$7,212 US$12,249 US$5,037

Table6demonstratesthateveryagencyinvolvedintheprogramrealizessomesavings

duetoanoffender’sparticipationintheDTCprogram.Inthisexample,lawenforcement

realizesthemostsavingsduetodecreasedincarcerationcosts.

137

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Conclusion – Cost Analyses

Thetypeofcostanalysisapproachdependstoalargeextentonthedataavailableand

thepolicyquestionsbeingaddressed,aswellasthebudgetavailabletoperformthecost

analysis.

�Program-cost analysiscanbeusefulwhenaprogramhasfewevaluation

resourcesandneedsonlyaminimalassessmentofprogramcosts.

�Cost-allocation analysis isvitalinsituationssuchasDTCswheremultiple

agenciescontributeresourcesandservices.Nonetheless,whilecost-allocation

analysisallowsyoutocalculateprogramandunitcosts,itcannotdetermine

whethertheprogramisanefficientuseofresources.

�Cost-effectiveness analysis isusefulwhenonehasclearpolicygoalsto

producethesameoutcomes,andoutcomedataforalltheoptions,butdoesnot

needtoplaceamonetaryvalueontheoutcomes.

�Cost-benefit analysis (suchastheTICAapproach)isusefulwhenoneneeds

todeterminewhetheraprogramactuallysavesmoneyandwantstocompare

theinvestmentcoststothesavings(oravoidedcosts)thatresultfrompositive

outcomes.

138

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Apart fromrelativelysimpletypesofanalyses, it isoftenadvisabletorelyonexpe-

riencedevaluators toperformappropriatestatistical tests,avoidcommonanalyticmis-

takes, and interpret thefindings. If thedatahavebeen recordedproperly and stored

intherightformat, itshouldnottakelongforanevaluatortoanalyzetheresults. The

bigquestion,however, ishowDTCstaffmembers—whohave typicallybeen trainedas

lawyers,clinicians,orcriminal justiceprofessionals—can identifycompetentevaluation

consultants.

Thefirststep inselectingtherightteamofevaluators istoreviewtheirpriorevalua-

tionreports,especiallyanyevaluationsrelatedtoDTCsorotherproblem-solvingcourtpro-

grams.FamiliaritywithhowDTCsoperateisoftenessentialforknowingwhatperformance

indicators,mediator variables,moderator variables andoutcomes toexamine. It is also

importantforknowinghowtointerprettheresultsanddescribethefindingsinapractical

andunderstandablewaytoDTCstaffandtreatmentprofessionals.

When reviewingpriorevaluation reports, it isuseful to consider thequestions listed

below. Thesearenotpresented inorderof importance,and somequestionsmightnot

berelevantforallevaluations. Forexample, if theevaluatorsdidnotcontrastoutcomes

againstthoseofacomparisonsample,thentheremightbenoneedtocontrolforbaseline

differencesbetweenthegroups.However,undersuchcircumstancestheevaluatorswould

beextremelylimitedintermsoftheconclusionstheycoulddrawfromthestudy.Lacking

acomparisonsample,itwouldnotbepossibletoconcludehowwelltheDTCperformedin

comparisontootherprograms.Iftheevaluatorswentbeyondthedatainmakingsuchan

interpretation,thatmightbodepoorlyfortheircompetenceasresearchers.

�Didtheevaluationteamanalyzeimportantandrelevantperformanceindicators?

�Didtheyproperlyaccountformissingdata?

SELECTING STATISTICAL CONSULTANTS

139

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

�Didtheyexamineoutcomesaccordingtoparticipants’riskfactorsormoderator

variables?

�Didtheyidentifythebesttargetpopulationfortheprogrambyexamininginteraction

effectsratherthanmerelyexaminingsignificantpredictorsofoutcomes?

�Didtheyruleoutpotentialconfoundingvariablesbeforeidentifyingdisparate

impactsbygender,race,ethnicorculturalgroups?

�Weretheanalysesconductedonanintent-to-treatbasis?

�Didtheyselectanunbiasedcomparisonsample?

�Didtheycheckforbaselinedifferencesbetweenthegroupsatentryand

statisticallycontrolforthemwherenecessary?

�Didtheyuseareasonablefollow-upwindowgiventheoutcomevariablebeing

measured?Forexample,didtheyexaminerecidivismoveralongenoughtime

periodfornewarrestsorconvictionstobeexpectedtooccur?

�Didtheystarttheclockrunningatanequivalenttimeforallparticipants?For

example,didtheystartitrunningatthepointofentryintoeithertheDTCoran

alternativedispositionsuchasprobation?

�Didtheystatisticallycontrol,whererelevant,fordaysatriskanddaysatlibertyin

thecommunity?

�Didtheytransformthedataorusenonparametrictestsforinfrequentevents?

�Weretheinterpretationsofthefindingsdefensiblegiventhelimitationsofthe

researchdesign?

Finally,themostimportantquestionofallis:Did the evaluators’ interpretations of the

findings make sense, and did they point to concrete actions the DTC could take to improve

its performance and enhance outcomes? Theprimarygoalofaprogramevaluation isto

improveoutcomesandcontinuouslymoveforward.Ifanevaluationreportsimplystates

whetherornottheprogramworkedinthepastbutdoessuggestshowitcouldworkbetter

inthefuture,itisofverylimitedutility.

140

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Thereaderhasundoubtedlyfoundthepreviousdiscussionsconcerningsuchtopicsas

participantcharacteristics,performance indicators,attitudinalvariables,mediationanaly-

ses, and cost analyses extremelydaunting, especiallywhen issuesof date-stamping and

theproblemofthemissingdenominatorareconsidered.Theunfortunaterealityisthatfew

DTCprogramscollectthenecessaryinformationforavalidevaluationinareliableorusable

manner.Whenprogramsdocapturetheappropriatedataelements,theyrarelyaccountfor

missedappointments,connecteventstothedatesonwhichtheyoccurredorweresup-

posedtohaveoccurred,orenterthedataintoanelectronicformatthatpermitsstatistical

analyses. Inmany instances,evaluatorsare required toextract information fromwritten

recordsorunwieldystatisticalspreadsheetswithlittlerecourseforreconcilinginconsisten-

ciesinthedataoraccountingformissingentries.Theresultisthatmanyevaluationsare

completedmonthsoryearsafterthefact—whentheresultsmaynolongerreflectwhatis

occurringintheprogram—andtheremaybesomanygapsorcaveatsinthedatathatthe

conclusionswhichcanbedrawnaretentativeatbest.

Date-stampingcanbeparticularlyunwieldyifevaluatorsusetraditionalformsofsta-

tisticalspreadsheets,suchasExcel.Mostspreadsheetsaretwo-dimensional,meaningthey

arecomprisedofrowsandcolumns.If,forexample,participants’namesarelistedbyrows

andtheservicesthatareprovidedarelistedbycolumns,howdoesoneaccountforthedates

onwhichtheservicesweredelivered?Itisusuallynecessarytocreateseparatecolumns

for each serviceon eachdate, resulting in hundreds of columns. Fortunately, new-

ergenerationsofdata-entrysystemsmayautomaticallydate-stampentries.Forexample,

data-entryscreensmightappear likeaprofessional’sappointmentcalendar. Information

canbeenteredontheappropriatedayinthecalendarandisautomaticallydate-stamped

foranalysis.DTCprogramsareencouragedtousemanagementinformationsystems(MISs)

MANAGEMENT INFORMATION SYSTEMS

141

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

thatautomaticallydate-stampentries.

Itcannotbestressedenoughthatthesecrettovalid,timely,andcost-efficientprogram

evaluationsdependsontheselectionofasuitableMIS.ThecostofpurchasingausefulMIS

willbeoffsetmanytimesoverbyprovidinggreaterefficienciesinoperationsandyielding

thetypeofperformancefeedbackthatisnecessarytocontinuouslyimproveandfine-tune

one’sDTCprogram.

SomeoftheolderandlesssophisticatedMISscanbeobtainedfreeofcharge.Forex-

ample,the“BuffaloSystem”—sonamedbecauseitwasdevelopedinaDTCinBuffalo,New

York,intheUnitedStates—canbeobtainedforfreebycontactingtheNADCPintheUnited

States. NewerMISsmustbepurchased,but theyare alsomore likely tobeweb-based

andcan,therefore,beaccessedsimultaneouslybymultipleprograms.Havingmultiplepro-

gramsusethesameMIS,eachwithitsownsecuredandencryptedaccess,canspreadthe

costsoftheMISacrossmultipleusersorcountries.TheAppendixprovidesinformationon

howtoaccesstheBuffaloSystemandotherproprietaryMISproducts.

Recommended MIS Features

NewergenerationsofMISsarecapableofstreamliningprogramevaluationsandre-

ducingtheburdenonstaffmembersandparticipants.Whereitisfeasible,itisrecommended

thatDTCprogramsselectMISswiththefollowingcharacteristics:

Web-Enabled. Staffmembers inDTCsareusuallyemployedbydifferentagencies

such as the court, probation department or treatment program, and theymay have

officesinseverallocations.Theymayalsoberequiredtovisitparticipantsinjail,residential

treatmentfacilitiesorattheirhomestoconductassessmentsordeliverservices.Thisre-

quiresstaffmemberstohaveaccesstotheMISwhiletheyaretravelingandfrommultiple

locations.IftheMISisweb-enabled,itcanbeaccessedfromanylocationthathasinternet

access,includingpersonalpalmdevicesorlaptopcomputers.

Security Protected. The data should be stored and transferred using indus-

142

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

try-standard128-bitSSLencryptionorbetter.Itisalsonecessarytoregulatestaffmember

accesstotheinformationbasedontheirjoblevelsanddescriptions.Onlyundercarefully

specifiedcircumstancesshouldonestaffmemberbeabletoalterdatathatwasenteredby

anotherstaffmember.Forexample,thejudgeshouldnotbeabletoalterinformationthat

wasenteredbythetreatmentprovider.Thejudgemight,however,haveread-onlyaccessto

certaininformationenteredbythetreatmentprovider,suchasaparticipant’sattendance

rateatcounselingsessions. Theauthorizedlevelofaccessforeachstaffmembershould

bespecifiedbyanMISAdministratorandcorrespondtothatstaffmember’spasswordand

username.

Less is More.Staffmembersshouldonlyberequiredtoviewdata-entryscreensthat

arerelevanttotheirjobs.Forexample,atreatmentproviderordinarilyshouldnotbefaced

withdata-entryscreensrelatingtoprobationcontactsorcourthearings.Thetreatment

providermight be permitted to view summary reports on probation contacts or court

hearingsbutshouldnotberequiredtoscrollthroughthatmaterial if it isnotdirectly

relevanttothetreatmentprovider’sduties.

Need to Know.DTCprofessionalshavearighttoknowwhytheyarebeingasked

tocollect informationandshouldavoidduplicationofeffort. Ifthere isnoobviousand

empiricallydefensiblereasonwhyparticular informationisbeingcollected,thenperhaps

itisunnecessarytocollectthatinformation.Redundanciesshouldalsobeeliminated.For

example,onceaparticipant’sageisenteredinonedata-entryscreen,itshouldautomatically

beenteredor“cross-walked”intotherespectivefieldsinotherscreens.

Minimal Burden. Itshouldordinarilyrequirenomorethantwotothreeminutes

toenteralldataelements thatare required foragivenparticipantduringagivenweek.

The data-entry screens should also be intuitive and easy to use.Most professionals are

accustomedtousingtheInternettopaytheirbills,purchasegoodsandservices,orgather

information.Agooddealofefforthasgoneintodevelopingcommercialwebsitesthatare

intuitiveandsimpletouse.TheMISfortheDTCshouldhaveasimilarlyintuitivedesign.The

143

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

morecomplex,confusing,and/ortime-consumingtheeffortrequiredtoenterdataproves

tobe,thelesslikelystaffwillenterdatacorrectlyandappropriately.

Automatic Date Stamping. Aswasmentionedpreviously,somedata-entryscreens

appearlikeaprofessional’sappointmentcalendar.Informationisenteredontheappro-

priatedateinthecalendarandisautomaticallydate-stampedforanalysis.

Reminders/Ticklers for Missing Data. ModernMISsmayroutinelyremindor“tick-

le”userstoentermissingorincompletedata.Forexample,ifaprobationofficeraccesses

theMIStoenterattendanceinformationfortheweekofOctober15,buthasnotentered

theattendanceinformationforthepreviousweekofOctober8,thesystemshouldalert

theprobationofficeraboutthemissingdata.Suchremindersorticklersmakeitlesslikely

thatsomuchtimewillpassthatitbecomesdifficulttoreliablyreconstructtheeventsfrom

memoryorhandwrittenrecords.

Flexible Input Screens.Itisoftennecessarytoaddnewitems,deleteor“grayout”

unwanteditems,orchangethewordingofitems.Forexample,aDTCmightcontractwith

anewtreatmentprogramtoprovidementalhealth services.Thismight requireanew

data-entry screen to be addedwith items pertaining to participants’ attendance at

mentalhealthcounselingsessions.Itshouldbepossibletoaddnewitems,deleteitems,

andchangethewordingofitemsinnomorethananhour.

Hidden Database. DTC professionals should not be required to wade through

columnsofnumericdata. Thedatashouldbestoredbehindthescenesinananalyzable

formatthatpermitsimmediatestatisticalanalysisandqueries,andallowseasyandquick

visualizationthroughaninterfacethatintuitivelymakessensetotheuser.Thedatashould

alsobestoredinspecifiedfieldsthatcanbeselectedreadilyandenteredinstatisticalmodels.

Longitudinal Database.Undernocircumstanceshouldthesystemoverwriteprevi-

ousdata.Forexample,ifaparticipantwasunemployedwhenheorsheenteredtheDTC,

andthenobtainedajobafewmonthslater,theparticipant’semploymentstatusshouldnot

beoverwritten.Doingsowouldmakeitdifficultorimpossibletodeterminelaterwhether

144

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

theparticipantobtainedajobwhileheorshewasintheDTC,orwhethertheparticipant

enteredtheprogramwithajobtobeginwith.Moreover,itwouldbedifficulttodetermine

howlongittooktheparticipanttofindajob.Thedatabaseshouldbearrangedlongitudi-

nally,meaningthatneweventsareappendedalongsideolderevents. If thedatabase is

arrangedlongitudinallyandtheentriesaredate-stamped,itwillbepossibletodetermine

whetherparticipants’employmentstatuschangedovertimeandhowlongittookforthe

changestooccur.

Descriptive Reports. ThereisnolimittotheresearchquestionsthatDTCstaffmem-

bersorstakeholdersmightask.However,therearealimitednumberofbasicdescriptive

analyses thatmostprogramswillwant to conduct. Forexample, virtuallyeveryDTCwill

want to know its graduation rate, re-arrest rate, attendance rate in counseling sessions,

andproportionofalcohol-anddrug-positivetests.Inaddition,manyprogramswillwant

toknowwhethertheseoutcomesdifferbetweenmalesandfemalesorbetweenracialor

ethnicgroups.Itispossibletowritestatisticalsyntaxsothatthesecommonanalysescan

beperformedatthepushofabutton.Formoresophisticatedresearchquestions,itmaybe

necessarytosendthedatatoastatisticianforanalyses;however,formanyroutineques-

tionsitshouldbepossibletogeneratereportsnearlyinstantly.Inlinewiththeadagethat

“apicture iswortha thousandwords,” the reports shouldalso includeeasy-to-interpret

graphs,tablesandpiecharts.

Continuous Performance Feedback.ThemostimportantreasonforevaluatingaDTC

istoimproveitsoperationsandoutcomes.Ifaninterventionisnotworking,itisimportant

tolearnthisfactquicklywhilethereisstilltimetoadjusttheparticipant’streatmentplan.

ThisrequirestheMIStocontinuouslymonitorthedataandissueautomatedalertstostaff

memberswheneveraparticularactionmightbecalledfor.Forexample,thesystemmight

automaticallyalertstaffwheneveraparticipantmissedtwocounselingsessionsinarowor

providedtwodrug-positiveurinespecimens.Thiswouldpromptthestafftodiscussthepar-

ticipantatthenextstaffmeetinganddecideonasuitableresponse.Researchindicatesthat

145

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

generatingautomatedfeedbackinthismannercansubstantiallyimproveaDTC’soperations

andparticipantoutcomes(Marloweetal.,2008b;Rempel,2007).

User Accountability Reports. Oneofthebiggestthreatstoavalidprogramevalua-

tionisthefailureofmanystaffmemberstoenterdatainatimelymanner.TheMISshould

generateuseraccountabilityreportswhichindicatehowlongittookforthedatatobeen-

teredaftertherelevanteventsoccurred.Forexample,supervisorsshouldbemadeaware

thatacounselorhasbeenenteringattendanceinformationanaverageoffourweeksafter

thesessionswereheld,orthatdrugtestresultsarebeinglistedas“pending”bythelabfor

anaverageofmorethantwoweeks.Thiswouldenablethesupervisortointervenerapidly

toimprovethequalityoftheevaluationandtheeffectivenessoftheDTC.

Attheriskofrepetition, if an MIS is easy to use, collects the essential performance

indicators, and stores the data in an analyzable format, the likelihood of completing a

successful and valid evaluation is high.AndthecostofpurchasingsuchanMISwillbe

offsetmanytimesoverbythefactthatthereisnolongeraneedforevaluatorstospend

hundredsofhoursattempting(oftenwithlimitedsuccess)toextractusableinformation

from written charts or spreadsheets. If the necessary information can be handed to

researchersinaproperformat,usefulfindingsshouldbeobtainableinashortperiodof

timeandataminimalcost.

Ofcourse,ifaDTCdoesnothaveadequateresourcestopurchaseanMISwiththeabove

characteristics,anyeffortatevaluation isbetter thannoeffort.TheDTCcoulduse the

BuffaloSystemdescribedearlier,whichisfreeofcharge,ordevelopwrittenchecklistsor

dataspreadsheetstocollectinformation.EvenanExcelspreadsheet,ifconfiguredandused

correctly,canhandleavarietyofdatacollectiontasks.Gettingstartedevaluatingone’sDTC

isthehardeststep.Onceevaluationactivitiesarebegun,DTCstaffmembersareapttoiden-

tifybetterwaysovertimetocollectusefulinformationandexaminetheservicesprovided

andimpactsproducedbytheirprogram.

146

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Much of the information that is required to evaluate a DTC is likely to be in the

possessionoftheprogram.Forexample,dataonparticipants’drugtestsresults,treatment

attendance informationandgraduationratesshouldbereadilyavailable incourtrecords

orparticipants’treatmentrecords. However,toconstructasuitablecomparisonsample,

anevaluatorwillalsoneedinformationaboutsimilarindividualswhowerenotenrolledin

theDTC,suchasprobationers.Forexample,theevaluatorwillneedinformationaboutrisk

factorsoftheprobationers,includingtheircriminalrecordsortreatmenthistories.Without

thisinformation,itwillnotbepossibletoconstructavalidmatched-comparisonsampleor

toperformavalidpropensity-scoreanalysis.

Evaluatorswillalsoneeddataonnewarrests,newconvictionsand incarcerationsfor

both theDTC and comparison samples. This recidivismdatawill often not be in the

possessionof theDTCprogram,but rathermaybe collectedand storedbyanother

governmentagency,suchasthepolice,departmentofcorrections,oradministrative

officeofthecourtsystem.Thiswillrequireevaluatorstoobtaininformationfromdata-

basesthatmaynotbewithintheircontrol,whichtheymaynotbeauthorizedtoaccess,

andwhichmaybeunfamiliartothem.

Oftentheremaybetechnologicalorattitudinalbarrierstoaccessingotherdatabases.

Lawsandregulationsconcerningthesharingofcriminalrecordsmaybecontradictoryor

difficultto interpret insomecountries.This can lead toparalysison thepartof some

agencies for fear of committing a breach of privacy or other legal protection. For

technologicalreasons,informationoftencannotbetransferredreadilyfromonedatabase

to another due to inconsistent data definitions, incompatible hardware or software,

or proprietary technologies that cannot interface with one another. Important terms

such as risk assessment may also have different meanings for agencies with different

ACCESSING ADMINISTRATIVE DATABASES

147

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

cultures. Toaprobationofficer,ariskassessmentmayrefertothelikelihoodofcriminal

reoffending,whereastoatreatmentprofessionalitmightrefertothelikelihoodofleaving

treatment prematurely or relapsing to substance use or mental illness. Agencies may

beunderstandably reluctant to share this informationoutofa concern that itmightbe

misinterpreted.

Thesebarriers,althoughdaunting,arenotinsurmountable.Evaluatorshavefoundeffec-

tiveandethicalwaystoaccesscriticallyimportantinformationfromgovernmentdatabases

withoutviolatinglegalrequirements,ethicalmandatesoreffectiveclinicalpractice(Treat-

mentResearchInstitute,2012).

Reaching Consensus

Thefirststeptosharingsensitiveinformationisreachingaconsensusaboutwhatin-

formationisavailable,whichdataelementsmaybeshared,andunderwhatconditionsthe

datamaybeshared.Representativesfromtheinterestedagenciesmustcometoamutual

understandingaboutthenatureofthedatathatiscollected,thedefinitionsandlimitations

ofthedata,andhowthedataisstoredelectronically.Fromthere,theagenciesmustex-

ecutedata-sharingagreementsandotherstandardsthatsatisfythelegalandethicalcon-

cernsofallinterestedparties.

Oneresourcethatmayassistagencies toreachthisconsensus is theNational Infor-

mationExchangeModel(NIEM).5TheNIEMisconsideredthestandardforinformationex-

changeamongcriminaljusticeagenciesintheUnitedStates.Ithasalsobeenappliedinter-

nationallyandisavailabletoanypublic,private,ornon-profitagency.TheNIEMoffersbasic

data-sharingtools,standardsandprocedurestohelpagenciessharesensitiveinformation.

Forexample, theNIEMcanprovidesamplesofdatadictionariescontainingagreed-upon

definitionsandrecommendedformatsforcriminaljusticedata.Thedatadictionariescan

help toensure that informationcarriesa consistentmeaningacrossagencies.TheNIEM

5 https://www.niem.gov/aboutniem/Pages/niem.aspx

148

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

mayalsoprovidesampledata-sharingagreementsandstandardsforthetransferofdata.

EvaluatorsandpolicymakersfromOASmemberstatescanusethesampledatadictio-

nariesanddata-sharingagreementsasstartingpointsforreachingagreementabouthow

datamightbedefined,formattedorsharedintheirownDTCprogramevaluations.Legal

andethicalmandateswill,ofcourse,differbetweennations,buttheessentialprinciplesand

practicesofeffectivedata-sharingarrangementshavecommonelementsthatarelikelyto

applytoawiderangeofevaluationcontexts.

After agencies have reached an agreement about how data should be defined and

shared,theNIEMcan,ifdesired,developcomputersyntaxthatallowsforthestandardiza-

tionandexchangeofthedatabetweenMISs.Thisdoesnotrequiretheagenciestochange

theirownMISordata-storagemethods. Rather,usingwhat issometimesreferredtoas

middlewaretheNIEMcandevelopelectronicconduitsforthetransferofinformationbe-

tweendatabases.Thisprocessislikelytobemorecomplicatedorcostlythaniswarranted

formanyDTCpilotprogramsinOASmemberstates,butitmaybecomemoreusefuland

relevantasdozensofDTCsaredevelopedinmultiplenationsovertime.

Linking Records

ItisessentialforrecordstocorrespondtothesameindividualacrossdifferentMISsor

databases. Ideally, unique identifying information shouldbeused to link records across

agenciesorsystems.Forexample,somecountriessuchasBarbadoshaveanationalregistry

numberthatcorrespondstoeachcitizen.

Alternatively,identifyingnumbersmaybeassignedtoindividualswhentheycontactthe

criminaljusticesystem.Someofthesenumbersmaycorrespondtotheperson,andthus

thesamenumbermaybeassignedtodifferentcasesoroffensesinvolvingthesameindivid-

ual.Othernumbersmaycorrespondtothecase,andthusthesamenumberwouldfollow

149

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

thepersonfromarrestthroughsentencingonthesameoffensebutwouldnotbeassigned

tosubsequentoffenses. Dependingonthecountry, these identifyingnumbersmightbe

usefulfor linkingcasesacrossagencieswithinthecriminal justicesystem,suchas linking

casesbetweenthecourtandprobationdepartment.Often,however,theycannotbeused

tolinkcriminaljusticerecordswithrecordsfromothertreatmentorsocialserviceagencies.

Ifuniqueidentifyinginformationisnotavailable,therearealternativeproceduresthat

evaluatorscanusetolinkrecordswithvaryingdegreesofcertaintythattherecordsrepre-

sentthesameindividual.Thereareseveralprobabilistic record linkagemethodswhichare

availablefreelyinthepublicdomain,suchas“LinkPlus”or“TheLinkKing”(Campbelletal.,

2008).Theseprobabilisticmethodslinkcasesfromdifferentdatasetsusingacombination

ofavailabledataelements,suchasbirthdatesandfirstandlastnames. Thelinkscanbe

accomplishedtoaspecifieddegreeofconfidence;forexample,with80percentconfidence

thatthecasesrepresentthesameperson.Thelevelofconfidencethatcanbeplacedinthe

linksisbasedonsuchfactorsasthenumberofpersonsinthedatabaseandthenumberof

variablesthatareavailabletocreatethematches.

Ofcourse,therewillalwaysbesomedegreeoferrorwhenusingprobabilisticrecord

linkages.However,themagnitudeoftheerrorshouldbedistributedevenlybetweenthe

DTCandcomparisongroups.Therefore,itshouldnotseriouslybiasthequestionofwhich

grouphadlowerrecidivismrates.However,estimationsofthemagnitudeoftheeffectof

theDTConrecidivismcouldbeoffbyseveralpercentagepoints.

Ifallelsefails,bearinmindthateventhemostrestrictivelawsandethicalmandateswill

usuallypermitde-identifieddatatobesharedortransferred.Manyethicsreviewcommittees

permitevaluatorstoassignanonymousresearchidentificationnumberstocases.Afterthecas-

eshavebeenlinkedtorecordsfromadministrativedatabases,theevaluatoragreestodestroy

thelistsconnectingthesubjects’namestotheiridentificationnumbers.Alternatively,theagen-

cythatisstoringtheadministrativedata,suchasthepoliceorprobationdepartment,mightbe

responsiblefordestroyingthelinkstosubjects’namesafterthedatahasbeentransferred.The

150

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

datamaythenbeanalyzedatthegrouplevelintermsofwhetheraparticularcasebelongstothe

DTCorcomparisongroupbutcannotbetracedbacktoanindividualparticipant.

Transferring DataGiventheseriousresourcechallengesthatconfrontmostcountries,data-sharingproce-

duresmuststartwithrelativelysimpleandlow-costmethodswhicharereadilyavailablefor

mostprograms.Forexample,Direct Secure Messaging(DSM)allowsprogramstotransmit

encryptedinformationtoknownandtrustedrecipientsovertheInternet.Similarly,“cloud”

technologymaybeattractivetosmallandunder-resourcedprogramsbecauseitdoesnot

requireMISdepartmentstohavesophisticatedhardware,networkingcapabilities,ormain-

tenancestaff.Finally,portableexternaldrivescanholdsubstantialamountsofinformation

andcanbeusedtophysicallytransferinformationbetweenlocations.

Ethics and Legal Training

AllprofessionalswhoareinvolvedintheevaluationofaDTCorwhohandlesensitive

datashouldfirstcompleteeducationalcoursesonethicalandlegalprotectionsinresearch.

Becauseapplicablelawsmaybedifferentindifferentnations,evaluatorsmightneedto

identifytrainingprogramswithintheirowncountries.Web-basedcoursesonresearchethics

areavailableinseveralcountries.Forexample,theU.S.NationalInstitutesofHealthprovide

web-basedtrainingsonlegalandethicalprotectionsforresearchparticipants.6 Comparable

web-basedprogramsorwritteneducationalmaterialsmaybeavailableinothercountries

aswell.

6http://phrp.nihtraining.com/users/login.php

151

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

MoreresearchhasbeenpublishedgloballyontheeffectsofDTCsthanonvirtually

allothercriminaljusticeandsubstanceusetreatmentprogramscombined.Theresultsof

thisintenselevelofscrutinyhavebeenalmostuniversallypositive.Inaneraofglobaleco-

nomicuncertaintyandshrinkingpublicdollars,DTCsarebeingfundedandexpandedata

steadilyincreasingrate.Therearenowmorethan3,000DTCsandothertreatmentcourts

intheUnitedStates,andnewDTCprogramsarebeingdevelopedorplannedinmorethan

30othercountries.

Theonlyway toexplain this success is tonote that the researchhaspaidoff.By

provingthatDTCsreducecrime,savelives,andsavemoney,evaluatorshavejustifiedthe

programs’worthtoaskepticalaudienceofpolicymakersandthepublicatlarge.Political

andsocialbeliefsmaychangewiththetimesandpetprojectsmayfadeinandoutoffavor,

butscientificproofisageless.Convincinglywordedargumentsandstronglyfeltemotions

mayswaypublicopiniontemporarily,buttheyhaveadifficulttimestandinguptoscientifi-

callyreliableandvalidcontradictoryevidence.Inthefinalanalysis,thefactshaveawayof

winningout.

Itwould be nice to say thatDTCs planned these events from thebeginning, but

theydidnot.ManyoftheearlieststudiesinDTCsviolatedfundamentalprinciplesofgood

science.Forexample,theyexaminedoutcomesonlyforDTCgraduatesorselectedunfairly

disadvantagedcomparisonsamples.IttooknearlytwodecadesfortheDTCfieldtogetit

rightandconductthepropertypesofstudiesthatwererequiredtoprovethatDTCswork.

Thereisnoneedforothercountriestorelivethesetime-consuminglessonsthatDTC

pioneershadto learnthroughpainstakingeffort. OtherOASmemberstatescanbenefit

frommorethanaquartercenturyofexperienceinAustralia,Canada,theUnitedStatesand

afewothernationstoconductproperresearchfromtheoutset.

CLOSING COMMENTS

152

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

ThisisnottosuggestthatothercountriesshoulddevelopandadministertheirDTCs

inthesamemannerasAustralia,CanadaortheUnitedStates.Differentcultureswillfind

differentwaystomeettheneedsoftheirowncitizens.Butthelawsofscienceareuniver-

sal.Therearecorrectwaysandincorrectwaystoevaluateaprogram.Performingflawed

evaluationsdoesnotmerelywastetime,resources,andopportunities.Itcanalsoresultin

lostcredibility,bymakingDTCsseemasiftheyareattemptingtocutcornersormanipulate

thetruth.Itisnotenoughtohaveaworthyconceptifonecannotproveitsworthinafair

andimpartialtest.

ToprofessionalsworkinginDTCprograms,rememberthis:Itiseasytobeagood

salespersonifyouhaveasuperiorproduct.AndDTCsare,indeed,asuperiorproduct.Do

nothesitatetostudythemcarefullyandfollowtheresultswherevertheymayleadyou.The

factsareonyoursideandgoodscientificmethodologywillenableyoutoproveit.

153

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Andrews,D.A.,&Bonta,J.(2010).The Psychology of Criminal Conduct (5thed.)NewProvidence,NJ:LexisNexis.

Ardito,R.B.,&Rabellino,D.(2011).Therapeuticallianceandoutcomeofpsychotherapy:Historicalexcursus,measurements,andprospectsforresearch.Frontiers in Psychology, 2,11.

Banks,D.,&Gottfredson,D.C.(2003).Theeffectsofdrugtreatmentandsupervisionontimetorearrestamongdrugtreatmentcourtparticipants.Journal of Drug Issues, 33,385-412.

Baron,R.M.,&Kenny,D.A.(1986).Themoderator-mediatorvariabledistinctioninsocialpsychologicalresearch–Conceptual,strategic,andstatisticalconsiderations. Journal of Personality & Social Psychology,51,1173–1182.

Belenko,S.,Patapis,N.,&French,M.T.(2005).Economic benefits of drug treatment: A critical review of the evidence for policy makers. MissouriFoundationforHealth,NationalRuralAlcohol&DrugAbuseNetwork.

Bhati,A.S.,Roman,J.K.,&Chalfin,A.(2008).To treat or not to treat: Evidence on the prospects of expanding treatment to drug-involved offenders.Washington,DC:TheUrbanInstitute.

Bonomo,E.(2012).Sanctionable behavior in a felony level drug court: Categorizing noncompliant behavior through a criminal-thinking lens[Master’sthesis].Atlanta:GeorgiaStateUniversity,Dept.ofCriminalJustice.

Bourgon,G.,&Armstrong,B.(2005).Transferringtheprinciplesofeffectivetreatmentintoa“realworld”prisonsetting.Criminal Justice & Behavior, 32,3-25.

Breckenridge,J.F.,Winfree,L.T.,Maupin,J.R.,&Clason,D.L.(2000).Drunkdrivers,DWI“drugcourt”treatment,andrecidivism:Whofails?Justice Research & Policy, 2,87-105.

Burke,K.S.(2010).Justwhatmadedrugcourtssuccessful?New England Journal on Criminal & Civil Confinement, 36,39-58.

Burke,K.,&Leben,S.(2007).Proceduralfairness:Akeyingredientinpublicsatisfaction.Court Review, 44,4-25.

Campbell,K.,Deck,D.,&Krupski,A.(2008).Recordlinkagesoftwareinthepublicdomain:AcomparisonofLinkPlus,TheLinkKing,anda“basic”deterministicalgorithm.Health Informatics Journal, 14(1),5-15.

Carey,S.M.,&Finigan,M.W.(2004).Adetailedcostanalysisinamaturedrugcourtsetting:acost-benefitevaluationoftheMultnomahCountyDrugCourt.Journalof

REFERENCES

154

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

ContemporaryCriminalJustice,20(3)292-338.

Carey,S.M.,Finigan,M.W.,Crumpton,C.D.,&Waller,M.S.(2006).CaliforniaDrugCourts:Outcomes,CostsandPromisingPractices:AnOverviewofPhaseIIinaStatewideStudy.JournalofPsychoactiveDrugs.SARCsupplement3.

Carey,S.M.,Mackin,J.R.,&Finigan,M.W.(2012).Whatworks?The10keycomponentsofDrugCourt:Research-basedbestpractices.Drug Court Review,8,6-42.

Chandler,R.K.,Fletcher,B.W.,&Volkow,N.D.(2009).Treatingdrugabuseandaddictioninthecriminaljusticesystem:Improvingpublichealthandsafety.Journal of the American Medical Association, 301, 183-190.

Cooper,C.S.(2004).Costs associated with the birth of drug and/or alcohol addicted/exposed infants.WashingtonDC:BureauofJusticeAssistance,DrugCourtClearinghouse&TechnicalAssistance Center at American University.

Cooper,C.S.,Franklin,B.,&Mease,T.(2010,April).Establishing drug treatment courts: Strategies, expe-riences and preliminary outcomes. WashingtonDC:JusticeProgramsOffice,SchoolofPublicAffairs,AmericanUniversity.Available at http://www.cicad.oas.org/fortalec-imiento_institucional/dtca/files/Establishing_DTC_%20Strategies_Ex-periences_Preliminary_Outcomes_volume%201.pdf.

Cresswell,L.S.,&Deschenes,E.P.(2001).Minorityandnon-minoritypercep-tionsofdrugcourtprogramseverityandeffectiveness.Journal of Drug Issues, 31,259-291.

Crumpton,D.,Carey,S.M.,&Finigan,M.W.(2004).EnhancingCostAnalysisofDrugCourts:TheTransactionalandInstitutionalCostAnalysisApproach.NPCResearch:Portland,OR

Dane,C.B.(2012).Examining the relationship between procedural justice and recidivism in a jail-based residential substance abuse treatment program [Master’sthesis].GeorgiaStateUniversity,AndrewYoungSchoolofPolicyStudies.

Dannerbeck,A.,Harris,G.,Sundet,P.,&Lloyd,K.(2006).Understandingandrespondingtoracialdifferencesindrugcourtoutcomes.Journal of Ethnicity in Substance Abuse5,1-22.

DeMatteo,D.S.,Marlowe,D.B.,&Festinger,D.S.(2006).Secondarypreventionservicesforclientswhoarelowriskindrugcourt:Aconceptualmodel.Crime & Delinquency, 52,114-134.

Deschenes.E.P.,Ireland,C.,&Kleinpeter,C.B.(2009).Enhancingdrugcourt success. Journal of Offender Rehabilitation, 48,19-36.

Edwards,S.J.L.,Lilford,R.J.,&Hewison,J.(1998).Theethicsofrandomisedcontrolledtrialsfromtheperspectivesofpatients,thepublic,andhealthcareprofessionals.British Medical Journal, 317,1209-1212.

155

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Farole,D.J.,&Cissner,A.B.(2007).Seeingeyetoeye:Participantandstaffperspectivesondrugcourts.InG.Berman,M.Rempel&R.V.Wolf(Eds.),Documenting Results: Research on Problem-Solving Justice (pp.51-73).NewYork:CenterforCourtInnovation.

Festinger,D.S.,Marlowe,D.B.,Lee,P.A.,Kirby,K.C.,Bovasso,G.,&McLellan,A.T.(2002).Statushearingsindrugcourt:Whenmoreislessandlessismore. Drug & Alcohol Dependence, 68,151-157.

Finigan,M.W.(2009).UnderstandingracialdisparitiesinDrugCourts.Drug Court Review, 6,135-142.

Flango,V.E.,&Cheesman,F.L.(2009).EffectivenessoftheSCRAMalcoholmonitoringdevice:Apreliminarytest. Drug Court Review, 6,109-134.

Frazer,M.S.(2006).The impact of the com-munity court model on defendant perceptions of fairness: A case study at the Red Hook Community Justice Center.NewYork:CenterforCourtInnovation.Availableathttp://www.courtinnovation.org/sites/de-fault/files/Procedural_Fairness.pdf.

FundaciónPazCiudadana,MinisteriodeJusticiadeChileandFiscalíadeChile(2014).Tribunales de Tratamiento de Drogas Compendio estadístico 2010, 2011 y 2012.[ebook]SantiagodeChile.Availableat:http://www.pazciudadana.cl/wp-content/up-loads/2015/03/ttd-analisis-estadisti-co.pdf[Accessed16Apr.2018].

FundaciónPazCiudadana/UldaOmarFigueroaOssa(2018).Tribunales de Tratamiento de Drogas: Evaluacion de su impacto en la disminucion de la reincidencia delictual en Chile.

Gallagher,J.R.(2012).Evaluating Drug Court effectiveness and exploring racial disparities in Drug Court outcomes: A mixed methods study [Doctoraldissertation].Arlington:Univ.ofTexas.

Gendreau,P.(1996).Theprinciplesofeffec-tiveinterventionwithoffenders.InA.Harland(Ed.),Choosing correc-tional options that work(pp.117-130).ThousandOaks,CA:Sage.

Gibbs,B.R.,&Wakefield,W.(2014).Theefficacyofenhancedalcohol-usemonitoring:Anexperimentexam-iningtheeffectsofanincreasedsupervisionprotocolinanadultdrugcourt.Drug Court Review, 9(1),1-22.

Goldkamp,J.S.,White,M.D.,&Robinson,J.B.(2002).Anhonestchance:Per-spectivesondrugcourts.Federal Sen-tencing Reporter, 6, 369–372.

Gossop,M.,Tradaka,K.,Stewart,D.,&Witton,J.(2005).Reductionsincriminalconvictionsafteraddictiontreatment:5-yearfollow-up.Drug&AlcoholDependence,79,295-302.

Gottfredson,D.C.,Kearley,B.W.,&Bush-way,S.D.(2008).Substanceuse,drugtreatment,andcrime:Anexaminationofintra-individualvari-ationinadrugcourtpopulation.JournalofDrugIssues,38,601-630.

156

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Gottfredson,D.C.,Kearley,B.W.,Najaka,S.S.,&Rocha,C.M.(2007).Howdrugtreatmentcourtswork:Ananalysisofmediators.JournalofResearchonCrime&Delinquency,44,3-35.

Gottfredson,D.C.,Najaka,S.S.,&Kearley,B.(2003).Effectivenessofdrugtreatmentcourts:Evidencefromarandomizedtrial.Criminology & Public Policy, 2, 171-196.

Gutierrez,L.,&Bourgon,G.(2012).Drugtreatmentcourts:Aquantitativereviewofstudyandtreatmentqual-ity.Justice Research & Policy, 14,47-77.

Harrell,A.,Cavanagh,S.,&Roman,J.(1999).Final report: Findings from the Evalu-ation of the D.C. Superior Court Drug Intervention Program.Washington,DC:TheUrbanInstitute.

Harrell,A.,&Roman,J.(2001).Reducingdruguseandcrimeamongoffend-ers:Theimpactofgraduatedsanc-tions.Journal of Drug Issues, 31, 207-232.

Hawken,A.,&Kleiman,M.(2009).Manag-ingdruginvolvedprobationerswithswiftandcertainsanctions:Eval-uatingHawaii’sHOPE[NCJRSno.229023]. WashingtonDC:NationalInstituteofJustice.Availableathttp://www.ncjrs.gov/pdffiles1/nij/grants/229023.pdf.

Heck,C.(2006).Local drug court research: Navigating performance measures and process evaluations [Mono-graphSeriesNo.6].Alexandria,VA:NationalDrugCourtInstitute.

Henrichson,C.,&Galgano,S.(2013).AGuidetoCalculatingJustice-SystemMarginalCosts.NewYorkVeraInstituteofJustice.https://www.bja.gov/Publications/Vera_Calculating-Justice-System-Marginal-Costs.pdf

Holloway,K.R.,Bennett,T.H.,&Far-rington,D.P.(2006).Theeffective-nessofdrugtreatmentprogramsinreducingcriminalbehavior.Psico-thema,18,620-629.

Huddleston,W.,&Marlowe,D.B.(2011).Painting the current picture: A national report on drug courts and other problem solving court programs in the United States. Al-exandria,VA:NationalDrugCourtInstitute.

Huebner,B.M.,&Cobbina,J.(2007).Theeffectofdruguse,drugtreatmentparticipation,andtreatmentcom-pletiononprobationerrecidivism.Journal of Drug Issues, 37,619-641.

Inciardi,J.,Martin,S.,&Butzin,C.(2004).Five-yearoutcomesoftherapeuticcommunitytreatmentofdrug-in-volvedoffendersafterreleasefromprison. Crime & Delinquency, 50,88-107.

Jones,C.(2011).Intensivejudicialsuper-visionanddrugcourtoutcomes:Interimfindingsfromarandomizedcontrolledtrial.Crime & Justice Bul-letin, 152,1-16.NewSouthWales,Australia:BureauofCrimeStatistics&Research.Available at http://www.courtwise.nsw.gov.au/lawlink/bocsar/ll_bocsar.nsf/vw-Files/CJB152.pdf/$file/CJB152.pdf.

157

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

King,M.S.(2009).The solution-focused judging bench book.Melbourne,Australia:AustralasianInstituteofJudicialAdministration&LegalSer-vicesBoardofVictoria.

Kirk,J.M.(2012).Assessing self determin-ing attitudes and behaviors in court mandated treatment clients [Doc-toraldissertation].OklahomaStateUniversity.

Kleinpeter,C.B.,Brocatao,J.,&Koob,J.J.(2010).Doesdrugtestingdeterdrugcourtparticipantsfromusingdrugsoralcohol?Journal of Of-fender Rehabilitation, 49,434-444.

Landenberger,N.A.,&Lipsey,N.W.(2005).Thepositiveeffectsofcognitive-be-havioralprogramforoffenders:Ameta-analysisoffactorsassociatedwitheffectivetreatment.Journal of Experimental Criminology,1(4),451-476.

Latimer,J.,Dowden,C.,&Muise,D.(2005).Theeffectivenessofrestorativejus-ticepractices:Ameta-analysis.The Prison Journal, 85,127-144.

Latimer,J.,Morton-Bourgon,K.,&Chre-tien,J.(2006).A meta-analytic ex-amination of drug treatment courts: Do they reduce recidivism?CanadaDept.ofJustice,Research&Statis-ticsDivision.

Leukefeld,C.,Webster,J.M.,Staton-Tin-dall,M.,&Duvall,J.(2007).Em-ploymentandworkamongdrugcourtclients:12-monthoutcomes.Substance Use & Misuse, 42,1109-1126.

Lindquist,C.H.,Krebs,C.P.,&Lattimore,P.K.(2006).Sanctionsandrewardsindrugcourtprograms:Implemen-tation,perceivedefficacy,anddeci-sion making. Journal of Drug Issues, 36,119-146.

Lowenkamp,C.T.,Flores,A.W.,Holsinger,A.M.,Makarios,M.D.,&Latessa,E.J.(2010).Intensivesupervisionpro-grams:Doesprogramphilosophyandtheprinciplesofeffectiveinter-ventionmatter?Journal of Criminal Justice, 38,368-375.

Lowenkamp,C.T.,Holsinger,A.M.,&La-tessa,E.J.(2005).Aredrugcourtseffective:Ameta-analyticreview.Journal of Community Corrections, 15 (1), 5-28.

Lowenkamp,C.T.,&Latessa,E.J.(2004).Understandingtheriskprinciple:Howandwhycorrectionalinter-ventionscanharmlow-riskoffend-ers. Topics in Community Correc-tions-2004, 3–8.

Lowenkamp,C.T.,Latessa,E.J.,&Smith,P.(2006).Doescorrectionalprogramqualityreallymatter?Theimpactofadheringtotheprinciplesofef-fectiveintervention.Criminology&PublicPolicy,5,575-594.

MacDonald,J.M.,Morral,A.R.,Raymond,B.,&Eibner,C.(2007).Theeffica-cyoftheRioHondoDUICourt:A2-yearfieldexperiment.Evaluation Review, 31, 4-23.

MacKinnon,D.P.(2008).Introduction to statistical mediation analysis.NewYork:Erlbaum.

158

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Marlowe,D.B.(2008).Applicationofsanc-tions.InC.Hardin&J.N.Kushner(Eds.),Quality improvement for drug courts: Evidence-based practic-es(pp.107-114)[MonographSeriesNo.9].Alexandria,VA:NationalDrugCourtInstitute.

Marlowe,D.B.(2011).Applyingincentivesandsanctions.InD.B.Marlowe&W.G.Meyer(Eds.),The drug court judicial benchbook(pp.139-157).Alexandria,VA:NationalDrugCourtInstitute.Availableathttp://www.ndci.org/sites/default/files/nad-cp/14146_NDCI_Benchbook_v6.pdf.

Marlowe,D.B.(2012a)(Ed.).SpecialIssueonbestpracticesinDrugCourts.Drug Court Review, 8 (1).

Marlowe,D.B.(2012b).Targeting the right participants for adult drug courts [PractitionerFactSheetVol.VII,No.1].Alexandria,VA:NationalDrugCourtInstitute.Athttp://www.ndci.org/sites/default/files/nadcp/Targeting_Part_I.pdf.

Marlowe,D.B.(2013).Achievingracialandethnicfairnessindrugcourts.Court Review,49(1),40-47.

Marlowe,D.B.,Festinger,D.S.,Arabia,P.L.,Dugosh,K.L.,Benasutti,K.M.,Croft,J.R.,&McKay,J.R.(2008b).Adaptiveinterventionsindrugcourt:Apilotexperiment.Criminal Justice Review, 33,343-360.

Marlowe,D.B.,Festinger,D.S.,Dugosh,K.L.,Arabia,P.L.,&Kirby,K.C.(2008a).Aneffectivenesstrialofcontingencymanagementinafel-onypre-adjudicationdrugcourt.

Journal of Applied Behavior Analy-sis, 41,565-577.

Marlowe,D.B.,Festinger,D.S.,Dugosh,K.L.,Lee,P.A.,&Benasutti,K.M.(2007).Adaptingjudicialsupervi-siontotherisklevelofdrugoffend-ers:Dischargeandsix-monthout-comesfromaprospectivematchingstudy.Drug & Alcohol Dependence, 88S,4-13.

Marlowe,D.B.,Festinger,D.S.,Foltz,C.,Lee,P.A.,&Patapis,N.S.(2005).Perceiveddeterrenceandoutcomesindrugcourt.Behavioral Sciences & the Law, 23,183-198.

Marlowe,D.B.,Festinger,D.S.,Lee,P.A.,Du-gosh,K.L.,&Benasutti,K.M.(2006).Matchingjudicialsupervisiontocli-ents’riskstatusindrugcourt.Crime & Delinquency, 52,52-76.

Marlowe,D.B.,Hardin,C.D.,&Fox,C.L.(2016).Painting the current pic-ture: A national report on drug courts and other problem-solving courts in the United States. Alex-andria,VA:NationalDrugCourtInstitute.Retrievedfromhttp://www.ndci.org/wp-content/up-loads/2016/05/Painting-the-Cur-rent-Picture-2016.pdf

Marlowe,D.B.,&Kirby,K.C.(1999).Effec-tiveuseofsanctionsindrugcourts:Lessonsfrombehavioralresearch.National Drug Court Institute Re-view, 2, 1-31.

Marlowe,D.B.,Patapis,N.S.,&DeMatteo,D.S.(2003).Amenabilitytotreat-mentofdrugoffenders.Federal Probation,67,40-46.

159

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Martin,S.S.,Butzin,C.A.,Saum,S.A.,&Inciardi,J.A.(1999).Three-yearoutcomesoftherapeuticcommu-nitytreatmentfordrug-involvedoffendersinDelaware.Prison Jour-nal, 79, 294-320.

McIvor,G.(2009).Therapeuticjurispru-denceandproceduraljusticeinScottishdrugcourts.Criminology & Criminal Justice, 9,29-49.

McKee,M.(2010).San Francisco drug court transitional housing program outcome study,athttp://www.sfsuperiorcourt.org/sites/default/files/pdfs/2676%20Outcome%20on%20SF%20Drug%20Court%20Transitional%20Hous-ing%20Program.pdf.

McLellan,A.T.,Lewis,D.C.,O’Brien,C.P.,&Kleber,H.D.(2000).Drugdepen-dence,achronicmedicalillness:Im-plicationsfortreatment,insurance,andoutcomesevaluation.JAMA, 284,1689-1695.

Mendoza,N.S.,Trinidad,J.R.,Nochajski,T.H.,&Farrell,M.C.(2013).Symp-tomsofdepressionandsuccessfuldrugcourtcompletion.Commu-nity Mental Health Journal,DOI:10.1007/s10597-013-9595-5.

Messina,N.,Calhoun,S.,&Warda,U.(2012).Gender-responsivedrugcourttreatment:Arandomizedcontrolledtrial.Criminal Justice & Behavior, 39,1539-1558.

Mitchell,O.,Wilson,D.B.,Eggers,A.,&MacKenzie,D.L.(2012).Assessingtheeffectivenessofdrugcourtsonrecidivism:Ameta-analyticreview

oftraditionalandnon-traditionaldrugcourts.Journal of Criminal Justice, 40,60-71.

NationalAssociationofDrugCourtPro-fessionals.(1997).Defining drug courts: The key components.Wash-ington,DC:OfficeofJusticePro-grams,U.S.Dept.ofJustice.

NationalInstituteofJustice.(2018,August23).DrugCourts.RetrievedDecem-ber20,2018,fromhttps://www.nij.gov/topics/courts/drug-courts/Pages/welcome.aspx.

NationalResearchCouncil.(2001).Inform-ing America’s policy on illegal drugs: What we don’t know keeps hurting us.WashingtonDC:NationalAcademyPress.

Newton-Taylor,B.,Patra,J.,&Gliksman,L.(2009).TorontoDrugTreatmentCourt:Participantintakecharacter-isticsaspredictorsof“successful”programcompletion.Journal of Drug Issues, Fall,965-987.

O’Hear,M.M.(2009).Rethinkingdrugcourts:Restorativejusticeasare-sponsetoracialinjustice.Stanford Law & Policy Review, 20,101-137.

OrganizationofAmericanStates,In-ter-American Drug Abuse Control Commission.(2010).Hemispheric drug strategy: Plan of action 2011-2015. Available at http://www.cicad.oas.org/Main/Template.asp?File=/main/about-cicad/basicdocuments/strate-gy_2010_eng.asp.

OrganizationofAmericanStates,In-ter-American Drug Abuse Control

160

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Commission.(2010b).Introduction to the New Hemispheric Drug Strat-egy. Available at http://www.cicad.oas.org/en/basicdocuments/Hemispheric%20Drug%20Strategy100603.pdf..

OrganizationofAmericanStates,In-ter-American Drug Abuse Control Commission.(2012b).Report on the first intensive technical work-shop on the development of meth-odologies and indicators for DTC evaluation. WashingtonDC:Author.

OrganizationofAmericanStates,Obser-vatoryonCitizenSecurity.(2011).Report on citizen security in the Americas. Available at http://www.oas.org/dsp/alertamerica/Aler-tamerica_2011.pdf.

OrganizationofAmericanStates,Obser-vatoryonCitizenSecurity.(2012a).Report on citizen security in the Americas. Available at http://www.oas.org/dsp/alertamerica/Report/Alertamerica2012.pdf.

Peters,R.H.(1996).Evaluating drug court programs: An overview of issues and alternative strategies.Wash-ingtonDC:JusticeProgramsOfficeatAmericanUniversity.Availableat http://www1.spa.american.edu/justice/documents/220.pdf

Peters,R.H.,Haas,A.L.,&Hunt,W.M.(2002).Treatment“dosage”effectsindrugcourtprograms.Journal of Offender Rehabilitation, 33,63-72.

Peters,R.H.,Kremling,J.,Bekman,N.M.,&Caudy,M.S.(2012).Co-occur-ringdisordersintreatment-based

courts:Resultsofanationalsurvey.Behavioral Sciences & the Law,DOI:10.1002/bsl.2024.

President’sCommissiononCombatingDrugAddictionandtheOpioidCrisis.(2017).Final Report to the President[PDFfile].Retrievedfromhttps://www.whitehouse.gov/sites/whitehouse.gov/files/images/Final_Report_Draft_11-1-2017.pdf.

Reingle,J.M.,Striley,C.W.,Small,E.,Cre-celius,R.,O’Leary,C.C.,&Cottler,L.B.(2012).Cancourtroombehaviorpredictrecidivism?AnassessmentoftheCourtroomBehaviorCheckListforwomenpresentingindrugcourt. American Journal of Criminal Justice,DOI:10.1007/s12103-012-9190-2.

Rempel,M.(2007).Actionresearch:Usinginformationtoimproveyourdrugcourt.InG.Berman,M.Rempel&R.V.Wolf(Eds.),Documenting re-sults: Research on problem-solving justice (pp.101-122).NewYork,NY:CenterforCourtInnovation.

Rempel,M.,Raine,V.,&Lambson,SH.(20142014).A diagnostic study of the Addiction Treatment Court in Guadalupe, Nuevo León, Mexico: Findings and recommendations. WashingtonDC:Inter-AmericanDrugAbuseControlCommission,OrganizationofAmericanStates,andCenterforCourtInnovation.

Richardson,M.,Katsakou,C.,Torres-Gon-zalez,F.,Onchev,G.,Kallert,T.,&Priebe,S.(2011).FactorialvalidityandmeasurementequivalenceoftheClientAssessmentofTreatment

161

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Scaleforpsychiatricinpatientcare–AstudyinthreeEuropeancoun-tries. Psychiatry Research, 188,156-160.

Roman,J.(2013).Cost-benefitanalysisofcriminaljusticereforms.NIJJournal,272.https://ncjrs.gov/pdffiles1/nij/241929.pdf

Rossman,S.B.,Rempel,M.,Roman,J.K.,Zweig,J.M.,Lindquist,C.H.,Green,M.,Downey,P.M.,Yahner,J.,Bha-ti,A.S.,&Farole,D.J.(2011).The Multi-Site Adult Drug Court Evalua-tion: The impact of Drug Courts, vol-ume 4. WashingtonDC:UrbanInsti-tuteJusticePolicyCenter.Availableat https://www.ncjrs.gov/pdffiles1/nij/grants/237112.pdf.

Rubio,D.,M.,Cheesman,F.,&Federspiel,W.(2008,July).Performance mea-surement of drug courts: The state of the art [StateTechnicalAssis-tanceBulletinVol.6]. Williams-burg,VA:NationalCenterforStateCourts.

Rysavy,P.,Cunningham,T.,O’Reilly-Mar-tinez,R.(2011).PreliminaryanalysisoftheNorthernTerri-tory’sillicitdrugcourtdiversionprogramhighlightstheneedtoexaminelowerprogramcom-pletionratesforindigenouscli-ents. Drug & Alcohol Review, 30,671-676.DOI:10.1111/j.1465-3362.2010.00275.x

Satel,S.(1998).Observationalstudyofcourtroomdynamicsinselecteddrugcourts.National Drug Court Institute Review, 1,43-72.

Sevigny,E.L.,Pollack,H.A.,&Reuter,P.(2013).Candrugcourtshelptoreduceprisonandjailpopulations?Annals of the American Academy of Political & Social Science, 647,190-212.

Shaffer,D.K.(2010).Lookinginsidetheblackboxofdrugcourts:Ameta-analyticreview.Justice Quarterly, 28, 493-521.

Somers,J.M.,Currie,L.,Moniruzzaman,A.,Eiboff,F.,&Patterson,M.(2011).DrugTreatmentCourtofVancouver(DTCV):Anempiricalevaluationofrecidivism.Addiction Research & Therapy, 2,1-7.Availableathttp://dx.doi.org/10.4172/2155-6105.1000117.

Somers,J.M.,Rezansoff,S.N.,&Moniru-zzaman,A.(2012).Mediators of drug treatment court outcomes in Vancouver Canada.BritishColum-bia,Canada:SimonFraserUniv.

Sperber,K.M.,Latessa,E.,&Makarios,M.D.(2013).Examiningtheinterac-tionbetweenlevelofriskanddos-age of treatment. Criminal Justice & Behavior, 40,338-348.

Sullivan,D.,&Tifft,L.(2008).Handbook of restorative justice.NewYork:Rout-ledge.

Summers,R.E.,&Barber,J.P.(2003).Therapeuticallianceasameasur-ablepsychotherapyskill.Academic Psychiatry, 27,160-165.

Taxman,F.S.,&Bouffard,J.A.(2005).Treatmentasapartofdrugcourt:Theimpactongraduationrates.Journal of Offender Rehabilitation, 42,23-50.

162

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Taxman,F.S.,&Marlowe,D.B.(Eds.)(2006).Risk,needs,responsivity:Inactionorinaction?[SpecialIssue].Crime & Delinquency, 52(1).

TreatmentResearchInstitute.(2012).In-creasing effective communication between criminal justice and treat-ment settings using health informa-tion technology.Philadelphia,PA:Author.

Turner,S.,Greenwood,P.Fain,T.,&De-schenes,E.(1999).Perceptionsofdrugcourt:Howoffendersvieweaseofprogramcompletion,strengthsandweaknesses,andtheimpactontheirlives.National Drug Court Institute Review, 2,61-85.

Tyler,T.R.(2007).Proceduraljusticeandthecourts.Court Review, 44,26.

U.S.GovernmentAccountabilityOffice.(2005).Adult drug courts: Evidence indicates recidivism reductions and mixed results for other outcomes [No.GAO-05-219].Washington,DC:Author.

VanWormer,J.(2010).Understanding op-erational dynamics of drug courts [Doctoraldissertation].Dept.ofCriminalJustice,UniversityofWashington.

Wiener,R.,Winick,B.J.,Georges,L.S.,&Castro,A.(2010).Atestabletheoryofproblem-solvingcourts:Avoidingpastempiricalandlegalfailures.International Journal of Law & Psy-chiatry, 33,417-427.

Wodahl,E.J.,Garland,B.,Culhane,S.E.,&McCarty,W.P.(2011).Utilizingbehav-ioralinterventionstoimprovesuper-visionoutcomesincommunity-basedcorrections.Criminal Justice & Behav-ior, 38,386-405.

Wolfer,L.(2006).Graduatesspeak:Aqual-itativeexplorationofdrugcourtgraduates’viewsofthestrengthsandweaknessesoftheprogram.Contemporary Drug Problems, 33,303-320.

Zarkin,G.A.,Cowell,A.J.,Hicks,K.A.,Mills,M.J.,Belenko,S.,Dunlap,L.J.,&Keyes,V.(2012).Lifetimebenefitsandcostsofdivertingsub-stance-abusingoffendersfromstateprison.Crime&Delinquency.DOI:10.1177/0011128712461904.

Zweig,J.M.,Lindquist,C.,Downey,P.M.,Roman,J.,&Rossman,S.B.(2012).Drugcourtpoliciesandpractices:Howprogramimplementationaf-fectsoffendersubstanceuseandcriminalbehavioroutcomes.Drug Court Review,8,43-79.

163

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

AddictionApsychiatricdiagnosisreflectingcompulsiveuseofalcoholorotherdrugs.Characteris-ticsymptoms includecravings for thesub-stance, uncontrolled usage, or uncomfort-ablewithdrawal symptomswhen levels ofthesubstancedeclineinthebloodstreamorcentralnervoussystem.InthemostrecentversionofDiagnosticandStatisticalManualofMentalDisorders(DSM-5orDSM-V),itisreferredtoas“substanceusedisorder.”Seealso:Substance dependence.

Adulterated test specimensBodilyspecimenssuchasurineorsalivathatshow evidence of having been tamperedwith or of being fraudulent or unreliable.Most laboratories routinely evaluate drugtestspecimensforevidenceofadulterationbyexaminingsuchindicatorsastemperature,pH, creatinine, and specific gravity. ManyDTCs assume adulterated specimens to besubstance-positive or require that a newspecimen be delivered. Evaluators shouldordinarily treat adulterated specimens assubstance-positive or as not having beendeliveredasdirected (i.e., asanunexcusedfailuretoprovideascheduledsample).

Attendance rateTheratioofthenumberofsessionsorser-vicesthatparticipantsreceived,dividedbythe number of sessions or services theywerescheduledtoreceive.

Best practicesSpecific services or interventions in DTCsthathavebeendemonstratedthroughem-piricalresearchtosignificantlyimproveout-comes.

Clinical assessment toolsStandardized and validated questionnairesor interviews that assess the diagnosticcriteria for or symptoms of substance de-pendenceandothermajorpsychiatricdis-orders.

CohortA group of participants who entered theDTCduringthesamespecifiedtimeperiod,oftendefinedasan intervalof12months.Forexample,allparticipantswhoenteredaDTCbetween January1 andDecember31ofagivenyearmightbedefinedasacohort. Community serviceA type of restorative justice interventionrequiringoffenderstoworkonavolunteerbasisinthecommunity.Commonexamplesofcommunityservicemayincludecleaninguplitteronthehighwayorworkinginasoupkitchenforthehomeless.

Comparison groupA sample of individuals who are similarto the DTC participants but did notparticipateintheDTC.InDTCevaluations,the comparison group is often comprisedof drug-involved offenders who weresentencedtoprobationorwhounderwentadjudication as usual. The comparisongroup should be as equivalent as possibletotheDTCgroupwithrespecttovariablesthatwouldbeexpectedtoaffectoutcomes.Without a comparison group, it cannotbe determined whether the outcomeswereaffectedbytheDTCorwhethertheymight have occurred anyway even if theparticipantshadnotenteredtheDTC.

GLOSSARY OF TECHNICAL TERMS

164

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Computer-assisted personal interview (CAPI)Acomputerizedsurveythatallowstheeval-uator to enter items directly into an ana-lyzable database. The computer presentsitemsoneatatime;mayofferhelp-menusindicatinghowtophrase,score,orinterpretan item;andskipsquestions thatare logi-cally inapplicable for a participant (e.g., aquestion concerningpregnancy for amaleinterviewee).

Computer syntax:See:Statistical syntax.

CorrelationThedegree towhich twovariablesareas-sociatedwitheachother.Acorrelationin-dicatesthereisarelationshipbetweenthevariables,butdoesnotprovecausality.Forexample,moretreatmentinaDTCmightbecorrelated with better outcomes, but thisdoesnotprovethattreatmentwasrespon-siblefortheoutcomes.

Cost evaluationAsystematicevaluationthatattachesmon-etaryvaluestotheresultsofanimpacteval-uationtoestimatethenetfinancialimpactsofaDTC.

Counterfactual hypothesisAn alternative hypothesis to the researchhypothesis which predicts that a DTC didnotaffectoutcomes.Inmostinstances,be-ingabletotestthecounterfactualhypoth-esisrequirestheevaluatortouseafairandunbiasedcomparisonsampleagainstwhichto compare the performance of the DTCparticipants. CovariateAvariablethatisenteredfirstintoastatis-tical analysis, andwhose influence is thenfactored out to evaluate the effect of thetrue variable of interest. This procedure

helps to rule out confounds or alternativeexplanationsthatmighthaveaccountedfortheeffectsbeingobserved.

Data & Safety Monitoring Board (DSMB)A multidisciplinary group of professionalswith expertise in research methods, re-search ethics and statistical analyses whoareresponsibleforoverseeingtheintegrityofdatacollectionanddataanalysesduringrandomized studies. Among other duties,theDSMBmayperformoroverseeinterimanalysestodeterminewhetheranypartici-pantsinthestudyarebeingdisadvantagedor suffering adverse events related to theinterventions.

Date-stampingThe practice of connecting services oreventstothedatesonwhichtheyoccurredorweresupposedtohaveoccurred.Forex-ample,indicatingthatacounselingsessionwasattendedonJanuary1,2013,isaformof date-stamping. Date-stamping is criti-cally important for measuring many per-formance indicators in DTC program eval-uations. Some computerized data-entrysystemsautomaticallydate-stampentries.

Density of servicesTheamountoftreatmentorotherservicesthatweredeliveredperunitoftime,suchaspermonthorperphaseoftheprogram.Forexample,attendanceatfivesessionsinonemonth(5÷1=5)reflectsagreaterdensityof services thanattendanceat15sessionsinsixmonths(15÷6=2.5)eventhough15sessionsaremorethanfivesessions.

Descriptive analysesStatisticalinformationthatdoesnotseektoinfercauseandeffectorattributethefind-ings to a larger population. For example,calculating thepercentageofwomenwhograduatedfromaDTCortheaveragenum-

165

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

beroftreatmentsessionsthatwereattend-edaredescriptiveanalyses.

Direct secure messaging (DSM)A method for transmitting encrypted in-formationtoknownandtrustedrecipientsovertheInternet.

Distal outcomesLong-term outcomes often occurring afterparticipants are no longer enrolled in theDTCprogram.

Dosage or doseTheamountofaserviceortreatmentthatparticipants actually received, as opposedto what they were scheduled to receive.Dosage is typicallymeasured by the num-ber of sessions thatwere attendedor thelengthoftimeparticipantswereexposedtotheservice.

EntryThespecificdateonwhichaparticipantoffi-ciallyenteredaDTCorcomparableprogramand the program obtained legal authorityovertheindividualtoordertreatmentandsupervision.ManyDTCsandprobationpro-gramshaveaformalentryhearingincourtatwhichtheparticipantpleadsorisfoundguilty (or in some jurisdictions,where thepre-trial/pre-plea criminal proceedings aresuspended),maywaivecertainlegalrights,andformallyenterstheprogram.

Field visitsSupervision activities conducted by pro-bationofficersoutsideoftheiroffices. Ex-amplesmay include visits to anoffender’shomeorplaceofemployment.

Follow-up windowThetimeperiodduringwhichoutcomesaremeasured. For example, if outcomes aremeasuredfor12monthsaftereachpartic-

ipant’sentryintoaDTC,thenthefollow-upwindow is 12 months in length. In mostanalyses, follow-up windows should beequivalent or comparable in length for allparticipants.

GeneralizabilityThe degree to which the same result isfound in different programsor amongdif-ferentpopulationsofparticipants.

GraduationSuccessful discharge from a DTC markingcompletionoftherequirementsofthepro-gram.OutcomesaresignificantlybetterforDTCs that have formal criteria for gradua-tionandholdaformalgraduationceremonyin court.

High-need offendersOffenderswith relatively severe substanceabuse or mental health disorders. High-need offenders have been found to havebetteroutcomesinDTCsascomparedtoal-ternativeprogramssuchasprobation.

High-risk offendersOffenderswithmoreseverecriminalhisto-riesorwhohaveahistoryoffailingintreat-mentoroncorrectionalsupervision.Theseindividualsareatahigherriskforre-offend-ing,andtendtohaveapoorprognosisforsuccess instandardcorrectionalrehabilita-tionprogramsandrequireintensivesuper-vision to succeed in treatment. High-riskaddictedoffendershavebeenfoundtohavebetteroutcomesinDTCsascomparedtoal-ternativeprogramssuchasprobation.

Illicit drugsDrugs that are legally banned and pre-scription medications that are used for anon-prescribed or non-medically-indicatedpurpose.

166

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Imputation proceduresStatisticalproceduresthatmaybeemployedtocompensateformissingdata.Someim-putationproceduresaccountforthepatternof results immediately before and after amissingresult.Othersassumetheaverageormostprevalentresultfortheparticipantorthesampletobethelikelyoutcomeofamissingscore.Expertconsultationisusuallyrequiredtodecidewhetherandhowtousetheseprocedures.

Institutional Review Board (IRB)A multidisciplinary group of professionalsandcommunityrepresentativeswithknowl-edgeorexpertiseinresearchmethods,re-search ethics, and the subjectmatterof astudy.TheIRBisresponsiblefordetermin-ingwhetherastudyisethicalandsafewithregardtosuchmattersasinformedconsent,confidentiality,andtherisk/benefitratioforparticipants. The IRBmaybeempoweredto stop a study or require changes to themethodsasaconditionofpre-approvalorannualre-approval.

Intent-to-treat analysisAnanalysisthatincludesdataonallindivid-uals who entered the DTC or comparisonprogram regardless of whether they com-pleted, dropped out, or were terminatedfromtheprograms.Intent-to-treatanalysesshouldordinarilybereportedastheprimaryanalysesformostDTCprogramevaluations.

Interaction analysesStatisticalprocedures thatexamine the in-fluence of more than one variable on anoutcomeanddeterminehowthevariablesaffectedtheoutcomealoneandincombina-tion.InDTCevaluations,interactionanaly-sesmightbeusedtodeterminewhichtypesofparticipantshadbetteroutcomes intheDTCasopposedtoacomparisoncondition.

Inter-rater reliabilityThedegreetowhichdifferentratersoreval-uators assign similar valueswhenmeasur-ingavariable.

Investment costsTheadditionalexpendituresrequiredtoad-ministeraDTCprogram,suchastheaddedcostsof treatmentor frequentcourthear-ings.

JailInsomejurisdictions,acorrectionalinstitu-tion that typically detains individuals on apre-trialbasisorforsentencesoflessthanoneyearinlength.

Log-linear transformationAmathematicalprocedurethatmaycom-pensateforaskewedornon-normaldistri-butionof scores. This procedurehas theeffect of smoothing out a skewed distri-bution ormaking it function as if itwerea normal distribution. Expert statisticalconsultation is often required to employthisadvancedstatisticalprocedureappro-priately.

Long-term outcome evaluationAsystematicevaluationofparticipants’per-formanceaftertheyarenolongerenrolledintheDTCprogram.Theevaluationtypical-lyfocusesonpost-programoutcomessuchasnewarrestorconvictionrates, re-incar-ceration rates, and employment rates. Itmayalsobereferredtoasadistaloutcomeevaluation.

Low-need offendersOffenders who do not have serious sub-stance abuse or mental health disorders.Low-needoffendersdonotneedtobetreat-edinprogramssuchasDTCsthatprovideahighleveloftreatmentandrehabilitation.

167

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Low-risk offendersOffenderswith relatively less severecrimi-nalhistorieswhohaveagoodprognosisforsuccess instandardsubstanceabusetreat-mentorcorrectionalrehabilitation.Suchin-dividualsareatalowerriskforre-offendingandoftendonotrequireintensiveprogramssuchasDTCstohavepositiveoutcomes.

Management information system (MIS)An automated computer system that col-lects standardizeddata elements andmayrun statistical analyses and provide out-comereports.InDTCs,anMISmightcollectandanalyzedataontheservicesdeliveredintheDTCandparticipants’performance.

Matched comparison groupIndividualswhoarepairedwithDTCpartic-ipantsbasedonmultiplevariablesthatareknowntoaffectoutcomes.Forexample,anevaluatormightmatcheachDTCparticipantwithaprobationerwho is similar in termsofcriminalhistory,demographiccharacter-istics,andsubstanceabuseproblems. Thegroupsshouldbematchedonvariablesthatpredictoutcomes,andnotmerelyvariablesthatareeasytomeasuresuchasbasicde-mographics.

Mediator variableAvariable thataffects the relationshipbe-tweenapredictorvariable (suchasenroll-ment in a DTC) and an outcome variable(suchasrecidivismorgraduation).Theme-diatorvariablecanbethoughtofasan in-terveningvariablethathelpstoexplaintherelationshipbetweenthepredictorvariableandoutcomevariable. Mediatorvariablesare identified through advanced statisticalanalysescalledmediationanalyses.

Missing denominatorReferstoaproblemcommonlyencounteredinDTCprogramevaluationsinwhichthere

is a failure to faithfully record informationabout events that should have transpiredbutdidnot.Forexample,datamightnotberecordedon treatment sessions thatwerescheduledtooccurbutwerenotattended.This can complicate the interpretation offindings. It is importantforstaffmembersto record information about whether ap-pointmentswerekept,notkept,excusedorrescheduled.Seealso:Problem of the miss-ing denominator.

Moderator variablesCharacteristicsofparticipantsthatdifferen-tiallypredictoutcomesinDTCsascomparedtoalternativeprograms,suchasprobation.Moderator variables may be identifiedthrough the use of statistical techniquescalled moderator analyses or interactionanalyses. Theseanalyses assist theevalu-ator to determine which types of partici-pantswerehelpedbyaDTCandwhichpar-ticipantsmightnothavebeenhelpedbytheDTC.

Motivation for changeReferstoabodyofresearchfindingbetteroutcomes in substance abuse treatment for patients who were intrinsically motivatedtoimprovetheir lives. IntrinsicmotivationforchangedoesnotappeartoberequiredwhenparticipantsfirstenteraDTCbutmaybecome necessary before they are dis-charged.Itmayalsobereferredtoasread-inessforchange.

Neutral dischargeA type of discharge from a DTC that indi-catesneitherfailurefromtheprogramnorsuccessfulcompletionoftheprogram.ItistypicallyreservedforparticipantswhohaveseriousmedicalorpsychiatricillnessesthattheDTCisunabletotreat,orwhomoveoutof the jurisdiction or enlist in themilitarywith the permission of the court. Partici-

168

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

pantswho receive a neutral discharge areremovedfromtheequationwhencalculat-ingmostperformanceindicatorsforaDTC.

Nonparametric statistical testsStatistical tests that do not require thescoresinthesampletobenormallydistrib-utedortohaveawiderange.

Operationalizing variablesThe process of defining variables such asthe services provided in a DTC or partici-pants’outcomesinobjective,concrete,andmeasurable terms.

Outcome costsThe expenditures incurred by taxpayers orthegovernmenttodealwithparticipantsaf-tertheycompletedtheDTCprogram,suchasthecostsofprosecutingnewoffensesorin-carceratingtheparticipantsfornewcrimes.

Outcome evaluationA systematic study of how a DTC affectedparticipants’ performance on outcomessuch as substance use or crime. See also:Short-term outcome evaluation or Long-term outcome evaluation.

Outcome savingsThemoniesrepaidtoorreclaimedbysoci-etyasaresultoftheimprovedsocialfunc-tioning of DTC participants. For example,DTC participants might pay more taxes,contributetothefinancialsupportoftheirchildren,orvolunteertoworkincharitiesorsocialserviceagencies.

Participant characteristicsAttributes of participants that pre-datedtheirentryintotheDTCorcomparisonpro-gram. Thesetypically includesocio-demo-graphicvariablessuchasage,gender,race,and employment status; clinical variablessuchasparticipants’primarysubstancesof

abuse,psychiatric/substanceusediagnoses,and history of substance abuse ormentalhealthtreatment;andcriminalhistoryvari-ablessuchaspriorarrests,convictions,andincarcerations.See also: Predictor variables and Risk factors.

Participant-level performance indicatorsQuantifiablemeasuresofhowparticipantsperformedwhiletheywereenrolledintheDTC and after discharge. Examplesmightincludehowoftenparticipants testedneg-ative for alcohol and other illicit drugs orgraduated from the DTC. They may alsobereferredtoasoutputs.See also: Perfor-mance indicators.

Perceived deterrenceReferstoabodyofresearchindicatingthatoffendershavebetteroutcomeswhentheyperceive a direct and rational connectionbetweentheirownconductandtheimpo-sitionofrewardsandsanctionsbycriminaljustice authorities. Better perceptions ofperceiveddeterrencehavebeenassociatedwithbetteroutcomesinDTCs.

Performance benchmarksSpecificthresholdsorlevelsofperformancethatDTCsshouldstrivetoachieve.Ideally,performancebenchmarksshouldbebasedonempiricalevidenceindicatingwhichprac-ticesproducedbetteroutcomesinDTCs.

Performance indicatorsPerformance indicators are quantifiablemeasuresoftheservicesprovidedinaDTC(called program-level performance indica-tors or inputs) and how participants per-formedintheprogramandafterdischarge(participant-levelperformanceindicatorsoroutputs).Examplesofprogram-levelperfor-mance indicatorsmight includehowoftencourthearingswereheldorhowoftenpar-

169

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

ticipants attended substance abuse treat-ment.Examplesofparticipant-levelperfor-mance indicatorsmight includehowoftenparticipantstestednegativeforalcoholandillicitdrugsorgraduatedfromtheprogram.Seealso:Participant-level performance in-dicators and Program-level performance indicators.

Policies and procedures manualAwrittenmanual documenting the essen-tialpractices,interventionsandpoliciesfortheDTC.ResearchindicatesDTCshavebet-teroutcomeswhenthey(1)haveapoliciesand procedures manual, (2) generally ad-heretowhatiswritteninthemanual,and(3)updatethemanualonaregularbasis.

Post-adjudication DTCADTCinwhichafinalconvictionhasbeenenteredandparticipantscompletethepro-gram in lieuof receivinga longerormoreseveresentence.

Pre-adjudication DTCA DTC in which successful graduates canhave their guilty plea or conviction with-drawnorvacatedandmayhavethearrestexpungedorerasedfromtheircriminalre-cord (or inother jurisdictionswhereapri-orguiltypleaorconviction isnot requiredforentry,aDTC inwhichthepre-trial/pre-plea criminal proceedings are suspended,andinwhichsuccessfulgraduateshavethechargesagainstthemdropped).

Predictor variablesCharacteristicsofparticipantsthatpre-datedtheirentryintotheDTCorcomparisonpro-gramandreliablycorrelatewithoutcomes.

PrisonInsomejurisdictions,acorrectionalinstitu-tion that typically incarcerates individualsforsentencesoflongerthanoneyear.

Probabilistic record linkageAmethod for linkingcases indifferentad-ministrative databases when no uniqueidentifyingnumberorvariable isavailable.Cases are linked using a combination ofavailabledataelementssuchbirthdatesandfirstand lastnames. Thedegreeof confi-dencethatcanbeplacedinthelinksisde-pendentonsuch factorsas thenumberofvariables that are available to create thelinkages,thedegreeofvariabilityonthosevariables,and thenumberofpersons thatareinthedatabase.

ProbationAcriminalsentencerequiringanoffendertobesupervisedinthecommunityinsteadofbeingdetainedinajailorprison.

Probation officerA criminal justice professional who is pri-marilyresponsibleforsupervisingoffenderswhohavebeen sentenced toprobation inthecommunity.

Problem of the missing denominatorSee:Missing denominator.

Procedural justice or procedural fairnessThe phenomenon, demonstrated by abody of research, in which litigants reactmorefavorablytoanadversejudgmentorpunitive sanction if they believe fair pro-cedureswerefollowedinreachingthede-cision. Greaterperceptionsofproceduraljustice have been associated with betteroutcomesinDTCs.

Process evaluationA systematic study indicating whether aDTC is functioning as originally planned,treating the intended target population,and delivering the types and dosages ofservices that are likely to produce favor-able outcomes.

170

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Program-level performance indicatorsQuantifiablemeasuresof theservices thatwereprovidedintheDTC.Examplesmightincludehowoftencourthearingswereheldor how often participants attended sub-stanceabusetreatment.Thesemayalsobereferredtoasinputs.

Propensity score analysisA statistical procedure thatmaybeusedto control for differences in participantcharacteristics between groups. It in-volves mathematically calculating theprobabilitythatanindividualwouldbeintheDTCgroupasopposedtothecompar-isongroup,ortherelativesimilarityoftheindividualtoonegroupasopposedtotheother. The analysis statistically accountsfor this relative probability when com-paring outcomes between the groups.Advancedstatisticalexpertiseisoftenre-quiredtoimplementandinterpretthere-sultsofthisprocedure.

Proximal outcomesShort term outcomes usually occurringwhile participants are still enrolled in theDTCprogram.Examplesmightincludecoun-selingattendanceorgraduationrates.

Quasi-experimental comparison groupA comparison sample of individuals whodidnotentertheDTCforreasonsthatareunlikelytohaveaffectedtheiroutcomes.Agood example might be drug-involved of-fenderswhowereeligibleforandwillingtoentertheDTCbutcouldnotgetinbecausetherewerenoemptyslotsavailable.

Random assignmentAprocedureforassigningparticipantstodif-ferentgroupsinanunsystematicandunbi-asedmanner,suchasbytheflipofthecoin.Random assignment provides the greatestassurancesthatthegroupsstartedoutwith

anequal chanceof success, and thus thatanypositiveoutcomescanbeattributedtotheeffectsof theDTCprogramandnottoextraneousfactors.

RecidivismThe incidenceofnewcriminalactivityoc-curringafterparticipantsenteredtheDTCor comparison program. This includescriminal activity occurring while partici-pants were enrolled in the program, andafter they graduated or were terminat-ed fromtheprogram. Recidivism ismostcommonly measured by the number orpercentageofnewarrestsornewconvic-tionsafterentry.Itmayalsobereferredtoasre-offending.

Redundancy or redundant variablesThe degree to which predictor variablesarecorrelatedwitheachotherandthusdonotprovideindependentpredictionofout-comes.

ReplicationTheprocessof repeatingor reproducingastudy to ensure the same results are ob-tained.Replicationincreasesthelikelihoodthatthefindingsarevalidandreliable.

Restorative justice interventionsRequirementsthatmaybeimposedonpar-ticipants inDTCstocompensatevictimsorsocietyfortheircrimes. Examplesmayin-cludevictimrestitution,communityservice,victimimpactpanels,andpaymentoffinesor fees.

RewardsConsequencesforbehaviorthataredesiredbyoffenderssuchasverbalpraise,applause,smallgifts,andreductionsinsupervisionre-quirements. Theymayalsobereferredtoasincentives.

171

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Risk factorsCharacteristicsofparticipantsthatpre-dat-ed theirentry into theDTCorcomparisonprogramandreliablycorrelatewithpooreroutcomes. Commonexamplesof risk fac-tors include a younger age, prior failuresin treatmentor rehabilitation,andamoreserious criminal or substance abuse histo-ry. Individualswith such risk factors typi-callyrequiremoreintensiveandstructuredinterventions to succeed in treatmentandrefrainfromcriminalactivity.

Risk assessment toolsStandardized and validated questionnairesorinterviewsthatmeasurethelikelihoodofparticipantsfailingintreatmentorcommit-tinganewoffense.Offenderswhoareiden-tifiedasbeinghighriskonthesetoolstendto have better outcomes in DTCs as com-paredtolessintensivedispositionssuchasprobation.

SanctionsConsequencesforbehaviorthataredislikedby offenders such as verbal reprimands,monetary fines, community service, in-creased supervision requirements, or in-carceration. Theuseofsanctions,andthetypes of sanctions used, varies from juris-dictiontojurisdiction.

SCRAM®Secure Continuous Remote Alcohol Moni-tor. An ankletmonitoringdevice that candetect alcohol in sweat vapors and trans-mitsdatawirelesslytoacentralmonitoringstation.

Secondary analysesDataanalysesthatexamineoutcomesonlyfor subgroups of participants (e.g., gradu-ates only). Secondary analyses are morelikely than intent-to-treat analyses to turnupunreliableorchancefindings.Therefore,

theyshouldordinarilybeperformedonlyifthe intent-to-treat analysis first indicatedsignificantresults.

Short-term outcome evaluationAsystematicevaluationofparticipants’per-formancewhiletheyareenrolledintheDTCprogram. The evaluation typically focusesonduring-treatmentoutcomesthatarelike-ly to predict post-program outcomes suchas graduation rates, treatment attendancerates,andratesofdrug-negativeurinetests.Itmayalsobereferredtoasaproximalorintermediateoutcomeevaluation.

Skewed distributionRefers to scores on a variable that have arestrictedrangeorsimilarvaluesforalargeproportionofparticipants.Manystatisticaltestsassumethatdistributionsarenormalorhaveawiderangeofvalues,andthere-forecannotbeusedtoanalyzedatawithaskewed distribution. The evaluator mightneedtouseaNonparametric statistical test or a Log-linear transformation to compen-satefordatathathasaskeweddistribution.

Starting clockThetimepoint fromwhichdata collectiononperformanceindicatorsbegins.ForDTCprogramevaluations,theclockshouldordi-narilybestartedon thedateofentry intotheDTCorcomparisonprogram.

Statistical imputation proceduresSee:Imputation procedures.

Statistical significance or statisticallysignificant differencesDifferences between groups that have ahigh mathematical probability (usually 95percentorhigher)ofbeingreproducibleinthefuture.Statisticallysignificantdifferenc-espermitgreaterconfidenceinthereliabil-ityofone’sfindings.

172

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Statistical syntaxPre-programmed statistical equations in-structinghowthedatashouldbeanalyzed.Forexample,acomputersystemmighthavea pre-programmed statistical formula forcalculating theaverageageofparticipantsinaDTCprogram.

Substance dependenceApsychiatricdiagnosisreflectingcompulsiveuseofalcoholorotherdrugs. Characteris-tic symptoms include cravings for the sub-stance, uncontrolled usage, or uncomfort-able withdrawal symptoms when levels ofthesubstancedeclineinthebloodstreamorcentralnervoussystem.Seealso:Addiction.

Survival analysisAstatisticalprocedurethatcomparesgroupsofparticipantsontheaveragelengthoftimeuntilaspecifiedeventoccurs,suchasanewcriminalarrestorrelapsetodruguse.

Target populationThe sub-group of drug-involved offenderswhoperformsignificantlybetterinDTCsascomparedtoalternativeprograms,andthusshouldbeprioritizedforentryintoDTCs.

Technical violationsViolationsofacourtorderthatdonotcon-stituteacrimeperse.Forexample,drink-ingalcohol is legalformostadults inmostcountries,butmaybeatechnicalviolationforaDTCparticipantandmayleadtoanar-restorconvictionforaprobationviolation.

Therapeutic allianceThephenomenon,demonstratedbyabodyofresearch, inwhichpatients’perceptionsofthequalityofthetherapist-patientrela-tionship reliablypredictsoutcomes inpsy-chotherapy.TheimpactofthetherapeuticalliancehasnotbeenwellstudiedinDTCs,buthasbeenhypothesizedtobeanimport-

antfactorforsuccessinDTCprograms.Itisalsoreferredtoastheworkingalliance.

Time at libertyTheproportionoftimeduringthefollow-upperiod when participants were relativelyfree in the community to engage in drugabuse,crime,orotherbehaviorsofinteresttoevaluators.Restrictionsonparticipants’timeatlibertytypicallyincludephysicalbar-rierssuchasjailorresidentialtreatment.

Time at riskThe length of time in which participantscould have engaged in drug abuse, crime,or other behaviors of interest to the eval-uator. Generally speaking, the longer thefollow-upperiod,thelongerthetimeatrisk.

VarianceThedegreetowhichparticipantsproduceda range of different scores on ameasure.Forexample,ifallparticipantsarebetweenthe ages of 21 and 23 years, the variancein age is low. For mathematical reasons,it iseasier todetect statistically significantdifferencesbetweengroupswhenthevari-anceonameasureishigh.

Victim impact panelA type of restorative justice interventionthatrequiresoffenderstomeetwithcrimevictimsandlearnhowtheywereaffectedbythecrime.Thegoalistohelpoffendersde-velopempathyandgainperspectiveonhowtheiractionsaffectedotherpeople.

Wait-list comparison groupDrug-involvedoffenderswhowere eligiblefor and willing to enter the DTC but whocould not get in because there were noemptyslotsavailable. Await-listcompari-sonsampleisoftenthebestalternativeaf-ter randomassignment foraDTCprogramevaluation.

173

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Performance Indicators and Performance MeasurementNationalCenterforStateCourts(NCSC)&JusticeProgramsOfficeatAmericanUniversityTranslatingDrugCourtResearchintoPractice(R2P)https://nicic.gov/series/r2p-translating-drug-court-research-practice

Performance Measurement of Drug Courts: The State of the ArtNationalCenterforStateCourts(2008)http://cdm16501.contentdm.oclc.org/cdm/ref/collection/spcts/id/171

Local Drug Court Research: Navigating Performance Measures and Process EvaluationsNationalDrugCourtInstitute(2006)http://www.ndci.org/publications/monograph-series/navigating-perfor-mance-measures-and-process-evaluations

Evaluating Drug Court Programs: An Overview of Issues and Alternative Strategies JusticeProgramsOfficeatAmericanUniversityhttps://jpo.wrlc.org/handle/11204/3306

Introductory Handbook for DWI Court Program Evaluations. NationalCenterforDWICourts(2010)http://www.dwicourts.org/sites/default/files/nadcp/DWI%20Ct%20Eval%20Manual%20REVISED-8-10.pdf

Recidivism 101: Evaluating the Impact of Your Drug Court CenterforCourtInnovation(2005)http://www.courtinnovation.org/sites/default/files/Recidivism_101%5B2%5D.pdf

Management Information Systems (MIS) TreatmentResearchInstituteCourtEvaluationProgram(TRI-CEP)http://triweb.tresearch.org//?s=tricep

DrugCourtCaseManagementSystem(DCCM)AdvancedComputerTechnologieshttp://www.actinnovations.com/Products/DCCM

eCourtSystemU.S.NationalInstituteofDrugAbuse

APPENDIX: RESOURCES FOR DRUG TREATMENT COURTS

174

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

CriminalJustice-DrugAbuseTreatmentStudies(CJ-DATS)http://www.gmuace.org/documents/prod-pub/cjdats/cjdats-summary-ecourt.pdf

Buffalo,NY,DrugCourtCaseManagementSystemhttp://www.ndci.org/[email protected]

Accessing Administrative DatabasesNationalInformationExchangeModel(NIEM)https://www.niem.gov/aboutniem/Pages/niem.aspx.

Staff Surveys of DTC Policies and PracticesU.S.NationalInstituteofDrugAbuseCriminalJustice-DrugAbuseTreatmentStudies(CJ-DATS)NationalDrugCourtSurveyhttp://www.gmuace.org/documents/prod-pub/cjdats/cjdats-summa-ry-drug-court.pdf

U.S.NationalInstituteofJustice MultisiteAdultDrugCourtEvaluation(MADCE)

https://www.ncjrs.gov/pdffiles1/nij/grants/237110.pdf(AppendixA)

NPCResearch,Inc.AdultDrugCourtTypologyInterviewhttp://npcresearch.com/wp-content/uploads/Drug-Court-typology-guide-NPC-Research-01-26-04-copyrighted.pdf

Rating Scales of Correctional Program QualityCorrectionalProgramAssessmentInventory(CPAI)http://www.state.in.us/idoc/files/CPAI_overview.ppt

CorrectionalProgramChecklist(CPC)https://www.uc.edu/content/dam/uc/gencounsel/docs/CPC%20Training%20MOU%2011.12.14.pdf

Risk-Assessment InstrumentsLevelofServiceInventory-Revised(LSI-R)* † ‡https://ecom.mhs.com/(S(zhkd5d55qlwc3lr2gzqq5w55))/product.aspx-?gr=saf&prod=lsi-r&id=overview

175

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

RiskandNeedsTriage(RANT)http://www.trirant.org/

CorrectionalOffenderManagementProfilingforAlternativeSanctions(COMPAS)https://www.cdcr.ca.gov/rehabilitation/docs/FS_COMPAS_Final_4-15-09.pdf

OhioRiskAssessmentSystem(ORAS)https://www.uscourts.gov/sites/default/files/74_1_2_0.pdf

FederalPostConvictionRiskAssessment(PCRA)http://www.uscourts.gov/FederalCourts/ProbationPretrialServices/Supervi-sion/PCRA.aspx RiskPredictionIndex(RPI)https://www.fjc.gov/content/328578/fjc-50th-improving-criminal-justice-ad-ministration-and-sentencing-federal-courts WisconsinRiskandNeedAssessmentScale(WRN)https://csgjusticecenter.org/corrections/projects/wisconsin-state-risk-assess-ment-validation/

Clinical-Assessment InstrumentsAddictionSeverityIndex(ASI) * † ‡TreatmentResearchInstitutehttp://triweb.tresearch.org/index.php/tools/download-asi-instruments-manu-als/

GlobalAppraisalofIndividualNeeds(GAIN) * GAINCoordinatingCenterhttp://gaincc.org/instruments/

AlcoholUseandAssociatedDisabilitiesInterviewSchedule(AUDADIS)* †U.S.NationalInstituteofAlcoholismandAlcoholAbusehttp://www.ncbi.nlm.nih.gov/pubmed/10606491

DrugAbuseScreeningTest(DAST) * †http://www.camh.ca/en/education/about/camh_publications/Pages/drug_abuse_screening_test.aspx

MultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)

176

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

StructuredClinicalInterviewfortheDSM-IV(SCID) * http://www.scid4.org/

TexasChristianUniversity(TCU)DrugDependenceScreen(availableinEnglish,Spanish,Russian,Chinese(simplifiedandtraditional),andVietnamese)* https://ibr.tcu.edu/forms/tcu-drug-screen/

Procedural Justice / Procedural Fairness ScalesMultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)

ProceduralJusticeQuestionnaire(PJQ)TreatmentResearchInstitutehttp://triweb.tresearch.org/index.php/about-us/contact-us/

Perceived Deterrence ScalesMultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)

PerceivedDeterrenceQuestionnaire(PDQ)TreatmentResearchInstitutehttp://triweb.tresearch.org/index.php/about-us/contact-us/

Motivation / Readiness for Change ScalesInventariodeProcesosdeCambio* †EuropeanMonitoringCentreforDrugsandDrugAddictionhttp://www.emcdda.europa.eu/html.cfm/index3662EN.html

PersonalConcernsInventory(PCI)http://pubs.niaaa.nih.gov/publications/AssessingAlcohol/InstrumentPDFs/47_PCI.pdf

UniversityofRhodeIslandChangeAssessment(URICA)https://elcentro.sonhs.miami.edu/research/measures-library/urica/index.htmlhttps://www.ncbi.nlm.nih.gov/books/NBK64976/table/A62309/

MultisiteAdultDrugCourtEvaluation(MADCE)ParticipantSurveyhttps://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf(AppendixA)

177

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

Circumstances,Motivation,Readiness,andSuitabilityScales(CMRS)http://www.ncbi.nlm.nih.gov/pubmed/7832182

Satisfaction with the Program / Treatment ScalesTreatmentPerceptionQuestionnaire(TPQ)EuropeanMonitoringCentreforDrugsandDrugAddictionhttp://www.emcdda.europa.eu/html.cfm/index4322EN.html

ClientSatisfactionQuestionnaire(CSQ)http://www.csqscales.com

ClientAssessmentofTreatmentScale(CATS) *http://www.ncbi.nlm.nih.gov/pubmed/21342706

Therapeutic Alliance ScalesHelpingAllianceQuestionnaire-II(HAQ-II)http://www.med.upenn.edu/cpr/instruments.html

WorkingAllianceInventoryhttp://wai.profhorvath.com/

Observer Rating Scales for Court SessionsMultisiteAdultDrugCourtEvaluation(MADCE)https://www.ncjrs.gov/pdffiles1/nij/grants/237109.pdf

On-Site Training and Technical Assistance InstitutionalStrengtheningandIntegralProgramsSectionInter-AmericanDrugAbuseControlCommission(CICAD)OrganizationofAmericanStateshttp://www.cicad.oas.org/Main/Template.asp?File=/fortalecimiento_insti-tucional/default_eng.asp

U.S.NationalDrugCourtInstitute(NDCI)http://www.ndci.org/training/advanced-training

DrugCourtClearinghouseandTechnicalAssistanceProjectatAmericanUni-versityhttps://www.american.edu/spa/jpo/initiatives/drug-court/

178

MAN

UAL

FO

R SC

IEN

TIFI

C M

ON

ITO

RIN

G A

ND

EVA

LUAT

ION

On-Line Webinars and Distance Learning Programs U.S.NationalDrugCourtInstitute(NDCI)http://www.ndci.org/training/online-trainings-webinars

NationalDrugCourtResourceCenter(NDCRC)https://www.ndci.org/resources/training/

CenterforCourtInnovationhttp://drugcourtonline.org/

NationalInstituteofJustice,NationalCenterforStateCourts(NCSC)&JusticeProgramsOfficeatAmericanUniversityTranslatingDrugCourtResearchintoPractice(R2P)https://www.nij.gov/topics/courts/drug-courts/pages/research2practice.aspx

Evidence-Based Treatments for Substance AbuseNationalRegistryofEvidence-BasedProgramsandPractices(NREPP)U.S.SubstanceAbuseandMentalHealthServicesAdministrationhttps://www.samhsa.gov/nrepp

Training on Research EthicsU.S.NationalInstitutesofHealthOfficeofResearchIntegrityhttp://phrp.nihtraining.com/users/login.php

*Spanishtranslationavailable.

†ValidatedamongindividualsofHispanicorLatinoethnicity.

‡ValidatedinatleastoneSouthAmericanorCaribbeanNation.


Man

ual f

or S

cien

tific

Mon

itor

ing

and

Eval

uati

on D

rug

Trea

tmen

t Cou

rts

in th

e A

mer

icas

ISBN 978-0-8270-6830-8

Organization of American StatesSecretariat for Multidimensional Security

Inter-American Drug Abuse Control Commission

This publication was prepared by the Executive Secretariat of the Inter-American Drug Abuse

Control Commission (ES-CICAD), Secretariat for Multidimensional Security of the Organization of

American States (SMS/OAS). It was developed under the aegis of the Drug Treatment Court

Program for the Americas, an initiative coordinated by ES-CICAD/SMS/OAS. This publication was

made possible through the financial contribution of the Government of the United States of

America, through the Bureau of International Narcotics and Law Enforcement Affairs (INL) of the

U.S. Department of State, and of the Government of Canada, through the Anti-Crime Capacity

Building Program (ACCBP) of Global Affairs Canada. The contents expressed in this document are

presented exclusively for informational purposes and do not necessarily represent the opinion

or official position of the Organization of American States, its General Secretariat, or its Member

States.

manual for scientific monitoring and evaluation … · oas cataloging-in-publication data...

Documents