research synthesis reliability and validity of · pdf fileresearch synthesis reliability and...

RESEARCH SYNTHESIS RELIABILITY AND VALIDITY OF COMPAS

Tim Brennan, Ph.D. Bill Dieterich Ph.D. Beate Ehret, Ph.D.

September 2007

Northpointe–EvolvingPracticeThroughScientificInnovation 2

©2007.Allrightsreserved.

TableofContentsExecutiveSummary .................................................................................3

1.Introduction .........................................................................................6

2.Reliability .............................................................................................8

2.1 RegionalfindingsregardingReliability ...................................................................... 82.2 ReliabilityacrossGenderbreakdowns .................................................................... 102.3 InternalStructuralValidation:FactorialValidity..................................................... 13

3.Criterion/ConcurrentValidity ............................................................14

3.1 ConcurrentValidity:CorrelationswithCriminalHistoryIndicators........................ 143.2 ConcurrentValidity:ReceiverOperatorCharacteristics(ROC)............................... 16

4.PredictiveValidity—SurvivalAnalysis................................................18

4.1 CommunityCorrectionsOhio.................................................................................. 184.2 NewYorkProbation—ScaleConstruction............................................................... 194.3 NewYorkProbation—CrossValidation................................................................... 244.4 CaliforniaParole ...................................................................................................... 25

5.ConstructValidity ..............................................................................29

6.ConclusionsandFutureResearch......................................................31



ExecutiveSummaryOverthepastseveralyearsasubstantialbodyofresearchonthereliabilityandvalidityoftheCOMPAS assessment system has accumulated across various corrections jurisdictionsnationwide. This report organizes and synthesizes the findings from a number of disparatepsychometricandvalidationstudies.Specifically,thisreportprovidesaconsistencyvalidationbyevaluatingthereliabilityandvalidityoftheCOMPASscalesacrossthosedifferentstudies.Theinstrumentistheoreticallyandempiricallysound,withpsychometricpropertiesgenerallyintheexcellentorsatisfactoryrange.AvailablefindingsindicatehighpredictivepoweroftheCOMPASRecidivismRisk scale with AUC1 scores above .7 formale and female correctionalpopulations. This integrated report demonstrates the emerging consistency andgeneralizabilityregardingthesefindings.The following psychometric properties of the COMPAS scales were tested across differentstudysites:

ScaleReliability.ThemajorityoftheCOMPASscaleshavehighinternalconsistencyasevaluated using Cronbach’s alpha with the balance providing measures of scalereliability in the satisfactory range. These findings generally hold across the sites,indicatingthatCOMPASisadministeredinaconsistentandreliablemanneracrossthedifferent jurisdictions included in this comparison.Genderdifferences inalphaswereonlydetectedforveryfewofthescalesusedtoratethesubjects.Forallsites,thevastmajority of the COMPAS scales were found to be equally reliable for males andfemales.

InternalFactorialValidity.Duetoourscaledevelopmentstrategy,factorialvalidityofthe COMPAS scales is ensured. Each COMPAS subscale was developed as aunidimensionallinearfunction.Ingeneral,itemswereselectedtobeboththeoreticallyrelatedtoacertainconstructandtoloadheavilyonthesame“component”orfactorapplyingrotatedprincipalcomponentanalysis.Variationsofthefactorialvalidityacrossdifferentsiteswere foundtobeonlyminorandarenotdiscussed in this report.TheCriminalInvolvementscaleispresentedasoneexampleforthegenerallyhighinternalvalidityanddimensionalityofthescales.

Criterion/ConcurrentValidity.ObservedrelationshipsbetweentheCOMPASsubscalesand criminal history indicators data provide good evidence for criterion‐related,concurrent validity of the scales. The general consistency of correlational patternswith criminal history indicators across different sites and datasets, adds additional

1 AUCscoresaremeasuresforthestrengthofassociation,forexamplebetweenascaleandanoutcomemeasure.AUCreferstothe“areaunderthecurve.”Thiscurveiscreatedbyplottingthetrueandfalsepositiveratesfordifferentcut‐offscoresofascale.Thesteeperthecurve,thelargertheareaunderthecurve,andthebetterthescale’sclassification.



empiricalsupporttothisconclusion.Interestingly,nomajordifferencesregardingthesepatterns were found between offenders on probation or parole. Supplementaryreceiver operator characteristics analyses evaluate the accuracy of the scales indiscriminating low and high‐risk cases and confirm these findingswith AUCs rangingbetween .71 and .82. These high AUC scores may not, however, support furthergeneralization(e.g.,topredictivevalidity),sincerealisticoutcomemeasures(collectedindependentlyfromthescalescores)wereunavailableforthisparticularanalysis.

PredictiveValidity. Thepredictiveutilityof theCOMPAS risk scales isexaminedwithsurvival models using different types of outcome measures. For a communitycorrections center inOhio it isdemonstrated that theRecidivism,FTA,andViolencerisk scales are strong predictors for the criterion of program failure. Inmatesparticipatinginacommunityserviceprogram,whoscoredhighontheRecidivismRiskscaleforexample,haveariskor“hazard”ofprogramfailurethatisthreetimeshigherthanthefailurehazardofinmatesscoringlowrisk.

More importantly, theCOMPASriskpredictionmodel,nowcalledtheRecidivismRiskscale is tested using different recidivism outcomemeasures such asNewCrime,AnyOffense,PersonOffense, andFelonyOffenses. These survival analysesdemonstrate ahigh predictive power of the COMPAS Recidivism Risk scale. An AUC of .72 asoriginallyfoundinthescaleconstructionsampleofNewYorkprobationerspredictingNewCrime(s) iscrossvalidated inanewsamplewithAUC’srangingbetween.69forAnyOffenseand.71forOffense(s)AgainstPersons.SincetheAUCs inthevalidationsample are only slightly smaller than those in the scale construction sample, theCOMPAS Recidivism Risk prediction model can be considered robust. This alsoindicatesthattheriskmodelcanbegeneralizedtootheroffenderpopulationswithlittlelossofpredictiveaccuracy.Moreover,thepredictiveutilityoftheCOMPASRecidivismandViolence‐RRiskscalesistestedforparoleesinaCaliforniasampleusingcause‐specificCoxproportionalhazardsmodels.SignificantdifferencesbetweenoffendersonthedifferentrisklevelsoftheRecidivismandViolence‐RRiskscaleswerefoundregardingtheriskofparolefailure.Forexample,inmatesclassifiedashigh‐riskontheRecidivismRiskscalehaveafailurehazardthatisabout4timeshigherthanthehazardforinmatesclassifiedaslowrisk.ParoleesonthehighriskleveloftheCOMPASMatrix‐Rhavea4.7timeshigherfailurehazardincomparisontotheircounterpartsonthelowrisklevel.ThisstudyyieldedratherlowbutneverthelessencouragingAUCsof.66forbothriskscales.Althoughthesampleisverylarge,withmorethan20,000parolees,thefollow‐upisnotmatureyetwithanaveragelengthofabout6months.Therefore,thefindingscouldchangeasadditionalfollow‐uptimeaccrues.



ConstructValidity:RecentresearchprovidesevidencethattheCOMPASriskscalesandtheir components are closely related to the construct of interest—recidivism. Themain risk scales (Recidivism and Risk Matrix) and their components (CriminalInvolvement, Vocational/Educational Problems, and Drug Problems) consistentlyemergeinvariousmetaanalysesasmajorriskfactorsforrecidivism(Lipsey&Derzon1998,Gendreau,Little,&Goggin,1996;Andrews&Bonta,1994).

Given that instrument validation is an ongoing process, numerous further tests andmodelsmaybeappliedtofurtherexaminetheconcurrentandpredictivevalidityoftheCOMPASriskscales. The present overview, however, summarizes encouraging evidence that the mainCOMPAS risk scales, Recidivism Risk and the Risk Matrix‐R, perform well in predictingrecidivism in a variety of different criminal justice populations. Our next steps to furtherconfirm these findings are additional outcome validation studies in larger samples andwithlongerfollow‐upperiods.Theseanalyseswillbeconductedatseveraldifferentsites.



AbstractOverthepastseveralyearsasubstantialbodyofresearchonthereliabilityandvalidityoftheCOMPAS assessment system has accumulated across various jurisdictions nationwide. Thisreportorganizesand synthesizes the findings fromanumberofdisparatepsychometric andvalidationreportsthatwereconductedatdifferenttimesforspecificstatesystems,counties,orregionalareas.Inthisrespect,thereportprovidesaconsistencyvalidationbyevaluatingthereliability and validity of the COMPAS scales across different studies. The instrument istheoreticallyandempiricallysound,withpsychometricpropertiesgenerallyintheexcellentorsatisfactory range. Available findings indicate the predictive power of COMPAS with AUCscores above .7 for male and female correctional populations. This integrated reportdemonstratestheemergingconsistencyandgeneralizabilityregardingthesefindings.

1.IntroductionTheCOMPASsystemisdesignedtomeasurerisk,needandmediatingfactorsforrecidivismasestablishedbyresearch(see, forexampleLipsey&Derzon1998,Gendreau,Goggin,&Little,1996,). Indeed, many of the items entering into the COMPAS scales are demonstratedpredictorsofrecidivism(e.g.,Cottle,Lee,&Heilbrun2001).TheCOMPASsystemcalculatesfourdifferentriskequationmodelsspecificallyfittedforeachofthefollowingriskdimensions: Violence Recidivism FailuretoAppear CommunityFailure

Moreover,thereareseveralCOMPASinstrumentsdesignedaccordingtothetypeofcriminaljusticepopulation: COMPASCore COMPASProbation COMPASRe‐Entry

These instruments account for variations in risk factor profiles at different stages of thecriminaljusticesystem.Asopposedtootherriskassessmenttoolsthatclaimtobevalidacrossdifferent populations, we acknowledge that using an assessment instrument for parolereleasees,whichhasbeenoriginallydevelopedforprobationers,weakensitspredictivepower.Giventhatmanyoftherisk/needdomainsandindividual itemsofCoreCOMPASorCOMPASProbationfocusontheoffender’slifeinthecommunity,theywouldassessthecurrentriskofprisonreleaseesratherinsufficientlyandnotveryaccurately(seealsoAustin,2006).Forthesereasons the COMPASRe‐Entry instrumentwas developed specifically for parole populations



andincludesadditionalscalessuchasEarlyOnset(ofcriminaloffending),occurrencesofPrisonMisconduct, or the risk of Housing Problems upon release. This report focuses on CoreCOMPAS.



2.ReliabilityNorthpointehasconductedseveralstudiesprovidingspecific testsofpsychometricreliabilityand factorial validity of all COMPAS scales. This section provides an overview of reliabilityfindingsacrossdifferentregions,andacrossgender.TheinternalscalestructureandfactorialvalidityisillustratedusingtheCriminalInvolvementscale.FormoredetaileddescriptionsofallCOMPASscalespleaseseeCOMPASTechnicalManual,2006.2.1 RegionalfindingsregardingReliabilityForascaletobeusefulitmustbereliable.Reliabilityisconsistentmeasurement.Generally,iftheitemsinascalearehighlycorrelated(internallyconsistent),thenthesummatedscalewillbe reliable. Cronbach’s alphaprovides a convenientmeasureof the reliability of a scale. Byconvention,alphacoefficientsof.70orhigherindicategoodreliability.Alpha coefficients for the Core COMPAS scales across various study sites are listed in Table2.1.1Theshadedcellsindicatealphasbelowthe.70thresholdforgoodreliability(alphasof.69areconsideredreasonablyclosetothatthresholdandstillsatisfactory—thereforethesecellsarealsonotshaded).TheNumberofcasesarelistedbysiteinTable2.1.1.Tablecountsvarybyscaleduetomissingvalues.2AlphacoefficientsarenotcomputedfortheRiskofTechnicalViolenceandRiskofRecidivismscalesbecausethesescalesarelinearequations.Themajority of the COMPAS subscales listed in Table 2.1.1 have alphas above .70. Inmostcases, thesefindingsholdacrosssites.Somescalesconsistentlydisplayalphasbelowthe .70threshold across sites. These are the History of Non‐Compliance, Current Violence, FamilyCriminality, and Social Adjustment scale. The majority of the scales in COMPAS have highinternalconsistencywiththebalanceprovidingsatisfactorymeasuresofinternalconsistency.ThesefindingsindicateCOMPASisadministeredinaconsistentandreliablemanneratallsitesincludedinthiscomparison.

2Foragivenscale,ifthevaluesforanyoftheitemsweremissing,thevalueforthescalewasconsideredmissing.



Table2.1.1:COMPASScaleReliabilityScaleReliabilityasindicatedbyCronbach’sAlphaProbation Combinedc Parole

COMPASScalesa Dallas

CountyCourts’PSIb.

SanBernardinoProbationAssessments

WyomingCommunity/Incarcerated

MIPre‐ReleaseAssessments

CAPre‐ReleaseAssessments

GAPre‐ReleaseAssessments

Numberofcases n=1,170n=1,534 n=1,065 n=1,071 n=1,077 n=3,905

CriminalInvolvement

0.89 0.90 0.87 0.85 0.79 0.83

HistoryofNon‐Compliance

0.56 0.62 0.68 0.66 0.57 0.56

HistoryofViolence

0.70 0.72 0.68 0.66 0.71 0.63

CurrentViolence

0.67 0.62 0.64 0.62 0.67 0.66

CriminalAssociates/Peers

0.71 0.76 0.76 0.74 0.80 0.70

SubstanceAbuse

0.78 0.79 0.70 0.78 0.74 0.76

FinancialProblems/Poverty

0.72 0.71 0.75 0.77 0.70 0.70

Vocational/EducationalProblems

0.69 0.68 0.65 0.69 0.67 0.67

FamilyCriminality

0.63 0.63 0.66 0.63 0.63 0.59

SocialEnvironment/Neighborhood

0.82 0.80 0.77 0.87 0.81 0.80

Leisure/Boredom

0.80 0.79 0.84 0.86 0.82 0.80

ResidentialInstability

0.63 0.68 0.69 0.68 0.71 0.65

SocialAdjustment

0.60 0.58 0.61 0.59 0.53 0.52

JuvenileSocializationProblems

0.70 0.70 0.71 0.68 0.68 0.65

CriminalOpportunity

0.66 0.63 0.68 0.71 0.68 0.63

SocialIsolation

0.79 0.80 0.84 0.84 0.78 0.77

CriminalAttitudes/Cognitions

0.79 0.82 0.82 0.77 0.76 0.78

CriminalPersonality

0.72 0.75 0.76 0.75 0.68 0.67

RiskofFailuretoAppear(FTA)

0.72 0.76 0.76 0.72 0.70 0.66

RiskofViolence

0.74 0.72 0.71 0.69 0.72 0.71

aAlphasarenotavailablefortheRiskofTechnicalViolenceandRiskofRecidivismscalesbecausethesescalesarelinearequations.bPre‐SentencingInvestigationcPrison/parole/probation/pretrial/jail/communitycorrections



2.2 ReliabilityacrossGenderbreakdownsToassess if theCOMPASscalesareequally reliable formalesand females,additionalalphasare provided by gender for probation (Table 2.2.1) and parole (Table 2.2.2) samples.Confidence intervals of the difference in alphas between females and males were alsocomputedtoevaluateforsignificanceofsuchgenderdifferences.Forallsites,thevastmajorityoftheCOMPASscaleswerefoundtobeequallyreliableforbothgenders. Few scales revealed gender differences in alphas. Thesewere theHistory of Non‐ComplianceandSocialEnvironment/NeighborhoodscalesintheDallasCountydata,theHistoryofViolenceandRiskofViolence(whichincludesitemsfromtheHistoryofViolencescale)scalesinSanBernardino,andtheLeisure/Boredom,ResidentialInstabilityandSocialIsolationscalesin Wyoming’s combined data. Within the California parole sample, we found differencesbetweenmalesandfemalesfortheCriminalPersonalityandRiskofViolencescales.NogenderdifferencesregardingscalereliabilitywerefoundintheMichiganparolesample.Overall, thescaledifferencesinalphasacrossgendersareinconsistentfromsitetosite.



Table2.2.1:COMPASScaleReliability–MalesandFemalesinProbationandCombinedSamplesScaleReliabilityasindicatedbyCronbach’sAlpha

Dallas CountyCourts’PSIb

San BernardinoProbationAssessments

WyomingCommunity/Incarcerated

COMPASScales

Female Male Female Male Female Male

CriminalInvolvement

0.90 0.89 0.91 0.90 0.88 0.85

HistoryofNon‐Compliance 0.40* 0.57* 0.62 0.62 0.65 0.61

HistoryofViolence

0.72 0.69 0.57* 0.72* 0.60 0.67

CurrentViolence

0.69 0.67 0.64 0.61 0.68 0.70


0.70 0.70 0.75 0.77 0.77 0.75

SubstanceAbuse

0.79 0.78 0.78 0.79 0.69 0.70


0.72 0.72 0.68 0.72 0.75 0.74


0.70 0.69 0.65 0.69 0.67 0.66

FamilyCriminality

0.60 0.62 0.61 0.63 0.64 0.67


0.77* 0.83* 0.79 0.81 0.73 0.78

Leisure/Boredom

0.82 0.79 0.79 0.79 0.88* 0.83*


0.62 0.64 0.67 0.69 0.63* 0.71*

SocialAdjustment

0.64 0.59 0.56 0.60 0.62 0.62


0.69 0.71 0.66 0.72 0.73 0.71

CriminalOpportunity

0.68 0.65 0.62 0.63 0.71 0.68

SocialIsolation

0.81 0.78 0.81 0.80 0.87* 0.83*


0.76 0.79 0.79 0.82 0.80 0.82

CriminalPersonality

0.72 0.71 0.72 0.77 0.73 0.76

RiskofFailuretoAppear(FTA)

0.72 0.72 0.77 0.77 0.76 0.77

RiskofViolence 0.72 0.74 0.65* 0.72* 0.67 0.70

*Thedifferenceinalphasbetweenmalesandfemalesisconsideredsignificantifthe95%confidenceintervaldoesnotincludezero.bPre‐SentencingInvestigationcPrison/parole/probation/pretrial/jail/communitycorrections



Table2.2.2:COMPASScaleReliability–MalesandFemalesinParoleSamplesScaleReliabilityasindicatedbyCronbach’sAlpha

MIPre‐Release

Assessments/winmates

CAPre‐ReleaseAssessments

COMPASScales

Female Male Female Male

CriminalInvolvement

0.82 0.86 0.81 0.79

HistoryofNon‐Compliance

0.70 0.65 0.54 0.58

HistoryofViolence

0.61 0.65 0.73 0.68

CurrentViolence

0.66 0.59 0.69 0.67


0.70 0.76 0.80 0.81

SubstanceAbuse

0.78 0.78 0.75 0.71


0.78 0.76 0.66 0.71


0.69 0.70 0.66 0.69

FamilyCriminality

0.64 0.61 0.59 0.62


0.86 0.87 0.83 0.81

Leisure/Boredom

0.85 0.86 0.82 0.81


0.70 0.67 0.71 0.71

SocialAdjustment

0.60 0.59 0.54 0.53


0.68 0.68 0.68 0.68

CriminalOpportunity

0.72 0.70 0.69 0.68

SocialIsolation

0.84 0.84 0.80 0.78


0.74 0.77 0.72 0.76

CriminalPersonality

0.75 0.76 0.73* 0.67*

RiskofFailuretoAppear(FTA) 0.70 0.74 0.74 0.69

RiskofViolence 0.67 0.66 0.76* 0.69*

*Thedifferenceinalphasbetweenmalesandfemalesisconsideredsignificantifthe95%confidenceintervaldoesnotincludezero.



2.3 InternalStructuralValidation:FactorialValidityEachCOMPASsubscalewasdevelopedasaunidimensional linear function. Ingeneral, itemswereselectedtobeboththeoreticallyrelatedtoacertainconstructandtoloadheavilyonthesame“component”or factor.The factorpatternsarebasedon rotatedprincipal componentanalysesforeachscale.Theimportanceofeachscaleitemisindicatedbythesizeofthefactorloading.Alargeloadingbyanitemonthefirstcomponent(i.e.,0.4andhigher)indicatesthattheitemisanimportantcontributor to the scale.Given that several itemswith strong loadingsare includedoneachscale, large eigenvalues for the first factor, or component are achieved. A large eigenvalue(greaterthan1)indicatesthatthefactoraccountsforalargeamountofoverallvarianceinthesummated score, which, in turn, suggests that the scale is measuring a single underlyingconstruct.Due to this scaledevelopment strategy, factorial validityof theCOMPAS scales isensured. A comparison of the factorial validity across different sites reveals only minorvariations(fordetailsseeCOMPASTechnicalManual).Toprovideoneexample,theCriminalInvolvementscaleispresented.Itconsistsoffouritemsindicating the number of: previous jail times “n.jails,” previous arrests, “n.prev.arrest,”previousconvictions,“n.prev.convict,”andpreviousepisodesonprobation,“n.probations.”The factor loadings of the Criminal Involvement scale are shown in Table 2.3.1 and theEigenvalues inTable2.3.2.Thisscale isclearlyunidimensionalwitha largeeigenvalueof thefirstcomponent(PC1)3.05,accountingfor76.2%ofthevarianceofthe itemset.Thestrongfactor‐loading pattern on the first component confirms the importance of all items in thescale.Table2.3.1:PrincipalComponentPattern:CriminalInvolvementinDallasCountyProbation.Items PC1 PC2 PC3 PC4

n.jails 0.495 0.488 0.653 −0.301n.prev.arrest 0.527 0.215 −0.192 0.799n.prev.conv 0.518 0.058 −0.676 −0.520n.probations 0.456 −0.844 0.282 −0.006Table2.3.2:EigenvaluesandVarianceExplainedinDallasCountyProbationData. PC1 PC2 PC3 PC4

Eigenvalue

3.047 0.483 0.283 0.188

StandardDeviation

1.745 0.695 0.532 0.433

ProportionofVariance

0.762 0.121 0.071 0.047



3.Criterion/ConcurrentValidityAs part of our standard psychometric reports, a section on criterion related “concurrent”validity is included. In addition to correlations with criminal history indicators, ReceiverOperator Characteristics are provided below, evaluating the accuracy of the scales indiscriminatinglowandhigh‐riskcases.3.1 ConcurrentValidity:CorrelationswithCriminalHistoryIndicatorsAcommonapproach toassessing theconcurrentvalidityofa scale is todeterminewhetherthe scale scores are correlated with the criterion behavior of interest such as Age‐at‐firstArrest,orNumberofPriorArrests.Twotablesareprovided(Table3.1.1and3.1.2)toexamineconcurrent validity separately for offenders on probation and parole. Shaded cells indicatesignificantcorrelations.For this concurrent, correlational approach, it is important to make sure that the criterionvariableisitselfnotincludedasanitemonthescale.Inthissection,SpearmanorPearson(forthe Michigan and Georgia samples) correlations of the COMPAS scales against selectedcriminal justice and criminological criterion variables (e.g. Age‐at‐first Arrest, Number ofPreviousArrests,etc)arepresentedacrossdifferentstudysites.PatternsofcorrelationsemergeattheprobationandparolesitescomparedinTables3.1.1and3.1.2. For example, Age‐at‐first Arrest correlates negatively with the personality scalesCriminal Personality and Criminal Attitudes. These results comport with findings indevelopmental research indicating offenders with early onset are more likely to have highscores on similar types of personality measures and more serious and persistent criminalinvolvement(Moffitt,1993).Theobservationalsocorrespondswiththepositivecorrelationofthe Criminal Personality scale and Total Number of Previous Arrests. Similar to the above,offenders with earlier Age‐at‐first Arrest are more likely to have higher scores on scalesmeasuring factors that have been identified as criminogenic in longitudinal criminal justiceresearch studies. These scales includeSocial Environment,Vocational/Educational Problems,andFamilyCriminality(Farringtonetal.,2001).Another pattern evident for probation and parole populations (Tables 3.1.1 and 3.1.2) isdefinedby the correlationsofPreviousArrests aswell asProbation/ParoleRevocationswiththescalesCriminalAssociates/PeersandSubstanceUsewhichrepresentrisk factors thatarehighly relevant for predicting criminal involvement as established by research (Stouthamer‐Loeberetal.,2002;Cottle,Lee&Heilbrun,2001).



Table 3.1.1: Spearman Correlations of COMPAS Sub‐scales with selected Criminal HistoryIndicatorsinProbationandCombinedSamples

Probation Age‐at‐First PreviousArrests ProbationRevocation

Dallas

SanBernar‐dino

Wyoming

combinedsample*

Dallas SanBernar‐dino

Wyoming

combinedsample*

Dallas SanBernar‐dino

Wyoming

combinedsample*

Scale N=1,148 N=1,047 N=988 N=1,148 N=1,047 N=988 N=1,148 N=1,047 N=988

CriminalAssociates

‐0.26 ‐0.14 ‐0.27 0.24 0.32 0.22 0.19 0.25 0.17

SubstanceAbuse

‐0.02 ‐0.04 ‐0.18 0.25 0.34 0.29 0.19 0.34 0.19

FinancialProblems

‐0.02 0.01 0.02 0.04 0.17 0.01 0.07 0.14 ‐0.03

VocationalEducational

‐0.29 ‐0.15 ‐0.26 0.09 0.19 0.12 0.12 0.15 0.12

FamilyCriminality

‐0.12 ‐0.09 ‐0.20 0.05 0.14 0.14 0.08 0.12 0.13

SocialEnvironment

‐0.17 ‐0.11 ‐0.18 0.07 0.17 0.14 0.06 0.19 0.08

Leisure/Boredom

‐0.10 ‐0.06 ‐0.24 0.03 0.18 0.10 0.06 0.17 0.11


‐0.03 ‐0.03 ‐0.10 0.05 0.04 0.14 0.02 0.07 0.10

SocialIsolation

0.01 ‐0.03 ‐0.07 0.09 0.18 0.12 0.07 0.12 0.07

CriminalAttitudes

‐0.13 ‐0.07 ‐0.17 0.01 0.10 0.06 0.01 0.12 0.06

CriminalPersonality

‐0.18 ‐0.11 ‐0.29 0.10 0.22 0.22 0.02 0.20 0.16

Note:WithN=1148,acorrelationof.058issignificant,withN=1047,acorrelationof.06issignificant,andwithN=988,acorrelationof.063issignificantatp<.05(2‐tailed).*Includingprison/parole/probation/pretrial/jail/communitycorrectionscases.

WhilesomeofthesegenerallyrobustpatternswerenotconfirmedintheMichiganparoledata(Table3.1.2),overall,theobservedrelationshipsbetweentheCOMPASsubscalesandcriminalhistoryindicatorsdataprovideevidenceforcriterion‐related,concurrentvalidityofthescales.Theirgeneralconsistencyacrossdifferentsitesanddatasets,addsadditionalempiricalsupportto this conclusion. Interestingly, nomajor differences in correlational patternswith criminalhistory indicators were found between offenders on probation and parole. On the whole,significantcorrelationsare lesscommonamongstparolees,while thedominantpatternsarethesamebetweenthetwocomparisongroups.



Table 3.1.2: Spearman/Pearson Correlations of COMPAS Sub‐scales with Selected CriminalHistoryIndicatorsinParoleSamples.

Parole Age‐at‐First PreviousArrests ProbationRevocation

MI CA*

GA MI CA* GA MI CA* GA

Scale N=920 N=785 N=3,809 N=920 N=785 N=3,809 N=920 N=785 N=3,809CriminalAssociates

−0.20 ‐0.20 −0.18 0.04 0.15 0.04 −0.20 0.07 0.04

SubstanceAbuse

−0.05 0.04 0.03 0.17 0.27 0.21 −0.05 0.19 0.15

FinancialProblems

−0.01 ‐0.03 0.01 0.08 0.11 0.10 −0.01 0.12 0.14

VocationalEducational

−0.23 ‐0.24 −0.28 0.03 0.04 −0.03 −0.23 0.05 0.01

FamilyCriminality

−0.13 ‐0.09 −0.09 0.03 0.05 0.03 −0.13 ‐0.05 0.06

SocialEnvironment

−0.17 ‐0.13 −0.10 −0.10 0.08 −0.06 −0.17 0.10 −0.03

Leisure/Boredom

−0.04 ‐0.08 −0.07 0.06 0.08 0.01 −0.04 0.06 0.01


0.01 ‐0.08 0.00 0.01 0.02 0.02 0.01 0.10 0.01

SocialIsolation

−0.15 ‐0.02 −0.01 0.06 0.10 0.04 −0.15 0.04 0.05

CriminalAttitudes

0.03 ‐0.17 −0.14 0.09 0.06 −0.06 0.03 0.11 −0.01

CriminalPersonality

−0.12 ‐0.19 −0.16 0.07 0.12 0.00 −0.12 0.07 0.03

With N=920 a correlation of .065 is significant, with N=785, a correlation of .07 is significant, and with N=3809, a correlation of .032 issignificant at p < .05 (2‐tailed). * For California Spearman correlations are presented, forMichigan andGeorgia Pearson correlations areprovided.

3.2 ConcurrentValidity:ReceiverOperatorCharacteristics(ROC)ForaNewYorkProbationsample,concurrentvaliditywasalsoevaluatedbymeansofReceiverOperatorCharacteristics(ROC).ThismethodcanbeappliedtoassesshowaccuratelytheriskscalesdiscriminatebetweenRecidivists andNon‐Recidivists. ROCanalysis produces aplotofthe“sensitivity”ofthescaleagainstoneminusthe“specificity”ofthescale,calledthe“ROCcurve.”Sensitivityisthepercentageoffailurescorrectlyclassified,alsocalledthe“truepositiverate,”andspecificityisthepercentageofsuccessescorrectlyclassified.Oneminusspecificityisthe false positive rate, in other words the percentage of offenders classified as Recidivistsalthoughtheyarenotrecidivating.Accordingly,theROCcurveisaplotofthetruepositiverateagainstthefalsepositiverateforeachcut‐pointofthescale.TheAreaUndertheCurve,(AUC)givesameasureoftheaccuracyofthescaleindiscriminatingbetweentrueandfalsepositiverates. Larger areas under the curve indicate better accuracy. The AUC indicator can beinterpretedastheprobabilitythatarandomlyselectedRecidivistwillhaveahigherriskscore



than a randomly selected Non‐Recidivist. An AUC of .70 or above indicates a strongassociation,whilemeasuresbetween.60and.70indicateamoderateassociationaccordingtorecentresearchinthefieldofoffenderclassification(Quinseyetal.,1998,Jones,1996,Aos&Barnoski2003).FortheROCanalysisofthescalesRiskofCommunityNon‐Compliance,RiskofFTA,andRiskofViolence the outcome variableswere excluded from the scales and used to test concurrentvalidity without true follow‐up data. For this purpose, these outcome measures weredichotomizedsothat0representsnofailureand1represents1ormorefailures.AspresentedinTable3.2.1,theAUCmeasuresofall threescalesareabovethe.70thresholdforastrongassociation, suggesting these scales have high concurrent validity. However, since morerealisticoutcomemeasureswereunavailable,itisimportanttobecautiousaboutgeneralizingthese results on concurrent validity to predictive validity. The AUC values may be biasedupward because of undesirable correlations between the scale items and the outcomevariablesselectedfortesting.Suchcorrelationscanbeduetoameasurementeffect, ifscaleitemsandoutcomevariablesusedfortheanalysisweremeasuredwiththesameinstrumentatthesamepointintime.Table3.2.1:COMPASConcurrentValidity–AreaundertheCurve

ConcurrentValidityasIndicatedbyAUC’s

COMPASScales

NewYorkProbation

SampleSize n=393

Risk‐CommunityNon‐Compliance(CNC) 0.79n.prob.rev*

RiskofFailuretoAppear(FTA)0.82n.fta*

RiskofViolence 0.71Violence**

*ThisItemwasremovedfromthescaleanddichotomized(0=nofailure;1=1ormorefailures).**This Item isacombinationof items thatwere removed fromthescaleand indicates if thecurrentoffense involvedviolenceat timeof the

assessmentinterview(0=no1=violent).



4.PredictiveValidity—SurvivalAnalysisSeveralagencyor regional‐specificstudieshavebeenconducted inwhichthebasicCOMPASscales or the risk prediction scales (Recidivism Risk, Violence Risk, FTA Risk, and Risk ofTechnicalViolation)havebeenevaluatedforpredictivevalidity.Techniquesofsurvivalanalysishave been used to estimate “Proportional Hazard” models. These models were selectedbecausetheynotonlyexaminetheoccurrenceofarecidivismeventbutalsothetimingofthatevent.Thissectionwillreviewthefindingsofthesestudiesandofferconclusionsregardingthegeneralizability of the predictive models that have been evaluated on different criminaloffenderpopulations(Probationers,Parolees,Inmates).

4.1 CommunityCorrectionsOhioA study in Ohio evaluated the utility of COMPAS risk scales in predicting communitycorrections program failure. The study sample was a subset of inmates (N=493) who wereassessedwiththeCOMPASinstrumentfromJanuary2002throughJuly2005andparticipatedin a community corrections program that places inmates at government and non‐profitagenciestoperformcommunityservice.

Programfailurewasusedastheoutcomeofinterestandprogramsuccessasthe“competing”event. Although itmaynot be readily apparent, theseoutcomes represent competing risks.Theanalyticapproachshouldadjustforthefactthatsubjectswhoexperiencethecompetingeventofprogramsuccessarenolongeratriskofprogramfailure,andviceversa.Therefore,aspecializedproportionalhazardssurvivalmodelforcompetingriskswasapplied(Fine&Gray,1999).

Therelative failureprobabilitiesofoffendersonthehigh,mediumand lowrisk levelsof theCOMPAS risk scales were compared using Hazard Ratios. Since the risk levels are coded asdummyvariables,theseHazardRatioscanbeinterpretedasmultipliersoftheriskor“hazard”ofprogramfailureforaparticulargroup(e.g.highriskoffenders)comparedtothereferencegroup(here:lowriskoffenders).

Results fromsurvivalmodelsaspresented inTable4.1.1showthat theRecidivism,FTA,andViolenceriskscalesarestrongpredictorsforthecriterionofprogramfailure.InmatesscoringhighontheRecidivismRiskscaleforexample,hadariskofprogramfailureor“hazard”thatisthreetimeshigherthanthefailurehazardof inmatesscoringlowrisk.Whiletheseincreasedhazard ratesofhigh compared to low riskoffenderswerenot as large for theViolence (2.4timeshigher)andFTA(1.9timeshigher)riskscales,asignificantdifferencewasfoundforthesescales as well. On the Technical Violation scale, medium and high risk offenders were notsignificantlydifferentfromoffendersclassifiedaslow.

ThefindingsdemonstrateclearlythattheCOMPASriskscalesarevalidtoolsfordeterminingeligibility for one of the core community corrections programs at this corrections center inOhio.



Table 4.1.1: Competing HazardsModel Regressing Program Failure on COMPAS Risk Scales:HazardRatiosandp‐values.N=493 ProgramFailure

Scales RiskLevel HazardRatio p‐valueMedium 0.96 0.872Technical

Violation High 1.42 0.268 HazardRatio p‐value

Medium 1.39 0.182RiskofFTA High 1.94 0.008 HazardRatio p‐value

Medium 1.76 0.017RiskofViolence High 2.36 0.043 HazardRatio p‐value

Medium 1.75 0.034RiskofRecidivismHigh 3.00 0.000

Note.Thereferencecategoryforthetestofmediumandhighriskcategoriesislowrisk.

4.2 NewYorkProbation—ScaleConstructionAStudyinaNewYorkProbationsampleexaminedthepredictivevalidityofseveralrecidivismriskscales.Theoutcomeorcriterionbehaviorofinterestforthisevaluationwasrecidivismormorespecificallya“NewCrime.”During 1999 COMPAS (Version 2) assessment data for offenders (n=513) was collected atcorrectional institutions of four sites in different counties in New York (Fulton, Monroe,Schenectady, and Suffolk). Subsequently, independent two‐year follow‐up data was madeavailableforoffendersinthisCOMPASsamplefromtheStateCriminalJusticeDatabase.Many of the offenders in the original assessment samplewere restricted fromentering thecommunityaftertheirCOMPASscreeningdate.Forexample,someweregivenprisontime.Forthisreason,103caseswithprisonsentenceswereexcludedfromthesample.Seventeencaseswithmissingdispositionswerealsoexcluded.Theremainingsampleiscomposedofprobation,jail, conditional discharge, and split sentence cases (n=393). Theseoffenderswere generallyunconstrainedfromenteringthecommunity.Inotherwords,theywere“atrisk”torecidivate.Althoughoffenderswithjailorsplitsentenceswereconstrainedforcertainunknownperiodsoftime,theywerestillconsidered“atrisk”torecidivate,sincetheseperiodsofdetentionwererelativelyshort.Thisassumptionwasfurtherconfirmedbytheirrelativelyhighrecidivismratesduringthetwo‐yearfollow‐up(e.g.jail55%,splitsentence39%).The follow‐updata indicated,whether anoffenderwas accusedof committing anewcrimeand the date on which this alleged crime occurred. Based on this information the twooutcomes variables “New Crime” and “Days until New Crime Occurred” were calculated.Predictive models of recidivism were developed with several different “candidate sets” of



variables using standard logistic regression. For the purpose of this report, a ProportionalHazardmodel based on techniques of survival analysis and a final logistic regressionmodelwithasimilarfinalsetofpredictorvariablesarepresentedbelow.TheCoxProportionalHazardmodelpresentedinTable4.2.1isbasedonalargecandidatesetofpredictorvariables,includingalloftheCOMPASbasicscalesandthethreeadditionalitemsAge, Age‐at First Arrest, andArrest Rate. This large set of variables was reduced using theAkaike Information Criterion (AIC) model selection procedure. AIC is a stepwise modelselection procedure, which reduces the number of predictor variables, similar to stepwiseregressionprocedures,accordingtothestrengthoftheireffect(VenablesandRipley,1999).TheCOMPASscalesaimatmeasuring risk,need,andmediating factorsof recidivism,hencetheywereallincludedinthismodel.ForamoredetaileddescriptionofthescalesincludedinthemodelseeAppendixA.Moreover,previousresearchhasshownthattheageofanoffenderatthetimeofthearrestisanimportantpredictorofrecidivism.Thenumbersofarrestsoverthe course of a criminal career are a good indicator of prior criminal involvement and alsopredictrecidivism.Therefore,thesevariableswereincludedinthemodel.ThevariableAge‐atFirst Arrest represents the age atwhich the offenderwas first arrested. The variableArrestRatestandsfortheaveragenumberofarrestsperyearexperiencedbytheoffenderpriortotheassessmentdate.Theeffectsofcovariatesontheriskor“hazard”ofrecidivismcaneasilybeinterpretedbasedontheantilogarithmoftheirestimatedcoefficient“exp(coeff),”alsocalledthe“alphaeffect”(TumaandHannan1984).Thisalphaeffectindicatesthepercentagechangeintheestimatedhazard rate if the value of the covariate is increased by one unit (change of hazardrate=(exp(α)–1)*100%). It takesthevalue1whenthecovariatehasnoeffect, it issmallerthan1whentheeffectisnegative,andit isgreaterthanonewhentheeffectispositive.Forexample,thealphaeffectoftheCriminalInvolvementscale(seeTable4.2.1)of1.276indicatesa27.6%increaseofthehazardtorecidivateforeachincreaseofthescale’sscorebyoneunit.Aunitonthisscalestandsforapriorinstanceofarrest,probation,conviction,orjailtime.TheAICmodelselectionprocedurewasappliedafterregressingthevariable“DaysuntilNewCrime Occurred” onto the predictors from the candidate set. “New Crime” served as the“censoring”variable,whichreferstothe“event”thatconstitutestheendofthetimelineforthis survival model.3 It is important to stress that the accuracy of “Days until New CrimeOccurred” can be questioned, especially for the offenders in the “jail” category, who wereconstrainedforunknownperiodsoftime.Thefindingsprovidereasonableevidencefor thepredictiveutilityofseveralof theCOMPASscales.Table4.2.1liststhesignificantpredictorvariablesthatremainedinthemodelaftertheAICselectionprocedure.

3ForanoverviewofsurvivalanalysisseeHosmerandLemeshow,(1998).



Table 4.2.1: Cox Proportional HazardModel Regressing Recidivism on COMPAS Risk Scales:RegressionCoefficientsandp‐values. Recidivism

PREDICTORS Coeff EXP(Coeff) P>|z|

CriminalInvolvement 0.244 1.276 1.4e‐02

FinancialProblems 0.331 1.392 8.5e‐02


0.427 1.532 1.3e‐02

UseofLeisureTime ‐0.364 0.695 2.4e‐02

ResidentialInstability 0.240 1.271 7.0e‐02

SocialAdjustment ‐0.475 0.622 4.3e‐02

CriminalAttitudes ‐0.275 0.774 1.1e‐01

CriminalPersonality 0.401 1.494 1.5e‐02

Ageat1stArrest ‐1.414 0.243 1.7e‐04

ArrestRate 0.611 1.842 3.6e‐05

The higher offenders score on the Criminal Involvement, Financial Problems,Vocational/Educational Problems, Residential Instability, and Criminal Personality scale, thehighertheirrisktorecidivate.Asoftenoccurswhenvariablesarecollinear(closelycorrelated),someofthesignsoftheparametersareinunexpecteddirectionssuchasfortheUseofLeisureTime,SocialAdjustment,andCriminalAttitudesscales.Theestimatedalphaeffectsindicatechangesofthehazardforeachincreasebyoneunit(scalescore) of the predictor variables. The higher an actual offender’s score, the higher theircumulative increase of the hazard per unit compared to low scores.Hence, COMPAS scaleswithpositiveeffects represent a substantially increased risk to recidivate foroffenderswithhighscores.As indicated by the negative coefficient of Age‐at First Arrest, an early onset of criminalinvolvement,predictsanincreasedrisktorecidivate.Theaveragenumberofarrestsperyear(Arrest Rate) can be seen as an indicator for a criminal history prior to the COMPASassessment. With every additional prior arrest per year, the hazard rate for recidivismincreasesby84%.Accordingtothep‐values,alleffectsaresignificantlydifferentthanzero.A comparable logistic regressionmodel predicting recidivism (derived froma reduced initialsetofvariablesenteringtheAICmodelselectionprocedure)includesanalmostidenticalfinalsetofpredictorvariablesasshowninTable4.2.2.Thisfinal,simplermodelwasselectedandemployed as the risk equation of the COMPAS Recidivism Risk scale. All effects are in the



expected direction. For a more detailed description of the predictor variable selection andmodel fittingprocess fortheRecidivismRisk scalepleaseseetheCOMPASTechnicalManualandPsychometricsReport.The results of the logistic regression (Table 4.2.2) are very similar to the Hazard modelindicating an increased risk of recidivism the higher offenders score on the CriminalInvolvement,Vocational/EducationalProblemsorDrugProblemsscale.Also,thehighertheiraverage numbers of prior arrests per year (Arrest Rate) the more likely offenders are tocommit anewcrime. In thismodel,Age‐at FirstArrest andCurrentAge are included,bothdisplayingeffectsintheexpecteddirection—theolderoffenderswereattheirfirstarrestoratthetimeoftheassessment,thelowertheirriskofrecidivism.AsopposedtotheHazardmodel,thefinalsetofthelogisticregressionincludesanadditionalDrugProblemsscale.Thisscalesummarizesindicatorswhethertheoffenderhashadproblemswithdrugs fromtheoffensehistory,currentcharges,and thecurrentarrest, incombinationwith information regarding a history of drug treatment or an expressed current drugtreatmentneed(formoredetailsseeAppendixA).Table4.2.2:PearsonCorrelationswithNewCrime(r)andLogisticRegressionModelPredictingNewCrime(PenalizedOddsRatios)PREDICTORS r OddsRatio(Penalized)

Intercept ‐0.35672CriminalInvolvement 0.20 0.26443Vocational/EducationalProblems

0.22 0.28935

Age ‐0.18 ‐0.04567Ageat1stArrest ‐0.28 ‐1.30161ArrestRate 0.29 0.46997

DrugProblems 0.22 0.54287

The sample selectionmayhave causeda samplebias sinceoffenderswithprison sentenceswhoaremorelikelytobeseriousoffenderswereexcluded.Inaddition,someoffenderswerenot“atrisk”torecidivatebutconstrainedforunknownperiodsoftime(jailorsplitsentences).However, both these sample selection issues make it less likely to detect differences inrecidivismbetweenlowandhighriskoffendersbecausethey limitthevarianceofrisk inthesample.Hencethesamplebiasisnotinfavorofthetestedhypothesis,thatthescaleshavethepotentialtodiscriminatebetweenlowandhighriskoffenders.Basedonthisassumption,thisstudycanbeseenasarather“conservative’testofthepredictivevalidityoftheCOMPASriskscalesandthefindingsareconvincing.



ReceiverOperatorAnalysisofRecidivismRiskScaleThelogisticregressionbasedonareducedsetofcovariatesaspresentedinTable4.2.2,leadtothe development of the Recidivism Risk Scale in the New York Probation sample. ThecomponentsoftheRecidivismRiskScalearetheCriminalInvolvement,VocationalEducationalProblems, and Drug Problems scales, together with the offender’s age at the time of theassessment,at the timeof the firstarrest,and theaveragearrest rateperyearprior to theassessment.Chart 4.2.1 below illustrates the relationship between the Recidivism Risk scale and theprobability of recidivism (committing a new crime) for the New York Probation sample(describedabove).Theriskofrecidivismclearlyincreasesacrossquartilesofthescale.The Risk of Recidivism scale was further validated applying ROC analysis in the New YorkProbationdata,basedontheoutcomemeasure“NewCrime”;hencetheROCanalysisinTable4.2.3 below evaluates thepredictive validity of this scale. The values in the column labeledAUC*areestimatesofAUCthathavebeencorrectedforbiaswithbootstrapvalidation.Thecorrectionaccountsforthevalidationbeingperformedinthescaleconstructionsample.Thescalemayperformlesswellinanewsample.Thefindingsarepositive,withanAUCof.74andacorrectedAUCof .72, indicatinghighpredictivevalidityoftheRiskofRecidivismscale.TheAUCvaluesaresimilarinmagnitudetothoseobtainedbyotherresearchersinthefield(Cottle,Lee,&Heilbrun,2001;Gran,Belfrage,&Tengstrom,2000;Quinseyetal.,1998);andhigherthanforcomparablefourthgenerationriskassessmentinstruments(Barnoski&Aos2003).Chart4.2.1:RecidivismRiskScaleandRecidivismRate



Table4.2.3:COMPASPredictiveValidity–AreaundertheCurve

COMPASScale AUC AUC*

RiskofRecidivism0.74New Arrest(within2yrs.)

0.72*New Arrest(within2yrs.)

*Correctedforbiasbybootstrapvalidation.

4.3 NewYorkProbation—CrossValidationAsecondrecidivismstudywasconductedtodeterminehowpredictivetheCOMPASRecidivismRiskscaleisinadifferent“validation”sample,astepthatisalsocalled“crossvalidation.”Therisk predictionmodel, as derived from the scale construction sample described above (seeTable4.2.2),isappliedto2,328offendersinNewYorkthatwereassessedwithCOMPASatthepointofPre‐SentencingInvestigation(PSI)orprobationintake.Thedatawerecollectedin19counties that administer the complete COMPAS assessment. The assessments wereconductedfromJanuary2000throughDecember2004.The COMPAS assessment data were matched with Computerized Criminal History (CHH)recordsby theNewYorkStateDivisionofCriminal JusticeServices. Multiple‐recordsurvivaldatasetswereconstructedusingassessmentdatesandeventdatesintheCCHdataincludingcrimedates,arrestdates,dispositions,dispositiondates,sentencetype,andsentencelength.Threetypesofrecidivismwereexamined includingAnyOffense,PersonOffense4,andFelonyOffense.SeparateCoxregressionmodelswerefittedforeachtypeofrecidivism.Thesemodelsdonotjustpredictfailurebutthetimeuntilarecidivismeventoccursstartingfromthetimeofthe assessment. Moreover, these models control for periods of incarceration during thefollow‐uptimebyremovingthesubjectfromtherisksetsduringsuchgaps,sincetheyarenot“atrisk”ofrecidivismwhileincarcerated.Results of receiver operator characteristics analysis (AUCs) for the COMPASRecidivismRiskCoxregressionmodelpredictinganyoffense,offensesagainstpersons,andfelonyoffensesarepresented inTable4.3.1below.Allof theAUCsare indicatingamoderate tohighpredictivevalidity.ItshouldalsobepointedoutthattheAUCsinthevalidationsampleareonlyslightlysmaller than those in the scale construction sample. Therefore theCOMPASRecidivismRiskprediction model can be considered robust. This also indicates that the risk model can begeneralizedtootheroffenderpopulationswithlittlelossofpredictiveaccuracy. 4Therecidivismtype“personoffense”isdefinedasafingerprintablearrestinvolvingachargeforanyCUCRcodeformurder,forciblerape,robbery,aggravatedassault,simpleassault,burglary(withweaponoroccupieddwelling),dangerousweapons,sexoffenses,extortion,arsonorkidnapping.Thiscategoryincludesmisdemeanorandfelonyoffenses.



Table4.3.1:COMPASPredictiveValidity–AreaundertheCurveinValidationSample AUCs

totalsample(n=2,328) AUCs

women(n=449) AUCs

men(n=1,879) any person felony any person felony any person felonyRiskof Recidivism* .68 .71 .70 .65 .76 .66 .68 .70 .71

*TheRecidivismRiskscaleiscodedasdecilesinthismodel.

4.4 CaliforniaParoleAnoutcomesstudy ina largesample (n=20,898)ofparolees in theCaliforniaDepartmentofCorrections and Rehabilitation (CDCR) examined the predictive validity of the COMPAS riskscales(Brennan&Dieterich,2007).TheparoleeswerereleasedontoparoleduringtheperiodMarch 2006 through June 2007. In this study the COMPAS Risk Matrix‐R and its twocomponents, the Violence‐R Risk scale and Recidivism Risk scale, were evaluated for theirutility in predicting parole failure. Parole failure is defined as a return to prison (RTP) for anontechnicalparoleviolation.Sinceoffenderscanreturntoprisonforotherreasonssuchastechnicalviolations,whileotherscan be discharged from parole, these events were modeled as “competing risks.” Thisapproach adjusts for the fact that subjectswho experience these competing events are nolongerat riskof returningtoprison foranontechnicalviolation.Totakethis intoaccount,aspecializedproportionalhazardssurvivalmodelforcompetingriskswasapplied(Fine&Gray,1999).TheRiskMatrix‐Rconsistsoftworiskdimensions;thefirstisdefinedbytheRiskofViolence‐Rdecilescore,andtheseconddimensionbytheRecidivismRiskdecilescore.TheRiskMatrix‐RasevaluatedinthisstudydiffersfromtheoriginalCoreCOMPASRiskMatrixinthatitincludestheViolence‐Rscale,anadaptationoftheViolencescalespecificallyforparolepopulations.Forthepurposeofthissynthesisofvalidationfindings,theevaluationsoftheRecidivismRiskscaleandtheRiskMatrix‐Rarepresented.Asanillustrationofthebasic logicofsurvivaltechniquesandtheresultingmodel,thechartsbelowdepictfailureratesovertimebyrisklevel.ThisisdonebyplottingthepredictedvaluesfromaregressionofthecumulativeincidencesofRTPonthecovariate(i.e.,thelevelsoftheRecidivismRiskScaleoroftheRiskMatrix‐R(Fine&Gray,1999).Chart4.4.1showsaplotofthefittedfailureprobabilitiesforthethree levelsofRecidivismRisk,andChart4.4.2doessofor theRiskMatrix‐R. As depicted for theRecidivism Risk scale (Chart 4.4.1), the three risklevelsdisplaysubstantialdifferences intheirfailuredynamics.Forexample,offendersonthehigh‐risk level reach a failure probability rate of almost 10%within 100 days upon releasecomparedtomuchlowerratesofthetwocomparisongroupsatmediumorlowrisk.WhilethechartfortheRiskMatrix‐R(Chart4.4.2)isinterpretedinthesamemanner,fourrisklevelsaredisplayed here. Offenders on all four risk levels reveal major differences in their failuredynamicsovertime.



Chart4.4.1:PredictedCumulativeIncidenceofNontechnicalViolationReturnstoPrison(RTP)fortheRiskLevelsoftheRecidivismScale.

Chart4.4.2:PredictedCumulativeIncidenceofNontechnicalViolationReturnstoPrison(RTP)fortheRiskLevelsoftheMatrix‐R.



ResultsfortheRecidivismRiskscalefromcause‐specificCoxproportionalhazardsmodelsarepresentedinTable4.4.1.SignificantdifferencesbetweenoffendersonthedifferentrisklevelsoftheRecidivismscalewerefoundregardingtheriskofparolefailure(here:areturntoprisonfor a nontechnical parole violation). The cause‐specific hazard for parole failure is 3.91 forhigh‐riskoffenders.Thisindicatesthatinmatesclassifiedashigh‐riskhaveafailurehazardthatisabout4timeshigherthanthehazardfor inmatesclassifiedas lowrisk. Incomparison,thehazardforthemediumcategoryrelativetothelowcategoryis2.25.Table4.4.1:ProportionalHazardModelRegressingReturntoPrisonforaNontechnicalViolationontheRecidivismRiskScale:RecidivismRiskLevel

Coeff.

SE

p‐value

HazardRatio

Lower95%CI

Upper95%CI

Medium 0.813 0.074 <.001 2.25 1.95 2.61High 1.364 0.068 <.001 3.91 3.42 4.47

Note.Thereferencecategoryforthetestofmediumandhighriskoffendersisthelowriskcategory.

Additionalreceiveroperatingcharacteristics(ROC)analysisfortheRecidivismRiskscaledecilescoreyieldsanAUCof.66.ThecorrespondingmodelfortheCOMPASRiskMatrix‐RispresentedinTable4.4.2.Themodelexaminestheriskofparolefailurespecifiedasthecause‐specifichazardforparoleestoreturntoprisonforanontechnicalviolation.ThedifferencesbetweenoffendersontherisklevelsoftheCOMPASRiskMatrix‐Raresomewhat larger thantheones foundfor theRecidivismRiskscale,indicatinganevenstrongerriskpredictionoftheCOMPASRiskMatrix‐R.ParoleesonthehighriskleveloftheCOMPASMatrix‐Rhavea4.7timeshigherfailurehazardincomparisontoparoleesonthelowrisklevel.Thehazardforthemediumcategoryrelativetothelowcategoryis2.42.Table4.4.2:ProportionalHazardModelRegressingReturntoPrisonforaNontechnicalViolationontheCOMPASRiskMatrix‐R.RiskMatrix‐R

Coeff.

SE

p‐value

HazardRatio

Lower95%CI

Upper95%CI

Medium 0.883 0.080 <.001 2.42 2.07 2.83Med.High 1.246 0.072 <.001 3.47 3.02 4.00High 1.542 0.073 <.001 4.67 4.05 5.39

Note. The reference category for the test of medium, medium‐high and high risk categories is the low riskcategory.Additionalreceiveroperatingcharacteristics(ROC)analysisindicatestheareaunderthecurve(AUC)fortheRiskMatrix‐Rmodelpresentedaboveis.66.These findings clearly support the predictive validity of the Recidivism Risk scale and theCOMPAS Risk Matrix‐R. While the AUC indicators are moderate, the proportional hazardsdiffersubstantiallyandsignificantlybetweentherisk levelsofboth,theRecidivismRiskscale



and the COMPAS Risk Matrix‐R. This indicates a high utility of the scales in discriminatingbetweenhighandlowriskoffenders.



5. ConstructValidityThis section will synthesize the accumulating body of evidence pertaining to the constructvalidity of selected COMPAS risk and needs scales. The empirical findings reviewed in theabove sections are integrated with the general findings from the recent literature on themeaningsandtheoreticalimportanceofriskandneedfactors.Constructvalidityincludesmanydifferenttypesofvalidityandaccordingtosomeresearchersevencriterion‐basedvalidity(asdiscussedabove)canbeviewedasaformofconstructvalidity(Judd&Kenney,1981;Judd&McClelland,1998).Essentially,ascalehashighconstructvaliditywhenittrulymeasurestheconstructofinterestandconsistentlybehavesinamannerthatistheoretically consistent with this construct. Recent research provides evidence that theCOMPAS risk scales and/or their components are related to the construct of interest—recidivism. For the sake of brevity, the construct validity of other COMPAS scales is notdiscussedhere;pleaserefertotheCOMPASTechnicalManualandPsychometricsReportforadetaileddiscussionofallCOMPASscales.CriminalInvolvementTheCriminalInvolvementscaleisacomponentoftheRecidivismRiskscaleandtheconstructvalidityofthelatterisdiscussedbelow.TheCriminalInvolvementscalewasalsoshowntobeagood predictor of recidivism in Section 4.2 of this report. Offenders that score high on thisscaleareatahigherrisktorecidivatethantheircounterpartswithlowscores.The“hazard”ofcommitting a new crime increases by 27.6% for each additional unit of this scale. Since itrepresents the number of prior instances of arrest, probation, conviction or jail time, thisindicatesthatwitheveryprioreventofthistypeanoffenderis27.6%morelikelytorecidivate.Theconcurrentvalidityofthisscalecouldnotbevalidated,becauseitincludescriminalhistoryindicators.ThedegreeofCriminalInvolvementhasconsistentlyemergedasamajorriskfactorfor predicting ongoing criminal behavior. It is themost important of themajor risk factorsaccordingtovariousmeta‐analyses(Lipsey&Derzon1998,Gendreau,Little,&Goggin,1996;Andrews&Bonta,1994).Vocational/EducationalProblemsAnother component of theRecidivismRisk scale, theVocational/Educational Problems scalealso demonstrated high predictive validity as presented in Section 4.2. This COMPAS scalecapturestheconceptof“legitimateeconomicopportunity”andisanamalgamofeducationalattainment, vocational skills, jobopportunities,employment stability, andgood income.Themoregeneralconceptof“socialachievement”isoneofthe“bigfive”riskfactorsforcrimeandrecidivism in theGendreau, Little,&Goggin (1996)meta‐analysis. It is closely related to thetheory of social capital (Hagan, 1998; Coleman, 1990). Basically, persons with more socialcapital have higher ”life chances” than others who may have very restricted successopportunities. Offenders differ greatly in access to social capital or other resources. SocialCapital is somewhatdynamic—itcanbebuiltordestroyed.Forexample,a recordofserious



criminalbehaviororhighschooldropoutwillclearlydiminishlifechancesandsocialresources,whereascompletingajobskillstrainingcourseorobtainingaGEDmayincreasethesechances.DrugProblemsThisscalesummarizesindicatorswhethertheoffenderhashadproblemswithdrugsasderivedfrom the offense history, current charges, or the current arrest, in combination withinformationregardingadrugtreatmenthistoryoranexpressedneedfordrugtreatment.Itisascaletransformspecificallydevelopedformodelingrecidivism.Becauseof itshighpredictivevalidity, theDrugProblems scalebecameacomponentof theCOMPASRecidivismRisk scale(where prior arrests/convictions for drug possession and/or drug traffickingwere excludedhowever). It’s strong predictive utility has been demonstrated throughout this report withresultsforthescaleitself(Table4.2.2)aswellasresultsfordrugproblemsasacomponentofthe Recidivism Risk scale as discussed below. Numerous published research studies haveestablishedthatsubstanceabuseisasignificantriskfactorforbothgeneralcriminalbehavioraswellasviolentbehavior.Substanceabusealsoemergedasoneofthemajorriskfactorsinthemeta‐analysisofGendreau,Little,&Goggin(1996).RecidivismRiskScaleandCOMPASMatrix‐RThe Recidivism Risk scale and COMPAS Matrix‐R were developed to predict re‐offendingsubsequenttotheCOMPASassessmentdate.TheyarecomposedofCOMPASitemsselectedthrough diagnostic modeling strategies. Whereas scale reliability and coherence wereemphasized for developing the other COMPAS scales, predictive validity was clearlyemphasized fordeveloping theRecidivismRisk scaleandtheCOMPASMatrix‐R.BothaimtopredictwhichoffenderswillcommitcrimessubsequenttotheirinitialCOMPASscreeningdate.Theirpredictivevalidityiscentraltothisreportanddiscussedintheconclusionsbelow.



6.ConclusionsandFutureResearchMajor findingsof this reportare summarizedalong thedimensionsof reliabilityandvaliditythathavebeenexaminedandevaluated.Reliability: Findings of good reliability were replicated across several sites and differentoffenderpopulationswithfewexceptions.Generallythefindingsweresatisfactory,indicatingthat the COMPAS scales have high internal consistency and COMPAS is administered in aconsistentandreliablemanneratallsitesincludedinthiscomparison.ConcurrentValidity:CorrelationalpatternscommonlyfoundinCOMPASpsychometricreportswereverifiedasbeingratherrobust.Overall,theobservedrelationshipsbetweentheCOMPASsubscales and criminal history indicators data provided evidence of the criterion‐concurrentvalidity of the scales. Their general consistency across different sites and datasets, addsadditionalempiricalsupporttothisconclusion.

Predictive Validity: The COMPAS risk scales are valid tools for determining eligibility forcommunitycorrectionsprograms.Theoffenderrisklevelsclearlypredictedprogramfailure.

ThepredictiveutilityofseveraloftheCOMPASsubscalesregardingtheriskofrecidivismhasalsobeendemonstrated inaNewYorkprobationsample.Logistic regressionconfirmedthatoffenders scoring high on the Criminal Involvement, Vocational/ Educational, and DrugProblemsscale,areatasubstantiallyhigherrisktorecidivate.ThisfindingledtotheselectionofthesescalesascomponentsoftheRecidivismRiskscale.

Further evaluations by means of Receiver Operator analysis confirmed the high predictivevalidityoftheRiskofRecidivismscaleintheNewYorksample.AUCsforthescales’accuracyindiscriminating between recidivists and non‐recidivists were similar in magnitude to thoseobtained by other researchers in the field (Cottle, Lee,&Heilbrun, 2001; Gran, Belfrage,&Tengstrom 2000; Quinsey et al., 1998); but higher than comparable fourth generation riskassessment instruments (Barnoski & Aos 2003). Cross validation in a different New Yorksample confirmed this assumption with AUCs only slightly smaller than those in the scaleconstruction sample. Therefore the COMPAS Recidivism Risk prediction model can beconsidered robust.Moreover, it canbegeneralized tootheroffenderpopulationswith littlelossofpredictiveaccuracy.

Additional findings in aCalifornia sampleofparoleesexamining theutilityof theRecidivismRisk scale and theCOMPASRiskMatrix‐R for predicting parole failures clearly support theirpredictive validity.While theAUC indicators in this sample are rather low, the proportionalhazardsdiffersubstantiallyandsignificantlybetweenoffendersontherisklevelsofboth,theRecidivism Risk scale and the COMPAS RiskMatrix‐R, indicating the scales’ effectiveness indiscriminating offender probabilities of parole failure. This finding is particularly remarkablebecausetheRecidivismRiskscalehasbeendevelopedinaprobationsampleandhadnotbeenevaluatedinaparolepopulation.



Given that instrument validation is an ongoing process, countless further tests andmodelscouldbeappliedtofurtherexaminetheconcurrentandpredictivevalidityoftheCOMPASriskscales. The present overview, however, summarized encouraging evidence that the mainCOMPAS risk scales, Recidivism Risk and the Risk Matrix‐R, perform moderately well inpredicting recidivism in a variety of different criminal justice populations.Our next steps tofurtherconfirmthesefindingsareadditionaloutcomevalidationstudiesinlargersamplesandwith longer follow‐up periods. These analyses will be conducted at several different sitesincludingMichiganandNewYork.Moreover, a basic or “global” COMPAS will be developed as an attempt to accommodatepractitioners’ need for a more concise assessment instrument that is neverthelesstheoretically and empirically informed. The comprehensiveness of COMPAS, its ability todistinguish different types of offender populations and recidivism risks, and its flexibility incustomization will be strengthened as additional options are activated as needed.



ReferencesAndrews,D.A.,&Bonta, J. (1994).Thepsychologyofcriminalconduct.Cincinnati:Anderson

Press.Aos,Steve&Barnoski,Robert (2003).Washington’sOffenderAccountabilityAct:AnAnalysis

of the Department of Correction’s Risk Assessment. Washington Institute for PublicPolicy:DocumentNumber:03‐12‐1202

Austin,James(2006)HowmuchRiskCanweTake?FederalProbation,Vol.70,No.2,58‐63Brennan, Tim, Dieterich,William (2007). A Test of the Validity of the COMPAS Risk Scales:

PredictingReturntoPrisoninaLargeParoleSample.UnpublishedReport.NorthpointeInstituteforPublicManagement.TraverseCity,Michigan.

Coleman,J.S.(1990).Foundationsofsocialtheory.Cambridge,Mass:HarvardUniversityPress.Cottle, Cindy C., Lee, Ria J., Heilbrun, Kirk (2001). The Prediction of Criminal Recidivism in

Juveniles.CriminalJusticeandBehavior,Vol.28No.3,367‐394.Farrington, D. P., Jolliffe, D., Loeber, R., Stouthamer‐Loeber, M., & Kalb, L. M. (2001). The

concentrationofoffendersinfamilies,andfamilycriminalityinthepredictionofboys’delinquency.JournalofAdolescence,24,579–596.

Fine, J. P., & Gray, R. J. (1999). A proportional hazards model for the subdistribution of a

competing risk. Journal of the American Statistical Association, 94 (446),496–509.

Gendreau,P.,Goggin,C.,&Little,T.(1996).Ameta‐analysisofthepredictorsofadultoffender

recidivism:Whatworks!Criminology,34,575‐607.Grann, M., Belfrage, H., & Tengstrom, A. (2000). Actuarial assessment of risk for violence:

Predictivevalidityof theVRAGandthehistoricalpartof theHCR‐20.Criminal JusticeandBehavior,27,97‐114.

Hagan,J.(1998).Lifecoursecapitalizationandadolescentdevelopment.InR.Jessor(Ed.),New

perspectives on adolescent risk behavior, 499–517. New York: Cambridge UniversityPress.

Hosmer Jr.,D.W.,&Lemeshow,S. (1998).Applied survivalanalysis:Regressionmodelingof

timetoeventdata.NewYorkCity:JohnWiley.



Jones, P.R., Risk Prediction in Criminal Justice, in A. T. Harland, ed., “Choosing CorrectionalOptionsThatWork,”33‐68.ThousandOaks,CA:Sage,1996.

Judd,C.M.,&Kenney,D.A.(1981).Estimatingtheeffectsofsocial interventions.NewYork:

CambridgeUniversityPress.Judd,C.M.,&McClelland,G.H.(1998).Measurement.InD.T.Gilbert,S.T.Fiske,&G.Lindzey

(Eds.),Thehandbookofsocialpsychology,180–232.Boston,MA:McGraw‐Hill.Lipsey,M.W.,&Derzon,J.H.(1998)Predictorsofviolentandseriousdelinquencyinadolscence

andearlyadulthood:Asynthesisof longitudinalresearch. In:Loeber,R.&Farrington,D.P. (Eds.), Serious and violent juvenile offenders: Risk factors and successfulinterventions,86‐105.ThousanOaks,CA:Sage.

Moffitt, T. E. (1993). Adolescence‐limited and life‐course persistent antisocial behavior: A

developmentaltaxonomy.PsychologicalReview,100(4),674–701.Quinsey,V.L.,Harris,G.T.,Rice,M.E.,&Cormier,C.A.(1998).Violentoffenders:Appraising

andmanagingrisk.Washington,DC:AmericanPsychologicalAssociation.Stouthamer‐Loeber,M., Loeber,R.,Wei,E.,Farrington,D.P.,&Wikstrom,P.H. (2002).Risk

and promotive effects in the explanation of persistent serious delinquency in boys.Journal of Consulting and Clinical Psychology, 70 (1),111–123.

Tuma,N.B.,Hannan,M.T.(1984),SocialDynamics.Modelsandmethods.NewYork:Academic

Press.Venables,W.N.,&Ripley,B.D.(1999).ModernappliedstatisticswithS‐Plus.NewYorkCity:

Springer‐Verlag.



APPENDIXA—ScaleDescriptionsTableA1:CriminalInvolvementScaleItems(criminv).Items ShortDescription(ResponseCategories)

n.jails Howmanytimeshastheoffenderbeenjailed? (0=0,1=1,2=2,3=3‐7,4=8‐12,5=13+)n.prev.convict Howmanytimeshastheoffenderbeenconvictedbefore? (0=0,1=1,2=2,3=3‐4,4=5‐10,5=11+)n.prev.arrest Howmanytimeshastheoffenderbeenarrestedbefore? (0=0,1=1,2=2‐3,3=4‐5,4=6+)n.probations Howmanytimeshastheoffenderbeenonprobation? (0=0,1=1,2=2,3=3,4=4‐5,5=6+)

TableA2:Vocational/EducationalProblems,ScaleItems(voced).Items ShortDescription(ResponseCategories)high.school Didoffendercomplete12thgrade? (1=yes,2=no)expelled Wasoffendereversuspendedorexpelledfromschool? (2=yes,1=no)grades.hs Whatweretheoffender’susualgradesinhighschool? (A=1,B=2,C=3,D=4,F=5)job Doestheoffndercurrentlyhaveajob? (1=yes,2=no)skill Haveskill,tradeorprofessiontousuallyfindwork? (1=yes,2=no)job.lastyear Howmuchworkorschoolthelast12months? (1=12monthsfulltime,2=12monthsparttime, 3=6+monthsfulltime,4=lessthan6mos.PT/FT)fail.or.repeat.grd Didtheoffnderfailorrepeatagradelevel? (2=yes,1=no)need.training Feelneedmoretraininginanewjoborcareerskill? (2=yes,1=no)wages.above.min Howhardtofindajobaboveminimumwage? (1=easier,2=same,3=harder,4=muchharder)chance.success.work Howwouldtheyratetheirchanceofbeingsuccessful? (1=good,2=fair,3=poor)haveempschool Doesoffenderhaveverifiedlocalemployerorschool? (1=yes,2=no)



TableA3:DrugProblemComponentItems(drgprob5).Items ShortDescription(ResponseCategories)d.current usingdrugswhenarrestedforcurrentoffense 2=Yes,1=Nocurr.drugp Currentchargedrugpossession

(0=notchecked,1=checked)curr.drugt CurrentchargedrugTrafficking

(0=notchecked,1=checked)want.rx.d Wouldbenefitfromtreatmentfordrugs 2=Yes,1=Noever.rx.d Hasoffendereverbeenintreatmentfordrugs 2=Yes,1=Nodrugposs* Numberofpreviousarrests/convictionsfordrugpossession

(0=0,1=1,2=2,3=3+)drugtraf* Numberofpreviousarrests/convictionsfordrugsales (0=0,1=1,2=2,3=3+)

*Note:Theseitemswereincludedintheoriginalscaleconstructionmodel(NewYorkProbation)butarenotacomponentoftheRecidivismRiskscaleanymore.

TableA4:HistoryofViolenceScaleItems(histviol).Items ShortDescription(ResponseCategories)n.prev.vfel Totalpriorjuvenile/adultfelonyassaultarrests (0,1,2,3,4,5+)assault Totalpriorassault(notmurder)arrests/convictions (0,1,2,3+)sex.force Totalpriorsexoffensewithforcearrests/convictions (0,1,2,3+)homicide Totalpriorhomicide/manslaughterarrests/convictions (0,1,2,3+)robbery Totalpriorrobberyarrests/convictions (0,1,2,3+)weapons.offense Totalpriorweaponsarrests/convictions (0,1,2,3+)family.violenc Totalpriorfamilyviolencearrests/convictions (0,1,2,3+)frq.injury Priortimesvictimhadphysicalinjuries (0,1,2,3,4,5+)fights.inmate Everwriteupsforfighting/threateninginmates? (2=yes,1=no)



TableA5:HistoryofNoncomplianceScaleItems(histnonc).Items ShortDescription(ResponseCategories)n.prob.rev Howmanytimesprobationsuspendedorrevoked? (0,1,2,3,4,5+)n.arrest.on.bail Howmanytimesarrested/chargedforcrimeonpretrialrelease? (0,1,2,3+)n.fta Howmanytimesfailedtoappearontimeforcourt? (0,1,2,3,4,5+)n.rec.prob Howmanytimesarrested/chargedwhileonprobation/parole? (0,1,2,3+)

research synthesis reliability and validity of · pdf fileresearch synthesis reliability and...

Documents