airs outcomes report

8/2/2019 AIRS Outcomes Report

1/42

AIRSINFORMATION&REFERRALOUTCOMESREPORT

1

AIRSOUTCOMESPROJECTREPORTJuly15,2011

RevisedAugust10,2011


2/42

Copyright2011AllianceofInformationandReferralSystemsandIntegerResearch&Consulting,LLC,

exceptasotherwisestated.

Allrightsreserved.Nopartofthisdocumentmaybereproducedinanyformorbyanymeanswithout

expresswrittenpermission,exceptforthenon-profitpurposesofeducationandscientificadvancement.

Forinformation,pleasecontacttheAllianceofInformation&ReferralSystems,P.O.Box33095,

Portland,OR97292.


2


3/42


3


4/42

TABLEOFCONTENTS

.............................................................................................................................................EXECUTIVESUMMARY 5

.......................................................................................................................................................I.BACKGROUND 7

.........................................................................................................................................II.PROJECTCOMMITTEE 8

........................................................................................................................................III.PROJECTLITERATURE 9

...........................................................................................................................IV.CONCEPTUALFRAMEWORKS 10

.......................................................................................................................................................1.InterventionFramework 10

...........................................................................................................................................................2.OutcomesFramework 12

...............................................................................................................................................................3.ProcessFramework 15

....................................................................................................................................................4.MeasurementFramework 18

..........................................................................................................................V.MEMBEROUTCOMESSURVEY 22

........................................................................................................................................................VI.SUMMARY 29

.............................................................................................................................................VII.MOVINGAHEAD 30

................................................................................................................................................................................KeyConcerns 30

............................................................................................................................................................................ProjectStrategy 31

............................................................................................................................................................APPENDICES 33

............................................................................................................................................................Appendix1:References 34

.........................................................................................................................................................Appendix2:LogicModels 37

........................................................................................Appendix3:211LADevelopmentalScreeningProjectFactSheet 40

.........................................................................................................................................Appendix4:AIRSOutcomesSurvey 41


4


5/42

EXECUTIVESUMMARY

InDecemberof2010,the AIRSBoardcommissionedIntegerResearch&ConsultingofChicagotodevelopafundingprospectusfor

I&R/Aoutcomesresearchthatwouldbesuitableforpresentationtopotentialfunders,participants,andthepublic. 1

Integer conducted a reviewof literaturepertinent toI&R/A outcomes, populateda projectwebsite withthe results,and beganworkingwitha ProjectTeamin January,2011. Withinthe firstmonth,it wasapparentthat theprojectwould haveto focus at a

foundational levelasa firststep towarddevelopinga fundingprospectus.Thisreportdefinesanddescribesthesefoundationsand

highlightsfactorsto considerabout movingthe workof I&R/Aoutcomesforward. It alsoprovides ample material withwhich to

buildaprospectusinthefuture.

ThefollowingobservationsareofferedfortheconsiderationoftheAIRSBoard.

TheliteratureonI&R/Aoutcomesis somewhat sparse, butseveralseminalsourceswereidentifiedandusedinthe project.

Theseincludeacomprehensive texton I&R/AwrittenbyRisha Levinson,oneofAIRSfounders,andthreeresearchpapersthat

provideageneralframeworkforI&R/Aoutcomes.

SolidconceptualfoundationsforI&R/Aoutcomesexistandofferexcellentdevelopmentpotential.Inexploringthese,weiden-

tifiedfourkeystructuralcomponentsthatrequirefurtherdevelopmentforourpurposes.

1. InterventionFramework: RishaLevinsondevelopedaninterventionframeworkthatencompassesallthemajor varia-

tionsofI&R/Aservices.Sheprovidesimportantsupportfortheideathatasingle,flexiblemodelcanbeusedtochar-

acterizeoutcomesacrossthe range ofI&R/Ainterventions,whetherthey arefor for agingand disabledpopulations,

thecommunityatlarge,otherspecialpopulations,orpeopleincrisis.

2. OutcomesFramework: TheresearchofMatthewSaxtonintheUSandAnnNettenintheUKprovidesarobustmodel

forunderstandingtheimpactofI&R/Aonthreelevels: on clients,oncommunityorganizations,andon society.Out-

comesdatafromallthreelevelswillofferthestrongestpossibleevidenceofthefar-reachingvalueofI&R/A.


5

1Throughoutthisreport,I&R/AisusedtodesignatethefullspectrumofinterventionsthatAIRSmembersofferthepublic:information,referral,

and/orassistance.


6/42

3. ProcessFramework:Webegandevelopinganend-to-end,client-centeredprocess framework.The processstarts with

the clientsrequestforhelpfrom theI&R/Aorganizationandencompasses allthekey stepshe orshetakesuntil the

needsorissuesareaddressedby theservicesystem.This processframeworkisessentialfordefiningthefullrangeof

I&R/A outcomes, becausethey extendbeyondthe I&R/Aorganizationto otherorganizationalandindividual actors in

thecommunity.4. MeasurementFramework: Findingameansofmeasuringoutcomes thatis understandable,affordable,practical,and

applicableacrosstherangeof AIRSmemberorganizationsandservices isa tremendous challenge. Weachievedsig-

nificant insightsby examiningseveralmeasurementandfeedbackmodels. These offerexcellentdevelopmentpoten-

tial,butthereisalongwaytogo.

Wedecidedtoassessthe interestofthe AIRSmembershipinthesubjectofI&R/Aoutcomesandrelateddevelopmentwork. We

mountedanonlinesurveyinApril;thecompiledsurveydatasuggeststhat:

- Thevastmajorityofrespondentsfeel thatI&R/Aoutcomesareveryimportanttodayandareessentialtosecuringfund-

ingfromallthemajorchannels.

- They ratethreefactorsestablishingcleardefinitions,methodsofmeasuringoutcomes,andinformationonbestprac-

ticesasveryimportantinI&R/Aoutcomesdevelopmentwork.

- Many respondentsstatedthathavinganindustrystandardforI&R/Aoutcomes isveryimportant,andtheyvoicedtheir

appreciationforAIRSconsideringsuchaninitiative.

Attheendofthispreliminaryphaseoftheproject,importantquesonsremainabouthowtoconnuethework. Theseprinci-

pallyconcern gatheringbuy-inand support fromkey memberconstuencies, defininga wayto manageextended development

work,andsecuringthenecessaryhumanandfinancialresources.


6


7/42

I.BACKGROUND

November of2010,IntegerResearchand Consultingmadeits finalpresentationto theAIRSBoard onthe EnvironmentalScanRe-

searchProject. In thecourseofthe presentation,Integermade foursuggestions(Figure1,left). TheBoarddecidedtoacton the

thirdsuggestion,FormulateauniversalandmeasurableI&Rprocess (Figure1, right). Integerwas retainedbyAIRStodevelopa

fundingprospectusforoutcomesresearchsuitableforpresentationtopotentialfunders,participants,andthepublic.

DuringDecember, 2010and January, 2011,Integer conducteda reviewof literaturepertinent to I&R/Aoutcomesand evidence-

basedpractice. Theresearchwaspursuedinseveralonlineacademicdatabases,inGoogleScholarandMicrosoftAcademic,andin

generalInternetsearchesusingGoogle. Preliminaryresearchmaterialwascompiledandloadedtoa privatewebsitethatwas de-

velopedfortheproject.

Figure1:SlidesfromIntegersPresenta\ontotheAIRSBoardinNovember,2010


7


8/42

II.PROJECTCOMMITTEE

DuringJanuary,the AIRSBoardandstaffwerecanvassedforsuggestionsonOutcomes Committeemembership.Thoughtleadersin

I&R/A,healthandhumanservicesresearch,andlibrarysciencewereinvitedtojointheCommittee,withtwelveaccepting.

Fiveteleconferences wereconductedfromFebruarythroughApril,withmembersparticipatingasfrequently as theirschedulesandinterests allowed. In the intervals betweenteleconferences,workpapersand researchmaterialswereadded tothe websiteand

teammemberswereinvitedtoreviewandcommentuponthem.Theagendaandnotesforeachteleconferencewereprovidedto

theteamseveraldaysinadvance,andservedasaguidelineforeachdiscussion.

Table1:OutcomesProjectCommi`ee

Commi`ee embers TeleconferenceDates

DR.SHERRYBAME,Professor,HealthPlanning&Policy

TexasA&MUniversity

LINDADAILY,Director2-1-1andDisasterServices

UnitedWayWorldwide

HELENELTZEROTH ,ChiefPrograms&CommunicaZons

Officer

NaZonalAssociaZonofAAA(n4a)

ANGELAFISHER,DirectorofClientServices

AAAforSouthwestFlorida/ADRC

MARYHOGAN,LibraryDirector

CoraJ.BeldenLibrary

DAVIDJOBE,Directorof2-1-1

UnitedWayofGreaterHoustonAMYLATZER, ChiefOperaZngOfficer

211LACounty

RANDYNICKLAUS,President

2-1-1BigBend

JOHNOHANIAN,ChiefExecuZveOfficer

211SanDiego

JACKYRODDY

InformCanada

PATRICKROGERS,ExecuZveDirector

InsZtuteforHumanServices

MICKITHOMPSON,ExecuZveDirector

2-1-1TampaBayCares,Inc.

CHARLENEHIPES,ChiefOperaZngOfficer

AllianceofInformaZonandReferralSystems

CLIVEJONES,Consultant

AllianceofInformaZonandReferralSystems

MARKH.NEUFFER,PresidentIntegerResearch&ConsulZng,LLC

February3,2011

February17,2011

March3,2011

March17,2011

April14,2011


8


9/42

III.PROJECTLITERATURE

Severalhundredarticles andpublications werescanned by theprojectconsultant duringDecember,2010and January,2011. The

searchproducedover 20 books,articles, and presentations published between1998 and2011 relevant to I&R/A outcomes, per-

formancemeasurement, quality,andcost/benefitanalysis.The Committeesuppliedseveral useful additionstotheliterature. Lit-

eraturewasuploadedtotheprojectwebsiteandmadeavailabletotheteam.AlistofreferencesappearsinAppendix1.

Fourresourceswereidentifiedashavingessentialimportancetotheproject:

Risha W.Levinsonis anAIRSfounderanda retiredprofessorofsocial work. Hersecondbook onthe I&R/A field,NewRoutesto

HumanServices: InformationandReferral,waspublishedin2002andisthemostauthoritativeandcomprehensiveoverviewwrit-

tentodate.HerdepictionofI&R/Ainterventionsasacontinuumhasbeenimportanttoourproject.

MatthewSaxtonisaprofessorattheInformationSchoolattheUniversityofWashington.HeandseveralcolleaguesattheInforma-

tionSchool werecommissionedto reviewthe states2-1-1systemandtosuggestanimpactframeworkthatwouldextendbeyond

the usual measuresofoutputsintotherealmofuseroutcomes. In2-1-1Informa@onServices:OutcomesAssessment,BenefitCost

Analysis,andPolicyIssues,Saxtonandhis colleagues presentamodel ofI&R/Aoutcomes astheyrelatetotheindividual,tocommu-

nityorganizations,andtosocietyingeneral.

AnnNettenandJulienForder,bothfromtheUniversityofKent(UnitedKingdom)receivedgovernmentfundingtoreviewInforma-

tionandAdvice(I&A)services andtorecommendhowoutcomescouldbemeasured.UsingtheSaxtonoutcomesmodelasafoun-

dation,NettenandForderextendedand detailedtheframework, anddevelopedan approachformeasuringI&Aoutcomesat the

leveloftheindividualclient. MeasuringtheOutputsofInformationandAdviceServices:InitialReport waspublishedin2008.

Inthetwoyearsfollowing,KarenWindleandAnn Nettenconducteda detailedstudyand authoredMeasuringtheOutcomesofIn-

forma@onandAdviceServices:FinalReportin2010.Todate,thisisthemostcomprehensiveandrelevantresearchfoundonI&R/A

outcomes. Theconceptualunderpinningsofthe outcomeslogicmodelfromNetten(andearlierfromSaxton)arereviewedand

amplified. All thedetails ofthestudy arepresented,andthe study closeswithfiverecommendationsforexpandingI&R/A out-

comesresearch.


9


10/42

IV.CONCEPTUALFRAMEWORKS

1.InterventionFramework

TheAIRSEnvironmentalScanunderscoredthediversityofAIRSmembership,operations,andservices.Thisraisesthequestionof

whetheritispossibletoevaluateoutcomesofthevariousI&R/Aserviceswithasingleoutcomesmodelthatusesstandardizeddefi-nitions,measures,andmethods.

ThisquestionisaddressedbyRishaLevinsonsworkin

NewRoutestoHumanServices.AsshowninFigure2,

shemapsawiderangeofI&R/Ainterventions.

Inourunderstanding,thehorizontalaxisshowsthe

degreeofinterventionactivity,withlessattheleftof

thematrixandmoreattheright.Onthefirstline,for

example,themostbasicinterventionisprovidingin-

formation(informationassistance).Thisisex-

tendedbyaddingareferral,andfurtherextended

withafollow-up.

Theverticalaxisshowsthedegreeofaninterven-

tionsprofessionalinvolvement,withtheleastin-

volvementatthetopandthemostatthebottom.

Forexample,thereisminimalornoinvolvementin

providinginformation(informationassistance)in

thetoprow.Bycontrast,caseadvocacy(thirdrow)requiresin-depthknowledgeoftheclientandsignifi-

cantprofessionalinvolvement.

ThismodelmakesitpossibletocharacterizeI&R/A

servicesasafamilyofrelatedinterventions.Iten-

ablesI&R/Aprofessionalstodepict,analyze,andquantifytheirworkwithinanextensibleframeworkthatdefinesthecommonfea-

turesof,andthedifferencesbetween,thevariousservicepermutationsofI&R/A.

Figure2:RishaLevinsonsModelofI&R/AInterven\ons

FromNewRoutestoHumanServices:Informa@onandReferral(p.63),

byR.W.Levinson,2002,NewYork,NY:SpringerPublishingCompany,Inc.,

ISBN0-8261-2393-7. 2002bySpringerPublishingCompany,Inc.Reprintedwithpermission.


10


11/42

Figure3adaptstheLevinsonmodeltodepictthe I&R/Ainterventions

practicedtodaybyAIRSmembers. Thediagramaims toshowthees-

sentialcontinuityoftheinterventions;theactualcontentandplace-

mentoftheinterventiontypes wouldbemoreaccuratelysuppliedby

I&R/Aprofessionals. The toprow,I&RBasicComponents,showsthestandardinformation,referral,andfollow-upservicesprovidedtothe

generalpublic.

Crisis Intervention2 requires greater professional involvement with

the caller(placingitfarther downthevertical axisasa supportserv-

ice) and isalsomoreextensivein termsof process, time, andre-

sources (thus extending it farther along the horizontal axis). Amy

Latzer introducedthe CareCoordination intervention inour project

work.Callerswithaparticularproblemareguidedbythe I&R/Apro-

fessional topre-qualifiedcommunityserviceproviderssuitedtodeal

withthat problem. Theextentof the interventionandprofessional

involvement aresomewhatbroaderand deeper thantheBasic I&R

model.3 The Assistance intervention (I&R/A) encompasses the

broadestsetofinterventiontransactionsandpotentiallythe greatest

professionalinvolvement. Itincludesalltransactionsthatsupport theclienton anongoingbasisthroughtheclientssearchfor

help;it thereforeencompasses services suchas advocacy,assistance withtransportation,helpgettinganappointment, motivational

support,andcasemanagement.

ThemodelshowsthatI&R,CrisisIntervention,CareCoordination,andAssistance areclosely-relatedmembersofthesamefamilyofprofessionalpractice whichdemanddifferentextentsofinterventionandprofessionalinvolvement.TheseI&R/Ainterventionsmay

becharacterizedandassessed witha commonoutcomesframeworkthatprovidesadditional definitions, measures, andmethods

applicabletospecificvariations.

Figure3:Adapta\onoftheLevinsonModel

FromNewRoutestoHumanServices:InformationandReferral (p.63),

byR.W.Levinson,2002,NewYork,NY:SpringerPublishingCompany,Inc.,

ISBN0-8261-2393-7. 2002bySpringerPublishingCompany,Inc.

Adaptedwithpermission.


11

2CrisisInterventionreferstoassistingcallerswithmentalhealthissues.

3TheCareCoordinationinterventionisdescribedonthe211LACountywebsiteat http://www.211la.org/?page_id=150 andintheFactSheetfor211LA

CountysDevelopmentalScreeningProjectinAppendix3ofthisreport.
http://www.211la.org/?page_id=150http://www.211la.org/?page_id=150


12/42

2.OutcomesFramework

MatthewSaxtonandhis colleagueshave made animportantcontributiontothe I&R/Afield byconceptualizinga three-levelout-

comesframeworkforindividualservice recipients,community organizations,and thecommunityorsocietyatlarge.AlthoughSax-

tondevelopedthemodelfora2-1-1systemstudy,wefeelitisentirely applicabletothefullcontinuumofI&R/Ainterventions.Fig-

ure4provides a high-levelrepresentationthat

isfurtherdetailedonpage13.

The terminology is probably familiar to most

readers: inputsincludefunding,information,

otherresources,andclients.Processesinclude

thefullrangeof I&R/Ainterventionsoutlined

onthe previous page. Outputsconsist ofthe

quantifiableresultsoftheprocesses:totalcli-

ent calls, client contacts, crisis interventionsandsoon. Outcomes are themeasurableim-

pactsof theseI&R/Ainterventions: crisesde-

escalated; service provider contacts madeby

theclientafter he or shereceived a referral;

andserviceuptakebytheclientafterreceiving

areferraland/orassistance.

This framework provides a means of broadly

quantifyingtheoutcomesandbenefitsofI&R/Afortheindividual,communityorganizations,andsocietyatlarge;itinvitesfurther

elaborationby I&R/Aprofessionals. I&R/Aorganizations andthe I&R/A professionwould thenhavea robust andrigorousframe-

workfordemonstratingthehelptheyprovidetoallthepeopleandorganizationstheytouch.

Figure4:Schema\za\onofSaxtonsThree-LevelOutcomesFramework

From2-1-1InformaZonServices:OutcomesAssessment,Benefit-CostAnalysisandPolicyIssues,

byM.L.Saxton,C.M.Naumer,andK.E.Fisher,2007,GovernmentInforma@onQuarterly,24,

pp.202-203.2007byElsevierInc.Adaptedwithpermission.


12


13/42

Thefollowingpageshowsthe Saxtonframeworkwithdetailslater suppliedby Netten. Herearea fewnotes tohelpthereaderre-

viewthediagrammoreeasily.

TheHorizontalAxisshowsthecompletespanofI&R/Aacvityfrominputs tooutcomes.Theinputsare showninonelargeboxto

suggestthattheyaresharedbythethreelevelsoftheindividual,organizaons,andcommunity.

TheVer\calAxisencompassesthethreelevelsofI&R/Aimpact:theindividual,communityorganizaons,andsociety.

Individual-LevelProcesses: this boxincludes mulpletypes ofI&R/Aintervenons. Thisis Neens conceptualizaonbased on

Informaon &AdviceservicesintheUnitedKingdom. Forthe purposesofour OutcomesProject,thisbox wouldbe populated

withthetypesofintervenonsprovidedbyI&R/AorganizaonsinNorthAmerica.

Outcomesare characterizedas short-term, intermediate,andlong-term;themodelaffordsthe flexibility todefine thelength of

theseintervalsonacase-by-casebasis.

TheOutcomesCommitteenotedthatitisimportanttodistinguishoutcomesthataremoredirectlyinthechainofI&R/Acause

andeffect(forexample,aserviceconnectionenabledbyanaccurateandappropriatereferral)fromoutcomesthatarefarther

downthechainandwhicharenot directly impactedby I&R/Aactivity(forexample,theclientsproblemisresolvedafter he

receivesservices).

Theredtrianglesymbol inthetop right cornerof someboxesindicates thatthe original Saxtoncontenthas beenreplaced with

subsequentrefinementsbyNeen.

TheOutcomesCommitteerecognizedthatthe Saxton-Nettenoutcomesframeworkinvitesfurtherelaboration;muchremains tobe

addedinspecificdetail. Yet,as alreadynoted,the frameworkprovidesanexcellent basisforbroadlycharacterizingthe beneficial

impactofI&R/AandisthusawayofbothdeepeningandbroadeningthediscourseonthevalueofI&R/Atoallstakeholders.


13


14/42

Figure6:CombinedSaxtonandNe`enOutcomesFrameworks

From2-1-1InformaZonServices:OutcomesAssessment,Benefit-CostAnalysisandPolicyIssues,byM.L.Saxton,C.M.Naumer,andK.E.Fisher,2007,

GovernmentInforma@onQuarterly,24,pp.202-203.2007byElsevierInc.;and

MeasuringtheOutputsofInformaZonandAdviceServices:IniZalReport,byA.NeenandJ.Forder,2008,PSSRUDiscussionPaper2543/2,pp.17-18.

PersonalSocialServicesResearchUnit,UniversityofKent,Canterbury.

Adaptedwithpermission.


14


15/42

3.ProcessFramework

TheSaxton-Netten modelsuppliesa general definition ofI&R/Aoutcomes forindividuals,organizations, andsociety.In ordertode-

fineandmeasureoutcomesatanoperationallevel,adetailedprocessandoutcomesframeworkisneeded.Thisframeworkwould:

Provideastandardizedapproachtodescribingprocesses,outcomes,andmetricsusableforallI&R/Aintervenontypes. DefinethekeyprocessstepsforI&R/Ainteraconswithindividuals,communityorganizaons,andsociety.

Definethecause-and-effectrelaonshipsbetweenprocessstepsandtheiroutcomes(alsoknownasalogicmodel.)

Provideadiagnosccapabilitytorevealwheretheprocess is notworkingasdesired(e.g.,duetobarriers)orshouldberedesigned

becausetheunderlyingprocessorlogicmodelsdonotproducedesiredresults.

TheProjectTeamtook several approaches todetailingthe process/outcomesmodels. Our first effortwas tobringthe Saxton-

Nettenmodeltoafinerlevelofdefinition(seeAppendix2).Itwasthendecidedtofocusthispreliminarythinkingonoutcomesat

the individuallevel. Theteamexperimentedwithseveraldifferentapproaches,includinga tabular format,flowcharts,andhybrids

ofthese.Severalkeydesignprincipleswerecrystallized:

Theprocessneedstobedefinedasanend-to-end,client-centeredframework.The processstartswiththeclientsrequestforhelp

fromtheI&R/Aorganizaonandencompassesall thekeysteps he orshe takesunltheneedsorissues areaddressedbytheserv-

icesystem.ThisframeworkdefinesalltheI&R/Aoutcomesfromthestartoftheprocesstoitscompleon.

Themodelneedstodistinguishclearlybetweenoutcomesthataredirectlyfacilitatedby I&R/Aactivities(forexample,contact

withaserviceproviderenabledbyanaccurateandappropriatereferral)andoutcomesforwhichtheclientorserviceorganiza-

tionshaveadirectresponsibility(forexample,theoutcomesresultingfromtheclientreceivingservices).

Theprimary aimoftheI&R/A processmodelistoimprove theeffectivenessinI&R/Aasa distinct,butintegral,partofthe

largerhumanservice system.I&R/Aprocesses and I&R/Aoutcomes contributeto,butare distinctfrom,thoseofthelargersys-temwhichareassessedthrough programand system evaluations. Clearly defining andcommunicating theI&R/A process/

outcomesmodel,itsterminology,anditspracticeswouldunderscoreitscompatibleroleinthelong-establishedfieldofhealth

andhumanserviceoutcomesevaluation.


15


16/42

Figure7on the followingpagewascreatedafterthe lastProjectTeamteleconferenceandreflectsasynthesisofthethinkingwhen

theprojectconcluded.

Theillustrationshowstheframeworkatthenextlevel ofdetailbelowtheSaxton-Nettenmodel.Itis stillverysimplifiedandis

presentedtoexemplifycurrentthinking,nottoprovideadefinitiveordetailedmodel.

Thetopsectionoftheframeworkincludesahigh-levelblockdiagramoftheend-to-endprocess.Notethatthe modelincludes

threeclientstates:the clientbeforegettinginformation,theclientafter receivingI&R/Ahelp,andtheclientafter receiving

helpfromthe directservice provider.Theseareplacedinthediagramtoshowexplicitlythe clients beforeand aftercondi-

tionsthatis,milestonesintheprocesswhereoutcomesshouldbedefinedandmeasured.

Thebottomportionoftheframeworkshowsa high-level logicmodelofthe processes withrepresentativeoutcomes.Forex-

ample,anoutcome ofprovidinginformationisclientknowledge:theclientsknowledgewilleitherbeincreasedorwillremain

thesameaftercontactwiththeI&R/Aspecialist.

Other outcomeshavea morecontingentcause-and-effectrelationshipwithI&R/Aactivities. Whetherthe client contactsthe

service providerafterreceivingareferraldepends uponmultiple factors,some ofwhichare withintheI&R/Ascontrol andoth-ersofwhicharenot. Ifa clientfailstoreachaserviceproviderbecausethereferralinformationis deficient,thisisa negative

outcomeoverwhichthe I&R/Ahas control. Onthe otherhand,aclientmay ormay notcontactthe providerbasedonfactors

(suchastheclientsmotivation,familyresponsibilities,andaccesstotransportation)thatareinmanycasesoutsidethecontrol

oftheI&R/A.

Definingresponsibility for outcomesenablesI&R/As tocorrectnegativeoutcomesforwhichthey areaccountable,andalsoto

identifyclient,process,serviceprovider,andsystemshortcomingswhichcanbecorrectedorimprovedtofosterbetterresults.

Outcomesinthemodelhavebeencharacterizedasshort-term,intermediate,andlong-termbasedontheelapsedtime,degree

ofcontrol,and numberof processstepsand actorsinvolved. For example,the I&R/Ahas significant controloverthe out-

comeof increasinga callersknowledgeat the timeofthe contact;hence itis a short-termoutcome. TheI&R/Ahas partial

controlover theclientcontactingaserviceproviderand itmaytake timefortheoutcometobe realized,hencetheoutcomeis

consideredintermediate. Finally,the I&R/Ahasnodirectcontroloverthe clientscourseofservice,whichmay be temporally

remotefromthe initialcontact.Totheextent,however,thattheI&R/Afacilitatedtheaccessiontoservices byprovidinginfor-

mation,areferral,oradditionalassistance,thislong-termoutcomeisessentialtorecord.


16


17/42


18/42

4.MeasurementFramework

TheProjectCommitteediscussedthesignificantchallengesassociatedwithcapturingandmeasuringI&R/Aoutcomesdata:

Inmanycases,intelligence gatheredaboutprocesseffecvenessandoutcomes is limitedtoad-hocfeedbackfromclients orserv-

icepartnersreporngamaerontheirowniniave.Suchfeedbackisuseful ona case-by-casebasis,butnotforgeneralprocessoroutcomeimprovement.

InsomeI&R/Aorganizaons, aqualityassurancefollow-upprocesshasbeendesignedandimplemented(StandardAssessment

andReferralwithFollow-Up).Localpraccedetermineshowfollow-upparcipantsareselected,thenumberandmingoffollow-

ups,thenatureoftheinformaongathered,andtheuseoftheinformaon.

SomeI&R/Aintervenonsincorporateassistanceservicesthatextendbeyondinformaon,assessment,and referral.In interven-

onslikeCareCoordinaonandI&Aforthe elderlyand disabled,connuedclientcontactandawarenessofprocessoutcomes are

intrinsicto the servicemodel. Consequently,outcomesdatacolleconand analysismayactually beincorporatedin theservice

model,asitisinthe211LADevelopmentalScreeningProject.4

DavidJobe,a memberofthe ProjectCommiee,describeda publichealthresearchprojectthatuses a Navigatorwho provides

movaonal supporttotheclientandhelpstheclientcontactandengageneededservices.TheNavigatoralsodevelopsinforma-

onaboutandinsightsintothepsychological,personal,andpraccalbarrierstheclientfacesintheserviceprocess.

TheProjectTeamdiscussedthepotenalofa navigator-like funconin the contextof I&R/Aoutcomes. Ahypothecal system

navigator couldassemble informaononprocess volumes, characteriscs,andcycle mesforeach segmentof the end-to-end

process. Thefunconcoulddeterminetherootcausesofprocessfailureand couldassistinimprovingprocessperformance,de-

finingbestpracces,and sengstandards. As well, thefuncon couldhighlightclientcharacteriscsthatfosterposive service

engagementorprocessfailure.

AsketchofthevariousI&R/Aintervenons,withtheirpotenaltomeasureoutcomes,isprovidedonthenextpageinFigure8.Itwasapparentthatconsiderableeffortwouldberequiredtodevelopanoutcomesmeasurementapproachthatwouldbeunder-

standable,affordable,andapplicabletothewidevariety ofI&R/ApracticesamongtheAIRSmembers.AIRSmembers are exploring

differentfeaturesofpossiblenewapproaches,butthereisalongwaytogo.


18

4TheDevelopmentalScreeningProjectisdescribedonthe211LACountywebsite: http://www.211la.org/?page_id=150
http://www.211la.org/?page_id=150http://www.211la.org/?page_id=150


19/42

2011IntegerResearch&ConsulZng,LLC

Figure8:MeasurementPoten\alofI&R/AInterven\ons(IndividualLevelofImpact)


19


20/42

Figure9(nextpage)didnotappearintheoriginalreporttotheAIRSBoard,butwasincludedintheBoardpresentationofJune4,

2011. Itbrings the thinkingofthe Reporttoa nextstageof possibledevelopment. Likethe precedingdiagrams,it isintendedto

representconceptsratherthanliteraldetail.

Theprocess blockdiagramappearsinayellowrectangleatthetopofFigure9.Belowitarehigh-leveloutcomesfora standard

assessmentandreferralintervention(greenrectangle).

Foreachoutcome,there isafailure/successsymbolinacascadingarrangement.Forexample,theclientattempts toreachan

I&R/Aspecialistandifheorshe issuccessful[O-1],the clientreceivesastandardassessmentandreferral fromtheI&R/Aspe-

cialist.A successfuloutcome [O-2]takesplacewhentheclientsneedis accurately assessedanda referralappropriatetothe

clientsneedcanbe madetoanexistingserviceprovider. Afailedoutcomemightoccurwhennoresourceexiststomeetthe

clients needandtherefore noreferral canbemade. Aftera standardreferralissuccessfullyprovided,the clientsknowledgeis

enhancedsuccessfully[O-3]ifheorsheisabletoassimilatesufficientinformationtomakecontactwithanappropriatepro-

vider.Afailedoutcomemightoccuriftheinformationisnotassimilatedduetoalanguagebarrier,orisnotsuccessfullypassed

ontoanotherpersonwhoneedstheservices.

Everysalientactivityinthe end-to-endprocesshasidentifiableandmeasurableoutcomes.If therearefeedbackmechanisms

thatcapturethese events,thedataaboutoutcomesuccessesandfailuresmaybegatheredandanalyzed.Thisis illustratedin

thediagram withthedottedverticallinesfromthe failure/successsymbolstothedrumsymbols( ),whichrepresentdata,

andtothesummationsymbols(),whichrepresentanalysisofthedata.Thiswouldmakeitpossibletogaugewhereaparticu-

laractivityneeds toberedesignedtoincreaseits effectiveness;whereclients needadditionalhelptooperationalize the infor-

mationtheyaregiven;orwhereotherfactors,suchasavailableservicecapacity,requireattention.

AsshowninFigure9,feedbackdataandanalysisfacilitatebetter processandoutcomes managementforI&R/Aorganizations,

forserviceproviderorganizations,andforthe servicesystemas a whole.This approachrecognizesthatcommunities,govern-

ment,andfundersaremostinterestedinthe end-to-endprocessofservingthe needsofthepublic. Gatheringand analyzing

dataontheend-to-endprocessenablesthisobjective;findingapracticableandvalidmeansofaccomplishingthisiskey.

Theincorporationofclientandcommunity dataintothefeedbackprocess(shownat theleftof Figure9)wouldalsohavea

transformativeeffecton process andoutcomes management. The abilitytoanalyzeservice systemperformance segmentedby

particular demographics,serviceneeds,and accesschannels wouldmakeitpossible tocalibratethesystemtobestmeetthe

serviceobjectivesforspecificproblems,needs,andpopulations.


20


21/42

2011IntegerResearch&ConsulZng,LLC

Figure9:ProcessandOutcomeMeasurementandManagementforaStandardAssessmentandReferralInterven\on


21


22/42

V.MEMBEROUTCOMESSURVEYTheProject Committee agreedthat itwas importantto determineAIRSmemberinterest

indevelopingI&R/Aoutcomes concepts andapplications. An initialdraftwas reviewed by

the CommitteeandthefinishedsurveywasmountedonlinefromApril27ththroughMay

5th.Thefollowingtextreviewsresponseratesandtheanswerstoeachsurveyquestion.

ResponseRates

Thesurveygathered117responsesoverthecourseofoneweek,amountingtoaresponse

rateof9%ofthemembership(topfigure,leftpie). Responsescamefrommembersin60

percentofthestates,provinces,andoverseaslocations(topfigure,rightpie).

Ananalysis of37AIRSsurveys during2009and 2010showeda medianresponserateof

99,placingthe responserateforthe outcomessurveywithinthethirdquartileforthispe-

riod(barchart).Asanotherbasisofcomparison,the2010AIRSBoardelectionsgathered

thesamenumberofresponses(117)astheoutcomessurvey.SURVEYQUESTION2:2.Pleasedescribetheservicefocusofyourorganiza\on.

Checkmorethanoneonlyifappropriate.

Themajority ofrespondents (81%) indicatedthatthey havea singleservicefocus(blue

pieslice); 19%indicatedthat they have multipleservicefocuses(greenpie slice). The

largestsegmentsofrespondentscamefromAIRSmembersprovidingI&R/Atotheaging/

disabledpopulation(40%)and tothe general community(37%),withsmaller segments

serving special populations and multiplepopulations, and providing crisis intervention

services.Thesereflectafaircross-sectionofthemembership.


22


23/42

SURVEYQUESTION3:Fortheseaudiences,howimportantisittodaytocreate

aframeworkfordefiningandmeasuringI&Routcomes?

Thevastmajorityofrespondents answeredthatitwasveryim-

portantto createanI&R outcomesframework atthe presentfor thepurposes of theindividualrespondent(74%), theI&R

organization(75%),andfunders(76%). Smaller fractionssawit

as very important to establish sucha framework for the pur-

poses of service providers(57%) andlocal government (56%).

Fractions of5percentorless oftherespondentsfeltthatcreat-

inganoutcomesframeworkwasunimportant.

SURVEYQUESTION4:Fortheseaudiences,how importantwillitbetwoyears fromnowto

haveaframeworkfordefiningandmeasuringI&Routcomes?

Theanswerswere verysimilartothepreviousquestion. Across

thefive audiences witha possiblestake inI&R outcomes, the

fraction of respondents deeming an outcomes framework to

beveryimportantincreasedoverQuestion3frombetween5%

and13%.


23


24/42

SURVEYQUESTION5:Forthesepurposes,howimportantisittoyourorganiza\ontocreateaframeworkfordefiningandmeasuringI&Routcomes?

Thisquestionsoughttodetermine the importanceofestablishinganoutcomesframeworkfor securingfunding,forsystemtrouble-

shooting, andfordemonstratingthevalueofI&Rtothepublic. Themajorityofrespondentsdeemeditveryimportantto havea

frameworkforeachofthesepurposes. Fractionsofmorethan70%feltthe frameworkwas veryimportanttosecurefundingfromstategovernment(78%),federalgovernment(77%),andfoundations(74%).Respondentsrangingbetween55%and69%deemedit

veryimportantforthe purposesofgettingfundingfromlocalgovernmentandtheUnitedWay,fortroubleshootingsystembarri-

ers,andfordemonstratingthevalueofI&Rtothepublic.


24


25/42

SURVEYQUESTION6:How helpful to your organiza\onwould itbe to for AIRS todevelopan

OutcomesFrameworkthatincludes...

Thevast majority of respondentsfelt itto be very important to

develop a framework that includes clear definitions of outcomes(87%),methods formeasuringoutcomes(90%),and informationon

bestpracticesandimprovingoutcomes(87%).

SURVEYQUESTION7:

Whatisthebestwayofdescribingthesubjectof"outcomes"inthecontext

ofInforma\onandReferral?Do you think "outcomes"shouldstandalone

asadis\nc\veconceptandword,ordoesitlogicallyfallundertheumbrella

ofI&R"QualityAssurance"?

Themajority ofrespondents(58%)feltthatI&ROutcomeswasthe

clearestway ofdescribingthesubjectinthecontextofI&R,withsub-

stantially smallerfractionsvoting forI&R QualityAssurance(21%)orbeinguncertain(18%).


25


26/42

Narra\veResponses

Respondentswereabletoprovidecommentsforeachquestion,andtheiranswerswererecordedandanalyzed.

Fivethemesemergedfromatotalof159narrative

responses. The most-frequently voiced of these

(#1 at right)is a desireto have a setof commonI&Routcomesmeasuresthatcanbeusedtocom-

pare the performance of I&R programs to each

otherortoaservicebenchmark.

The secondgroup (#2) warrants a follow-up be-

cause the respondents indicate they are already

using anoutcomes protocolof somekind. Valu-

ableinsightsorpractices mayberevealed byinves-

tigatingthesefurther.

Thethirdtheme(#3)encompassedfivesmallerresponseclusters thatexpressedsupportforoutcomemeasuresasawayofimprov-

ingservice,demonstratingvalue,gainingfunding,andevaluatingeffectiveness.

TwolesserthemeswerethedesiretoincorporateoutcomemeasureswithexistingI&Rsoftware(#4)andsupportforsuchmeasures

providedtheydidntburdenstaffthatarealreadystretchedtothelimit(#5).

ASamplingofResponses

1. QuantitativeandQualitativeEvaluationiscritical...weneedtoknownowmorethaneverthatoureffortsaretruly

makinga difference-a documentabledifference. Wecan'taffordtospend timeandmoneyon thingsthataren't

reallygettingtheinformationtothepeoplewhoneedit.Thanksfortakingupthisissueofevaluation.(UnitedWayAgency/2-1-1)

2. Ittakesourserviceto thenextlevel-helps usbe moreaccountableandis criticaltoprogramevaluationandim-

provement...(PublicHealthAssociation)


26


27/42

3. Twoyears fromnowthefundingavailableisgoingtobeevenmoredifficulttoaccess.Allthemorereasontodoit

today...(CommunityActionPartnership)

4. Our stateis coveredby aconsortiumof providersand weneed thisframeworkto drive serviceuniformity and

demonstrateatruesystem.(UnitedWayAgency/2-1-1)

5. Arewejustproposingdevelopinga frameworkforI&Rperse (asanindustrytool)orareweproposingamodelto

beusedtodeterminetheeffectivenessofI&RANDtheservicestowhichwerefer?Thelatterbeingpreferableasit

wouldprovideforamore'world-view'modelthatfunderscanusetobetterservepeople/communities/aswellasto

bettercommunicatethevalueofourservices.(ComprehensiveCommunityI&R/2-1-1)[italicsadded]

6. Wedefinitelyneed a frameworktomeasure I&Routcomes. Wehave beentryingtodevelopa processforyears

withoutsuccess.(HospitalMedicalCenter)

7. I'mconstantly ongrantstoexplainthe outcomes andI finditvery difficultespeciallywhen you're not following

throughtofindoutifpeoplecalledthereferralsorwerehelped.(SeniorInformation&AssistanceAgency)

8. Iwouldliketoseeafewdifferenttoolsdevelopedforconsideration;thenpossiblyallowAIRSmemberstovoteon

utilization.I wouldthinkitwouldbeimportantforall tousethesametools toensurecredibilityand uniformityfor

reviewers/fundingsources. ThereshouldalsobesufficienttrainingprovidedfreeofchargetoallI&Rprovidersre-

gardingthe toolanditsfunction/utilization.Thankyouforallowingustoparticipateinthissurvey!(CountyCoun-

cilonAging)

9. Wearecontinuouslyaskedbyfunders foroutcomesotherthantheactivity performed(referralsandcallshandled).

Theinterest is inknowingresultsofthe activity,such as the percentageofcallerswhoconnectedwithservices or

whoselives wereimprovedindefinable ways.AIRSstandards foroutcome-basedfollow-up questions andadequate

samplesizeswouldlendvalidityasweproposeoutcomestofunders,inthatwecouldpointtonationallyrecognizedachievementstandards.(ComprehensiveCommunityI&R/2-1-1)


27


28/42

Thesurvey comments express strongsupport for developinga standardizedI&R/A outcomesframework. They alsoraisesome

trenchantquestions.

Forexample,comment#5asks about thegoalof anI&Routcomes framework: is itto improvethedelivery ofinformationwithin

the scopeofI&R/A,orisit,morebroadly,toenablesystem-wideimprovementcapabilityforthe largerhumanservicesystem?This

echoesaquestionraisedbyUniversityofNebraskaProfessorandAIRSmemberNancyC.Shankinherpaper ExaminingthePotentialBenefitsofa2-1-1System:QuantitativeandOtherFactors :

Additionalexplorationoftherolethat2-1-1s playinlinkingcallerstoappropriate servicesneedstobeundertaken.

Whodetermines whatkind of serviceis appropriate andefficaciousand, therefore,should beincluded ina 2-1-1

database?Should2-1-1sbeinthebusinessofmeasuringoutcomes,suchasclientsachievingself-sufficiency?Ifso,

whatistheirroleinevaluatingprograms?(Shank&Rosenbaum,2003,p.21;italicsadded).

Comment#8suggestsadevelopmentprocess thatinvolves the AIRSmembershipandwhichincludesavote onoutcomesmeasure-

mentalternativesinthe processofdecidinganationalstandard.This kindofenthusiasmwouldbeinstrumentalinmovingthispro-jectaheadsuccessfully.

Thiscomment,and otherslikeit,alsoremindus ofthevariety ofI&R/Aorganizationsinthe AIRSmembership. Suppose fora mo-

mentthataninclusivedevelopmentprocess producedacoresetofI&R/Aoutcomespractices andstandardsthatwasacceptedby

the majorityof AIRSmembers;andfurthersuppose thatwell-designedvariationscould bebuiltinto thismodel toaccommodate the

widerangeof the members institutionalcapacities,technologies,and funderdemands. Perhapsdevelopingbasic,intermedi-

ate,andadvanced outcomesmeasurementmodelswouldgoa longwayto securingthesupportofthe membershipfor this im-

portantwork.


28


29/42


30/42

VII.MOVINGAHEAD

KeyConcerns

Astheprojectunfolded,questions naturallyemergedaboutwhethertheworkshouldproceedand,ifso,how. Hereareseveral of

thekeyconcernsthatwereraised.

FeasibilityfromaMemberPerspec\ve:The ProjectTeamcompiledandgeneratedideasofpotenalvaluetoanI&R/AOutcomes

Framework. Giventhe membershipsgeneral supportofanI&R/Aoutcomesframework(as reflectedin surveyresponses), how

woulditappraisetheTeams specificideas?Are theseideasworthyof furtherdevelopment? Wouldfocus groups bea logical

nextstepinansweringthisquestion?

Member Commitment: Howcommi[ed areAIRSmemberstodevelopinganI&R/AOutcomesFramework? The onlinesurvey

shows thatthereis stronginterestinanOutcomesiniave,and thatmembers havegoodideastocontribute. Theprojectstrat-

egypresentedinthenextfewpagesmakes itclearthata commitmentof memberme,experse,andresourcesisneededifthe

projectistoconnue.

TheRoleofAIRS: Itis clearthatthescopeofthis projectis significantin termsofits impacton I&R/A,its needfor parcipaon,resourcesandfunding,andthemerequiredfordevelopmentandimplementaon.WhatrolecanAIRStakeinthisprocess?As

theleadership, training,andcredenaling body forI&R/Aprofessionals,AIRSholds apreeminentplace inany OutcomesProject

forI&R/A.ShouldAIRSbethedevelopmentenginetodrivethisprocess?

ProjectStrategy:As notedabove,anOutcomesProjectis significantintermsofresource,funding,andmerequirements.What

strategywouldbestbesuitedforsuchaproject?

Insomeways,thisis themost readily answeredoftheseimportantquestions. Avariationofthe classicsystems analysisand

designmodel wouldprovide anexcellentmeans ofconceptualizingand managingan OutcomesProject. Thefollowingtwo

pagespresentanoutlineofaprojectstrategywhichcouldberefinedandexpandedtomeettheneedsofanOutcomesProject.


30


31/42

ProjectStrategy

StatusIndicators:=complete;=startedbutnotcomplete;=notstarted

Pha

Dev

tot

se0:I

lopp

eAIR

dea\on

reliminaryideasfortheOutcomesFramework.GaugememberinterestintheOutcomesFramework.ReviewfindingsandmakerecommendaZons

Board.

1 ProjectTeamgeneratesiniZalOutcomesFrameworkideas

2 ProjectTeamgaugesmemberinterestinOutcomeswithonlinesurvey

3 AIRSstaffprovidesOutcomesProjectReporttoBoard(5/25/11)

4 IntegerpresentsOutcomesProjectReporttoBoard(6/4/11)

5 MakeOutcomesProjectReportavailabletoAIRSMembers(July,2011)

Pha

The

and

se1:

Feasib

extern

easibility

ilityPhaseestablisheswhetherdevelopingandimplemenZnganOutcomesFrameworkisfeasiblefromtheperspecZvesoftheAIRSmembership

alconsZtuents.IftherefinedOutcomesFrameworkisdeterminedtobefeasibleandfundingisavailable,theprojectproceedstotheDesignPhase.

R

eprese

AIRS

Proje

Proje

Proje

Proje

Proje

AIRS

nta\veAc\vi\es:

BoardapprovesFeasibilityPhaseproposal,funding,andprojectplan;appointsProjectTeam

tTeamdefineskeyinternalandexternalconsZtuentsegments

tTeamconductsFocusGroupsandOutreachtointernalconsZtuents;revisesOutcomesFrameworkbasedoninput

ctTeamconductsFocusGroupsandOutreachtoexternalconsZtuents;revisesOutcomesFrameworkbasedoninput

tTeamcanvassespotenZalfunders,humanserviceorganizaZons,andotherparZcipants,andcompilessalientinformaZon

tTeampresentsfindingsandrecommendaZonstoAIRSBoard

BoardapprovesDesignPhaseproposal,funding,andprojectplan;addsresourcesasneededtoProjectTeam


31


32/42

StatusIndicators:=complete;=startedbutnotcomplete;=notstarted

Pha

The

are

Ifth

se2:

Desig

review

eAIRS

esign

PhasetransformsthefindingsfrompreviousphasesintooneormoredesignsthatincorporatekeydefiniZons,metrics,andprocesses.Thedesigns

edwithinternalandexternalconsZtuencies,revisedasnecessary,andpresentedtotheAIRSBoard.

Boardapprovesthedesignsandfundingisavailable,theprojectproceedstotheDevelopmentPhase.

Pha

The

eac

Ade audi

fied,

Ifth

se3:

Devel

pote

tailed iencea

conta

eAIRS

evelopment

pmentPhaseproducesworkingprototypesfromthedesign(s)ofthepreviousphase.EachprototypeisreviewedwithmulZplerepresentaZvesof

Zaluseraudience.TheuseraudienceincludesI&R/AorganizaZonsandmayincludestakeholders,suchasfunders,thatarewillingtoparZcipate.

walkthroughofeachprototypeisconducted,andrevisionsaremadeonthebasisofuserfeedback.I&R/Afieldtestagenciesareselectedfromeach ndpreparedfortesZng.Foreachtestsite,representaZvesfromlocalfundingorganizaZons,serviceproviders,andotherstakeholdersareidenZ-

cted,andengagedinreviewingtheprototypes.

Boardapprovestheprototypesandfundingisavailable,theprojectproceedstotheFieldTesZngPhase.

Pha

The

eac

tors

Bas

req

Ifth Pha

se4:

Field

field

that

dont

ireme

eAIRS e.

ieldTes\ng

esZngPhaseengagesI&R/AagenciesintesZngtheOutcomesFrameworkandMethodssuitableforaparZcularaudience.Adetailedaccountof

testiscreatedandusedtoguidecorrecZonandrefinementofdefiniZons,metrics,andprocesses.Stakeholdersprovidedetailedfeedbackonfac-

illenableorpreventsuccessfuluseofthenewFramework.Acost/benefitmodelisdevelopedtojusZfytheinvestmentinthenewmethods.

heFieldTestresults,theProjectTeamdevelopsanimplementaZonstrategythatincludesphasingbyaudienceorothermembersegment,training

nts,resourcerequirements,andotherfactors.

BoardapprovestheresultsoffieldtesZngandtheimplementaZonstrategy,andiffundingisavailable,theprojectproceedstotheImplementaZon

Pha

The

take

se5:I

AIRS

ntom

plementa\on

utcomesFrameworkisimplementedinthephasesspecifiedbytheimplementaZonstrategy.Ongoingmonitoringandqualityassuranceareunder-

aximizetheeaseandeffecZvenessofthetransiZon.


32


33/42

APPENDICES


33


34/42

Appendix1:References5

Flowers,A.(2009,June23).I&RInnovations:Discoveringthefoundationsofcustomerservice. PresentationforNational AgingI&R

SupportCenterLecture Series,NationalAssociationof StatesUnited forAging andDisabilities,Washington, D.C. Retrieved from

http://www.nasuad.org/documentation/I_R/PP_IRSupportCenter_LectureSeries.pdf

Harris,E.(2008,May21).ABCsofsatisfaction&impactsurveys .PresentationattheAllianceofInformation&ReferralSystems30th

AnnualTrainingandEducationConference,Houston,TX.

Retrievedfromwww.airs.org/files/public/Conference08_Management_SatisfactionImpactSurveys.ppt

Lampkin,L. M.&Hatry,H.P.(2003).Keystepsinoutcomemanagement.SeriesonOutcomeManagementforNonprofitOrganiza-

tions.Washington,DC:TheUrbanInstitute.Retrievedfrom http://www.urban.org/publications/310776.html

Latzer,A.(2008,May21).211Centers:Doingit,thehowofmetrics;whattocollectandhowtouseit! PresentationattheAllianceof

Information&ReferralSystems30thAnnualTrainingandEducationConference,Houston,TX.

Retrievedfromwww.airs.org/files/public/Conference08_211_HowOfMetrics.ppt

Latzer,A.(2010,May24).Quality,training,andcoachingBestpracticesforI&Rcontact centers.PresentationattheAllianceofIn-

formation&ReferralSystems32ndAnnualTrainingandEducationConference,Rochester,NY.

Retrievedfromwww.airs.org/files/public/Conference2010/ServiceDelivery_QualityTrainingCoaching.pptx

Levinson,R.W.(2002).Newroutestohumanservices:Informationa ndreferral.NewYork,NY:Springer.[Springer Publishing Companyoffersa 20%discountto AIRSmembersfor purchaseof Levinson'sbook. It isavailable atthe Springer PublishingCompany

website.AIRSmembersshoulduse PromoCodeAIFS.Shipping withintheUnited Statesisfreewhenordering throughtheSprin ger website.Usedcopiesmay

beavailableatamazon.com,PowellsBooks,orotherbooksellers.]

Marin,M.(2007,May7).Metricsfor2-1-1centersandsystems:A policy dialogue. presentationattheAllianceofInformation&

ReferralSystems29thAnnualTrainingandEducationConference,Jacksonville,FL.

Retrievedfromwww.airs.org/files/public/Metrics_for_211_Centers_and_Systems.ppt

Michigan211,Inc.(2011).Michigan211businessplan.Lansing,MI:Author.


34

5Wherepossible,linkstoarticles,books,andpresentationshavebeenprovided.ThelinkswereoperativeasofJuly15,2011,butmaysubsequentlychange.
http://www.airs.org/files/public/Metrics_for_211_Centers_and_Systems.ppthttp://www.airs.org/files/public/Metrics_for_211_Centers_and_Systems.ppthttp://www.amazon.com/New-Routes-Human-Services-Information/dp/0826123937/ref=sr_1_2?ie=UTF8&qid=1310479865&sr=8-2http://www.powells.com/s3?kw=&title=new+routes+to+human+services&author=&publisher=&section=&class=0&binding=0&sort=by_relevance&location=all&received_date=0&perpage=25&isbn=http://www.springerpub.com/product/9780826123930http://www.airs.org/files/public/Conference2010/ServiceDelivery_QualityTrainingCoaching.pptxhttp://www.airs.org/files/public/Conference2010/ServiceDelivery_QualityTrainingCoaching.pptxhttp://www.airs.org/files/public/Conference08_211_HowOfMetrics.ppthttp://www.airs.org/files/public/Conference08_211_HowOfMetrics.ppthttp://www.urban.org/publications/310776.htmlhttp://www.urban.org/publications/310776.htmlhttp://www.airs.org/files/public/Conference08_Management_SatisfactionImpactSurveys.ppthttp://www.airs.org/files/public/Conference08_Management_SatisfactionImpactSurveys.ppthttp://www.airs.org/files/public/Metrics_for_211_Centers_and_Systems.ppthttp://www.airs.org/files/public/Metrics_for_211_Centers_and_Systems.ppthttp://www.powells.com/s3?kw=&title=new+routes+to+human+services&author=&publisher=&section=&class=0&binding=0&sort=by_relevance&location=all&received_date=0&perpage=25&isbn=http://www.powells.com/s3?kw=&title=new+routes+to+human+services&author=&publisher=&section=&class=0&binding=0&sort=by_relevance&location=all&received_date=0&perpage=25&isbn=http://www.amazon.com/New-Routes-Human-Services-Information/dp/0826123937/ref=sr_1_2?ie=UTF8&qid=1310479865&sr=8-2http://www.amazon.com/New-Routes-Human-Services-Information/dp/0826123937/ref=sr_1_2?ie=UTF8&qid=1310479865&sr=8-2http://www.springerpub.com/product/9780826123930http://www.springerpub.com/product/9780826123930http://www.springerpub.com/product/9780826123930http://www.springerpub.com/product/9780826123930http://www.airs.org/files/public/Conference2010/ServiceDelivery_QualityTrainingCoaching.pptxhttp://www.airs.org/files/public/Conference2010/ServiceDelivery_QualityTrainingCoaching.pptxhttp://www.airs.org/files/public/Conference08_211_HowOfMetrics.ppthttp://www.airs.org/files/public/Conference08_211_HowOfMetrics.ppthttp://www.urban.org/publications/310776.htmlhttp://www.urban.org/publications/310776.htmlhttp://www.airs.org/files/public/Conference08_Management_SatisfactionImpactSurveys.ppthttp://www.airs.org/files/public/Conference08_Management_SatisfactionImpactSurveys.ppthttp://www.nasuad.org/documentation/I_R/PP_IRSupportCenter_LectureSeries.pdfhttp://www.nasuad.org/documentation/I_R/PP_IRSupportCenter_LectureSeries.pdf


35/42

Miloff,M. & Associates(2007).Tappingthe211InformationGoldmine.[Reportpreparedfor 211Ontario].Toronto,Ontario,Can-

ada.

MunicipalBenefits Committee,Ontario211 Team,UnitedWays ofOntario(2007).WhobenefitsfromOntario211?[Reportpre-

paredforAssociationofMunicipalitiesofOntario].Toronto,Ontario,Canada.

Nayyar-Stone,R.,&Hatry,H.P. (2003).Findingoutwhathappenstoformerclients.Series onOutcomeManagementforNonprofit

Organizations.Washington,DC:TheUrbanInstitute.Retrievedfrom http://www.urban.org/publications/310815.html

Netten, A., &Forder, J.(2008).Measuringthe outputsof informationand adviceservices:Initialreport. (PSSRU Discussion Paper

2543/2)RetrievedfromPersonalSocialServicesResearchUnit,UniversityofKent,Canterburywebsite:http://www.pssru.ac.uk/pdf/dp2543_2.pdf

Neuffer,M.H.(2011).AIRSresearchatlas .Retrievedfromhttp://integerconsult.com/airs/

Reisz,I.,&Goodman,G. (2010).Whyeldersdonotfollowuponsuggestionstheyrequested:AnexploratorystudybyCareforElders

fundedbytheCityofHouston,DepartmentofHealthand HumanServices[ReportforCityof Houston, Department ofHealth and

HumanServices].

Saxton,M.L.,Naumer,C.M.,&Fisher,K.E. (2007).2-1-1Informationservices:Outcomesassessment,benefit-costanalysis

andpolicyissues. GovernmentInformationQuarterly,24,186-215.Purchasedat:http://www.sciencedirect.com/science/article/pii/S0740624X06000463

Shank,N. C.,&Rosenbaum,D. I.(2003).Examiningthepotentialbenefitsofa2-1-1system: Quantitativeandotherfactors.Infor-

mationandReferral,25,1-26.Retrievedfrom:http://digitalcommons.unl.edu/publicpolicyshank/2/

Sheppard, V.&Christopher,J. (2001).Measuringoutcomesinthedeliveryofinformationandreferralservices: Maternalandchild

healthexample. InformationandReferral,23 ,189-235.RetrievedfromAIRSarchives.

Sim,S.C.,&Rocha,K.A.(1998).Aqualitativeandquantitativeapproachinevaluatingatelephonebasedinformationandreferral

program.InformationandReferral,20 ,1-14.RetrievedfromAIRSarchives.


35
http://www.sciencedirect.com/science/article/pii/S0740624X06000463http://www.sciencedirect.com/science/article/pii/S0740624X06000463http://www.sciencedirect.com/science/article/pii/S0740624X06000463http://integerconsult.com/airs/http://www.pssru.ac.uk/pdf/dp2543_2.pdfhttp://www.pssru.ac.uk/pdf/dp2543_2.pdfhttp://www.urban.org/publications/310815.htmlhttp://digitalcommons.unl.edu/publicpolicyshank/2/http://digitalcommons.unl.edu/publicpolicyshank/2/http://www.sciencedirect.com/science/article/pii/S0740624X06000463http://www.sciencedirect.com/science/article/pii/S0740624X06000463http://integerconsult.com/airs/http://integerconsult.com/airs/http://www.pssru.ac.uk/pdf/dp2543_2.pdfhttp://www.pssru.ac.uk/pdf/dp2543_2.pdfhttp://www.urban.org/publications/310815.htmlhttp://www.urban.org/publications/310815.html


36/42

Thompson,M.,&Luther,J.(2009,June1).Qualityassurance,customersatisfactionandfollow-up:Definitionandpracticalexperi-

ence.PresentationattheAllianceofInformation&ReferralSystems31stAnnualTrainingandEducationConference,Reno,NV.

RetrievedfromAIRSarchives.

UnitedWayofAmerica. (2000).Agencyexperienceswithoutcomemeasurement:Surveyfindings .

Retrievedfromhttp://www.unitedwayslo.org/ComImpacFund/10/Experiences_Outcome_Measure.pdf

U.S.DepartmentofHealthandHumanServices,CentersforDiseaseControland Prevention,OfficeoftheDirector,OfficeofStrat-

egy and Innovation. (2005). Introduction to programevaluation forpublic health programs: A self-study guide. Retrieved fromhttp://www.cdc.gov/eval/evalguide.pdf

Windle, K.,Netten, A.,Caiels, J.Masrani,R., Welch, E.,& Forder,J. (2010). Measuringtheoutcomesof informationandadvice

services:Finalreport. (PSSRUDiscussion Paper2713). RetrievedfromPersonal Social Services ResearchUnit, UniversityofKent, Can-

terburywebsite: http://www.pssru.ac.uk/pdf/dp2713.pdf

Wood,C.,&Fay,P.(2010,May24).Qualitynotquantity:Afocusontheresourcedatabase .PresentationattheAllianceofInforma-

tion&ReferralSystems32ndAnnualTrainingandEducationConference,Rochester,NY.RetrievedfromAIRSarchives.

Woods,D., & Eyre, J.(2003). 211forall Ontario: Bringingpeopleandservicestogether. FinalReportJuly2003.Part2:Torontos

211experience.[ReportpreparedforInformOntario,UnitedWaysof Ontario,CommunityInformationToronto,and UnitedWayof

GreaterToronto].Toronto,Ontario,Canada.


36
http://www.pssru.ac.uk/pdf/dp2713.pdfhttp://www.pssru.ac.uk/pdf/dp2713.pdfhttp://www.cdc.gov/eval/evalguide.pdfhttp://www.cdc.gov/eval/evalguide.pdfhttp://www.unitedwayslo.org/ComImpacFund/10/Experiences_Outcome_Measure.pdfhttp://www.unitedwayslo.org/ComImpacFund/10/Experiences_Outcome_Measure.pdf


37/42

Appendix2:LogicModels

DETAILEDLOGICMODELSBASEDONSAXTON-NETTENFRAMEWORK

Becauseofitssize,themodelhasbeendividedintothethreelevelsofoutcomesasdefinedbySaxtonandNeen.Thisporonofthe

modelshowsshowstheindividualclientseekingandreceivinghelp.Posiveandnegaveoutcomesaresuppliedforeachstage;the

detailofprocessstepshasbeenomied.(A}ercomplengthismodel,theOutcomesCommieedecidedthatworkingatahigherlevelofabstraconwouldbemorehelpful,andthatthisdegreeofdetailwouldhavetobeelaboratedatalaterme.)


37


38/42


Thesecondporonofthemodeldealswiththeorganizaonallevelofoutcomes.Toconservespace,onlytheposiveoutcomeshave

beenshowninthisdiagram.


38


39/42


Thethirdporonofthemodeldealswiththesocietallevelofoutcomesverybriefly.Toconservespace,onlytheposiveoutcomes

havebeenshowninthisdiagram.


39


40/42

Appendix3:211LADevelopmentalScreeningProjectFactSheet


40


41/42

Appendix4:AIRSOutcomesSurvey

1.PLEASEIDENTIFYYOURSELFANDYOURORGANIATION

2.PLEASEDESCRIBETHESERVICEFOCUSOFYOURORGANIATION.CHECKMORETHANONEONLYIFAPPROPRIATE.

I&RandI&R?AforAgingand/orDisabledPopulaZons I&RforOtherSpecialPopulaZons(e.g.,Youth,Families) I&RforGeneralCommunity

CrisisIntervenZonServices Other(specify)

3.FORTHESEAUDIENCES,HOWIMPORTANTISITTODAYTOCREATEAFRAMEWORKFORDEFININGANDMEASURINGI&ROUTCOMES?

(a)YouasanI&RorI&R/AProfessional (b)YourOrganizaZon (c)HumanServiceProvidersinYourCommunity(d)YourFunders (e)YourLocalGovernment

Scale:NoOpinionNotImportantSomewhatImportantVeryImportant

4.FORTHESEAUDIENCES,HOWIMPORTANTWILLITBETWOYEARSFROMNOWTOHAVEAFRAMEWORKFORDEFININGANDMEASURINGI&ROUTCOMES?

(a)YouasanI&RorI&R/AProfessional (b)YourOrganizaZon (c)HumanServiceProvidersinYourCommunity(d)YourFunders (e)YourLocalGovernment


5.FORTHESEPURPOSES,HOWIMPORTANTISITTOYOURORGANIATIONTOCREATEAFRAMEWORKFORDEFININGANDMEASURINGI&ROUTCOMES?

( a) G eng funding fromloca lgove rnme nt( city,county) ( b) G eng funding fromstategove rnme nt ( c) G eng funding fromfe de ra lgove rnme nt(d)GengfundingfromUnitedWay (e)Gengfundingfromotherprivatesources(e.g.,foundaZons) (f)TroubleshooZngservicesystembarriersandproblems(g)DemonstraZngthevalueofI&Rtothegeneralpublic (h)Other(specify)


6.HOWHELPFULTOYOURORGANIATIONWOULDITBETOFORAIRSTODEVELOPANOUTCOMESFRAMEWORKTHATINCLUDES:

(a)CleardefiniZonsofwhati&Routcomesare (b)GuidelinesandmethodsformeasuringI&Routcomes (c)InformaZononbestpracZces,methodsofimprovingoutcomes,etc.


7.WHATISTHEBESTWAYOFDESCRIBINGTHESUBJECTOF"OUTCOMES"INTHECONTEXTOFINFORMATIONANDREFERRAL?DOYOUTHINK"OUTCOMES"SHOULDSTANDALONEASADISTINCTIVECONCEPTANDWORD,ORDOESITLOGICALLYFALLUNDERTHEUMBRELLAOFI&R"QUALITYASSURANCE"?

Scale:NoOpinionNotCertainI&ROutcomesI&RQualityAssurance

8.WHATKINDSOFOUTCOMESMEASUREMENTTOOLSWOULDBEUSEFULTOYOURI&RORGANIATION?


41


42/42


42

airs outcomes report

Documents