WHAT FOLLOWS BELOW IS AN EXAMPLE OF AN OUTCOME EVALUATION

THIS WAS AN EXPERIMENTAL “CLINICAL CASE MANAGEMENT PROGRAM” RUN IN COLLABORATION WITH THE DEPT. OF HUMAN RESOURCES AND THE DEPT. OF COMMUNITY AFFAIRS (HOUSING).

The goal of this particular “Clinical Case management Program” was to help people with chronic and severe psychiatric diagnoses to ‘live successfully and independently within the community’.

Research questions were : 1. How effective is out program in meeting objectives one through five (did the program make a difference question – see last weeks notes)? 2. Which services contributed most to program effectiveness (association or connection question).

Program Objectives were. 1. to reduce the number of unplanned disruptions to independent living. 2. To reduce ‘breaks’ in independent living. 3. to reduce the need for unplanned and/or emergency services. 4. to reduce the need for changes to or interventions in housing situation. 5. to reduce the number of critical incidents in people’s lives. 6. to provide services that would be effective in helping clients live independently.

Because the objectives were measurable and the questions involved the effectiveness of the program, the evaluation was summative – an outcome evaluation!!!!

IN ORDER TO ANSWER EACH RESEARCH QUESTION, THE PROGRAM OBJECTIVES NEEDED TO BE OPERATIONALIZED!

THIS WAS DONE BY CREATING OUTCOME MEASURES FOR EACH OBJECTIVE! BECAUSE THE PRGRAM OBJECTIVES WERE VERY SPECIFIC, IT WAS EASY!

Outcomes measures for program objective – one through 5 -were: #1 scores on the use of crisis intervention services #2 number of e.r. visits. #3. Number of days in hospital, number of hospitalization, alternative housing or respite care. #4. number of housing interventions or housing changes #5 number of total critical incidents .

• This was a pilot program

• If successful, it would lead to more programs like it.

• On the next slide is the Program Logic Model that was “developed” for the program model.

• Ultimately it was not used.

INPUTSAlbany & gainesvile

OBJECTIVES ACTIVITIES OUTPUTS OUTCOMESAnd measures of them

IMPACTSAka long term outcomes

Mhc staff1st step staffDhr fundsDhr plannersHousing authority fundsHousing auth planners,TrainingHousing vouchersLandlords,ApartmentsScreening tools,Clients,Evaluator, Program supervisorsHospital authorities. E.R. managers.

1. reduce the number of unplanned disruptions to independent living.

2. reduce ‘breaks’ in indept. living.

3. reduce need for unplanned and/or emrgcy. services.

4. reduce need for changes to or interven. in housing sit.

5. reduce the number of critical incidents in people’s lives.

6 have clients live stably and in one place

7. provide services that would be effective in helping clients live independently.

Training, planning, supervision,Hospitalization, case magemnt.Crisis int. e.r. services. Med. Mgmt.daily living asst.. Linking and referral, housing advkcacy. Program advocacy, peer support grp., med. Educ. Family work/intervention, partial hospitalization (day prgram), job training, job asst., , transportation asst.,Job coaching, alternate hosuing, crisis resolution services, Landlord managament, landlord education, dual dx services, substance abuse tx, financial mgmt., financial assistance, counseling, living skill training, supportive visits, clinical case mgmt., respite care services

# clients going through program#housing vouchers# new landlords & housing

1 scores on the use of crisis intervention services 2 the number of e.r. visditsor unplanned MHC visits

#3. Number of days in hospital, number of hospitalization, alternative housing or respite care#4 number of landlord or neigbor complaints

#5. number of housing interventions or housing changes

#6 number of domestic interventions with roomates, family, neighbors

#7 number of total critical incidents .

#8 identify services used most frequently or intensively that correlate most strongly with reductions in 1 thru 7 above.

1. Stable alternative housing for people with sever psych. Dx,. In the state of Georgia2. Establishment of a working model between housing authority and DHR3. Reduce stigma of mental illness in community4. Increase independent, stable and respectful living for those with chronic and severe dx within GA.5. Reduce housing discrminitaion within the state.

Note that the PLM can ‘capture’ all aspects of the Program that planners and staff wanted

• The next slide shows part of the form that was filled out monthly on each client in the program – is designed to measure the ‘operationalized outcomes’. (in outcomes column # 1 thru 7 on PLM)

• It was the first part of a larger form

Crisis intervention services

Housingcomplaints

Domestic intervention(roommates)

Housing intervention(includes advocacy,

mediation and education with landlord or tenants)

Overall Face to UNPLANNED face contact

Number of hospitalizations,

alternative housing or respite care. (please specify number, type and total days)

Number of emergency room visits or unplanned visits to MHC

MONTHLY Total

Number ofCriticalincidents

5= high level of CI. ongoing or major crisis4 = moderate level

of CI3 = minor level of CI

interventions2 = isolated minor

CI1= no CI

6= Frequent or ongoing major complaints concerns

5=occasional major concerns

4= frequent minor concerns

3= occasional minor concerns

2= infrequent minor concerns

1= no complaints

6= frequent or ongoing major

issues.interventions

5= occasional major issues or intervention

4 = frequentminor issues orintervention3 = occasional

minor issues or

intervention2 = infrequent

minor issues or intervention

1 = none

6= frequent or ongoing major

issues.interventions

5= occasional major issues or intervention

4 = frequentminor issues orintervention3 = occasional

minor issues or

intervention2 = infrequent

minor issues or intervention

1 = none

5 - daily 4 = several times

per week3 = once a week2 = less than one

week1 = no contact or

once per month

______Housing_____Legal_____Medical_____Familial______Domestic______Agency_____Personal_______Other

The second question - which services were most helpful - was more difficult!

6. to provide services that would be effective in helping clients live independently. HERE IS HOW IT WAS OPERATIONALIZED AS AN OUTCOME MEASURE

#6 correlations between the use of certain services listed on the form above (in bold black) and the variables listed on the form above in bold red.

BECAUSE IT IS A QUESTION ABOUT ASSOCIATION (CONNECTIONS), WE HAD TO LOOK AT CORRELATIONS (CONNECTIONS)!!!

In order to answer the question, “Which services contributed most to program effectiveness?” we had to look at two ‘sets’ of variables 1. THE FIRST FIVE OBJECTIVES OPERATIONALIZED BY THEIR OUTCOME MEASURES AND 2. THE USAGE FOR EACH SERVICE/INTERVENTION COMPONENT OF THE PROGRAM.

Again I sat down with the service providers from two different agencies. At first they did not think they provided the same services. The more we talked, it became clear that they did MANY of the same things, they just called them different things.

So what we did was to develop the second part of the instrument and take MONTHLY LOGS of the services provided (the likert scales on the form above measures usage of service in FREQUENCY AND INTENSITY (IN BOLD BLACK ON THE FORM BELOW) AND SEE WHICH SERVICES CORRELATED THE MOST WITH A DECREASE IN THE ITEMS IN BOLD RED ( # OF HOSPITALIZATIONS, HOUSING COMPLAINTS, HOUSING INTERVENTION, CRITICAL INCIDENTS ETC)

Medication management

Daily livingAssistance

Linking and referrals

Advocacy(other than

housing)

Peer supportGroups

Family intervention

Medication education

PartialHospitalization

6= major med difficulties or change w/ assist.

5= regular/daily med. assistance

4= occasional med. Assistance

3= regular/daily med. monitoring

2= occasional med. monitoring

1= no monitoring

5= high level of service, daily or prolonged

4= moderate level daily or more

3= mild level weekly or less

2= minimal level

1= none

5 = high degree of complex linking done

4 = moderate degree of linking

3 = minor degree of linking

2= minimal degree of linking

1 = no linking

5 = high level of ongoing advocacy

4 = moderate level of advocacy

3 = minor level of advocacy

2 = minimal level of advocacy

1 = no advocacy

5 = high level of daily participation

4 = moderate level of daily participation

3 = attends daily2 = attendsoccasionally1 = no attendance

5 = major or prolonged

family intervention4= moderate level

of family intervention

3= a family intervention

2= family contact1 = no family

contact

5 = intensive4 = moderate but

ongoing3 = occasional with

regular check in

2. very infrequent understands meds well

1 = none

6= daily attendance, high level of activity

5= daily attendance, moderate level of activity

4= daily visits low level of activity

3= occasional attendance

2= infrequent attendance

1= no attendance

Transportation assistance

job coaching

Dual dx service

Substance abuse tx

Financial management

Counseling Daily living skill training

Supportive visits

5 = high, daily or almost daily provision

4 = moderate, regular or weekly need

3 = occasional assistance provide

2 = isolated, infrequent assistance

1 = no assistance

5 = ongoing training or support

4= regular job skill development

3= some assistance with job hunting

2= some linking or educatio

1 = no job coaching

5 = high level of service, included daily/weekly group?

4= moderatelevel of, service 3= occasional

service2= infrequent

service1= no service

5= major intervention required

4= major focus in service

3 = moderate focus in service (monitoring but no tx)

2= minimal focus in service

1 = no focus

5= daily or high level of assistance

4 = weekly or moderate level of assistance

3 = intermittent but regular assistance

2= occasional, infrequent assistance

1= no assist.

5 =regular counseling w/ focus on major life issues

4= regular problem-solving s not related to above issues

3= intermittent counseling recurrent issues

2= intermittent counseling different issues

1= no counseling

5 = high degree of training needed

Uses regularly4 = moderate degree

of training3 = intermittent

training2 = infrequent1 = none

6= daily visits or string of daily visits, high levelf support5= weekly/bi-

weekly visits high level of

support4= occasional visits

or moderate level of

support3= infrequent visits

or mild level of support2= minimal level of supportive contact1 = none

Daily living education

Medication education

______Housing_____Legal_____Medical_____Familial______Domestic______Agency_____Personal_______Other

5 - daily 4 = several times per week3 = once a week2 = less than one week1 = no contact or once per month

6= frequent or ongoing majorissues.interventions5= occasional major issues or intervention4 = frequentminor issues orintervention3 = occasional minor issues orintervention2 = infrequent minor issues or intervention1 = none

6= frequent or ongoing majorissues.interventions5= occasional major issues or intervention4 = frequentminor issues orintervention3 = occasional minor issues orintervention2 = infrequent minor issues or intervention1 = none

6= Frequent or ongoing major complaints concerns5=occasional major concerns4= frequent minor concerns3= occasional minor concerns2= infrequent minor concerns1= no complaints

6= daily attendance, high level of activity5= daily attendance, moderate level of activity4= daily visits low level of activity3= occasional attendance2= infrequent attendance1= no attendance

Total Number ofCriticalincidents

Number of hospitalizations, alternative housing or respite care. (please specify number, type and total days)

Overall Face to face contact

Housing intervention(includes advocacy, mediation and education with landlord or tenants)

Domestic intervention(roommates)

Housing complaints

PartialHospitalization

6= daily visitsor string of daily visits,high level of support5= weekly/bi-weekly visitshigh level of support4= occasional visits or moderate level of support3= infrequent visits or mildlevel ofsupport2= minimallevel of supportive contact1 = none

5 =regular counseling w/ focus on major life issues4= regular problem-solving sessions not related to above issues3= intermittent counseling recurrent issues2= intermittent counseling1= no counseling

5= daily or high level of assistance4 = weekly or moderate level of assistance3 = intermittent but regular assistance2= occasional, infrequent assistance1= no assist.

5= major intervention required4= major focus in service3 = moderate focus in service(monitoring but no tx)2= minimal focus in service1 = no focus

5 = high level of service, included daily/weekly group?4= moderatelevel of, service3= occasional service2= infrequent service1= no service

5 = ongoing job training or job support4= regular job skill development3= some assistance with job hunting2= some linking or education1 = no job coaching

5 = high, daily or almost daily provision4 = moderate, regular or weekly need3 = occasional assistance provide2 = isolated, infrequentassistance1 = no assistance

Supportive visitsCounselingFinancial management

Substance abuse treatment

Dual dx group or service

job coachingTransportation assistance

5= high level of CI. ongoing ormajor crisis4 = moderate level of CI3 = minor level of CI interventions2 = isolated minor CI1= no CI

5 = major or prolonged family intervention4= moderate level of family intervention3= a family intervention2= family contact1 = no family contact

5 = high level of daily participation4 = moderate level of daily participation3 = attends daily2 = attendsoccasionally1 = no attendance

5 = high level of ongoing advocacy4 = moderate level of advocacy3 = minor level of advocacy2 = minimal level of advocacy1 = no advocacy

5 = high degree of complex linking done4 = moderate degree of linking3 = minor degree of linking2= minimal degree of linking1 = no linking

5= high level of service, daily or prolonged4= moderate level daily or more than once per week 3= mild level weekly or less2= minimal level weekly or less1= none

6= major med difficulties or change w/ assist.5= regular/daily med. assistance4= occasional med. Assistance3= regular/daily med. monitoring2= occasional med. monitoring1= no monitoring

Crisis interventionFamily intervention

Peer supportGroups

Advocacy(other than housing)

Linking and referrals

Daily livingAssistance

Medication management

• Finally, you might be evaluating a program like “ adult outpatient services in mental health”.

• In a program like this, all of the clients are working on individual issues or objectives

• The program objective then is “to help each client to achieve their individual objectives”

• Because you have “individualized outcomes”, you need some way to “translate” these individual outcomes into some “programmatic measurement”

• We will cover that next week. See chapter 7 in your text