1. 2 design and funding forum: emerging models for innovation september 5, 2008 an rbs evaluation...

2

Design and Funding Forum:Emerging Models for Innovation

September 5, 2008

An RBS Evaluation Framework

3

Goals for Today

To review the values and principles underlying the evaluation framework

To introduce the individual outcomes component

To explain how the evaluation framework will fit into the RBS program design

To get your ideas about implementing the evaluation framework

4

Values and Principles Underlying the Proposed Evaluation FrameworkThe evaluation should:

Yield the information needed to meet the evaluation mandates of AB1453

“Follow and not drive” the RBS program design

Be as unobtrusive as possible

Be affordable

Have minimal administrative and fiscal burden on participating providers and counties

Provide some of the clinical and management information needed by the local RBS program stakeholders to monitor local program operations

Be acceptable to all RBS stakeholders (children and families, providers, counties, CDSS, and members of the RBS Reform Coalition)

5

The Evaluation Mandates of AB1453Excerpted from the Legislation (emphasis added):

“Voluntary agreements by counties and nonprofit agencies shall …

(3) Provide for an annual evaluation report, to be prepared jointly by the county

and the private nonprofit agency. The evaluation report shall include analyses

of the outcomes for children and youth, including achievement of

permanency, average lengths of stay, and rates of entry and reentry into

group care. The evaluation report shall also include analyses of the

involvement of children or youth and their families, client satisfaction, the

use of the program by the county, the operation of the program by the private

nonprofit agency, payments made to the private nonprofit agency by the county,

actual costs incurred by the nonprofit agency for the operation of the program,

and the impact of the program on state and county AFDC-FC program costs.

The county shall send a copy of each annual evaluation report to the

director, and the director shall make these reports available to the Legislature

upon request.”

6

AB1453 Evaluation Mandates Table

8. The impact of the operation of the program on state and county AFDC-FC costs

7. Actual costs incurred by the private non-profit agency for the operation of the program

6. Payments made to the private non-profit agency by the county

Fiscal Outcomes

5. The operation of the program by the non-profit agency

4. The use of the program by the countySystems Operations

3. Client satisfaction

2. Analyses of the involvement of children or youth and their families

1c.Rates of entry and reentry into group care

1b.Average lengths of stay

1a.Achievement of permanency

1. Analyses of the outcomes for children and youth includingClient Outcomes

7


8. The impact of the operation of the program on state and county AFDC-FC costs

7. Actual costs incurred by the private non-profit agency for the operation of the program

6. Payments made to the private non-profit agency by the county

Fiscal Outcomes

5. The operation of the program by the non-profit agency

4. The use of the program by the countySystems Operations






1. Analyses of the outcomes for children and youth includingClient Outcomes

The evaluation framework for these elements

will be developed based on the RBS program

designs proposed in the Voluntary Agreements

due in mid-October

8

Client Outcomes





1. Analyses of the outcomes for children and youth including[but not limited to]




9

Client Outcomes




1. Analyses of the outcomes for children and youth including[but not limited to]



RBS evaluation Challenge - meet the requirements of:

AB1453 mandated evaluation client outcomes (1, 1a-c., 2, and 3 above) and

client outcomes identified by the local RBS teams during the July Technical Assistance Workshops (1d and 1e above) while

remaining true to the seven values and principles presented earlier


County Identified Outcomes

1d.Well Being (e.g. safety, stability, behavioral improvement, sense of hope, others)

1e.Educational progress

and progress toward permanency

10

Meeting the Challenge: The Proposed Framework

Three pre-existing data collection instruments

County Level CMS/CWS Data currently collected quarterly

The Youth Services Survey (YSS) and

The Youth Services Survey for Families (YSS-F) promulgated by the California Department of Mental Health (DMH)

An assessment instrument with a national track record

The Child and Adolescent Needs and Strengths (CANS)assessment instrument

11

Maximum Data Collection Model

Minimum Data Collection Model

Treatment Planning /Utilization Review

YSS/YSS-F

CANS

CWS/CMS

YSS/YSS-F

CANS

CWS/CMS

RBS Treatment Program

Year TwoYear One

Meeting the Challenge: How This Might Work

Intake Exit90

days180 days

270 days

365 days

90 days

180 days

270 days

Residential Treatment Community Based Care

2 data points

2 data points

9 data points

9 data points

9 data points

2 data points

Children & Families

12

What Degree of Data Collection is Manageable?

To be determined:

The frequency of data collection needed by RBS stakeholders to understand what is happening with children and families during RBS enrollment

Costs associated with and funding available to support:

Staff and client time spent in instrument administration

Data collection and compilation

Data reporting

13

RBS Evaluation FrameworkDesign Values and Principles CANSYSS & Y

SS-FExisting CWS/CMS Measures

4. Is it affordable?

Meeting the Challenge: Why These Instruments?

5. Does its use minimize the administrative and fiscal burden on participating providers and counties?

3. Can it be or is it implemented “unobtrusively”?

2. Does it follow - and not drive - the RBS program design?

1. Will it yield the information needed to meet the legislative mandates of AB1453 - the legislation authorizing the RBS initiative? Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes

Yes Yes Yes

Yes Yes Yes

6. Will it provide some of the clinical and management information needed by the local RBS program stakeholdersto monitor their program operations?

Yes Yes Yes

14

How the Data Collection Could be Operationalized

1. County Level CWS/CMS Data currently collected quarterly Work with participating RBS counties and The Center for Social Services

Research (CSSR) to establish standard procedures for identifying or “flagging” children enrolled in RBS in CWS/CMS

Generate automated reports on RBS children on current CWS/CMS measures relevant to RBS including:

RBS Stakeholders’ Roles in this Process Continue to report CWS/CMS data accurately and on time

Flag the records of children enrolled in RBS according to the standard procedures we establish

◆ Permanency◆ Lengths of Stay◆ Rates of entry and reentry

into group care

◆ Safety◆ Placement stability◆ Restrictiveness, and◆ Educational achievement

15

2. The Youth Services Survey (YSS) and theYouth Services Survey for Families (YSS-F) Assess the current use of the instruments and how data are collected

and compiled

Conduct an item level review of the instruments to determine which items are most relevant to RBS

If appropriate, create abbreviated versions of the instruments

RBS Stakeholder’s Role in this Process Provide feedback to above results

Imbed client satisfaction instruments in the ongoing periodic treatment planning and/or utilization review processes

Collect, compile and submit client satisfaction data to the RBS Consortium


16

3. Child and Adolescent Needs and Strengths Assessment Instrument (CANS) Assess the current use of the CANS and how data are collected and

compiled

Conduct an item level review of the CANS to determine which items are most relevant to RBS

If appropriate, create an abbreviated version of the CANS

RBS Stakeholder’s Role in this Process

Provide feedback to above results

Imbed the CANS in the ongoing periodic treatment planning and/or utilization review processes

Collect compile and submit CANS data to the RBS Consortium


17

RBS Evaluation Next Steps Proposed Timeline

Client Outcomes Measures

September October November December

1. Contact RBS county decision makers to assess the acceptability of the proposed evaluation framework

3. Work with CSSR and RBS counties to establishprocedures for identifying RBS clients in CWS/CMS

2. Assess the current use of the YSS, YSS-F and CANS in the RBS counties

4. Report on findings from 1-3 to the RBS Consortium and Steering Committee

6. Create modified instruments as per 1-4 above

7. Submit modified instruments for review and comment to RBS Consortium and Steering Committee and RBS county decision makers

8. Finalize instruments based on 6 and 7 above

1-15 16-30

1-15 16-31

5. Convene RBS Evaluation Subcommittee

1-15 16-31 1-15 16-30

18

Voluntary Agreements Submitted

Systems Operations Measures

Fiscal Outcomes Measures

September October November December

1-15 16-31

1. Determine proposed Systems Operations Measures based on October Evaluation Subcommittee meeting and Voluntary Agreement

2. Propose Systems Operations Evaluation Framework at November RBS Forum

1. Determine proposed Fiscal Outcomes Measures based on October Evaluation Subcommittee meeting and Voluntary Agreement

2. Propose Systems Operations Evaluation Framework at November RBS Forum

1-15 16-30 1-15 16-31

1-15 16-30

RBS Evaluation Next Steps Proposed Timeline

19

Your Feedback

1. Your name

2. The RBS location you’re from

3. Whether you’re representing a county or a provider

Please sharing your thoughts on what you’ve just seen and heard by answering the following questions:

1. What, in your opinion, are the strengths of the proposed RBS evaluation design?

2. What are your concerns about the evaluation design?

3. What additional questions do you have?

Please introduce your comments by telling the group:

Thank you for participating!

Your feedback will help the RBS Evaluation Team and Steering Committee as we develop a practical and affordable evaluation approach

21

Themes from Demonstration Site Presentations

California RBS Reform ForumCalifornia RBS Reform ForumSacramento, CASacramento, CASeptember 5, 2008September 5, 2008

22

Convergence of Technologies

Resource Utilization

Management

Comprehensive Care Coordination

Practice Based on Evidence

23

Maximize revenue to create best array of services possible for each child and family

Arc of enrollment

Family voice and choice

Vehicles for continuity

Pass-off the funds, rather than pass-off the service relationship

Comprehensive care coordination themes

24

Connections for youth/family/community

Use of standard assessment measures

Enhanced referral process & matching

Flex worker roles

Increase worker time for team participation

Open up the “black box” of facility-based care

Provider involvement in planning selected providers informing operational details

Evidence based practice themes

25

Capture cost savings with initial innovations

Utilize flexible pool of AFDC-FC funds

Creative tension between “packaged” funding & “blended” funding

RCL Alternative

Treatment incentives and outcome incentives

Creating a better niche to fill the space currently occupied by group care

Resource utilization themes

27

Proposed CDSS Review Process

What would you attempt to do if you knew you could not fail?

-Unknown

28

Deliverables Submitted to CDSS

•RBS Analyst will coordinate internal routing and reviews.•Counties identify contact(s).•Electronic MS Word format.

3 Week ReviewInternal CDSS areas reviewing deliverables will work with county contact(s) to resolve minor issues.

If needed, schedule meeting or conference call with County to clarify and resolve issues.

Meet with County during 4th week

(optional)

Tentative ApprovalCounty finalizes deliverables and resubmits.

Final CDSS Approval

10 Day Review of Resubmitted Deliverables

30

Date Key MilestoneSept 2008 Technical assistance support to demonstration sites

• Sept 9-10 Sacramento• Sept 15-16 Bay Area

Oct 2008 Demonstration sites submit RBS reform deliverables to CDSS for review• Oct 3 Sacramento• Oct 17 Los Angeles, San Bernardino, Bay Area

Nov – Dec 08 CDSS Review Process

Nov 2008 Technical assistance support to demonstration sites

Nov 5-6, 2008 RBS Implementation Forum

Jan - Feb 09 CDSS authorizes regulatory & fiscal waivers needed for approved RBS Reform Plans and secures agreements

Feb – Mar 09 RBS reforms implemented in demonstration sites

Timeline for 2008

31

All proceedings from Forum will be posted on website www.RBSreform.org

Specific technical assistance topics to address in Sept & Oct?

Confirm need for, timing and format for next LIC gathering?

Questions?

Feedback & next steps

32

Karen Gunderson, MSWChief, Child and Youth Permanency BranchCalifornia Department of Social Services

Email: [email protected]

Phone: (916) 651-7464

Fax: (916) 651-0673

1. 2 design and funding forum: emerging models for innovation september 5, 2008 an rbs evaluation...

Documents

evaluation frameworkto

evaluation frameworkvalues

rbs program designbe

rbs program designto

rbs evaluation frameworkdesign

private nonprofit agency

private nonprofit agency

annual evaluation report