1 creating strong reports new mexico americorps april 20, 2006 sue hyatt, project star coach

11

Creating Strong Reports

New Mexico AmeriCorpsApril 20, 2006

Sue Hyatt, Project STAR Coach

22

AgendaAgenda

Learn some “do’s and don’ts” Learn some “do’s and don’ts” forforperformance measures (PM’s) in performance measures (PM’s) in the APR.the APR.

Reporting TipsReporting Tips

Review of Your Most Recent Review of Your Most Recent ReportReport

33

Project STAR Findings:Common PM and Reporting

Issues

Outputs versus OutcomesOutputs versus Outcomes Intermediate versus End OutcomesIntermediate versus End Outcomes ““Creative Writing” - Indicators and Creative Writing” - Indicators and

TargetsTargets Targets Not Related to InstrumentsTargets Not Related to Instruments Weak Community Need StatementsWeak Community Need Statements

44

REVIEW: Types of Performance Measures OutputsOutputs Intermediate OutcomesIntermediate Outcomes End OutcomesEnd Outcomes

55

OutputsOutputs

Are counts of the amount of services Are counts of the amount of services delivered, work, or products created that delivered, work, or products created that members or volunteers have completed.members or volunteers have completed.

Usually number of people servedUsually number of people served

Do Do notnot answer the question, “what answer the question, “what changed as a result of our work?” changed as a result of our work?”

Do Do notnot provide information on benefits or provide information on benefits or other changes in the lives of beneficiaries.other changes in the lives of beneficiaries.

Examples:Examples:30 students tutored in reading100 tress planted25 houses built benefiting 25 families

66

End OutcomesEnd Outcomes

Answer the question, “So what?”Answer the question, “So what?”

Changes that have occurred in the Changes that have occurred in the lives of beneficiaries that are lives of beneficiaries that are significantsignificantExamples:Examples:Students will not engage in drug or alcohol useIncreasing homeownership among clientsStudents demonstrate better in school behavior

77

Intermediate Intermediate Outcomes Outcomes

Answer the question, “what difference did it Answer the question, “what difference did it make”?make”?Do Do notnot represent the final result you hope to represent the final result you hope to achieve for beneficiariesachieve for beneficiariesLikely preconditions for longer-term gainsLikely preconditions for longer-term gains

When positive are usually a sign that your project When positive are usually a sign that your project is on track to achieve the related end outcomes.is on track to achieve the related end outcomes.

Examples:Examples:

Students report feeling safe and not pressured to engage in Students report feeling safe and not pressured to engage in risky behavior.risky behavior.

Decrease in volunteer attrition.Decrease in volunteer attrition.Students will complete homework assignments.Students will complete homework assignments.

88

Possible Beneficiary Possible Beneficiary ChangesChanges

Outputs:Outputs: Number of People Served Number of People Served Outcomes:Outcomes: How People Changed How People Changed

KnowledgeKnowledge

AttitudesAttitudes

Behavior or SkillsBehavior or Skills

IntermediateIntermediate EndEnd

99

What is an Indicator?What is an Indicator?

Information you collect to determineInformation you collect to determineprogress toward achieving resultsprogress toward achieving results

Concrete and measurableConcrete and measurable Answers the question, “What is myAnswers the question, “What is my

evidence that something has occurred?”that something has occurred?” Output indicator usually begins with Output indicator usually begins with

“Number of...”“Number of...” Outcome indicator usually begins with Outcome indicator usually begins with

“Percent of...”“Percent of...”

1010

Internal Alignment: End Outcome

Members will provide 1 on 1 reading tutoring to third graders at least 3 times a week for 20

minutes/session End Outcome Result: Students have End Outcome Result: Students have

improved proficiency in readingimproved proficiency in reading Indicator: Percentage of students tutored Indicator: Percentage of students tutored

who have improved proficiency in readingwho have improved proficiency in reading Target: 75% (263 of 350) students Target: 75% (263 of 350) students

tutored will have improved proficiency in tutored will have improved proficiency in readingreading

Instrument: Reading GradesInstrument: Reading Grades

1111

Internal Alignment: End Outcome (NOT!)

Members will provide 1 on 1 reading tutoring to third graders at least 3 times a week for 20

minutes/session End Outcome Result: Students have End Outcome Result: Students have

improved proficiency in readingimproved proficiency in reading Indicator: Teachers will report that 75% Indicator: Teachers will report that 75%

students have completed more reading students have completed more reading homework.homework.

Target: Students tutored will show Target: Students tutored will show improved behavior in the classroom and improved behavior in the classroom and be sent to the Principal’s office less be sent to the Principal’s office less often.often.

Instrument: logs and surveysInstrument: logs and surveys

1212

APR Do’s and Don’ts:Do create performance

measurement plans

1313

What is a Performance Measurement What is a Performance Measurement Plan?Plan?

Written description of steps you will take to obtain data to measure outputs and outcomes

The PM plan specifies:• Who will do each step• When they will do it

Also known as data collection plan PM plan is instrument-focused The PM plan is your roadmap during the

program year. It helps you obtain reliable, useful information about the effect of your program on beneficiaries.

1414

Performance Measurement Plans Performance Measurement Plans (continued)(continued)

1515

Performance Measurement Plans Performance Measurement Plans (continued)(continued)

1616

APR Do’s and Don’ts:Do include strong descriptions

of service activities

1717

APR Do’s and Don’ts:Do clearly identify instruments

1818

APR Do’s and Don’ts:APR Do’s and Don’ts:DoDo talk about talk about changechange in outcome in outcome

resultsresults

1919

APR Do’s and Don’ts:Don’t get carried away with

output codes

2020

Why Do Progress Reports?Why Do Progress Reports?

Enhances program accountabilityEnhances program accountability Opportunity for reflection and Opportunity for reflection and

strategic thinking—see the forest strategic thinking—see the forest for the treesfor the trees

Helps you become a “great Helps you become a “great communicator”—the more youcommunicator”—the more youdo it, the better you get at it.do it, the better you get at it.

Program staff turnover: Progress Program staff turnover: Progress report is a report is a permanent permanent document.document.

Helpful in identifying TA needs.Helpful in identifying TA needs.

2121

Calculating Targets: Calculating Targets: Who Should I Report Who Should I Report On?On?

Entire “needy

population”

Your beneficiarie

s

Beneficiaries you

measured

“Successful” beneficiaries

2222

Example of Target CalculationExample of Target Calculation

““Successful” Successful” Beneficiaries You Beneficiaries You

MeasuredMeasured

Beneficiaries You Beneficiaries You MeasuredMeasured

287287

428428

TargetTarget: : Sixty percent of neighborhood Sixty percent of neighborhood residents will report an increase in recycling.residents will report an increase in recycling.

67%67%

2323

Completing theCompleting theAPR Progress ReportAPR Progress Report

2424

Well-written Progress Well-written Progress Reports Answer These Reports Answer These

Questions:Questions:

How much work is complete?How much work is complete? What work is currently in What work is currently in

progress?progress? What work hasn’t started yet?What work hasn’t started yet? What challenges have arisen?What challenges have arisen? What steps are being taken to What steps are being taken to

address these challenges?address these challenges?

These same questions apply to reporting on performance

measurement!

2525

Describing Your Work Describing Your Work Before Results Are Before Results Are

AvailableAvailable

“...We don’t have any results yet.”

2626

Describing Your Work Describing Your Work Before Results Are Before Results Are

AvailableAvailable Report with respect to:Report with respect to:

– Developing logic models and Developing logic models and indicatorsindicators

– Refining data collection plansRefining data collection plans– Data access/confidentiality (human Data access/confidentiality (human

subjects)subjects)– Developing and pilot testing Developing and pilot testing

instruments instruments – Training data collectorsTraining data collectors

2727

Describing Your Describing Your ResultsResults

Tell the reader:Tell the reader:– How and when data were collectedHow and when data were collected– How data were analyzedHow data were analyzed

Account for any data that were Account for any data that were inaccessible or unusable.inaccessible or unusable.

Include numbers, not just Include numbers, not just percentages.percentages.

Don’t delay analyzing and reporting Don’t delay analyzing and reporting baseline data!baseline data!

2828

Describing Challenges & Describing Challenges & SolutionsSolutions

Worst thing to report: “No challenges”Worst thing to report: “No challenges”

Explain how challenges affect program Explain how challenges affect program operation.operation.

Report steps taken to address challenges.Report steps taken to address challenges.

Always have a backup plan (in case your Always have a backup plan (in case your proposed solution doesn’t work)proposed solution doesn’t work)

Bring past challenges to closureBring past challenges to closure

2929

ExerciseExercise

Review Your Recent Progress Review Your Recent Progress Report and Note What You Want Report and Note What You Want to Remember to Upgrade Next to Remember to Upgrade Next

Time You Do a ReportTime You Do a Report

3030

Additional Additional Resources::

Contact Project STARContact Project STAR• http://nationalserviceresources.org//siteshttp://nationalserviceresources.org//sites

/star//star/

• 1-800-548-36561-800-548-3656• star@aiweb.comstar@aiweb.com