ppa 502 – program evaluation

PPA 502 – Program Evaluation

Lecture 9 – Making a Splash: Reporting Evaluation Results Effectively

Six Overall PrinciplesRemember the burden for effectively reporting our results is on us, the evaluators, not on our audiences.

Six Overall PrinciplesBe aggressive. Instead of waiting for audiences to request information, actively look for chances to report results. Report regularly and frequently, appear in person if at all possible, and target multiple reports and briefings to specific audiences and/or issues.Simplify, simplify! Our audiences are usually busy and their interest is pulled in different directions, so pare ruthlessly to determine the key points. If the core message creates interest, quickly follow up with more details.

Six Overall PrinciplesStudy the audience. Learn about their backgrounds, interests, concerns, plans, pet peeves, and so on. Even something as simple as including examples from home states or regions of the audience members can make reporting much more effective.

Six Overall PrinciplesFocus on actions. Audiences are rarely interested in general information; They usually want guidance that will help them decide what to do next.Report in many different ways. Rather than using only one reporting technique or other, produce several written products, give personal briefings, develop a screenshow presentation, produce a videotape, and so on – all with powerful graphics and helpful recommendations.

Reporting Via Final ReportsTraditional versus action-oriented evaluation reports.

Structure. One report, comprehensive, long, inductive, sometimes

executive summary, table of contents appendices. More than one report, focused, short, deductive, always

executive summary, sometimes table of contents, no appendices.

Reporting Via Final ReportsTraditional versus action-oriented evaluation reports.

Style. Detached, third person, passive voice, dry language, past

tense, medium to long sentence length. Involved, first or second person, active voice, rich

language, present tense, short sentence length.

Reporting Via Final ReportsTraditional versus action-oriented evaluation reports.

Content. Detailed background, formal hypotheses, detailed

methodology, full analyses, all findings, some graphics and figures.

Brief background, no hypotheses, brief methodology, overall approach, selected findings, many graphics and figures.

Reporting Via Final ReportsTraditional versus action-oriented evaluation reports.

Perspective. Neutral point of view, author as outsider. Audience’s point of view, author as insider.

Resolution. Medium to slow length of study, conclusions, sometimes

interpretations, sometimes recommendations, no follow-up, publicly disseminated.

Fast length of study, conclusions, interpretations, recommendations, suggested follow-up, initially privately disseminated.

Reporting Via Other Written ProductsDraft reports.

Especially if subject to change.

Other documents.Internal memoranda, interim progress reports, talking papers, question-and-answer statements, written responses to other documents, etc.

Index cards.

Reporting Via Personal Briefing.Step 1 – prepare the briefing materials.

Select information to include in the briefing.Prepare six to ten briefing charts.Prepare a handout for each member of the audience.

Step 2 – set the stage for the briefing.Determine the presenter, assistant, and liaison.Invite a small, select audience.Practice, practice, practice.Establish and distribute a written agenda.Make all miscellaneous arrangements.

Reporting Via Personal Briefing.Step 3 – deliver the briefing.

Explain the purpose of the briefing.Grab attention immediately.Encourage interaction at any time.Provide a balanced briefing.Ask for a response from those being evaluated.Facilitate a lively discussion.

Reporting Via Action Recommendations.

Suggestion 1: allocate sufficient time and resources to developing recommendations.Suggestion 2: consider all aspects of the issue fair game.Suggestion 3: draw possible recommendations from a wide variety of sources.Suggestion 4: work closely with agency personnel throughout the process.

Reporting Via Action Recommendations.

Suggestion 5: consider the larger context within which the recommendations must fit.Suggestion 6: generally offer only realistic recommendations.Suggestion 7: decide carefully whether to be general or specific.Suggestion 8: think twice before recommending fundamental change.

Reporting Via Action Recommendations.

Suggestion 9: show the future implications of recommendations.Suggestion 10: make recommendations easy to understand.Suggestion 11: stay involved after recommendations have been accepted.Suggestion 12: if a recommendation is not accepted, look for other opportunities to recommend it again.

Reporting Via GraphicsWrite a message sentence.

Reporting Via GraphicsDecide what type of comparison this message implies.

Different parts of a whole.Different units.Different points in time.Two different variables.

Reporting Via GraphicsExperiment with different graphics to show this comparison.

Different parts of the whole – pie charts, 100 percent column charts, sliding bar charts.Different units – horizontal bar charts, histograms, maps, small multiples.Different points in time – line charts, mountain charts, vertical bar charts, pictographs, or historical timelines.Two different variables – scatterplots, paired bar charts.

Reporting Via GraphicsSelect the graphic that conveys the message best.

Accuracy.Simplicity.Clarity.Attractiveness.

Actually construct the graphic.Pilot-test the graphic and revise if necessary.Insert the final graphic into the report or briefing.

Reporting Via Other Techniques.Videotapes.

Local conditions.Briefing.

Computerized presentations.

Conclusions.Adopt the proper attitude.Celebrate variety.Tailor our reporting very carefully.Stay focused on the bottom line.