evaluating ongoing programs: a chronological perspective to include performance measurement...
TRANSCRIPT
![Page 1: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/1.jpg)
Evaluating Ongoing Programs:
A Chronological Perspective to Include Performance Measurement
Summarized from Berk & Rossi’s Thinking About Program Evaluation, Sage, 1990; Martin & Kettner’s Measuring the Performance of Human Service Programs, Sage, 1996
![Page 2: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/2.jpg)
Stages of Assessment
• Stage 1: Determining Whether the Program is Reaching the Appropriate Beneficiaries.
• Stage 2: Making Sure the Program is Being Properly Delivered
• Stage 3: Ensuring Funds Are Being Used Appropriately
• Stage 4: Ensuring Effectiveness can be Estimated• Stage 5: Determining Whether the Program
Works• Stage 6: Determining Program Worth
![Page 3: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/3.jpg)
Stage One: Program Impact
Program Impact Research is designed to identify who is actually served by a program to determine the number being served that meet program service criteria and those that are being served that do not meet service criteria.
![Page 4: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/4.jpg)
Stage Two: Program Integrity
Program Integrity Research analyzes the essentials of program delivery such as:
• personnel qualifications & skill assessment
• consistency of program services with program mission
• targeting & marketing of services
• service coordination
![Page 5: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/5.jpg)
Stage Three: Fiscal Accountability
Accountant Perspective:
• Out of Pocket Expenses
• Historical Costs
• Depreciation
• Current & Anticipated Revenues
• Product Inventory
• Income & Outgo of Funds
![Page 6: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/6.jpg)
Stage Three: Fiscal Accountability
Economist Perspective & Opportunity Costs
• Opportunity Costs May be Considered as What was Given Up to Direct Resources in a Particular Direction
• Opportunity Costs May Also Be Construed as the “Next Best Use” of Resources
![Page 7: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/7.jpg)
Stage Four: Evaluability
Criteria for Evaluability
• Clarifying Goals
• Specifying Program Goals
• Determining Possible Outcomes in the Absence of the Program
![Page 8: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/8.jpg)
Stage Five: Program Effectiveness
• Comparisons Across Subjects
• Comparison Across Settings
• Comparison Across Time
• Comparison Across Criterion
• Pooled Comparisons
![Page 9: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/9.jpg)
Research Designs for Estimating Effectiveness
• Random Assignment: Comparing Mean Outcomes of Control & Experimental S’s
• Interrupted Time Series: (Pre- and Post-Assessment Model)
• Cross Sectional Designs: Comparisons of Different Types of Units (e.g. comparing smaller & larger cities) with Comparisons Occurring at Only One Point In Time
![Page 10: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/10.jpg)
Research Designs for Estimating Effectiveness
• Regression Time Series: Assignment of S’s by variables (Criterion Based). Objectives:– To provide estimates of values of the dependent
variable (outcome variable) from values of the independent variable (assignment Variable)
– To obtain measures of the error involved in using the regression line as a basis of estimation (I.E. Standard Error of Estimate)
– To obtain a measure of the degree of association or correlation between the two variable
![Page 11: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/11.jpg)
Research Designs for Estimating Effectiveness
• Pooled Cross Sectional & Time Series: – Randomized Experiments &– Regression Designs
May Be Compared
– Across Units (Cross Sectional) &– Across Time (Time Series)
![Page 12: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/12.jpg)
Stage Six: Cost Effectiveness
Ongoing Versus New Programs:
• Ongoing Programs Have Historical Data to work With.
• New programs lack such historical data from which to determine cost effectiveness
![Page 13: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/13.jpg)
Performance Measurement
![Page 14: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/14.jpg)
Defining Performance Measurement
The regular collection of and reporting of information about the efficiency, quality, and effectiveness of human service programs. (Urban Institute, 1980)
![Page 15: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/15.jpg)
Perspectives of Performance Measurement
• Efficiency Perspective
• Quality Perspective
• Effectiveness Perspective
![Page 16: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/16.jpg)
Systems Model Essentials
• Inputs: Includes anything used by a system to achieve its purpose
• Process: Involves the treatment or delivery process in which inputs are consumed to produce outputs
• Outputs: That which is produced
• Feedback: System information reintroduced into the process to improve quality, efficiency & effectiveness
![Page 17: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/17.jpg)
Efficiency Perspective
• Productivity = ratio of outputs to inputs
• Efficiency = maximizing outputs to inputs– Efficiency can not reflect whether program
goals are being met – Inefficiency is how many programs are
regarded by the public - often in the absence of a full understanding of the goals, mission, clientele, resources, and services of the agency
![Page 18: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/18.jpg)
Quality Perspective
• Typically involves benchmarking against standards and criteria of excellence (as in TQM, or Total Quality Management)
• TQM now defines productivity as the ratio of outputs that meet a specified quality standard
![Page 19: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/19.jpg)
Effectiveness Perspective
• Focuses on outcomes such as the results, impacts and accomplishments of programs
• Effectiveness is the highest form or performance accountability
• Focuses upon which intervention works in which settings
• Effectiveness accountability is primarily concerned with ratios of outcomes to inputs.
![Page 20: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/20.jpg)
Reasons for Adopting Performance Measurement
• Performance measurement has the potential to improve the management of human service programs
• Performance measurement has the potential to affect the allocation of resources to human service programs
• Performance measurement may be a forced choice for many, if not, most human service programs
![Page 21: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/21.jpg)
Key Questions in Performance Measurement
• Who are the clients?
• What are their demographic characteristics?
• What are their social or presenting problems?
• What services are they receiving?
• In what amounts?
• What is the level of service quality?
• What results are being achieved?
• At what costs?
![Page 22: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/22.jpg)
Performance Measurement as a Management Tool
• Performance Measurement promotes client centered approaches to service delivery
• Provides a shared language for comparing human service programs for quality, efficiency, & effectiveness
• Allows administrators to continuously monitor programs to identify areas for improvement
• Provides direct feedback to personnel, allowing them to improve their service provision
![Page 23: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/23.jpg)
Performance Measurement Programs
![Page 24: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/24.jpg)
Government Performance & Results Act (1993)
• Effective 1998, all federal agencies must begin reporting effectiveness data for their services & products
• This requirement will be passed on to agency contractors & subcontractors
• Increasingly Federal block - grant programs also have this requirement
![Page 25: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/25.jpg)
National Performance Review
• Refers to governmental efforts at instituting program effectiveness, efficiency, and quality, to implement the 1992 report on government practices entitled Reinventing Government (Osborn & Gaebler, 1992)
![Page 26: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/26.jpg)
Total Quality Management Movement
• National Movement to Improve Quality
• Focuses upon:– consumer satisfaction– outputs as measured against a quality standard
![Page 27: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/27.jpg)
Managed Care
• Emanates from health care
• Promotes efficiency to assist health care industry shift from cost-based to capitated reimbursement
![Page 28: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/28.jpg)
(SEA) Service Efforts and Accomplishments Reporting
• Standard introduced by the Governmental Accounting Standards Board (GASB)
• SEA is GASB’s term for performance measurement
![Page 29: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/29.jpg)
SEA Reporting Model
• Built upon an expanded system model including:
• inputs
• outputs
• quality outputs, &
• outcomes BUT
• Excludes Process
![Page 30: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/30.jpg)
SEA’s Lack Of Emphasis Upon Process
Absence of the Process component reflects SEA’s primary emphasis upon performance & performance cost considerations
![Page 31: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/31.jpg)
SEA Reporting Elements
• Service Efforts
• Service Accomplishments
• Measures or Ratios Relating Service Efforts to Service Accomplishments
![Page 32: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/32.jpg)
Service Efforts
Service Efforts are inputs utilized in a human service program, which are measured by the GASB in terms of
• Total Program Costs
• Total Full Time Equivalent Staff (FTE)
• Total Number of Employee Hours
![Page 33: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/33.jpg)
Service Accomplishments
Outputs:Total Volume of Total Service Provided
Proportion of Total Service Volume Meeting Quality Standard
Outcomes:
Measures of results, accomplishments, impacts
![Page 34: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/34.jpg)
Service Accomplishment Ratios
• Efficiency (output measures) cost per unit of service– cost per FTE– cost per service completion– service completions per FTE
• Effectiveness (outcome measures)– cost per outcome– outcome per FTE
![Page 35: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/35.jpg)
Output Performance Measures
• Intermediate: – episode or contact unit of service– material unit of service
• Final: Service completions
![Page 36: Evaluating Ongoing Programs: A Chronological Perspective to Include Performance Measurement Summarized from Berk & Rossi’s Thinking About Program Evaluation,](https://reader036.vdocument.in/reader036/viewer/2022081603/56649f295503460f94c42c98/html5/thumbnails/36.jpg)
Outcome Performance Measures
• Intermediate:– numeric counts,– standardized measures– level of functioning scales– client satisfaction
• Ultimate:– numeric counts– standardized measures– level of functioning scales