inspection, evaluation, and performance auditing presented by george f. grob center for public...
Post on 16-Dec-2015
212 Views
Preview:
TRANSCRIPT
Inspection, Evaluation, and Performance Auditing
Presented By
George F. GrobCenter for Public Program Evaluation
To the Association of Inspectors GeneralNew Orleans
November 2013
Center for Public Program Evaluation
Essentials of Evaluations and Inspections
• What are they?• Comparison to Audits• Other Related Professional
Disciplines• Evaluation Types and Approaches• Analytic Frameworks• Possible Organizational Locations• Managing an Evaluation Office• Quality Standards
2
Center for Public Program Evaluation
Center for Public Program Evaluation
3
What Is an Evaluation
Center for Public Program Evaluation
Evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.
4
Evaluation—What Is That?
American Evaluation Association
Center for Public Program Evaluation
Examples of Government Sponsored Evaluations
• Early implementation of emergency response systems after 9-11
• Effects of programmatic earmarks for international HIV/AIDS relief programs
• Case study of Freddie Mac’s loss of $1.2 billion from Lehman Brothers bankruptcy
• Frequency of and criteria for home visits to foster care families
• Performance standards for homeless youth centers
5
Center for Public Program Evaluation
More Examples• Unlicensed fire sprinkler contractors performing
inspections in Chicago
• Potential use of a fiscal intermediary by the Chicago department of public health
• Assessment of New Orleans' System of City Courts and Performance Review of New Orleans Traffic Court
• Evaluation of the City of New Orleans Delinquent Property Tax Collection Program
• Promising approaches for State and local government enforcement of prohibition on sale of cigarettes to youth
• Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs
6
Center for Public Program Evaluation
Websites of OIG Evaluation ReportsFederal Agencies
City Government
• Department of Health and Human ServicesOIG Office of Evaluation and Inspections
• Department of CommerceOIG Audits and Evaluations
• Department of StateOIG Audits and Program EvaluationsOIG Inspections
• Department of Veterans Affairs
OIG Office of Audits and Evaluations Oversight Reports
• Department of Housing and Urban DevelopmentOIG Evaluations and Inspections
• Federal Housing Finance AgencyOIG Audits and Evaluations
• City of ChicagoAudits, Reviews, Advisories
• City of New OrleansAudits, Inspections, and Evaluations
7
U.S. Congress• Government Accountability Office (GAO)
Reports and Testimonies
Audits, Inspections, and Evaluations
Center for Public Program Evaluation
Center for Public Program Evaluation
8
Evaluation Standards
Center for Public Program Evaluation
• Association of Inspectors General
• Council of the Inspectors General on Integrity and Efficiency
• Government Accountability Office
• Institute for Internal Auditors
• American Evaluation Association
9
Government and Professional Standards for Evaluations and Inspections
Center for Public Program Evaluation
10
Association of Inspectors GeneralQUALITY STANDARDS FOR INSPECTIONS, EVALUATIONS, AND
REVIEWS
• General Standards Staff Qualifications Independence Due Professional Care
Qualitative Standards Quality Control Planning Data Collection and Analysis Evidence Timeliness Fraud and Other Illegal Acts Reporting Confidentiality Follow-Up
http://inspectorsgeneral.org/files/2012/06/IGStandards_revised_july2012.pdf
Center for Public Program Evaluation
CIGIE Standards for Inspection and Evaluation
11
• Timeliness• Fraud, Illegal Acts, Abuse• Reporting• Follow-Up• Performance Measurement• Working Relationships and
Communications
• Competency• Independence• Quality control• Planning• Data Collection and
Analysis• Evidence• Records maintenance
http://www.ignet.gov/pande/standards1.html
Center for Public Program Evaluation
GAO Standards for Performance Audits
• Chapter 3: General Standards▫ Independence▫ Professional Judgment▫ Competence▫ Quality Control
• Chapter 7: Field Work▫ Planning▫ Supervision▫ Evidence▫ Documentation
• Chapter 8: Reporting▫ Form▫ Contents▫ Quality Elements▫ Report Issuance
and Distribution
12
http://www.gao.gov/govaud/yb2003.pdf
Center for Public Program Evaluation
• Systematic Inquiry• Competence• Integrity/Honesty• Respect for People• Responsibilities for General & Public
Welfare
13
AEA Guiding Principles for Evaluators
http://www.eval.org/p/cm/ld/fid=51
Center for Public Program Evaluation
•Standards for the Professional Practice of Internal Auditing
14
Institute of Internal Auditors
https://na.theiia.org/standards-guidance/mandatory-guidance/Pages/Standards.aspx
Discusses evaluation of risk and controls, and requires audit results to be based on analysis and evaluation.
No “Evaluation” standard as such
Center for Public Program Evaluation
• OIGs may have responsibilities that go beyond audits and investigations.
• This additional work includes inspecting, evaluating, reviewing, studying, and/or analyzing government operations and programs for the purposes of providing information for decision-making, and of making recommendations to improve programs, policies, or procedures.
• The objectives of these processes include: providing a source of factual and analytical information, monitoring compliance, measuring performance, and assessing the efficiency and effectiveness of operations.
15
Inspections, Evaluations, and Reviews—AIG Definition
Source: Association of Inspectors General
Center for Public Program Evaluation
Systematic and independent assessments of the design, implementation, and/or results of an Agency’s operations, programs, or policies.
They can be used to determine efficiency, effectiveness, impact, and/or sustainability of agency operations, programs, or polices.
16
Inspections and Evaluations—Federal IG Definition
Source: Council of the Inspectors General on Integrity and Efficiency
Center for Public Program Evaluation
• They provide information that is timely, credible, and useful for agency managers, policymakers, and others.
▫ providing factual and analytical information; ▫ measuring performance; ▫ determining compliance with applicable law, regulation, and/or
policy ▫ identifying savings and funds put to better use; ▫ sharing best practices or promising approaches; ▫ assessing allegations of fraud, waste, abuse, and mismanagement.
17
Inspections and Evaluations—Federal IG Definition
Source: Council of the Inspectors General on Integrity and Efficiency
Center for Public Program Evaluation
Center for Public Program Evaluation
18
The Proving Professions
Center for Public Program Evaluation
19
Auditors and Evaluators
EvaluationAudit
Center for Public Program Evaluation
20
Auditors and Evaluators
EvaluationAudit
Center for Public Program Evaluation
21
Auditors and Evaluators
EvaluationAudit
Performance Audit
Performance Evaluation
Financial Audits
Attestation Audits
Non-Audit Services
Issue Analysis
Inspections
Reviews
A Multitude of Analysis Types
Center for Public Program Evaluation
22
Other Professional DisciplinesEvaluators/Inspectors Come From Many Professions
• Evaluation• Inspection• Audit• Social Science• Economics• Operations
Research Analysis
• Policy Analysis• Public Administration• Program Specialties
▫ Health Care▫ Housing▫ Defense▫ Trade▫ International Assistance
Center for Public Program Evaluation
23
LegalAnalysis
Inspections
Economics
Evaluation
OperationsResearch
Applied Social
Research
Psychology
Statistics
Public Administration
Center for Public Program Evaluation
24
LegalAnalysis
Inspections
Economics
Evaluation
OperationsResearch
Applied Social
Research
Psychology
Statistics
Public Administration
Center for Public Program Evaluation
What About Investigators?
25
Evaluators Investigators?
Center for Public Program Evaluation
Useful Interplay
26
Investigators
Evaluators
Fraud leads
Fraud trends
Tech
assist
Center for Public Program Evaluation
Center for Public Program Evaluation
27
Evaluation Types and Approaches
Center for Public Program Evaluation
Evaluation Types and Approaches
• There are many legitimate and useful forms of evaluations
• Each has its purpose. The methodological approach must be adapted to it
• Generally accepted professional guidelines are available to guide the work.
28
Center for Public Program Evaluation
Center for Public Program Evaluation
29
As A Program MaturesThe Evaluation Questions Change
How is the implementation going?Any Issues?
Are service providers in compliance with program rules?
Are some service providers doing better than others?How so?
Are the required services being delivered?
Are impacted citizens or businesses satisfied with program results?
How do the services stack up to goals and standards?
Have conditions improved as intended?
Center for Public Program Evaluation
Center for Public Program Evaluation
30
And So Do the Analytic Approaches and Methods Used
Some inspections and evaluations need to be done fast
Others need more time
Some require more resources than others
All must be done professionally
Following are some evaluation and inspections types commonly used today.
Center for Public Program Evaluation
Center for Public Program Evaluation
31
Inspection/Evaluation Types
• Compliance Review• Performance Review• Landscape Survey• Early Implementation
Reviews• Impact Assessment• Report Card• Process Review
Center for Public Program Evaluation
Center for Public Program Evaluation
32
More Inspection/Evaluation Types
• “Just the Facts” Review• Report from “On the
Ground”• “Try it out” Reviews• Client Satisfaction Surveys• Before and After
Assessment• Control System Survey• Program Logic Model
Center for Public Program Evaluation
Center for Public Program Evaluation
33
Analytic Frameworks
Center for Public Program Evaluation
Center for Public Program Evaluation
34
The Standard IG Analytic Model
•FINDINGS -- Criteria -- Condition -- Effect -- Cause
•RECOMMENDATIONS
Center for Public Program Evaluation
What the Federal IG Blue Book Says
Center for Public Program Evaluation
35
“Findings often have been regarded as containing the elements of criteria, condition, effect, and, when problems are found, cause. However, the elements needed for a finding depend entirely on the objectives of the inspection. Thus, a finding or set of findings is complete to the extent that the inspection objectives are satisfied and the report clearly relates those objectives to the applicable elements of a finding.”
Center for Public Program Evaluation
Two Analytic FrameworksPerformance Audit Performance Evaluation
• Findings▫ Criteria▫ Condition▫ Effect▫ Cause
• Evidence▫ Physical▫ Documentary▫ Testimonial▫ Analytic
• Recommendations
• Objective
• Questions to Be Answered
• Analytic Approach▫ Mixed Methods▫ Qualitative▫ Quantitative
• Findings
• Recommendations
36
Center for Public Program Evaluation
Center for Public Program Evaluation
37
METHODOLOGIES
Center for Public Program Evaluation
Center for Public Program Evaluation
38
The Fundamentals• The Correct Methods to Use Are
the Ones Most Compatible With the Purpose of the Study
• Always Use More Than One Method▫ At least one quantitative▫ At least one qualitative
Center for Public Program Evaluation
Center for Public Program Evaluation
39
Quantitative and Qualitative Methods• Quantitative methods involve things you can count
▫ Even if they are subjective answers to a closed ended questionnaire
• Qualitative methods are applied to the interpretation of statements▫ Especially from interviews▫ But also from observations and in documents
• Reviews are far more compelling if both methods are used▫ Otherwise people will say: “Numbers always lie” or “They always say that”
Center for Public Program Evaluation
Center for Public Program Evaluation
40
The Most Common Methods Used by Evaluators
• Background Search• On-Site Inspections• Review of Administrative
Files• Sampling• Data Collection
Instruments• Surveys• Focus Groups
Center for Public Program Evaluation
Center for Public Program Evaluation
41
Some Exercises
Center for Public Program Evaluation
You Be the Evaluator
• Early implementation of emergency response systems after 9-11
• Frequency of and criteria for home visits to foster care families
• Pharmacies with extreme outlier filling of prescriptions for dangerous, addictive drugs
42
Center for Public Program Evaluation
The Quick Start1. State the purpose
2. Ask two or three key questions 3. Choose the methods for each question
Quantitative Qualitative
4. Conduct the Field Work5. Analyze the data6. Construct Findings and Recommendations7. Draft the report
Center for Public Program Evaluation
43
Center for Public Program Evaluation
Discussion1. Would your approach work as either an
audit or evaluation?
2. Who will use the findings and recommendations?
3. How will you know that the recommendations have been acted upon?
4. What can you expect will be the result?
44
Center for Public Program Evaluation
Center for Public Program Evaluation
45
Organizing and Managing the Evaluation Function
Center for Public Program Evaluation
Possible Organizational Location
46
Inspector General Evaluation
AuditInvestigatio
n Evaluation
Evaluation Evaluation
Center for Public Program Evaluation
Managing an Evaluation Office
47
Work Planning
Operating Procedures
Quality Control
Quality Assurance
Professional Development
These are generally similar to
audit practices
Take Them Seriously !
Center for Public Program Evaluation
Training Priorities for EvaluatorsHighest Priority Other Worthwhile Skills
• General Evaluation Concepts
• Writing to the Point
• Survey Design• Interviewing• Descriptive Statistics• Sampling• Spread Sheets• Graphs
• Focus Groups• Data Base Development• Randomized Control Trials
• Team Leadership• Programmatic Knowledge
48
Center for Public Program Evaluation
Work Planning
•Combine Audit and Evaluation•Focus on risk and return on investment•Use a Tactical Approach
▫ To systematically fill gaps▫ To develop staff capacity▫ To keep up the attention on important areas
49
Center for Public Program Evaluation
Center for Public Program Evaluation
50
Program Area A
Program Area B
Program Area C
Program Area D
A strategic plan might consist of several families of program studies, some in the early stages, some further along, even in final stages
Center for Public Program Evaluation
51
LegalAnalysis
Inspections
Economics
EvaluationAndAudit
OperationsResearch
Applied Social
Research
Psychology
Statistics
Public Administration
Listen to What They Have to Say
Center for Public Program Evaluation
52
Good Luck!
Center for Public Program Evaluation
George F. Grob
Center for Public Program Evaluation540-454-2888
GeorgefGrob@CS.com
53
top related