Environmental Results and Grants

National Tribal Forum on Air QualityLas Vegas, Nevada

June 4, 2008

Today’s Presentation

Understand resource and performance measurement terminology.

Understand relationship of performance measures to workplan requirements.

Identify challenges in linking environmental results to resources.

Encourage dialogue to improve state-of-the-art for grant measures.

Terms: Indicators, Measures and Results

An indicator is often expressed as a snapshot of an environmental condition expressing an overall trend (e.g., CO2 emissions have increased by 1.5% over the previous year).

A measure typically expresses progress or change (i.e., results) towards a specific goal or a target (e.g., 20% reduction in particulate emissions towards 2012 target level).

Grant results are usually related to a specific program or activity, are measured relative to the grant’s period of performance, and can be either output or outcome-oriented. Measures, rather than indicators, tend to be associated with grant agreements.

Types of Results (and Measures)

Output: an activity, product or deliverable related to an environmental goal or objective that will be produced or provided by a specific date during the grant funding period. Can be qualitative or quantitative. In short – a product.

Outcome: result, effect or consequence that occurs from carrying out an environmental or programmatic activity that is related to an environmental goal or objective. Must be quantitative but may not occur within grant period. In short – an impact.

Grants can support a wide range purposes within an overall environmental strategy or program (e.g., research, outreach, mitigation plan) and the type of results to be achieved (and their relevant measure) will vary according to purpose.

Examples of Measures by Type of Activity

SELECTED PHASE or ELEMENT

PURPOSE EXAMPLE MEASURE and TYPE

Research Determine Health Effects of Fine Particles on Sensitive Population

- Change in Aerobic Capacity in Test and Control Groups (Outcome)

Air Quality Assessment Conduct Baseline Air Quality Monitoring for Fine Particulates

- Number of Quality-Assured Air Quality Data Reports Submitted on Time (Output)

- Air Quality Design Value (Outcome)

Capacity-Building Equip Agency with Expertise, Knowledge, Equipment and Resources to Maintain a Continuing Program

- Number of Employees Trained (Outputs)- Number of Tribes with Approved Eligibility

Determinations (Output/Outcome)- Tribes with Approved TIPs to Address AQ Conditions on

Tribal Lands (Outcome)

Mitigation Develop and Implement Plan to Control Sources and Meet NAAQS

- Number of TIP Elements Submitted on Time (Output)- Number of Tribal Reservations Formerly in Non-

Attainment now Monitoring Clean Air (Outcome)

Prevention / Maintenance

Maintain Clean Air Status - Annual Number of Inspections (Output)- Tons of Emissions Prevented through Compliance

(Outcome)

GPRA: Funds Should Produce Intended Results

CAA grants are intended for our common air quality goals: achieve and maintain clean, healthy air and mitigate climate change.

Grants support strategies needed to reach the goal.

Progress on various aspects of the strategy is expressed via measures or indicators specific to those aspects and helps convey the strategy’s overall performance (logic model concept).

Performance influences the targeting and use of funds and vice versa.

EPA Provisions for Linking Grants, Performance Measures and Results

Part 31 and Part 35 Grant Regulations Annual Commitments EPA Environmental Results Order PART and State Grant Template Measures Other Information

Applicable EPA Grant Regulations

Part 31.11 and 31.40 – General requirements for recipient workplans; monitoring and reporting of grantee performance.

Part 35 Preamble – Discusses and encourages States’ and Tribes’ use of GPRA Goal/Objective format in categorical and PPG workplans.

Part 35, Subpart B – Pertains to Tribes’ use of workplan commitments, components, reporting schedule and performance evaluation process; consistency with NPM Guidance.

Annual Commitments

Contained in the annual National Program Manager Guidance. Reflect the most important priorities, activities, milestones and

measures for the year. Used as a basis for the negotiation of performance between

EPA HQ and the Regions. Used as a basis for the negotiation of continuing program grants

with recipients. Results by region are recorded in the electronic Annual

Commitments System. Includes PART and Grant Template Measures.

EPA Environmental Results Order

Bolsters Existing Requirements in Response to External Criticism OIG Audits/GAO Reviews OMB PART Reviews Congress

Established EPA Policy for Addressing Environmental Results Under Assistance Agreements (2006 - on). Ensure that all grants are results-oriented including

categorical, project, and competitive grants. Align grants with strategic plan/GPRA architecture. Establish reporting of results as part grant reporting and

evaluation process.

Results Order: Impact on Recipients

Applies to all forms of grants (i.e., categorical, project, competitive, earmarks, etc.) throughout entire grant process.

EPA must be able to relate recipient’s workplan to EPA’s strategic architecture.

WPs must have well-defined outputs and, to the extent practical, well-defined outcomes.

Must identify how and when performance is going to be gauged and reported.

EPA and recipient should jointly develop a process for evaluation of progress.

PART Findings

In 2007 NAAQS PART Review, OMB couldn’t easily associate recipient categorical or PPG grant workplans with EPA mission.

Recipients’ results were not transparent for NAAQS or Radon.

OMB directed EPA to create a template for use by States in their categorical/PPG WP development that enables: (a) clear WP linkages to Agency’s strategic architecture; and (b) consistent and regular performance reporting.

Template should: (a) allow for meaningful comparisons between various states’ past and planned activities and performance; and (b) include language and mechanisms that clearly hold states accountable for meeting performance goals.

Grant Template Measures

Applies to categorical and performance partnership grants. EPA stresses use of existing performance measures in template

to reduce recipient burden. Template measures are a subset of ACS information. For OAR, uses mostly environmental results measures

identified by OMB during the NAAQS and Radon PART reviews (including grants) for template.

Most OAR measures calculated at national level without recipient-specific targets (e.g., change in population-weighted ambient O3 concentration).

Handful of key activity measures also included (e.g., submission of monitoring data).

Issues with Current Template Measures

Measures reflect accommodation with OMB and include influences beyond grantee’s contribution.

Due to timing and data lag, results still pending. ‘National measures’ cause confusion about what recipients report.

Applies only to categorical and PPG grant programs and not project grants (e.g., DERA, Tribal projects).

Doesn’t adequately reflect other important work and progress captured in ACS related to grants.

Many question whether this is best approach to truly reflect recipient performance.

Additional Thoughts

Agency looking at grant workplan standardization: addresses consistency in format but not substance of measures.

Project grants, where results can be measured within, or immediately after, grant period (e.g., DERA and diesel emissions) can show more direct cause-effect relationship.

Inclusion of indicators (e.g., cumulative progress or ad hoc data like AirNOW, AQI) as an indirect measure of environmental progress is possible.

If a meaningful relationship between grants and short-term environmental performance is not possible for categorical programs will EPA and recipients have to go back focusing only on key activity measures?

More Information

Strategic Plan and Program Guidance Government Performance and Results Act (GPRA) -

http://www.epa.gov/budget/planning/gpra.htm EPA Strategic Plan - http://www.epa.gov/ocfopage/plan/plan.htm National Program and Grant Guidance - http://www.epa.gov/budget/npmguidance Regional Plans and Guidance - http://www.epa.gov/budget/regionplans EPA Annual Plan (and Budget) - http://www.epa.gov/budget/budget/

Grants Information Environmental Results Order, Appendix A, Appendix B

http://www.epa.gov/ogd/grants/award/5700.7.pdf CFDA Site (Grant Program Descriptions) http://12.46.245.173/cfda/cfda.html Grant Template: http://www.epa.gov/ocfo/npmguidance/index.htm

Logic Modeling http://www.uwex.edu/ces/lmcourse; http://evaluationwebsite.org/bja/guide/pe4.shtml;

http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

ContactBill Houck@ houck.william@epa.gov; Phone – 202-564-1349