review of evaluation, measurement and verification ... · review of evaluation, measurement and...
TRANSCRIPT
LBNL-3277E
ERNEST ORLANDO LAWRENCE BERKELEY NATIONAL LABORATORY
Review of Evaluation, Measurement and Verification Approaches Used to Estimate the Load Impacts and Effectiveness of Energy Efficiency Programs Mike Messenger*, Ranjit Bharvirkar*, Bill Golemboski*, Charles A. Goldman**, Steven R. Schiller*** *Itron, Inc., **Lawrence Berkeley National Laboratory, ***Schiller Consulting, Inc.
Environmental Energy Technologies Division April 2010
The work described in this report was funded by the Office of Electricity Delivery and Energy Reliability (OE); Permitting, Siting, and Analysis Division of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.
Disclaimer
This document was prepared as an account of work sponsored by the United States Government. While this document is believed to contain correct information, neither the United States Government nor any agency thereof, nor The Regents of the University of California, nor any of their employees, makes any warranty, express or implied, or assumes any legal responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by its trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or The Regents of the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof or The Regents of the University of California.
Ernest Orlando Lawrence Berkeley National Laboratory is an equal opportunity employer.
iii
LBNL-3277E
Review of Evaluation, Measurement and Verification Approaches Used to
Estimate the Load Impacts and Effectiveness of Energy Efficiency Programs
Prepared for the Ernest Orlando Lawrence Berkeley National Laboratory
1 Cyclotron Road, MS 90R4000 Berkeley CA 94720-8136
Principal Authors
Mike Messenger, Ranjit Bharvirkar, Bill Golemboski Itron, Inc.
Charles A. Goldman
Lawrence Berkeley National Laboratory
Steven R. Schiller Schiller Consulting, Inc.
April 2010
The work described in this report was funded by the Office of Electricity Delivery and Energy Reliability (OE); Permitting, Siting, and Analysis Division of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231.
iv
Acknowledgements
The work described in this report was funded by the Office of Electricity Delivery and Energy Reliability (OE); Permitting, Siting, and Analysis Division of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231 The authors would also like to thank Denise Mulholland of the U.S. Environmental Protection Agency, Larry Mansueti and Michael Li of U.S. Department of Energy and the members of the National Action Plan for Energy Efficiency EM&V Technical Work Group (TWG) for their useful input to the research. Members of the TWG are: Commissioner Dian Grueneich, Diane Munns, and Phil Giudice, Patrick Oshie, Marsha Smith, Larry Pakenas, Dub Taylor, Susan Stratton, Fred Gordon, Gene Rodrigues, Jeremy Newberger, Val Jensen, Alan Shedd, Julie Michals, Joe Loper, Marc Hoffman, Don Gilligan, Mark Lauby, Kevin Kelly, Janet Streff, and Matthew McCaffree. The authors would also like to thank the following individuals for providing comments and input on a review draft of this study: Lynn Anderson (Idaho PUC), Tom Ballinger (Florida PSC), Kathy Kuntz (WECC), Marjorie McCrae (Research Into Action), Larry Pakenas (NYSERDA), and Elizabeth Titus (NEEP).
v
Table of Contents
Acknowledgements ........................................................................................................................ iv
Table of Contents .............................................................................................................................v
List of Figures and Tables............................................................................................................. vii
Acronyms and Abbreviations ...................................................................................................... viii
Executive Summary ....................................................................................................................... ix
1. Introduction .................................................................................................................................1 1.1 Objectives ............................................................................................................................2 1.2 National Action Plan for Energy Efficiency (Action Plan) .................................................2 1.3 Report Organization .............................................................................................................3
2. Approach .....................................................................................................................................4
3. Uses and Users of EM&V Studies ..............................................................................................7 3.1 Key Users of Evaluation Studies .........................................................................................7 3.2 Key Uses of Evaluation Studies ...........................................................................................8
4. EM&V Planning Practices, Methods and Protocols .................................................................10 4.1 Evaluation Planning Practices: Management Roles and Responsibilities and Oversight ..10 4.2 Evaluation Planning Practices: Types of EM&V studies ..................................................13 4.3 Allocation of funding for EM&V activities .......................................................................14 4.4 Evaluation Practices: EM&V protocol and Technical Reference Manual of deemed savings........................................................................................................................................16 4.5 Evaluation Methods to estimate Savings and Load Impacts .............................................17
4.5.1 Guidance on Required Components in Program Savings Report ..............................17 4.5.2 Net and Gross Savings ...............................................................................................18 4.5.3 Estimating Load Impacts from Energy Efficiency: Dealing with uncertainty ..........21
4.6 Implementation of EM&V studies .....................................................................................23 4.7 Review and reporting of EM&V study results ..................................................................24
5. Current and Emerging Issues in EM&V Practices ....................................................................26 5.1 Emerging issues in evaluation practices and methods: Key drivers ..................................26 5.2 Summary of Key Evaluation Issues ...................................................................................28
5.2.1 Relative Importance of EM&V Issues under alternative future scenarios for energy efficiency 29
6. Recommendations .....................................................................................................................32
7. Conclusion .................................................................................................................................34
References ......................................................................................................................................35
vi
Appendix A. Interview and Discussion protocol .....................................................................36
Appendix B. Description of recommended EM&V activities ................................................47
vii
List of Figures and Tables
Figure 1. Most important audiences for EM&V studies ................................................................ 7 Figure 2. Respondent’s rating of relative importance of different types of evaluation studies ... 14 Table 1. Ratepayer-funded energy efficiency budgets and EM&V expenditures for selected
states .................................................................................................................................. 4 Table 2. Interviewees by type of respondent and state .................................................................. 6 Table 3. Relative importance of current use of EE program evaluations ...................................... 8 Table 4. How state policymakers and regulators use information obtained from evaluation
studies ................................................................................................................................ 9 Table 5. Suggestions made by policymaker/regulators to improve the usefulness of EM&V
studies .............................................................................................................................. 10 Table 6. Evaluation planning processes: Roles and responsibilities ............................................ 12 Table 7. Evaluation planning processes: Respondent views ....................................................... 12 Table 8. EM&V 2008 funding levels and allocation among activities ........................................ 15 Table 9. EM&V Protocols and Technical Reference Manual ..................................................... 17 Table 10. Market influences and program effects included in estimates of net savings from
energy efficiency programs ............................................................................................. 19 Table 11. Reporting and definition of net and gross savings in EM&V studies ......................... 20 Table 12. Respondent views regarding reporting of net and/or gross savings ............................ 20 Table 13. Comparability of EM&V load impact results among states ......................................... 21 Table 14. How is certainty defined and addressed ...................................................................... 22 Table 15. Who sets precision, accuracy, or robustness requirements for savings estimates? ..... 23 Table 16. Typical cycle time for savings estimates from energy efficiency programs ............... 24 Table 17. Availability and review process for load impact studies ............................................. 25 Table 18. Differences in evaluation practices and methods among states: Potential implications
if move towards regional or national protocols ............................................................... 27 Table 19. Alternative future scenarios for energy efficiency ...................................................... 30 Table 20. Relative importance of addressing emerging EM&V issues* ...................................... 30
viii
Acronyms and Abbreviations
CEE Consortium for Energy Efficiency EE Energy efficiency (end-use, non-transportation) EERS Energy Efficiency Resource Standard EIA U.S. Energy Information Administration EM&V Evaluation, measurement, and verification NAPEE National Action Plan for Energy Efficiency M&V Measurement and verification NEEA Northwest Energy Efficiency Alliance TWG Action Plan EM&V Technical Work Group TRM Technical Reference Manual ISO Independent System Operator
ix
Executive Summary
Public and private funding for end-use energy efficiency actions is expected to increase significantly in the United States over the next decade. Increased funding should result in more benefits as well as more scrutiny of these results. Therefore, evaluating the effectiveness and energy saving impacts of energy efficiency programs is likely to become increasingly important for policymakers and private and public funders of efficiency actions. This report focuses on evaluation, measurement, and verification (EM&V) issues relating to estimating savings and impacts of energy efficiency programs and includes:
A review of the strengths and weaknesses of current EM&V methods and practices;
EM&V issues associated with improving and scaling up EM&V practices under several future federal and regional efficiency funding and implementation scenarios; and
Suggested activities and projects that address these emerging EM&V issues and support more consistent and standardized approaches to estimating the savings, impacts, and effectiveness of energy efficiency programs.
While EM&V can be a subject of interest for all types of energy efficiency investments, this report focuses on EM&V of efficiency programs (versus projects) funded with ratepayer monies.
The work underlying this report was conducted to support an EM&V Technical Work Group formed from the Leadership Group of the National Action Plan for Energy Efficiency (The Action Plan). The Action Plan is a private-public initiative established to create a sustainable, aggressive national commitment to energy efficiency through the collaborative efforts of gas and electric utilities, utility regulators, and other partner organizations. The EM&V Technical Work Group, made up of 21 EM&V experts and policy makers from across the country, is researching possible projects that the Action Plan can undertake to (1) improve the practice of determining the effectiveness and savings of end-use (non-transportation) energy efficiency activities and (2) standardize the evaluation methods and terms used to report on gross and net program savings in different jurisdictions. As part of this research project, we interviewed over 50 energy efficiency policy experts and regulatory staff, program administrators, evaluation project managers, and evaluation practitioners in 14 states and the Pacific Northwest region. These states account for ~80-85% of current spending on ratepayer-funded energy efficiency. A key objective of this activity was to catalog the range of evaluation practices and methods used in different jurisdictions and identify important emerging evaluation issues. We also had discussions with 11 national evaluation experts, and 21 members of the Action Plan EM&V Technical Work Group. We found that policymaker, technical experts and stakeholder views on EM&V issues depend to some extent on how the energy efficiency market and efforts to develop national savings goals evolve. Thus, we defined three alternative energy efficiency future scenarios as part of our work with the EM&V Technical Work Group in order to obtain their input on the relative importance of different EM&V issues under each scenario: (1) state policy (Business as Usual), (2) national policy (National Drivers), or (3) regional initiatives (Regional Drivers).
x
Key findings of this study with respect to current EM&V methods and practices are:
EM&V uses and users - Respondents identified the three most important current uses of evaluations of energy efficiency programs as: 1) quantification of level of savings (both energy and peak demand); 2) assessment of program cost-effectiveness; and 3) future program planning, design and implementation. Regulatory agencies, energy efficiency program managers, and third-party implementers were identified as the most important users of evaluation studies, followed by state energy offices.
Usefulness of EM&V studies - Policymakers and regulatory agency staff indicated that EM&V studies were useful overall (average rating of 3.8 on a 1-5 scale with 5 being extremely useful). For those that indicated that the studies were not useful, reasons given were that they were not timely (i.e. too late) or did not include discussion of factors that cause uncertainty in the savings results.
Importance by type of EM&V study - Process, load impact and benefit cost evaluations of energy efficiency programs were seen by experts to be highly important at present and in the future. Respondents also indicated that their assessment of the relative importance of load impact, process and market evaluations was unlikely to change in the future irrespective of the alternative scenario considered.
Analysis of net program load impacts - About two-thirds of respondents indicated that most or all energy efficiency programs evaluations in their jurisdictions include consideration of free-ridership. About 60% of state respondents reported that spillover/market effects caused by efficiency programs are analyzed.
Use of deemed savings - The vast majority of respondents also reported that deemed savings estimates are frequently used in evaluations of mass market energy efficiency programs and that these ex-ante savings estimates are usually trued-up based on ex-post evaluations; with regulatory decisions indicating whether they are updated every year or less often.
Allocation of EM&V budgets - The allocation of the total EM&V budget among process, load impact, and market research studies varies significantly among the jurisdictions. Impact studies accounted for 75% or more of total EM&V funding in six states and between 50-60% of total EM&V funding in 4 states and 30% in one state.
Respondent’s recommendations on useful projects to improve EM&V practices - Respondents were also asked what type of activities would help mitigate differences in the methods and terms used to describe program savings between states in the event a national energy efficiency resource standard was adopted. Sixty-four percent of the respondents supported the development of a glossary of program savings terms, while 46% supported the development of a national EM&V protocol. However, what was meant by national protocols, how detailed they would be, or whether they should be in the form of voluntary guidelines or mandatory requirements was not assessed.
xi
Key EM&V issues Based on interviews and our research, we identified six EM&V issue categories that are important to address in order to improve EM&V practices and methods. It will be important to address these EM&V issues because the landscape for energy efficiency is evolving fairly rapidly. In our view, in the future, energy efficiency savings and impacts are likely to be assessed in a market and policy environment that includes explicit linkage to greenhouse gas mitigation policies, innovative mechanisms for rewarding performance of program administrators, and in which there may be multiple administrators and/or sources of funding for energy efficiency programs. The six EM&V issue categories are:
1. Consistency in reported savings or load impacts - Differences between jurisdictions in how program savings are estimated and how they are defined (e.g., net versus gross) makes it very difficult at present to undertake meaningful comparisons of reported savings among states.
2. Measurement methods used to estimate net savings – EM&V methods are well documented and relatively standardized for determining gross (direct) energy savings for energy efficiency programs or projects. In contrast, there is much less agreement on the value and methods that should be used to estimate net savings. Key areas where differences exist on issues relating to net savings include: (1) how, if at all, to address program attribution; (2) how to define and set standards for rigor and accuracy for net savings given different policy objectives, and (3) how to assess broader “net” market effects of energy efficiency programs on future spillover savings in the market and the demand for energy services.
3. Quality control and accuracy - The quality of program evaluation and review processes varies widely among states because of differences in the level of independent review of program saving estimates required by state regulatory commissions and because relatively few states require that the level of uncertainty associated with program savings estimates be reported, driven in part by the level of technical expertise (e.g., lack of understanding on how to or the importance of reporting estimation certainty).
4. Allocation of evaluation resources -. Some states tend to focus on reporting only first year gross program savings and tend to underfund comparative analysis on the effectiveness of alternative program designs (e.g., process evaluations) and estimates of the market changes caused or attributed to energy efficiency programs.
5. Independence of program evaluators – EM&V has two primary objectives, (1) to assess the savings resulting from an efficiency program or portfolio and (2) to provide feedback for program/savings improvement. In a regulatory environment where program savings levels affect administrator compensation, there is a natural friction between the need for independence between program administrators and evaluators responsible for estimating program savings and the need for a close working relationship between these two entities in order to provide useful feedback on programs. We found that a number of states have moved away from having program administrators manage and/or oversee load
xii
impact and cost-effectiveness studies in order to reduce perceived or potential conflicts of interest or undue influence by program administrators. These solutions can create a “firewall” between program administrators and evaluation firms, which while addressing independence issues can cause delays in timely and effective feedback on program design and modification and may also limit the ability to integrate process and impact evaluations.
6. Integration of load impact results from energy efficiency programs into utility planning and forecasting – We found that states have made limited progress in addressing the analytic challenges associated with aggregating estimates of gross program savings into load forecasting frameworks. Failure to fully account for and reconcile program savings estimates with load forecasts may result in under- or over-counting of savings which may adversely affect major resource planning decisions and/or estimates of impacts on greenhouse gas emissions (GHG).
Recommendations We provide a number of specific recommendations for activities that can be undertaken to address the EM&V issues identified in this report. The relative priority or importance of these activities varies to some extent on future energy efficiency policy and market environment (e.g., Business as Usual driven by state/local policy, National Drivers, and Regional Drivers). However, many of the proposed activities appear to be beneficial irrespective of the future scenario(s) that comes to pass. The specific activities recommended are:
Develop new method(s) or enhance existing methods for estimating total program savings that includes both the near term effects of the installation of high-efficiency technologies or systems and the longer-term behavioral impacts caused by energy efficiency programs. This can include user issues, such as changes in behavior and operations and maintenance practices induced by the program, as well as broader impacts on best practices in the construction trades that install energy efficiency measures.
Develop and share best practices guides and case studies on evaluation methods and planning through webinars and regional seminar series with state, utility, regulatory staff, and industry. Possible topic areas include:
1. compendium of methods used to develop forecasts of baseline energy usage and to integrate reported savings results from energy efficiency programs into future load forecasts;
2. processes used to verify the load impact results of EE programs;
3. compendium of approaches used to structure EE evaluation planning processes;
4. review of the methods and criteria used to select ex-ante measure specific savings values (deemed or stipulated savings);
xiii
5. review of different methods used to estimate net savings from efficiency programs; and
6. review methods used to estimate gross and net GHG impacts.
Develop an accessible national or set of regional databases of standardized ex-ante savings estimates for energy efficiency measures. These efforts would reduce evaluation costs in states just starting up energy efficiency programs and help to standardize savings estimates and methods across regions.
Design and implement a national, searchable data base that provides access to energy efficiency evaluation plans and studies in various states;
Develop and disseminate best practices guides on processes and techniques used to improve quality control and accuracy in reported program savings; include recommendations on data collection, quality control, and analysis;
Develop a short program savings reporting format (i.e. one page) for all states and regions to use in reporting program savings and seek voluntary adoption by region(s) via regional workshops or other strategies; and
Develop a glossary of standardized EM&V and measure, program, or portfolio terms for voluntary use by all states and seek voluntary adoption by region(s) through regional workshops or other strategies.
1
1. Introduction
Public and private funding for end-use energy efficiency actions is expected to increase significantly in the United States over the next decade. For example, Barbose et al (2009) estimate that spending on ratepayer-funded energy efficiency programs in the U.S. could increase from $3.1 billion in 2008 to $7.5 and 12.4 billion by 2020 under their medium and high scenarios. This increase in spending could yield annual electric energy savings ranging from 0.58% - 0.93% of total U.S. retail sales in 2020, up from 0.34% of retail sales in 2008. Interest in and support for energy efficiency has broadened among national and state policymakers. Prominent examples include ~$18 billion in new funding for energy efficiency programs (e.g., State Energy Program, Weatherization, and Energy Efficiency and Conservation Block Grants) in the 2009 American Recovery and Reinvestment Act (ARRA). Increased funding for energy efficiency should result in more benefits as well as more scrutiny of these results. As energy efficiency becomes a more prominent component of the U.S. national energy strategy and policies, assessing the effectiveness and energy saving impacts of energy efficiency programs is likely to become increasingly important for policymakers and private and public funders of efficiency actions. Thus, it is critical that evaluation, measurement, and verification (EM&V) is carried out effectively and efficiently, which implies that:
Effective program evaluation, measurement, and verification (EM&V) methodologies and tools are available to key stakeholders (e.g., regulatory agencies, program administrators, consumers, and evaluation consultants); and
Capacity (people and infrastructure resources) is available to conduct EM&V activities and report results in ways that support program improvement and provide data that reliably compares achieved results against goals and similar programs in other jurisdictions (benchmarking).
The National Action Plan for Energy Efficiency (2007) presented commonly used definitions for EM&V in the context of energy efficiency programs:
Evaluation (E) - The performance of studies and activities aimed at determining the effects and effectiveness of EE programs;
Measurement and Verification (M&V) – Data collection, monitoring, and analysis associated with the calculation of gross energy and demand savings from individual measures, sites or projects. M&V can be a subset of program evaluation; and
Evaluation, Measurement, and Verification (EM&V) - This term is frequently seen in evaluation literature. EM&V is a catchall acronym for determining both the effectiveness of program designs and estimates of load impacts at the portfolio, program and project level.
2
This report is a scoping study that assesses current practices and methods in the evaluation, measurement and verification (EM&V) of ratepayer-funded energy efficiency programs, with a focus on methods and practices currently used for determining whether projected (ex-ante) energy and demand savings have been achieved (ex-post).1 M&V practices for privately-funded energy efficiency projects (e.g., ESCO projects) or programs where the primary focus is greenhouse gas reductions were not part of the scope of this study.2 1.1 Objectives
We identify and discuss key purposes and uses of current evaluations of end-use energy efficiency programs, methods used to evaluate these programs, processes used to determine those methods; and key issues that need to be addressed now and in the future, based on discussions with regulatory agencies, policymakers, program administrators, and evaluation practitioners in 14 states and national experts in the evaluation field.3 We also explore how EM&V may evolve in a future in which efficiency funding increases significantly, innovative mechanisms for rewarding program performance are adopted, the role of efficiency in greenhouse gas mitigation is more closely linked, and programs are increasingly funded from multiple sources often with multiple program administrators and intended to meet multiple purposes. 1.2 National Action Plan for Energy Efficiency (Action Plan)
The work underlying this report was conducted primarily to support an EM&V Task Force formed from the Leadership Group of the National Action Plan for Energy Efficiency (The Action Plan). The Action Plan is a private-public initiative established to create a sustainable, aggressive national commitment to energy efficiency through the collaborative efforts of gas and electric utilities, utility regulators, and other partner organizations. The Action Plan was established in 2005 as a private-public initiative for creating a sustainable and aggressive national commitment to EE. The Department of Energy and Environmental Protection Agency have acted as facilitators of the Action Plan. The Action Plan’s Leadership Group consists of representatives from electric and gas utilities, state regulators, and other EE organizations. Leadership Group members are identifying key barriers limiting greater U.S. investment in
1 With the passage of ARRA, state and local governments have about $11 billion in funding to implement large-scale energy efficiency programs and activities over the next 2-3 years under the State Energy Program, Weatherization Assistance Program, and Energy Efficiency Conservation Block Grant Program. 2 We focus on EM&V practices in the context of accountability for public funds expended primarily for efficiency’s energy resource benefits. Work on the role of EM&V in private sector transactions may be appropriate in the future if the lack of savings documentation or existing savings documentation practices are seen as a barrier to such transactions. In addition, if there is national energy legislation that includes energy efficiency resource standards (EERS), the determination of whether a state meets an EERS requirement may require new “top-down” approaches and/or consideration of savings achieved with or without public funds. If there is regional or national climate legislation, the documentation of both energy savings and avoided emissions from activities that have a goal of climate mitigation, and/or use “climate funds” such as taxes or allowance allocations, will require additional EM&V considerations. 3 Evaluation efforts sometimes include measurement and verification, although project- or measure- level M&V analysis issues are generally not covered in this report.
3
energy efficiency, and developing and documenting sound business practices for removing these barriers.4 The Action Plan’s EM&V Task force, made up of 21 EM&V experts and policy makers from across the country, is researching possible projects that the Action Plan can undertake to (1) improve the practice of determining the effectiveness and savings of end-use (non-transportation) energy efficiency activities and (2) standardize the evaluation methods and terms used to report on gross and net program savings in different jurisdictions. The EM&V Task Force is continuing its work while this report has been prepared to document the research conducted and conclusions reached to date by the report’s authors. The EM&V Task Force may initiate actions or recommend the same or other actions. 1.3 Report Organization
The report is organized as follows:
In Section 2, we describe the overall approach and methods used in this study, including the audience targeted for interviews and response rate;
In Section 3, we discuss respondent’s views on current and future uses of EM&V studies, key users, and present perspectives of various states on EM&V approaches;
In Section 4, we discuss EM&V practices with respect to planning evaluation activities, present information on EM&V budgets in various states, summarize practices and methods used to estimate and report savings from energy efficiency programs, precision/accuracy requirements, and review of EM&V studies.
In Section 5, we summarize key differences in evaluation practices and methods among the 14 states and NEEA, identify factors that may account for these differences, and identify current and emerging EM&V issues pertaining to evaluating the effectiveness of energy efficiency programs.
In Section 6, we present a set of recommended activities that can be pursued to improve the quality and consistency of EM&V practices in estimating load impacts.
4 http://www.epa.gov/cleanenergy/energy-programs/napee/index.html
4
2. Approach
In this section, we describe the methods used to collect information on evaluation practices and issues. Because of time and budget constraints, the project team focused on reviewing information from 14 selected states and a regional energy efficiency organization (Northwest Energy Efficiency Alliance or NEEA): California, Connecticut, Florida, Idaho, Illinois, Iowa, Maine, Massachusetts, Minnesota, New York, Oregon, Pennsylvania, Texas and Wisconsin. These states vary significantly in the experience with and maturity of ratepayer-funded EE programs, ranging from 1 to 20 years.5
Table 1. Ratepayer-funded energy efficiency budgets and EM&V expenditures for selected states
State
2008 Total EE Budget
(million $)6
2008 Total EE Budget per Capita ($)
2008 EM&V Budget
(million $)
2009 Total EE Budget (million $)7
2009 Total EE Budget per Capita
($)
2009 EM&V Budget
(million $)8
CA 1014.2 28 80.2 1376.7 37 79.3
CT 113.6 32 1.7 107.4 31 1.4
FL 124.3 7 ? 138.9 7 0.1
IA 58.9 20 3 90.5 30 3.2
ID 19.7 13 0.9 33.1 21 0.5
IL 41.0 3 1.3 67.4 5 2.2
MA 148.9 23 5.1 208.5 32 7.8
ME 16.8 13 0.2 20.8 16 0.2
MN 136.5 26 1 73.7 14 1.3
NEEA9 97.5 NA 1 105.2 NA 1.8
NY 287.9 15 7.7 421.2 22 7.6
OR 76.8 20 1.6 105.4 28 2.210
PA ? ? ? 8.7 1 0.1
TX 106.4 4 01 101.8 4 0.2
WI 140 25 2.4 162.4 29 4.8 Source: Consortium for Energy Efficiency (2008), Consortium for Energy Efficiency (2009) and U.S. Census Bureau.11
5 We typically refer to the 14 states and NEEA as ‘jurisdictions.’ 6 CEE (2008), sum of electricity and gas program budgets: http://www.cee1.org/ee-pe/2008/us_gas.php, http://www.cee1.org/ee-pe/2008/us_electric.php. No gas program data reported for PA and TX. 7 CEE (2009), figure 26. Data are missing from at least one electric program administrator in MA and MN. At least one organization did not allow release of gas data at the state level in IL, MN, NY and OR. 8 CEE (2009); total of Electric Program EM&V Budget (Figure 46) and Gas Program EM&V Budget (Figure 48). No reported gas program EM&V data for FL, ME, OR and TX. Only a small number of utilities reported gas program EM&V data for IL. 9 The energy efficiency program budget of Bonneville Power Authority (BPA) is also included; includes electricity programs only. 10 Estimated EM&V spending for Oregon accounts only for Energy Trust of Oregon’s (ETO) expenditures, including both external contracts (~55%) and internal resources (~45%) committed for EM&V activities (Gordon 2010). Data for utility EM&V expenditures in 2009 was not available.
5
Energy efficiency budgets for electric and gas utilities are shown for the targeted jurisdictions and accounted for approximately 70% of the total U.S. budget for ratepayer-funded energy efficiency in 2007 (see Table 1). Ten of these states have energy efficiency budgets that are greater than $10 per capita in 2009, except for Pennsylvania, Illinois, Florida, and Texas. These states and NEEA account also for about 84% of the EM&V budgets for ratepayer-funded energy efficiency programs in 2008. An interview protocol was developed for three types of respondents in each jurisdiction (see Appendix A):
State regulatory commission staff and policymakers in each of the 14 states; Administrators of ratepayer-funded energy efficiency programs (e.g., utilities, Energy
Trust of Oregon, NEEA); and Practitioners (e.g. program evaluation firms and consultants familiar with EM&V
activities of specific jurisdictions). Topics that were discussed with each of these groups included: the context, uses, and users of EM&V studies; resources devoted for EM&V studies; evaluation planning process and selection of EM&V methods; methods used to estimate load impacts from energy efficiency programs; and emerging policy issues (see Appendix A; section A.1). We also collected additional information from state regulators and policymakers on how they used evaluation studies and their ideas on how to improve the usefulness of evaluation studies in their state (see Appendix A, section A.2). We also contacted and had discussions with national-level EM&V and/or energy efficiency policy experts to obtain their perspective on evaluation planning processes, methods used to estimate load impacts, emerging policy issues, and the effectiveness of EM&V studies to policymakers (see Appendix A, section A.3). The interview topics and questions were sent via email to 90 people in the first three cohorts (state regulators/ policy makers, program administrators and local evaluation practitioners) and 17 in the fourth cohort (national EM&V practitioners). Contacts in each state were recommended by members of the TWG or the project team. Some respondents chose to complete the questionnaire electronically, while others discussed their responses during phone conversations with project team members. The actual response rate was 57% for policymakers and regulators, 77% for program administrators, and 43% for local evaluation practitioners (see Table 2). We contacted 17 national experts and 11 responded, seven of whom were evaluation consultants and four of whom were regulators or policymakers. Overall, we conducted 64 interviews with state or national experts out of 107 persons contacted.
11 Table 1 includes electric and gas programs; does not include load management.
6
Table 2. Interviewees by type of respondent and state
Respondents Respondents Respondents Respondents
State Regulatory
commission and Policymakers
Energy Efficiency Program
Administrators
Local Evaluation
Practitioners
Total Responses
Total Target
population CA 2 3 1 6 6
CT 1 0 1 2 6
FL 1 1 1 3 6
IA 2 2 0 4 6
ID 1 2 0 3 6
IL 1 1 1 3 6
MA 1 1 1 3 6
ME 1 1 0 2 6
MN 1 2 0 3 6
NEEA 1 4 2 7 6
NY 1 1 0 2 6
OR 1 2 3 6 6
PA 1 0 1 2 6
TX 1 2 1 4 6
WI 1 1 1 3 6
TOTAL 17 23 13 53 90
Some respondents provided answers only to questions that drew upon their individual expertise and experience in the field. Phone discussions were likely to lead to more comprehensive responses compared to those respondents who provided written responses to the informational questions via email. The results of these discussions with state and national experts were presented to the Action Plan TWG in a web-based seminar in September 2009; identification of emerging EM&V issues was a key outcome of this meeting.
7
3. Uses and Users of EM&V Studies
In this section, we present respondent’s views and ranking of key uses and users of evaluation studies of energy efficiency programs.12 3.1 Key Users of Evaluation Studies
Respondents indicated the two most important users of and audiences for EM&V studies include regulators and utility program managers with scores greater than 4 (see Figure 1). Third-party service providers and implementers and state energy offices are also an important audience for EM&V studies with scores greater than three. Consumer groups, environmental agencies, ISOs, and Energy Services Companies (ESCOs) received lower rankings between two and three (somewhat important).
Figure 1. Most important audiences for EM&V studies
12 State-level respondents were asked to rate the importance of key uses and users on a scale of 1 to 5 with 1 representing “not important” and 5 representing “most important” (see Questions 2-3 in Appendix A).
8
3.2 Key Uses of Evaluation Studies
About 80% of the 53 state experts responded to questions about the use of EM&V studies; see Table 3 for their rankings of relative importance of uses of EE program evaluations. Respondents indicated that quantifying the level of savings (both energy and peak demand), assessing cost-effectiveness of programs, and planning, designing, and implementing programs were the highest ranking uses of EM&V studies (4.2 or higher on a scale of 5). Other uses such as goal setting, identifying target market segments, cost recovery, and Integrated Resource Planning (IRP) were ranked in the middle in importance. Meeting environmental goals, obtaining participation payment from Independent System Operators (ISOs) in wholesale markets, and assessing environmental permit allocation were ranked lower in importance. There are also a few examples where the ratings of current uses of evaluation studies were quite different among policymakers/regulators and program administrators (e.g., rankings of relative importance differed by more than 1.2 on a scale of 5 for environmental permit allocation and performance incentives).13
Table 3. Relative importance of current use of EE program evaluations
Current Uses of EE Program Evaluations Average Rating (N = 43)
All Policymakers/
Regulators Program
Administrators Evaluators
Quantify level of program energy and peak savings
4.7 4.9 3.8 4.5
Assessing cost-effectiveness 4.4 4.5 4.6 4.0 Program planning, budgeting, design, and refinement
4.2 4.2 4.4 4.3
Performance incentives, penalties 3.8 4.1 2.9 4.0 Goal setting (Electricity or Natural Gas Savings goals, potential studies)
3.6 3.6 3.4 3.4
Identifying or Targeting receptive market segments for EE programs
3.5 3.5 3.7 3.3
Cost Recovery for Program Expenses 3.3 3.8 3.5 2.8
Integrated resource planning 3.2 2.9 3.1 3.6 Meeting environmental goals (e.g. CO2 emission reduction)
2.7 2.9 1.9 3.0
Participation payments for EE program results from ISOs in wholesale markets
1.8 2.1 1.5 1.4
Environmental permit allocation 1.4 1.3 3.6 1.2 Note: Scale - 1 = Not important; 2 = Somewhat important; 3 = Important; 4 = Very important; 5 = Most important
13 Sample sizes are relatively small; thus ascribing detailed nuances to these results versus broad trends is not suggested.
9
When asked to rate the importance of these evaluation uses in the future, 36% of respondents indicated that their ratings for current and future importance of evaluation uses would not change. Among those respondents (64% of sample) that indicated that their ratings would change for future evaluation applications, the largest change was in their ranking for meeting environmental goals: average ranking of 4.1 in future compared to a ranking of 2.7 (somewhat important) currently. We also asked several additional questions of policymakers and state regulatory agency staff on how they typically use information from evaluation studies (see Table 4).14 Of the 10 policymakers/regulators who responded to this question, 71% indicated that they read, review, and rely on the results described in EM&V studies during the normal course of their work.15 Almost all responded that they use the findings from program evaluations as a basis for program funding decisions and guide for improving program designs. Approximately half use these studies as basis for determining awards/penalties and to resolve disputes (where applicable).
Table 4. How state policymakers and regulators use information obtained from evaluation studies
Uses Number of Respondents
(N = 10)
As basis for program funding decisions or policy changes 9
As the basis for determining performance awards or penalties 5 As a guide to developing better program designs 10 To resolve disputes about program impacts and benefits among different stakeholder groups
6
Used by program administrators or staff to review the reports and make the changes supported by the analysis
4
We also asked policymakers/regulators to characterize the usefulness of evaluation studies on a scale of 1 to 5 with 5 representing “extremely useful.” The average rating among policymakers/regulators was 3.8 on the usefulness of evaluation studies.. Reasons given by policymakers/regulators that did not find the evaluation studies useful (i.e. ranked them <3) included timing of the results available from the EM&V study (i.e. too late), lacked appropriate discussion of various effects on results (e.g. small sample sizes), and use of inappropriate methods. This group of respondents offered two main suggestions to improve EM&V studies: greater use of non-technical language and a standardized set of “EM&V” terms in program impact evaluations (see Table 5).
14 See Appendix A, questions 1-5 in section A-2 asked only of state-level policymakers and regulatory agency staff. 15 There is potential for significant self-selection bias in the response to this question in that only people who read or rely on EM&V studies might respond to this type of survey.
10
Table 5. Suggestions made by policymaker/regulators to improve the usefulness of EM&V studies
Ideas for making evaluation studies more useful Number of Respondents
(N = 10)More quality control and providing context for numbers 5 Greater use of non-technical language in executive summary to convey key study results
7
Require the reporting of key information in standardized terms 7 Provide more education to policy makers on how program savings are estimated and why results may be relevant
4
4. EM&V Planning Practices, Methods and Protocols
In this section, we review approaches that various states have taken to planning their evaluation activities, including roles and responsibilities of various entities (e.g., program administrators, regulatory agency, stakeholders, evaluation consultants). We summarize respondents’ feedback on processes used to allocate resources among different types of studies (e.g. load impact, process, and market) and present information on EM&V budgets in various states and how EM&V budgets are allocated among different types of EM&V activities. We then summarize current practices and methods used in states to estimate savings from energy efficiency programs (e.g. definition and use of gross and net savings), including development of Technical Reference Manuals, use of “deemed” savings approach and requirements for the precision and accuracy of savings estimates. Finally, we summarize current practices for reviewing EM&V studies as well as public availability of EM&V study results in our sample of states. 4.1 Evaluation Planning Practices: Management Roles and Responsibilities and
Oversight
In order to obtain a sense of the institutional process used by states to plan and prioritize EM&V studies and allocate budgets, we defined three alternative processes that the research team has observed based on our work:16
Approach 1: Evaluation planning is initiated by evaluation contractor hired to conduct evaluation studies based on objectives and goals set by the EE Program Administrator (PA); PA is final decision-maker on evaluation studies
Approach 2: Evaluation planning is initiated and conducted by the EE Program Administrator (PA) in a public setting. Process can be collaborative but the PA is the final decision-maker on evaluation studies;
Approach 3: PUC staff or an advisory board (e.g., the Connecticut Energy Conservation Management Board) leads a public evaluation planning process, supported by an independent evaluation contractor; PUC makes final decisions regarding priorities and funding.
16 See Appendix A, Evaluation Planning Process, question 1.
11
The evaluation planning processes differ with respect to who initiates and manages the planning process and which entity is responsible for prioritizing EM&V studies and funding. In the first two cases, the program administrators manage and lead the evaluation planning process (either through an internal process that involves their evaluation contractor and utility program staff or through a public process), typically hire a third party evaluation contractor that provides input to this process and conducts the studies, and the program administrator prioritizes and makes decisions about budget allocations among EM&V studies. In Approach 3, regulatory commission staff initiate and manage a public evaluation planning process, typically directing independent evaluation contractors that have been hired to provide input and conduct studies, and the regulatory commission will make final decisions regarding priorities and budgets for EM&V studies.17 Survey respondents were then asked to identify which of the three processes were used in their state and/or comment on ways in which roles and responsibilities differed from our prototypes (see Table 6) for compilation of responses from state respondents. Although, the evaluation planning processes were intended to be more or less exclusive, survey respondents from seven states indicated that more than one evaluation planning process occurs in their jurisdiction. Some respondents selected multiple processes because they believed that different program administrators (e.g. investor-owned utilities vs. municipal utilities) in their state use different types of evaluation planning processes or because planning processes vary somewhat by program type or evaluation activity (e.g. process vs. impact). Respondents from Connecticut, Maine, NEEA, Pennsylvania, Texas and Wisconsin selected one option. A number of respondents indicated that their evaluation planning process was evolving over time, often with regulatory commission staff taking more responsibility for managing and overseeing the process.
17 These three approaches represent somewhat stylized prototypes that illustrate roles and responsibilities. In practice, evaluation planning is a collaborative process in some states with substantial input and involvement by program administrators, stakeholders, and regulatory staff.
12
Table 6. Evaluation planning processes: Roles and responsibilities
State
Evaluation planning is initiated and completed by
Evaluation firm hired to conduct evaluation studies based on the
objectives or goals from the program administrator (PA);
Final decision made by PA
Program administrator in a public setting; Final decisions
are made by PA.
Regulatory staff or oversight body who will ultimately
approve or adopt a research plan & budget;
Final decisions made by state PUC
CA X X
CT Evaluation Planning performed in
a collaborative setting FL X X IA X X (but not public setting) ID X X IL X X X MA ME X MN X X NEEA X NY X X X OR X X PA X TX X WI X Respondents reported that Program Administrators managed the process and made the final decision on which evaluation studies and activities to fund 68% of the time (i.e. the first two approaches) while regulatory agency staff managed and led evaluation processes in 21% of the cases (see Table 7). In most cases, the regulatory body may set the overall budget for EM&V, but then give the Program Administrator the discretion to decide the budget and the scope of the individual EM&V studies to execute. ‘Other’ was indicated if the respondent did not choose one of the other methods or indicated another type of planning process is used.
Table 7. Evaluation planning processes: Respondent views
Evaluation Planning Alternatives Percent Approach 1: Evaluation planning is initiated and completed by evaluation contractor hired to conduct evaluation studies based on the objectives or goals from the program administrator. PA is final decision maker
30%
Approach 2: Evaluation planning is initiated and completed by the program administrator (PA) in a public setting. Process can be collaborative but the PA is final decision maker
38%
Approach 3: PUC staff led a public evaluation planning process, supported by an independent evaluation contractor; PUC makes final decisions regarding priorities and budgets for EM&V studies
21%
Other evaluation processes 11%
13
States with adopted energy savings goals and related incentives or penalties tend to have more involvement of regulators in both planning and review of load impact studies and portfolio level evaluation reports (e.g., California, Minnesota, and Massachusetts). Jurisdictions without explicit savings goals and/or less experience tend to have evaluation planning processes led by the Program Administrators. Respondents in several states also indicated that they are in the process of transitioning from evaluation planning process led by evaluation contractors working on behalf of program administrators to evaluation planning processes managed and led by regulatory commission staff (or independent evaluation contractors working at their direction). This may explain why some survey respondents may have selected more than one evaluation planning alternatives. In summary, there is a range of evaluation planning processes across different jurisdictions that have evolved to meet the needs of Program Administrators and policy makers over time. We believe that the major differences are probably driven by several factors: (1) the size of program (and evaluation) budgets (e.g. states that spend less on energy efficiency tend to rely on program administrators to manage evaluation), (2) regulatory or legislative policies such as Energy Efficiency Resource Standards or performance-based incentives that place more responsibilities on regulatory commissions to assess and report program impacts, (3) technical expertise on energy efficiency program evaluation among regulatory agencies, and (4) the degree of trust between Program Administrators and state regulators. 4.2 Evaluation Planning Practices: Types of EM&V studies
We also asked respondents a series of questions on the type of evaluation activities and studies that are being conducted in their state and their assessment of the relative importance of these studies. EM&V activities were grouped into three broad categories: impact (including benefit cost analysis), process, and market effects (e.g., market baseline studies, market assessment, and potential studies).18 About 80% of the state-level respondents rated the importance of different types of evaluation studies at present and in the future (see Figure 2). Benefit cost analyses are separated out from the impact evaluation category since they are sometimes considered separately. In aggregate, respondents rated impact studies and benefit cost analysis as more important than process evaluation and market effects studies. In terms of projecting into the future, respondents perceive that the relative importance of process and benefit-cost studies is likely to decrease while market effects/market transformation studies are likely to become more important in the future.
18 See Appendix A, Evaluation Planning process, questions 11 and 12
14
1
2
3
4
5
Impact Process Benefit/Cost Analysis
Market Efects/ Market
Transformation
Avergae
Level of Im
portance
Existing
Future
Most Important
NotImportant
N=42
Figure 2. Respondent’s rating of relative importance of different types of evaluation studies
Respondents ranking of the relative importance of various EM&V activities were influenced by their relative experience level with energy efficiency programs and EM&V. Respondents from states with less experience tended to believe that implementing impact and benefit-cost analysis studies may be more important in the near term than approving studies to estimate market effects. In contrast, respondents from states that have implemented large-scale energy efficiency programs for many years tended to place a higher level of importance on estimating market effects from energy efficiency programs, comparable to load impact studies (i.e., 4.2 for market effects in future vs. 4.3 for load impact studies). This result could be driven by differences in program mix among less and more experienced states and/or respondents’ views on the relative importance of assessing long-term market effects of energy efficiency programs. 4.3 Allocation of funding for EM&V activities
The Consortium for Energy Efficiency (CEE 2009) compiles data on EM&V budgets and energy efficiency programs provided by program administrators and reports state-level information. We asked respondents to provide additional information on EM&V budgets allocated among three main types of EM&V activities (e.g. process, impact evaluations, and market research) as well as updated information on the total energy efficiency program budget and projected EM&V budget.19
19 See Appendix A, Section A.1; Resources for EM&V, questions 5-10
15
EM&V budgets in the 14 states vary from about $45 million in California to $500,000 in Maine (see Table 8). Based on information provided by respondents, EM&V budgets currently account for 5% or more of the total ratepayer-funded energy efficiency program budget in two states (CA and IA) and less than 1% of the total energy efficiency budget in two states (PA and TX). The budget for EM&V accounts for 2-5% of the total energy efficiency budget in eight states and the NEEA.
Table 8. EM&V 2008 funding levels and allocation among activities
State EM&V Funding
(million $) EM&V Funding as % of
Total EE Funding
Allocation of EM&V Budget (% Impact / % Process /
% Market Research) CA 25 to 45* > 5%* 75 / 15 / 10 CT 2.0 3% NA FL NA NA NA IA 8 > 5% 50 / 30 / 20 ID NA NA NA IL 2.6 2-3% 75 / 15 / 10
MA 6.3 3-5%* 75 / 15 / 10 ME 0.5 2-3% 30 / 50 /20 MN 0.7 3-5% NA
NEEA NA 3-5% 60 / 30 / 10 NY 820 3-5% 80 / 10 / 10 OR 2* 2-3%* 50 / 30 / 20 PA NA <1% 50 / 30 / 20 TX 0.8 <1% 75 / 15 / 10
WI21 3 to 4 3-5% 100 / 0 / 0 * Range depicts answers provided by different respondents from California; funding also includes evaluations activities for codes and standards. NA = not available The allocation of EM&V budget among different types of activities also varies substantially across jurisdictions. Six states currently allocate 75% or more of their EM&V budget for impact evaluation while three states and NEEA allocate between 50-60% of their EM&V budget to impact evaluations. Respondents in Maine indicated that process evaluations account for 50% of their EM&V budget and it appears that many states allocate between 10-20% of their EM&V budget to market research. Wisconsin does not allocate any budget to either process or market research studies. Respondents from four states (Connecticut, Florida, Idaho, and Minnesota) were unable to provide estimates of budget allocations by type of EM&V activity. Respondents were also asked to rank the relative importance of various factors in selecting the type of evaluation studies to pursue (load impact, process, and market effects) during the planning phase and in determining allocation of EM&V budget. The overall budget allocated for EM&V and the perceived degree of uncertainty in existing estimates of program savings and costs were ranked as the most important factors by respondents.
20 Annual funding for the 2009-2013 cycle. 21 Program administrators may conduct some market research during program implementation phase; market research is not carried out as part of EM&V activities.
16
4.4 Evaluation Practices: EM&V protocol and Technical Reference Manual of deemed savings
Three states (California, New York, and Wisconsin) have developed their own EM&V protocols which require the use of specific methods to evaluate the impacts of efficiency programs (see Table 9).22 Respondents indicated that Florida and Iowa rely primarily on the IPMVP protocol, while Illinois and Pennsylvania are considering developing an EM&V protocol. Eight states (CA, CT, MA, ME, MN, NY, TX, and WI) and the Pacific Northwest region currently have developed a standard energy efficiency measure data base (or Technical Reference Manual) and several states are planning or considering developing Technical Reference Manuals (e.g. PA, IL).23 Practices differ in terms of whether the use of deemed savings is mandatory for program administrators or encouraged and whether deemed values are verified, ex-ante or ex-post, by an independent party. Standard EM&V practice indicates that when using deemed savings it is crucial to verify a sample of installations to ensure the measures were actually installed and working per the specifications defined for using the deemed savings value. Eight states have or are considering an audit requirement to verify a sample of installations resulting from efficiency programs.
22 The approach and scope of EM&V protocols varies somewhat among states. For example, in some states, EM&V protocols define the planning and analysis processes to use for EM&V of specific types of energy efficiency programs. Some protocols offer guidance while others lay out mandatory methods and/or certainty/reliability criteria for reported results. 23 In Wisconsin, TRM has been developed for half of the programs and is under preparation for the remaining.
17
Table 9. EM&V Protocols and Technical Reference Manual
State
EM&V protocols exist
to guide or require use of
specific methods to
evaluate program or
market savings?
Is there a Technical Resource
Manual or Deemed Savings
database?
Audit requirement to verify sample of
installations?
CA Yes Yes Yes CT No Yes Part of program
FL No; Rely on
IPMVP No Yes
IA No; Rely on
IPMVP No No
ID No No No
IL Not yet, in
process Not yet Yes
MA No Yes; in next two
months Yes for most
programs ME No Yes No
MN Yes for custom
projects Yes Yes
NEEA No No No NY Yes Yes Yes OR No No No PA Not yet Planned Under review TX No Yes In practice yes WI Yes Planned Yes
4.5 Evaluation Methods to estimate Savings and Load Impacts
In this section, we summarize current practices and methods used to estimate savings from energy efficiency programs. 4.5.1 Guidance on Required Components in Program Savings Report
Evaluation consultants have chosen to estimate the savings from programs at different levels of granularity depending on the needs of program administrators and state regulatory policies. We asked respondents to provide information on current practices in their jurisdiction with respect to the level of granularity with which savings are reported:
Average Savings over a population for specific energy efficiency measures; Average Savings at the end use level where more than one measure may have been
installed; Average Savings at the program level; or Average Savings at the portfolio level.
18
The level of detail chosen in each jurisdiction depends in part on regulatory policy and the extent to which program administrators rely on ex-ante deemed savings values for specific efficiency measures or systems. Jurisdictions with significant experience in implementing large-scale energy efficiency programs tend to rely on estimating savings at the measure level because these inputs are needed to assess program cost-effectiveness and assess differences between planned and achieved program savings. However, in some states that are ramping up ratepayer-funded energy efficiency, evaluation efforts tend to focus on reporting savings at the program level initially and then over time to report savings at the end use and measure level. 4.5.2 Net and Gross Savings
We provided a short glossary of commonly used definitions of net and gross savings from previous Action Plan documents (Schiller, 2007), as part of the interviews and discussions with respondents:
Gross savings - The change in energy consumption and demand that results directly from program-related actions taken by participants in an efficiency program, regardless of why they participated.
Net savings - The total change in energy consumption and demand that is attributable to an EE program or efficiency standard. This change may include, implicitly or explicitly, the effects of free-drivers, free-riders, state or federal energy efficiency standards, changes in the level of energy service and natural change effects, spillover, and market effects.
During the literature review, it became clear that there is some variation among states in how gross and net savings are defined and in the relative priority placed on reporting gross and/or net savings in estimating load impacts from energy efficiency programs. In order to assess current practices in this area, respondents were asked to identify which savings metrics are reported and provide definitions used in their states.24 The greatest variation in practices is in the types of market influences and program effects that are included or required by states in estimating the net load impacts of ratepayer-funded energy efficiency programs (see Table 10). For example, free-ridership effects are analyzed in estimating net program savings in ten states, while eight states require estimates of spillover or market effects in estimating net program savings. California is the only state in our sample that attempts to account for leakage effects in selected energy efficiency programs, which involves estimating the fraction of incented products that move across state lines. In many states the choice of whether to measure free riders or market effects is an ad hoc decision influenced by the size of the program and available evaluation budgets. For example, in California free ridership is estimated for all energy efficiency programs but the positive (or negative) market effects caused by energy efficiency programs are only periodically evaluated on an ad hoc basis. Market effects and the related load impacts of programs are sometimes
24 See Appendix A; Section A.1 Load Impact Methods, Questions 8 and 9. Some respondents provided definitions for net and gross savings; many respondents agreed with the definitions that were provided in the glossary.
19
measured for large programs although currently, utilities in California are not allowed to include these market effects estimates in annual reports of program savings.
Table 10. Market influences and program effects included in estimates of net savings from energy efficiency programs
State Free-ridership Spillover/ Market
Effects Leakage to Other
States?
CA Yes Yes in few casesYes, but just for
selected programsCT Yes Yes in some cases NA FL Yes Yes No IA No No NA
ID25 No No No IL Yes Yes NA
MA Yes Yes No ME Yes No No MN Yes No No
NEEA No No No NY Yes Yes No OR Yes Yes No PA No NA No TX No No No WI Yes Yes in few cases No
Total Yes 10 8 1 In assessing the effects of these parameters, an understanding of how they are measured, how frequently they are measured, and how the results are used are important considerations, and would be helpful companion questions in any future comparative research on net savings. Program administrators and evaluators are required to report both gross and net savings in seven states (CA, FL, IL, IA, MA, ME, TX, and WI) and NEEA (see Table 11). Only gross savings are required to be reported in Illinois, Minnesota, and Texas, while only net savings are required to be reported in Connecticut and New York. Five of seven states that require both gross and net savings to be reported indicated that the net savings include spillover and/or market effects caused by the program. In Maine, the definition for gross and net is the same, while in NEEA and California the net savings definition does not include spillover and/or market effects.
25 Idaho PUC staff has notified each of Idaho’s utilities that net-to-gross estimates are necessary for credible reports of savings and cost-effectiveness and that at least some of the utilities program reports do include net-to-gross adjustments. However, there is no formal requirement for including estimates of free-ridership, spillover, and leakage by the PUC.
20
Table 11. Reporting and definition of net and gross savings in EM&V studies
State
Requirement to report gross or net program savings or
both?
Do net program savings include spillover and/or market effects
caused by program? CA Both No CT Net NA FL Both Yes IA Sometimes both No ID Net26 No IL Both Yes MA Both Yes
ME Both Gross and Net are
defined as the same MN Gross NA
NEEA Gross No
NY Net Yes OR Both No PA Under Review Under Review TX Gross NA WI Both Yes27
We also asked state respondents for their views on what type of savings should be reported for ratepayer-funded energy efficiency programs (see Table 12).28 About 57% of respondents favored reporting both gross and net savings, while 15% favored reporting only gross savings. Some respondents that favored reporting only gross savings indicated that conducting extensive net-to-gross type of studies is a waste of resources (i.e. time and money) given the uncertainties in attributing savings to a single program. About 10% of respondents favored reporting only net program savings.
Table 12. Respondent views regarding reporting of net and/or gross savings
Should the energy savings from utility programs be reported as net program savings,
gross program savings, or both?
Responses (N=50)
Report Net savings only 6% Report Gross savings only 15%
Report Both 57%
Neither or No response 22%
26 This is not a formal requirement by the PUC, however, PUC staff has notified all the utilities to estimate both net and gross savings – especially, net savings – since, they are necessary for estimating cost-effectiveness of EE programs. 27 Often these studies are without quantification. 28 See Appendix A, Section A.1; Load Impact Methods, Questions 10 and Section A.3, Question 11.
21
About two-thirds of the state respondents indicated it was definitely (or possibly) a problem if some states estimated only gross savings while other states estimated and reported only net savings (see Table 13).29
Table 13. Comparability of EM&V load impact results among states
From your perspective, is it a problem if utility program savings are being reported using gross savings estimates in
some states and using net savings estimates in others?
Responses (N=49)
Yes 35% Maybe 33%
No 15% No answer 17%
About 68% of respondents indicated that inconsistencies in program savings terms would become a “very important problem” in the event that energy efficiency resource standards were adopted at the national level (ranking of 3.89 on a scale of 1 to 5 where 1 represented “not important” and 5 represented “extremely important”). Respondents were then asked to propose initiatives that could address this consistency problem. About 60% of the survey respondents indicated that a glossary of program savings terms could be developed at a national level to ensure that states and program administrators use the same savings definitions or terms when reporting program savings. For example, precise definitions of net and gross savings could be used by the Energy Information Agency (EIA) as part of its annual data collection efforts through forms such as EIA-861. About 47% of the respondents also supported the development of national EM&V protocols that could be made available to states. However, a number of respondents from states that have implemented large-scale ratepayer funded energy efficiency programs for a long time raised concerns about developing national EM&V protocols as they felt their existing state-level protocols on gross and net savings are likely to be better suited to their respective regulatory frameworks and policies than a national protocol. 4.5.3 Estimating Load Impacts from Energy Efficiency: Dealing with uncertainty
We also asked questions about how precision and accuracy requirements for estimating savings from energy efficiency were set and by who as part of our effort to understand current evaluation practices.30 We found that only four states (CA, CT, NY, and MA) require that their EM&V methodologies incorporate estimates of the uncertainty in program load impact estimates, while Illinois is considering adding this requirement (see Table 14). These reporting requirements are usually described in an EM&V protocol or public utility commission decision on ratepayer-funded energy efficiency programs.
29 See Appendix A, Section A.1 Emerging Policy Issues, Questions 14 and 15 and Section A.3, Questions 15 and 16. 30 See Appendix A; Section A.1 questions 13 and 14.
22
In terms of which entities are involved, respondents indicated that requirements are set on a case by case basis through a collaborative process involving multiple entities in seven states (see Table 15). For example, in Illinois and Oregon all three entities (i.e., oversight body, program administrator and evaluator) are involved in setting the precision and accuracy requirements (see Table 12). In six states (Connecticut, Massachusetts, Maine, New York, Pennsylvania, Texas, and Wisconsin), the precision and accuracy requirements are set by only one entity; the entity varies by state. In California, the accuracy and precision requirements are codified as part of the adopted evaluation protocols and enforced by the staff of the California Public Utility Commission.
Table 14. How is certainty defined and addressed
State
Requirement to identify range of uncertainty in program savings estimates?
EM&V protocols require (or provide guidance) on specific methods to use in evaluating and reporting savings uncertainty?
California Yes Yes
Connecticut Yes; 80/20 for
surveys No
Florida No No; rely on IPMVP
Idaho No No Illinois No, but likely Not yet, in process
Iowa No No; rely on IPMVP
Maine No No Massachusetts Yes No
Minnesota No Yes for custom projects NEEA No No
New York Yes Yes Oregon No No
Pennsylvania No Not yet Texas No No
Wisconsin No Yes
23
Table 15. Who sets precision, accuracy, or robustness requirements for savings estimates?
Precision, accuracy, and robustness requirements for savings
estimates are set by:
State Oversight Body and codified in protocols
Program Administrators on a
case by case basis
Evaluator based on professional judgment, prior experience and/or
budget constraints CA Program & Portfolio Program & Portfolio
CT By evaluator but reviewed
by Collaborative Board
FL Program Program IA Program Program ID Program & Portfolio Program IL Program & Portfolio Program & Portfolio Program & Portfolio
MA Program & Portfolio
ME Program MN Program & Portfolio
NEEA Program & Portfolio Program NY Program & Portfolio
OR Program Program & Portfolio Program PA Program & Portfolio TX Program & Portfolio WI Program Program & Portfolio Program & Portfolio
4.6 Implementation of EM&V studies
Respondents in the case study states were also asked to indicate how long it typically took for program savings results to be available in their state after energy efficiency programs were implemented.31 About 30% of the respondents indicated that savings estimates were available between 12-18 months after programs were implemented and another 30% indicated that it took longer than 18 months for these results to be available (see Table 16). We found that the answers on typical cycle time for saving estimates varied among respondents from the same state, in part because respondents had different interpretations of the term savings estimates. Respondents that interpreted these two words to mean estimates of program savings from the tracking systems tended to report the cycle time as less than six months while other respondents interpreted savings estimates to mean the time it took for an independent party to complete an evaluation of gross program savings. These respondents indicated that it took from 12 to 18 months from program end date until the verification report was complete.
31 See Appendix A, Section A1.; Load Impact methods, question 21.
24
Table 16. Typical cycle time for savings estimates from energy efficiency programs
Time Period Number of Survey Respondents
(N = 40) Less than 6 months 7
6-12 months 8 12- 18 months 12
More than 18 months 12 The frequency of evaluations for the same (or similar) energy efficiency programs implemented in consecutive years varies across jurisdictions and program types. Twenty respondents noted that energy efficiency program evaluations usually take more than a year to complete. Fourteen respondents indicated that programs are evaluated annually or more frequently.” A number of respondents indicated that the time spent conducting the load impact evaluation varies by program and also may change from program cycle-to-cycle as EE programs and policies evolve. 4.7 Review and reporting of EM&V study results
Table 17 summarizes current practices for reporting EM&V study results in these jurisdictions. In virtually all states (except Idaho), annual reports that summarize program savings are currently available in the public domain or planned for the future (see Table 17). In most states (except Iowa, Idaho and Texas), program savings are aggregated in order to determine whether the annual energy efficiency savings goals are met. In most states (except Florida and Minnesota), a public review is normally conducted of draft load impact studies. In Connecticut, Illinois, and New York, an advisory group conducts the review or a third-party is appointed specifically for conducting the review.
25
Table 17. Availability and review process for load impact studies
State
Are annual reports that summarize
program savings available in the public domain?
Do annual reports include
program savings summed to
determine if annual savings
goals were met?
Is a public review of draft load
impact studies conducted?
CA Yes Yes Yes
CT Yes Yes Board appoints 3rd parties to review
FL Yes Yes Yes IA Yes No Yes ID Yes No Sometimes
IL Not yet Planned Yes; Evaluation
group MA Yes Yes Yes ME Yes NA Yes MN Yes Yes No NEEA Yes Yes Yes
NY Yes Yes Yes; Evaluation Advisory Group
OR Yes Yes Yes PA Not yet Will be In process
TX Yes No Yes for some
utilities; No for some utilities
WI Yes Yes Yes
26
5. Current and Emerging Issues in EM&V Practices
In this section we first summarize key differences in evaluation practices and methods among the 14 states and NEEA, discuss key factors that may account for these differences, and identify EM&V issues that arise if the policy objective is to move towards more consistency across states and regions in assessing load impacts of energy efficiency programs. We then identify six EM&V issue categories based on discussions with respondents and the Action Plan EM&V Technical Work Group. 5.1 Emerging issues in evaluation practices and methods: Key drivers
Table 16 summarizes key differences in evaluation practices and methods identified among the jurisdictions in this study. The project team identified three key factors as having a significant influence in the evolution of evaluation practices over time among states:
1) Length of Experience - the number of years of experience that a state has in implementing and evaluating energy efficiency programs or the relative size of a states’ energy efficiency program activity, which is often correlated with where program administrators and regulatory commissions are in terms of the “learning curve” of evaluation practices;
2) Regulatory Culture - the degree of regulatory intervention and oversight by regulatory staff (and involvement by stakeholders) in energy efficiency program planning, development and evaluation efforts, including the degree of trust and cooperation between regulators and program administrators; and
3) Incentive Structure - the existence of statewide or utility-specific energy savings goals with consequences for non-performance (e.g., Energy Efficiency Resource Standard) or performance incentives offered to program administrators for successfully achieving program savings or other impact targets.
We identify these factors and list those that have in our judgment the largest impact in influencing differences among states (see middle column of Table 18). For example, we think that the fact that states have varying levels of experience implementing energy efficiency programs is probably the biggest reason that accounts for lack of consistency in reporting conventions (#2). Conversely, energy efficiency program administrators face varying legislative requirements (e.g. state EERS vs. requirement to do a DSM plan) or have different financial incentives/consequences which we think is a key driver that accounts for varying definitions of program savings (#1) and methods used to measure net savings (#4), and often increased scrutiny and review of load impact studies (#8). In the last column of Table 18, we highlight key evaluation issues identified by respondents (and the project team) that are likely to arise if the policy objective (or legislative requirement) was to ensure more consistency in reporting of savings and load impacts from energy efficiency programs across different jurisdictions.
27
Table 18. Differences in evaluation practices and methods among states: Potential implications if move towards regional or national protocols
Differences in Evaluation Planning and Practices among states
Key Factors Driving Differences: 1: Length of Experience 2: Regulatory Culture 3: Incentive Structure
EM&V Issue
1. Definition of program savings – e.g., are savings reported as gross, narrow net, or wide net?
1, 3 Comparability of savings reports among states and impact of EE program savings on load forecasts
2. Reporting conventions (e.g., program Savings = forecast savings, or reported savings (from the Program Administrator) or verified savings (as reported by independent evaluator)
1 Comparability of savings reports among states and impact of EE program savings on load forecasts
3. Existence and Use of Evaluation Guidelines or Protocols
1, 2 Is there a need for regional or national guidelines? Is there a need for mandatory EM&V protocols?
4. Methods used to measure net savings for use in program reporting and cost-effectiveness screening
1, 3 Disagreement on the value, accuracy and ability to estimate net savings rather than measuring gross savings & stipulating to ex ante net-to-gross (NTG) values
5. Use of ex ante compared to ex post evaluation approaches to estimate verified savings; relative importance of determining program attribution among similar programs in same market; frequency of performing market change or market effects evaluations
1, 2 What is the relative priority of measuring total savings in markets compared to net program savings? What are the potential costs and benefits of extensive use of deemed savings to estimate program savings?
6. Public disclosure requirements for EM&V study results and/or inputs and outputs used in evaluation models
1, 2 Comparability of savings reports among states; potential need to encourage more third party review of evaluations sponsored by program administrators
7. Varying or ad hoc processes used to allocate evaluation resources between load impact, process, and market evaluations
1, 3 Over-allocation of resources to load impact studies can crowd out resources for process or market evaluations
8. Level of effort used to review evaluation methods and results from load impact studies varies from no third party review to intensive review and public hearings.
1, 2, 3 Can the savings results from all states be treated as equally reliable or rigorous given the wide range in intensity of review processes? Does it matter?
9. Responsibility for estimating GHG impacts of EE programs is spread between program administrators, air quality agencies and utility commissions,
1, 3 Should EE program administrators be responsible for reporting GHG impacts? Who should vet GHG calculations and what is desired level of accuracy?
10. Integration of EE portfolio savings estimates (gross or net) to modify statewide or regional load forecasts at nodal levels is sometimes required, often neglected and usually executed poorly because of difference in baseline assumptions between EE programs and load forecasting models
1, 2, 3 In some cases, it is not correct to simply subtract EE program savings from the load forecast. Better understanding of forecast of baseline usage and naturally occurring savings is needed.
Note: Key factors found to drive these practice differences are: 1 = Amount of EE Program Experience; 2 = Regulatory culture; 3 = Incentive Structure for Program Administrators
28
5.2 Summary of Key Evaluation Issues
Discussions and interviews with state-level respondents, national experts, and the EM&V Technical Work Group identified a broad set of issues that need to be addressed to improve the consistency of EM&V practices, methods, and reporting among states. The project team consolidated and grouped these emerging issues into six broad categories, in part to provide a framework for a set of suggested activities and projects that begin to address these key EM&V issues:
1. Consistency in Reported Savings or Load Impacts - Differences between jurisdictions in how program savings are estimated and how they are defined (e.g., net versus gross) makes comparison of reported savings between jurisdictions and their comparison against benchmarks very difficult if not impossible.
2. Measurement Methods Used to Estimate Net Savings – EM&V methods are well documented and relatively standardized for determining gross (direct) energy savings for energy efficiency projects. However, there is much less agreement on the value and methods that should be used to estimate net savings. Key areas where differences exist on issues relating to net savings include: (1) how, if at all, to address program attribution; (2) how to define and set standards for rigor and accuracy for net savings given different policy objectives, and (3) how to assess broader “net” market effects of energy efficiency programs on future spillover savings in the market and the demand for energy services.
3. Quality Control and Accuracy – The quality of program evaluation and review processes varies widely among states because of differences in the level of independent review of program saving estimates required by state regulatory commissions and because relatively few states require that the level of uncertainty associated with program savings estimates be reported, driven in part by the level of technical expertise (e.g., lack of understanding on how to or the importance of reporting estimation certainty).
4. Evaluation Resource Allocation - Some states with less experience with energy efficiency tend to focus on reporting first year gross program savings and tend to underfund comparative analysis on the effectiveness of alternative program designs (e.g. process evaluations) and estimates of the market changes caused or attributed to energy efficiency programs. States with more experience often focus on estimating net savings attributable to a set of programs offered in various market segments and spend more dollars on estimating net savings. Making a decision on what type of evaluation planning process is likely to yield the optimal allocation of evaluation funds and what level of stakeholder involvement is needed in these funding decisions is an unresolved issue in this field.
29
5. Independence of Program Evaluators – EM&V has two primary objectives, (1) to assess the savings resulting from an efficiency program or portfolio and (2) to provide feedback for program/savings improvement. In a regulatory environment where program savings levels affect administrator compensation, there is a natural friction between the need for independence between evaluation consultants responsible for estimating program savings and program administrators. In contrast, a close working relationship between program administrators and evaluators facilitates evaluators being able to provide useful feedback on programs. We found that a number of states have moved away from having program administrators manage and/or oversee load impact and cost-effectiveness studies in order to reduce perceived or potential conflicts of interest or undue influence by program administrators. In some cases, program administrators are only allowed to manage process evaluations of programs. These institutional solutions can create additional challenges in sharing evaluation results in a timely fashion and limit the opportunities for integrating process, market and impact evaluations into existing and future program improvement efforts.32
6. Integration of Load Impact Results From Energy Efficiency Programs Into Utility Planning and Forecasting – – We found that states have made limited progress in addressing the analytic challenges associated with aggregating and incorporating estimates of net or gross program savings into load forecasting frameworks.33 Failure to fully account for and reconcile program savings estimates with load forecasts may result in under- or over-counting of savings which may adversely affect major resource planning decisions and/or estimates of impacts on greenhouse gas emissions (GHG).
5.2.1 Relative Importance of EM&V Issues under alternative future scenarios for energy efficiency
A number of respondents also indicated that the relative priority of addressing these key EM&V issues depends to some extent on the energy efficiency policy and market environment over the next three to five years. We defined three scenarios for energy efficiency that characterize future policy and market drivers (see Table 19):
1. “Business as Usual” where EM&V priorities and practices in the future continue to be driven primarily by state (and local) policies;
2. “Regional Policy Drivers,” where EM&V priorities and practices are significantly influenced by regional transmission organizations or policy initiatives; or
3. “National Policy Drivers” where EM&V priorities and practices are driven by national energy and/or climate legislation requiring state level reporting of savings levels.
32 Some respondents indicated a preference to reintegrate process and impact evaluation into one integrated series of evaluation reports for the use of program managers. 33 In many cases, respondents reported that it was unclear whether a particular type of forecasting model requires estimates of gross or net savings to be consistent with the implicit forecast of naturally occurring energy efficiency investments contained in the model’s drivers.
30
Table 19. Alternative future scenarios for energy efficiency
Scenario #1 – Energy Efficiency Markets under “Business as Usual” future
Interest and funding for ratepayer-funded EE programs continues to grow at the state and local level driven by:
- legislative and regulatory requirements
- high cost and environmental impacts of conventional supply-side resources
- growing environmental concerns about climate change, local economic development (job creation/retention
Scenario #2 – Energy Efficiency Markets driven by Regional Policies
EM&V practices and reporting requirements are driven by the efforts of regional planning and/or reliability organizations (e.g. ISO, RTO) to standardize evaluation methods for use in regional transmission planning processes and wholesale markets (e.g. forward capacity markets).
Program Administrators begin to recognize and achieve economies of scale in EM&V by working with regional EE organizations (e.g. NEEP or NEEA) to develop common evaluation methods and data.
Scenario #3 – Energy Efficiency Markets under a national EERS or Climate Change legislation
Congress enacts some form of a Combined Energy Efficiency and Renewable Electricity Standard (CERES) that includes energy efficiency as an eligible resource, which may or may not be part of national carbon legislation.
A CERES or separate Energy Efficiency Resource Standard (EERS) includes requirement for nationwide reporting of program savings
A federal agency is given the responsibility of developing consistent EM&V protocols that are used by entities to comply with the CERES or EERS requirements.
The six EM&V issue categories are all important issues to address; however, we also provide our assessment of the relative importance of addressing these EM&V issue areas (e.g. High, Medium, or Low) under alternative scenarios for energy efficiency, drawing from discussions with the EM&V Technical Work Group members (see Table 20).
Table 20. Relative importance of addressing emerging EM&V issues*
Decision Criteria
Evaluation, Measurement and Verification Emerging Issues
Consistency in
Reporting Issues
Measurement Issues in
estimating Net Savings
Quality Control
& Accuracy
Evaluation Resource Planning
Timeliness of and Feedback on Evaluation Study results
Integration of program impacts into load forecasts
EE under “Businesas Usual” Future
Low High Medium Medium Low to
Medium (**)
Low to High (depending on state’s savings
goals) Regional Initiatives as Driver
Medium High Low to High
Low Medium Very High (due to forward capacity
markets)
National Policy as Driver
Very High High High Medium Medium
Low to High (depending on
adopted savings goals metrics)
* EM&V issue #5 – This table does not include addressing evaluators’ independent versus cooperative relationship with program managers, which is a generic issue common to all future scenarios. ** When a range of relative importance is shown that indicates the importance is contingent on multiple factors.
31
As Table 20 illustrates, we believe that the relative importance of addressing emerging EM&V issues depends to some extent on the future policy and market environment for energy efficiency. For example, addressing EM&V issues associated with estimating net savings from energy efficiency program savings is a relatively high priority under any future scenario. In contrast, developing a consistent approach among states on the definition(s) and reporting of program energy savings is a very high priority under future scenarios in which there are national energy efficiency resource standards or carbon markets compared to a future that involves a continuation of the current policy and market environment for ratepayer-funded energy efficiency in which state policies play a key role. If regional initiatives (e.g. regional transmission planning processes or ISO/RTO markets) are a primary driver for energy efficiency in the future, then greater attention may be focused on issues involved in integrating and accounting for load impacts of energy efficiency in long-term load forecasts used by regional organizations to define transmission and generation market needs. If there were a national EERS or carbon legislation in which energy efficiency was a compliance strategy in a cap and trade environment, then we believe it would be a high priority to have consistent approaches to reporting savings and quality control and accuracy protocols that provided increased confidence that states had procedures for independent (or public) review of program savings estimates and required reporting of uncertainties associated with savings estimates.
32
6. Recommendations
In this section, we provide a number of recommendations for activities that can be undertaken to address the EM&V issues identified in this report and describe several project concepts.34 The relative priority or importance of these activities varies to some extent on future energy efficiency policy and market environment (e.g., Business as Usual driven by state/local policy, National Drivers, Regional Drivers). However, many of the proposed activities appear to be beneficial irrespective of the future scenario(s) for energy efficiency. The specific EM&V activities recommended are:
1) Develop new method(s) or enhance existing methods for estimating total program savings that includes both the near term effects of the installation of energy-efficient technologies or systems and the longer-term behavioral impacts caused by the energy efficiency program. This can include user issues, such as changes in behavior and long-term maintenance induced by the program, as well as broader market messaging and impacts associated with programs.
2) Develop and share best practices guides and case studies on evaluation methods and planning through webinars and regional seminar series with state, utility, regulatory staff, and industry. Possible topic areas include:
compendium of methods used to develop forecasts of baseline energy usage;
processes used to verify the load impact results of EE programs;
compendium of approaches used to structure EE evaluation planning processes;
review of the methods and criteria used to select ex-ante measure specific savings values (deemed or stipulated savings) and;
review of different methods used to estimates net savings from efficiency programs; and
review methods used to estimate gross and net GHG impacts.
3) Develop an accessible national or set of regional databases of standardized ex-ante savings estimates for energy efficiency measures.
4) Design and implement a national, searchable data base that provides access to energy efficiency evaluation plans, tools and reports developed and filed by program administrators in various states during the past two or three years.
5) Develop and disseminate a best practices guide that reviews the range of processes and techniques used to improve quality control in savings reports, including recommendations on data collection, quality control, and analysis.
34 These recommendations are limited in part because the focus of this effort was gathering information for the Action Plan EM&V Task Force to assess.
33
6) Develop a short program savings reporting format (i.e. one page) for all states and regions to use in reporting program savings and seek voluntary adoption by region(s) via regional workshops or other strategies.
7) Develop a glossary of standardized EM&V and measure, program, or portfolio terms for voluntary use by all states and seek voluntary adoption by region(s) through regional workshops or other strategies.35
See Appendix B for a more detailed description of several of these recommended EM&V activities. To refine these recommendations further, effort is needed that should include evaluating the following for any proposed activity:
What is the near and long term objective of the activity? How is success defined and assessed?
Who is the target audience for each activity? What strategies will be used to effectively reach actors in each of these audiences?
What strategies will be used to convince these actors to actually use and incorporate the information provided through these activities in their daily work?
At what level should the proposed activity be implemented and by whom to improve the chances of achieving support from key stakeholders at regional or national levels?
35 See P. Horowitz 2009. “Glossary of Evaluation Terms: Version 1.0, NEEP Regional Evaluation, Measurement and Verification Forum,” March. This glossary prepared for NEEP is a good example of a regional EM&V glossary.
34
7. Conclusion
This report has identified significant differences in the practices and reporting requirements used to evaluate the load impacts and effectiveness of energy efficiency programs in the United States. Regional efforts to learn from best practices are just beginning to bear fruit. We hope that this inventory of EM&V practices and methods will, at a minimum, provide a foundation for efforts to share information, foster best-practices, standardize reporting formats and improve consistency in reporting of savings from energy efficiency programs. The report also highlights the need to expand or perhaps improve EM&V methods and approaches, particularly for addressing impacts of efficiency on total capacity and energy requirements, savings attribution among multiple programs that target similar markets, environmental benefits of efficiency, and/or the impacts of behavior, operational changes, or information feedback actions.
35
References
Barbose, G., C. Goldman, and J Schlegel 2009. “The Shifting Landscape of Ratepayer-Funded Energy Efficiency in the U.S.,” LBNL-2258E, Lawrence Berkeley National Laboratory Berkeley, October. Consortium for Energy Efficiency 2009. “2008 Annual Industry Report,” Available at URL - http://www.cee1.org/ee-pe/2008/index.php, accessed on January 20, 2010. Eldridge, M., Neubauer, M., York, D., Vaidyanathan, S., Chittum, A., and S. Nadel 2008. “The 2008 State Energy Efficiency Scorecard.” American Council for an Energy-Efficiency Economy Report No. E086. Washington, DC: ACEEE. Erhardt-Martinez, K., J. A. Laitner, and K. M. Keating 2009. “Pursuing Energy-Efficient Behavior in a Regulatory Environment: Motivating Policymakers, Program Administrators, and Program Implementers," White paper prepared for California Institute for Energy Efficiency and the California Public Utilities Commission, September. Gordon, Fred 2010. Personal communication for Energy Trust of Oregon, April. Horowitz, P. 2009. “Glossary of Evaluation Terms,” facilitated by the Northeastern Energy Efficiency Partnership, March. Hopper, N., G. Barbose, C. Goldman, and J. Schlegel 2008. “Energy Efficiency as a Preferred Resource: Evidence from Utility Resource Plans in the Western United States and Canada,” LBNL-1023E, Lawrence Berkeley National Laboratory Berkeley, September. Kushler, Marty 2009. Personal communication, December. National Action Plan on Energy Efficiency 2007. “Model Energy Efficiency Program Impact Evaluation Guide,” prepared by Steven R Schiller, Schiller Consulting, Inc, www.epa.gov/eeactionplan TecMarket Works 2006. “California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals,” April.
36
Appendix A. Interview and Discussion protocol
Appendix A includes questions used in our interviews and discussions with program managers, policymakers, regulatory staff, evaluation managers, and evaluation consultants. A.1. State-level Respondents: Policymakers/regulatory staff, program
administrators/managers, evaluation consultants Data and Information Questions Context, Uses, and Users
1. Who currently evaluates the energy savings from or the effectiveness of Energy Efficiency (EE) programs in your state? (Check one or list alternate)
Utility Internal staff 3rd Party Evaluation contractors (Describe) State organization staff Others: Please list here
2. Please rank the importance of the audiences for evaluations of energy efficiency programs (evaluation
studies). (Please use the ranking scale for each group or indicate Don’t Know (DK)) Utility Regulator Utility Program Managers Consumer groups State Energy Offices Environmental Agencies Reliability Organizations Independent System Operators Market Monitors Energy Service Companies Third Party Implementers (hired by program managers) Other: Please list
3. Please rank the importance of the current uses of Energy Efficiency evaluations. (Please use the ranking
scale for each group or indicate Don’t Know (DK)) Goal setting (Electricity or Natural Gas Savings goals, potential studies) Cost Recovery for Program Expenses Performance incentives, penalties Integrated resource planning Meeting environmental goals (e.g. CO2 emission reduction) Participation payments for EE program results from ISOs in wholesale markets Environmental permit allocation Program planning, budgeting, design and refinement Assessing cost-effectiveness Quantify level of program energy and peak savings Identifying or Targeting receptive market segments for EE programs Other: Please list
37
4. Please rank the importance of future uses of Energy Efficiency evaluations. (Please use the ranking scale for each group or indicate Don’t Know (DK))
NO CHANGE Goal setting (Electricity or Natural Gas Savings goals, potential studies) Cost Recovery Performance incentives, penalties Integrated resource planning Meeting environmental goals (e.g. CO2 emission reduction) Participation payments for EE program results in wholesale markets Environmental permit allocation Program planning, budgeting, design and refinement Assessing cost-effectiveness Quantify level of program energy and peak savings Identifying or Targeting receptive market segments for EE programs Other: Please list
Resources for EM&V
5. What is the source of EM&V funding used to evaluate EE programs in this state? (Check all that apply) Utility program funding State agencies or organizations Other: Please list
6. What is your estimate of the amount of money spent each year to evaluate EE programs in your state? $ millions
Please provide reference to the budget estimate if available The budget estimate developed by CEE is one potential reference
7. Does this amount include the cost of evaluating state codes and standards or just utility programs? (Check
one) Just utility programs Includes codes and standards and Programs
8. Of the following list of budget allocations, which best represents the $ allocation between impact & process
evaluations: 15% Impact / 75% Process / 10% Market Research 30% Impact / 50% Process / 20% Market Research 50% Impact / 30% Process / 20% Market Research 75% Impact / 15% Process / 10% Market Research ___% Impact / ___% Process / __ % Market Research (Provide your own) Not Sure
9. How does the total budget for EM&V activities compare to the dollars spent each year designing and
delivering EE programs? (Check one) less than 1% 1% to 2% 2% to 3% 3% to 5% Greater than 5% Please provide a reference for this number if available
10. How frequently is the same or similar utility program design run that has been run over consecutive
program years evaluated? (Check one) Less than 1-year (specify) Annually (once a year) > 1 year (specify) Please provide reference
38
Evaluation Planning Process & Choice of Evaluation Methods
11. Please rank the importance of the following types of evaluations in your job? (Please use the ranking scale for each group or indicate Don’t Know (DK))
Impact Process Benefit/Cost Analysis Market effects/Market transformation
12. Please rank the importance of the types of evaluations in your job in the future. (Please use the ranking
scale for each group or indicate Don’t Know (DK)) NO CHANGE Impact Process Benefit Cost Analysis Market effects/Market transformation
13. Who sets the precision, accuracy, or robustness requirements for developing program savings estimates?
(Check all that apply) Requirements adopted by Oversight Body and codified in protocols Requirements set by Program Administrators on a case by case basis Requirements set by the evaluator after winning the contract based on professional judgment, prior experience and/or budget constraints Please describe how the requirements are set or if these requirements are required to be disclosed in a final program savings report
14. Who sets the precision, accuracy, or robustness requirements for developing portfolio savings estimates?
(Check all that apply) Requirements adopted by Regulator or Oversight Body and codified in EM&V protocols Requirements set by Program Administrators on a case by case basis Requirements set by the evaluator after winning the contract based on professional judgment, prior experience and/or budget constraints No requirements are explicitly set in my state Don't Know Please describe how the requirements are set or if these requirements are required to be disclosed in portfolio savings reports
Load Impact Methods
15. Please rank the importance to you in your job of reviewing information on the following parameters that are or can be a part of load impact evaluations of EE programs in your state? (Please use the ranking scale for each group or indicate Don’t Know (DK))
Ex-ante and Ex-Post Gross Program Savings impacts Ex-ante and Ex-Post Net Program Savings Impacts (with or without spillover effects) Market effects (savings or price impacts for non participants caused by utility programs) Attribution of observed savings to specific programs or causal factors Impact of new building Codes and appliance standards on savings from utility programs Impacts of Programs on End User behavior and Energy consumption Ex-ante and Ex-Post Measure Life and Savings Persistence Other - Please List:
39
16. Please rank the importance of the same parameters in the future. (Please use the ranking scale for each group or indicate Don’t Know (DK))
NO CHANGE Ex-ante and Ex-Post Gross Program Savings impacts Ex-ante and Ex-Post Net Program Savings Impacts (with or without spillover effects) Market effects (savings or price impacts for non participants caused by utility programs) Attribution to specific programs or causal factors Impacts of new building Codes and appliance standards on savings from utility programs Impacts of Programs on End User behavior Ex-ante and Ex-Post Measure Life and Savings Persistence Other - Please List:
17. Is analysis of the level of free-ridership included in most program load impact evaluations?
Yes No Don't know Describe the types of methodologies or techniques used in your state here
18. Is analysis of the potential leakage of program subsidized or is more efficient products across state lines
included in EM&V studies? Yes No Don't know Describe methodology
19. Is analysis of spillover or market effects included in some or all program evaluations?
Yes No Don't know Describe methodologies used in your state to estimate these effects
20. Are ex ante program savings estimates at the measure level trued up based on ex post evaluation results
and published? Yes No Don't know If, yes, describe the process used to perform the true up
21. What is the average or typical time interval observed between the implementing a program and the
availability of program savings results in your state? Less than 6 months 6-12 months 12- 18 months More than 18 months
22. Is the time interval or delay between implementing a program and verifying the annual Peak demand
savings achieved by a program be shorter or longer than the time it takes to verify the annual energy savings from the program?
It takes a longer amount of time to develop and verify estimates of peak savings compared to energy savings It takes a shorter amount of time to develop and verify estimates of peak savings compared to energy savings The time it takes to develop and verify energy and peak savings estimates is about the same
40
Discussion Questions Evaluation Planning Process & Choice of Evaluation Methods
1. Which of the following best describes the process that is used to select the types of evaluation research to be carried out on EE programs or EE portfolios for the next year or multiple years in this region? (Check all that apply)
Evaluation planning is initiated and completed by evaluator or firm hired to conduct evaluation studies based on the objectives or goals from the program administrator Evaluation planning is initiated and completed by the program administrator in a public setting Evaluation planning is initiated and operated by the regulatory or oversight body who will ultimately approve or adopt a research plan & budget for the research List other ideas:
2. Please rate the relative importance of the following factors in selecting which types of evaluation studies
(process, impact, or market research) will receive the highest priority and or budget? (Check all that apply or add new issues)
Overall Budget Time to complete studies or regulatory deadlines Regulatory policies Quality of the Program Tracking Systems High degree of uncertainty in existing estimates of program savings or costs Interveners have raised concerns about a program's or measure's cost effectiveness Please provide additional answers or your rationale.
3. Please rate the relative importance of the following factors in selecting EM&V methods for evaluating
gross or net program savings? Overall Available Budget Time to complete studies to meet management or regulatory deadlines Regulatory policies Quality of the Existing Program Tracking Systems Availability of Evaluation Firms to complete the work on time Please provide additional answers or your rationale here
4. Are the precision or accuracy requirements the same or very similar for all ex post (or after the fact ) load
impact studies in your region or do they vary based upon characteristics of the market and confidence in existing estimates of measure or program savings?
Same or similar precision requirements for each program Vary by program and market
5. Please rank the relative importance of the following drivers of the decision to either use deemed savings
estimates for measures promoted by a program or to measure or verify the savings using some form of field visit or billing analysis
Quality and or age of the deemed savings estimates Extent that program design has changed from year to year Cost of performing some form of ex post analysis of savings Regulatory policies or guidelines with respect to how often a given program type must be evaluated Availability of Quality Evaluation Firms to perform the ex post load impact studies Other drivers that are important
6. How frequently is a baseline study conducted to estimate average energy use for key end uses targeted by
programs and used in deemed savings estimates? Please specify (in years) Don't Know
41
7. Who decides when such a study is needed? Utility or Program Administrator Regulatory Body or State Agency Master Evaluation firm or Technical Advisory Group Don't know or is not relevant to my state Other- List here
Load Impact Methods
8. What is the accepted definition of Gross Program Savings in this jurisdiction? Do evaluators need to use some form of control group to estimate changes in naturally occurring or baseline usage for non participants?
Please provide definition of gross savings
9. What is the accepted definition of Net Program Savings in this jurisdiction? Does it include Corrections for free riders only or does it also include participant and non participant spillover?
Please provide definition of net savings
10. Should the energy savings from utility programs be reported as net program savings, gross program savings, or both?
Report Net savings only Report Gross savings only Report Both Neither
11. Please identify the EM&V methods currently being used to estimate market effects or market
transformation induced by energy efficiency programs? ( Check all that apply) Surveys of Key Market Agents to determine what would have happened in absence of program Market assessments that document changes in market structure (prices, number of firms buying and selling efficient measures, etc) Analysis of sales or shipment data of more efficient products List other methods (e.g., conjoint analysis)
12. Does your state or utility add any incremental energy savings associated with market effects or changes
in the market structure induced by the program to the traditional estimates of net savings from participants in the same program? (Check one and provide reasons)
Yes and describe why No and describe why Not sure, it depends on quality of the market effects study
Emerging Policy Issues
13. What are the principal issues or uncertainties related to the use of evaluation results in making energy policy or resource decisions? ( check all that apply)
Use of inappropriate EM&V methods to measure program savings Lack of information about the uncertainties underlying program savings estimates Lack of quality control to ensure sampling procedures and data collection was unbiased Lack of understanding of the meaning or policy relevance of the savings results Lack of explanation of what was actually done in the evaluation Other thoughts or ideast:
14. From your perspective, is it a problem if utility program savings are being reported using gross savings
estimates in some states and using net savings estimates in others? Yes Maybe No Please describe
42
15. If it is a problem, what steps should be taken, if any, to encourage more consistent usage of program "savings" terms?
Develop national EM&V protocols and make them easily available to all states Develop a glossary of program savings terms and make sure states or utilities use these same terms when program savings are reported to EIA or other national agencies Other thoughts or ideas:
16. To what extent will the use of inconsistent definitions for estimating program savings at the state level
become a bigger problem if a National Energy Efficiency Resource Standard (EERS) is developed? Other thoughts or ideas about this question
17. Are you aware of any existing organizations working to resolve some of the problems mentioned above
related to the use of consistent terminology and methods? Yes- Please list the organization and the problems they are working on in next blue cell No
18. What other actions, if any, should NAPEE technical working group consider taking to increase the
quality or usefulness of EM&V studies at the state or national level? Please list actions and describe their potential benefits here Or indicate that no actions are recommended
A.2 Questions for State Regulators and Policymakers only
1. Do you read, review, or rely on the results of program savings estimates in the course of your work? Yes No If no, please skip to question 12
2. If yes, how would you characterize the usefulness of these studies?
3. If your ranking was less than 3 (useful) - Why are some or all the studies not useful?
Please provide reasons
4. How do you typically make use of information gathered from the review of an evaluation study that estimates program savings or cost effectiveness? (Check all that apply)
I use the EM&V results as basis for program funding decisions or policy changes I use the EM&V results as the basis for determining performance awards or penalties I use the EM&V results as a guide to developing better program designs I use EM&V results to resolve disputes about program impacts and benefits that arise between different stakeholder groups. I don't use the EM&V results directly, I expect program administrators or staff to review the reports and make the changes supported by the analysis Please List other uses
5. In your opinion, what could be done to make these studies more useful to you in making decisions?
More quality control and providing context for numbers Greater use of non-technical language in executive summary to convey key study results Require the reporting of key information in standardized terms Provide more education to policy makers on how program savings are estimated and why results may be relevant Other thoughts or ideas - Please List:
43
6. If you do not read/review/rely on results of program savings estimates, is there someone on your staff or organization who is in charge of reading these results and deciding what is relevant for you?
Yes No
7. If yes, ask for a referral to this person and we will attempt to interview them.
Please provide name(s) and contact information
8. How frequently have estimates of program savings from evaluation studies been disputed by stakeholders in a regulatory context?
Never Some times Many times Every time Don't know
9. If there are disputes, what is the process for reviewing those disputes and who ultimately decides on the
final savings number? Process is driven by regulatory rules and the Regulatory Commission makes the final decision Process is driven by utility project managers and they make the final call Process of review of program savings estimates only happens on rare occasions when a third party disputes a savings estimate because it has financial consequences to ratepayers Program Savings estimates are not disputed Don't know Describe Other Processes
A.3 National-level Experts Evaluation Planning Process & Choice of Evaluation Methods
1. Which of the following options represents the best process that you would recommend be used to select the types of evaluation research to be carried out on EE programs or EE portfolios in a given state or region ? (Check all that apply)
Evaluation planning is initiated and completed by the evaluator or evaluation firm hired to conduct evaluation studies based on the objectives or goals of the program administrator Evaluation planning is initiated and completed by the program administrator in a public setting Evaluation planning is initiated and operated by the regulatory or oversight body who will ultimately approve or adopt a research plan & budget for the research There is no best practice or method, depends on the circumstances No need for uniform evaluation process across all states
2. Please rate the relative importance of the following factors in selecting which types of evaluation studies
(process, impact, or market research) should receive the highest priority and/ or budget to evaluate the impacts and effectiveness of a given portfolio of efficiency programs? (Check all that apply or add new issues)
Overall EM&V Budget Time needed to complete studies or regulatory deadlines Regulatory policies Quality of the Program Tracking Systems High degree of uncertainty in existing estimates of program savings or costs Interveners have raised concerns about a program's or measure's cost effectiveness
44
3. Please rate the relative importance of the following factors in selecting EM&V methods for evaluating program savings for an existing energy efficiency program?
Overall Available EMV Budget Time to complete studies to meet management or regulatory deadlines Regulatory policies Quality of the Existing Program Tracking Systems Availability of Evaluation Firms to complete the work on time No answer
4. Is uncertainty reported in the load impact studies you have reviewed recently? Should precision or
accuracy be reported? Requirements adopted by Oversight Body and codified in protocols Requirements set by Program Administrators on a case by case basis Requirements set by the evaluator after winning the contract based on professional judgment, prior experience and/or budget constraints Generally there are no accuracy or precision requirements set before the work begins
5. Are the precision or accuracy requirements the same or very similar for all ex post (or after the fact ) load
impact studies in the states you have worked in or do the requirements vary based upon the characteristics of the market and level of confidence in existing estimates of measure or program savings?
Please provide answer and probe rationale 6. Please rank the relative importance of the following drivers of the decision to either use deemed savings
estimates for measures promoted by a program to report program savings or to measure or verify the Program savings achieved using some form of field visit or billing analysis after the program has been implemented
Quality and or age of the deemed savings estimates Extent that program design has changed from year to year Cost of performing some form of ex post analysis of savings Regulatory policies or guidelines with respect to how often a given program type must be evaluated Availability of Quality Evaluation Firms to perform the ex post load impact studies Deemed savings makes sense for most programs. Lots of uncertainty in field measurements, states should have latitude to decide when to use deemed savings
7. In your experience how frequently are baseline studies conducted to estimate the average energy use for
key end uses for customer segments targeted by programs as the basis for estimating program savings and or developing a deemed savings database?
Possibilities - Once a year, once every five years, once every decade, etc. Don't Know Too much variation among states to estimate an average frequency
8. Ideally, Who should decide when such a baseline study is needed? Possible answers include
Regulatory Body or State Agency Master Evaluation firm or Technical Advisory Group Program administrators Probably the regulatory body but in some cases the utility administrator
Load Impact Methods
9. What definition of Gross Program Savings should be used at the national level in the event a minimum energy efficiency resource standard is set for all states? Do evaluators need to use some form of control group to estimate changes in naturally occurring or baseline usage for non participants?
Please answer the question here and provide a citation or reference to definition of gross savings you prefer if you do not want to type it out here
45
10. What definition of Net Program Savings should be used at the national level in the event energy efficiency resource standards are set for all states via national legislation? Should it include adjustments or methods to separate out free riders only or does the definition also need to include the possibility of savings from participant and non participant spillover?
Please answer the question and provide a citation or reference to definition of net savings you prefer
11. Should the energy savings from utility programs be reported as net program savings, gross program savings, or both?
12. Please identify the EM&V methods currently being used to estimate market effects or market
transformation induced by energy efficiency programs? Are these methods adequate to develop reasonable estimates of net program savings?
Surveys of Key Market Agents to determine what would have happened in absence of program Market assessments that document changes in market structure (prices, number of firms buying and selling efficient measures, etc) Analysis of sales or shipment data of more efficient products List other methods (e.g., conjoint analysis) No answer
13. Should states or utilities add any incremental energy savings estimated to occur as a result of market
effects or changes in the market structure induced by EE programs, to the more traditional estimates of net savings from participants in the same program?
Yes and describe why No and describe why Not sure, it depends on quality of the market effects study
Emerging Policy Issues
14. What are the principal issues or uncertainties related to the use of evaluation results in making energy policy or resource decisions? ( check all that apply)
Use of inappropriate EM&V methods to measure program savings Lack of information about the uncertainties underlying point estimates of program savings Lack of quality control to ensure sampling procedures and data collection was unbiased Lack of understanding of the meaning or policy relevance of the program savings results Lack of explanation of what was actually done in the evaluation Little discussion of Sampling Bias and or Measurement Error No opinions
15. From your perspective, is it likely to be a problem in the future if utility program savings are being
reported using gross savings definitions in some states and using net savings definitions in others? Yes and why Maybe and why No and why Probe rationale
16. If it is a problem, what steps should be taken, if any, to encourage more consistent usage of program
"savings" terms?
17. To what extent do evaluation approaches and processes need to change or evolve to meet the changing needs of new EM&V users, the increasing level of program maturity, and the current emphasis on estimating the GHG impacts of energy efficiency programs?
46
Usefulness of Current Evaluation Studies that support Energy Efficiency Programs 18. In your opinion, what should be done to make Energy Efficiency evaluation studies more useful to
decision makers or policy makers? More quality control and providing context for numbers Greater use of non-technical language in executive summary to convey key study results Require the reporting of key information in standardized terms Provide more education to policy makers on how program savings are estimated and why results may be relevant no answer-
19. How frequently have estimates of program savings from evaluation studies been disputed by stakeholders
in a regulatory context where you were working? Never Some times Many times Every time Don't know
20. If there are disputes, what is the best or recommended process for reviewing those disputes and who
ultimately decides on the final program savings number(s)?
21. Are you aware of any existing organizations working to resolve some of the problems mentioned above related to the use of consistent terminology and methods in evaluating the load impacts of Energy Efficiency programs?
Yes- Please list the organization and the problems they are working on 22. What other actions, if any, should EM&V technical work group consider taking to increase the quality,
effectiveness, or usefulness of EM&V studies at the state or national level?
47
Appendix B. Description of recommended EM&V activities
Appendix B includes a more detailed description for several of the recommended EM&V activities and includes initial responses to several key questions.
What is the near and long term objective of the activity? How is success defined and assessed?
Who is the target audience for each activity? What strategies will be used to effectively reach actors in each of these audiences?
What strategies will be used to convince these actors to actually use and incorporate the information provided through these EM&V activities in their work?
EM&V Activity area: Improving methods used to estimate energy efficiency program savings Project 1: New Methods of estimating program savings including behavioral dimensions and long-term impacts
Objective: Support efforts to improve evaluation methods used for estimating long-term energy savings by integrating the physical and human behavior impacts of programs on long term energy use
Brief Description: Develop a new method or set of methods for estimating total program savings that includes both the near term effects of the installation of EE technologies or systems and the longer-term behavioral impacts caused by the EE program. This can include user issues, such as changes in behavior and long-term maintenance induced by the program, as well as broader market messaging and impacts associated with programs. This effort should be integrated with other state or federal efforts around efficiency and behavior research, as well as long-term growth in energy service demand.
Project 2: Develop and disseminate “best practices” guides and case studies on evaluation planning and methods for energy efficiency programs
Objective: Improve the accessibility and reduce the transaction costs currently associated with finding descriptions of state-of-the art evaluation methods, methods used to incorporate program savings into energy consumption forecasts, and the processes used to develop portfolio level evaluation plans.
Brief Description: Develop and share best practices guides and case studies on evaluation methods and planning through webinars and regional seminar series with state, utility, regulatory staff, and industry. Possible guide topics include:
a. A compendium of methods used to develop forecasts of baseline energy usage (e.g., projected energy use in the absence of programs) under different regulatory and market
48
conditions.
b. A review of “best practice” processes used to verify the load impact results of EE programs.
c. A compendium of approaches used to structure EE evaluation planning processes designed to achieve specific energy policy objectives. The compendium could also include the identification of best practices in evaluation planning and the development of a standard list of twenty questions to be asked and answered during an evaluation planning review process.
d. A document that reviews and references the methods and criteria used to select ex-ante measure specific savings values (deemed or stipulated savings).
e. A maintained database of standardized ex-ante savings estimates for EE measures in an accessible national data base format. This database can include regional values with variations in weather, common practices, baselines, etc.
f. A document that reviews the different methods used to estimates net savings from EE programs with inclusion of different environmental or market factors, such as free drivers, free riders, snap-back, and behavioral effects; and
g. A document that reviews the methods used to estimate gross and net GHG impacts given a set of EE program energy savings impacts.
Many of the deliverables defined for this second project could be combined into an activity to design and implement a national, searchable data base that provides access to EE evaluation plans, tools and reports developed and filed by program administrators (or independent evaluators) in various states over the previous two years. Existing public websites (e.g. CALMAC, NYSERDA, CEE) focus on compiling completed EM&V studies. This activity could be modeled after those websites but would focus on providing visitors with searchable access to evaluation plans, tools and reports for EE programs or portfolios which were developed under different regulatory conditions (e.g., jurisdictions with annual energy savings goals, shareholder incentives).
EM&V Activity area: Improving quality control and accuracy of reported program savings estimates Project 3: Develop and disseminate “best practices” guide in quality control and accuracy in reporting program savings Objective: Support efforts to widely disseminate best practices in quality control and accuracy in reporting program saving results.
Brief Description: Develop and disseminate a best practices guide that reviews the range of processes and techniques used to improve quality control in savings reports. This would include recommendations on the best practices in EM&V data collection, quality control, and analysis. The guide would review current best practices used to transform draft load impact reports into final reports and provide details on the processes used to estimate the level of uncertainty in
49
program load impact estimates under different evaluation methods. This guide could also include a standardized list of ‘twenty questions’ that should be asked and answered for any report indicating estimates of program energy and/or demand savings. The guide could be distributed through webinars and/or regional seminars to state, utility, and regulatory staff.
EM&V Activity Area: Improve the Consistency in Program Energy Savings Terms and Reporting Project 4: Standardized program savings reporting format Objectives: Promote the use of voluntary standardized program savings formats that would reduce confusion and increase the consistency of savings being reported to state regulatory commissions, EIA, EPA and DOE.
Brief Description: Develop a short program savings reporting format (i.e. one page) for all states and regions to use in reporting program savings and seek voluntary adoption via regional workshops or other strategies. This format would include instructions, definitions, and examples.
Project 5: Glossary of savings terms
Brief Description: Develop a glossary of standardized EM&V and measure, program, or portfolio terms for voluntary use by all states. Include annex with standardized terms targeted to region-specific needs with goal of having all states in a targeted region(s) to adopt these terms on a voluntary basis. Start with draft document based on references and input from industry participants. Include annex with one or more definitions (with examples) of net savings (e.g., narrow net, market net) for different program or regulatory objectives.