b o n n e v i l l e p o w e r a d m i n i s t r a t i o n draft: methods for evaluating residential...
Post on 13-Dec-2015
215 Views
Preview:
TRANSCRIPT
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT:Methods for Evaluating
Residential Behavior-based Programs
RTF PresentationJanuary 5, 2010
Lauren Gage (lsmgage@bpa.gov)Bobbi Wilhelm (bobette.wilhelm@pse.com)
Slide 2
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Goals for RTF Meeting
Introduce methods developed
Gain feedback from RTF members
Bring final methods to RTF in February for action/ approval
Slide 3
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Background• Residential behavioral-based programs are gaining popularity and promise
significant savings potential• Multiple programmatic approaches are being tested throughout the country
• In-home feedback devices• Energy benchmarking information • Others (e.g., community-based programs).
• Some research was conducted in the early 1980s on similar topics, but there is a lack of recent focused research in behavioral-based energy programs, particularly research designed to:
• Quantify savings achieved • Measure the persistence of that savings.
Slide 4
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Process A group of Northwest utilities and energy organizations have held
multiple meetings over the last several months to discuss regional evaluation strategies for behavior-based energy programs.
– Puget Sound Energy, BPA, Seattle City Light, Snohomish County PUD, Energy Trust of Oregon, and Eugene Water & Electric Board.
The group agreed on the need for standard methods for evaluation of behavior-based conservation programs within the Pacific Northwest.
Slide 5
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Outcomes and Caveats These are recommended guidelines and/or methods for
evaluating behavioral programs for Pacific Northwest utilities.
They consider the research completed to date and are designed to evolve as more studies are completed.
They should not be viewed as the only acceptable approaches to estimating savings.
Once reviewed and approved by the RTF, they would be an agreed-upon approach that provides sufficient rigor to estimate savings.
Slide 6
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Residential Behavior-Based Programs (RBBP) Definitions
Programs designed to save energy through:– Changes in behavior - e.g., turning off lights, setting thermostats– Increasing investments in energy-efficiency measures
Program Examples:– Energy benchmarking compares participant consumption to historical consumption or to
peers. • OPower’s Home Energy Reports, Snohomish Energy Challenge)
– Feedback devices use monitoring or metering devices to provide information on instantaneous demand or consumption over time at the whole-house or end-use level
• Blue-line, TED
– Others include: Information and training programs, Schools-based programs and marketing/community-based programs
Proposed evaluation methods are most relevant for energy benchmarking and feedback device programs due to large sample sizes and ease of finding a comparison group
Slide 7
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
The Power Act and RBBP Historically, the Council has excluded measures and practices that reduced the level of
service or utility (in economic terms) provided to consumers by the current measure or practice.
– 839a(3). "Conservation" means any reduction in electric power consumption as a result of increases in the efficiency of energy use, production, or distribution. [Northwest Power Act, §3(3), 94 Stat. 2698.]
– Savings from lowering thermostats for space heating have not been considered conservation under the Act
– Savings from lowering the thermostat on water heaters from 140 F or 130 F to 120 F have been.
Rationale for behavioral change programs as “utility neutral”, or non-sacrificial?– Persistent savings are unlikely to be sacrifices– Programs are asking customers to reduce behavior when it does not change utility (e.g.,
turning off lights or thermostat down when not in the room)– Recent interest in sustainability has created utility for reducing energy consumption,
particularly enabled through technology or information
Slide 8
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Contents of the Methods
Methods contain the following key areas:
– Data Collection, Sampling and Data Quality
– Estimating Overall Program Savings
– Attribution of Savings
• Quantify energy savings from utility-sponsored measure installations
– Determining Persistence
Slide 9
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
Data Collection, Sampling and Data Quality
Slide 10
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Data Needs Billing data: Daily (or monthly or bi-monthly) kWh and/or therm data for
comparison and participant groups - during the study period, at least one year prior to the study period, after participation
Weather data: HDD and CDD for all months/days used in the billing analysis Participation group information: Participant or comparison group assignments,
frequency of participation, participation start date Energy-efficiency measure installation: Utility-sponsored installations, Optional
survey data on other measure installation for a sub-sample Household and building characteristics data
The Evaluation Methods paper addresses best practices approaches for preparing data for analysis (data cleaning).
Slide 11
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Identification and Selection of Comparison Group
Essential to have a comparison group to control for secular trends affecting energy consumption not related to the program (e.g., price, economy).
Relative to the participant group, the comparison group should be similar in the following ways: – Baseline energy consumption (heating, cooling and baseload)– Home characteristics– Heating fuels– Household size – Motivations or demographics
Slide 12
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Sample Sizes Methods encourage organizations to obtain a sample large enough to
provide a savings estimate with plus or minus 5% at a 95% confidence level. Methods need more input on:
– Default assumptions• 1% change on base usage of 10,000kWh = 100 kWh, base usage standard deviation of
2,000 kWh
– What needs to be measured (appropriate means tests)• Savings different from 0 relatively small sample size• Mean savings are significantly different from a comparable comparison group
medium-sized sample• Mean savings are plus or minus 5% of population mean HUGE sample size
– Population Correction Factors– Distribution of savings
Slide 13
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
Estimating Program Savings
Slide 14
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Estimating Savings Recommended approaches utilize billing analyses that include a mix of
participant and comparison data.
Program savings should, at a minimum, be measured annually.
Three models are defined in the methods:
– Simplified Annual Model
– Monthly Panel Model
– More Sophisticated Approaches
Slide 15
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Simplified Annual Model Model estimates the weather-normalized difference in the change in annual
consumption between participant and comparison group. Its value is in its simplicity and ability to tell a high-level story to policy-makers.
This model assumes that the difference of differences is due to the program and that the comparison group is similar to the participant group.
General calculation approach:
– Calculate weather-normalized Normalized Annual Consumption (NAC) for pre- and post- periods for participant and comparison groups
– Calculate Difference in NAC (DNAC) as the PreNAC – PostNAC for participant and comparison groups
– Calculate Savings = DNACparticipant - DNACcomparison
Slide 16
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Simplified Annual Model, 2 The advantages of this model are:
– Minimal data requirements– Descriptive results are easier to analyze and identify sample/research issues– Base loads, weather sensitive loads and reference temperatures can be estimated– Very easy to estimate with available software programs
The drawbacks of this model are: – Requires a year of data– NAC has error in its estimate – Cannot give short-term results (e.g. impacts after a month)– Heating loads often contain seasonal non-heating loads– More difficult to incorporate data from participants that installed multiple measures
over time.
Slide 17
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Monthly Panel Model
This model uses monthly billing data and includes several descriptive variables and weather adjustments.
Monthly energy usage as a function of weather, participation, price, household chars, and participant characteristics, where:
– Monthly energy use: Energy used per day during month for each household– Weather: HDD and CDD for each observed time period for the closest weather station– Participation: Indicator variable equal to one if the household is a participant or zero
otherwise. This variable should be included alone and interacted with the Period (delete, they can read this)
– Period: Indicator variable equal to one if reading is in testing period (post-period) or zero otherwise.
– Household characteristics: May include square footage, age of home, value of home, measure installation
Slide 18
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Monthly Panel Model, 2 The advantages of this model are:
– Can include all billing data– Can more easily incorporate data from participants that installed multiple measures over
time– No error associated with estimation of NAC– Does not require a full year’s worth of data for entire sample– Relatively easy to estimate and a one-step procedure
The drawbacks of this model are:– Typically estimated using average weather effects– Coefficients of some variables are often not very descriptive or intuitive– Inappropriate research designs often used (e.g. one model estimated instead of more) – Subgroups/Strata do not get sufficient scrutiny
Slide 19
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
More Sophisticated ApproachesEach of the models can be expanded to include additional variables to improve the model’s explanatory power Fixed Effects Model– Model allows the intercept to vary for each house and each month observed in the data.
The fixed-effects model essentially allows each household to serve as its own control, by making comparisons within individuals and averaging those differences among individuals.
Enhanced Annual Model – This approach can be used to provide greater detail on the factors that are affecting
energy use, e.g., savings from specific measures, what buildings characteristics are correlated with energy consumption and savings, how measures interact, and if the engineering estimates are consistent with actual savings.
Slide 20
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
Savings Attribution
Slide 21
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Savings Attribution between Behavior and Energy-Efficiency
Measure Savings The value of estimating the allocation of savings between behavioral and
energy-efficiency measure is to:– Understand changes in investment behavior in participants– Allow organizations to avoid double-counting of savings captured in program
tracking– Facilitate understanding of persistence estimates of the program
Three approaches defined in the methods:
– Calculate energy-efficiency realized savings (prorate savings)
– Dummy Variables for Measure Categories
– Elimination from the Model
Slide 22
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
Calculate Energy-efficiency Realized Savings (Proration)
Match participant and comparison group information with utility rebate information. Calculate a pro-rated savings (accounting for timing of installation versus RBBP program participation). Discount the savings obtained in the model by the sum of savings from participation.
Advantages
– Keeps every observation in the model
– Allows comparison of program participation rates Disadvantages
– Cumbersome and time-consuming process.
DRAFT for RTF DISCUSSION
Slide 23
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Dummy Variables for Measure Categories
Uses models described above to estimate the average savings for the measure category to be estimated. Assuming no correlation, the estimated coefficient for the dummy variable can then be directly interpreted as the change in consumption associated with that measure.
Requires thoughtful definition of measure categories and clearly delineated pre- and post participation billing periods
Dummy variables can be used in tandem with engineering estimates and should be used to support any direct estimation of realization rates.
Slide 24
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Elimination from the Model
Another method is to run a model with only participant and comparison group homes that did not participate in other utility-sponsored programs during experiment.
Advantages
– Simple – no pro-rated savings and no double-counting. Disadvantage
– Excludes an important sub-set of the sample because those participants who installed equipment may be more likely to change energy efficiency behaviors and therefore it would underestimate the behavioral effect.
Slide 25
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Distinguishing between Behavioral and Energy-efficiency Measures Installed
Outside of Utility Programs Valuable to understand various behaviors that affect energy
consumption (e.g., non-incented equipment, turning off lights, appliances and electronics when not in use, thermostat setting) due to likely persistence differences
Above-methods allocate savings between utility-sponsored equipment savings and “all other savings”
Dividing the savings further between non-sponsored equipment and behavioral is likely to require a considerable amount of research and time.
This is a promising area of research that will require more thought and planning before a standard method is developed.
Slide 26
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
Persistence
Slide 27
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Persistence Persistence is critical to understanding the role RBBPs will play as a resource
for meeting/managing future demand for energy.
At this point it is unclear how long expected behavior modifications, and the savings associated with those behavior modifications will last.
Approaches for measuring persistence defined in the methods:
– Utilizing the same models over time
– Sample adjustment
– Addition of survey data
Slide 28
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Persistence:Same Models over Time
Impact evaluations will be conducted one year after program launch, and each year following.
Differences in the year-to-year savings of the participant group will indicate how the savings change over time.
This type of analysis can continue on an annual basis until no measurable impacts between participant and comparison groups can be estimated.
Slide 29
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Persistence:Sample Adjustment
Discontinue participation in the behavior-based program for a portion of the participant group after a year of delivery.
Keep large portion of participants on the program and make no changes the comparison group. – DNACparticipants – DNACcomparisongroup. This comparison will, like #1 above,
show the savings associated with ongoing participation. – DNACdiscontinuedparticipants – DNACcomparisongroup: This comparison will
show the savings associated with receiving the program services for a year. – DNACParticipants – DNACdiscontinuedparticipants. This comparison will show if
receiving the program services for an extended period of time will help maintain or increase the program’s effect.
Slide 30
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Addition of Survey Data Surveys of the participant and comparison groups gives information on why an
impact is occurring (e.g., energy using behavior and its longer term investment behavior in regards to energy efficiency, and how it was influenced by the program).
Survey results will also provide guidance on how program marketing and implementation could be enhanced.
Slide 31
B O N N E V I L L E P O W E R A D M I N I S T R A T I O N
DRAFT for RTF DISCUSSION
Q&A
top related