b o n n e v i l l e p o w e r a d m i n i s t r a t i o n draft: methods for evaluating residential...

B O N N E V I L L E P O W E R A D M I N I S T R A T I O N

DRAFT:Methods for Evaluating

Residential Behavior-based Programs

RTF PresentationJanuary 5, 2010

Lauren Gage ([email protected])Bobbi Wilhelm ([email protected])


DRAFT for RTF DISCUSSION

Goals for RTF Meeting

Introduce methods developed

Gain feedback from RTF members

Bring final methods to RTF in February for action/ approval



Background• Residential behavioral-based programs are gaining popularity and promise

significant savings potential• Multiple programmatic approaches are being tested throughout the country

• In-home feedback devices• Energy benchmarking information • Others (e.g., community-based programs).

• Some research was conducted in the early 1980s on similar topics, but there is a lack of recent focused research in behavioral-based energy programs, particularly research designed to:

• Quantify savings achieved • Measure the persistence of that savings.



Process A group of Northwest utilities and energy organizations have held

multiple meetings over the last several months to discuss regional evaluation strategies for behavior-based energy programs.

– Puget Sound Energy, BPA, Seattle City Light, Snohomish County PUD, Energy Trust of Oregon, and Eugene Water & Electric Board.

The group agreed on the need for standard methods for evaluation of behavior-based conservation programs within the Pacific Northwest.



Outcomes and Caveats These are recommended guidelines and/or methods for

evaluating behavioral programs for Pacific Northwest utilities.

They consider the research completed to date and are designed to evolve as more studies are completed.

They should not be viewed as the only acceptable approaches to estimating savings.

Once reviewed and approved by the RTF, they would be an agreed-upon approach that provides sufficient rigor to estimate savings.



Residential Behavior-Based Programs (RBBP) Definitions

Programs designed to save energy through:– Changes in behavior - e.g., turning off lights, setting thermostats– Increasing investments in energy-efficiency measures

Program Examples:– Energy benchmarking compares participant consumption to historical consumption or to

peers. • OPower’s Home Energy Reports, Snohomish Energy Challenge)

– Feedback devices use monitoring or metering devices to provide information on instantaneous demand or consumption over time at the whole-house or end-use level

• Blue-line, TED

– Others include: Information and training programs, Schools-based programs and marketing/community-based programs

Proposed evaluation methods are most relevant for energy benchmarking and feedback device programs due to large sample sizes and ease of finding a comparison group


The Power Act and RBBP Historically, the Council has excluded measures and practices that reduced the level of

service or utility (in economic terms) provided to consumers by the current measure or practice.

– 839a(3). "Conservation" means any reduction in electric power consumption as a result of increases in the efficiency of energy use, production, or distribution. [Northwest Power Act, §3(3), 94 Stat. 2698.]

– Savings from lowering thermostats for space heating have not been considered conservation under the Act

– Savings from lowering the thermostat on water heaters from 140 F or 130 F to 120 F have been.

Rationale for behavioral change programs as “utility neutral”, or non-sacrificial?– Persistent savings are unlikely to be sacrifices– Programs are asking customers to reduce behavior when it does not change utility (e.g.,

turning off lights or thermostat down when not in the room)– Recent interest in sustainability has created utility for reducing energy consumption,

particularly enabled through technology or information



Contents of the Methods

Methods contain the following key areas:

– Data Collection, Sampling and Data Quality

– Estimating Overall Program Savings

– Attribution of Savings

• Quantify energy savings from utility-sponsored measure installations

– Determining Persistence


Data Collection, Sampling and Data Quality



Data Needs Billing data: Daily (or monthly or bi-monthly) kWh and/or therm data for

comparison and participant groups - during the study period, at least one year prior to the study period, after participation

Weather data: HDD and CDD for all months/days used in the billing analysis Participation group information: Participant or comparison group assignments,

frequency of participation, participation start date Energy-efficiency measure installation: Utility-sponsored installations, Optional

survey data on other measure installation for a sub-sample Household and building characteristics data

The Evaluation Methods paper addresses best practices approaches for preparing data for analysis (data cleaning).



Identification and Selection of Comparison Group

Essential to have a comparison group to control for secular trends affecting energy consumption not related to the program (e.g., price, economy).

Relative to the participant group, the comparison group should be similar in the following ways: – Baseline energy consumption (heating, cooling and baseload)– Home characteristics– Heating fuels– Household size – Motivations or demographics



Sample Sizes Methods encourage organizations to obtain a sample large enough to

provide a savings estimate with plus or minus 5% at a 95% confidence level. Methods need more input on:

– Default assumptions• 1% change on base usage of 10,000kWh = 100 kWh, base usage standard deviation of

2,000 kWh

– What needs to be measured (appropriate means tests)• Savings different from 0 relatively small sample size• Mean savings are significantly different from a comparable comparison group

medium-sized sample• Mean savings are plus or minus 5% of population mean HUGE sample size

– Population Correction Factors– Distribution of savings


Estimating Program Savings



Estimating Savings Recommended approaches utilize billing analyses that include a mix of

participant and comparison data.

Program savings should, at a minimum, be measured annually.

Three models are defined in the methods:

– Simplified Annual Model

– Monthly Panel Model

– More Sophisticated Approaches



Simplified Annual Model Model estimates the weather-normalized difference in the change in annual

consumption between participant and comparison group. Its value is in its simplicity and ability to tell a high-level story to policy-makers.

This model assumes that the difference of differences is due to the program and that the comparison group is similar to the participant group.

General calculation approach:

– Calculate weather-normalized Normalized Annual Consumption (NAC) for pre- and post- periods for participant and comparison groups

– Calculate Difference in NAC (DNAC) as the PreNAC – PostNAC for participant and comparison groups

– Calculate Savings = DNACparticipant - DNACcomparison



Simplified Annual Model, 2 The advantages of this model are:

– Minimal data requirements– Descriptive results are easier to analyze and identify sample/research issues– Base loads, weather sensitive loads and reference temperatures can be estimated– Very easy to estimate with available software programs

The drawbacks of this model are: – Requires a year of data– NAC has error in its estimate – Cannot give short-term results (e.g. impacts after a month)– Heating loads often contain seasonal non-heating loads– More difficult to incorporate data from participants that installed multiple measures

over time.



Monthly Panel Model

This model uses monthly billing data and includes several descriptive variables and weather adjustments.

Monthly energy usage as a function of weather, participation, price, household chars, and participant characteristics, where:

– Monthly energy use: Energy used per day during month for each household– Weather: HDD and CDD for each observed time period for the closest weather station– Participation: Indicator variable equal to one if the household is a participant or zero

otherwise. This variable should be included alone and interacted with the Period (delete, they can read this)

– Period: Indicator variable equal to one if reading is in testing period (post-period) or zero otherwise.

– Household characteristics: May include square footage, age of home, value of home, measure installation



Monthly Panel Model, 2 The advantages of this model are:

– Can include all billing data– Can more easily incorporate data from participants that installed multiple measures over

time– No error associated with estimation of NAC– Does not require a full year’s worth of data for entire sample– Relatively easy to estimate and a one-step procedure

The drawbacks of this model are:– Typically estimated using average weather effects– Coefficients of some variables are often not very descriptive or intuitive– Inappropriate research designs often used (e.g. one model estimated instead of more) – Subgroups/Strata do not get sufficient scrutiny



More Sophisticated ApproachesEach of the models can be expanded to include additional variables to improve the model’s explanatory power Fixed Effects Model– Model allows the intercept to vary for each house and each month observed in the data.

The fixed-effects model essentially allows each household to serve as its own control, by making comparisons within individuals and averaging those differences among individuals.

Enhanced Annual Model – This approach can be used to provide greater detail on the factors that are affecting

energy use, e.g., savings from specific measures, what buildings characteristics are correlated with energy consumption and savings, how measures interact, and if the engineering estimates are consistent with actual savings.


Savings Attribution



Savings Attribution between Behavior and Energy-Efficiency

Measure Savings The value of estimating the allocation of savings between behavioral and

energy-efficiency measure is to:– Understand changes in investment behavior in participants– Allow organizations to avoid double-counting of savings captured in program

tracking– Facilitate understanding of persistence estimates of the program

Three approaches defined in the methods:

– Calculate energy-efficiency realized savings (prorate savings)

– Dummy Variables for Measure Categories

– Elimination from the Model


Calculate Energy-efficiency Realized Savings (Proration)

Match participant and comparison group information with utility rebate information. Calculate a pro-rated savings (accounting for timing of installation versus RBBP program participation). Discount the savings obtained in the model by the sum of savings from participation.

Advantages

– Keeps every observation in the model

– Allows comparison of program participation rates Disadvantages

– Cumbersome and time-consuming process.




Dummy Variables for Measure Categories

Uses models described above to estimate the average savings for the measure category to be estimated. Assuming no correlation, the estimated coefficient for the dummy variable can then be directly interpreted as the change in consumption associated with that measure.

Requires thoughtful definition of measure categories and clearly delineated pre- and post participation billing periods

Dummy variables can be used in tandem with engineering estimates and should be used to support any direct estimation of realization rates.



Elimination from the Model

Another method is to run a model with only participant and comparison group homes that did not participate in other utility-sponsored programs during experiment.

Advantages

– Simple – no pro-rated savings and no double-counting. Disadvantage

– Excludes an important sub-set of the sample because those participants who installed equipment may be more likely to change energy efficiency behaviors and therefore it would underestimate the behavioral effect.



Distinguishing between Behavioral and Energy-efficiency Measures Installed

Outside of Utility Programs Valuable to understand various behaviors that affect energy

consumption (e.g., non-incented equipment, turning off lights, appliances and electronics when not in use, thermostat setting) due to likely persistence differences

Above-methods allocate savings between utility-sponsored equipment savings and “all other savings”

Dividing the savings further between non-sponsored equipment and behavioral is likely to require a considerable amount of research and time.

This is a promising area of research that will require more thought and planning before a standard method is developed.

Jill Steiner

Good example of how the methods/guidelines can and should evolve over time.


Persistence



Persistence Persistence is critical to understanding the role RBBPs will play as a resource

for meeting/managing future demand for energy.

At this point it is unclear how long expected behavior modifications, and the savings associated with those behavior modifications will last.

Approaches for measuring persistence defined in the methods:

– Utilizing the same models over time

– Sample adjustment

– Addition of survey data



Persistence:Same Models over Time

Impact evaluations will be conducted one year after program launch, and each year following.

Differences in the year-to-year savings of the participant group will indicate how the savings change over time.

This type of analysis can continue on an annual basis until no measurable impacts between participant and comparison groups can be estimated.



Persistence:Sample Adjustment

Discontinue participation in the behavior-based program for a portion of the participant group after a year of delivery.

Keep large portion of participants on the program and make no changes the comparison group. – DNACparticipants – DNACcomparisongroup. This comparison will, like #1 above,

show the savings associated with ongoing participation. – DNACdiscontinuedparticipants – DNACcomparisongroup: This comparison will

show the savings associated with receiving the program services for a year. – DNACParticipants – DNACdiscontinuedparticipants. This comparison will show if

receiving the program services for an extended period of time will help maintain or increase the program’s effect.



Addition of Survey Data Surveys of the participant and comparison groups gives information on why an

impact is occurring (e.g., energy using behavior and its longer term investment behavior in regards to energy efficiency, and how it was influenced by the program).

Survey results will also provide guidance on how program marketing and implementation could be enhanced.



Q&A

b o n n e v i l l e p o w e r a d m i n i s t r a t i o n draft: methods for evaluating residential...

Documents

t i o n draft

b o n n e v

e p o w e r

behaviorbased energy

behavioral programs

schoolsbased programs

based conservation programs

training programs