BPA EnergySmart Grocer Program Evaluation

Deborah Swarts, Josh Arnold, Mary Klos, Roger Hill, Kevin Cooney, and

Gary Cullen

Summit Blue Consulting

Program Description

• A BPA third-party program that promotes efficiency improvements in commercial refrigeration. Customers include: o National chain supermarketso Independent grocerso Corner grocerso Specialty shops (ice cream, butchers, etc.)o Regional chain grocerso Convenience storeso Floristso Liquor storeso Restaurants

2

Measure Categories

• Program Measures fall into three categories:o Deemed savings: The costs and savings for these

measures have been deemed by the Regional Technical Forum (RTF).

o Deemed calculated savings: The RTF has reviewed the EnergySmart Grocer software and determined that given reliable inputs, the software appropriately calculates savings.

o Provisionally deemed savings: This designation means that the RTF has uncertainty in these measures and they require more research before a “deemed” status can be approved.

3

Measures Within Categories

• Deemed savings measureso Auto-closureso CFLs and general lightingo Gasketso PC Controls

• Deemed calculated savings measureso Energy efficient caseso Doorso Efficient compressor – low tempo Electronically Commutated Motorso Air cooled condensero Floating head controlso Refrigeration – VFD – Motors

• Provisionally deemed measureso Case lightingo Anti sweat controlso Night coverso Vending machine controls

4

Program Savings Information by Year

Jan 2007 - Sep

2007Oct 2007 - Sep

2008Oct 2008 -

March 2009Total

Participating Utilities 31 72 76 76

Audits Conducted 218 704 155 1,077

Number of Free Measures 62 1,022 580 1,664

Number of Rebated Measures 48 178 190 416

Installed Savings from Free Measures (kWh/year)

137,775 2,417,956 1,249,392 3,805,123

Installed Savings from Rebated Measures (kWh/year)

2,044,227 7,946,050 7,736,950 17,727,227

Incentive Costs $132,413 $689,034 $749,441 $1,570,888

Non-Incentive Costs $999,907 $3,058,575 $2,180,045 $6,238,527

5

Program Savings by Measure Type

6

Anti-Sweat Heat Controls, 3.1%

Auto-Closer, 6.6%

Case Lighting, 0.1%

Case Lighting - LED, 0.4%

CFLs, 7.1% Commercial RF/Freezer, 0.03%

Compressor/Condenser, 0.8%

ECM, 7.5%

Gaskets, 10.4%

General Lighting Retrofit, 21.6%

High Efficiency Cases, 12.3%

Doors, 0.03%

Night Covers, 3.5%

PC Controls, 1.5%

Refrigeration Controls, 24.1%

Vending Machine Controllers, 0.9%

VFDs, 0.1%

Program Savings Breakdown

Evaluation Objectives

• Assess program operation and barriers • Assess opportunities for application to other third-

party programs• Estimate measure and program realization rates

• Assess the strengths/weaknesses of two evaluation approaches:o Billing analysiso Engineering analysis

7

Process EvaluationGoals

• Develop a comprehensive understanding of the program

• Identify programmatic or organizational barriers

• Make recommendations for possible improvement

• Assess opportunities and barriers for potential future third-party program administration

8

Process EvaluationMethodology

• Review program materials, including logic model

• Interview program stakeholders

• Research commercial refrigeration programs for benchmarking and best practices

• Submit early findings reports to BPA

9

Process EvaluationStakeholder Interviews

• 36 Stakeholders Interviewed (February—April 2009)o Utility representativeso End-use customerso Trade Allieso PECI staffo BPA Energy Efficiency Representativeso BPA Program Staff

• Interviews included questions about program roles, communication, delivery, and satisfaction

10

Process Evaluation Major Findings

• Program found to be highly regarded by all parties, with some fine-tuning opportunities

• There is a stakeholder perception of a need to clarify roles and responsibilities

• Economic downturn has affected end-use customer uptake

• The program’s use of one-on-one and personal visits (I2I) are critical to program success

• Program success provides strong evidence to support adding more third-party programs

11

Process EvaluationRecommendations:

Roles and Responsibilities

• Review proposed logic model and discuss roles• Clarify communication channels among program

stakeholders• Engage PECI to address administrative

requirements• Review service agreements with utility

representatives

12

Process Evaluation Recommendations:

Marketing & Outreach

• Field Energy Analysts (FEAs) and Utility Representatives are the most trusted source for program information.

• To improve implementation even more:o Develop FEA presentation scriptso FEAs leverage local utility repso FEAs identify local success stories

• Create local case studies for different target markets

• Utilize the local utility names and contacts

13

Process Evaluation: Opportunities

Future Third-Party Initiatives• History and Vision of BPA

o Successful history of innovation provides favorable framework

• Structure of BPAo The wide geographic range of both the BPA structure and

member utilities along with the diverse population and socio-economic characteristics of the member utilities lend themselves to third-party initiatives

• Third-party implementer expertise with current program• Access to Research & Resources

o BPA’s internal expertise and access to other research organizations in the NW

• Human Resourceso BPA and member utilities have experienced staff 14

Process Evaluation: Recommendations

Future Third-Party Initiatives

• Engage BPA member utilities in a voluntary advisory group

• Fully Develop “New Initiatives Checklist”• Implement pilot phase with review prior to “open

enrollment”• Incorporate workplan development period into

contract to leverage third-party implementer expertise

15

Impact Evaluation Approaches

• Both an engineering and billing analysis are useful

• Comparative results from the two are similar

• Billing analysis:o Cannot assess small-scale impact measures o Can assess hard to separate multiple measures that

are installed togethero Lower cost

• On-site engineering analysis:o Can assess small impact loadso Some difficulty with hard to separate multiple

measureso On-site verification of installed measures

16

Billing Analysis Methodology

• Fixed-Effects Regression Approacho Pre-installation bills are compared to post-installation

billso Each customer serves as his or her own controlo Fixed effects control for differences between customers

• Statistically Adjusted Engineering Modelo Savings estimates from tracking system were usedo Coefficients represented realization rate on the savings

17

Billing Analysis Sample• Collecting billing data was a challenge

o Many utilities were involvedo Billing data was in a different format from each utility

• Had to balance cost of data collection with value of additional cases

• Final sample design covered 80% of savings by including 15 utilities

18

Stores

Measures

First-Year KWH

Savings

Utilities

All Participating Stores 466 1,296 11,048,855 66

Billing Analysis Sample 142 479 8,874,442 15

Billing Analysis Results• Three measures had both sufficient savings and a sufficient

number of cases to estimate statistically significant realization rates at the 90% confidence level.

19

0.60

0.80

1.00

1.20

1.40

1.60

Energy Efficient Lighting Energy Efficient Cases Floating Head Pressure Control

Rea

liza

tion

Rat

e

0.77

1.27

1.07

Engineering Analysis Methodology

• Assess the EnergySmart Grocer Software• Spot measurements combined with logging of

temperatures and current• Use of system data logs where available• Three weeks of monitoring for weather dependent

measures• Comparison of results with previous studies

commissioned by BPA• Specifications for installed systems used along

with measured data

20

Assessment of the EnergySmart Grocer

Software• Modeling logic is sound• Fundamental engineering principles with arrays of

DOE 2.1 simulations• Quality input data is utilized by the model• User friendly interface• Reasonable results based on review of model

inputs• The proprietary nature of the model algorithms

leaves uncertainty on how the estimates are derived

21

Program without sample

65%

Case Lighting0.1%

Condenser0.3%

ECMs3%

General Lighting

9%

High Efficiency Cases 8%

Refrigeration Controls

15%Sample, 35%

Engineering Evaluation Sample

• Goal was to focus on the largest savings in order to estimate program cost-effectiveness

• Ten largest sites by deemed calculated savings were chosen• Represented 35% of program savings and 50% of non-

deemed program savings• High efficiency cases and refrigeration controls made up the

largest portion of both the sample and non-deemed program savings

• Night covers and auto-closers were primary missing measures within the ten largest sites

22

Engineering Site Visit Information

Measures not monitored were verified during site visits

23

Type of Engineering Number Comments

Total Site Visits 10

Floating Head Pressure – visits 9 Obtained data from 7 systems

Electrically Commutated Motors 7 5 sites assessed

High Efficiency Cases 4

Site-Monitoring Performed

• Floating Head/Suction Pressure Controls and VFD Modulating Compressors and Condensers – nine siteso Pre-installation data not availableo Spot measurements of temperature and power

factorso Installed data loggers or acquired trend log data from

the building automation and control systemo Data collection covered minimum of 16 days

• Electronically Commutated Motors – seven siteso Pre-installation data not availableo On-site measurement of post-installation true powero Installed data loggerso Data collection covered minimum of 16 days

24

Site-Monitoring Performed (cont.)

• Anti-Sweat Heater Controls and Low ASHC Doors – two siteso Spot measurements used to verify which breakers

operated anti-sweat heaters, the supply voltage, and the range of switching

o Multi-week logging employed

• Long term weather data from nearest available station for weather dependent measures

• The data logging was primarily of single phase current, but with some three phase logging also performed for comparison purposes

• The spot measurements used Fluke 43B and Amprobe ACD-31P devices

25

Engineering Analysis Results

• Overall program savings corresponded well with predicted savings

• ECMs showed substantially higher savings than predicted

• High efficiency low temperature reach in cases also exhibited high savings, primarily due to the inclusion of ECMs

• VFDs contributed substantially less savings than predicted, probably due to the presence of staged controls previously

26

Measure Realization Rate

Floating head and suction

pressure controls

119%

Floating head and suction

pressure controls plus

VFDs

84%

Electronically Commutated

Motors312%

High efficiency low temp reach

in cases

157%

Summary of Measure Realization Rates

27

• Measure group realization rates ranged from 0.77 for lighting to 3.12 for Electronically Commutated Motors

• The overall program realization rate was found to be 1.15.

• Recommendation to apply measure-level recommended realization rates to evaluated period. Make program and engineering adjustments in future to align RRs.

Measure CategoryBilling Analysis Realization Rate

Engineering Analysis

Realization Rate

Recommended Realization Rate

Energy Efficient Cases 1.27 1.57 (low temp only) 1.27

Electronically Commutated Motors N/A 3.12 3.12

Lighting 0.77 N/A 0.77

Floating Head Controls - w/o VFDs1.07

1.19 1.19

Floating Head Controls - w/ VFDs 0.84 0.84

Lighting and Electronically Commutated Motor Realization Rates

• For lighting, there did not appear to be any significant issues with the calculation methodology or assumptions. A few possible reasons for the realization rate include:o The effects of other measures and equipment

(interactive effects)o Stores did not actually install the lamp and ballast

combinations listed for the retrofito Stocking hours have been extended since the lighting

retrofit. o Emergency lights could be in use and operating 8,760

hours per yearo Additional lights have been added since the retrofit

• It is our assessment that existing motors are running at higher loads than anticipated by the GrocerySmart model

28

Program Cost-Effectiveness

Cost Effectiveness Test

Benefits(NPV)

Costs(NPV)

Net Benefits B/C RatioCost of

Conserved Energy

Utility (UTC $10,373,401 $7,358,602 $3,014,799 1.41 $0.036

Participant (PCT) $13,413,849 $664,697 $12,749,152 20.18 $0.003

Ratepayer Impact (RIM)

$10,373,401 $20,772,451 -$10,399,050 0.50 $0.101

Total Resource (TRC)

$11,410,741 $8,023,299 $3,387,442 1.42 $0.039

29

• Using the Total Resource Cost (TRC) test, the program was found to be cost effective with a TRC of 1.42

• Other cost-effectiveness test results in table below

Summary• The program level measure realization (1.15) was

close to the ex-ante impact estimates. • Based on the TRC test, the program is cost

effective (TRC=1.42)• The combination of a billing analysis and

engineering analysis proved effectiveo Engineering approach best for measures with impacts

that are relatively small compared to whole building loads

o Billing analysis less costly and provides good results for measures with relatively large impacts compared to whole building loads

• The program is well received among the member utilities

30

Discussion

• Contact Information:

o Gary Cullen: gcullen@summitblue.com, 360-718-8392, x 304

o Deborah Swarts: dswarts@summitblue.com, 360-314-6298, x 302

31