Performance Metrics for Weatherization
David Carroll
State WAP Evaluations
ACI Home Performance Conference
April 30, 2014
What is Performance Measurement?
2
Step 1 - Goals• What are your goals?
– Are you trying to serve a certain population?– Are you trying to affect …
• Energy use?
• Energy costs?
• Payment problems?
• Emissions?
• Health problems?
– How will you know when you are successful?
3
Step 2 – Measurement• Overall Research
– Who are you serving?– How are you serving them?– What are they saving?– What other benefits are you delivering?
• Comparative Research– What approach maximizes energy, cost,
payment, emissions, health impacts?
4
Step 3 - Assessment
• Assessment– What are your goals for your program?– Does your performance match your goals?– How can you better align your performance
with your goals?
5
Step 4 – Improvement
• Strategies– Changing who you serve
• Targeting
• Partnerships
– Changing what you deliver• Assessments
• Measures
6
Step 4 - Improvement
• Strategies (continued)– Changing your performance
• Comparative analysis
• Mentoring / partnerships
– Changing funding available• Leveraging
• Advocacy
7
A Comprehensive Approach to Performance Measurement
8
Benchmark
• Use data on ….– Population / Census Data (ACS)– Recipients / LIHEAP or Energy Assistance– WX Clients / Demographics, Housing Units,
Baseline Conditions
9
Observe
• Observations of ….– Intake– Audits– Service Delivery– Inspections
10
Measure
• Collect data on ….– Installed Measures– Post-WX Conditions– Pre/Post Energy Usage– Client Outcomes – Payments, Housing, Health
11
Analyze
• Study ….– Gross changes in weather-normalized usage– Net changes using a comparison group– Variations in outcomes by key client, building,
and program factors
12
Assessment
• Investigate…– Why are some subgroups seeing high impacts?– Why are some subgroups seeing low impacts?– What strategies might be tested to better align
outcomes with goals?
13
Who are you Serving? Example of using Census and LIHEAP
data as a Benchmark
14
Minnesota Households - 2010
About one in four Minnesota households were income-eligible for the LIHEAP Program in 2010.
15
Eligibility Group Number of Households Percent of Households
Not Income Eligible 1,564,187 75%
Income Eligible 527,359 25%
All Households 2,091,546 100%
Source: 2010 American Community Survey
Minnesota LIHEAP and WAP Recipients in 2010
About 38% of income-eligible households received LIHEAP and 2% received WAP.
National averages are 17% and 0.5%, respectively.
16
Received BenefitsLIHEAP
RecipientsPercent of
HouseholdsWAP
RecipientsPercent of
Households
Yes 199,969 38% 11,207 2%
No 327,390 62% 516,152 98%
Income-Eligible Households
527,359 100% 527,359 100%
Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database
Housing Unit Type
LIHEAP is less likely to serve households in Large Multi-Family buildings. WAP serves very few households in Multi-Family buildings of any type.
17
Housing Unit Type Low-Income LIHEAP Recipients WAP Recipients
Single Family 52% 59% 80%
Mobile Home 5% 11% 14%
Small Multi-Family 8% 7% 3%
Large Multi-Family 35% 23% 3%
Total 100% 100% 100%
Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database
Housing Unit Type
LIHEAP is less likely to serve households in Large Multi-Family buildings. WAP serves very few households in Multi-Family buildings of any type.
18
Housing Unit Type Low-Income LIHEAP Recipients WAP Recipients
Single Family 52% 59% 80%
Mobile Home 5% 11% 14%
Small Multi-Family 8% 7% 3%
Large Multi-Family 35% 23% 3%
Total 100% 100% 100%
Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database
Ownership Status
LIHEAP serves both owners and renters. WAP is less likely to serve renters.
19
Ownership Status Low-Income LIHEAP Recipients WAP Recipients
Owner 46% 44% 84%
Renter/Other 54% 56% 16%
Total 100% 100% 100%
Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database
Ownership Status – Single Family Units
Even if we control for housing unit type, WAP is more likely to serve owners than renters.
20
Ownership Status Low-Income LIHEAP Recipients WAP Recipients
Owner 75% 60% 87%
Renter/Other 25% 40% 13%
Total 100% 100% 100%
Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database
Ownership Status of WAP Participants – Single Family Units
Control for housing unit type reduces that variation in service to renters by substate area.
21
Ownership Status
State NE NW Central Metro SE SW
Owner 87% 93% 94% 89% 80% 88% 88%
Renter/ Other
13% 7% 6% 11% 20% 12% 12%
Total 100% 100% 100% 100% 100% 100% 100%
Source: 2010 American Community Survey and 2010 MN LIHEAP/WAP Database
How are you serving them? Example of furnishing details for
the state, region, and agency.
22
23
Table 4 PY 2010 Clients in Single Family Homes
Housing Unit Characteristics by Region and Agency
Statistic STATE Central Region
Lake Wobegone
Number of Units 8,863 1,292 100
Housing Unit
Median Heated Space 1,340 1,235 1,200
Mean Heated Space 1,451 1,336 1,350
% One Story 40% 46% 49%
Housing Vintage
% pre 1940 38% 38% NC
% 1940-1969 33% 27% NC
% 1970 or later 29% 35% NC
PreWX Status
Mean CFM 50 2,337 2,163 2,855 NC=Not Collected
24
Table 7 PY 2010 Clients in Single Family Homes
Air Sealing and Shell Measures by Region and Agency
Statistic STATE Central Region
Lake Wobegone
Number of Units 8,863 1,292 100
Air Sealing
< 500 CFM Reduction 54% 58% 50%
500 - < 1000 CFM Reduction 23% 20% 15%
1000+ CFM Reduction 19% 17% 32%
No Information 5% 5% 3%
Any bypass sealing or caulking 81% 75% 82%
Attic Insulation
% Installed 86% 83% 90%
Wall Insulation
% Installed 36% 36% 39%
Other Insulation
% Floor insulation 5% 7% 6%
% Rim/band joist insulation 19% 41% 15%
% Foundation insulation 6% 6% 10%
Mean CFM reduction 607 548 840
% Installed Mechanical Ventilation 46% 57% 55%
How much energy are you saving and how does that vary by important program factors?
25
Table 1WAP Energy Impacts for Single Family Site-Built Homes
Gross and Net Gas Savings by End Use (therms/year)
Group/Breakout # HomesGas Use Pre-WAP
Gas Use Post-WAP
Gross Savings Net Savings % of Pre
Total Use 937 983 845 138 (±10)
130 (±11) 13.2% (±1.1%) Comparison 1,295 929 921 8 (±4)
Heating Use 937 775 658 117 (±9)
100 (±11) 13.0% (±1.4%)
Comparison 1,295 741 724 16 (±6)
Baseload Use 937 208 187 21 (±7)
30 (±8) 14.2% (±17)
Comparison 1,295 188 196 -8 (±5)
Table 3WAP Energy Impacts for Single Family Site-Built Homes
Gas Savings for Homes with Natural Gas Main HeatBy Measure Combination (therms/year)
Group/Breakout # HomesGas Use Pre-
WAP Net Savings % of Pre
No Major Measures 202 877 37 (±15) 4.2% (±1.7%)
One Major Measure 298 957 121 (±17) 12.7% (±1.8%)
Two Major Measures 211 1,003 162 (±20) 16.1% (±2.0%)
Three Major Measures 115 1,111 236 (±29) 21.2% (±2.6%)
All Four Major Measures 33 1,179 382 (±71) 32.4% (±6.1%)
Table 4
WAP Energy Impacts for Single Family Site-Built HomesNet Gas Savings for Natural Gas Main Heat by Pre-Weatherization
Gas Usage (therms/year)
Pre-WAP Gas Use (therms/yr)
Major Measures # Homes
Gas Use Pre-WAP Net Savings % of Pre
All Clients 1.4 937 983 130 (±10) 13.2% (±1.1%)
<750 th/yr. 1.1 245 640 64 (±12) 10.0% (±1.9%)
750-1000 1.3 296 880 105 (±14) 12.0% (±1.6%)
1000-1250 1.6 226 1,097 142 (±22) 12.9% (±2.0%)
1250-1500 1.6 101 1,355 219 (±42) 16.2% (±3.1%)
>=1500 th/yr. 2.0 69 1,731 269 (±65) 15.6% (±3.7%)
Table 12Gas Impact Results by Agency for Gas Heated
Single Family Site-Built Homes
Agency IDGas Use Pre-WAP Net Savings % of Pre
# of Measures
A 1,077 187 (± 21) 17.3% (±1.9) 2.2B 992 122 (± 32) 12.3 % (±3.2) 0.9C 1,028 119 (± 40) 11.6% (±3.9) 1.2D 948 118 (± 19) 12.4% (±2.0) 1.3E 1,012 113 (± 64) 11.1% (±6.3) 0.7F 937 109 (± 44) 11.6% (±4.7) 1.2G 945 107 (± 50) 11.3% (±5.3) 0.9H 875 94 (± 26) 10.7% (±3.0) 0.7I 929 94 (± 41) 10.1% (±4.4) 1.1J 889 58 (± 27) 6.5% (±3.0) 0.5
Total 983 130 (±11) 13.2% (±0.7) 1.4
Make sure you consider other program impacts.
30
Table 26Emissions Benefits
Estimated Value of Avoided Emissions per Home for Single Family Site-Built Homes (2013 Dollars)
Pollutant First Year Lifetime
CO2 Equivalent $53.65 $1,009.63
NOX $23.45 $432.11
PM2.5 $2.49 $42.85
SO2 $44.48 $684.39
VOC $0.41 $8.25
Total $124.49 $2,177.22
31
Dwelling Quality Indicators
32
IndicatorPre-Audit Incidence
Post-WX Incidence
Percent Change
Home sometimes at unhealthy temperature
20% 9% -55%
Observed standing water in home
33% 19% -42%
Frequent mildew odor or musty smell
29% 16% -45%
Home somewhat or very infested with insects
25% 17% -32%
Home has CO monitor 41% 74% +80%
Affordability Indicators
33
IndicatorPre-Audit Incidence
Post-WX Incidence
Change
Got disconnect notice 45% 33% -26%
Had natural gas, electricity, or bulk fuel service terminated
18% 9% -50%
Went without food to pay energy bills
34% 23% -33%
Went without food in the last four weeks
9% 6% -33%
Used high interest loan to pay energy bills
20% 12% -40%