Download - Presented at the American Evaluation Association/Canadian Evaluation Society Joint Conference
Crossing Methodological Borders to Develop and Implement an
Approach for Determining the Value of Energy Efficiency R&D
Programs
Crossing Methodological Borders to Develop and Implement an
Approach for Determining the Value of Energy Efficiency R&D
ProgramsPresented at the
American Evaluation Association/Canadian Evaluation Society Joint Conference
Toronto, Canada October 28, 2005
Scott Albert, GDS AssociatesHelen Kim, NYSERDA
Rick Ridge, Ridge & AssociatesGretchen B. Jordan, Sandia National Laboratory
2
The NYSERDA PortfolioThe NYSERDA PortfolioTotal New York Energy $mart Budget, By Program Area ($ millions)
Total 8-Year Budget = $961.8 million*
NYS Cost Recovery Fee
$9.0 0.9%
Research and Development
$210.8 21.9%
Business and Institutional
$359.1 37.3%
Residential (non Low-Income)
$170.717.7%
Low-Income $128.4 13.4%
Program Administration
$64.6 6.7%
Environmental Disclosure
$2.9 0.3%
Metrics and Evaluation $16.2 1.7%
Source: New York Energy $martSM P rogram - Financial Status Report. As of December 31, 2004.*Including Estimated Interest Earnings
3
R&D Budget Through 12/31/04R&D Budget Through 12/31/04
$13$11
$18$13 $13
$2
$32
$21
$60
$20 $21
$4
$52
$28
$67
$22
$30
$7
0
10
20
30
40
50
60
70
80
WholesaleRenewables
End-UseRenewables
DG-CHP EMEP NextGeneration
Power &Storage
Invoiced Committed Budgeted
4
ObjectiveObjective• Develop and pilot-test an evaluation model for NYSERDA’s
R&D program area covering 1998 through 2004 that recognizes:– R&D programs and their societal impacts are difficult to
evaluate by their nature.– The outcomes are subject to multiple and uncontrollable
influences that are difficult to foresee. – The cycle for product development is 5 to 15 or 20 years and
many of the energy and economic impacts of R&D projects may not be fully realized and measured for many years.
– Given the multiple and compounding effects that happen along the way, it is also very difficult to be exact about the attribution of impacts to any one particular effort.
– When evaluating an entire portfolio of R&D projects, objectives and outcomes vary by project.
5
R&D Portfolio Logic ModelR&D Portfolio Logic ModelNYSERDA Select & Manage R&D Projects to:-drive portfolio changes over time to respond to current needs, and-Provide public benefits
Develop new or improved product
Study, Prove Concepts
Demonstrate products, inform markets
Study to inform policy & R&D community
Test & improve products
Dissemination builds common knowledge base-Lab prototypes-Future R&D & product options
-Investment/interest growing-Commercial scale product developed -Potential demonstrated
Product proven/ introduced in market
Informed policies & programs;R&D opportunities & standards identified, publicized
- Data from tests- Establish standards- Hands on experience (industry)-Feedback to R&D
White papers, workshops;Policy-relevant research
-Intermediate scale prototypes- Performance/cost specifications improving
Producers, consumers,
policy makers see value
New knowledge:-papers, articles -data
- Data from tests in different context- Feedback to R&D & policy makers- Visibility & data from showcases
Inputs:Funds, staff, NYSERDA competencies, partnerships
Activities
Outcomes
Outputs
Policy and Product development and pre-deployment process (5-10 years)
Knowledge for future R&D and productsFirms have credibility & market infrastructure is supportive
Products manufactured as replacement, stand alone, or part of system and purchased by early adopters
External Influences:Cost, Performance of existing technologies; Industry willingness to take risks; Uncertainty of R&D; Energy prices; Government policies
Research for Policy DemonstrationProduct Development Pre-deployment
Educate, provide incentives to supply & delivery
-Training, certification-Production incentives-Innovative designs-Other barriers lowered
Business infrastructure supports
the product
Environmental benefits Energy benefits- generation, energy/load management, efficient use
Economic benefits -cost of compliance, NY jobs
- New- Accelerated- Expanded
6
Six Stages of the R&D ModelSix Stages of the R&D Model
• Information for policy makers and R&D community• Product development stage 1 – study and prove
concepts • Product development stage 2 – develop new or
improved products • Product development stage 3 – product testing • Demonstration• Pre-deployment
7
The Value/Cost MethodCombines Two Approaches
The Value/Cost MethodCombines Two Approaches
• Aggregate approach– Analyzed data collected for each of NYSERDA’s 638 R&D
projects (since 1998) in the portfolio.– Basic statistics, such as the number of projects, expenditures
by technology type, leveraged funds, and the stage of development were calculated to describe the entire R&D portfolio.
• Peer Review– Analyzed using an adaptation of the Composite Performance
Rating System (CPRS) used to evaluate the U.S. Department of Commerce’s Advanced Technologies Program (ATP).
– Peer review approach was applied to a small sample of successful R&D projects, covering each of the six R&D stages (project types).
8
ATP: Composite Performance Rating System Constructed Bottom-up; Used
Top-down
ATP: Composite Performance Rating System Constructed Bottom-up; Used
Top-down Performance Distribution for the Portfolio
Distributionby
Tech Area
Distributionby
Firm Size
Distributionby
Locationetc.
Project 1Case Study
Project 2Case Study
Project 3Case Study
Project 4Case Study
Project nCase Study
...
• Unique cases• Aggregate statistics• Composite scores• Performance distributions• Minimum net portfolio benefits
CPRS 1 CPRS 2 CPRS 3 CPRS 4 CPRS n
ATP MethodR.Ruegg, Nov. 2002
9
AGGREGATE ANALYSISAGGREGATE ANALYSIS
• Expanded and updated R&D database in order to carry out a comprehensive descriptive analysis of the entire R&D portfolio.
• Variables Considered– Funding– Technology Area– Co-Funding Entity– Project Status– Expected Benefits from R&D Projects
10
Questions Addressed by Aggregate Analysis
Questions Addressed by Aggregate Analysis
• How does NYSERDA funding per project vary by project type?
• How does NYSERDA funding per project vary by program?
• What is the frequency of the various project types?• What goals are being served by the various project
types?• What are the primary goals served by the portfolio?• What are the sources of funding, by project type?• What is the funding share contributed by partners?• How does NYSERDA funding and co-funding vary by
project type over time?• How does the mix of technologies and issues examined
change over time?
11
Results:Aggregate Analysis
Results:Aggregate Analysis
12
NYSERDA Funding, by Project Type
NYSERDA Funding, by Project Type
Test/Improve Product16%
Demonstration42%
Research for Policy15%
Pre-deployment22%
Develop/Improve New Product
64%
Proof of Concept20%
Product Development
21%
13
Funding by GoalsFunding by Goals
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Energy
Environment
Economic
Percentage of Projects with Expected Benefit
14
Co-Funding Sources Co-Funding Sources
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Research forpolicy
Study, proveconcept
Develop newor improved
product
Test andimproveproduct
Demonstration Pre-deployment
Project Type
Allo
cate
d F
undi
ng
NYSERDA GOVERNMENT ACADEMIC COMMERCIAL OTHER
15
Percent of Projects by Technology and Year Percent of Projects by Technology and Year
0%
20%
40%
60%
80%
100%
1999 2000 2001 2002 2003 2004
Energy Using Technology Environment Materials
Other Generic Power Systems Alternative Power
Cogeneration Energy Storage Transportation
16
Peer Review Focused on Six Success Stories as a Pilot TestPeer Review Focused on Six
Success Stories as a Pilot TestR&D Stage Technology Area Project Name
Information for Policy and R&D Community
HVAC 21st Century HVAC Research Consortium
Product Development Stage 1 – study and prove concepts
Demand Resposne Aggregating Distributed Generators (also in R&D Demonstration stage)
Product Development Stage 2 – develop new or improved products
Environmental EMEP – Development of Continuous Ambient Paticulate Monitor
Product Development Stage 3 – product testing
Transportation Evaluation of Truck Stop Electrification for NYS (also in Product Development Stages 1 and 2, Demonstration, and Pre-Deployment)
Demonstration Industrial Turnkey Pump and Compressed Air Program
Pre-deployment Power generation Green Power Marketing Program
17
Indicator VariablesIndicator Variables
• Choice of indicator variables for the R&D portfolio guided by the R&D portfolio logic model.
• Six categories of outcomes, identified in the logic model were selected: – Knowledge creation,– Knowledge dissemination,– Commercialization progress,– Energy benefits,– Economic benefits, and – Environmental benefits.
18
Accomplishment PacketsAccomplishment Packets• Project-specific accomplishment packets were then
developed to document objective evidence regarding the six outcomes:– Knowledge creation– Knowledge dissemination– Commercialization progress– Realized and potential energy benefits – Realized and potential economic benefits– Realized and potential environmental benefits – Value versus cost (not a specific outcome, but this
item was also included in the peer-reviewer response packet for 0 to 4 rating)
19
Review ProcessReview Process
• Reviewers willing to participate were sent:– Peer Review Instructions,– Conflict of Interest Form– Peer Review Assessment Form, and– the Peer Review Information Packet for their
specific project.• Over a period of five weeks, the reviewers
completed their assessment and returned them for data entry.
20
Results:Peer Review
Results:Peer Review
21
Weighted Rating By ProjectWeighted Rating By Project
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
TruckstopElectrification
Compressed AirProgram
21st CenturyHVAC
EMEP AirParticulateMonitoring
Aggregating DG
Me
an
Ra
tin
g (
We
igh
ted
)
22
Overall Ratings by OutcomeOverall Ratings by Outcome
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Knowledg
e Cre
ation
Knowledg
e Diss
eminati
on
Commerci
aliza
tion
Progr
ess
Energ
y Ben
efits
Econom
ic Bene
fits
Enviro
nmen
tal B
enef
its
Value
vers
us C
ost
Mea
n R
atin
g (
Un
wei
gh
ted
)
23
Overall Ratings, by Project, by Outcomes
Overall Ratings, by Project, by Outcomes
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
KnowledgeCreation
KnowledgeDissemination
CommercializationProgress
Energy Benefits Economic Benefits EnvironmentalBenefits
Overall Value
Un
wei
gh
ted
Rat
ing
Truckstop Electrification Compressed Air Program 21st Century HVAC EMEP Air Particulate Monitoring Aggregating DG
24
Conclusions: Aggregate AnalysisConclusions: Aggregate Analysis
• Assumes more risk than the commercial sector in the earlier stages of technology development, while in the latter stages, the reverse is true.
• Covers a wide range of technologies that are aimed at achieving potentially significant energy, economic and environmental benefits.
• Leverages funds on a 4.3 to 1 ratio.• Partners with a wide range of public and private
organizations and institutions.• Evolves over time in response to the societal
needs and opportunities to address them (i.e., the technologies and issues addressed in the R&D portfolio are not static).
25
Conclusions: Peer ReviewConclusions: Peer Review
• Peer review scores from the pilot test averaged 3.34 (on a 0-to-4 scale) across all assessment categories.
• There are substantial benefits across all documented accomplishment areas for the five projects assessed.
• Significant progress is being made toward the eventual achievement of measurable 3-E benefits.
26
Conclusions: Peer Review ProcessConclusions: Peer Review Process
• The information provided in the review packets for the five selected projects was adequate
• The instructions provided were clear• The criteria used in the assessments were clearly
defined• The criteria used in the assessments were the right
ones• It is very important for NYSERDA to assess the value
of its R&D programs• The results of the peer review process should be
useful for NYSERDA decision-makers• Reviewers can assess a fair amount of information if
the information is presented in a clear and organized format.
• Statistical analyses revealed that the ratings provided by the peer reviewers were reliable.
27
Next StepsNext Steps
• Routinize the collection of key indicator data for all R&D projects.
• Perform aggregate analysis on all projects • Focus significant effort on a more
representative sample of projects