wap monitoring & quality control
DESCRIPTION
WAP Monitoring & Quality Control. Thursday, September 23, 2010. Agenda. DOE Monitoring Approach IG/GAO Trends DOE Field Implementation Quality Assurance Grantee Monitoring. DOE Monitoring Approach. Monthly Timeliness Accuracy Production Planned vs. Actual - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/1.jpg)
Weatherization Assistance Program eere.energy.gov
WAP Monitoring & Quality Control
Thursday, September 23, 2010
![Page 2: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/2.jpg)
Weatherization Assistance Program eere.energy.gov
Agenda
• DOE Monitoring Approach
• IG/GAO Trends
• DOE Field Implementation
• Quality Assurance
• Grantee Monitoring
![Page 3: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/3.jpg)
Weatherization Assistance Program eere.energy.gov
DOE Monitoring Approach
• Monthly
– Timeliness
– Accuracy
– Production
• Planned vs. Actual
• Previous Month vs. Current Month
– Expenditures
• Planned vs. Actual
• Previous Month vs. Current Month
![Page 4: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/4.jpg)
Weatherization Assistance Program eere.energy.gov
DOE Monitoring Approach
• Quarterly
– Completeness
• OMB 1512
• DOE 504.3 Program Report
• SF-425 Federal Financial Report
– Timeliness
– Accuracy of Data
– Performance Targets
• Production
• Expenditures
• Monitoring
• Training
![Page 5: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/5.jpg)
Weatherization Assistance Program eere.energy.gov
DOE Monitoring Approach
• On-Site Visits (Minimum frequency/Year)
– Grants over $100 Million – Quarterly
– Grants between $100 Million - $40 Million – Three visits
– Grants under $40 Million – (Semi-Annual)
• On-Site Visit Content
– Program Administrative Management
– Program Technical Management
– Program Financial Management
![Page 6: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/6.jpg)
Weatherization Assistance Program eere.energy.gov
IG/GAO Trends
• Quality of Oversight of Programs, Contractors, and Program Staff
• Substandard installation of energy saving materials
• Davis-Bacon non-compliance
• Low Spending Rates/Grant recipients fell short of goals to weatherize homes
![Page 7: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/7.jpg)
Weatherization Assistance Program eere.energy.gov
IG/GAO Trends
• Inadequate systems capable of tracking overall performance by individual contractors and local agencies
• Insufficient Internal Controls to Prevent Fraud, Waste, and/or Abuse of Funds:
– Poor workmanship
– Incomplete client files
– Lack of qualified staff
– Incomplete vehicle and equipment inventories
– Onsite monitoring and/or inspection
![Page 8: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/8.jpg)
Weatherization Assistance Program eere.energy.gov
A Snapshot of IG Conclusions and Observations
The State X CAA had not implemented financial and reporting controls needed to ensure Weatherization Program funds are spent effectively and efficiently.
Specifically, State X CAA had not:
• Performed on-site financial monitoring of any of its sub-grantees under the Recovery Act;
• Reviewed documentation supporting sub-grantee requests for reimbursements to verify the accuracy of amounts charged;
• Periodically reconciled amounts paid to sub-grantees to the actual cost to weatherize units;
• Maintained vehicle and equipment inventories as required by Federal regulations and state and Federal program directives; and,
• Accurately reported Weatherization Program results to the Department.
![Page 9: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/9.jpg)
Weatherization Assistance Program eere.energy.gov
DOE Field Implementation
• Grantee Level State Plan Implementation
Grantee Policy and Procedures Manual and Field Standards
Reporting Systems (Financial, Administrative, and Programmatic)
Training and Technical Assistance
• Subgrantee Level Program and Financial Management
Production Planning and Quality Control Process
Procurement
File Management
![Page 10: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/10.jpg)
Weatherization Assistance Program eere.energy.gov
DOE Field Implementation
• Dwellings
Training Requirements
Application Process
Initial Inspection/Audit
Health and Safety
Technical Standards and Installation
Quality Control
Job Costs and Funding Sources
![Page 11: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/11.jpg)
Weatherization Assistance Program eere.energy.gov
Federal – QA Reviews
• All agencies to be visited by contractors
• Contractors may NOT be “weatherizers”
• Home Inspectors that are BPI credentialed; RESNET/HERS; or Weatherization Trained
• Captures information for PO to determine where attention needs to be focused
• Strategy
• “SMS version” -- Hit the state with 3-6 staff – disrupt operations for a week in lots of agencies – then get out
• “QA Contractor” – PO schedules reviews, depending on need, could be continuously (xx% a week)
• Results
• Aggregated results/discussion will lead to focused attention by the Project Officer
![Page 12: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/12.jpg)
Weatherization Assistance Program eere.energy.gov
The Process is a Cycle
![Page 13: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/13.jpg)
Weatherization Assistance Program eere.energy.gov
Quality Assurance Reviews
• The unit review form has 3 major sections:
1.Agency/unit identification
2.File review
3.On-site work assessment
1.Attic
2.Sidewalls/kneewalls
3.HVAC
4.Subspace
5.Windows/doors
6.Other measures
7.Additional attention needed?
![Page 14: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/14.jpg)
Weatherization Assistance Program eere.energy.gov
Unit identification and file review
• The unit identification and desk file review sections should be completed at the State or sub-grantee office prior to the site visit.
– We usually spend half a day reviewing 8-12 files then a day and a half in the field
• 45 minutes for the first one; 35 minutes for the next; 15-20 minutes for the rest… (we are trainable!)
• 45 minutes (on average) going through a unit
![Page 15: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/15.jpg)
Weatherization Assistance Program eere.energy.gov
Example:
FILE REVIEW
Unit Assessed Using: (select one) Energy Audit Priority List None Evidenced
Energy Audit or Priority List is included and aligns with expected audit/list (approved by DOE)
Energy Audit or Priority List is included, but does NOT align with approved audit/list
No clear indicator of use of site specific audit or approved priority list in assessment
Work Order/Building Weatherization Report Yes No
DOE Investment: $_________________________ Total Job Investment: $_____________________
Yes, work order is included in file and appears to follow the audit/list
Yes, work order is included but does NOT follow approved audit/list
No, work order is not in file
Quality Assurance
![Page 16: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/16.jpg)
Weatherization Assistance Program eere.energy.gov
Trends in Quality within State
• With reviews of 4-6 different agencies in a state
– We look for isolated issues for specific agencies
• Agency may be mis-entering information in the audit tool
(e.g., door = infiltration measure reduction)
• Missed opportunities for savings -- sometimes the “low hanging fruit” (e.g., no refrigerator replacements allowed)
– We look at what may be an area for the state to affect change (peer-to-peer, training needs)
![Page 17: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/17.jpg)
Weatherization Assistance Program eere.energy.gov
Electric Base Load Measures
![Page 18: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/18.jpg)
Weatherization Assistance Program eere.energy.gov
Regional Trends and Opportunities
• With reviews of 4-6 different states in a region
– We look for isolated issues for specific states
• Statewide concerns (e.g., windows sans justification)
– We look for trends that might be housing stock or climate based
• Air sealing challenges based on a specific housing stock
• How solar screens are being implemented in the program in some of the hot climates
– We think about how best to pass information from one state to another
![Page 19: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/19.jpg)
Weatherization Assistance Program eere.energy.gov
Window Sunscreens
![Page 20: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/20.jpg)
Weatherization Assistance Program eere.energy.gov
National Trends and Opportunities
• After seeing about 20 states, you start to think about national opportunities/conversations
– Audits and Priority Lists
• Clarifying when states with a priority list need to run a site specific audit
• Multi-family misunderstandings – what states need to do if they do not have an approved multi protocol in place
– Health and Safety
• What does that mean, how does it get implemented
![Page 21: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/21.jpg)
Weatherization Assistance Program eere.energy.gov
Passing along “Best Practices”
• We see opportunities to share with others
– Colorado – Passport for crew advancement
– Iowa – Agency photo files were superb
– Idaho – Best LSW documentation we’ve seen
– North Carolina – Agency has amazing file documentation, should be replicated
– Vermont – Great use of two-part foam
– Mississippi, Alabama, Illinois, Georgia, New (all)…
![Page 22: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/22.jpg)
Weatherization Assistance Program eere.energy.gov
And, building up our “can you believe” library
Cardboard box used for fan enclosure
![Page 23: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/23.jpg)
Weatherization Assistance Program eere.energy.gov
Can you figure this one out??
![Page 24: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/24.jpg)
Weatherization Assistance Program eere.energy.gov
Personal Favorite
Agency assured us, this is not their typical method
![Page 25: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/25.jpg)
Weatherization Assistance Program eere.energy.gov
What do we know
• We know:
– 30,000 houses are going to be reviewed
• To date 2000 have been done
– Every local agency is going to be visited
• Maybe multiple times, depending on production levels
– Local agencies are going to be asked to go with the contractor on reviews (Grantee is welcome)
• At their discretion, agency can send the contractor out without local agency staff; that is the choice of the agency
![Page 26: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/26.jpg)
Weatherization Assistance Program eere.energy.gov
Grantee/Subgrantee Scheduling is Not Random
Factors• Levels of Grantee and mandatory
visits/year requirement and % requirements
• Factors in last (most recent) review dates
• Current state of Actual production, units weatherized vs. Planned
• Number of units to be inspected per Subgrantee based on the “inherited” schedule from the Grantee – even considers some Subgrantees with disproportionately higher number of units than others in state
• Factors in the net number of reviews done already
• Need to balance to @ 2000-2200 units/month over several years
Scheduling Approach• Schedule 3 months in advance• Advise PO’s and Contractor
– PO’s can confirm schedules with Grantees/sub-grantees• Problems on timing or availability
can be accommodated with schedule adjustment (which, when)
• Vacations can be worked around– Contractor can identify resources
as required• Finalize schedule @2 weeks prior to
start of next month
![Page 27: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/27.jpg)
Weatherization Assistance Program eere.energy.gov
What we DON’T Know
• We don’t know:
– Who – contract hasn’t been awarded
– When – when is predicated on determining who…
– What – (I just threw this in since I was on a roll)
– Where – DOE hasn’t determined priority points of what states will go first
• Will it be states that haven’t been visited by SMS?
• Will it be large states with high production?
– How – training of contractors STILL must occur, even after award
• Let’s make sure they know what they are looking at
![Page 28: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/28.jpg)
Weatherization Assistance Program eere.energy.gov
We’re all in this together
• Weatherization Plus goal
– The quality of work would be the same regardless of locale
• Evaluation… Quality Assurance Reviews… PO increased monitoring efforts…
– These equal lots of opportunities to get it right and affect change… together!
![Page 29: WAP Monitoring & Quality Control](https://reader035.vdocument.in/reader035/viewer/2022062517/56813b16550346895da3c3f0/html5/thumbnails/29.jpg)
Weatherization Assistance Program eere.energy.gov
• Questions/Comments?