chapter 6. making success measurable through quality control

1

Chapter 6.Making Success Measurable

Through Quality Control

2

Chapter 6. Section 1. Introduction to Quality Control

3

Introduction• How do you know the quality and accuracy

of your rent calculations? – Wait for HUD to tell you?– Close your eyes and hope for the best?– Assume that good people lead to good

outcomes?– Wait until you tally SEMAP scores?

• Or …..

4

Introduction

• Are you interested in a better way that could:– Enable you know when to take action on

performance issues,– Help document the improvements in error

reduction you are making,– Enhance the quality of the services you

provide?

5

Introduction

• A good quality control program can be that better way that helps you:– Prevent errors– Documents compliance with program

requirements– Demonstrate performance improvement over

time

6

Introduction

• An effective quality control program:

– Facilitates conversation about expectations and performance for individuals and the organization as a whole.

– Enables program managers to be proactive, rather than reactive

7

Introduction

In this session, we will talk about:

• Three types of quality control

• Illustrate the thought process needed to develop a QC program

• Provide sample QC tools

• Provide guidance on communicating with your staff about quality issues.

8

Introduction

• We will focus on HUD’s expectations related to income and rent determination

• BUT, the QC concepts we discuss apply equally well to other parts of a PHA operation.

9

Why Quality Control?

• Quality control can help to confirm what you already know. It can:– Provide documentation that supports

recommendations you want to make,– Generate statistics that document staff

performance,– Provide valuable information on a systematic

basis to share within and outside your organization.

.

10

Why Quality Control?

• QC can be even more helpful if the results are a surprise. Then the results may highlight:– Problems of which you were not aware– Good news about improved performance or

excess capacity.

• Were there aspects of the RIM review for your agency that were a surprise?

11

Chapter 6. Section 2.Types of Quality Control

12

Types of Quality Control

Preventive

Continuous

Periodic

Types of Quality Control

13

Preventive QC

• Preventive QC is checking on the quality before the transaction is completed. Use Preventive QC when:– The consequences of errors are high, or– The error is not recoverable.

• Examples:– Supervisors sign termination letters even though

specialists sign other correspondence– Occupancy supervisor reviews 100% of rent

calculations for new staff (for some period of time).

14

Preventive QC

• More examples:– Intake specialist presents a “dry run” of the briefing to

staff before doing the real thing in front of participants.

• What examples of preventive QC occur at your PHA?

• Try to identify at least one additional area (related to income and rent) where preventive QC might be helpful.

15

Limits of Preventive QC

• Preventive QC can be an important tool but what are its limitations?

– PHA’s don’t have time or staffing to double and triple check every transaction –you need to prioritize.

– Use Common Sense: Not everything requires this level of effort.

16

Periodic Quality Control

• In contrast, periodic quality control happens after the fact. Work has already been done and reported.

• It’s purpose is to take a “picture-in-time” assessment of PHA operations that will help improve the program and prevent future errors.

• Sound familiar? Yes – HUD’s RIM reviews and SEMAP audits are kinds of periodic QC.

17

Periodic Quality Control

• During a periodic review HUD systematically looks at:– Policies, procedures, and practices and– Individual files.

• But you don’t have to wait for HUD. You can also do your own. You could:– Look at the same things HUD will be

checking, or– Focus your periodic review differently to learn

even more.

18

Limits of Periodic QC

• Periodic QC can provide an accurate point-in-time assessment, but:– It doesn’t help managers see trends– Results come infrequently so it doesn’t help

managers take corrective action during the year.

– The level of effort required may seem overwhelming (but HUD’s RIM Review guide provides a good model for the review).

19

Continuous Quality Control

• Continuous QC is used to help with day-to-day management.

• It involves analyzing a few (but very important) factors on a regular, continuing basis. For example:– Tracking on-time reexaminations each month– Looking at accuracy rates by individual or

transaction

20

Continuous QC

• Continuous QC helps managers:– Focus more quickly on vulnerable areas.– Look at trends, rather than point-in-time

information.

• HUD does some continuous QC of 50058 submissions and uses the data to:– Send error reports to PHAs, and– Assess in general terms how PHAs are doing.

• But only the PHA can implement an effective continuous QC program.

21

Limits of Continuous QC

• Continuous QC is a powerful tool.• But, remember, the scope of the

Continuous QC is limited.• This means:

– Continuous QC provides only a limited picture (hopefully of the key areas)

– Continuous QC should be supplemented by Periodic Reviews that take a broader view.

22

Continuous QC Example

Page 6-11 and 6-12

• Look at the “Error Analysis” chart– QC results are shown by quarter & individual– Caseload = annual caseload– Errors = things HUD would call findings– Comments = errors that would not be findings– Types of errors are defined on page 6-12

23

Continuous QC Example

• What issues does this report raise for you?

• What are your concerns about:– Productivity?– Types of errors?– Error Rates?

• What are the implications of the information you see for this PHA?

24

Chapter 6. Section 3.Designing a QC Program

25

Designing a QC Program

• Now that we have discussed each type, it’s time to think about how to design your program.

• Before you begin it is very important to think through: – What you want your QC program to look at. – What you will do with the information.

26

Designing a QC ProgramWhere to Start

• To focus your program ask three questions:– What are the outcomes you want to achieve?– What would success look like for each

outcome?– What is the best way to confirm (track,

measure) performance in each area? • Then decide:

– Which measures are useful and feasible.– How you are going to record the information.– Who needs to know the results.

27

A Simple Example- Sleep

• What is my desired outcome?– A good night’s sleep

• What would success look like?– Eight hours of uninterrupted sleep

• What is the best way to measure?– An independent 3rd party could observe my

sleep.– I could track myself the time I go to bed, time I

wake up, and number of interruptions.

28

Simple Example (Cont)

• Decision:– Having someone watch me sleep could be

expensive and disruptive– Instead, I will track sleep patterns myself– I’ll write this down in a notebook– No one needs to see it but me (unless I think

maybe my doctor should know)

29

A More Serious Example – Utility Allowances

• What is my desired outcome?– Current and correct utility allowances

will always be used in rent calculations.

• What would success look like?– Utility allowances would be used

correctly in 100% of the files.

• Are there other ways you would describe success re: utility allowances?

30

A More Serious Example – Utility Allowances

• What are possible ways to measure the outcome?– 100% file review in advance (preventive)– 100% file review after the fact– Sample file reviews after the fact

• What else could we do to see if the correct utility allowances are being used consistently?

31

A More Serious Example

• Decision:– 100% review would be too time consuming

(before or after)– Therefore, review a sample of files (after the

fact) to see how often staff have used the correct utility allowance

– Generate a report that summarizes errors by employee, by program type, by transaction type.

32

Thinking About Outcomes

• Choosing the outcomes you want to use is an important part of the design process.

• Outcomes can be expressed in very general or very specific terms.

• Some “big picture” outcomes might include:– Consistency in data collection, documentation – Complete and accurate verifications– Accuracy in rent calculations– Timeliness – Complete, accurate reporting

33

Thinking About Outcomes

• Even broader outcomes might include:– No findings on the next RIM review– Informed decision making by applicants,

participants and HCV landlords– Fair and equitable treatment– Equitable workload– Being a SEMAP high performer– Positive perceptions – we’re a good agency.

34

Thinking About Outcomes

• Caution: Generally more specific outcomes are better understood and easier to measure. For example:– Reduce departmental error rate to less than

one percent– Complete 100% of annual reexaminations

accurately and on-time

• The more specific the outcome, the easier it is to track performance.

35

Designing a Program to Support Outcomes

• Page 6-20 through 6-21 show examples of turning desired outcomes into a QC plan.

• Follow along on the chart

• We will walk through Row 1 and Row 2 together

36

The End Result

• The end result of this thinking will be a Quality Control Plan. (see page 6-25).

• The planning itself is more important that the format for the plan.– Think through the process before you begin

data collection– Collect information about the items that have

the most effect on error reduction– Be realistic about what data you can collect.

37

What’s Next?

• The balance of this chapter discusses 5 important “how to” aspects of QC:– Conducting file reviews– Conducting a “general” review of operations– Using reports to track quality– Addressing staff skills and skill gaps– Communicating about error reduction and

quality control

38

Chapter 6. Section 4.How to Aspects of Quality

Control

39

Conducting File Reviews• Files reviews are used:

– To determine accuracy and completeness– To identify potential policy and management

issues

• “Files” for this purpose include the 50058 record as well as paper files.

• Caution: A file review can tell you what has happened, but not necessarily why.

40

Conducting Tenant File Reviews

• Sample size and characteristics of files selected depend upon the purpose.– Generally PHAs use a “directed” sample. – Directed means selecting the files that contain

the characteristics you want to monitor.– Page 6-28 shows characteristics that may be

important to consider for the sample.

• Not every characteristic must be represented each time you review files.

41

Conducting File Reviews

• For Continuous QC – Purpose is timely feedback on few issues. – File Selection Priorities would be:

• Each specialist• Basic transaction types• Files completed over time

42

Conducting File Reviews

• For Periodic QC– Purpose is a more comprehensive review.– File Section Priorities:

• Each specialist (but perhaps more files)• More transactions types• Larger cross section of income sources and

household types

43

Conducting File Reviews

• Tools for Selecting Files:– PIC reports can generate lists for some

transactions. (See Appendix A, A-25)– PHA vendor software may be able to generate

reports that list files in categories.

• How many files do I need to review?– QC sample should be sufficient to meet

SEMAP requirements

44

SEMAP File Sampling

• Useful guideline for determining number of files to review

Universe Minimum Sample Size

50 or less 5

51-6005 plus 1 for each 50 (or part of 50) over 50

601-200016 plus 1 for each 100 (or part of 100) over 600

Over 200030 plus 1 for each 200 (or part of 200) over 2000

45

Conducting File Reviews

• Logistics – Use a file checklist to document your review.

(RIM checklists are a good start.)

– Complete 50058 calculations without looking at the completed calculations in the files.

– Look for consistency between the paper and electronic files.

46

What The Reviews May Show

• Key RIM Review programmatic findings:– Earned income/pension/public assistance

calculations– Incomplete/outdated PHA policies– Data not being transmitted to PIC– Lack of verification of income and deductions– File data doesn’t match data used to calculate

rent– Improper calculation of utility allowances

47

What the Reviews May Show

• RIM reviews also found errors related to automation.

• In general, these errors include:– Programming problems(software vendor) – Errors related to the way PHAs use

automated systems such as:• Not attending to error reports and• Inadequate training for staff on the system

• See page 6-32 for details

48

Conducting the “General” Portion of a Periodic Review

• Periodic Reviews enable PHAs to take the same broad look that HUD takes during a RIM Review.

• The agenda:– Look at policy and procedural documents– Conduct an expanded file review

• HUD’s RIM checklists are good models.

C. Admin Plans H. Rent and HAP

D. Payment Standards I. Reexaminations

E. Applications J. HUD 50058

F. Income

49

Conducting the “General” Portion of the Review

• You may need to add to HUD’s checklists:– Questions covering PHA-specific policies or

performance standards– Protocols for checking how PHA-specific

software handles income, rent and subsidy determinations.

50

Using Reports to Track Quality

• Perspective is important.– Identifying individual errors is important but

it’s only one aspect of QC.– Looking at trends (performance over time)

provides important insights that lead to error reduction.

– Looking at trends can also help set performance standards.

51

Using QC Reports to Track Quality

• What should be tracked regularly?– How often things happen– How long things take– Whether improvement is being made– How individuals are doing.

• Caution: Tracking reports can raise flags but may not answer the question: “Why?”

52

Analyzing Data

• Pages 6-36 – 6-38.• Study the three reports

– On-time reexaminations– Transactions by type– Error rate trends

• What flags does each report raise?• What additional information would you like

to have?

53

Addressing Staff Capacity Issues

• Error reduction will not happen without staff capacity.

• No one is born knowing the rules of assisted housing! Most PHAs must grow their own “experts.”

• Staff development is a joint responsibility of the organization and the individual.

• Capacity building begins with a mutual understanding of both the job requirements and current performance.

54

Addressing Staff Capacity

• A good QC program can support staff development efforts if it includes an analysis of: – The knowledge and abilities that each job

requires – The actual knowledge and abilities of the staff.

• And, if the PHA uses this information to create:– A strategy for addressing any gaps in capacity– A plan for developing staff and monitoring staff

progress in meeting performance goals.

55

Addressing Staff Capacity

• Skills Set is the term used to describe what people need to know, understand, and be able to do.

• See pages 6-40 & 6-41 for a list of the skill sets required by staff who perform income and rent determinations.

• Review the list briefly. Do you think these items represent the skills needed by staff at your PHA?

56

Addressing Staff Capacity

• How to use your analysis of skills sets:– Your analysis can improve your recruiting

efforts and hiring decisions– Be aware of skill gaps when considering

changes in staff roles or position – Use your understanding of required skills vs.

available staff skills to help diagnose performance problems

– Use your understanding of skill sets to select the right employees and to develop employee training plans.

57

Addressing Staff Capacity

• Consider using Individual Development Plans to:– Help set expectations– Demonstrate a commitment to staff

development• See page 6-49 for a sample IDP. Note how it

sets expectations and specifies development activities that will help the employee succeed.– Goal #2: Employee will become proficient….– Goal #2, Activity 3. Attend a 3-day occupancy

training program.

58

Communicating About Error Reduction and QC

• The Challenge:– Reducing errors requires staff commitment.– A QC program must be seen as a positive

activity to staff as well as management.• To succeed, staff need to know:

– What good performance is– How it will be measured– How they are doing as performers.

59

Communicating About Error Reduction and QC

• Performance expectations help staff understand what is required. They generally relate to:– Quality and Quantity of Work– Timeliness– Customer Service

• Quality control program helps managers to:– Set expectations – Share information about actual performance in

comparison to goals– Recognize and reward good and improving

performance

60

Communicating About Error Reduction and QC

• Sharing information about performance helps staff keep focused on:– What is important– What needs to be done– What they have accomplished

• Most organizations have too many reports and not enough information.

• Finding effective ways to share this information is challenging.

61

Communicating About Error Reduction and QC

• Considering using “Scorecards” to help staff maintain perspective

• Keep them simple (see page 6-45) and focus on:– Goals and. Progress– Information that makes paperwork real

(household demographics, neighborhoods)– Quality and quantity of work– Progress over time– Accomplishments – things to brag about

62

Communicating About Error Reduction and QC

• Scorecards for individuals also can help reinforce expectations and give feedback to staff.

63

QC Recap

• A good quality control system should:– Be comprehensive but not burdensome– Cover key performance areas– Fairly and accurately measure performance– Be perceived as fair by staff and managers

• Few organizations are perfect. QC will find problems.

• But a good QC program will also show the pathway to improvements.

64

QC Recap

• The next session talks about how QC data can help you to develop strategies for:– Responding to RIM findings– Solving performance problems, and– Reducing errors.