session #65 draft results of quality assurance program data from award year 2006-07 david rhodes and...
TRANSCRIPT
![Page 1: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/1.jpg)
Session #65
Draft Results of Quality Assurance Program Data from
Award Year 2006-07
David Rhodes and Anne Tuccillo
![Page 2: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/2.jpg)
2
Goals• Share draft results of program-
wide analysis of 2006-07 data
• Illustrate additional ways to analyze ISIR data
![Page 3: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/3.jpg)
3
Background• Quality Assurance (QA) Program
• Participants granted regulatory flexibility to decide which ISIR data they verify
• ISIR Analysis Tool
![Page 4: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/4.jpg)
4
The 2006-07 Analysis• 146 Quality Assurance Program
institutions• Each school drew a random
sample of at least 350 applicants• Each school verified the ISIR
information for the entire random sample of applicants
• We analyzed 68,077 records
![Page 5: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/5.jpg)
5
Previous data collections
2005-06 2004-05
Population
ISIR Records subject to school verification
Random sample of all applicants
Number of schools
140 133
![Page 6: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/6.jpg)
6
See a pattern?• Alternating focus
– Random sample of all applicants– Institutionally verified applicants
• This year (2007-08)– All QA Program institutions– ISIRS subject to institutional
verification
![Page 7: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/7.jpg)
7
Is analysis over time appropriate?
• For individual schools, yes with care
• For program-wide analyses the answer “should” be no, but…..
![Page 8: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/8.jpg)
8
When Looking Across Years Keep in Mind:
• The two different populations• Changes to institutional verification
criteria between years• Other differences between award
years• Percents and averages not counts
![Page 9: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/9.jpg)
9
Why we “shouldn’t” • QA schools supplying data differ
slightly from year to year (146 ≠140 ≠133)
• Unmeasured differences in institutional verification across years
![Page 10: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/10.jpg)
10
Key areas of analysis• Description of population• Critical ISIR fields• How changes affect aid eligibility• Improper payments in the Pell
Grant program
![Page 11: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/11.jpg)
11
Characteristics of Aid Applicants at QA Program Schools 2006-07
62%
44%
40%
58%
0% 25% 50% 75% 100%
Dependent
Selected for School Verification
Selected for CPS Verification
Eligible for Pell
![Page 12: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/12.jpg)
12
Dependent and independent record counts over time
42,455
94,866
24,310
25,622
42,297
14,785
0
25,000
50,000
75,000
100,000
125,000
150,000
2006-07 2005-06 2004-05
Dependent Independent
![Page 13: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/13.jpg)
13
Dependent and independent percentages over time
62% 69% 62%
38% 31% 38%
0%
25%
50%
75%
100%
2006-07 2005-06 2004-05
Dependent Independent
![Page 14: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/14.jpg)
14
Percentage of dependent applicants with changes to the most commonly changed ISIR Fields: 2006-07
22%
23%
26%
29%
29%
0% 25% 50% 75% 100%
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total fromWorksheet B
Parent AGI
![Page 15: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/15.jpg)
15
Percentage of independent applicants with changes to the most commonly changed ISIR Fields: 2006-07
11%
12%
14%
18%
19%
0% 25% 50% 75% 100%
Student Total from Worksheet C
Student Tax Filing Status
Student Total from Worksheet B
Student AGI
Student Income from Work
![Page 16: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/16.jpg)
16
Changes to critical ISIR fields among dependent students over time
29% 29%33%27%30% 32%
0%
25%
50%
75%
100%
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06 2004-05
![Page 17: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/17.jpg)
17
Changes to critical fields among independent students over time
18% 14%25%
15%19% 14%
0%
25%
50%
75%
100%
Student AGI Student Total fromWorksheet B
2006-07 2005-06 2004-05
![Page 18: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/18.jpg)
18
Dependent records: percent of ISIR fields experiencing an EFC change - 2006-07
87%
89%
88%
90%
89%
0% 25% 50% 75% 100%
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
![Page 19: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/19.jpg)
19
Independent records: Percent of ISIR fields experiencing an EFC change - 2006-07
71%
62%
68%
68%
66%
0% 25% 50% 75% 100%
Student Total fromWorksheet C
Student Tax Filing Status
Student Total fromWorksheet B
Student AGI
Student Income from Work
![Page 20: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/20.jpg)
20
Comparison of changes to EFC among dependent students with a change to the indicated ISIR field
89% 90%88% 90%91% 93%
0%
25%
50%
75%
100%
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06 2004-05
![Page 21: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/21.jpg)
21
Comparison of changes to EFC among independent students with a change to the indicated ISIR field
66% 68%67% 66%71% 72%
0%
25%
50%
75%
100%
Student AGI Student Total fromWorksheet B
2006-07 2005-06 2004-05
![Page 22: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/22.jpg)
22
Percentage of dependent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07
43%
42%
46%
38%
49%
0% 25% 50% 75% 100%
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
![Page 23: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/23.jpg)
23
Percentage of independent records with a change to the indicated ISIR field and a change to a Pell Grant: 2006-07
35%
31%
34%
38%
36%
0% 25% 50% 75% 100%
Student Total fromWorksheet C
Student Tax Filing Status
Student Total fromWorksheet B
Student AGI
Student Income from Work
![Page 24: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/24.jpg)
24
Comparison of change to a Pell Grant among dependent students with a change to the indicated field
49%38%
58% 54%42%
35%
0%
25%
50%
75%
100%
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06 2004-05
![Page 25: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/25.jpg)
25
Comparison of change to Pell Grants among independent students with a change to the indicated field
38% 34%48% 45%
37% 34%
0%
25%
50%
75%
100%
Student AGI Student Total fromWorksheet B
2006-07 2005-06 2004-05
![Page 26: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/26.jpg)
26
Average EFC change among dependent records with a change to the indicated ISIR field: 2006-07
1,039
913
645
819
648
0 400 800 1,200
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
![Page 27: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/27.jpg)
27
Average EFC change among independent records with a change to the indicated ISIR field: 2006-07
470
490
387
186
100
0 500 1,000 1,500
Student Total from Worksheet C
Student Tax Filing Status
Student Total from Worksheet B
Student AGI
Student I ncome from Work
![Page 28: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/28.jpg)
28
Average EFC change among dependent students with a change to the indicated field over time
648819
980
1,214
0
500
1,000
1,500
Parent AGI Parent Total fromWorksheet B
2006-07 2005-06
![Page 29: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/29.jpg)
29
Average EFC change among independent students with a change to the indicated field over time
103
387371
591
0
500
1,000
1,500
Student AGI Student Total fromWorksheet B
2006-07 2005-06
![Page 30: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/30.jpg)
30
Average Pell change for dependent records with a change to the indicated ISIR field: 2006-07
-$344
-$305
-$257
-$314
-$270
-$600 -$400 -$200 $0
Parent Tax Filing Status
Father I ncome from Work
Mother I ncome from Work
Parent Total from Worksheet B
Parent AGI
![Page 31: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/31.jpg)
31
Average Pell change for independent records with a change to the indicated ISIR field: 2006-07
-$368
-$347
-$257
-$122
-$102
-$600 -$400 -$200 $0
Student Total from Worksheet C
Student Tax Filing Status
Student Total from Worksheet B
Student AGI
Student I ncome from Work
![Page 32: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/32.jpg)
32
Average Pell Grant change for dependent records with a change to the indicated field over time
-$270-$314
-$469-$504
-$600
-$400
-$200
$0
Parent AGIParent Total from
Worksheet B
2006-07 2005-06
![Page 33: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/33.jpg)
33
Average Pell Grant change for independent records with a change to the indicated field over time
-$122
-$257
-$344
-$459
-$600
-$400
-$200
$0
Student AGIStudent Total from
Worksheet B
2006-07 2005-06
![Page 34: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/34.jpg)
34
Potential improper payments in Pell Grants prior to verification at QA Schools: 2006-07
6.7% potential under-payments
9.1% potential over-payments
84.2% Pell Grant dollars not at risk
prior to verification
![Page 35: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/35.jpg)
35
Improper Payments in Pell Grants before and after school verification
-9.1%
6.7%
-2.4%-0.7%
2.9%
-3.6%
-10.0%
-5.0%
0.0%
5.0%
10.0%
Potentialunder-awards
Perc
en
t o
f P
ell D
ollars
Before verification After school verification
Potenial over-awards
Net potenial over-awards
![Page 36: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/36.jpg)
36
Improper Payments in Pell Grants before and after CPS verification
-9.1%
6.7%
-2.4%
2.4%4.1%
-1.7%
-10.0%
-5.0%
0.0%
5.0%
10.0%
Potentialunder-awards
Perc
en
t o
f P
ell D
ollars
Before verification After CPS verification
Potenial over-awards
Net potenial under-awards
![Page 37: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/37.jpg)
37
Lessons Learned• While verification is crucial in some
cases, many ISIR records do not experience a meaningful change from their initial transaction value
• QA school verification procedures target larger EFC and Pell changes
• QA school verification reduces improper payment risks in Pell
![Page 38: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/38.jpg)
38
What Do the Findings Mean for Schools and FSA?
Given that relatively few ISIR records experience a change that affects eligibility for need-based aid, verification efforts should strive to focus on the records that matter.
![Page 39: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/39.jpg)
39
Two Ways to Focus• Look for the records that matter• Look for the records that don’t
![Page 40: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/40.jpg)
40
Suggested Additional Analysis
Look for ways to reduce the number of school verified records that experience no or only trivial changes to aid eligibility after verification
![Page 41: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/41.jpg)
41
Additional Analysis• ISIR Analysis Tool reports• Exporting data from the Tool
![Page 42: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/42.jpg)
42
ISIR Analysis Tool• Key Filters
–EFC change < 400 & EFC change > -400
–EFC change > 400 –EFC change < -400–Institutional verification criteria
![Page 43: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/43.jpg)
43
ISIR Analysis Tool• Key Reports
–Sample summary–Field Increment (EFC or AGI)
• Drill down to refine understanding
![Page 44: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/44.jpg)
44
New Analytic “Recipe”
See handout
![Page 45: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/45.jpg)
45
Exporting Data
See handout
![Page 46: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/46.jpg)
Institutional Profiles
Comparison of Program-wide and school specific
data
![Page 47: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/47.jpg)
47
2006-07 Institutional Profile
Example of a 2006-2007 Institutional Profile
![Page 48: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/48.jpg)
48
Profile Differs from Year to Year
• 2005-06 provided data of improper payments PREVENTED by institutional verification
• 2006-07 provides data on estimated levels of POTENTIAL improper payments in a schools applicant population
![Page 49: Session #65 Draft Results of Quality Assurance Program Data from Award Year 2006-07 David Rhodes and Anne Tuccillo](https://reader030.vdocument.in/reader030/viewer/2022032605/56649e7d5503460f94b7ff5d/html5/thumbnails/49.jpg)
49
Contact Information
We appreciate your feedback and comments. We can be reached at: