aggregate reporting and data disclosure avoidance techniques

2012 SLDS P-20W Best Practice Conference 1

AGGREGATE REPORTING AND DATA DISCLOSURE

AVOIDANCE TECHNIQUES

Monday, October 29,2012

Kim Nesmith, Louisiana Department of Education

Adrian Peoples, Delaware Department of Education

Baron Rodriguez, Privacy Technical Assistance Center

2012 SLDS P-20W Best Practice Conference

• Louisiana Process and Types of Suppression

• Delaware Public Reporting Rules and Strategy

• Contact Information and Resources

OVERVIEW

2

2012 SLDS P-20W Best Practice Conference 3

LOUISIANA

2012 SLDS P-20W Best Practice Conference 4

• Determining a “n” size

• Determining a percentage threshold

• Limiting student Level Reports and establishing MOUs

FIRST STEPS

2012 SLDS P-20W Best Practice Conference 5

• Determining what is most important

• Determining how to handle complementary suppression

NEXT STEPS

2012 SLDS P-20W Best Practice Conference 6

• If you can “back into a number”, the suppression is not effectiveo When only one number in a row or column is

suppressed and the total is present

o When all suppressed numbers are 0s and the total is present

o If numerator, denominator, and percentage are all present

COMPLEMENTARY SUPPRESSION

2012 SLDS P-20W Best Practice Conference 7

N SIZE EXAMPLE

Scholarship School Name Enrollment by Grade

K 1 2 3 Total

School A 16 8 8 6 38

School B 15 0 0 0 15

School C 3 0 0 0 3

School D 62 20 13 15 110

School E 3 1 0 1 5

School F 31 22 8 15 76

School G 14 14 15 12 55

TOTAL 144 65 44 49 302

2012 SLDS P-20W Best Practice Conference 8

N SIZE EXAMPLE

Scholarship School Name Enrollment by Grade

K 1 2 3 Total

School A 16 <10 <10 <10 38

School B >=10 <10 <10 <10 15

School C <10 <10 <10 <10 <10

School D 62 20 13 15 110

School E <10 <10 <10 <10 <10

School F >=30 >=20 <10 >=10 76

School G 14 14 15 12 55

TOTAL 144 65 44 49 302

2012 SLDS P-20W Best Practice Conference 9

PERCENT EXAMPLE

LEA Name

All Students Special Ed.Dropouts Total Rate

Dropouts Total Rate

District A 61 2,499 2.4% 12 265 4.5%

District B 10 1,210 0.8% 2 94 2.1%

District C 45 5,919 0.8% 6 457 1.3%

District D 34 1,167 2.9% 3 93 3.2%

District E 7 388 1.8% 4 41 9.8%

District F 23 409 5.6% 2 22 9.1%

2012 SLDS P-20W Best Practice Conference 10

PERCENT EXAMPLE

LEA Name

All Students Special Ed.Dropouts Total Rate

Dropouts Total Rate

District A 61 2,499 2.4% 12 265 4.5%

District B >10 >1,210 <1% <10 >90 2.1%

District C >40 >5,910 <1% <10 >450 1.3%

District D 34 1,167 2.9% <10 >90 3.2%

District E <10 >380 1.8% <10 >40 9.8%

District F 23 409 5.6% <10 >20 9.1%

2012 SLDS P-20W Best Practice Conference 11

• Talking with the requestor

• Creative solutions

MAINTAINING TRANSPARENCY

2012 SLDS P-20W Best Practice Conference 12

DELAWARE

2012 SLDS P-20W Best Practice Conference 13

Rule of X• Delaware masks all data for a particular

demographic if its group size is less than or equal to X

• 15 for Assessment, Enrollment, Teacher Quality

• 40 for Accountability

5/95 Rule• If demographic performance is calculated

to be either at or below 5% OR at or above 95%, Delaware masks the data.

DELAWARE PUBLIC REPORTING RULES

2012 SLDS P-20W Best Practice Conference

large images

14

Database

Data

Application

STRATEGY: LEVEL OF IMPLEMENTATION

2012 SLDS P-20W Best Practice Conference 15

Data

Level of Maintenance

Database

Application

MAINTENANCE GRADIENT

2012 SLDS P-20W Best Practice Conference 16

Data Suppression• DO NOT SHOW data to any constituent group

(e.g. public, administrators, teachers, etc.)• DO NOT ALLOW aggregate data to be used as

input to any data-driven decision-making

Effect Suppression• SHOW data to appropriate constituent group

(e.g. public, administrators, teachers, etc.)• DO NOT ALLOW aggregate data to be used as

input to any data-driven decision-making

STRATEGY: DATA VS. EFFECT SUPPRESSION

2012 SLDS P-20W Best Practice Conference 17

IMPLEMENTATION EXAMPLE: DATABASE LEVEL/DATA

SUPPRESSION

2012 SLDS P-20W Best Practice Conference 18

IMPLEMENTATION EXAMPLE:DATABASE LEVEL/DATA

SUPPRESSION

2012 SLDS P-20W Best Practice Conference 19

IMPLEMENTATION EXAMPLE: APPLICATION LEVEL/EFFECT

SUPPRESSION

2012 SLDS P-20W Best Practice Conference 20

BIGGEST PITFALL: INCONSISTENT

IMPLEMENTATIONPolicy

Accountability Assessment Enrollment Teacher Quality

• Small constant team• Long history

• One point of contact• Both policy and data

• Multiple transient contractors

• Newcontractor

• Bringing new reports to the public

2012 SLDS P-20W Best Practice Conference

• PTAC State-by-State analysis of public reports:

2PM today in the Burnham room. Please send a representative from your state to receive your sealed copy!

• Case Study 5: Minimizing Access to PII…

• Data De-identification: A Glossary of Terms

RESOURCES/SESSIONS

21

2012 SLDS P-20W Best Practice Conference 22

Frequently Asked Questions:

1. If I am only publishing aggregate data tables, do I still need to be concerned about disclosure avoidance?

2. What issues should educational agencies and institutions consider to successfully balance privacy protection requirements with data disclosure requirements?

3. Is public reporting of data for small groups (“small cells”) the same thing as a disclosure?

4. What standard is used to evaluate disclosure risk? 5. Does the U.S. Department of Education require

educational agencies and institutions to use specific data disclosure avoidance techniques?

6. And many more…

PTAC GUIDANCE FAQ’S

2012 SLDS P-20W Best Practice Conference

Contact information:Adrian Peoples, apeoples@doe.k12.de.us

Kim Nesmith, kim.nesmith@la.gov

Baron Rodriguez, Baron.Rodriguez@aemcorp.com

For more information on Aggregate Reporting:Resource 1: Presentation: Protection of Personally Identifiable Information through Disclosure Avoidance Techniques

Resource 2: PTAC Privacy Toolkit – Case Studies, etc.

Resource 3: Tech Brief #3: Statistical Methods for Protecting Personally Identifiable Information in Aggregate Reporting (DRAFT; Dec 2010)

CONTACTS & ADDITIONAL RESOURCES

23