9. Weighting and Weighted Standard Errors

9. Weighting and Weighted Standard Errors

2

Prerequisites• Recommended modules to complete before viewing

this module 1. Introduction to the NLTS2 Training Modules 2. NLTS2 Study Overview 3. NLTS2 Study Design and Sampling NLTS2 Data Sources, either

• 4. Parent and Youth Surveys or• 5. School Surveys, Student Assessments, and Transcripts

6. Implications for Analysis and either• 7. Parent and Youth Surveys or• 8. School Surveys, Student Assessments, and Transcripts

9. Weighting and Weighted Standard Errors

3

Overview Purpose Sampling weights overview Creating NLTS2 sampling weights Sampling weight example Which sampling weight to use Obtaining correct standard errors by correcting for design

effects Type 1 error Analysis recommendations Closing Important information

9. Weighting and Weighted Standard Errors

4

Purpose

• Learn how to obtain valid point estimates by using sampling weights.

• Learn how to obtain valid standard errors by adjusting for “design effects.”

9. Weighting and Weighted Standard Errors

5

Sampling weights overview• Purpose of sampling weights

To produce point estimates that are representative the universe (i.e., national population of students with disabilities in age group; each disability category).

9. Weighting and Weighted Standard Errors Sampling weights overview

• Why do we need to weight? When a population includes some low-incidence groups of

interest, those groups typically are oversampled.• Oversampling ensures that the sample includes some members of

those groups. But unless weighted, that sample does not represent the total population results.

Demographic groups may have differential response rates.

6

9. Weighting and Weighted Standard Errors Sampling weights overview

If all students in a disability group had been equally likely to be selected, then to make results represent the full population, weight each observation by the number of individuals in the population that it represents.

Example using synthetic data for illustrative purposes:

7

2008 Population Sample Weight

Students with learning disability

2,522,735 500 2,522,735/500 = 5,045.47

Students with visual impairment

25,790 500 25,790/500 = 51.58

9. Weighting and Weighted Standard Errors Creating NLTS2 sampling weights

• Example Local education agency (LEA) had 100 students with learning disability.

• Our sample had 10 students with learning disability, drawn with equal probabilities.

• We obtained responses from 5 students with learning disability.• Students’ within-LEA weight would be 100/5 = 20.

The universe of LEAs in the cell served 400,000 students.• Sampled LEAs in a cell, drawn with equal probabilities, served 2,000

students. • Each student in the sampled LEAs represents 400,000/2,000 = 200

students. Students’ total sampling weight would be 20 x 200 = 4,000.

8

9. Weighting and Weighted Standard Errors

9

Creating NLTS2 sampling weights• State schools were treated as a sampling cell. All

were sampled; not all responded. For each disability category, a weight was calculated by

multiplying the number of students with that disability on the rosters of the responding schools by the inverse of the proportion of state schools that submitted rosters.

9. Weighting and Weighted Standard Errors

10

Creating NLTS2 sampling weights• Adjustments

So that weighted sample matches the number of students in each disability category, as reported to OSEP by the states for the 1999–2000 school year.

So that the weighted sample matches known characteristics of the population, such as age group and race/ethnicity.

9. Weighting and Weighted Standard Errors

11

Things to note about NLTS2 samplingweights• The youth is the unit of analysis.

Research questions must pertain to youth only. With the weights provided by NLTS2, data cannot be used

to represent the universe of teachers, classrooms, schools, districts, or states.

• There is clustering as a result of the sample design. Clustering has implications for standard errors. This is

covered a little later.• Results using NLTS2 data must always be weighted for

reporting or publishing.• Weights differ for each wave and each instrument.

9. Weighting and Weighted Standard Errors

12

Sampling weight example

Disability CategoryNumber in

SampleParticipated in

Group ActivitiesWeight for Category

Weighted Valuefor Category

Learning disability 1 14.3

4.3

Speech/language impairment 1 13.0

3.0

Mental retardation 1 11.0

1.0

Emotional disturbance 1 0.8

.0

Hearing impairment 1 1.1

.1

Visual impairment 1 1.1

.1

Orthopedic impairment 1 0.1

.0

Other health impairment 1 1.4

.4

Autism 1 0.1

.0

Multiple disabilities 1 0.1

.0

TOTAL 10 610.0

8.9

Unweighted sample percentage = 60% (Column B total divided by Column A total)

Weighted population estimate = 89% (Column D total divided by Column C total)

Synthetic data for illustrative purposes.

9. Weighting and Weighted Standard Errors

Sampling weights exampleDifferences in weights across waves and data collection instruments

Youth Direct Assessment Parent Interview School Program Teacher SurveyWave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2

ID Wt_na np1Weight n2ParentWt wt_npr1 wt_npr2 wt_nts1 wt_nts2xx1 560.3 381.9 499.9 605.0 454.3xx2 157.5 160.1 314.9 312.0 314.9xx3 137.9 175.9 839.4 442.1 776.7xx4 2,343.6 975.4 1,891.0 3,385.8 2,131.3 2,914.0 2,322.1xx5 560.3 381.9 604.0 655.9 499.9 605.0xx6 87.1 36.2xx7 560.3 381.9 604.0 499.9 454.3xx8 1,132.7 916.0 2,565.3 1,681.1 2,304.9xx9 560.3 381.9 499.9 605.0 454.3x10 130.3 88.1 113.8 381.5 381.5

Individual Student Weights

13Synthetic data for illustrative purposes.

9. Weighting and Weighted Standard Errors

14

Which sampling weight to use?

“Help! I have more than one weight to choose from; which one do I use?”

When combining data from multiple sources (instruments or waves) in analyses that require data from all of those sources, a general rule is to use the weights from the source for which the sum of the weights of the individuals is largest.

When combining data from sources with a lot of nonoverlapping data, proceed with caution and consult a statistician.

9. Weighting and Weighted Standard Errors

15

Correcting standard errors for design effects

Reason: In a multistage sample, multiple observations are drawn a given location. Observations may be correlated.

Sample weights will produce correct point estimates but may produce incorrect standard errors.

Statistical adjustments for clustering “design effects” produce correct standard errors.

9. Weighting and Weighted Standard Errors

16

Correcting standard errors for design effects

• Use Taylor linearization Replicate weights For NLTS2, SRI’s approximation algorithm.

9. Weighting and Weighted Standard Errors

17

Correcting standard errors for design effects:Taylor linearization approach• With Taylor linearization, you specify the strata and

the cluster variables. Stratum variable has 64 values (geographic region, size, and

wealth) Cluster (first-level PSU) variable: LEA

• A problem arises: when there is only one observation per cluster. Solution: Collapse these clusters within their stratum.Generally, we recommend using replicate weights.

9. Weighting and Weighted Standard Errors

18

Correcting standard errors for design effects:Replicate weights• For any data collection activity, NLTS2 provides 32

replicate weights.• Each replicate is calculated as follows:

Half the LEA sample is used. Respondents in selected LEAs are weighted up to the

universe. Differences across replicates reflect true variability in

the full sample.

9. Weighting and Weighted Standard Errors

19

Correcting standard errors for design effects:SRI’s simple approximation

SE = Standard errorM = Mean of the sampling weightsV = Variance of the sampling weights

2

2

25.1MVM

Adjusted SE = SE *

9. Weighting and Weighted Standard Errors

20

Correcting standard errors for design effects:• What does an adjusted standard error buy you?

Accurate confidence intervals around estimates Greater confidence that differences observed are truly

differences in the population

9. Weighting and Weighted Standard Errors

21

Weighted standard errors: Example means and standard errors

Type Mean Standard Error

No weight 82.2 0.32

Sampling weight (erroneously using a frequency weight)

81.9 0.02

Sampling weight and adj. for design effects

81.9 1.20

Randomly selected subset of the NLTS2 data used in all examples

9. Weighting and Weighted Standard Errors

22

Weighted standard errors: Example estimates and p values for gender difference

Type Estimate p Value

No weight 0.43 .56

Sampling weight(erroneously using frequency weight)

1.2 <.001

Sampling weight and adj. for design effects

1.2 .40

Randomly selected subset of the NLTS2 data used in all examples

9. Weighting and Weighted Standard Errors

Examples of statistical packages thatcalculate weighted standard errors• SAS • STATA • WESVAR Taylor linearization or replicate weights• SUDAAN

• SPSS – Taylor linearization only

23

9. Weighting and Weighted Standard Errors

24

Weighted standard errors• Examples of statistical packages that calculate weighted

standard errors NLTS2 training modules use SAS and SPSS for analysis examples. SPSS – General statistical package

• Requires SPSS Complex Samples Module for calculating accurate standard errors

• GUI interface; Taylor linearization SAS – General statistical package

• Requires the SAS Statistics Module for calculating accurate standard errors in PROC SURVEY procedures

• Taylor linearization or replicate weights

9. Weighting and Weighted Standard Errors

25

Type 1 error• Applies to NLTS2 as to any other study• Temptation to perform many tests drastically

increases the probability of spurious results

• Use caution

9. Weighting and Weighted Standard Errors

26

Analysis recommendations

Two analysis recommendations• Start simple!• Before you start more complicated analyses,

Understand who’s in and who’s out of the analyses. Understand the distributions. Understand the bivariate relationships.

9. Weighting and Weighted Standard Errors

27

Closing

• Topics discussed in this module Sampling weights overview Creating NLTS2 sampling weights Sampling weight example Which sampling weight to use Obtaining correct standard errors by correcting for design

effects Type 1 error Analysis recommendations

9. Weighting and Weighted Standard Errors

28

Closing

• Next module: 10. NLTS2 Documentation Overview

9. Weighting and Weighted Standard Errors

29

Important information NLTS2 website contains reports, data tables, and other

project-related information http://nlts2.org/

Information about obtaining the NLTS2 database and documentation can be found on the NCES website http://nces.ed.gov/statprog/rudman/

General information about restricted data licenses can be found on the NCES website http://nces.ed.gov/statprog/instruct.asp

E-mail address: nlts2@sri.com