nonresponse bias in the 2003 survey of small business finances y. michael yang, norc at the...
TRANSCRIPT
Nonresponse Bias in the 2003 Survey of Small Business Finances
Y. Michael Yang, NORC at the University of Chicago
Traci Mach, Federal Reserve Board
Lieu Hazelwood, Federal Reserve Board
Disclaimer
The views expressed herein are those of the authors. They do not necessarily reflect the opinions of the Federal Reserve Board or its staff.
Overview
• Bias due to Nonresponse
• Data Source: 2003 Survey of Small Business Finances
• Analytical Methods for Assessing Bias
• Results
• Discussions
Bias Due to Nonresponse (1)
• Deterministic model
• Probabilistic model
Bias Due to Nonresponse (2)
• Implications– Nonresponse bias is variable-specific;– Nonresponse does not necessarily lead to bias;– Nonresponse bias increases as response rate
decreases;– Nonresponse bias depends on the response
mechanism;– Key is to examine the correlation between
response propensity and the survey variable;
2003 SSBF (1)• Target population
– U.S. for-profit small businesses
• Frame– Dun’s Market Identifier (DMI)
• Sample design– Stratified systematic sampling– Screener nonresponse subsampling– Main interview nonresponse subsampling
• Data collection– Telephone Interview
2003 SSBF (2)1. SSBF Target Population
2. DMI Frame
3. Initial Sample
Sample Selection
4. ScreeningInterview Pass 1
13. ScreeningInterview Pass 2
7. Ineligible 8. Eligible 10. Noncontacts9. Nonresponse
16. Ineligible 17. Eligible
5% Subsample
20. TotalEligible
Subsample
21. ReserveSample
Screening
11. ExpectedIneligible
12. ExpectedEligible
18. ExpectedIneligible
19. ExpectedEligible
15. Elig. Unknown(Nonresponse)
5. Eligibility Known6. EligibilityUnknown
14. EligibilityKnown
22. Main InterviewPass 1
Subsample
25. Main InterviewPass 2
24. Incompletes
26. Completes28. Total
Completes
Main Interview
23. Completes
27. Incompletes
2003 SSBF (3)
• Scope of analysis– Eligible screener respondents (n=6,520)– Focus on bias due to main interview
nonresponse– Adjustments for screener nonresponse and
subsampling have been incorporated in the base weight
Analytical Methods
• Comparing response rates across subgroups;• Comparing early responders and late
responders on survey variables;• Comparing first phase responders and follow-
up responders on survey variables;• Comparing unadjusted and adjusted
estimates;• Estimate nonresponse bias analytically
Main Interview Outcome Rates
0
10
20
30
40
50
60
Complete Refusal Noncontact Other
Response Category
Per
cent
of S
ampl
e
Unweighted % Weighted %
Outcome Rates by Business Size: Unweighted
0
20
40
60
80
100
Response Rate Contact Rate Cooperation Rate Refusal Rate
0 - 19 20 - 49 50 - 99 100 - 499
Outcome Rates by Business Size: Weighted
0
20
40
60
80
100
Response Rate Contact Rate Cooperation Rate Refusal Rate
0 - 19 20 - 49 50 - 99 100 - 499
Outcome Rates by Urbanicity: Unweighted
0
20
40
60
80
100
Response Rate Contact Rate Cooperation Rate Refusal Rate
Rural Urban
Outcome Rates by Urbanicity: Weighted
0
20
40
60
80
100
Response Rate Contact Rate CooperationRate
Refusal Rate
Rural Urban
Early (<10 calls) Vs. Late (>=10 calls) Completes: Unweighted (1)
0 20 40 60 80 100
White
Minority or Hispanic
Mortgage loans
Motor vehicle loans
Equipment loans
Capital leases
Other loans
Line of credit
Early Complete Late Complete
Early (<10 calls) Vs. Late (>=10 calls) Completes: Unweighted (2)
0 10 20 30 40 50 60
Transaction svcs
Cash management svcs
Credit related svcs
Trust svcs
Brokerage svcs
Card processing svcs
Early Complete Late Complete
Early (<10 calls) Vs. Late (>=10 calls) Completes: Unweighted (3)
$0
$1,000,000
$2,000,000
$3,000,000
$4,000,000
$5,000,000
$6,000,000
$7,000,000
Total sales Profit Total Assets
Early Complete Late Complete
Early (<10 calls) Vs. Late (>=10 calls) Completes: Weighted (1)
0 20 40 60 80 100
White
Minority or Hispanic
Mortgage loans
Motor vehicle loans
Equipment loans
Capital leases
Other loans
Line of credit
Early Complete Late Complete
Early (<10 calls) Vs. Late (>=10 calls) Completes: Weighted (2)
0 10 20 30 40 50
Transaction svcs
Cash management svcs
Credit related svcs
Trust svcs
Brokerage svcs
Card processing svcs
Early Complete Late Complete
Early (<10 calls) Vs. Late (>=10 calls) Completes: Weighted (3)
$0
$200,000
$400,000
$600,000
$800,000
$1,000,000
$1,200,000
Total sales Profit Total Assets
Early Complete Late Complete
Early (<19 calls) Vs. Late (>=19 calls) Completes: Unweighted (1)
0 20 40 60 80 100
White
Minority or Hispanic
Mortgage loans
Motor vehicle loans
Equipment loans
Capital leases
Other loans
Line of credit
Early Complete Late Complete
Early (<19 calls) Vs. Late (>=19 calls) Completes: Unweighted (2)
0 10 20 30 40 50 60
Transaction svcs
Cash management svcs
Credit related svcs
Trust svcs
Brokerage svcs
Card processing svcs
Early Complete Late Complete
Early (<19 calls) Vs. Late (>=19 calls) Completes: Unweighted (3)
$0
$1,000,000
$2,000,000
$3,000,000
$4,000,000
$5,000,000
$6,000,000
Total sales Profit Total Assets
Early Complete Late Complete
Early (<19 calls) Vs. Late (>=19 calls) Completes: Weighted (1)
0 20 40 60 80 100
White
Minority or Hispanic
Mortgage loans
Motor vehicle loans
Equipment loans
Capital leases
Other loans
Line of credit
Early Complete Late Complete
Early (<19 calls) Vs. Late (>=19 calls) Completes: Weighted (2)
0 10 20 30 40 50
Transaction svcs
Cash management svcs
Credit related svcs
Trust svcs
Brokerage svcs
Card processing svcs
Early Complete Late Complete
Early (<19 calls) Vs. Late (>=19 calls) Completes: Weighted (3)
$0
$200,000
$400,000
$600,000
$800,000
$1,000,000
$1,200,000
Total sales Profit Total Assets
Early Complete Late Complete
Main Sample Vs. Follow-up Subsample Completes: Unweighted (1)
0 20 40 60 80 100
White
Minority or Hispanic
Mortgage loans
Motor vehicle loans
Equipment loans
Capital leases
Other loans
Line of credit
Main Sample Complete Subsample Complete
Main Sample Vs. Follow-up Subsample Completes: Unweighted (2)
0 10 20 30 40 50 60
Transaction svcs
Cash management svcs
Credit related svcs
Trust svcs
Brokerage svcs
Card processing svcs
Main Sample Complete Subsample Complete
Main Sample Vs. Follow-up Subsample Completes: Unweighted (3)
$0
$1,000,000
$2,000,000
$3,000,000
$4,000,000
$5,000,000
$6,000,000
Total sales Profit Total Assets
Main Sample Complete Subsample Complete
Main Sample Vs. Follow-up Subsample Completes: Weighted (1)
0 20 40 60 80 100
White
Minority or Hispanic
Mortgage loans
Motor vehicle loans
Equipment loans
Capital leases
Other loans
Line of credit
Main Sample Complete Subsample Complete
Main Sample Vs. Follow-up Subsample Completes: Weighted (2)
0 10 20 30 40 50
Transaction svcs
Cash management svcs
Credit related svcs
Trust svcs
Brokerage svcs
Card processing svcs
Main Sample Complete Subsample Complete
Main Sample Vs. Follow-up Subsample Completes: Weighted (3)
$0
$200,000
$400,000
$600,000
$800,000
$1,000,000
$1,200,000
Total sales Profit Total Assets
Main Sample Complete Subsample Complete
Indirect Estimate of Bias
• (unadjusted estimate)-(adjusted estimate)
• Measures the amount of nonresponse adjustment;
• Measures nonresponse bias if the adjusted estimate is unbiased;
Indirect Estimates of BiasCharacteristics Estimated Bias Characteristics Estimated Bias
Native American 0.14 Equipment loans -1.27
White 0.07 Capital leases -0.18
White non-Hispanic 0.24 Other loans -0.38
Asian -0.13 Line of credit -0.65
Asian or Pacific Islander -0.1 Transaction service -0.57
Black 0.04 Cash management service -0.51
Pacific Islander 0.03 Credit related service -0.53
Hispanic -0.12 Trust service -0.71
Owner managed 0.77 Brokerage service -0.16
Average owners education 0.02 Card processing service -0.33
Firm age 0.07 Total sales -158570.05
Minority or Hispanic -0.1 Profit -20781.18
Total employment -1.01 Total Assets -77824.28
Mortgage loans 0.07 D&B credit percentile -0.96
Motor vehicle loans -1.02 Total employees -1.31
Direct Estimate of Bias• Estimate response propensity
• Compute the covariance between response propensity and survey variable;
• Estimate bias analytically
Direct Estimates of Bias (1)Survey Variable Covariance Estimated Bias Weighted Estimate Bias Ratio
Native American 0.01 0.02 1.41 1.4%
White 0.08 0.16 90.82 0.2%
White non-Hispanic 0.15 0.31 86.43 0.4%
Asian -0.05 -0.11 4.17 -2.6%
Asian or Pacific Islander -0.06 -0.11 4.29 -2.6%
Black -0.04 -0.07 3.63 -1.9%
Pacific Islander 0 0 0.12 0.0%
Hispanic -0.09 -0.18 4.28 -4.2%
Owner managed 0.15 0.29 94.16 0.3%
Average owners education 0.01 0.01 4.61 0.2%
Firm age -0.03 -0.05 14.3 -0.3%
Minority or Hispanic -0.17 -0.35 13.31 -2.6%
Total employment -1.91 -3.85 8.73 -44.1%
Mortgage loans -0.11 -0.22 13 -1.7%
Motor vehicle loans -0.28 -0.57 25.33 -2.3%
Direct Estimates of Bias Bias (2)
Survey Variable Covariance Estimated Bias Weighted Estimate Bias Ratio
Equipment loans -0.27 -0.54 10.59 -5.1%
Capital leases -0.21 -0.42 8.9 -4.7%
Other loans -0.21 -0.41 10.08 -4.1%
Line of credit -0.53 -1.08 32.45 -3.3%
Transaction service -0.4 -0.8 38.96 -2.1%
Cash management service -0.55 -1.11 6.43 -17.3%
Credit related service -0.25 -0.5 4.82 -10.4%
Trust service -0.45 -0.92 17.62 -5.2%
Brokerage service -0.05 -0.11 5.45 -2.0%
Card processing service -0.16 -0.31 36.67 -0.8%
Total sales -331788.2 -669119.94 1,177,572 -56.8%
Profit -31853.62 -64239.45 175,965 -36.5%
Total Assets -156104.5 -314817.2 535,133 -58.8%
Direct vs. Indirect Bias Estimates
-2
-1.5
-1
-0.5
0
0.5
Indirect Direct
Discussion• There is significant correlation between nonresponse
and key survey variables– Larger and urban businesses have lower response rate;– Larger and urban businesses use more services, have more
sales, assets, and profit
• Unadjusted estimates of the mean/total are biased downward for most variables
• Response propensity-based weighting appears to be effective, but only a portion of the bias is corrected
• Weighting method may be improved in future surveys
