february 2020 evaluation reportcarolinasmallbusiness.org/.../02/csbdf-2019-survey...carolina small...

FEBRUARY 2020 EVALUATION REPORT

2019 Loan Client Survey Cycle Analysis of Generalizability

[Grab your reader’s attention with a great quote from the document or use this space to emphasize a key point. To place this text box anywhere on the page, just drag it.]

About the Research Program This product has been created as part of Carolina Small Business’s commitment to provide innovative and objective research on issues of preeminent concern for policy leaders, academic thought leaders, development practitioners, and small firm entrepreneurs. To learn more, visit carolinasmallbusiness.org/research.

For More Information

Jamie McCall Vice President of Policy & Research Carolina Small Business Development Fund [email protected]

Carolina Small Business Development Fund

3128 Highwoods Boulevard, Suite 170 Raleigh, N.C. 27604

telephone 919.803.1437 fax 919.897.8612

www.carolinasmallbusiness.org

mailto:[email protected]

of 19

Overview Carolina Small Business Development Fund (CSBDF) conducts yearly surveys of lending clients. Each year’s survey includes clients receiving loans within the last two calendar years. For 2019, that means the survey instrument invited responses from clients that were issued CSBDF loans in calendar years 2017 and 2018. There were 195 individual borrowers surveyed about 208 loans issued during this period. A total of 101 responses were received, which is a response rate of 52%. Based on meta-analyses of similar types of surveys, this response rate exceeds expectations. Meta-analysis research on organizational surveys suggests an expected response of between 31%1 and 36%2.

The goal of this analysis is to assess the extent to which responses from the 2019 survey can be generalized to CSBDF’s population of lending clients. CSBDF’s entire population of lending clients (724 loans through June 2019) was not surveyed for two reasons. First, the extant literature has long noted it is difficult to assess economic impacts through surveys. Due to the bounded rationality of clients, this problem is likely to be more severe when surveyed about development interventions that occurred more than 2 years ago.3 Second, methodology research suggests that surveying clients more frequently could lead to survey fatigue and cause systemic issues related to data reliability and accuracy.4 However, if the characteristics of survey respondents are materially similar to the characteristics of survey non-respondents, then it is likely that the findings of the survey broadly apply to all CSBDF lending clients.

From inception through June 2019, CSBDF had issued 724 loans to 577 unique business clients. If the 2019 survey data are generalizable, confidence interval analysis suggests the results can be applied to all lending clients plus or minus a 8.9% margin of error.5 We assess generalizability through two ways. First, we consider whether there are statistically significant differences between the individual characteristics of client respondents and non-respondents. Second, we build a basic model to determine whether these characteristics are reliable predictors of response rates. The desired result is a lack of statistically significant differences between survey respondents and non-respondents. The more the group of respondents resembles non-respondents, the more confident we can be about the generalizability of the survey data.

Analysis

Table 1 assesses whether characteristics of respondent clients is materially similar

to the characteristics of non-respondent clients. When examining loan or borrower

characteristics one at a time, the respondent group is generally not dissimilar from the

non-respondent group. However, there are a few notable exceptions. Survey respondents

tended to be recipients of loans for significantly higher amounts ($104k) versus non-

respondents ($77k). While it is important to keep this in mind when assessing the survey

results, it does not represent a material flaw to generalizability. The 2019 survey assessed

of 19

respondents for loans closed in 2017 and 2018. The average loan for the survey period

was $103,682 (2017-2018) but $72,781 for earlier time periods (2011-2016). While

CSBDF’s average loan amount by calendar year has varied, it has generally trended

upwards. Respondents in the survey period were simply far more likely to have received

loans for higher dollar amounts.

There were two other major dissimilarities between respondents and non-

respondents. First, clients with a negative payment status (being delinquent or having a

charged off loan) are more likely to respond. Although the survey process encouraged

participation from this subset of borrowers, it is unsurprising that those with a negative

payment status are less likely to respond. Second, respondents were more likely to have

lower levels of current employment (3.7 FTEs) than non-respondents (5.3 FTEs). While

this is a notable difference, the finding is likely an artifact of how CSBDF has historically

collected data. Current levels of employment were not captured at regular intervals until

recent years, so mean employment for clients in the 2011-2016 period is likely being

skewed by missing data. For all of the other 9 variables tested, there is no meaningful

difference between respondents and non-respondents.

0

40

80

120

160

2011 2012 2013 2014 2015 2016 2017 2018

Th

ou

san

ds

of 19

Metric Respondents Non-Respondents Differences? Loan Characteristics Loan Amount (Mean) $104,632 $77,683 Yes6, p < 0.05 Negative Payment Status (%) 10% 24% Yes7, p < 0.01 Business Characteristics Startup Firm (%) 45% 38% No8 Current Employment (FTEs) 3.7 5.3 Yes9, p < 0.05 Jobs Created (FTEs) 2.1 3.2 No10 Jobs Retained (FTEs) 1.8 1.7 No11 Demographic Characteristics Minority-Owned Firm (%) 58% 58% No12 Women-Owned Firm (%) 35% 39% No13 Veteran-Owned Firm (%) 17% 19% No14 Geographic Characteristics (County-Level) Rural Location (%) 37% 36% No15 Economically Distressed (%) 31% 39% No16 Persistent Poverty (%) 8% 8% No17

Although the data suggest survey respondents are substantially similar to all other

types of CSBDF lending clients, it is important to also consider possible variable

interactions. While Table 1 considered each listed variable in isolation through a

statistical test, Table 2 examines all variables in Table 1 through a single model.18

Omnibus testing suggests that – when considering all variables at once – response rates

can be somewhat predicted by the below characteristics. Similar to Table 1, the model

suggests that having a higher loan amount and a positive payment status increases the

likelihood of responding to the survey. Unlike Table 1, the model suggests no impact on

response rates based on the businesses level of current employment (when controlling for

all variables). Finally, when controlling for all other factors, two additional variables

appear to increase chances of responding. Respondents with higher levels of projected job

creation are less likely to respond. Currently, clients with businesses located in more

economically distressed counties are also more likely to respond.

of 19

19

Metric B S.E. Wald Sig. Exp(B)

Loan Characteristics

Loan Amount .000 .000 4.349 .037 1.000

Negative Payment Status 1.039 .464 5.002 .025 2.826

Business Characteristics

Startup Firm .360 .298 1.461 .227 1.433

Current Employment (FTEs) -.028 .030 .877 .349 .972

Jobs Created (FTEs) -.122 .053 5.318 .021 .885

Jobs Retained (FTEs) -.005 .046 .010 .920 .995

Demographic Characteristics

Minority-Owned Firm .297 .284 1.096 .295 1.346

Woman-Owned Firm -.353 .300 1.389 .238 .703

Veteran-Owned Firm -.379 .357 1.124 .289 .685

Geographic Characteristics (County-Level)

Rural Location .376 .358 1.104 .293 1.456

Economically Distressed .706 .297 5.646 .017 2.026

Persistent Poverty .017 .535 .001 .974 1.017

Constant -3.521 1.027 11.756 .001 .030

Conclusions

Analysis of survey response factors is imperfect – the number of variables which

might influence response rates are difficult to capture. Interactions between variables

that might promote or suppresses responses are even more complex. Still, CSBDF

believes that the assessment of impact data collection is an important and vital

component of community economic development. While survey respondents tended to

be recipients of larger loans and have positive repayment statuses, overall, our analysis

suggests the data are broadly generalizable.

To conclude, CSBDF recognizes that surveys are frequently used as a tool to measure impact in community economic development. The use of surveys to measure certain impacts – particularly employment – is not ideal. However, surveys represent a method of data collection that are often more cost effective than other tools like economic impact modeling. At the same time, we note that data collection in this arena is often plagued by low response rates and serious concerns about data validity. Carolina Small Business Development Fund is committed to transparency and accountability in program evaluation, and it is in that spirit that we provide this report as a public research product.

of 19

Appendix: Differences in Respondents and Non-Respondents

Loan Values Independent Samples T-Test (Table 1)

Respondent N Mean Std. Deviation Std. Error

Yes 101 $104,632.21 $184,657.982 $18,374.156

No 623 $77,682.84 $94,842.463 $3,799.783

Levene's Test t-test for Equality of Means

F Sig. Sig.

Mean

Difference

Std. Error

Difference

95% CI of the Difference

Lower Upper

Equal Variance 9.620 .002 .025 $26,949.368 $11,979.346 $3,430.855 $50,467.881

No Equal Variance .154 $26,949.368 $18,762.941 -$10,239.307 $64,138.043

of 19

Loan Repayment20 Status Chi-Square Test (Table 1)

Respondent

Total No Yes

Negative Repayment Status 149 10 159

Positive Repayment Status 474 91 565

Total 623 101 724

Chi-Square Tests

Statistic Value df Sig.

Pearson Chi-Square 9.961a 1 .002

Continuity Correction 9.160 1 .002

Likelihood Ratio 11.583 1 .001

Linear-by-Linear Association 9.948 1 .002

N of Valid Cases 724

a. The minimum expected count is 22.18.

Symmetric Measures

Value Sig.

Nominal by Nominal Phi .117 .002

Cramer's V .117 .002


of 19

Startup Firm21 Status Chi-Square Test (Table 1)

Respondent

Total No Yes

Start-Up Firm 384 55 439

Not a Start-Up Firm 239 46 285

Total 623 101 724

Chi-Square Tests



Continuity Correction 1.589 1 .207





Symmetric Measures

Value Sig.




of 19

Current Employment/Jobs Created/Jobs Retained (FTEs) Independent Samples T-Test (Table 1)

Respondent N Mean Std. Deviation Std. Error

Current Employment

Yes 100 3.724 3.7532 .3753

No 221 5.324 7.5238 .5061 Jobs Created

Yes 92 2.086 3.8028 .3965

No 437 3.222 8.9451 .4279 Jobs Retained

Yes 92 1.878 2.9760 .3103

No 439 1.734 4.2056 .2007

Levene's Test t-test for Equality of Means

F Sig. Sig. Mean

Difference Std. Error

Difference

95% CI of the Difference

Lower Upper

Current Employment

Equal Variance 18.273 .000 .045 -1.6000 .7941 -3.1623 -.0378

No Equal Variance .012 -1.6000 .6301 -2.8397 -.3603

Jobs Created

Equal Variance 2.793 .095 .233 -1.1359 .9507 -3.0036 .7318

No Equal Variance .052 -1.1359 .5833 -2.2834 .0116

Jobs Retained

Equal Variance .207 .650 .755 .1438 .4611 -.7620 1.0495

No Equal Variance .698 .1438 .3695 -.5855 .8730

of 19

Minority-Owned Firm22 Status Chi-Square Test (Table 1)

Respondent

Total No Yes

Not a Minority-Owned Firm 262 42 304

Minority-Owned Firm 361 59 420

Total 623 101 724

Chi-Square Tests


Pearson Chi-Square .008a 1 .929

Continuity Correction .000 1 1.000

Likelihood Ratio .008 1 .929

Linear-by-Linear Association .008 1 .929



Symmetric Measures

Value Sig.




of 19

Women-Owned Firm23 Status Chi-Square Test (Table 1)

Respondent

Total No Yes

Not a Women-Owned Firm 382 66 448

Women-Owned Firm 241 35 276

Total 623 101 724

Chi-Square Tests



Continuity Correction .440 1 .507





Symmetric Measures

Value Sig.

Nominal by Nominal Phi -.029 .439

Cramer's V -.029 .439


of 19

Veteran-Owned Firm24 Status Chi-Square Test (Table 1)

Respondent

Total No Yes

Not a Veteran-Owned Firm 506 84 590

Veteran-Owned Firm 117 17 134

Total 623 101 724

Chi-Square Tests








Symmetric Measures

Value Sig.

Nominal by Nominal Phi -.017 .640

Cramer's V -.017 .640


of 19

Rural County25 Status Chi-Square Test (Table 1)

Respondent

Total No Yes

Urban County Location 399 64 463

Rural County Location 224 37 261

Total 623 101 724

Chi-Square Tests








Symmetric Measures

Value Sig.




of 19

North Carolina County Tier Status26 Chi-Square Test (Table 1)

Respondent

Total No Yes

Tier 1 County 82 7 89



Total 615 100 715

Chi-Square Tests







Symmetric Measures

Value Sig.




of 19

Distressed County Status27 Chi-Square Test (Table 1)

Respondent

Total No Yes




Total 615 100 715

Chi-Square Tests







Symmetric Measures

Value Sig.




of 19

Persistent Poverty County Status28 Chi-Square Test (Table 1)

Respondent

Total No Yes

Not a Persistent Poverty County 571 93 664

Persistent Poverty County 52 8 60

Total 623 101 724

Chi-Square Tests



Continuity Correction .000 1 1.000





Symmetric Measures

Value Sig.




of 19

All Metrics Survey Response Model Binominal Logistic Regression (Table 2)

Omnibus Tests of Model Coefficients

Chi-square df Sig.

Step 25.655 12 .012

Block 25.655 12 .012

Model 25.655 12 .012

Classification Tablea Predicted

Survey Respondent Percent

Correct No Yes

Step 1 Observed

Survey Respondent

No 167 15 91.8

Yes 74 15 16.9

Overall Percentage 67.2

a. The cut value is .500

of 19

1 The 50th percentile response rate for surveys mailed to top managers in organizational surveys is 31.0%. See Anseel, F., Lievens, F., Schollaert, E., & Choragwicka, B. (2010). Response rates in organizational science, 1995-2008: A meta-analytic review and guidelines for survey researchers. Journal of Business and Psychology, 25(3), 335-349. 2 Response rate research for organizational surveys of all types shows a mean response rate of 35.7%. See Barauch, Y., & Holtom, B. (2008). Survey response rate levels and trends in organizational research. Human Relations, 61(8), 1139-1160. 3 Dequech, D., (2001). Bounded rationality, institutions, and uncertainty. Journal of Economic Issues, 35(4), 911-929. 4 Rogelberg, S.G. & Stanton, J.M. (2007). Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10(2), 195-209. 5 The confidence interval is 8.9% with a confidence level of 95%, based on a sample size of 101 and a population size of 577. 6 An independent samples T-test for differences between the committed loan values for respondents and non-respondents, assuming equality of variances (Levene’s test is significant at p = 0.002) has a T value of 2.250 (significant at p = 0.025). 7 A comparison between loan repayment status for respondents and no-respondents has a Pearson Chi-Square value of 9.961 (significant at p = 0.002) and a weak Phi value of 0.117 (significant at p = 0.002). 8 A comparison between loans to start-up businesses and all other types of businesses has a Pearson Chi-Square value of 1.878 (not significant at p = 0.171) and a Phi value of 0.051 (not significant at p = 0.171). 9 An independent samples T-test for differences between in current business employment for respondents and non-respondents, assuming equality of variances (Levene’s test is significant at p = 0.000) has a T value of -2.015 (significant at p = 0.045). 10 An independent samples T-test for differences between jobs created by respondents and non-respondents, not assuming equality of variances (Levene’s test is not significant at p = 0.095) has a T value of -1.947 (not significant at p = 0.052). 11 An independent samples T-test for differences between jobs retained by respondents and non-respondents, not assuming equality of variances (Levene’s test is not significant at p = 0.650) has a T value of 0.389 (not significant at p = 0.698).

of 19

12 A comparison of loans to minority-owned businesses and all other types of businesses has a Pearson Chi-Square value of 0.008 (not significant at p = 0.929) and a Phi value of 0.003 (not significant at p = 0.929). 13 A comparison of respondent and non-respondent loans to women-owned businesses and all other types of businesses has a Pearson Chi-Square value of 0.598 (not significant at p = 0.439) and a Phi value of -0.029 (not significant at p = 0.439). 14 A comparison of respondent and non-respondent loans to veteran-owned businesses and all other types of businesses has a Pearson Chi-Square value of 0.219 (not significant at p = 0.640) and a Phi value of -0.017 (not significant at p = 0.439). 15 A comparison of respondent and non-respondent loans to businesses in rural counties and all other types of businesses has a Pearson Chi-Square value of 0.017 (not significant at p = 0.895) and a Phi value of 0.005 (not significant at p = 0.895). 16 A comparison of respondent and non-respondent loans to businesses by county tier has a Pearson Chi-Square value of 3.878 (not significant at p = 0.144) and a Phi value of 0.074 (not significant at p = 0.144). 17 A comparison of respondent and non-respondent loans in persistent poverty counties has a Pearson Chi-Square value of 0.021 (not significant at p = 0.885) and a Phi value of -0.005 (not significant at p = 0.885). 18 See the Appendix for a detailed explanation of all individual variable tests and the response model. 19 This table lists the results of Step 1 of a binominal logistic regression, with a dichotomous response variable (respondents are coded as 1, non-respondents coded as 0). 20 Firms are classified as having a negative repayment status if the loan is delinquent (more than 30 days late) or charged off. Loans that have been paid in full or are currently on-time and in repayment have a positive repayment status. 21 Firms that have been in operations for 2 years or less by the date of loan closure. 22 Firms that are at least 51% owned or controlled by individuals who self-identify as African American, Asian/Pacific Islander, Native American, and/or Hispanic. 23 Firms that are at least 51% owned or controlled by individuals who are women. 24 Firms that are at least 51% owned or controlled by individuals who are veterans and/or spouses of veterans.

of 19

25 Firms located in a rural county must be classified as non-metropolitan (codes 4 through 9) under the rural-urban continuum system administered by the US Department of Agriculture Economic Research Service. 26 Tier 1 represents the most economic distress, while Tier 3 represents the least economic distress. Loans to areas outside of North Carolina have been coded as missing data and are excluded from the analysis. 27 Firms located in Tier 1 or Tier 2 counties at the time of loan closure, as determined by the North Carolina Department of Commerce. Tier 1 represents the most economic distress, while Tier 3 represents the least economic distress. Loans to areas outside of North Carolina have been coded as missing data and are excluded from the analysis. 28 Firms located in counties with 20% or higher poverty rates during the last 4 decennial Census cycles.

february 2020 evaluation reportcarolinasmallbusiness.org/.../02/csbdf-2019-survey...carolina small...

Documents