up504 winter 2008 prof. campbell university of michigan
DESCRIPTION
UP504 Winter 2008 Prof. Campbell University of Michigan EXAMPLE OF USING REGRESSION TO TEST A POLICY IMPACT. Example: A city has 30 low-income housing projects . - PowerPoint PPT PresentationTRANSCRIPT
UP504 Winter 2008
Prof. Campbell
University of Michigan
EXAMPLE OF USING REGRESSION TO TEST A POLICY IMPACT
Housing Project
Vacancy Rate
1 33%2 33%3 18%4 23%5 28%6 19%7 32%8 20%9 15%
10 32%11 19%12 19%13 25%14 29%15 17%16 25%17 5%18 28%19 19%20 26%21 15%22 8%23 11%24 30%25 14%26 27%27 21%28 13%29 21%30 31%
Passersby walk past the Bromley-Heath Housing Complex in the Jamica Plain district of Boston, Monday, Nov. 2, 1998. What began as a 1960's experiment in self-government ended this past weekend when management of Bromley-Heath Housing Complex, Boston's only tenant-run public housing, was seized by the city after a number of drug related arrests. (AP Photo/Steven Senne)
Example:A city has 30 low-income housing projects.
A large number of vacant units in these projects creates a wide variety of problems (reduced revenues, vandalism, lower morale of existing tenants, etc.)
There is a wide range of vacancy rates, from less than 10 percent to over 30 percent.
The city officials believe that drug trafficking in the housing projects is discouraging people from either moving into or staying in the projects.
http://accuweather.ap.org/cgi-bin/aplaunch.pl
To prove the key role of drug dealing in shaping housing project vacancy rates, the city releases data showing that vacancy rates in projects with anti-drug programs (run by the police department) have a lower vacancy rate:
Vacancy rate (projects with anti-drug programs): 19 percent
Vacancy rate (projects without anti-drug programs): 25 percent
Housing Project
Vacancy Rate
1 33%2 33%3 18%4 23%5 28%6 19%7 32%8 20%9 15%
10 32%11 19%12 19%13 25%14 29%15 17%16 25%17 5%18 28%19 19%20 26%21 15%22 8%23 11%24 30%25 14%26 27%27 21%28 13%29 21%30 31%
Police Anti-Drug
Program?
001001110110001010101011100110
t-Test: Two-Sample Assuming Equal Variances
vacancy rate NO
drug program)
vacancy rate (drug program)
Mean 0.248 0.189Variance 0.005 0.006Observations 15.000 15.000Pooled Variance 0.005Hypothesized Mean Difference 0.000df 28.000t Stat 2.215P(T<=t) one-tail 0.018t Critical one-tail 1.701P(T<=t) two-tail 0.035t Critical two-tail 2.048
And just to be sure, they ran a difference of means test to demonstrate that the results were statistically significant at the 0.05 level.
To further demonstrate the significant role that this policy anti-drug program plays, the city also collects data on
housing expenses
family structure
(since these two variables also affect vacancy rates).
Housing Project
Vacancy Rate
Percent of Gross Hhd
Income Spent on
rent
percent 2-parent
families
Police Anti-Drug
Program?
1 33% 35% 12% 02 33% 52% 14% 03 18% 36% 15% 14 23% 51% 41% 05 28% 40% 18% 06 19% 50% 21% 17 32% 37% 22% 18 20% 19% 24% 19 15% 38% 25% 0
10 32% 21% 26% 111 19% 20% 27% 112 19% 25% 27% 013 25% 38% 33% 014 29% 35% 29% 015 17% 44% 46% 116 25% 47% 46% 017 5% 19% 57% 118 28% 50% 46% 019 19% 49% 30% 120 26% 24% 30% 021 15% 39% 33% 122 8% 50% 41% 023 11% 41% 36% 124 30% 22% 39% 125 14% 38% 39% 126 27% 37% 40% 027 21% 19% 37% 028 13% 19% 44% 129 21% 28% 22% 130 31% 20% 30% 0
An unidentified Long Beach Police officer questions a suspect during a drug raid on an apartment complex as a child watches, Tuesday July 15, 1997 in Long Beach, Calif. An 80-member police task force arrested nearly a dozen people in a four-block area targeted for rock cocaine sales and drug safe houses. The operation culminated six weeks of surveillance and was called, "Operation Clean Streets."(AP Photo/MIchael Caulfieldhttp://accuweather.ap.org/cgi-bin/aplaunch.pl
The city then releases the results of their own in-house multiple regression analysis, controlling for these two variables.
SUMMARY OUTPUT
Regression StatisticsMultiple R 0.8067R Square 0.6508Adjusted R Square 0.6105Standard Error 0.0448Observations 30
ANOVA
df SS MS FSignificance F
Regression 3 0.0974 0.0325 16.1513 0.0000Residual 26 0.0523 0.0020Total 29 0.1496
Coefficients
Standard Error t Stat P-value
Intercept 0.4925 0.0443 11.1171 0.0000Percent of Gross Hhd Income Spent on rent -0.3746 0.0862 -4.3448 0.0002percent 2-parent families -0.3860 0.0776 -4.9768 0.0000Police Anti-Drug Program? -0.0593 0.0166 -3.5711 0.0014
Even controlling for the other two variables, the police anti-drug program is statistically significant, and seems to reduce the vacancy rate by 6 percentage points -- and then asks for more money for the program.
Housing Project
Vacancy Rate
Percent of Gross Hhd
Income Spent on
rent
percent 2-parent
families
Police Anti-Drug
Program?
Active Tenants
Group? (1 = yes; 0 =
no)
1 33% 35% 12% 0 02 33% 31% 14% 0 03 18% 25% 15% 1 14 23% 22% 41% 0 05 28% 34% 18% 0 06 19% 51% 21% 1 17 32% 42% 22% 1 08 20% 29% 24% 1 19 15% 33% 25% 0 1
10 32% 37% 26% 1 011 19% 46% 27% 1 112 19% 45% 27% 0 113 25% 31% 33% 0 014 29% 25% 29% 0 015 17% 18% 46% 1 116 25% 39% 46% 0 017 5% 23% 57% 1 118 28% 21% 46% 0 019 19% 29% 30% 1 120 26% 49% 30% 0 021 15% 27% 33% 1 122 8% 47% 41% 0 123 11% 21% 36% 1 124 30% 46% 39% 1 025 14% 51% 39% 1 126 27% 27% 40% 0 027 21% 40% 37% 0 128 13% 24% 44% 1 129 21% 18% 22% 1 130 31% 23% 30% 0 0
http://www.nhi.org/online/issues/95/phorg.html
NOT SO FAST, cries a tenants organization, which has been skeptical of the police anti-drug program’s effectiveness, and instead argues that the strong presence of organized tenants groups makes the difference in keeping vacancy rates lower.
Controlling also for this new variable, the police anti-drug program is no longer statistically significant, an instead the presence of the active tenants group makes the dramatic difference. (and look at that great R square!). However, we are no quite done…
Regression StatisticsMultiple R 0.995R Square 0.990Adjusted R Square 0.989Standard Error 0.008Observations 30
ANOVA
df SS MS FSignificance F
Regression 4 0.164 0.041 628.372 0.000Residual 25 0.002 0.000Total 29 0.165
Coefficients
Standard Error t Stat P-value
Intercept 0.500 0.008 60.294 0.000Percent of Gross Hhd Income Spent on rent -0.399 0.016 -24.610 0.000percent 2-parent families -0.288 0.015 -19.422 0.000Police Anti-Drug Program? -0.004 0.004 -1.238 0.227Active Tenants Group? (1 = yes; 0 = no) -0.102 0.004 -28.827 0.000
Since the police variable now has a statistically insignificant t-score, we remove it from the model. (We also remove the income variable, since it also becomes insignificant after we remove the police variable.) We are left with two independent variables: percent of 2-parent families and active tenants group.
SUMMARY OUTPUT
Regression StatisticsMultiple R 0.928R Square 0.861Adjusted R Square 0.850Standard Error 0.030Observations 30
ANOVAdf SS MS F Significance F
Regression 2 0.149 0.074 83.484 0.000Residual 27 0.024 0.001Total 29 0.173
Coefficients
Standard Error t Stat P-value BETA
Intercept 0.36582 0.017 20.908 0.000percent 2-parent families -0.2565 0.051 -5.017 0.000 -0.362Active Tenants Group? (1 = yes; 0 = no) -0.1246 0.011 -11.347 0.000 -0.821
Write out the equation:Predicted vacancy rate = + 0.366 - 0.256 [percent 2-parent families]- 0.125 [active tenants group]
Example (50% 2-parent & active tenants group]:Predicted vacancy rate = + 0.366 - 0.256 [.50]- 0.125 [1] = .113 or 11.3 percent
Vacancy Rate
Percent of Gross Hhd
Income Spent on
rent
percent 2-parent
families
Active Tenants
Group? (1 = yes; 0 =
no)
Police Anti-Drug
Program?Vacancy Rate 1.00Percent of Gross Hhd Income Spent on rent 0.10 1.00percent 2-parent families -0.44 0.20 1.00Active Tenants Group? (1 = yes; 0 = no) -0.86 0.12 0.10 1.00Police Anti-Drug Program? -0.39 -0.08 0.03 0.53 1.00
Looking at a correlation matrix can help understand the interrelationships between variables.
Moral of the story?
•Multiple regression can be a powerful tool in evaluation research
•One should be careful of generalizing from under-specified models (with omitted variables). This is especially true when the R-square is dramatically less than 1.00.
•Evaluation research is the effort to isolate the specific (or unique) impact of a specific policy/program/influence on a dependent variable (an outcome).
•A great challenge is to estimate the “counter-factual”, in this case, what would the vacancy rate have been without the police anti-drug program. (Here we used regression to make a prediction -- one can never know for sure.)
•Why might the police program initially seemed to have a strong positive effect? Perhaps because of self-selection: the police -- either intentionally or by accident -- may have targeted their program on housing projects that already had the characteristics of projects with lower vacancy rates.
•How do you get around this self-selection bias: use random assignment (a sometimes difficult but powerful approach in evaluation research).