the schedule moving forward…
DESCRIPTION
The schedule moving forward…. Today Evaluation Research Remaining time = start on quantitative data analysis Thursday Methodology section due Quantitative analysis continued (start Exercise 10) Next Tuesday Exercise 10 Due Maahs returns Exercise #9 - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/1.jpg)
The schedule moving forward…• Today
– Evaluation Research – Remaining time = start on quantitative data analysis
• Thursday– Methodology section due– Quantitative analysis continued (start Exercise 10)
• Next Tuesday – Exercise 10 Due– Maahs returns Exercise #9
• Research proposal (everything is now online) is due on Dec 21st (Monday)
![Page 2: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/2.jpg)
Evaluation Research
• PURPOSE rather than specific method• Much more popular in last 20 years• A form of “applied” research • Results, though designed to impact decision-
making, often have no impact
![Page 3: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/3.jpg)
Types of Evaluation Research
• Needs assessment• Cost-benefit study• Monitoring study • Program evaluation
![Page 4: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/4.jpg)
Doing evaluation (program) research
• Research questions / measurement as Job #1– Political context how do measure – Operationalize “outcome” of interest
• “Response” variable • NEED AGREEMENT: what is the program trying to do?
– Operationalize “processes” • Intermediate objectives
– Context of – ”
![Page 5: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/5.jpg)
Research designs for evaluation
• Experimental– Problems, ethics (placebo)– The “black box” issue– Quasi-experimental designs
• Time Series (multiple time series)• Qualitative evaluations
– Low birth weight study
![Page 6: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/6.jpg)
Evaluation Research ISSUES
• Evaluation research as “MESSY”– Administrative control, context of “real life”
• Importance of looking at “black box”
• Ethics– The intervention itself can raise issues– Control group members– Violating Ethics 101 (Tuskegee) – Evaluation impacts people’s lives
![Page 7: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/7.jpg)
The (non) use of findings
• Purely rational/scientific world vs. the one we inhabit– Nixon panel on porn– Scared Straight – 3 Strikes and You’re Out legislation
• Studies gone wrong…– Fire the evaluator (or dismiss as pointy headed idiot) and
keep the program– Why not trust/use the results?
• Scientist/practitioner gap• “True believers”• Vested interests
![Page 8: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/8.jpg)
Bivariate Analysis
Backed up with a little inferential statistics……yeah baby!
![Page 9: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/9.jpg)
Review
• Descriptive statistics– Purpose?– Types?– These are “univariate” statistics
• Explanatory research– Attempt to demonstrate cause-effect
• Necessary Criteria?
![Page 10: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/10.jpg)
Demonstrating Associations
• Bivariate (2 variables) analysis– There are a number of ways to do this
• The method you choose depends largely on characteristics of the two variables
– Levels of Measurement?
![Page 11: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/11.jpg)
Contingency Tables (cross tabs)
• Some find these very intuitive…others struggle– It is very easy to misinterpret these critters
• Convention: the independent variable is on the top of the table (dictates columns) and the dependent variable is on the side (dictates rows). – What is in the individual “cells?”
• Frequencies (number of cases that fit criteria)• Convert to Percentages: a way to standardize cells and make
relationships more apparent
![Page 12: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/12.jpg)
Example
• A survey of 10,000 U.S. residents• Research question: is one’s political view
related to attitudes towards police?– What is the DV and IV?– In constructing a table, what goes where?
![Page 13: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/13.jpg)
An Example
Attitude TowardsPolice
Political Party Total
Repub Democrat Libertarian Socialist
Favorable 2900 2100 180 30 5210
Unfavorab 1900 1800 160 28 3888
Total 4800 3900 340 58 9098
![Page 14: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/14.jpg)
The Percentages of Interest
Attitude TowardsPolice
Political Party Total
Repub Democrat Libertarian Socialist
Favorable 2900 (60%)
2100(54%)
180(53%)
30(52%)
5210
Unfavorab 1900 1800 160 28 3888
Total 4800 3900 340 58 9098
![Page 15: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/15.jpg)
Inferential Statistics
• Researchers are typically not interested in whether there is a relationship in the sample– Want to know about the population– Why might there be a difference between what
you find in the sample and what actually exists in the pop?
• Even with a probability sample, there is always sampling error
• Without a probability sample = error + bias
![Page 16: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/16.jpg)
Inferential Statistics II
• Cannot assume that the relationship in your sample is true in population– BUT: probability theory allows us to estimate:
“The likelihood of obtaining a particular finding if in the population, there was no relationship” OR
“The likelihood of obtaining a particular finding assuming the null hypothesis”
![Page 17: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/17.jpg)
Test Statistics and Significance Tests
• What does “statistically significant” mean?– Not due to sampling error
• What do you need to do to test for statistical significance?
1. A “test statistic” • An indicator of how different the sample is from “no
relationship”
2. The level of error that is acceptable (.05, .01)3. Degrees of Freedom
![Page 18: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/18.jpg)
The Test Statistic for Contingency Tables
• Chi Square, or χ2 – Calculation
• Observed frequencies • Expected frequencies
– Can get confusing calculating these critters by hand
– Intuitive: how different are the observed cell frequencies from the expected (under null hypothesis) cell frequencies
– Degrees of Freedom:• (# of Rows -1) (# of Columns -1)
![Page 19: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/19.jpg)
Political View (3 Category) * Respondent's Sex Crosstabulation
157 229 386166.9 219.1 386.0
215 312 527227.9 299.1 527.0
252 278 530229.2 300.8 530.0
624 819 1443624.0 819.0 1443.0
CountExpected CountCountExpected CountCountExpected CountCountExpected Count
Liberal
moderate
conservative
Political View(3 Category)
Total
Male FemaleRespondent's Sex
Total
![Page 20: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/20.jpg)
The Chi-Square Sampling Distribution (Assuming Null is True)
![Page 21: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/21.jpg)
Conventional Significance Testing
1. Calculate the test statistic2. Set your “Type II” error level
– The risk of being wrong that you are willing to live with
3. Find the “critical region” within the distribution for your sample statistic
– How far out on the curve does your test statistic have to get before you can reject the null hypothesis
4. Decide whether to reject, or fail to reject the null hypothesis
![Page 22: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/22.jpg)
Significance Testing in SPSS
• Within your crosstabs, select “Chi-Square”– SPSS gives you the χ2 value
• What you would calculate based on the observed and expected cell frequencies
– SPSS also gives you “p”• The exact probability of obtaining this χ2, if indeed the
null hypothesis was correct• In newer versions of SPSS, the p values are listed under
“sig” or “significance” or something similar
![Page 23: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/23.jpg)
Review of Contingency Tables
• Level of Measurement– Both IV and DV are Nominal or Ordinal
(Categorical data)• Constructing a table
– IV on top (columns), DV on bottom (rows)• With this format, select “column percentages”
![Page 24: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/24.jpg)
Review of Inferential Statistics• Purpose of inferential statistics
– Figure out the odds that a finding from sample is due to “chance” or “sampling error”
• Process – Assume null hypothesis is correct
• Calculate the odds of obtaining your particular finding under this assumption
• If the odds are very low, you may be suspicious that the null hypothesis is incorrect
• At some point (research sets level), you say that the odds are so low that you are going to go ahead and reject the null hypothesis
![Page 25: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/25.jpg)
Review of Chi-Square
• A measure of how different observed (from your sample data) cell frequencies are from what would be expected under the null hypothesis (e.g., no relationship)– The Chi-square distribution changes shape with
different degrees of freedom• This is because, by definition, the more cells you have,
the higher χ2 gets– df = (R-1)(C-1)
![Page 26: The schedule moving forward…](https://reader036.vdocument.in/reader036/viewer/2022062310/56816674550346895dda0d07/html5/thumbnails/26.jpg)
Interpreting Chi-Square
• Chi-square has no intuitive meaning, it can range from zero to very large– The real interest is the “p value” associated with
the calculated chi-square value– This is the exact probability of obtaining the chi-
square if in the population there was no relationship
• In other words, the exact probability of finding that chi-square, under the null hypothesis