center for social services research school of social welfare, uc berkeley data use vs. misuse: the...
TRANSCRIPT
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Data Use vs. Misuse:The challenging nature of publicly available
data
Emily Putnam-Hornstein, MSWBarbara Needell, MSW, PhD
Center for Social Services ResearchUniversity of California at Berkeley
National Child Welfare Data and Technology Conference
July 20, 2007
The Performance Indicators Project at CSSR is supported by the California Department of Social Services and the Stuart Foundation.
(a.k.a., “Numbers Gone Wild”)
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Overview…
Child Welfare Data in California: History
The Good Stuff: Public Website
The Bad Stuff: (Real) Examples of Data Misuse
Strategies for educating users / discussion
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
California’s Child Welfare Data
State supervised, county administered child welfare system 58 Diverse CountiesLongstanding Interagency Agreement Quarterly Data Reports to State and County officialsFunding from CDSS and Stuart FoundationData Publicly Available…
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
http://cssr.berkeley.edu/ucb_childwelfare/
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Public Data: Putting it all out there
• PROS:
– Greater performance accountability– Community awareness and involvement,
encourages public-private partnerships– Ability to track improvement over time, identify
areas where programmatic adjustments are needed
– County/County and County/State collaboration – Transparency – Dialogue
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
• CONS:
– Potential for misuse, misinterpretation, and misrepresentation
– Available to those with agendas or looking to create a sensational headline
– Misunderstood data can lead to the wrong policy decisions
– “Torture numbers, and they’ll confess to anything”
(Gregg Easterbrook)
Public Data: Putting it all out there
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
There are three kinds of lies: Lies, Damned Lies and Statistics
Misused Statistics
^
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Six Ways to Misuse Data (without actually lying!):
1) Rank Data2) Compare Apples and Oranges3) Use ‘snapshots’ of Small Samples4) Rely on Unrepresentative Findings5) Logically ‘flip’ Statistics 6) Falsely Assume an Association to be Causal
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
1) Rank Data
Two streets in Anytown, CA….
It’s all relative…And SOMEONE will
always be ranked last (and first)
Poverty Blvd
$$ Ave
“Jane Doe is the poorest person living on Moneybags Avenue.”
“Joe Shmoe is the wealthiest person living on Poverty Blvd.”
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“San Francisco ranks 55 out of 58 counties when it comes to state and national performance measures…” SF Chronicle, “No refuge. For Foster youth, it’s a state of chance”, November
15, 2005
23.8%
18.5%
9.8%
9.6%
5.7%
0.1%1 or 2 Placements (at 12m, cohort) (+)
Recurrence w/ in 12m (- )
Adopted w/ in 24m (cohort) (+)
Recurrence w/ in 12m of Subst. (- )
Reunified w/ in 12m (cohort) (+)
Re- Entries w/ in 12m (cohort) (- )
San Francisco:AB636 UCB State Measures (Used in NCYL Ranking)
% IMPROVEMENT Jan ‘04 compared to June ‘06
(+) indicates a measure where a % increase equals improvement. (-) indicates a measure where a % decrease equals improvement. indicates a measure where performance declined.
“San Francisco ranks 55 out of 58 counties when it comes to state and national performance measures…” SF Chronicle
• Rankings mask improvement over time.• However, even improvement over time and relatively high rankings can be misleading.
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
2) Compare Apples and Oranges
Two doctors in Anytown, CA…Doctor #1 Doctor #2
What if the populations served by each doctor were very different?
2/1000 20/1000
Doctor of the Year?
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”
SF Chronicle, “Accidents of Geography”, March 8, 2006
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“Foster Children in Fresno County are three times more likely to remain in foster care for more than a year than in Sacramento.”
1. Different families and children served?
2. Different related outcomes?• First entry rates in Fresno are consistently lower
• Re-entries in Fresnoare also lower…
3. Other considerations…• Resources available, resource allocation choices• Performance trends over time
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Number of Crimes Period 1: 76Period 2: 51Period 3: 91Period 4: 76
3) Data snapshots…
Crime jumped by 49%!!No change.
Crime dropped by 16%
Average = 73.5
Crime in Anytown, CA…
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay area...”
SF Chronicle, “No refuge. For foster youth, it’s a state of chance”, November 15, 2005
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
Abuse in Care Rate
Period 1: 1.80% Period 2: 1.64% Period 3: 0.84% Period 4: 0.00%
Responsible use of the data prevents us from making any of these claims
(positive or negative).
The sample is too small; the time frame too limited.
“A foster child living in Napa County is in greater danger of being abused in foster care than anywhere else in the Bay Area…”
100% improvement!0 Children Abused!
= 2/111
= 0
= 2/122= 1/119
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
4) Unrepresentative findings…
Survey of people in Anytown, CA…
90% of respondents stated that they support using tax dollars to build a new football stadium.
The implication of the above finding is that there is overwhelming support for the stadium…
But what if you were then told that respondents had been sampled from a list of season football ticket
holders?
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“Some reports indicate that maltreatment of children in foster care is a serious problem, and in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.”
“My Word”, Oakland Tribune, May 25, 2006
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“…in one recent large-scale study, about one-third of respondents reported maltreatment at the hands of their caregivers.” Oakland Tribune
Factually true?• Yes.
Misleading?• Yes.
– This was a survey of emancipated foster youth
– Emancipated youth represent a distinct subset of the foster care population
– This “accurate” statistic misleads the reader to conclude that one-third of foster children have been maltreated in care…
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
5) Logical “Flipping”…
Headline in The Anytown Chronicle:
60% of violent crimes are committed by men who did not graduate from high school.
“Flip”
60% of male high school drop-outs commit violent crimes?
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”
Bath and Haapala, 1993 as cited in “Shattered bonds: The color of child welfare” by Dorothy Roberts
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“One study in Washington State found that 75 percent of a sample of neglect cases involved families with incomes under $10,000.”
• In reading statistics such as the above, there is a tendency to want to directionally “Flip” the interpretation
• But the original and flipped statements have very different meanings!
75% of neglect cases involved families with incomes under $10,000
DOES NOT MEAN75% of families with incomes under $10,000
have open neglect cases
• Put more simply, just because most neglected children are poor does not mean that most poor children are neglected
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
6) False Causality…
A study of Anytown residents makes the following claim:
Adults with short hair are, on average, more than 3 inches taller than those with long hair.
Finding an association between two factors does not mean that one causes the other…
Hair Length Height
Gender
X
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“A number of child characteristics have previously been shown to be associated with risk of maltreatment. Prematurity or low birth weight is frequently reported…” As reported in Sidebotham and Heron’s 2006 article
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
“A number of child characteristics have previously been shown to be associated with risk of maltreatment. Prematurity or low birth weight is frequently reported…”
• Should one conclude that prematurity is a causal factor in maltreatment?
(Drug use?)a third factor
prematurity maltreatment
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
California’s Response to Data Misuse?
• CA has had the will to weather the storm(s)…
• Continued efforts to frame the data, educate interested media, policymakers, and others– What do these findings mean?– How can these data be used to gain insight into where
improvements are needed?
• Counties have been proactive in discussing both the “good” and the “bad” (be first, but be right).– Be transparent– If not playing offense…playing defense
• Data still public!! (Thank you to the CWDA for these bullets!)
CENTER FOR SOCIAL SERVICES RESEARCH School of Social Welfare, UC Berkeley
CSSR.BERKELEY.EDU/UCB_CHILDWELFARE
Needell, B., Webster, D., Armijo, M., Lee, S., Cuccaro-Alamin, S., Shaw, T., Dawson, W., Piccus, W., Magruder, J., Exel, M., Conley, A., Smith, J. , Dunn, A., Frerer, K., & Putnam-Hornstein, E., (2007). Child Welfare Services Reports for California. Retrieved [month day, year], from University of California at Berkeley Center for Social Services Research website. URL: <http://cssr.berkeley.edu/UCB_CHILDWELFARE/>
Barbara [email protected] Putnam-Hornstein