domestic violence network people matching
DESCRIPTION
Domestic Violence Network People Matching. October 22, 2014 Jay Colbert, Indianapolis. Domestic Violence Network. Phase1: Developed “Domestic Violence in the Criminal Justice System” report in 2013-2014 Phase 2: Updating report - PowerPoint PPT PresentationTRANSCRIPT
Domestic Violence NetworkPeople Matching
October 22, 2014
Jay Colbert, Indianapolis
www.savi.org
Domestic Violence Network
• Phase1: Developed “Domestic Violence in the Criminal Justice System” report in 2013-2014
• Phase 2: Updating report• There are many different sources of DV
data and the question was “how many unique people are involved?”
www.savi.org
Data Sources
Notes1Domestic Violence Shelter2No names given3Special IMPD domestic violence program
Source Table Date Range RecordsJulian Center1 Julian Outreach 2008 - 2013 32,540
IndianapolisMetro PD
IMPD Victims2 2009 - 2013 341,558
IMPD Perpetrators 2009 - 2013 520,148 (38,780 linked to other DV Data)
Baker One3 2013 2,715 Marion County Prosecutor’s Office
MCPO Cases 2008 - 2013 32,885 MCPO Victims 2008 - 2013 36,133
Indiana Supreme Court
Protective Orders (Protected Person) 2009-2013 123,630 Protective Orders (Respondent) 2009-2013 123,630
Total (with names) 871,681Marion County DV Total
(with names) 177,545
www.savi.org
Phase 1 people matching
• In-house solution using (mainly) python scripts.
• Looked at combination of name, race, gender, age.
• Gave pretty good final results, but…• It took 10 days to run.
www.savi.org
Phase 2 people matching
• Researched multiple commercial software solutions geared towards data deduplication.
• Some enterprise solutions as much as $30,000 and up.
• Some desktop solutions as cheap as $500.
www.savi.org
Phase 2 name matching
• Purchased MatchIT software. $4,000 initial, $2,000 annual renewal Discounted since we are a nonprofit Flexibility on matching algorithms.
We match on name and date of birth first. Then we move on to matching on name, race,
gender, and year of birth (not all sources give us exact DOB)
We previously used age, but that becomes problematic with increasing number of years of data.
www.savi.org
Example
DVNMATCHFIRST NAME LAST NAME RACE GENDER BIRTH_DATE BIRTH_YEAR
Tammy Copelan Black Female 11/25/1983 1983
Tammy Copland Black Female 11/25/1983 1983
Tamara Copeland Black Female 11/25/1983 1983
Tamy Copeland Black Female 11/25/1983 1983
Tammy Copeland Black Female 11/25/1983 1983
Tammy Copeland Black Female 11/25/1983 1983
Tammie Copeland Unknown Female 11/25/1983 1983
Tammy Copeland Black Female 1983
Tammy Copeland Black Female 1983
Tammy Copeland Black Female
These records identified as same person.
Names have been changed to protect the innocent (and the guilty).
www.savi.org
Old Method
Old ResultsPOLISID FIRST_NAME LAST_NAME RACE GENDER DOB BIRTH_YEAR
6486Eric Jones White Male 10/1/1980 19806486ERIC JONES White Male 10/1/1980 19806486ERIC JONES White Male 10/15/1982 19826486ERIC JONES White Male 1/15/1985 19856486ERIC JONES White Male 8/20/1981 19816486Eric Jones White Male 19826486Eric Jones White Male 10/30/1982 19826486Eric Jones White Male 12/17/1985 1985
81210Eric Jones White Male 7/3/1988 198881210Eric Jones White Male 6/10/1989 198981210ERIC JONES White Male 2/5/1986 198681210ERIC JONES White Male 2/5/1986 198681210Eric Jones Unknown Male 7/3/1988 198881210ERIC JONES White Male 7/3/1988 1988
174231ERIC JONES White Male 2/5/1986 1986174231ERIC JONES White Male 2/5/1986 1986
Old Method had a hard time seeing people with different birthdates within a few years of one another as different people.
www.savi.org
New MethodNew Results
MATCH_REF FORENAMES SURNAME RACE GENDER BIRTH_DATE BIRTH_YEAR0008050 ERIC JONES White Male 1/15/1985 19850008050 Eric Jones White Male 1/15/1985 19850008050 Eric Jones White Male 1/15/1985 19850039351 Eric Jones White White 2/5/1986 19860039351 ERIC JONES White Male 2/5/1986 19860039351 ERIC JONES White Male 2/5/1986 19860039351 ERIC JONES White Male 2/5/1986 19860039351 ERIC JONES White Male 2/5/1986 19860039351 ERIC JONES White Male 2/5/1986 19860039351 ERIC JONES White Male 2/5/1986 19860141208 Eric Jones White Male 10/15/1982 19820141208 ERIC JONES White Male 10/15/1982 19820254712 ERIC JONES White Male 10/1/1980 19800254712 ERIC JONES White Male 10/1/1980 19800310884 Eric Jones White Male 7/3/1988 19880310884 ERIC JONES White Male 7/3/1988 19880310884 Eric Jones White Male 7/3/1988 19880310884 Eric Jones Unknown Male 7/3/1988 19880351920 Eric Jones White Male 19820427720 Eric Jones White Male 6/10/1989 19890465350 ERIC JONES White Male 8/20/1981 1981
New Method much better since it looks at exact dates of birth
www.savi.org
New Method
New ResultsSURNAME FORENAMES RACE GENDER BIRTH_DATE BIRTH_YEAR
STEPHENS CLARENCE Black Male 12/3/1969 1969
Stephens Clarence Black Male 1969Stephens Clarence Black Male 1969Stephens Clarence Black Male 12/3/1969 1969STEPHENS CLARENCE Black Male 12/3/1969 1969STEPHENS CLARENCE Black Male 12/3/1969 1969STEPHENS CLARENCE Black Male 12/3/1969 1969STEPHENS CLARENCE Black Male 12/3/1969 1969STEPHENS CLARENCE Black Male 12/3/1969 1969STEPHENS CLARENCE Black Male 12/3/1969 1969Stephens Clarence Black Male 12/31/1969 1969STEPHENS CLARENCE L Black Male 12/3/1969 1969
Even gets it right when both names are accidently in one field or there is a typo in the birthdate
www.savi.org
Many other ways to match
• Name• Date of Birth/Age• Race, Gender• Address• Phone numbers• Anything else you can think of
www.savi.org
What did it boil down to?
• The total 871,681 records boiled down to 400,736
• The 177,545 Marion County DV people boiled down to 92,908.
www.savi.org
Example Output
Data Year
Julian Outreach
Marion County
Prosecutor’s Office
Protective Orders
IMPD Domestic Violence Officer
Information Sheets
Total Unduplicated
Count
2009 4,499 5,471 3,888 12,9442010 5,457 6,532 3,802 13,9662011 5,081 5,899 3,178 12,1552012 5,354 5,958 3,058 12,0492013 6,560 5,890 2,653 2,543 11,559
Unique Victims
www.savi.org
Example Output
188101105
5331,8601,7501,4571,039
799553362
17766
332214
63
11484
5879
300381360256203144
12984
30175430
2,000 1,500 1,000 500 0 500
Under Age 55-9
10-1415-1920-2425-2930-3435-3940-4445-4950-5455-5960-6465-6970-7475-7980-84
85+
Marion County2013 Domestic Violence Victims
n = 11,559
Male
Female
www.savi.org
Example Output
26.7%
9.4%
2.4%0.0%
62.7%
44.4%
0.6% 0.4%3.6%
50.9%
0%
10%
20%
30%
40%
50%
60%
70%
Marion CountyComparing Race of Domestic Violence Victims
and Race of General Population, 2013
Race of General Population as % of Total Population
Race of Victims as % of All Victims