statistical validation and data analytics in e discovery

Statistical Validation And Data Analytics In eDiscovery

Geoff BlackDirector, High Tech Investigations

Prudential

The views expressed in this presentation are solely those of the presenter and do not necessarily reflect the views of the presenter’s employer.

Recommended Reading

Why do we need Statistics?(Ensuring Quality in eDiscovery)

Professional standards

Savvy judges already require sampling

Defensibility

Types of Sampling

Judgmental

Statistical*

A Recent Experience with Sampling

Setting the stage

A Recent Experience with SamplingThe Challenge

Select appropriate filters for a large data set

Audit reviewers without double reviewing everything

Test our processing tools

Accomplish all of these with a high confidence level and low confidence interval

Statistics for eDiscoveryConfidence Interval

The “confidence interval” or margin of error

How closely our results will reflect the general population

Lower is better

Statistics for eDiscoveryConfidence Interval Example

We have 100 documents and our confidence interval is ± 2%.

Testing shows 10% responsiveness

General population should show between 8% and 12% responsiveness, or

8 to 12 documents.

Statistics for eDiscoveryConfidence Level

The “confidence level”

Does our sample accurately represent the results of general population?

Higher is better

Statistics for eDiscoverySample Sizes for Population of 1,000,000

± 10% ± 5% ± 2%0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

99% Confidence Level95% Confidence Level90% Confidence Level

Margin of Error

[Scaling] Statistics for eDiscovery

10,000 100,000 1,000,000 10,000,0002,800

3,000

3,200

3,400

3,600

3,800

4,000

4,200

4,400

Sample Sizes at 99% Confidence ± 2%

Population Size

A Recent Experience with SamplingFiltering Selection

Finding a good search method is difficult

Who chooses search terms?

Requires iterative testing and validation

A Recent Experience with SamplingValidating Filters

Began with around 10,000,000 documents

A 99% confidence level with a ± 2% confidence interval dictated a sample size of 4,150 documents

Chose a random sample and searched

Reviewed all the results (positive and negative)

A Recent Experience with SamplingValidating Filters

Results did not match expectations

Revised the list of search terms

Tested the filtering again, and…

A more accurate search with less responsive data!

A Recent Experience with SamplingValidating Filters

Wait a minute, I always test my keywords!

Not whether you test, but how much data…

A Recent Experience with SamplingValidating Review

After filtering about 120,000 documents to review

Reviewers often disagree about relevance or simply don’t understand the material

Double and triple review kills budgets

Instead, sample a random set of 4,010 reviewed documents

A Recent Experience with SamplingValidating Review

Subject matter expert noted a few anomalies

Re-reviewed items with the confusing term

One reviewer’s results could not be trusted

A Recent Experience with SamplingKeeping Your Vendors Honest

How do they test their tools?

How were automated tools used in your matter?

Do you know what they cannot do?

How did you use the results in your decisions?

What’s Next?

Built-in iterative review with statistical sampling

Relying solely on “Concept Searching” is a black box and a dead end

Advanced search techniques must offer explanatory reasoning

What does all this mean?(The Benefits of Using Statistics)

Small dataset for testing

Minimize false positives

More accurate search, reduced data volume

Defensibility of statistically validated testing

One last thing…

Technologies will always differ and change rapidly,

but statistical validation is a timeless truth.

References & Related Cases—The Sedona Conference Working Group Series, “Commentary

on Achieving Quality in the E-Discovery Process,” May 2009.—Losey, Ralph. “The Multi-Modal ‘Where’s Waldo?’ Approach

to Search…,” 2010. http://e-discoveryteam.com/2010/02/27/—William A. Gross Construction Associates, Inc. v. American

Manufacturers Mutual Insurance Co., 256 F.R.D. 134, 134 (S.D.N.Y. 2009)

—Victor Stanley v. Creative Pipe, 250 F.R.D. 251 (D. Md. 2008) —In re Seroquel Products Liability Litigation, 244 F.R.D. 650, 662

(M.D. Fla. 2007)

Statistical Validation And Data Analytics In eDiscovery

Geoff Blackgeoff@geoffblack.com

www.geoffblack.com/ediscovery