using technology assisted review for insurance defense
TRANSCRIPT
877.557.4273catalystsecure.com
A primer on the law and technology for insurance defense professionals
WEBINAR
TAR for Smart People Gordon J. Calhoun, Esq.
John Tredennick, Esq.
Presenters
&
We welcome your questions via the chat feature.
Our Speakers
Gordon serves as Chair of the firm's E-Discovery Practice and specializes in representing clients in a variety of
complex, large exposure litigated matters. He also manages the Insurance Regulatory and Reinsurance Division of
the firm’s Insurance Coverage and Bad Faith Practice Group.
Partner, Lewis Brisbois Bisgaard & Smith LLP
Gordon J. Calhoun
John founded Catalyst, which designs, hosts and services the world’s fastest and most powerful document
repositories for large-scale discovery and regulatory compliance. He was named by the American Lawyer as one of
the top six e-discovery trail blazers in its special issue on the “Top 50 Big Law Innovators” of the past 50 years.
Founder and CEO, Catalyst
John Tredennick
What is Technology Assisted Review?
1. A process through which humans work with a computer to teach it to identify relevant documents.
2. Ordering documents by relevance for more efficient review.
3. Stopping the review after you have reviewed a high percentage of documents. [Optional]
Is This New?
We Already Use It
What is the Process?
1. Collect and process your files
Shredding the Documents
12
3
What is the Process?
1. Collect and process your files
2. Train the system (seeds)
3. Rank the documents
WHY RELEVANT?
How Does it Work?
How Does it Work?
Support Vector Machines Naïve Bayes K-Nearest Neighbor Geospatial Predictive Modeling Latent Semantic
"I may be less interested in the science behind the "black box” than in whether it produced responsive documents with reasonably high recall and high precision.“
Magistrate Judge Andrew Peck
How Does it Work?
Support Vector Machines Naïve Bayes K-Nearest Neighbor Geospatial Predictive Modeling Latent Semantic
What Goes on Under the Hood?
The computer builds a big, complex search!
What terms are most likely to be associated with good documents?
What terms are most likely to be associated with bad documents?
What is the Process?
1. Collect and process your files
2. Train the system (seeds)
3. Rank the documents
4. Continue your review
5. Test as you go
6. Stop when finished
A Circular Process
What Are the Savings?
What Are the Savings?
Number of documents in the review
What Are the Savings?
Percentage of relevant documents found
What Are the Savings?25
Linear Review
What Are the Savings?
Review 12% and get 80% recall
What Are the Savings?
Review 24% and get 95% recall
What Are the Savings?
When Can I Stop?
1. When you run out of relevant documents
2. Courts have approved 70% and higher
3. Prove recall by sampling the non-reviewed documents
4. Sample size and margin of error
Case Study: Large Production Review
Collection: 2.1 million documents Initial richness: 1% Review team used CAL Review richness: 25 to 35%
Result: 98% Recall
Review: Less than 10%
Japanese client faces U.S. litigation
Case Study: IP Litigation
Over 3.6 million documents collected
Six relevant per 1,000 documents
Started with 10,000 already-tagged training documents
Used Insight Predict and its CAL protocol
Batch richness grew to 35%
Stopped review when relevant documents stopped coming
Results
Found 80% relevant documents after just 20% review
Reviewed 500,000 of 3.6 million total
Saved 80% of the review costs
Saved 80% of the review time
How Can I Use it?
Prioritized Review
Studies show that review teams get through documents more quickly when similar documents are reviewed together.
How Can I Use it?
Outbound productions
Reduce review costs and time dramatically by establishing a cutoff point and stopping review.
Save up to 90% on review costs!
How Can I Use it?
In-bound productions
In-bound productions allow your team to find hot documents for depositions and trial.
Find relevant documents in a fraction of the time!
How Can I Use it?
Early case/document assessment
The SEC and DOJ actively use TAR to get a quick handle on document produced in their investigations.
Greg Buckles, eDJ Analyst
How Can I Use it?
Focus team on multiple issues during review to save time and effort
Issue review
How Can I Use it?
Non-English documents
TAR works effectively on any language so long as you tokenize.
裁判所はどこにありますか ?
Where is the courthouse?
TAR 1.0: One-Time Training
1. Rolling uploads?
2. Subject matter experts?
3. Low richness collections?
4. Only for big cases?
Problems
Review = training
TAR 2.0: Continuous Training
CAL: Solving Real-World Problems
1. One time training
2. Rolling uploads
3. Subject matter experts
4. Low richness collections
5. Only for big cases
Simple and flexible
CAL Training
1. Train with anything you want
2. Use as many or as few as you want
3. Keep searching and feeding throughout the
process
4. Makes use of every attorney decision on
documents
5. Contextual diversity will help find what you
don’t know
6. QC helps ensure consistent training
Continuous active learning (CAL) is more effective than one-time
training used in TAR 1.0
Continuous Active Learning vs. One-Time Training
TAR 2.0 Savings
What Have the Courts Said?
Go Ahead, Dive In!
“This judicial opinion now recognizes that computer-assisted review is an acceptable way to search for relevant ESI in appropriate cases.”
Da Silva Moore v. Publicis Groupe (S.D.N.Y. 2012)
Go Ahead, Dive In!
“This judicial opinion now recognizes that computer-assisted review is an acceptable way to search for relevant ESI in appropriate cases.”
Da Silva Moore v. Publicis Groupe (S.D.N.Y. 2012)
Approving Predictive Coding
Global Aerospace v. Landau Aviation (Virginia) April 23, 2012
Unanimous Approval
Nat’l Day Laborer Org. Network v. U.S. Immigration & Customs Enforcement Agency, No. 10 Civ. 2488 (SAS), 2012 WL 2878130 (S.D.N.Y. July 13, 2012).
EORHB, Inc. v. HOA Holdings, LLC, No. 7409-VCL (Del. Ch. Oct. 15, 2012).
Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL 61040 (Vir. Cir. Ct. Apr. 23, 2012).
In re Actos (Pioglitazone) Prods. Liab. Litig., MDL No. 6:11-MD-2299 (W.D. La. July 27, 2012).
2012
Unanimous Approval
2013Gordon v. Kaleida Health, No. 08-CV-378S(F), 2013 WL 2250579 (W.D.N.Y. May 21, 2013).
In re Biomet M2a Magnum Hip Implant Prods. Liab. Litig., 2013 U.S. Dist. LEXIS 84440 (N.D. Ind. Apr. 18, 2013).
In re Biomet M2a Magnum Hip Implant Prods. Liab. Litig., 2013 U.S. Dist. LEXIS 172570 (N.D. Ind. Aug. 21, 2013).
EORHB, Inc. v. HOA Holdings, LLC, No. 7409-VCL, 2013 WL 1960621 (Del. Ch. May 6, 2013).
Gabriel Techs., Corp. v. Qualcomm, Inc., No. 08CV1992 AJB (MDD), 2013 WL 410103 (S.D. Cal. Feb. 1, 2013).
Unanimous Approval
2014
Bridgestone Americas, Inc. v. Int. Bus. Machs. Corp., No. 3:13-1196 (M.D. Tenn. July 22, 2014).
Dynamo Holdings Ltd. P’ship v. Comm’r of Internal Revenue, Nos. 2685-11, 8393-12 (T.C. Sept. 17, 2014).
FDIC v. Bowden, No. CV413-245, 2014 WL 2548137 (S.D. Ga. June 6, 2014).
In re Bridgepoint Educ., Inc., No. 12cv1737 JM (JLB), 2014 WL 3867495 (S.D. Cal. Aug. 6, 2014).
Progressive Cas. Ins. Co. v. Delaney, No. 2:11-cv-00678-LRH-PAL, 2014 WL 2112927 (D. Nev. May 20, 2014).
The Law
“The case law has developed to the point where it is black letter law that if a party wants to utilize TAR for document review, courts will permit it.”
Discussion and Q&A
Let’s continue the discussion.
Thank you!
Gordon J. Calhoun
John Tredennick