from%informa*on%retrieval%(ir)%to% …...list of 60 results for varicella vaccine can cause...

Copyright Kevin D. Ashley, Vern R. Walker. 2013 1

From Informa*on Retrieval (IR) to Argument Retrieval (AR) for Legal Cases:

Report on a Baseline Study

Kevin D. Ashley a and Vern R. Walker b

a University of Pi;sburgh School of Law, Intelligent Systems Program

b Maurice A. Deane School of Law, Hofstra University Research Laboratory for Law, Logic and Technology

With thanks to Matthias Grabmair and Prof. Eric Nyberg, CMU.

Summary

n  Commercial legal informaIon retrieval (IR) system users oLen want argument retrieval (AR), retrieving: q  not just sentences with highlighted terms, but q  arguments and argument-‐related informaIon.

n  We conducted baseline study of how two legal IR systems q  responded to standard queries q  using a corpus of argument-‐annotated legal cases.

n  We idenIfy ways in which IR systems do not meet the need for AR, q  illustrate how addiIonal argument-‐relevant informaIon could address some

inadequacies, and q  briefly describe how to develop an AR system to retrieve argument-‐related

informaIon from legal decisions.


Vaccine/Injury Project (V/IP) Corpus* n  Legal decisions: does claim comply with NaIonal Vaccine Injury

CompensaIon Program? n  Claimant compensated IFF vaccine caused the injury.

q  Althen v. Secr. of Health and Human Services, 418 F.3d 1274 (Fed.Cir. 2005): q  the peIIoner must establish, by a preponderance of the evidence, that:

1.  a “medical theory causally connects” the type of vaccine with the type of injury; 2.  there was a “logical sequence of cause and effect” between the parIcular vaccinaIon

and the parIcular injury; and 3.  a “proximate temporal relaIonship” existed between the vaccinaIon and the injury.

n  Court of Federal Claims “Special Masters”: q  decide which evidence is relevant to which issues of fact, q  evaluate plausibility of evidence in the legal record, q  organize evidence and draw reasonable inferences, and q  make findings of fact.

n  Corpus = all decisions in 2-‐years applying Althen test of causaIon-‐in-‐fact q  35 decision texts, 15-‐40 pages per decision

Copyright Kevin D. Ashley, Vern R. Walker. 2013 3 * Research Laboratory for Law, Logic and Technology (LLT Lab), Maurice A. Deane School of Law at Hofstra University in New York.

Default Logic Framework (DLF)


!ParIal Rule Tree for Vaccine Decisions, Showing Three CausaIon CondiIons of Althen

1. DLF AnnotaIons


Ms. Cusati has provided more than preponderant evidence that Eric’s intractable seizure disorder led to Eric’s death…. Dr. Kinsbourne and Dr. Kohrman agree that MMR vaccine causes fever…. Dr. Kinsbourne and Dr. Kohrman agree that fever causes seizures…. Dr. Kinsbourne and Dr. Kohrman agree that a child who suffers a complex febrile seizure has a greater chance of developing epilepsy…. As such, Dr. Kohrman’s reports and testimony, and the medical literature, do not assist the special master in evaluating Ms. Cusati’s “legal cause” claim.

2. PresupposiIonal Concepts: EnIIes, Events & RelaIons Seman*c Rela*ons Meaning (objects or event referents)

1. Covered-‐vaccine a vaccine covered by the VICP

2. Specific-‐date a specific month, day, year

3. Specific-‐vaccinaIon a vaccinaIon with a Covered-‐vaccine on a Specific-‐date

4. Generic-‐injury a type of injury, adverse condiIon or disease

5. Injury-‐onset a symptom, sign or test result associated with the onset of a Generic-‐injury

6. Onset-‐Iming Ime interval between Specific-‐vaccinaIon and the earliest Injury-‐onset

C1. Medical-‐theory-‐asserIon asserIon that a medical theory causally connects vaccinaIon with a Covered-‐vaccine with the occurrence of a Generic-‐injury

C2. Causal-‐chain-‐asserIon asserIon that a Specific-‐vaccinaIon caused an instance of a Generic-‐injury


Noun Phrases: 1-‐6; Causal Asser*ons: C1-‐C2

Baseline Study n  Ten cases in V/IP Corpus involving the first Althen condiIon:

q  five for peIIoner (Cusa7, Casey, Werderitsh, Stewart, Roper) q  five for government (Walton, Thomas, Meyers, Sawyer, Wolfe).

n  Each case used as a “source case” to construct a standard search query based on its facts by: q  subsItuIng values from each “source” case for Covered-‐vaccine, Generic-‐

injury, and Onset-‐Iming into two templates: q  Q1. <Covered-‐vaccine> can cause <Generic-‐injury> q  Q2. <Covered-‐vaccine> can cause <Generic-‐injury> <Onset-‐Iming> q  For example:

n  Casey Q1: “Varicella vaccine can cause encephalomyeloneuriIs”. n  Casey Q2: “Varicella vaccine can cause encephalomyeloneuriIs within four

weeks”.

Copyright Kevin D. Ashley, Vern Walker. 2013 7

Westlaw Next Results List & Case Reports: Casey Q2


Ashley, Kevin 08/11/2013For Educational Use Only

List of 60 results for Varicella vaccine can cause encephalomyeloneuritis within four weeks

© 2013 Thomson Reuters. No claim to original U.S. Government Works. 1

1. Doe/17 v. Secretary of Health and Human ServicesUnited States Court of Federal Claims. December 16, 2008 84 Fed.Cl. 691 2008 WL 5330496

HEALTH - Vaccines. Special Master in Vaccine reasonably found lack of credibility as to claimant, given evidentiarycontradictions.

... Petitioner's theory of causation posited that she had a pre-existing condition of CVID, which was significantly aggravatedby her varicella vaccinations, as evidenced, in part, by the onset of diarrhea and other flu-like symptoms approximately twoweeks after receiving the second varicella vaccination on June 4, 2001....

...Special Master did not act arbitrarily and capriciously, in “off-table” Vaccine Act case in which claimant allegedaggravation of preexisting condition after receiving varicella vaccine, by relying on claimant's Social Security disabilityrecords preceding vaccinations; records showed that claimant had suffered from chronic irritable bowel syndrome (IBS)before receiving vaccinations, which was contrary to her representations in Vaccine Act claim, and undermined her theoryof causation and her credibility. National Childhood Vaccine...

...Claimant in “off-table” Vaccine Act case must present: (1) medical theory causally connecting vaccination and injury; (2)logical sequence of cause and effect showing that vaccination was reason for injury; and (3) showing of proximate temporalrelationship between vaccination and injury. National Childhood Vaccine Injury Act, 42 U.S.C.A. §§300aa–11(c)(1)(C)(ii),300aa–13(a)(1)(A)....

... At a scheduled interview twenty-four weeks after receiving the second vaccination there were no adverse effects of thevaccine recorded....

2. Stapleford ex rel. Stapleford v. Secretary of Dept. of Health and Human ServicesUnited States Court of Federal Claims. October 05, 2009 89 Fed.Cl. 456 2009 WL 3380616 03-234V

HEALTH - Vaccines. Child's seizures and developmental delays were not caused by varicella immunization.

... Having concluded that the varicella vaccine could not have caused Devon's injury, the Special Master held that thevaricella vaccine did not cause Devon's injury in this case....

...To establish causation-in-fact for an off-table claim under the National Childhood Vaccine Injury Act, petitioner's burden isto show by preponderant evidence that the vaccination brought about her injury by providing: (1) a medical theory causallyconnecting the vaccination and the injury; (2) a logical sequence of cause and effect showing that the vaccination wasthe reason for the injury; and (3) a showing of a proximate temporal relationship between vaccination and injury. NationalChildhood Vaccine Injury Act of 1986, §311(a), 42 U.S.C.A. §300aa–13(a)(1)(A)...

... In sum, the Special Master concluded that the varicella vaccine did not cause Devon's injury and, therefore, petitioner wasnot entitled to compensation under the Vaccine Act....

...Medical record and treating physicians' opinions contained therein did not provide substantial evidence that child's seizureswere caused in fact by varicella immunization, as required for compensation, under National Childhood Vaccine Injury Act;medical record reflected general consensus among treating physicians that cause of child's seizures was unknown and thatscientific literature found no association between varicella vaccine and seizures. National Childhood Vaccine Injury Act of1986, §311(a), 42 U.S.C.A. §300aa–13(a)(1)(A)....

3. Althen v. Secretary of Health and Human ServicesUnited States Court of Appeals, Federal Circuit. July 29, 2005 418 F.3d 1274 2005 WL 1793399 04-5146

Lexis Advance Results List & Case Reports: Casey Q2


6,484,948forVaricella vaccine can cause encephalomyeloneuritis...

Document Title Jurisdiction Court Date

1. Casey v. Sec’y of HHS, 2005 U.S. Claims LEXIS 391

In sum, petitioner was vaccinated against varicella on June 9, 1995. The attenuated virusin the varicellavaccine both directly attacked petitioner’s nervous system and caused animmune-mediated inflammatory response in her nervous system. As a result, withinfour-weeks of her varicella vaccination, petitioner began to experience the onset of symptomsof her encephalomyeloneuritis. One such symptom was cerebellitis, a condition particu-larly associated with a natural varicella infection, which ...

Overview:Petitioner was able to prove by a preponderance of evidence that avaricella vaccine was the cause in fact of her neurological injuries, rendering herentitled to compensation pursuant to he National Childhood Vaccine Injury Act,42 U.S.C.S. §§ 300aa-1 to 300aa-34.

U.S. Federal Court ofFederalClaims

December12, 2005

2. Coppola v. Sec’y of HHS, 2012 U.S. Claims LEXIS 321

After the hearing, Petitioner identified as the primary issue whether the varicellavaccin-ecancause SIRS 72 hours after its administration in a 56-year old man. Pet’r’s Post-Hr’gBr. at 2. Petitioner stated that ! [t]he only peer-reviewed literature available on thesubject simply states that adverse reactions to the varicellavaccine have not beenreported within seventy-two hours in petitioner’s age group.! Id .

Overview:In action under Vaccine Act, petitioner had not established primafacie case of causation because, inter alia, there was no reliable theory explain-ing how varicella vaccination could have caused symptoms of which petitionercomplained, and there was not preponderant evidence of a logical sequence ofcause and effect between vaccination and illness.


March 07,2012

3. Tompkins v. Sec’y of the HHS, 2013 U.S. Claims LEXIS 842

This news release, coupled with Dr. Pike’s assertions and VAERS research, does notconstitute preponderant evidence that the meningococcal vaccinecancause GBS, particu-larly in light of the absence in the ensuing seven years of any case reports, studies, orother evidence suggesting that the spike in cases was more than coincidence. Further-more, a 2008 study based on a mass meningococcal vaccination campaign in Canadashowed no increased risk of GBS within eight weeks of vaccination. Haber ...

Overview:Petition for compensation under National Vaccine Injury Compensa-tion Program was dismissed because decedent’s respiratory infection, whichbegan two weeks before onset of Guillain-Barré syndrome (GBS) symptoms,was well-recognized cause of GBS, occurred at appropriate temporal intervalbefore onset of symptoms, and was most likely cause for GBS.


June 21,2013

4. Bast v. Sec’y of HHS, 2012 U.S. Claims LEXIS 1721

... received IPV, DTaP, HIB, and the two attenuated vaccines--MMR and varicella. Incontrast, Makena received the IPV, DTaP, HiB and hepatitis B vaccinations. See Pet’r’sEx. 1 at 166. Third, Hannah’s symptoms, including a high fever, presented within 48hours after her vaccinations. Makena’s symptoms, which did not include fever, did notpresent until more than fourweeks after she received the vaccines at issue. Fourth,Hannah’s onset symptoms of inconsolable crying, irritability, ...

Overview:Petitioner was not entitled to compensation in a case brought underthe Vaccine Act where petitioner alleged that her child suffered seizures, anencephalopathy, and liver damage as a result of a hepatitis B vaccinationbecause petitioner’s theories were neither scientifically tenable nor factuallysupported by the record.


December20, 2012

5. Kathaleen v. Sec’y of HHS, 2009 U.S. Claims LEXIS 456

... the MMR vaccination. (Tr. 41.) In the case of DTP vaccinations, the increased risk ofseizures was within the first 48 hours post-vaccination, but in the case of MMR the risk


May 01,2009

Kevin Ashley

User Name: Kevin AshleyDate and Time: 08/11/2013 12:43 PM EDTJob Number: 4181459

Results List (includes up to 250)

1. Results List for: Varicella vaccine can cause encephalomyeloneuritis within four weeks

Content Type: CasesTerms: Varicella vaccine can cause encephalomyeloneuritis within four weeks

Search Type: Natural Language

| About LexisNexis | Privacy Policy | Terms & Conditions | Copyright © 2013 LexisNexis.

Kevin Ashley

Results of Baseline Study


Results of Baseline Study (cont.)


Results: Rank-‐Ordered Lists of Cases Retrieved

n  Goal: searches should return most relevant cases at top of rank-‐ordered results list. q  For all 40 queries, % of returned cases in top-‐10 cases that were federal

vaccine compensaIon cases always at least 90% (Col.4). ✔ n  Goal: source case, whose facts formed basis for the query, should

be among highly relevant cases. q  LA: source case in top-‐10 cases of Results List 75% of Ime (Col. 5) , ✔

n  in top-‐2 cases 60% of Ime. n  For 2 queries, LA returned source case but not in the top-‐10 cases.

q  WN: returned source case in top-‐10 50% of Ime (Col. 5), ✔− n  in top-‐2 15% of Ime. n  For remaining 10 queries, WN returned source case, but not in top-‐10 cases.


Results: Results-‐List Case Reports

n  For all 40 queries’ Results Lists: q  we determined if Case Reports for source case and top-‐10 cases q  included all elements sought by query (Col. 6 & 7).

n  Goal: Source’s Case Report should be Q1-‐ and Q2-‐complete: q  Q1-‐complete: three elements:

1.  a parIcular vaccine, 2.  a parIcular alleged injury, 3.  an indicaIon that the vaccine caused the injury.

q  Q2-‐complete adds: 4.  an indicaIon of Ime-‐to-‐onset between the vaccinaIon and the

manifestaIon of the injury.

n  Goal: Top-‐10 cases should include some Q1-‐ and Q2-‐complete Case Reports.


Results: Results-‐List Case Reports (cont.) Policies for determining if Case Report is Q1-‐ and Q2-‐complete: Report should include: 1.  vaccine by:

q  name (e.g., measles-‐mumps-‐rubella), q  well-‐known iniIals (e.g., MMR), or q  alternaIve name (e.g., commercial brand name);

2.  injury by name (e.g., myocardiIs) or alternaIve names; 3.  for Q2, some specific Ime period; 4.  Regarding causaIon, report should include asserIon or

direct implicaIon, q  for Q1, that the vaccine can[not] cause the injury. q  for Q2, that the vaccine can[not] cause the injury in the specific Ime

period.


Results: Results-‐List Case Reports (cont.) For 40 queries: n  Re source case:

q  On 4 occasions, WN’s Case Report for source case was not Q2-‐complete even though the query was based on its facts (Col. 7). ✔− n  For LA, this occurred twice. ✔− n  WN reported Roper source case at rank Q1(21) and Q2(11) even though it was

only case re gastroparesis in the WN Results List for Q1 or Q2.

n  Re top-‐10 cases: q  LA always returned at least one case in top-‐10 whose Case Report was

“Q1-‐complete”or “Q2-‐complete” (Col. 6). ✔ q  WN did not find any Q1-‐complete cases in top-‐10 for 3 queries or any Q2-‐

complete cases in the top-‐10 for 4 queries. ✔−


Results: Case Report Decision Abstracts

n  Goal: Case Reports for Top-‐10 cases should make clear which side won: 1.  ulImate claim for vaccine compensaIon (i.e., peIIoner or the

government); and 2.  causaIon sub-‐issue under Althen CondiIon 1.

n  For all 40 queries, informaIon about winners much less frequent in Case Reports in WN than in LA. q  WN reported ulImate-‐claim outcomes about 29% of the Ime and

causaIon sub-‐issue outcomes about 47% of the Ime. ✔− q  LA reported ulImate-‐claim outcomes about 88% of the Ime and

causaIon sub-‐issue outcomes about 93% of the Ime. ✔


From IR Systems to AR Systems

Baseline study shows: n  Legal IR systems return relevant cases with natural language

queries and probabilisIc criteria. n  Ways IR systems’ performance is not ideal for AR –

q  retrieval precision (e.g., source cases not ranked highly in Results Lists) q  Case Reports do not focus on retrieved case features relevant to query. q  Decision Abstracts do not make clear who won what claims or issues.

n  Hypothesis: AR tasks possible if system can: q  idenIfy / use semanIc, presupposiIonal and DLF informaIon relevant

to legal argument (i.e., semanIc and pragmaIc legal informaIon).


If AR can idenIfy presupposiIonal info re vaccines: (i.e., Covered-‐vaccine, Specific-‐vaccinaIon)

1.  Address some co-‐reference problems. Cases refer to vaccines ito:

q  generic names (“varicella”) q  popular names (“Chickenpox”) q  commercial brand names (“VARIVAX”) q  in composite vaccines (Quadrigen vaccine combines DPT and polio

vaccines)

n  In Thomas queries (“DPT vaccine can cause acute encephalopathy and death”), cases involving asserIons re Quadrigen-‐caused injuries may be relevant.


If AR can idenIfy presupposiIonal info re temporal relaIons: (i.e., Specific-‐vaccinaIon, Injury-‐onset, Onset-‐Iming) 2. Perform simple temporal reasoning.

q  In Wolfe Q2 queries, neither WN nor LA ranked source case in top-‐10 cases. n  Query: “HepaIIs B vaccine can cause intractable seizure disorder aLer about

one day” n  Wolfe does not menIon “one day” or “day”, but

q  menIons “12 hours” in the following sentence: q  “The temporal relaIonship between the immunizaIon and the chain of seizure acIvity

which followed, starIng within the 12 hours aLer the immunizaIon, compel [sic] one to conclude that there is a causal relaIonship between the two.”

q  In Walton Q2, following seem to have misled LA: n  Query: “MMR vaccine can cause myocardiIs aLer over three weeks” n  Cases IR returned:

q  “three test cases”, “all three special masters”, “three theories”, q  “several weeks of evidenIary hearings”, q  “aLer the vaccine (i.e., no Ime specified)”, “aLer the designaIon of this case”, “aLer

determining the evidence”.


If AR system can idenIfy presupposiIonal info re injuries: (i.e., Generic-‐injury, Injury-‐onset) 3.  Filter sentences / cases not relevant to reasoning about

injuries. q  Sawyer queries sought cases involving injuries to the arm that might have

been caused by the tetanus vaccine. q  Query: “Tetanus can cause hand, wrist and arm injuries at any Ime” q  Cases IR returned:

n  Garcia involved vaccinaIons “in the right and leL arms”; n  Pociask involved a vaccinee with a chronic arm problem; and n  Hargrove decision referred to “the site that the anIgen was introduced [the

arm].”


Use logical structure & pragmaIc legal context 4.  to focus AR on key sentences:

q  OLen, few sentences in lengthy opinion capture significant reasoning. q  User argument goals makes some sentences more relevant than others.

n  For example, purpose behind queries like Q1 or Q2 is fact-‐oriented. n  In Casey search, Case Reports for cases WN ranked higher than source case

contained re-‐statements of Althen rule. q  They contain search terms, but no info about specific vaccinaIons, injuries, or

duraIons. n  AR could disInguish sentences that re-‐state the law from those

that apply the law to facts using: q  DLF (Default Logic Framework) rule tree showing Althen rule q  presence / absence of presupposiIonal enIIes/events (e.g., Covered-‐

vaccine, Generic-‐injury, and Onset-‐Iming) q  citaIon text analyzer idenIfying appellate courts (rule-‐se;ers)


Use logical structure & pragmaIc legal context…

… so that AR can 5.  rank cases more effecIvely and improve precision.

q  Some knowledge of pragmaIc legal context (e.g., which presupposiIonal info fulfilled or not) may improve precision where

q  source case is only Q1-‐ or Q2-‐complete case returned, but source is sIll not ranked at top of Results List (i.e., Roper, Casey, Stewart, Meyers, and Walton, Table 2, Col. 5).

6.  generate more informaIve summary of case decisions. q  determine issues addressed from DLF structure & presupposiIonal info, q  conclusion reached (the finding of fact) on each decided issue (i.e., which

party won or lost).


Challenges: A;ribuIng (Embedded) AsserIons: n  “A;ribuIon problem”: determining parIcipant to whom AR

system should a;ribute the asserIon. q  Sentence that expresses causal asserIon may be:

n  recounIng of allegaIons in the case, n  entry in a medical record, n  tesImony of an expert witness, n  finding of the fac{inder.

q  E.g., in Walton Q2 search, LA ranked source case first: n  Sentence recounted causal tesImony of witness. n  But, Special Master discounted tesImony, held against peIIoner.

n  Analyze “Dr. Smith tesIfied that the vaccine can cause the injury”: a)  embedded asserIon (“the vaccine can cause the injury”) and b)  the person to whom the asserIon should be a;ributed (e.g., Dr. Smith).


Challenges: presupposiIonal info phrased differently:

n  If user’s goal seeks arguments relevant to proving or disproving Althen CondiIon 1, then q  asserIons about general causaIon (Medical-‐theory-‐asserIons) and

specific causaIon (Specific-‐causaIon-‐asserIons) are important.

n  Presents lexical challenges: q  recognizing alternaIve ways to express causal relaIons, such as

“results in” and “brings about”.

n  PragmaIc challenges: q  recognizing successful / unsuccessful ways of reasoning to causal

conclusions.


Argument Retrieval Pipeline


Integrated pipeline of open-‐source UIMA soLware components that: 1.  takes full-‐text legal decisions re scienIfic evidence of causaIon, 2.  extracts syntacIc and argument-‐related semanIc and pragmaIc (contextual/discourse)

informaIon, and 3.  uses it to improve IR precision, report extracted arguments, and suggest new evidence and

arguments.

References: re automaIc semanIc processing of case decision texts for legal IR: n  SPIRE retrieved cases and highlighted passages relevant to bankruptcy law factors

(Daniels, Rissland 1997). n  SMILE+IBP classified case texts in terms of factors and predicted outcomes (Ashley,

Brüninghaus 2009). n  Assigned rhetorical roles to case sentences based on 200 manually annotated Indian

decisions (Saravanan, Ravindran 2010). n  Categorized legal cases by Westlaw categories (e.g., bankruptcy, banking) (Thompson

2001) or general topics (e.g., excepIonal services pension, reIrement) (Gonçalves, Quaresma 2005).

n  Extracted treatment history (e.g., ‘‘affirmed’’, ‘‘reversed in part’’, or ‘‘overruled’’) (Jackson, et al. 2003).

n  Classified sentences as argumentaIve based on manually classified sentences from court reports and generated argument tree structures (Mochales, Moens 2011).

n  Determined role of sentence in legal case (e.g., as describing the applicable law or the facts) (Hachey, Grover 2006).

n  Extracted from criminal cases, offenses raised and legal principles applied to generate summaries (Uy;endaele 1998).

n  Extracted holdings of legal cases (McCarty 2007). Copyright Kevin D. Ashley, Vern R. Walker. 2013 26

Conclusions n  Baseline study suggests:

q  current legal IR systems are effecIve at returning relevant cases, but q  do not support AR.

n  Module added to full-‐text legal IR system could: q  extract semanIc / pragmaIc legal informaIon from top n cases returned

and q  analyze them to improve retrieval precision and q  construct be;er Case Reports and Decision Abstracts.

n  Future work: q  Using V/IP corpus as training data and test bed, q  program could learn to extract semanIc / pragmaIc legal informaIon

from new case texts with UIMA-‐based mulI-‐level annotaIon approach q  similar to DeepQA architecture of IBM Watson quesIon-‐answering

system (Ashley & Walker, ICAIL 2013).


from%informa*on%retrieval%(ir)%to% …...list of 60 results for varicella vaccine can cause...

Documents