when is reproducibility an ethical issue? genomics ...€¦ · genomic signatures 2 using cell...
TRANSCRIPT
-
When is Reproducibility an EthicalIssue? Genomics, Personalized
Medicine, and Human Error
Keith A. BaggerlyBioinformatics and Computational Biology
UT M. D. Anderson Cancer [email protected]
Hasselt Hexa-Symposium, Nov 15, 2013
-
GENOMIC SIGNATURES 1
Why is Reproducibility Important in Genomics?With “Big Data” in General?
Our intuition about what “makes sense” is very poor in highdimensions.
To use “genomic signatures” as biomarkers, we need to knowthey’ve been assembled correctly.
Without documentation, we may need to employ forensicbioinformatics to infer what was done to obtain the results.
Let’s examine a case study involving an important clinicalproblem: can we predict how a given patient will respond toavailable chemotherapeutics?
-
GENOMIC SIGNATURES 2
Using Cell Lines to Predict Sensitivity
Potti et al (2006), Nature Medicine, 12:1294-1300.
The main conclusion is that we can use microarray data fromcell lines (the NCI60) to define drug response “signatures”,which can be used to predict whether patients will respond.
* They provide examples using 7 commonly used agents.
* This got people at MDA very excited.
-
GENOMIC SIGNATURES 3
Their Gene List and Ours
> temp
-
GENOMIC SIGNATURES 4
Predicting Docetaxel Response
Potti et al, Nat Med 2006, 12:1294-300, Fig 1d
Chang et al, Lancet 2003, 362:362-9, Fig 2 top
-
GENOMIC SIGNATURES 5
Predicting Adriamycin Response
Potti et al, Nat Med 2006, 12:1294-300, Fig 2c
Holleman et al, NEJM 2004, 351:533-42, Fig 1
-
GENOMIC SIGNATURES 6
Trying it Ourselves
When we try it, it doesn’t work. This took 6 months.
-
GENOMIC SIGNATURES 7
A Repro Theme: Don’t Take My Word For It!
Read the paper: Coombes, Wang & Baggerly, Nat Med, Nov6, 2007, 13:1276-7, author reply 1277-8.
Try it yourselves: All of the raw data, documentation*, andcode* is available from our web site (*and from Nat Med):
http://bioinformatics.mdanderson.org/Supplements/ReproRsch-Chemo.
http://bioinformatics.mdanderson.org/Supplements/ReproRsch-Chemohttp://bioinformatics.mdanderson.org/Supplements/ReproRsch-Chemo
-
GENOMIC SIGNATURES 8
Potti/Nevins Rebuttal (Nat Med 13:1277-8)
Labels for Adria are correct – details on their web page.
They’ve gotten the approach to work again. (Twice.)
-
GENOMIC SIGNATURES 9
Adriamycin 0.9999+ Correlations (Reply)
Redone Aug 08, “using ... 95 unique samples”.
-
GENOMIC SIGNATURES 10
Validation 1: Hsu et al
J Clin Oncol, Oct 1, 2007, 25:4350-7.
Same approach, using Cisplatin and Pemetrexed.
For cisplatin, U133A arrays were used for training.ERCC1, ERCC4 and DNA repair genesare identified as “important”.
With some work, we matched the heatmaps. (Gene lists?)
-
GENOMIC SIGNATURES 11
The 4 We Can’t Match (Reply)
203719 at, ERCC1,210158 at, ERCC4,228131 at, ERCC1, and231971 at, FANCM (DNA Repair).
Another problem –
The last two probesets aren’t on the U133A arrays used.They’re on the U133B.
-
GENOMIC SIGNATURES 12
Validation 2: Bonnefoi et al
Lancet Oncology, Dec 2007, 8:1071-8. (early access Nov 14)
Similar approach, using signatures for Fluorouracil,Epirubicin, Cyclophosphamide, and Taxotere to predictresponse to combination therapies: FEC and TET.
Potentially improves ER- response from 44% to 70%.
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 13
Experimental Design Matters
High Sample Correlationsafter Centering by Gene
Array Run Dates
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 14
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TETand FEC. Let P() indicate prob sensitive. The rules used areas follows.
-
GENOMIC SIGNATURES 14
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TETand FEC. Let P() indicate prob sensitive. The rules used areas follows.
P (TFAC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
-
GENOMIC SIGNATURES 14
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TETand FEC. Let P() indicate prob sensitive. The rules used areas follows.
P (TFAC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
P (ET ) = max[P (E), P (T )].
-
GENOMIC SIGNATURES 14
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TETand FEC. Let P() indicate prob sensitive. The rules used areas follows.
P (TFAC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
P (ET ) = max[P (E), P (T )].
P (FEC) =5
8[P (F ) + P (E) + P (C)]− 1
4.
-
GENOMIC SIGNATURES 14
How Are Results Combined?
Potti et al predict response to TFAC, Bonnefoi et al to TETand FEC. Let P() indicate prob sensitive. The rules used areas follows.
P (TFAC) = P (T )+P (F )+P (A)+P (C)−P (T )P (F )P (A)P (C).
P (ET ) = max[P (E), P (T )].
P (FEC) =5
8[P (F ) + P (E) + P (C)]− 1
4.
Each rule is different.
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 15
Predictions for Individual Drugs?
Does cytoxan make sense?
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 16
The Story Takes A Darker Turn
Jun 2009: we learn clinical trials had begun.2007: pemetrexed vs cisplatin, pem vs vinorelbine.2008: docetaxel vs doxorubicin, topotecan vs dox (Moffitt).
-
GENOMIC SIGNATURES 16
The Story Takes A Darker Turn
Jun 2009: we learn clinical trials had begun.2007: pemetrexed vs cisplatin, pem vs vinorelbine.2008: docetaxel vs doxorubicin, topotecan vs dox (Moffitt).
Sep 1, 2009: We submit a paper describing case studies tothe Annals of Applied Statistics.
Sep 14, 2009: Paper accepted and available online at theAnnals of Applied Statistics.
Sep-Oct 2009: Story covered by The Cancer Letter.NCI raises concerns with Duke’s IRB behind the scenes.Duke starts internal investigation, suspends trials.
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 17
Nov 2009: New Data
In mid-Nov (mid-investigation), the Duke team posted newdata for cisplatin and pemetrexed (in lung trials since ’07).
These included quantifications for the 59 ovarian cancer testsamples (from GSE3149, which has 153 samples) they usedto validate their predictor.
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 18
We Tried Matching The Samples
43 samples are mislabeled.16 samples don’t match because the genes are mislabeled.All of the validation data are wrong.
We reported this to Duke and to the NCI in mid-November.c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 19
Jan 29, 2010
Their investigation’s results “strengthen ... confidence in thisevolving approach to personalized cancer treatment.”
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 20
A Catalyzing Event: July 16, 2010
Jul 19/20: Letter to Varmus; Duke resuspends trials.Oct 22/9: First call for paper retraction.Nov 9: Duke terminates trials.Nov 19: call for Nat Med retraction, Potti resigns
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 21
Other Developments
IOM Review (1.5yrs); documentation.
117 patients were enrolled in the trials.Sep, 2011: Patient lawsuits filed (11+ settlements).
Misconduct investigation (ongoing).10 retractions, 6 corrections/partial retractions to date.
Jul 8, 2011: Front page, NY Times.Feb 12, 2012: 60 Minutes.http://www.cbsnews.com/8301-18560_162-57376073/deception-at-duke/
Mar 23, 2012: IOM Report Released.http://www.iom.edu/Reports/2012/Evolution-of-Translational-Omics.aspx
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
http://www.cbsnews.com/8301-18560_162-57376073/deception-at-duke/http://www.cbsnews.com/8301-18560_162-57376073/deception-at-duke/http://www.iom.edu/Reports/2012/Evolution-of-Translational-Omics.aspxhttp://www.iom.edu/Reports/2012/Evolution-of-Translational-Omics.aspx
-
GENOMIC SIGNATURES 22
Some Cautions/Observations/Lessons
This case is pathological.
But we’ve seen similar problems before.
The most common mistakes are simple.
Confounding in the Experimental DesignMixing up the sample labelsMixing up the gene labelsMixing up the group labels(Most mixups involve simple switches or offsets)
This simplicity is often hidden.
Incomplete documentation
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 23
This is not an Isolated Problem
Ioannidis et al. (2009), Nat. Gen., 41:149-55. Testedreproducibility of microarray papers. Could reproduce 2/18.
Begley and Ellis (2012), Nature, 483:531-3. Amgenattempted validation of clinical “breakthroughs” prior tofurther study. Validated 6/53.
Subject of an NCI focus meeting in Sep 2012.
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 24
Reasons for Hope: Doing Better
1. Our Own (Evolving!) Experience
2. Better tools (knitr, Markdown, Pandoc)
3. The IOM, the FDA, and IDEs*
4. The NCI and Trials it Funds
5. Journals, Code and Data
6. The Power of Ridiculehttp://www.youtube.com/watch?v=GgMV4KV6Qw0
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
http://www.youtube.com/watch?v=GgMV4KV6Qw0
-
GENOMIC SIGNATURES 25
Recent Links
OSTP, Feb 22, 2013 http://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdf
Science, March 6, 2013 http://www.aaas.org/news/releases/2013/0311_alberts.shtml
Nature, April 24, 2013 http://www.nature.com/news/announcement-reducing-our-irreproducibility-1.12852
Nature, BMC Medicine, Oct 17, 2013http://www.nature.com/nature/journal/v502/n7471/full/nature12564.html,http://www.biomedcentral.com/1741-7015/11/220
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
http://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdfhttp://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdfhttp://www.whitehouse.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdfhttp://www.aaas.org/news/releases/2013/0311_alberts.shtmlhttp://www.aaas.org/news/releases/2013/0311_alberts.shtmlhttp://www.nature.com/news/announcement-reducing-our-irreproducibility-1.12852http://www.nature.com/news/announcement-reducing-our-irreproducibility-1.12852http://www.nature.com/news/announcement-reducing-our-irreproducibility-1.12852http://www.nature.com/nature/journal/v502/n7471/full/nature12564.htmlhttp://www.nature.com/nature/journal/v502/n7471/full/nature12564.htmlhttp://www.biomedcentral.com/1741-7015/11/220
-
GENOMIC SIGNATURES 26
Acknowledgements
Kevin Coombes
Shannon Neeley, Jing WangDavid Ransohoff, Gordon MillsJane Fridlyand, Lajos Pusztai, Zoltan Szallasi
M.D. Anderson Ovarian, Lung and Breast SPOREs
For updates:http://bioinformatics.mdanderson.org/Supplements/ReproRsch-All/Modified
http://bioinformatics.mdanderson.org/Supplements/ReproRsch-All/Modified/StarterSet
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
http://bioinformatics.mdanderson.org/Supplements/ReproRsch-All/Modifiedhttp://bioinformatics.mdanderson.org/Supplements/ReproRsch-All/Modifiedhttp://bioinformatics.mdanderson.org/Supplements/ReproRsch-All/Modified/StarterSethttp://bioinformatics.mdanderson.org/Supplements/ReproRsch-All/Modified/StarterSet
-
GENOMIC SIGNATURES 27
Thanks!
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes
-
GENOMIC SIGNATURES 28
Index
TitleCell Line StoryTrying it OurselvesThe ReplyAdriamycin FollowupHsu et al. (Cisplatin)Bonnefoi et al.Timeline, Trials, Cancer LetterTrial Restart and ObjectionsFinal Lessons
c© Copyright 2007-2013, Keith A. Baggerly and Kevin R. Coombes