rigor and reproducibility - massachusetts general hospital · rcr: rigor and reproducibility in...

20
Office for Research Career Development - RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility Dennis Brown, Ph. D. Professor of Medicine Massachusetts General Hospital and Harvard Medical School, Boston, USA Director: MGH Program in Membrane Biology Former study section Chair, NIDDK Former journal editor Rigor and Reproducibility Rigor Robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results Reproducibility Ability of multiple scientists to “duplicate” (validate) original results

Upload: others

Post on 23-Jun-2020

5 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 1

Rigor and Reproducibility

Dennis Brown, Ph. D. Professor of MedicineMassachusetts General Hospital and Harvard Medical School, Boston, USA

Director: MGH Program in Membrane BiologyFormer study section Chair, NIDDKFormer journal editor

Rigor and Reproducibility

• Rigor

Robust and unbiased experimentaldesign, methodology, analysis,interpretation and reporting of results

• Reproducibility

Ability of multiple scientists to“duplicate” (validate) original results

Page 2: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 2

What is the problem?

Pre-clinical results “crisis”

Page 3: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 3

Pre-clinical results “crisis”

Pre-clinical results “crisis”

Page 4: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 4

Pre-clinical results “crisis”

Pre-clinical results “crisis”

Page 5: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 5

Why does this happen (1)?

• Pressure to publish –poor/superficial experimental approach

– competition

– seeking a “breakthrough” story

– funding stress

–promotion anxiety

–ego………..

Funding stress

Page 6: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 6

Why does this happen (2) – fraud!

Is science like following a recipe?

• Take any recipe book

• Follow the instructions exactly

• What does your cake look like?

• Is it what you expected?

• If not, why not?

Page 7: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 7

My results are not like yours!

Why is the recipe not reproducible?

• biology is complicated

• interpretation of data (e.g., Mueller report!)

• skills/competence of colleagues

• lab environment (temperature, humidity)

• different reagents/cells (quality, sources)

• animal strains and housing (e.g., microbiome)

• technical issues, equipment variability

• leadership, expectations, training

Page 8: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 8

How to address this problem?

1) Improving experimental design

2) Clearer reporting in publications

3) Changes to the grant writing process

R & R in Grant Applications

1) Address rigor of previous research as part of “premise” for your work

2) Include rigorous experimental design in “approach” section

3) Discuss biological variables

4) Authentication of key resources

Page 9: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 9

1) Rigor of previous research

Old way: provide background information and described how you will fill knowledge gaps

New way: assess strengths and weaknesses of published literature (including your own work). Discuss design, sex differences, species, reagents etc. How will you address and correct these prior issues

R and R - review concerns

“Basing one's proposed research on previous publications that lacked statistical power, were not blinded, lacked detail on the sex of animals or authentication of cell lines would be considered a weakness of the application if it does not identify these weaknesses and propose ways to improve going forward”

Page 10: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 10

2) Rigorous experimental design

• “Approach” section of grant

• I use a separate R and R heading

• Follow “ARRIVE” guidelines for animals

• Follow Good Laboratory Practices (GLP)

• Address sex and gender requirements

• Describe statistics, power analysis

2a) Rigorous design - animals

Animal studies: we will follow the ARRIVE guidelines for reporting animal studies (species, strain, sex, age, source, genetic modifications, housing, diet); controls used in the study (littermates, whether mice are purchased or not, conditions of experiment); precise details of procedures (drug formulation, dose, anesthesia, analgesia, euthanasia);minimization of bias (blinding of researchers, randomization).

Page 11: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 11

ARRIVE

• Animal

• Research:

• Reporting of

• In

• Vivo

• Experiments• https://www.nc3rs.org.uk/arrive-guidelines

2b) Rigorous design - statistics

This calculation was completed based upon some of our preliminary data (e. g., Fig. 15) showing a 15-45% change in V-ATPase assembly at a power of 96% at 5% type 1 error rate (two-tailed P < 0.05, n = 5). A stringent statistical analysis will always be performed: ANOVA followed by post-hoc Bonferroni or Tukey’s tests for multi-group analysis, and Student t test for paired or unpaired samples when only two groups are included. Collection, analysis and interpretation of data will be conducted in a “blinded manner” in de-identified samples.

Page 12: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 12

3) Biological variables - sex

“We are very conscious of this issue (our refs). Age-matched adult males and females will be included in our experimental groups for all initial experiments. Only if no significant differences are found will we resort to using males alone, a strategy recommended by NIH during numerous informational events”

4) Authentication of key resources

• Briefly describe methods to ensure the identity and validity of key biological and/or chemical resources used in the proposed studies.

• One page or less – no actual data

• Reviewers will be asked to assess the adequacy of the plans you propose

Page 13: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 13

4a) Authentication of key resources

• Antibodies

• Cell lines

• Serum (for cell culture media)

• Drugs

• Exotic chemicals (not NaCl, sucrose etc)

• Can extend to modified animals

• Reagents that may differ from lab to lab

4b) Authentication: antibodies

• We have developed a systematic process by which all antibodies, whether home-made, shared by other investigators, or commercial are validated.

• Full length westerns

• Series of negative controls (e. g., antigen absorption)

• Cells + or – the target protein

• KO mouse if available

Page 14: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 14

4c) Authentication: cell lines

• ATCC number if available (e.g., CRL2038)

• Purity of cell lines (STR - “Authentikit”)

• Confirm genome modification (PCR, Southerns)

• Other published standards (e.g., morphology, function)

• Sex of cell lines

• https://grants.nih.gov/grants/guide/notice-files/NOT-OD-17-068.html

4d) Medications (human study)

• The medication was provided by Foundation Care Pharmacy (Earth City, MO) for our pilot clinical trial (CT.gov: NCT02479451), and will continue to be provided by Foundation Care for the double-blind randomized controlled clinical trial.

• The methods for authentication of theophylline methylpropyl paraben in a 0.4-mL saline solution and corresponding placebo include a series of tests that have been applied to the medication in our pilot clinical trial and will be applied to each batch of medication used for the proposed double-blind randomized controlled clinical trial:

• Theophylline (20 mcg/0.4 mL) nasal spray will undergo microbiology testing. Purity and sterility will be assessed. A fungal report will also be issued. The medication will be considered acceptable if endotoxin is below 10500 EU/mL and pH is between 5.5 and 8.0. After the initial sterility and the initial fungal report, the samples will be incubated for 14 days and if there is any change in the sample a supplemental report will be issued.

Page 15: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 15

The problem with journals

• Tactics to increase/maintain impact factor

• Reluctance to publish negative findings

• Push for “fashionable” science

• Inexperienced “gatekeeper” editors making quick, superficial decisions

• It’s partially our own faultWe are study sections

We are referees for journals

Page 16: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 16

The quest for high impact

Data overload – is more better?

Page 17: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 17

How are journals helping R & R?

Many journals have new procedures:

• Allowing extensive methods section

• Not included in page limitations

• Pre-submission checklists

• Post-acceptance requirements

• New guidelines for reviewers

• More pre-screening of figures

Specific requirements

• Datasets in public repositories

• Statement on sharing datasets

• Provide original gels, images etc

• Multiple replicates often required

• Validation of reagents (antibodies, cells)

• Sex and gender requirements

Page 18: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 18

STAR Methods (Cell Press)

STAR Methods (Cell Press)

Page 19: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 19

Registered Reports

Promoted by the Center for Open Science

New type of manuscript (like clinical trials)

1) Develop idea, design study

2) Submit for stage 1 peer review

3) Upon provisional acceptance, collect, analyze data – write and submit

4) Publish report after 2nd review

Registered Reports (172 journals so far)

Benefits:

• Supports rigorous methods and analysis

• Eliminates bias against negative results

• Reduces tendency to show “beautiful” rather than an accurate story

Downsides?

Let’s discuss

Page 20: Rigor and Reproducibility - Massachusetts General Hospital · RCR: Rigor and Reproducibility in Scientific Research April 23, 2019 Dennis Brown, MGH, 2019 1 Rigor and Reproducibility

Office for Research Career Development -RCR: Rigor and Reproducibility in Scientific Research

April 23, 2019

Dennis Brown, MGH, 2019 20

R & R Resources

https://www.nih.gov/research-training/rigor-reproducibility

https://grants.nih.gov/policy/reproducibility/index.htm