reproducible research chapter 4: central tendency … · dipak das (u conn)-- wine researcher...

REPRODUCIBLE RESEARCH&

CHAPTER 4: CENTRAL TENDENCY AND DISPERSION

profberg.com

Upcoming Conferences

March 19-23San Francisco

April 1-4Washington, DC

July 28 - August 2San Diego, CA

(Abstract submission deadline: Feb 1)

Wa! Street JournalDecember 2, 2011

“Reproducibility of scientific results and data is essentially the gold standard of the foundation of all modern research.”

--Gautam Naik (WSJ)

Reproducible Research“The goal of reproducible research is to tie specific instructions to data analysis and experimental data so that scholarship can be recreated, better understood and verified.”

CRAN Task View: Reproducible Research

Victoria Stodden, PhD, MLS Keith Baggerly, PhD

February 11, 2011 December 2, 2011

(V. Stodden, Computing in Science & Engineering, 2009)

Only 2 of the 18 articles on microarray expression gene profiling published in Nature Genetics from 2005-2006 were reproducible.

Reasons Not to Share

(Victoria Stodden)

Reasons To Share

(Victoria Stodden)

ScandalsClimategate (Climatic Research Unit email controversy)-- Eight committees investigated the allegations and published reports including many email exchanges involving Michael Mann, Director of Penn State’s Earth System Science Center.

Dipak Das (U Conn)-- Wine researcher accused of fraud in 26 articles.

Geoffrey Chang (Scripps Research Institute)--“unintentional” mistakes led to the retraction of 3 Science papers and 1 PNAS paper.

Jan Hendrik Schön (previously at Bell Labs)-- made-up data led to the retraction of 9 Science papers, 7 Nature papers, 6 Physical Review papers, 4 Applied Physics Letters papers, 2 Advanced Materials papers, and 8 other journal articles were flagged.

...journals should demand that authors submit sufficient detail for the independent assessment of their paper’s conclusions. We recommend that all primary data are backed up with adequate documentation and sample annotation; all primary data sources, such as database accessions or URL links, are presented; and all scripts and software source codes are supplied, with instructions. Analytical (non-scriptable) protocols should be described step by step, and the research protocol, including any plans for research and analysis, should be provided. Files containing such information could be stored as supplements by the journal.

Havard Arts & Sciences Faculty Adopt an Open-Access Policy

In legal terms, the permission granted by each Faculty member is a nonexclusive, irrevocable, paid-up, worldwide license to exercise any and all rights under copyright relating to each of his or her scholarly articles, in any medium, and to authorize others to do the same, provided that the articles are not sold for a profit.

To assist the University in distributing the articles, each Faculty member will provide an electronic copy of the final version of the article at no charge to the appropriate representative of the Provost’s Office in an appropriate format (such as PDF) specified by the Provost’s Office.

The Provost’s Office may make the article available to the public in an open-access repository.

Others Follow SuitStanford School of Education:

Faculty members grant to the Stanford University permission to make publicly available their scholarly articles and to exercise the copyright in those articles. They grant to Stanford University a nonexclusive, irrevocable, worldwide license to exercise any and all rights under copyright relating to their scholarly articles, in any medium, and to authorize others to do the same, provided that the articles are properly attributed to the authors not sold for a profit. The policy will apply to all scholarly articles authored or co-authored while a faculty member of the School of Education, beginning with articles for which the publisher’s copyright agreement has yet to be signed.

H1N1Of the 566 people who are known to have contacted H1N1, 322 have died. (~60% kill rate)

The virus consists of 8 pieces of RNA (~10 genes) that replicated independent of each other. There’s no built in error-correction.

Some random mutation might just bring back a major flu pandemic like the 1918 flu pandemic--“we better investigate”.

Dutch virologist Ron Fouchier altered the H1N1 genome in two places supposedly allowing the virus to pass from ferret to ferret.

He was blocked from publishing his results.

Back in 2005, researchers at Mount Sinai reconstructed the 1918 influenza virus from formalin-fixed autopsy material and frozen lung tissue from an Alaskan influenza victim.

Front Page of the NY TimesJuly 8, 2011

Other Potti Retractions:-NEJM-JAMA-PNAS-JCO-The Lancet Oncology-PLOS ONE-Blood

Data is Publicly Available

Drug Response: NCI-60 sensitivity/resistancehttp://dtp.nci.nih.gov/docs/cancer/cancer_data.html

Training Data: NCI-60 triplicate Affy expression arrayhttp://dtp.nci.nih.gov/mtargets/download.html

Testing Data: 24 breast cancer patients (published in Cheng et al (2003), Lancet)

http://dtp.nci.nih.gov/mtargets/download.html

http://dtp.nci.nih.gov/mtargets/download.html

The Drama...Potti et al get the wrong gene signatures, off-by-one gene lists, made up data, switch-up sensitive/resistant labels

Clinical trials initiated at Duke (2007) and Moffitt (2008) -- completely bogus design

Baggerly & Coombes publish their findings in the Annals of Applied Statistics (2009)

Duke launches internal investigation; suspends three trials

Trials resume because treatments are all standard of care therapies (six different chemotherapy drugs)

The Drama (continued)Duke’s internal review report was kept confidential (but provided to NCI).

The Cancer Letter uses the Freedom of Information Act to obtain a redacted version from the NCI.

NCI stops one trial, has no control over the others.

The Cancer Letter reports Potti falsified his CV (claiming he was a Rhodes scholar)

Duke terminates all related trials; Potti resigns (was fired)

Duke is now dealing with the ramifications of its cover-up.

document markup language and document preparation system

Created by Leslie Lamport (built upon the TeX system developed by Donald Knuth)

It is not WYSIWYG

It is free, open, and powerful (just like R).

SweaveSweave allows S (“R”) to be weaved inside a markup language (like LaTeX).

Reproducible ReportsMAdCAm1, VCAM1, NKX2-31

Arthur Berg

November 12, 2011

1Analysis provided for Wei Yu, PhD (correspondence) and Zhenwu Lin, PhD

/Users/ab/Documents/communications/wei/2011-11-09 wei/stat-report-2011-11-09.Rnw

MAdCAm1, VCAM1, NKX2-31

Arthur Berg

November 12, 2011

1Analysis provided for Wei Yu, PhD (correspondence) and Zhenwu Lin, PhD

/Users/ab/Documents/communications/wei/2011-11-09 wei/stat-report-2011-11-09.Rnw

Contents

1 Analysis 2

1.1 MAdCAm1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.1 Two-Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.1.2 Di�erences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.1.3 Quotients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.2 VCAM1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.2.1 Two-Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.2.2 Di�erences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.2.3 Quotients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.3 NKX2-3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

1.3.1 Two-Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

1.3.2 Di�erences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

1.3.3 Quotients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

A The Data 19

Arthur Berg, PhD Page 6 October 5, 2011

3.2 Covariate analysisvar3b

23 Variables 1324 Observationsid

n missing unique Mean .05 .10 .25 .50 .75

1324 0 1323 41700415 21630153 21630309 35630056 50630030 51630209

.90 .95

54630034 54630099

lowest : 15630005 15630009 15630011 18630004 18630007

highest: 54630162 54630163 54630164 54630165 54630166

cc n missing unique1305 19 2

case (710, 54%), control (595, 46%)

gendern missing unique

1305 19 2

female (567, 43%), male (738, 57%)

agen missing unique Mean .05 .10 .25 .50 .75 .90 .95

1305 19 61 65.41 46 51 58 65 74 80 84

lowest : 19 24 27 28 33, highest: 88 89 90 94 98

race n missing unique1305 19 6

asian black multi-racial native american other white

Frequency 2 36 4 2 1 1260

% 0 3 0 0 0 97

smoken missing unique1305 19 3

ex-smoker (557, 43%), never (605, 46%), smoker (143, 11%)

packyearsn missing unique Mean .05 .10 .25 .50 .75 .90 .95

1305 19 249 112.2 0.0 0.0 0.0 1.2 25.0 56.0 76.2

lowest : 0.00 0.05 0.10 0.15 0.20

highest: 8998.20 9498.10 9998.00 15996.80 21995.60

kcaln missing unique Mean .05 .10 .25 .50 .75 .90 .95

1140 184 1140 1951 729.1 882.5 1218.9 1722.0 2297.7 3215.6 3874.0

lowest : 175.2 190.3 201.9 227.8 244.6

highest: 8980.5 8996.7 10954.6 12144.4 14260.4

Proteinn missing unique Mean .05 .10 .25 .50 .75 .90 .95

1140 184 1094 72.5 26.19 31.72 45.53 64.81 90.58 119.21 142.44

lowest : 4.72 6.54 8.44 9.31 10.01

highest: 273.53 275.58 305.39 322.68 496.87

TotalFatn missing unique Mean .05 .10 .25 .50 .75 .90 .95

1140 184 1081 72.81 23.21 28.74 42.97 64.75 90.88 124.79 150.17

lowest : 2.66 6.05 6.87 6.99 7.05

highest: 311.37 321.07 345.60 415.59 637.43

3.17 rs2248733

countAA 558AT 214TT 14NA 93

HWE p-value: 0.2050813

AA AT TTcase 215 89 6

control 343 125 8

fisher test p-value: 0.7127 (n=786)

‰2-based p-value for snp rs2248733 from logistic regressionwith covariates age, sex, and cig: 0.45 (n=762)female: 0.2071 (n=321)male: 0.7472 (n=441)adenocarcinoma: 0.7906 (n=569)sequamous cell carcinoma: 0.007446 (n=520)light smokers: 0.3278 (n=275)heavy smokers: 0.4912 (n=268)

3.18 rs12988520

countCC 233AC 431AA 169NA 46


CC AC AAcase 97 181 62

control 136 250 107


‰2-based p-value for snp rs12988520 from logistic regres-sion with covariates age, sex, and cig: 0.655 (n=808)female: 0.715 (n=345)male: 0.8732 (n=463)adenocarcinoma: 0.903 (n=595)sequamous cell carcinoma: 0.8183 (n=539)light smokers: 0.8936 (n=289)heavy smokers: 0.6435 (n=292)

3.19 rs7608713

countGG 467AG 310AA 55NA 47


GG AG AAcase 201 119 20

control 266 191 35



3.20 rs7578153

countCC 789TC 44TT 4NA 42

HWE p-value: 0.0002462722

CC TC TTcase 331 12 0

control 458 32 4



7

4.2 overall (with covariates)

glmrs4663333 0.0113rs7564935 0.0791

rs17868323 0.0873rs28969678 0.0891rs17868322 0.0963rs11695770 0.0999rs6725478 0.1004

rs17863762 0.1125rs6759892 0.1381rs1500482 0.1436

4.3 female (with covariates)

femalers10929251 0.0115rs7578153 0.0318rs6747843 0.0411rs4663333 0.0492rs887829 0.1060

rs4148325 0.1077rs12474980 0.1143rs1875263 0.1213

rs11891311 0.1228rs10210058 0.1331

4.4 male (with covariates)

malers7578153 0.0101

rs11695770 0.0237rs2741029 0.0453rs7608713 0.0476rs2070959 0.0636

rs17868323 0.0672rs10189426 0.0773rs1500482 0.0892rs2602373 0.0972

rs17863762 0.1004

4.5 adenocarcinoma (with covariates)

adenors17868322 0.0162rs17864678 0.0701rs11888492 0.0711rs2302538 0.1295rs4663333 0.1387rs7586006 0.1404

rs10189426 0.1549rs1817154 0.1560

rs17868337 0.1941rs11695770 0.1998

4.6 sequamous cell carcinoma (with co-variates)

seqrs2248733 0.0074

rs10929251 0.0083rs17863773 0.0276rs10221563 0.0414rs4663333 0.0637

rs12474980 0.0805rs7608713 0.0844rs1604144 0.0953rs7564935 0.1154

rs10189426 0.1371

4.7 light smokers (with covariates)

lightrs7569014 0.0251rs3771342 0.0347

rs17863762 0.0376rs10929251 0.1010rs1604144 0.1322rs4663333 0.1405

rs12474980 0.1436rs17862859 0.1487rs28898590 0.1535rs28969678 0.1540

23

Arthur Berg, PhD Page 25 November 14, 2011

no yes

12

34

5

obs. mean median s.d. min. max.

469 3.187 3.1 0.76 1.1 5.3

For cc = no


388 3.17 3.1 0.778 1.1 5.3

For cc = yes


81 3.265 3.3 0.674 1.7 4.9

t-test p-value: 0.262Mann-Whitney p-value: 0.3

peak (minimum)

Estimate Std. Error z value Pr(>|z|)(Intercept) 2.5878 0.4330 5.98 0.0000

x -0.3754 0.1483 -2.53 0.0113

●●

●

●

no yes

12

34

5

Recurrence (§ Labs)

Arthur Berg, PhD Page 25 November 14, 2011

no yes

12

34

5


469 3.187 3.1 0.76 1.1 5.3

For cc = no


388 3.17 3.1 0.778 1.1 5.3

For cc = yes


81 3.265 3.3 0.674 1.7 4.9

t-test p-value: 0.262Mann-Whitney p-value: 0.3

peak (minimum)

Estimate Std. Error z value Pr(>|z|)(Intercept) 2.5878 0.4330 5.98 0.0000

x -0.3754 0.1483 -2.53 0.0113

●●

●

●

no yes

12

34

5

Recurrence (§ Labs)

With the limited data size, variable selection is rather sensitive, but no variable is particularly significant in this analysis.Note that it is expected that the inclusion of covariates would decrease the variation explained by the genetics, whichthen reduces the significance of the genetic e�ect. With large sample sizes and only a couple of covariates, these changein e�ect is typically very small, but with small sample sizes and many covariates, this e�ect can be significantly larger.

2 Results

The following are the covariate-adjusted results, adjusting for smoking status, location, behavior, and sex.

p-value snp name gene name0.001923 rs6927210 U60.003754 rs13361189 IRGM0.011122 rs4958847 IRGM0.047845 rs2241880 ATG16L10.055233 rs2522057 IBD5

Table 1: Top SNPs – covariate adjusted

If the behavior variable is removed as a covariate, the follow results are yielded:

p-value snp name gene name0.003207 rs13361189 IRGM0.008045 rs4958847 IRGM0.017559 rs12704036 U70.018937 rs6927210 U60.026161 rs3024505 IL.10

Table 2: Top SNPs – covariate adjusted without the behavior variable

3 Session Info

• R version 2.11.1 (2010-05-31), x86_64-apple-darwin9.8.0

• Locale: C

• Base packages: base, datasets, grDevices, graphics, methods, stats, utils

• Other packages: xtable 1.5-6

2

T h e n e w e ngl a nd j o u r na l o f m e dic i n e

n engl j med 366;3 nejm.org january 19, 2012 225

original article

Bone-Density Testing Interval and Transition to Osteoporosis in Older Women

Margaret L. Gourlay, M.D., M.P.H., Jason P. Fine, Sc.D., John S. Preisser, Ph.D., Ryan C. May, Ph.D., Chenxi Li, Ph.D., Li-Yung Lui, M.S., David F. Ransohoff, M.D.,

Jane A. Cauley, Dr.P.H., and Kristine E. Ensrud, M.D., M.P.H., for the Study of Osteoporotic Fractures Research Group

From the Departments of Family Medi-cine (M.L.G.), Biostatistics ( J.P.F., J.S.P., R.C.M.), and Medicine (D.F.R.) and the North Carolina Translational and Clinical Sciences Institute (C.L.), University of North Carolina, Chapel Hill; the Research Institute, California Pacific Medical Cen-ter, San Francisco (L.-Y.L.); the Depart-ment of Epidemiology, University of Pittsburgh, Pittsburgh (J.A.C.); and the Department of Medicine and Division of Epidemiology, University of Minnesota, and the Department of Medicine, Minne-apolis Veterans Affairs Health Care Sys-tem (K.E.E.) — both in Minneapolis. Ad-dress reprint requests to Dr. Gourlay at the University of North Carolina–Chapel Hill, Aycock Bldg., Manning Dr., Chapel Hill, NC 27599-7595, or at [email protected].

N Engl J Med 2012;366:225-33.Copyright © 2012 Massachusetts Medical Society.

A BS TR AC T

BackgroundAlthough bone mineral density (BMD) testing to screen for osteoporosis (BMD T score, !2.50 or lower) is recommended for women 65 years of age or older, there are few data to guide decisions about the interval between BMD tests.

MethodsWe studied 4957 women, 67 years of age or older, with normal BMD (T score at the femoral neck and total hip, !1.00 or higher) or osteopenia (T score, !1.01 to !2.49) and with no history of hip or clinical vertebral fracture or of treatment for osteopo-rosis, followed prospectively for up to 15 years. The BMD testing interval was defined as the estimated time for 10% of women to make the transition to osteoporosis before having a hip or clinical vertebral fracture, with adjustment for estrogen use and clinical risk factors. Transitions from normal BMD and from three subgroups of osteopenia (mild, moderate, and advanced) were analyzed with the use of paramet-ric cumulative incidence models. Incident hip and clinical vertebral fractures and initiation of treatment with bisphosphonates, calcitonin, or raloxifene were treated as competing risks.

ResultsThe estimated BMD testing interval was 16.8 years (95% confidence interval [CI], 11.5 to 24.6) for women with normal BMD, 17.3 years (95% CI, 13.9 to 21.5) for women with mild osteopenia, 4.7 years (95% CI, 4.2 to 5.2) for women with moderate os-teopenia, and 1.1 years (95% CI, 1.0 to 1.3) for women with advanced osteopenia.

ConclusionsOur data indicate that osteoporosis would develop in less than 10% of older, post-menopausal women during rescreening intervals of approximately 15 years for wom-en with normal bone density or mild osteopenia, 5 years for women with moderate osteopenia, and 1 year for women with advanced osteopenia. (Funded by the Na-tional Institutes of Health.)

The New England Journal of Medicine Downloaded from nejm.org at PENN STATE UNIVERSITY on January 19, 2012. For personal use only. No other uses without permission.

Copyright © 2012 Massachusetts Medical Society. All rights reserved.



original article






A BS TR AC T








n engl j med 366;3 nejm.org january 19, 2012230

sis in which we used the secondary definition of osteoporosis, based on the BMD at the femoral neck alone, the covariate-adjusted times for 10% of women to make the transition to osteoporosis were 1.0 years for women with advanced osteo-penia, 4.7 years for those with moderate osteope-nia, and more than 15 years for those with mild osteopenia or normal BMD. Although these es-timates were similar to the estimates in the pri-mary analysis (which was based on the BMD at the total hip or femoral neck) for women with osteopenia, the time estimate in this sensitivity analysis for women with normal BMD was more than twice as long as that in the primary analy-sis, and it was much longer than the maximum follow-up time of 15 years.

A total of 121 women (2.4%) had a hip or clini-cal vertebral fracture before the transition to os-teoporosis, as defined by WHO diagnostic criteria, or before the receipt of treatment for osteoporosis. The adjusted estimated time for 2% of women to have a hip or clinical vertebral fracture was more than 15 years for women with normal BMD or mild osteopenia, and approximately 5 years for those with moderate or advanced osteopenia. Complete results of the sensitivity analyses are presented in the Supplementary Appendix.

Discussion

We conducted a study of rates of transition to osteoporosis in order to help clinicians decide on BMD testing intervals for older women with nor-mal BMD or osteopenia at the initial assessment. Our results suggest that the baseline T score is the most important determinant of a BMD testing interval. During the 15-year study period, less than 1% of women with T scores indicating normal BMD and 5% of women with T scores indicating mild osteopenia at their first assessment made the transition to osteoporosis, with an estimated testing interval of about 15 years for 10% of wom-en in each of these groups to make the transition. This finding suggests that if BMD testing is de-ferred for 15 years among women with T scores greater than !1.50, there is a low likelihood of a transition to osteoporosis during that period. We found that 10% of women with moderate osteo-penia and 10% of women with advanced osteope-nia made the transition to osteoporosis in 5 years and 1 year, respectively. Although clinical risk fac-tors had a minimal effect on the time estimates as a whole, a significant trend for age supported shorter testing intervals as women age. The esti-mated time for only 2% of women to make the

Normal BMDT score, !1.00 or higher(N=1255)

Mild osteopeniaT score, !1.01 to !1.49(N=1386)

Moderate osteopeniaT score, !1.50 to !1.99(N=1478)

Advanced osteopeniaT score, !2.00 to !2.49(N=1351)

Cum

ulat

ive

Inci

denc

e of

Ost

eopo

rosi

s(%

)

100

80

90

70

60

40

30

10

50

20

00 2 4 6 8 10 12 14 16 18

Years since Baseline Study Visit

Figure 2. Unadjusted Cumulative Incidence of Osteoporosis According to Baseline T-Score Range.

The proportion of women who had a transition to osteoporosis is shown as a function of time. The cumulative inci-dence curves were estimated by means of parametric cumulative incidence models for interval-censored data. The dashed horizontal line marks the 10% threshold for the transition to osteoporosis; where this line intersects each cu-mulative incidence curve, a vertical dashed line to the x axis marks the estimated testing interval. The analysis of wom-en with osteopenia at baseline is based on three T-score groups and included the 513 women who made the transition from normal BMD to osteopenia and had at least one subsequent examination with BMD recorded.





original article






A BS TR AC T











Discussion






Cum

ulat

ive

Inci

denc

e of

Ost

eopo

rosi

s(%

)

100

80

90

70

60

40

30

10

50

20

00 2 4 6 8 10 12 14 16 18










Discussion






Cum

ulat

ive

Inci

denc

e of

Ost

eopo

rosi

s(%

)

100

80

90

70

60

40

30

10

50

20

00 2 4 6 8 10 12 14 16 18




