how many scientists fabricate and falsify research? a systematic review and meta-analysis of survey...
TRANSCRIPT
How Many Scientists Fabricate and
Falsify Research?
A Systematic Review and Meta-Analysis
of Survey Data
Daniele Fanelli
““The million dollar question”….The million dollar question”….
Data source Frequency (%) of misconduct
Government-confirmed cases in the US
0.001 – 0.01 (Marshall 2000, Steneck 2006)
Retractions due to misconduct, in PubMed
0.02 - 0.2 (Claxton 2005)
Manipulated images submitted to The Journal of Cell Biology
1 (Steneck 2006)
Investigators disqualified by
FDA data audits (1977-1990)
2 (Glick 1992)
-Only misconduct that has been discovered,-Only misconduct that has been discovered,and (presumably) proven to be intentionaland (presumably) proven to be intentional
Ultimately, only scientists know about Ultimately, only scientists know about their own intentions! their own intentions!
NeverSometimesFrequently
Over the years, many surveys Over the years, many surveys have asked scientists directly…have asked scientists directly…
“Since entering medical school have you…?”
0 1-5 >5
“Have you participated in research involving […] during the last 10 years?”
“Fabricated data”
“Modified research or experimental results to improve the outcome”
“Failing to present data that contradict one's own previous research” “Indicate the number of
[…] members you have observed/experienced exhibiting […] within the last 5 years”
“Seriously misleading interpretation of results”
Yes No
QuestionQuestion Form of misconductForm of misconduct OutcomeOutcome
Results appeared inconclusive and dificult to compare
…different things, in different ways…
““Tricks” in the analysisTricks” in the analysis
• How many committed or observed X at least once• Only questions on fabrication, falsification, alteration and
QRP that distort scientific knowledge. No plagiarism, professional misconduct etc…
• Mixed questions were excluded
• No other measure of study quality (it’s controversial)– Included all eligible studies that specified their methods– Entered methodological factors in inverse variance weighted regression
ES=Log
)1( p
p
W=2
1
SE= )1( pn p
Question by question: effect size and weight
42 literature databases, 14 journals, 8 grey literature db, 2 internet scientific search engines, and references lists
The search…The search…"research misconduct" OR "research integrity" OR "research malpractice" OR "scientific fraud" OR "fabrication, falsification" OR "falsification, fabrication"
Potentially relevant studies obtained from electronic search (n=3276)
Studies retrieved for examination of full text (n=69)
Studies included in review (n=21)
Studies excluded for one of the following reasons (n=48):-Did not have any relevant or original data -Sample not exclusively composed of researchers-Misconduct not related to research (e.g. cheating on school projects)-Does not distinguish fabrication and falsification from other forms of misconduct not relevant to this review-Presents data only in format not usable in this review
Studies excluded because were not surveys on research misconduct (n=3207)
Studies included in meta-analysis (n=18)
Studies excluded from meta-analysis because did not meet quality criteria (n=3)
Characteristics of studiesCharacteristics of studies
• Conducted between 1986-2005
• USA (15), UK (3), multinational (2), and Australia (1)
• Medical/clinical (8), biomedical (6), multidisciplinary (6), economists (1)
• In total 85 questions:
– about fabrication, falsification, alteration, modification (meta-analysis)
– Questionable research practices (systematic review only)
(Full data set available soon in PLoS ONE)
Scientists who admit Scientists who admit fabrication, falsification, fabrication, falsification, or alteration of resultsor alteration of results
b= -0.14±0.05P=0.006
1.97% (N=7, 95%CI: 0.86-4.45) If only asked “fabrication, falsification”
1.06% (N=4, 95%CI: 0.31-3.51)
Scientists who know a Scientists who know a colleague who fabricated, colleague who fabricated, falsified, or altered resultsfalsified, or altered results
14.12% (N=12, 95% CI: 9.91-19.72) If only asked “fabrication, falsification”
12.34% (N=11, 95%CI: 8.43-17.71)
Questionable Research Practices
(e.g. “failing to publish data that contradicts one’s previous research”“dropping data points based on a gut feeling”)
Inverse variance-weighted regression
What influences admission rates?What influences admission rates?
Asking about self vs colleagues: b±SE P
-4.53±0.81 <0.0001
Handed-out surveys vs mailed: 1.17±0.4 0.0032
Using “fabrication” or “falsification” vs “alteration” or “modification”
-
+- -1.02±0.39 0.0086
82% of variance explained (N=15)
USA / other
Researcher / other
Biomedical / other
Social Sc. / other
Medical / other
Controlling for these factors, tested for differences between:
b=0.85±0.28
P= 0.0022
n.s.
Year
Martinson 2005 is Martinson 2005 is outstanding, as outstanding, as conservative!conservative!
Leave-one-out sensitivity analysisLeave-one-out sensitivity analysisScientists who admit Scientists who admit
fabrication, falsification, fabrication, falsification, or alteration of resultsor alteration of results
Scientists who know a Scientists who know a colleague who fabricated, colleague who fabricated, falsified, or altered resultsfalsified, or altered results
““Repairing Repairing misconduct” misconduct”
IDID N casesN cases Action takenAction taken %%
Tangney, 1987
78 Took some action to verify their suspicions of fraud or to remedy the situation
46
Rankin, 1997 31 (incl. Plag.)
In alleged cases of scientific misconduct a disciplinary action was taken by the dean
32.4
Some authority was involved in a disciplinary action 20.5
Ranstam, 2000
49 I interfered to prevent it from happening 28.6
I reported it to a relevant person or organization 22.4
Kattenbraker, 2007
33 Confronted individual 55.5
Reported to supervisor 36.4
Reported to Institutional Review Board 12.1
Discussed with colleagues 36.4
Titus, 2008 115 (incl. Plag.)
The suspected misconduct was reported by the survey respondent
24.4
The suspected misconduct was reported by someone else 33.3
Around half of recalled cases hadno action whatsoever taken against them
Summary of key findingsSummary of key findings
• Data fabrication, falsification and alteration was– admitted on average by around 2% (1% - 4%)
– directly observed by 14% (10% - 20%)
• Questionable Research Practices were– admitted on average by up to 34%
– directly observed by up to 72%
• Overall admission rates (self-/non-self) were higher in– Non-self reports, questions not using “fabrication” or
“falsification”, handed out questionnaires
– Medical/clinical and related research
Unlike surveys for other criminal behaviour,Scientists always lose by admitting misconduct
Unclear- Risk of multiple reporting- “Muhammad Ali” effect- Unaware of all cases- Unwilling to damage their field
Self-reports
Non-self-reports
Conservative
Regression analysis:Regression analysis:-Medical research not robust to all sensitivity-Differences in methodology masked most effects
-Non-significant effects not necessarily non-significant
How Reliable Are These Numbers?How Reliable Are These Numbers?
ConclusionsConclusions
• On average, 2% of scientists admitted misconduct, and 34% QRP– Actual frequencies probably higher– Probably vary depending on field and many
other factors, which meta-analysis currently cannot detect
• Future surveys might benefit by– Focusing on correlates of misconduct– Common methodologies
(tip of the iceberg)
(this too)