Thomas Songer, PhD

Introduction to Research MethodsIn the Internet Era

Critically Reviewing anEpidemiologic Study

Learning Objectives:

1. Describe the approach to reviewing a manuscript2. Identify the research hypothesis of a manuscript3. Identify the quality of the research and the

validity of the findings of a manuscript4. Describe the factors which may raise concern

about the truth of a research finding

2

Reviewing a Paper is about Asking Questions

3

Is the study adequately powered?

What level of measurement error exists?

What is the study hypothesis?

What population do the study subjects originate from?

Are there unstated confounding factors?

Were appropriate statistical procedures used?

What is my overall impression?

4

• Critique of data collection

• Critique of data analysis

• Critique of data interpretation

Key Areas of Focus

5

Example Manuscript

• Fitzpatrick AL, Kuller LH, Lopez OL, Diehr P, O’Meara ES, Longstreth Jr. WT, Luchsinger JA. Midlife and Late-Life Obesity and the Risk of Dementia. Archives of Neurology 66(3):336-342, 2009.

6

• Study Context

• Study Objectives– Is a hypothesis stated?

• Exposure and Outcome Variables

• Study Design

• Study Population

• Potential for– Selection Bias– Information Bias– Confounding

What to Examine When Evaluating Data Collection

7

• Several Issues to consider–What is the public health significance of

this study?–Does this study generate new hypotheses

or confirm previous results with improved methods?

–Is the study hypothesis biologically plausible?

Study Context

8

- What is the public health significance of this study?

• Rising obesity levels in US• Dementia increasing in US• Aging population

–Does this study generate new hypotheses or confirm previous results with improved methods?• Seeks to clarify conflicting results in the literature

by examining a large study sample longitudinally

–Is the study hypothesis biologically plausible?

Study Context – Fitzpatrick, et. al.

9

• What do the investigators want to achieve in this research?–What is the hypothesis of this study?

• There may be more then one• Is the hypothesis specific or too general to

refute?

Study Objectives

10

• Primary exposure– How was variable defined?

• How was information on exposure collected?– Best method?

– Sensitivity/specificity of this method?

– Potential for misclassification?

• Primary outcome– Conceptual vs. operational

outcome? • e.g. breast cancer vs. malignant

neoplasm of the breast tissue

• How was information on outcome collected?– Best method?– Sensitivity/specificity of this

method?– Potential for misclassification?

Exposure and Outcome Variables

11

• What study design was employed?

• Is it an appropriate design?– Exposure or outcome rare?– New hypothesis?

• What are the limitations and strengths of this design?

Type of Study

12

• What study design was employed?– Cohort study (retrospective)

• Is it an appropriate design?– Exposure or outcome rare? Neither– New hypothesis? No, but conflicting study results

• What are the limitations and strengths of this design?– Strengths: longitudinal assessment, incidence of

dementia, uses previously collected data– Limitations: short period of assessment

Type of Study -- Fitzpatrick

13

• What was the source of study population?– How does the study population compare to the general

population?

• How were subjects selected?– Could this method introduce selection bias?

• What was the sample size?– Is the statistical power of the study identified?– Out of the projected study sample, how many persons

participated?

The Study Population

14

• What was the source of study population?– Community dwellers who were Medicare eligible

over age 65 years - 4 sites

• How were subjects selected? Not stated– Could this method introduce selection bias?

• What was the sample size? 2798 out of original cohort of 5888 adults– Is the statistical power of the study identified? Yes– Out of the projected study sample, how many persons

participated? Unknown from original study.

The Study Population -- Fitzpatrick

15

INFORMATION BIAS

• Could there have been bias in the selection of subjects?

• What type of bias would this be? – e.g. healthy worker bias

• In which direction would this bias affect the measure of association?

• Could there have been bias in the collection of information?

• What type of bias would this be? – e.g. interviewer bias

• In which direction would this bias affect the measure of association?

Potential for Bias

SELECTION BIAS

16

• What factors were potentially confounding the study relationship?

• What methods did the authors use to minimize the influence of confounding when planning the study?–E.g. restriction, matching, randomization, etc.

• Is there still residual confounding?

Potential for Confounding

17

• What factors were potentially confounding the study relationship?–Table 1, others

• What methods did the authors use to minimize the influence of confounding when planning the study?–E.g. restriction, matching, randomization,

etc.

• Is there still residual confounding? likely

Potential for Confounding - Fitzpatrick

18

• Confounding

• Measures of Association

• Measures of Statistical Stability

What to Examine When Evaluating Data Analysis

19

• What methods were used to control confounding?

-Standardization: Indirect and Direct. Usually used to control for differences in age distribution among populations.

-Stratification: Allows you to examine data more closely. However, it is difficult to control for more than 1 confounder.

-Matching: Done in Case-Control Studies.

-Multivariate Analysis: Linear Regression, Logistic Regression, Poisson Regression, Cox Proportional Hazards model. Allows you to control for multiple confounders simultaneously.

Data Analysis -- Confounding

20

• What Measures of Association were reported in the study? Was the correct measure used?

• Cohort Study: Relative Risk (RR), Odds Ratio (OR), Hazard Ratio (HR), Incidence Rate Ratio (IRR.

• Case-Control: Exposure or Disease OR (if nested). Can not use RR. However, the OR is a good estimate of the RR when the prevalence of the disease in the study population is very low.

• Cross-sectional Study: Prevalence Ratio.

• Ecologic Study: Correlation coefficient.

Data Analysis – Measures of Association

21

• How was the potential for random error accounted for in the study?

• Hypothesis Testing: Can use p-values or confidence Intervals (CI) to test the null hypothesis.

• P-value: The probability of observing the study results given that the null hypothesis is true. P<0.05 is a standard value that investigators use to reject the null hypothesis of no association and declare that there is a significant relationship between 2 variables.

Data Analysis – Statistical Stability

22

• 95% CI: This measure can be used for hypothesis testing and interval estimation. Can be defined as, if one will repeat the study 100 times the true association will lie inside the interval 95% of the time.

• We fail to reject the null hypothesis when a confidence interval contains the null value of 1 between its lower and upper limits for relative measures.

Data Analysis – Statistical Stability

23

• Large confidence intervals indicate that the standard error is high. A high standard error is often related to a small sample size. Underpowered studies normally have wider confidence intervals and thus difficulty in rejecting the null hypothesis.

• The problem, therein, lies that it is difficult to know if the non-association is real or false.

Data Analysis – Statistical Stability

24

• Major findings of the research

• Influence (on the results) of:– Bias and confounding

– misclassification

What to Examine When Interpreting the Results of the Study

25

• The first paragraph of the discussion section in a manuscript should summarize the main findings of the study.

• Example: Sedentary individuals in this study have 3 (95% CI:1.5-4.9) times the risk of developing a Myocardial (MI) compared to active individuals after controlling for potential confounders.

Major Findings

26

• Reader should be able to recognize information bias, selection bias, or confounding in the study and assess their magnitude and direction in the study.

• Bias or confounding that is large in magnitude signals that the findings in this sample may not approximate what you would expect to see in the population.

Influence of Bias and Confounding

27

• Misclassification of the exposure or the outcome (or both) can influence study results

Non-Differential: Misclassification is similar in the exposure or outcome groups. This would bias the results to the null making it unlikely for investigators to reject the null hypothesis.

Misclassification

28

• Differential: Misclassification occurs at a different rate in exposure or outcome groups.

• Example of differential misclassification, a larger number of individuals are classified as high stress instead of medium stress than individuals classified as medium stress instead of high stress. This type of misclassification can bias results away or towards the null hypothesis.

29

• What are the strengths and limitations of the report?

• How do these balance?

• Can the results be generalized to the whole population?

Formulating an Overall Impression of the Manuscript

30

• Examine the overall issues related to data collection, data analysis, and data interpretation.

• What conclusions do you draw from the results based upon your interpretation of the strengths and limitations of the study?

• Do the strengths outweigh the limitations?

• They are often mentioned in the discussion section of a manuscript.

Strengths and Limitations

31

• Goal is to have a study where the results can be used to infer what is going on in the population

• Major problems with the internal validity of the study make it difficult to for the results to be generalized to any population.

• Example, the study population excluded a certain groups, minorities, women, blacks, or low income individuals. The results would not be generalizable to these groups.

Generalizability

32

• The conclusions are a brief summary of the findings.

• Authors tend to include recommendations for future studies or policy.

• It is essential that the recommendations do not stray far from the study findings. Recommendations should be made in the context of the findings or the readers may be deceived and make incorrect conclusions about the actual results of the study.

Conclusions and Justification

33

The Big Picture of Research Findings

• Publication bias

• John P.A. Ioannidis– Why Most Published Research Findings

are False. PLoS Medicine 2(8):e124, 2005.

34

Publication bias

• Definition:

“Publication bias refers to the greater likelihood that studies with positive results will be published”

JAMA 2002;287:2825-2828

Abbasi 35

Publication bias may ….

• Distort the scientific record• Hide the “truth” of association/no

association• Influence doctors’ decision making• Mislead policy makers• Etc.

36

Ioannidis Corollaries

• The smaller the studies conducted, the less likely the research findings are to be true

• The smaller the effect size, the less likely the research findings are to be true

• The greater the financial interest and prejudice, the less likely the research findings are to be true

• The hotter a topic interest, the less likely the research findings are to be true

37