overview of critiquing a paper

Child Obesity Research Group International (CORGI)

[email protected]

Objectives • To improve understanding of key research

methodology concepts• To increase knowledge of key research in the topic

area• To generate and develop new research ideas• To carry out collaborative research • To serve as a support group• To provide a legitimate excuse to go to the pub

each week

Methods

• Journal club to appraise published research covering – Range of research methodologies– Range of topics

• After-appraisal discussion to review existing knowledge and generate new ideas

• Internal peer review of manuscripts, grants, talks in preparation

• Offer expert support and collaborate on research

Journal club • In the meeting preceding, members put up papers for

possible papers for future discussion • In the week preceding, members appraise papers using

structured template• Member proposing the paper leads discussion, presenting

his/her own findings with others contributing• At the end of the meeting, lead member circulates the

agreed appraisal • Following 40 minutes of appraisal, 20 minutes of

discussion on existing knowledge and new research ideas initiated by the lead member through presentation of a brief summary of ‘what is already known’ on the topic

How to systematically approach published studies?

Brief overview

Critical appraisal format: why, how, what and so what?

• Introduction: Why was the study carried out?• Methodology: How was the study carried out?• Results: What are the results of the study?• Discussion: So what are the implications of the

results?

• Abstract: Does the abstract give a fair summary of the main findings?

Introduction

Why was the study carried out?• Is there a clear and justified research question?• Is the chosen study design most appropriate?

Hierarchy of evidenceFeasibility (opportunity, ethics, costs, and time)

Methodology

How was the study carried out?• Is the study population representative?

e.g. response rate (selection bias) • Are the comparison groups comparable?

e.g. other risk factors (confounders)• Are measurements appropriate?

e.g. recall bias (information bias)• Was the follow-up sufficiently long and complete?

e.g. differential loss to follow-up

Bias

• Bias– “Error which produces results or conclusions

that differ systematically from the truth”– Design or analysis

• Commonest biases:– Selection bias (sample selection errors)– Information bias (measurement errors)

Confounding

The exposure (E) is associated with a third factor, the confounder (C), which is also linked to the disease (D)

The association of the disease with the exposure may simply reflect the association with the confounder

CF

E D??

Confounding

The association of the alcohol with lung cancer simply reflects the fact that those who drink higher amounts

of alcohol also tend to smoke

smoking

Alcohol Lung cancer??

Results

What are the results of the study?• How strong is the association?

i.e. magnitude and precision (relative risk & 95%CI)• How likely that observed association is by chance?

i.e. p-values• Is there dose response?

e.g. smoking and lung cancer• Have confounders been adjusted for?

e.g. social class in studies

Discussion

So what are the implications of the results?• Are results believable?

i.e. chance, bias, confounding, reverse causality• Is there a plausible explanation?

e.g. disliking eastenders and getting lung cancer• Are results consistent?

i.e. results from other studies• Should things be changed?

i.e. population impact

Others

Abstract• Does it give a fair summary of findings?

Ethics• Were appropriate approvals taken?

Conflicts of interest• Could the authors be biased?

Cross-sectional StudiesIntroduction:Is there a clear and justified research question?Is the cross-sectional study design the best approach to investigating this particular research question?

Methodology:Was the sample studied representative of the target population?Were the exposure and outcome accurately measured to minimise bias?

Results:What was the response rate? Is there any reason to suspect that the non-responders differ significantly from the responders?How strong is the association between the exposure and outcome?How precise is the estimate of prevalence or risk?Is there evidence of a dose-response association?Have appropriate confounding factors been controlled for?

Discussion:Could the results be explained by (a) chance; (b) bias; (c) confounding; (d) reverse causality? Is there a plausible explanation for the results?Are the results consistent with other available evidence?Should policy or practice change as a result of this study?

Abstract:Does the abstract give a fair summary of the main findings?

Case Control Studies

Introduction:Is there a clear and justified research question?Is the case control study design the best approach to investigating this particular research question?

Methodology:Were the cases precisely defined?Are the controls representative of the population that gave rise to the cases?Was exposure measured in the same way in cases and controls?If a matched design has been used are the statistical methods used to analyse the data appropriate i.e. conditional logistic regression/McNemars?

Results:How strong is the association between the exposure and outcome?How precise is the estimate of risk?Have appropriate confounding factors been controlled for?Is there evidence of a dose-response association?

Discussion:Could the results be explained by (a) chance; (b) bias; (c) confounding; (d) reverse causality? NB Recall and control selection biases are common problems with case control studies.Is there a plausible explanation for the results?Are the results consistent with other available evidence?Should policy or practice change as a result of this study?


Cohort StudiesIntroduction:Is there a clear and justified research question?Is the cohort study design the best approach to investigating this particular research question?

Methodology:Is the cohort representative of the population to which the results will be generalised?Was the follow-up sufficiently long and complete?Is there any reason to suspect that those lost to follow-up differ from those in whom follow-up is complete? (e.g. are they more likely to have died?)Was exposure measured in the same way in all people?Is exposure likely to have changed over follow-up?Is the likelihood of disease detection the same in exposed and unexposed individuals?

Results:How strong is the association between the exposure and outcome?Is there evidence of a dose-response association?How precise is the estimate of risk?Have appropriate confounding factors been controlled for?

Discussion:Could the results be explained by (a) chance; (b) bias; (c) confounding; (d) reverse causality? Is there a plausible explanation for the results?Are the results consistent with other available evidence?Should policy or practice change as a result of this study?


Randomised Controlled TrialsIntroduction:Is there a clear and justified research question?Is the randomised controlled trial the best approach to investigating this particular research question?

Methodology:How representative of those who would receive treatment are people who took part in the trial?Did the researchers perform a sample size calculation and specify a primary outcome measure?Was the assignment of patients to treatments randomised?Could the method of randomisation have resulted in bias e.g. could the investigator have predicted what treatment the next subject would receive or interfere with treatment allocation?Aside from the experimental intervention, were the group treated equally?Were the patients, health workers and study personnel (particularly those assessing outcome) “blind” to the treatment?Was the follow-up sufficiently long and complete?Were all patients who entered the trial properly accounted for and attributed at its conclusion?Did the presentation of methods and results follow the CONSORT guidelines?

Results:Were the groups similar at the start of the trial?How large was the treatment effect on the primary outcome measure?How precise is the estimate of treatment effect?Were the patients analysed in the groups to which they randomised (intention to treat)?

Discussion:Are the main conclusions based on primarily defined endpoints or based on sub-group analyses? Could the results be explained by (a) chance; (b) bias?Are the results consistent with other available evidence?Should policy or practice change as a result of this study?


overview of critiquing a paper

Documents