Introduction to the Design and Analysis of Trials can be found on:

http://www-users.york.ac.uk/~djt6/

Observational Studies

Background Stronger ‘quasi-experimental’

research methods involve control groups. Most health research use none randomly formed control groups.

Need to consider the advantages and disadvantages of non-randomised controlled studies.

Case control study In a case control study ‘cases’ are

identified (e.g., patients with angina) and these are matched with similar controls (e.g., patients of a similar age without angina).

The medical/lifestyle history of the two groups are then compared.

Case Control-study

Case control Once cases have been matched to

controls differences between the groups are examined. Any differences, (e.g., fruit and vegetable consumption) are then seen as potentially causative of the disease (e.g., Angina).

Case control data Typically in a case control study

we will observe a range of ‘risk factors’ for the disease. Among women with angina we will see: Smoking; Low oestrogen use; Early menopause; Low fruit and vegetable consumption.

Weaknesses Case control studies have a number of

weaknesses. Recall bias – patients in the intervention

group may selectively recall ‘risk factors’ more often than the control group giving a spurious correlation with outcome.

For example, people with Angina may selectively recall lower amounts of vegetable consumption compared with controls.

Weakness Case control studies can be biased

because of mortality effects. If a treatment incurs a mortality at the start of intervention then these patients won’t be available for a case control study.

Selection Bias All non randomised studies are

susceptible to selection bias. Case control studies are particularly susceptible. Selection of both the cases and controls by the researcher may be biased.

Controls may be selected from the same hospital which can lead to bias.

Interpretation of case - control Assocations should NEVER been

assumed to be causal in case-control studies.

Associations are pointers to further research – preferably using random allocation.

Why do case control studies?

Speed and Economy Case control studies can usually be

done quickly and relatively inexpensively;

For rare events (e.g. DVT among women on the pill) case control methods may be the only feasible approach.

Association of biochemical markers and Down’s syndrome was observed using case control methods.

Cohort Studies These avoid some of the biases of

the case control approach. A cohort of people of selected and

followed for a period of time and people who have an event are then compared with people who did not have an event.

Cohort study

Cohort Because we measure various risk

factors BEFORE the disease event then this eliminates some biases, such as recall bias, and therefore, cohort studies are more rigorous than case control methods.

Advantages Relatively inexpensive (but not quick)

compared to an RCT; Can sometimes be quicker than an RCT as

ethics committees ironically usually allow less scientifically rigorous studies through more quickly than the rigorous RCT;

Are an alternative when RCTs are not possible (either ethically or not appropriate).

Historical Controls Some studies use historical

controls to compare treatment effects.

This approach has even more potential for bias than when using contemporaneous controls (ie., usual care patterns might have changed over time).

Time Series analysis When data are available over a

long period of time we can look at changes due to policy changes.

For example, data on percent of women being screened for cervical cancer showed little change for 25 years until government changed the payment mechanism.

Glue ear No evidence that the use of

grommets is an effective treatment for the treatment of glue ear.

A systematic review showing this was widely disseminated.

Did the review have an effect on practice? Mason et al, in a time series analysis suggested it did.

Does NICE work? NICE guidance for hernia

treatments does not support expensive laparoscopic surgery for this condition.

Not cost effective. Did guidance reduce laparoscopic

surgery? Bloor et al in a time series analysis

suggested it did not.

Problems with Time Series The assumption is that any change in

the trend is due to the intervention. There may be confounding due to other simultaneous policy initiatives.

Often when a problem is identified several initiatives may be introduced at around the same time and it is difficult to untangle what is the main effect.

Perils of Confusing Association with Causation An association was noted that

smokers who had high consumption of vegetables were less likely to develop lung cancer than smokers who did not. They also had higher blood levels of beta-carotene a natural anti-oxidant present in many vegetables.

D e d uc tionIn c rea se B e ta -caro ten e leve ls

R e du ce lu n g can cer

B lo o d te s ts co n firm lowle ve ls o f b e ta -ca ro te ne

a m o ng sm o kers w h o ge t can cer

O b se rva tionH ig h ve ge tab le co n su m p tion

a sso c ia te d w ith lo w e r risko f lun g ca ncer

Association gives rise to hypothesis

High vegetable consumption leads to high consumption of beta-carotene, which as an anti-oxidant neutralises ‘dangerous’ free radicals produced by the cells, which are a cause of cancer. Supplement with beta-carotene in pills among smoking men will reduce free radicals and reduce lung cancer risk.

Test hypothesis in RCT Several RCTs launched to test this

hypothesis. Male smokers randomised to receive beta-carotene supplements, several trials – Finland and USA.

What happened? Men taking beta-carotene INCREASED their risk of lung cancer.

Association does NOT mean causation.

Vitamin E Another anti-oxidant, so beloved of

‘alternative’ practitioners, is the use of high dose vitamin E.

Cannot be harmful – BUT if it doesn’t have the potential for harm cannot do GOOD.

A meta-analysis of all the RCTs of vitamin E shows an INCREASE in mortality.Miller et al, Ann Intern Med 2005; 142:37-46.

Miller, E. R. et. al. Ann Intern Med 2005;142:37-46

Risk difference in all-cause mortality for randomized, controlled trials of vitamin E supplementation and pooled results for low-dosage (<400 IU/d) and

high-dosage ([&ge;]400 IU/d) vitamin E trials

High dose vitamin E Most people who take vitamin E

supplements (>60%) take daily doses above 400 IU a day, which is the dose associated with increased mortality.

Education Hypothesis – use of ICT improves

literacy and numeracy? Observational data suggest ICT is

ineffective for language teaching (in UK & Israel) and HARMFUL for numeracy acquistion.

Need RCT to test negative association?

Selection Bias ALL observational data are

susceptible to selection bias. Selection bias is when people are

‘selected’ on characteristics that affect outcome.

This can often give a ‘spurious’ association.

Examples of selection bias Women taking HRT are healthier than non-

users; Schools using ICT have more resources

different teachers and pupils compared with schools not using ICT.

Male smokers with ‘naturally’ high beta-carotene intake are ‘different’ in some other way that makes them more resistant to carcinogenic effects of smoking.

Direction of Association Use of dummies (pacifiers) are

associated with early cessation of breast feeding.

25% of children who use a dummy daily are weaned by 3 months compared with 13% who do not use a dummy (RR = 1.9, 1.1 to 3.3).

Conclusion: DON’T use Dummies?

Trial of Pacifier(Dummy Use) WHO recommends that women should

avoid using dummies as ‘observational’ data suggests a STRONG association with early weaning or no breast feeding with their use.

Kramer et al tested this hypothesis in a trial. In an intervention they reduced dummy use from 84% to 61%.

Kramer et al, JAMA 2001;286:322.

Did this affect breast feeding? NO – at 3 months 19% of those

women with LOW dummy use had weaned their babies compared with 18% with HIGH dummy use.

HOWEVER, when the data were analysed as an observational study women using dummies were more likely to wean their babies. WHY?

Direction of Association Women who for what ever reason

did not or could not continue with breast feeding were using dummies to ‘pacify’ their infants. The decision to give up breast feeding CAUSED the association NOT vice-versa.

Can observational data ever infer causation?

Many epidemiologists believe it can, 6 requirements to assess causation: Consistency of association Proper time sequence Dose response Strength of association Change in risk with change in exposure Biological plausibility

Example, lung cancer and smoking

Consistent association development of lung cancer and smoking in both

sexes across numerous countries; Time sequence

Smoking occurs before lung cancer (development of cancer does not preceed smoking);

Dose-response Heavy smokers have higher risk than light

smokers.

Smoking (cont) Strength of association

Very high relative risk of lung cancer associated with smoking (e.g., 19/20 cancers in smokers).

Change in risk with exposure Former smokers have lower risk than

current smokers; Biological Plausibility.

Animal and tissue data show that exposure to contents of tabacco smoke increase cell proliferation and cancer.

BUT The power of selection bias means

that a treatment can pass all these criteria and the relationship is still not causal.

Whither Observational Data? Observational data is good for

hypothesis forming, cheap easily collected before an expensive trial.

Good for looking at ‘risk factors’. Helpful in areas where trials are

not possible (e.g., policy changes).

Disadvantages Prone to selection and other biases. May often give the ‘wrong’ answer. Cannot be used to ‘prove’ an causal

effect. If at all possible findings from

observational data need confirming using trial data.

Observational Studies Any controlled study IS better than

an uncontrolled evaluation. BUT non-randomised studies are

susceptible to a wide range of bias and should only be used to convict a suspect if no randomised data are available.

Comparing cohort and case-control

Cohort Complete source

population Estimate incidence

rates and relative risks Takes long time Expensive Useful for broad range

of diseases and exposures

Case control Sampling from source

population Estimate only odds

ratios Shorter and cheaper Focused around one

disease

Threats to internal validity

• Selection bias- Self selection- Investigator selection

• Information bias- Biased effect size because some subjects are more

likely to attain a positive or negative outcome score than others

• Confounding- Effect size explained by other factors

Selection bias

When the association between the disease/event and exposure/intervention differs for those who participate in the study and those who don’t

Effect of breast cancer screening by comparing breast cancer rates in volunteers and those not tested

Selection bias through self-selection (patient/clinician) Those who get the intervention are more likely to benefit from

it (observed and unobserved characteristics) Selection bias due to choices made by the investigator

Comparison of workers health with those of general population (healthy worker effect)

Volunteer Bias Breast Cancer Incidence Rates in a Screening Trial (per 1000/yr)

2 .1 0

2 0 ,0 0 0 a tten d

1 .4 5

1 1 ,0 0 0 re fu se

3 1 ,0 0 0 in vitedfo r sc reen in g

1 .8 7

3 1 ,0 0 0 u su a l ca re

3 1 ,0 0 0 n o t in vitedfo r sc reen in g

6 2 ,0 0 0 w om enran d om ized

Investigator selection

Case-control studies Selection of cases

Exposed patients are more likely to be be included in the study than non-exposed patients

Selection of controls Exposed controls are less likely to be included

in the study than non-exposed controls

Information bias

Information collected about exposure or outcome are wrong

Misclassification Non-differential

All epidemiological studies to some extent Always biases estimate of association towards 1

Differential Recall bias (e.g. maternal recall bias) Biased follow-up (e.g. unexposed people under

diagnosed compared to exposed) Biases estimate of association in different directions

Information bias

Measurement errors in variables• cohort study

- more adequate measurement of results in exposed in comparison to non-exposed

• case-control- more adequate detection of exposure in

cases in comparison to controls

Confounding

Exposure/intervention

Outcome

Association

Confounder

Association (causal relation)

Do I do ‘observational’ studies Yes, quite often. They are useful for:

Getting peer reviewed papers (which boosts your CV and keeps you in a job);

Generating hypotheses; Identifying ‘risk factors’; BUT cannot be used to establish

causation.

Example – Fracture Cohort Study

We wanted to find out what were the risk factors for perimenopausal women having a fracture.

We invited 1,857 women to have a bone mass measure and answer questionnaires about their lifestyle, sociodemographics and medical history.

Followed them for two years by questionnaire.

.

Fracture Risk Factors - Results We found that:

Low bone mass; Early menopause; Prior fracture Family history

All predicted fracture risk.

Two types of risk factor Changeable risk factors

Low bone mass; Low oestrogen (early menopause).

Irreversible risk factors Family history Prior fracture.

Where now? All risk factors could be used to target

EFFECTIVE treatment. Reversible risk factors could be used to

inform a trial of agents (e.g, drugs to increase bone mass).

Irreversible factors can be used to target EFFECTIVE treatment and perhaps inform basic science (e.g., genetics).

Best Evidence Synthesis Large rigorous RCT (preferably

several with meta-analysis). Meta-analysis of several small RCTs. Cohort studies. Case control. Before and after. Professional opinion.