how to kill your study an introduction to planning studies · “making your mark on the world is...

How to kill your studyAn introduction to planning studiesSEMINAR SERIES: HOW TO RUIN YOUR CAREFULLY PLANNED STUDY? TIPS FOR IMPROVING DATA ANALYSIS – SESSION 1

BART JACOBS

2

Seminar Series: How to ruin your carefully planned study? Tips for improving data analysis

3

Seminar Series: How to ruin your carefully planned study?Tips for improving data analysis

Organised by the statisticians of ITM …

Jozefien Buyze (DCS, CTU)

Bart Karl Jacobs (DCS)

Meryam Krit (DBS)

Tom Smekens (DPH)

Achilleas Tsoumanis (DCS, CTU)

… but not about statistics

Focus on what you can do, what questions you can ask yourself.

Introduction of concepts linked to high quality data-analysis

Limited previous knowledge required

4

Question: What’s the first memory that comes to mind when youthink about your statistics classes at university/college?

Boxplots, histograms and scatterplots

The normal distribution / the Bell curve

Something about t-tests and confidence intervals

p < 0.05

A difficult class that many failed

Advice on how to collect and analyse real-world data.

6

Let’s talk about real-world data

It’s messy!Often difficult to collect, potentially costly

Full of errors and inconsistencies

Time-consuming to clean up

And it often doesn’t answer the questions we asked ourselves.… but there were some interesting accidental findings we can talk about!

Is this helpful?

Can we do better?Let’s answer that with an Obama quote.

7

Let’s talk about real-world data

Step 1: Admit to ourselves that working with real data is hard.

Step 2: Don’t be pessimistic, just be aware of challenges and tackle them one at the time.

Step 3: Success!

8

“Making your mark on the world is hard. If it were easy, everybody would do it. But it's not. It takes patience, it takes

commitment, and it comes with plenty of failure along the way. The real test is not whether you avoid this failure, because you

won't. it's whether you let it harden or shame you into inaction, or whether you learn from it; whether you choose to persevere.”

– Barack Obama

Discussion – what do you expect from this seminar series?

12

Your questions help drive the discussion!

An introduction to planning studies

13

Analysing data starts at the planning stage

Most critical moment for data-analysis!

Every choice has direct impact on the quality of future data.

Rest of this session will give an (incomplete) overview of some important choices and considerations at the planning stage.

14

Type of study – Observational

The observational study

Easier – no active involvement of researcher needed

Can be done prospectively (future data will be collected) or retrospectively (already existed data will be obtained).

Increasingly convenient in the upcoming “age of Big Data”

Choices are limited, especially with retrospective collection.

15

Type of study – Experimental

The observational study

The experimental study

Requires active involvement of researcher

In medicine often in the form of a clinical trial

Extensive planning typically required.

Choices will have major impact.

16

Type of study – comparison

Observational

No active role for researcher

Prospectively or retrospectively

Convenient timelines

Limited choices

Experimental

Active involvement required

Typically prospective

Extensive planning required

Extensive choices

17

Observational or experimental?

The choice will be made for you in most cases.

Type of study in practice

Experimental is the “gold standard” for estimating the effect of anintervention.

Session 8 will introduce clinical trials.

Why not always perform experimental studies?

18

Type of study in practice

Why not always perform experimental studies?

May not be practical

Provide costly treatment

Long follow-up of patients

May not be ethical

Force people to do something they don’t want

Give placebo when proven treatment exists

May be impossible

Can’t change gender, date of infection, country of origin, …

Concept of intervention irrelevant (e.g. anthropology, epidemiology)

19

Key question – Why will I do this study?

Descriptive vs. inference

Estimation vs. prediction

Exploratory (pilot?) vs. confirmatory

Superiority / non-inferiority / equivalence

The type of data-analysis depends on goal of the study

21

Researchquestion

Data

How to kill your study from the start

Sloppy review of relevant literature

Extensive and critical review can uncover many issues!

Pay attention to all limitations of previous studies

How robust are results? (session 14)

Is there potential literature bias? (session 15)

Adapt expectations accordingly – be realistic, not optimistic.

22



Don’t think about collecting data, just do it.

Plan in advance! Ask both “Why?” and “Why not?”

Make sure you will have everything you need

But not too much either: collecting (and analysing) data costs money

Is information about confounders available? (session 7)

Can precision be (correctly) estimated? (session 5)

23




Let everyone just send data in a format of their choosing to your email

Consistent reporting?

What if there are different sources?

Data management?

How will variables be coded, what types are there? (seminar 2)?

How much precision is possible, how much is needed? (session 4)

24




Let everyone just send data in a format of their choosing to your email

Only consider quality and availability of the data after collection

When done before starting the study, you may avoid several issues:

Were there enough observations? (session 6)

Do we have missing data and can we find it, or know why? (seminar 12)

How likely are errors and can we still detect them? (seminar 11)

Were our timelines realistic?

Should we have done an interim analysis instead of wasting all this money?

25

Final tip – You don’t have to be something you are not

“To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say

what the experiment died of.” (Fisher)

If someone enters my office and coughs repeatedly, I assume they are trying to get my attention.If they are looking for medical advice, I will forward them to a doctor.

Your job description doesn’t state “statistician” or “data-analyst”.

27

Final tip – You don’t have to be something you are not

Your job description doesn’t state “statistician” or “data-analyst”.

Self-reflection

Do I really understand what I am doing?

Why did I make certain choices: be aware of your own biases – seminar 3

Discuss your ideas with peers.

Do not hesitate to contact a statistician to assist you with the data-analytical challenges, even at the planning stage!

28

Overview of topics

Types of data (October 3)

Cognitive biases (October 17)

Sources of variation (November 7, November 21)

Sample size (December 5)

Association and causation (December 19)

Clinical trials (January 16)

P-values (January 30)

Bayesian thinking (February 13)

Outliers (March 5)

Incomplete data (March 19)

Graphics (April 16)

Publication bias (April 30, May 14)

29

Acknowledgements

Jozefien, Achilleas, Tom & Meryam

Jan Kennis, Carlos Kiyan and all colleagues providing technical support

Maria Zolfo

Daniëlle Van Melle, Ann Verlinden, Nathalie Brouwers, Saskia Decuypere

Lut Lynen

All of you

30

Questions / Comments?

Next seminar is on October 3

When 1 is not larger than 0 – Types of data and how to use them

Presenter: Jozefien Buyze

Data come in a lot of shapes and sizes and often from a variety of sources ranging from vague notes in a patient file over precise lab values and imprecise estimates of dates of events. In this session, we will look into the advantages and disadvantages of different types of data and how they can be correctly analysed. Additionally, we offer advice on which choices (not) to make and when to dichotomize.

31

how to kill your study an introduction to planning studies · “making your mark on the world is...

Documents