Designing data collection

The two types of study!

• Objectives and hypotheses always (really?) reduce to…– …there is a difference between…– …there is an association between…– …change X leads to response Y…

Implications for methods

Primary response

What changes it?

What needs to be held constant?

What may be confounding it?

Experiment or survey?

Individual Population Research question Measurement tool

farmer Farmers with < 3ha in Embu district, Kenya

Does the proportion of income derived from fruit depend on distance to a tarred road?

Questionnaire, map

tree Baobabs in Kibwesi Do the trees fall into groups with different flowering dates?

Visual observation

wood trader Wood traders selling timber in Kampala

What proportion of timber currently comes from farms?

Questionnaire

tree Grevillea trees in upper Embu

How does pruning affect growth rate?

Tape measure

mother Women around Zomba who have planted sclerocarya

How do the farm grown fruit compare with those in the wild?

Bao board and stones

Question What changes?

Unit observed Measurement tool

Can time to germination for species x be reduced by using a higher temperature?

temperature Dish of 100 seeds Visual count

Can reducing the density of potting mix increase root growth rates?

Potting mix density

seedling Ruler, balance

How much variation in growth is there between 6 provenances of x?

Provenance Line plot of 8 trees Tape measure

What advantages are there of planting timber trees mixed with crops rather than along boundaries?

Planting position

10 trees on a farm Questionnaire

Surveys Experiments

Define units Define units

Select sample Select sample

Impose changes (treatments)

Control variation

Take measurement Take measurement

Conclude:

1. Status of population

2. Difference between groups

3. Relationship between variables

Conclude:

Treatments cause observed changes

“Cigarette smoking is harmful to your health”

• How do we know?

Experiment:– Take 2000 young people who have not

smoked– Randomly allocate to smoking and non-

smoking groups– Monitor health outcomes for 40 years

• Survey (prospective)– Sample of smokers and non-smokers– Monitor health– Compare two groups

• Survey (retrospective)– Sample of sick and health people– Elicit smoking history– Compare two groups

• Result clear

Health

Smoking

F

?

•BUT maybe there are other factors (F)?

- lifestyle - diet - genetics - …

• Reaction: control for F– eg F=overweight. – Do the study for overweight and normal

people

• Hypothesis: it is F2– Do a study controlling for F and F2

• ….

• Look for further supporting evidence– Surveys in very different contexts that gave the same

results– Experiments on animals and in vitro

• considered ethical

– Causal explanations• biochemical

• Eventually evidence is considered overwhelming (even by tobacco companies?)

Your problem

• You have to probably 1 chance to do the study

• Therefore:– Think of those alternative explanations and

control for them– Collect the supporting evidence– Get familiar with the causal/theory

explanations

Common design principles

Principle What? Example problems

Objectives Drive all of design

Too vague

Not internally consistent

Units The objects being studied

Mixed within one study

Wrong level used

Comparison Differences or relationships

No variation in key variable

Principle What? Example problems

Confounding In ability to separate effects of two variables

Failure stratify

Management varying between treatments

Repeatability and uncertainty

Consistent results shown by repeating

No repetition (case study)

Repetition at the wrong level

Measurement Variables to measure

Many measurements not related to objective

Key measurements missing

Any more?

Principle What? Example problems

Protocol Written details of all procedures

Not done

Not reviewed and critiqued

Not kept uptodate

Design hierarchy of unitsHow many?

Which ones?

Which treatments?

Who decides?

Site

Community/ farmer group

Farm

Field

Plot

Tree

Sample

Comparison…

• …of treatments• …of niches• …of farmers• …of farm types• …of communities• …of farmer groups• …of group typesThe more which are planned the better!

Uncertainty

• Aim to make some generalisable conclusions Variation and uncertainty make this difficult.• Answer: repeat observations to spot general

patterns = Replication– decreases the uncertainty

• 40 out of 50 farmers or 4 out of 5 farmers?– quantify the uncertainty

• 40 out of 50 means 67% to 93%

• Replicate at the appropriate 'layer‘• And remember the domain to which conclusions

must apply

Bias

• Bias in allocation – randomise– measure

• Bias in selection– Randomise measure

• Bias in measurement– use improved tool– triangulate

= consistent under or over estimate of real effect

Awareness

Training

Confounding

Solution:• Check objectives - it may not be a problem• Control the NEV• Include extra plots of other combinations

= confusingSystemcontinuous IF

variety

Local

improved

Eg: compare continuous cropping with improved fallow

The protocol

written plan of the activity

A protocol should be...

• prepared for every research activity, however small

• written, not in your head• shared with others who can help improve it

– experience from similar problems, methods, species, ecozones,...

• detailed enough for someone else to take over the study

• kept up to date– plans change during the execution– a record of what actually done

• archived with the data

The protocol contains...• Identification (name of trial, people,...)• Justification• Objectives• Methods

– Type– Location– Treatments– Field layout (sites, blocks, plots)– Management– Measurements– ….– Analysis methods

• Implementation

Experiments with Farmers:

 Checklist for Preparing Protocols

- A list of items to think about

- Not a form to fill in