Validity = accuracy of study Reliability = consistency of results

VALIDITY- Does your study reveal accurate and representative results?

Reliability

• Is the measure able to give consistent measures over time?

Reliability checks?

• Test-retest• Split-half• Inter-observer

• Convergence…replications

VALIDITY- Does your study reveal accurate and representative results?

• Internal Validity and External Validity

– External Validity- generalizability• Issues

» Sampling» Return rates» …..

…External Validity

• Generalizeability– Across population– Across environments– Across time

Internal Validity

Convergent validity

• Different manipulations• Different measuring devices• Different populations…etc

• Replication

COMMON THREATS TO VALIDITY

• Selection bias– random selection and

assignment– Pretesting and matching

– Recruitment technique

History and Maturation Effects

• Changes in the DV over time• Between pretest and post-test • Or between test sessions

– constancy

Mortality

• Subject loss Over time

• May reflect unintended effect of IV or DV• Subject loss may result in selection bias

STOP FOR FUN!

• ESP in PSYCHOLOGY??– Somewhat critical– Lacking empirical validation– Should we remain open-minded?

– Let’s give it a shot

One of these symbols on each trial

STOP until test complete

Trial Number Symbol

1 red 2 blue 3 green 4 green 5 red 6 yellow 7 yellow 8 blue 9 red 10 blue 11 blue 12 yellow 13 red 14 green 15 green 16 green 17 blue 18 red 19 yellow 20 yellow

What is Chance Performance?

-1 out of 4 for each trial…-20 trials

Chance performance = 5

Just for fun let’s try again..this time seriously!

answers

Trial Number Symbol

1 Blue 2 Blue 3 Yellow 4 Red 5 Green 6 Yellow 7 Yellow 8 Red 9 Red 10 Green 11 Green 12 Blue 13 Red 14 Green 15 Yellow 16 Red 17 Blue 18 Blue 19 Yellow 20 green

Statistical Regression

• Regression to the mean with repeated testing

• Watch out for extreme scores..they may be due to chance

Instrumentation

• Changes in DV due to changes in measurement, not due to IV

Sequence Effects-repeated testing effects

• Practice effects• Test sensitization• Boredom/fatigue• Carry-over

– Randomization and counterbalancing

Subject Bias

• Subjects can have attitude! RESPONSE SETS that produce inaccurate results– Soc desireability– Cooperitivity– Negativity– Volunteerism– -blindin-recruitment

-deception or disguiselie detectors

Experimentor Bias

• Experimentors like to be correct! CONFIRMATIONAL BIAS

• Demand characteristics– constancy– Automation– Double blinding

How would you use blinding in a MJ drug study?

PLACEBO EFFECTS ARE REAL

• Expectation

• What might someone expect MJ to do?

• Beliefs/expectations can lead to real results….that are not related to the real effects of the IV.

DOUBLE-BLIND PLACEBO CONTROL DESIGNS

• Fake MJ?/ Ciggarette/ Pill other• Constancy• Can tease apart placebo effects from real

drug effects.

RANGE EFFECTS

• Don’t forget the inability to see an affect of an IV on a DV does not necessarily mean there isn’t one.

• Ceiling effects• Basement effects

ULTIMATE CONTROL

• Control procedures may be tailored for unique experiments

• Ultimate control may never be completely achieved

BEING A REAL PSYCHOLOGICAL SCIENTIST??

• The crux- identification of potential confounds

• AND• RIVAL HYPOTHESIS

RIVAL HYPOTHESIS

• Alternative explanations for experimental results

• Alternative to the research hypothesis

Consider the following

Hampton Court Maze

The radial arm maze for rats

Training of rats on the RAM

• Initial freezing• Adaptation and exploration• Neophobia to food pellet• Adaptation to food• Rapid acquisition of task

RAM memory errors?

• Entrance into previously entered arm• One foot? Two? Half-way? All the way?

Rats become quite good

• Rarely make a memory error• Win-shift strategy

Maybe they take an algorithmic strategy

• Right-right-right etc

To control for strategies..forced choice procedures

Using forced-choice procedures

• No indication of algorithm• Performance still very good

SO…

• What does rat performance in RAM show us?

Rival Hypothesis?

• Memory or sensory guidance?• Intra-maze or extramaze guidance?

• Spatial memory or…..?

Motivation-deficitsSensory-deficitsAttention-deficits