methods for tina

89
Some experimental methods in Educational Research Questions Design Analysis Adam Dubrowski, HSC Learning Institute

Upload: adam-dubrowski

Post on 22-May-2015

414 views

Category:

Technology


2 download

TRANSCRIPT

Page 1: Methods for Tina

Some experimental methodsin Educational Research

QuestionsDesign

Analysis

Adam Dubrowski, HSC Learning Institute

Page 2: Methods for Tina

Not enough?

Page 3: Methods for Tina

You need to

perform a systematic review of literature

Page 4: Methods for Tina

You need to

generate an answerable question

Page 5: Methods for Tina

You need to

generate a

Page 6: Methods for Tina

You need to

design of a “clean” experiment

Page 7: Methods for Tina

You need to

be able to use the stats

Page 8: Methods for Tina

You need to

generate an answerable question

Page 9: Methods for Tina

Types of Questions

• Health Care Intervention (or treatment, prevention therapy): Determining the effect of different interventions

• Harm (causation): Ascertaining the effect of potentially harmful agents

• Prognosis: Estimating the future course of a patient’s disease or condition

• Diagnosis (or assessment): Establishing the power of a diagnostic tool

• Meaning: Describing, exploring, and explaining phenomena

• Economics: Studying the economic efficiency of health care programs or interventions

Page 10: Methods for Tina

Types of Questions

• Education: Determining the effect of educational interventions

Page 11: Methods for Tina

Matching Question to Study Design

• Quantitative studies are most useful for answering questions of “how many” or “how much”

Qualitative studies are most appropriate for answering questions about how people “feel about” or “experience” certain situations and conditions.

Page 12: Methods for Tina

Matching Question to Study Design

• Quantitative studies are most useful for answering questions of “how many” or “how much”

• Qualitative studies are most appropriate for answering questions about how people “feel about” or “experience” certain situations and conditions.

Page 13: Methods for Tina

Asking Structured Questions (WHY?)

Page 14: Methods for Tina

Asking Structured Questions (WHY?)

Page 15: Methods for Tina

Asking Structured Questions (WHY?)

Page 16: Methods for Tina

Asking Structured Questions (WHY?)

Page 17: Methods for Tina

Asking Structured Questions (WHY?)

Page 18: Methods for Tina

Quantitative Questions (PICO)

• Population• Intervention or exposure• Comparison• Outcome

Page 19: Methods for Tina

Quantitative Questions (PICO)

• The Population

– Who are the trainees?– Why are they enrolled?– Is there a particular age or sex grouping?– Is there a particular profession grouping?– What else?

Page 20: Methods for Tina

Quantitative Questions (PICO)

• The Intervention

– What interventions are we interested in?– Examples?

Page 21: Methods for Tina

Quantitative Questions (PICO)

• The Comparison

– What are the current educational practices or what are other possible educational practices that we are interested in comparing?

Page 22: Methods for Tina

Quantitative Questions (PICO)

• The Outcome

– What are the relevant consequences of the intervention in which we are interested?

• Satisfaction• Performance

– Individual– Team– Patient

• System changes• Economics

Page 23: Methods for Tina

Quantitative Questions (PICO)

• Group work

Page 24: Methods for Tina

Hypothesis

Page 25: Methods for Tina

Hypothesis

A hypothesis [from Greek] consists either of a suggested explanation of a phenomenon or of a reasoned proposal suggesting a possible relationship between multiple phenomena.

In science: The Scientific Method requires that one can test a scientific hypothesis.

Generally base such hypotheses on previous observations or on extensions of scientific theories.

Page 26: Methods for Tina

Hypothesis

• For evaluation of a hypothesis, one needs to define operational terms.

“I predict that A will lead to D only if B and C are in place”

• A hypothesis should enable predictions by testing.

• A hypothesis requires work by the researcher in order to either accept or reject it.

Page 27: Methods for Tina

Hypothesis

• A hypothesis may take the form of asserting a causal relationship (such as "A causes D") but not always.

• For example, if a particular independent variable changes, then a certain dependent variable also changes.

However …… …. what we do not know is whether there is a direct causal relationship, or whether there is another variable in the middle that causes the change.

Page 28: Methods for Tina

Hypothesis: Variables

An independent variable is the variable whose value one actively controls and can change.

A dependent variable is the thing whose value then changes as a result.

Example:

In a study of how introduction of eLearning modules affect the adherence to a program, a researcher could compare the course drop out rate (the ? variables) with the presence of eLearning module (the ? Variable), and attempt to draw a conclusion.

Page 29: Methods for Tina

Hypothesis: Variables

An independent variable is the variable whose value one actively controls and can change.

A dependent variable is the thing whose value then changes as a result.

Page 30: Methods for Tina

Hypothesis testing

In statistics, there are only two kinds of hypothesis that are being tested:

The null hypothesis or H0. The alternative hypothesis or H1.

Page 31: Methods for Tina

Hypothesis testing

In education research, the null hypothesis may be used to test differences in intervention and control groups at the end of a program, and the assumption is that no difference exists between the two groups for the outcome variable being compared.

Page 32: Methods for Tina

Not guilty unless proven otherwise

Page 33: Methods for Tina

Hypothesis testing: Statistical significance

The “p” value or alpha is the chance you are willing to take to reject the Ho [not guilty], while you actually should have retained it.

The choice of the level is up to the experimenter and may be determined as a function of the phenomenon being studied.

Page 34: Methods for Tina

PausePause

Page 35: Methods for Tina

Experimental design

Page 36: Methods for Tina

You have

performed a systematic review of literature

Page 37: Methods for Tina

You have

generated an answerable question

Page 38: Methods for Tina

You have

generated a

Page 39: Methods for Tina

What is the next step?

Page 40: Methods for Tina

You need to

design of a “clean” experiment

Page 41: Methods for Tina

Example

You may wish to examine if a new hands on simulation based educational program leads to adherence to hand washing protocols [by nurses ….newly hired nurses …etc].

Is this a good research question?

Does it have all the parts?

What are the hypotheses?

Ho

H1

Experimental design

Page 42: Methods for Tina

Example

You may wish to examine if a new hands on simulation based educational program leads to adherence to hand washing protocols.

Is this a good research question? OK, but not great

Does it have all the parts? No control, participants, outcomes.

What are the hypotheses?

Ho: Old and new programs are the same

H1: Old and new are different (note no directionality)

Experimental design

Page 43: Methods for Tina

To test this hypothesis requires 2 conditions to be met (Cook

and Campbell; 1979).

1. Changes in the outcome occur after, rather than before the institution of the program/intervention.

2. The program/intervention are the only reasonable explanation for the changes in the outcome.

If there are any other explanations for the observed changes in outcomes the researcher cannot be confident that the presumed cause effect relationship is correct.

How do you know if there are other explanations?

Experimental design

Page 44: Methods for Tina

Eliminating these alternative explanations is the purpose of a proper experimental design.

These are also known as threats to internal validity (Cook and Campbell 1979).

Experimental design

Page 45: Methods for Tina

Minimizing Threats to Internal Validity:

Argument. This is the least effective ways to argue threats to internal validity.

- In a paper it is in the intro

Design. This is by far the most powerful method to rule out alternative explanations.

- In a paper it is in the methods

Analysis. The researcher can use various statistical analysis performed on the collected data.

- In a paper it is in the methods and results

Experimental design

Page 46: Methods for Tina

The three ways of Minimizing Threats to Validity are not mutually exclusive and a good research plan should make use of multiple methods for reducing threats.

Experimental design

Page 47: Methods for Tina

Design construction

• Most research designs can be conceptualized and represented graphically from four basic elements

• Time, randomization, groups, interventions, observations, etc.

XOXOXOXO

Experimental design

Page 48: Methods for Tina

Time. In design notation time is represented horizontally.

Intervention (s). In design notation the intervention is depicted with the symbol "X".

Observation (s). Assessments and observations are depicted as the symbol "O".

XOXOXOXOX

Experimental design

Page 49: Methods for Tina

Groups. Each group is indicated on a separate line.

Most importantly however, the manner in which groups are assigned to the conditions can be indicated by a letter:

• "R" represents random assignment, • "N" represents non-random assignment (i.e., a

nonequivalent group or cohort) • "C" may represent an assignment based on a cutoff

score.

Group 1: R O X OGroup 2: N O X O

Experimental design

Page 50: Methods for Tina

The most basic causal relationship between educational intervention and outcome can be described using the following annotation:

XO

This is the most simple design and serves as a starting point for the development of better strategies.

What is wrong with it?

No control for threats to internal validity: • Level before intervention?• Historical events?

Experimental design

Page 51: Methods for Tina

When the researcher is faced with being able to deliver an intervention to all participants one can include additional observations either before or after the intervention.

O O X O O

• Provides a "baseline"

• No intervention vs. intervention

• Additional posttest assessments: decay, or a lag.

What is wrong with this deign?

Still we do not know if X (or the intervention) caused change in O

Experimental design

Page 52: Methods for Tina

Use a control group!

N O X ON O O

• The interpretation: The study was comprised of two groups, with non-random assignment of participants to each of the groups. There was an initial assessment before the intervention implementation. Subsequently, participants in the first group receive the intervention (indicated by X), while participants in the second group do not. Finally, all participants are reassessed.

Why is the first “O” important?

To ensure no bias

Experimental design

Page 53: Methods for Tina

• In education especially the initial tests may be viewed as a contaminating factor.

R X OR O

• The posttest-only randomized experimental design allows the researcher to assume that there are randomized assignment of the participants to the two groups will ensure similar patient distribution.

Experimental design

Page 54: Methods for Tina

• These designs assess the effectiveness of intervention when compared to a lack of alternative intervention.

• That is the C group did not get any intervention.

R X1 OR X2 O

Experimental design

Page 55: Methods for Tina

• Inclusion of additional groups in the design may be necessary in order to rule out specific threats to validity.

• The researcher may be inclined to add an additional nonequivalent group from a similar institution.

R O X O R O ON O O

Experimental design

Page 56: Methods for Tina

Another possibility is to use pre-post cohort groups:

N O X ON O

N O

The treatment group consists of current learners, the first comparison group last year’s learners assessed in the same year, and the second comparison group consists of the following year's learners.

Experimental design

Page 57: Methods for Tina

The Nature of Good Design

• Be linked with theory (know your threats)

• Be innovative

• Be realistic

• Be flexible

• Do not over-design

• Word of the ay: parsimonious

Experimental design

Page 58: Methods for Tina

PausePause

Page 59: Methods for Tina

You have

performed a systematic review of literature

Page 60: Methods for Tina

You have

generated an answerable question

Page 61: Methods for Tina

You have

generated a

Page 62: Methods for Tina

You have

designed a “clean” experiment

Page 63: Methods for Tina

What is the next step?

Page 64: Methods for Tina

You need to

be able to use the stats

Page 65: Methods for Tina

• The differences are important, but more important is the variability (noise)

• Control confounding

variables!

Basic stats - variability

Page 66: Methods for Tina

How do we measure that?

Page 67: Methods for Tina

Independent-Samples T Test• This procedure compares means for two groups of cases

R X OR X O

• Example. Learners are randomly assigned to a new educational approach group and the rest follows standard curriculum.

T-test

Page 68: Methods for Tina

Standard

New

Skill

T-test

Page 69: Methods for Tina

Standard

New

Skill

T-test

Page 70: Methods for Tina

Standard

New

Skill

T-test

Page 71: Methods for Tina

Paired-Samples T Test• This procedure compares the means of two variables for

a single group. The procedure computes the differences between values of the two variables for each case and tests whether the average differs from 0.

O X O

• Example. Learners participate a new curriculum.

T-test

Page 72: Methods for Tina

Post

Pre

Skill

O X O

T-test

Page 73: Methods for Tina

One-Sample T Test• This procedure tests whether the mean of a single

variable differs from a specified constant.

• Examples. A researcher might want to test whether the skill improvement for the group participating in a new curriculum differs from a known average.

T-test

Page 74: Methods for Tina

LimitationsOnly 2 groups or valueDoes not correct for number of tests

T-test

Page 75: Methods for Tina

The One-Way ANOVA

• This technique is an extension of the independent-sample t test.

• In addition to determining that differences exist among the means, you may want to know which means differ. Post hoc tests are run after the experiment has been conducted to test for trends across categories.

ANOVA

Page 76: Methods for Tina

Example. Learners are participating in 2 new programs. You want to know which one is better, but you are also interested to know how does the standard curriculum compare.

Independent variable:Program (New A, New B and Standard)NOTE: This is also known as between group or between

subject variable

Dependent variable:Skill

ANOVA

Page 77: Methods for Tina

Standard

Skill

New A New B

ANOVA

Page 78: Methods for Tina

LimitationAllows for only one independent variable

Q: What if one of the two programs is an eLearning module and the other is f2f? You may suspect that the cyber natives and cyber shy people may have different benefits.

ANOVA

Page 79: Methods for Tina

Multifactor ANOVA

• Using this procedure, you can test for the effects of individual factors.

• You can also investigate interactions between factors.

• After an overall test has shown significance, you can use post hoc tests to evaluate differences among specific means.

ANOVA

Page 80: Methods for Tina

Cyber shy

Cyber native

Skill

ANOVA

Page 81: Methods for Tina

Standard

Skill

New A New B

ANOVA

Page 82: Methods for Tina

Standard

Skill

New A

New A

New BNew B

ANOVA

Page 83: Methods for Tina

Each of these tests has a non-parametric equivalent.

Why? When?

If 2 assumptions are violated:

1. The data does not follow normal distribution.

2. N<15

Non-parametric test

Page 84: Methods for Tina

You have

performed a systematic review of literature

Page 85: Methods for Tina

You have

generated an answerable question

Page 86: Methods for Tina

You have

generated a

Page 87: Methods for Tina

You have

designed a “clean” experiment

Page 88: Methods for Tina

What is the next step?

Page 89: Methods for Tina

You have

Been introduced to stats