1

Handout Ten: Mixed Design Analysis of Variance

EPSE 592Experimental Designs and Analysis in

Educational ResearchInstructor: Dr. Amery Wu

Today, we will introduce the analysis and assumptions of a mixed design.This design explains/predicts quantitative data using two independent variables- one (explicit) between groups and one (implicit) within-subjects (repeated measures). Each IV has two or more conditions.

Where We Are TodayMixed Design Analysis of Variance

Measurement of Data

Quantitative Categorical

Type ofthe

Inference

DescriptiveA B

InferentialC D

2

3

case diet before warm-up after1 0 95 134 1862 0 66 109 1503 0 69 119 1774 0 93 151 2175 0 77 122 1786 0 78 119 1737 0 58 99 1318 0 85 132 1869 0 78 110 16410 1 112 166 22511 1 111 166 23512 1 89 132 19913 1 125 177 25114 1 85 117 19615 1 97 137 19516 1 81 134 21517 1 88 133 19018 1 88 157 234

An Example to Contextualize the Learning

2x3 Mixed Design ANOVA

4

An Example to Contextualize the Learning

2x3 Mixed Design ANOVAQuestions:1. What is the DV?2. What is the between-groups factor? How many levels are there?3. What is the within-subjects factor? How many levels are there?

Lab Activity:Using your own words, state as many (non-redundant) research questions as you can based on this research design.

5

1. Does exercise has an effect on the pulse? (Dose the pulse change over time (before, during, and after exercise)?2. Is there a difference in the pulse between the two dietary groups?3. Dose the pattern of temporal change in pulse due to exercise differ (be moderated) between dietary groups?Alternatively, is the relationship between diet and pulse differs (be moderated) by the timing of when the pulse is taken?

An Example to Contextualize the Learning

2x3 Mixed Design ANOVA

6

DesignExperimentalObservational

DataQuantitativeCategorical

ModelDescriptive/SummativeExplanatory/Predictive

InferenceDescriptive vs. InferentialRelational vs. Causal

ResearchQuestion

Quantitative Methodology Network

7

case diet before warm-up after1 0 95 134 1862 0 66 109 1503 0 69 119 1774 0 93 151 2175 0 77 122 1786 0 78 119 1737 0 58 99 1318 0 85 132 1869 0 78 110 16410 1 112 166 22511 1 111 166 23512 1 89 132 19913 1 125 177 25114 1 85 117 19615 1 97 137 19516 1 81 134 21517 1 88 133 19018 1 88 157 234

Lab ActivityThere are six groups of samples separated by three columns (purple, blue, and yellow) and by the 2 categories of the diet column (in green and red texts) . Questions: In what way, the observation of the six groups is supposed to be dependent and independent?

An research design that incorporates both between- groups (i.e., independent) factor and within-subject (i.e., dependent, correlated, or repeated measures) factors.Hence, for a mixed design, there is at least two IVs with a possibility of inclusion of another interaction IV: “between* within” (i.e., two-way factorial design with interaction).Mixed design is also known as split-plot design

Mixed (between & within) Experimental Design

8

9

For the between-groups factor, the group membership of the participants are independent (i.e., exclusive). Participants in one group differ from those in another groups. Hence, the observation of the DV scores are unrelated across the groups.Examples for between-group factors are gender groups, randomly assigned treatment groups, or teachers in different school districts.

Mixed (between & within) Experimental Design

10

For the within-subjects factor (e.g., repeated measures), the observation of the participants under one condition is dependent of that of another because the participants are selected to be related by design. Examples for within-subject factors are repeated measures groups, matched groups, twin groups, group scores by the same raters, or dyads groups.

Mixed (between & within) Experimental Design

What is a Mixed Design AnalysisMixed Design ANOVA is an inferential statistics

that tests experimental treatment effects (or simply mean differences for observation data) with a mix of between-group and within-subject factors.A full factorial mixed design with one between and one within factors is similar to a two-way factorial ANOVA. The only difference is that one of the IV now is a within-subjects factor.A full two-way factorial ANOVA may include testing of the following effects: 1. the main effect of the between-groups factor2. the main effect of the within-subjects factor3. interaction effect- “between x within” factor

11

12

See Excel file “Exercise_Diet_Pulse_Mixed Design with Interaction.xlsx” for computing F statistics for the mixed design ANOVA.

13

    Within (Time)  

  Before Warm-up After

Between Vege 77.67 121.67 173.56 113.96

 (Diet) Meat 97.33 146.56 215.56 110.51

  94.88 143.44 210.31 138.72

Means – Cross Tabulation between by within

Questions:1. By comparing which values, is the main

effect of the between group factor (diet) examined?

2. By comparing which values, is the main effect of the within-subject factor (time) examined?

3. By comparing which values, is the interaction effect of between*within (diet*time) examined?

Example- 2x3 Mixed Design

14

The between-factor effect is investigated by comparing the between-factor marginal means- the mean DV scores among the groups for the between-factor(e.g., treatment groups or vegetarians vs. meat eaters). The F statistics is calculated using the error term for the between-factor (see Excel file).

Between-factor Effect

Vege 77.67 121.67 173.56 113.96Meat 97.33 146.56 215.56 110.51

15

The within-factor effect is investigated by comparing the within-factor marginal means- the mean DV scores among the groups for the within-factor (e.g., repeated measures of same individuals; comparing their own repeated DV scores under different treatment conditions).The F statistics is calculated using the error term for the within-factor (see Excel file).

Within-factor Effect

Before Warm-up After77.67 121.67 173.5697.33 146.56 215.5694.88 143.44 210.31

16

“Between x Within” Interaction Effect

The “between x within” interaction effect is investigated by conditional mean differences (i.e., simple main effects)- comparing the mean differences in the DV score among the between-factor groups for each of the within-factor groups (alternatively, among the within-factor groups for each of the between-factor groups).Take the Pulse data for example, the interaction effect is investigated by comparing the mean differences between the diet groups (vege vs. meat) for each of the three within groups (before, warm-up, and after exercise).The F statistics is calculated using the error term for the within-factor.

    Within  Before Warm-up AfterBetween Vege 77.67 121.67 173.56  Meat 97.33 146.56 215.56

diff 19.67 24.89 42.00

Data Assumptions about Mixed Effect ANOVA

The DV scores for each of the PxQ groups are all normally distributed (P= # of groups for the between-factor and Q= # of groups for the within-factor). Independence and equal (homogeneity) variances assumption for the between-factor groups. Normal distribution and sphericity assumption for the within-factor groups. The difference scores between any pairs of the repeatedly measured are normally distributed. Also, the variances of the difference scores are equal (sphericity). With Q groups, there are Q(Q-1)/2 difference scores. 17

18

Interpretation of the Mixed Design ANOVA-BetweenInterpretation of the between-factor effect

makes sense only if the interaction effect is non-significant.The between-factor effect is interpreted the same way as is the one-way independent ANOVA. The omnibus F test for the between-factor examines whether there is any statistically significant mean difference in the DV between any pairs of between-factor groups. If the omnibus F test is statistically significant, do post-hoc pairwise t-tests to find out which pair(s) of mean difference is statistically significant. If the equal variances assumption is not met, remember to choose a post-hoc pairwise t-test that does not assume equal variances.

19

Interpretation of the within-factor main effect makes sense only if the interaction effect is non-significant.The within-factor main effect is interpreted the same way as is the repeated measures ANOVA (see Handout Four). The omnibus F test for within-factor examines whether there is any statistically significant mean difference in the DV between any pairs of within-factor groups. If the sphericity assumption is violated (Greenhouse-Geisser’s ἓ less than 0.75), pick one of the alternative F-tests that corrects for violation from sphericity (i.e., Greenhouse-Geisser, Huynh-Feldt, or Lower-bound in SPSS). If the selected omnibus F test is statistically significant, do post-hoc pairwise t-tests to find which pair(s) of mean difference is statistically significant. If sphericity assumption is violated, choose Bonferroni or Sidak’s post hoc t-test for correction(see Handout Four).

Interpretation of the Mixed Design ANOVA- Within

20

Interpretation of the Mixed Design ANOVA- Interaction

If the omnibus F test for the interaction effect is non-significant, interpret the each of the significant main effect separately. Note that once the omnibus F test for the interaction effect is significant, it is inappropriate to interpret the between- or within-factor main effects even if these main effects are significant (always interpret the highest order of the effect that is significant).SPSS drop-down menu does not provide a function to test the simple main effects. With SPSS, to test and interpret the interaction effect in terms of simple main effects (i.e., conditional mean differences). See later slides.

21

Mixed Design ANOVA by SPSS

Remember to hit “Add”

22

Mixed Design ANOVA SPSS Output First, examine whether the interaction effect is significant

within the F table of within-subjects effects. To do so, we need to check whether the sphericity

assumption (for with-factor) is met. Because the assumption is not met (using Greenhouse-Geisser Epsilon <0.75 criterion), we should use an F test that does not assume sphericity.

23

Mixed Design ANOVA SPSS OutputBecause the “Time x Diet” Interaction effect is significant, the main effects of time and diet should be interpreted in the context of the other by interpreting the simple mail effects.

24

One has to use the following syntax to produce the output:

GLM PulseBefore PulseWarmUp PulseAfter BY Diet

/WSFACTOR=Time 3 /MEASURE=Pulse /METHOD=SSTYPE(3) /EMMEANS=TABLES(Diet*Time)

compare (diet) adj (Bonferroni) /CRITERIA=ALPHA(.05).

Testing Simple Main Effects Using SPSS Syntax

25

Interpretation of the Mixed Design ANOVA- InteractionThis is the extra table produced by the bolded line

in the syntax for testing the interaction effect in terms of simple main effects.For each of the within-factor groups (before, warm-up, and after), the mean difference in pulse between the two dietary groups is compared and tested.

Simple main effects

26

If the Interaction Effect is Non-significant

When the interaction effect is non-significant, one can interpret the results separately for each of the between- and within-factor if they are significant.

One can test the planned contrasts or conduct post hoc t tests, correcting for type-one error rate and correcting for violation of equal variances assumptions (for both each between- and within- effects). See Power Points slides course handout #8 and #9.