Research Methods ReviewResearch Methods Review

Advanced Cognitive Advanced Cognitive PsychologyPsychology

PSY 421 - Fall, 2004PSY 421 - Fall, 2004

OverviewOverview The BasicsThe Basics Experimental MethodsExperimental Methods

DesignsDesigns Within-subjectsWithin-subjects Between-subjectsBetween-subjects FactorialFactorial

Statistical ReviewStatistical Review Example of ExperimentExample of Experiment Don’t let me forget a simple Don’t let me forget a simple

assignment to give you before you assignment to give you before you leaveleave

What are Research What are Research Methods?Methods?

Application of the scientific method Application of the scientific method to studying behaviorto studying behavior

Scientific Method = observe, Scientific Method = observe, hypothesize/predict, test, concludehypothesize/predict, test, conclude

Work with structured observationsWork with structured observations Deal with solvable problemsDeal with solvable problems Produce publicly verifiable Produce publicly verifiable

informationinformation Hypotheses, theories, and Hypotheses, theories, and

explanations MUST BE FALSIFIABLEexplanations MUST BE FALSIFIABLE

General Methods – The General Methods – The BasicsBasics Watch people in public settings and drawing Watch people in public settings and drawing

conclusions = conclusions = Observational/DescriptiveObservational/Descriptive Conducting a survey, questionnaire, Conducting a survey, questionnaire,

interview, poll, etc. and comparing interview, poll, etc. and comparing responses on one question to other responses on one question to other questions = questions = Relational/CorrelationalRelational/Correlational

Randomly forming 2 or more groups, treat Randomly forming 2 or more groups, treat them all differently, see how the outcomes them all differently, see how the outcomes differ = differ = ExperimentalExperimental

Compare two or more groups (that are not Compare two or more groups (that are not randomly formed and are different to begin randomly formed and are different to begin with) in a variety of ways = with) in a variety of ways = Quasi-Quasi-ExperimentalExperimental

Experimental MethodExperimental Method This type of method attempts to This type of method attempts to

answer questions about CAUSEanswer questions about CAUSE VariablesVariables

Independent = manipulated or changed Independent = manipulated or changed for various subjects; IVfor various subjects; IV

Dependent = the way to measure a Dependent = the way to measure a change in behavior (or not change); DVchange in behavior (or not change); DV

Control = a potential IV that is held Control = a potential IV that is held constant at one level – all subjects are constant at one level – all subjects are exposed to that one levelexposed to that one level

Manipulation = changeManipulation = change

Manipulating Variables - Manipulating Variables - DesignsDesigns

One variable – multiple levelsOne variable – multiple levels Univariate or one-way designUnivariate or one-way design Level = one aspect of the variable; Level = one aspect of the variable;

conditioncondition ExampleExample

Manipulation TypesManipulation Types Between Subjects = ONEBetween Subjects = ONE

To expose some subjects to one level of the To expose some subjects to one level of the IV and other subjects to another level of the IV and other subjects to another level of the IV – subjects are not exposed to all aspects of IV – subjects are not exposed to all aspects of the variablethe variable

Within Subjects = ALLWithin Subjects = ALL To expose all subjects to all the levels of the To expose all subjects to all the levels of the

IVIV

The BOXThe BOX Boxes with rows and columns – Boxes with rows and columns –

essential for understanding essential for understanding experiment design and statisticsexperiment design and statistics

Level 1

Level 2Level 1

Level 2

Variable A

Variable B

Row 1

Row 2

Column 2Column 1

Factorial DesignsFactorial Designs When more than one variable is When more than one variable is

manipulated in an experimentmanipulated in an experiment 2 Variables = Two way2 Variables = Two way 3 or more variables = multivariate3 or more variables = multivariate Between subjects designBetween subjects design = all variables = all variables

are manipulated between/across the are manipulated between/across the levelslevels

Within subjects designWithin subjects design = all subjects = all subjects receive all the levels of all the variablesreceive all the levels of all the variables

Mixed designMixed design = at least one variable is = at least one variable is manipulated between subjects and at manipulated between subjects and at least one variable is manipulated within least one variable is manipulated within subjectssubjects

Statistical ReviewStatistical Review

Populations and SamplesPopulations and Samples Hypothesis TestingHypothesis Testing Using methods and statistical tests Using methods and statistical tests

togethertogether

Population vs. SamplePopulation vs. Sample Population = everyone that you are interested in Population = everyone that you are interested in

studying or at the very least, generalizing your studying or at the very least, generalizing your results toresults to Ex: Women; children; Men over 40Ex: Women; children; Men over 40 You can’t possibly hope to study the entire populationYou can’t possibly hope to study the entire population

Sample = a subset of the population that contains Sample = a subset of the population that contains the important characteristics of the population; a the important characteristics of the population; a sample is representative of its populationsample is representative of its population Various techniques for sampling from a populationVarious techniques for sampling from a population Ex: PSU Women; children at BFC; Male faculty in Ex: PSU Women; children at BFC; Male faculty in

psychology over 40psychology over 40 Why does this matter?Why does this matter?

Allows research to occur without the impossible task of Allows research to occur without the impossible task of studying everyone studying everyone

Important assumptions for statisticsImportant assumptions for statistics

Hypothesis TestingHypothesis Testing Comparing the Null and Experimental Comparing the Null and Experimental

hypotheses to predict the likelihood hypotheses to predict the likelihood of one being show to be trueof one being show to be true

Null Hypothesis = There is no Null Hypothesis = There is no difference; nothing will change; zerodifference; nothing will change; zero

Experimental Hypothesis = There will Experimental Hypothesis = There will be a difference; something will be a difference; something will change; non-zerochange; non-zero

Hypothesis TestingHypothesis Testing

Null Null HypothesisHypothesis

There is no There is no difference in difference in GRE scores GRE scores between between males and males and femalesfemales

Experimental Experimental HypothesisHypothesis

There is a There is a difference in difference in GRE scores GRE scores between between males and males and femalesfemales

In the real world, the Null is

Your decision

True False

Reject H0

Type I error

Correct Decision

Retain H0

Correct Decision

Type II error

Hypothesis Testing ExampleHypothesis Testing ExampleBarbie and Kendall – Chocolate Eaters

(to be read in class) From this, what could we conclude about this contradiction?

1. According to our assumption, in the real world, H0 is true. Therefore, if Barbie rejected H0 because she thought it was wrong (based on her study's results), what has happened? Did she commit an error or make a correct decision?

2. If Barbie would have retained H0 (and to do that, her study would have resulted in no differences between the mean exam scores from the two groups), and we assume that in the real world, H0 is true, did she commit an error or make a correct decision?

3. If we now assume that Kendall is wrong and in the real world, H0 is false, and Barbie rejected H0because she thought it was wrong (based on her study's results), what has happened? Did she commit an error or make a correct decision?

4. Again, assume that Kendall is wrong, and H0 is false in the real world. If Barbie would have retained H0 (and to do that, her study would have resulted in no differences between the mean exam scores from the two groups), and we assume that in the real world, H0 is true, did she commit an error or make a correct decision?

Hypothesis TestingHypothesis TestingAlpha Beta Effect Size Statistical Power

Definition The probability of committing a Type I error

The probability of committing a Type II error

The size of the effect (difference/

relationship)

The probability of rejecting a false null hypothesis (or the probability of finding an effect if one exists)

When to use Set prior to collecting data

Based on how much power you want (determined before collecting data) or how much power you actually have (determined after collecting data)

To be determined every time you run an inferential statistic (correlation, t-test, ANOVA, chi-square)

Prior to collecting data: estimate power and effect size to determine how many subjects you need to achieve certain level of power. After analyzing data: determine effect size and combine that with sample size to determine power

Interpre-tation

Percent chance of actually committing a Type I error (if the null is true)

Percent chance of actually committing a Type II error (if the null is false)

Small, medium, or large (quantitative values depend on type of effect size test you used)

Percent chance that you will find a statistically significant result given your sample size (N) and effect size (and assuming the null is false)

Example α = .05 means a 5% chance of committing a Type I error

β = .20 means a 20% chance of committing a Type II error

d = .50 means a medium effect size for the effect size corresponding to a t-test

Power = .80 means an 80% chance of finding a statistically significant result given your N, ES, and assumption that the null is false

Scales of MeasurementScales of Measurement Nominal = used to identify a particular Nominal = used to identify a particular

characteristics of the scale; also called categorical characteristics of the scale; also called categorical (categories are mutually exclusive)(categories are mutually exclusive) EX: Sex (M/F); ZIP CodesEX: Sex (M/F); ZIP Codes

Ordinal = numbers indicate whether there is more Ordinal = numbers indicate whether there is more or less of the measured variable; order is or less of the measured variable; order is importantimportant EX: Levels of education (Freshman, Sophomore, Junior, EX: Levels of education (Freshman, Sophomore, Junior,

Senior); Olympic medalsSenior); Olympic medals Interval = numbers correspond exactly to changes Interval = numbers correspond exactly to changes

in the measured variable and there are equal in the measured variable and there are equal distances between numbers that correspond to distances between numbers that correspond to equal changes in the measured variableequal changes in the measured variable EX: IQ; Temperature (Fahrenheit, Celcius)EX: IQ; Temperature (Fahrenheit, Celcius)

Ratio = like an interval scale (equal intervals) but Ratio = like an interval scale (equal intervals) but also includes a true zero point (the absence of the also includes a true zero point (the absence of the measured variable). This allows for multiplication measured variable). This allows for multiplication and division of scale values.and division of scale values. EX: Weight; Height; Temperature (degrees Kelvin)EX: Weight; Height; Temperature (degrees Kelvin)

Descriptive StatisticsDescriptive StatisticsDecide how to summarize and represent data based on Decide how to summarize and represent data based on

the the TYPETYPE of data that you have (its scale of of data that you have (its scale of measurement)measurement)

Measures of Central TendencyMeasures of Central Tendency

1.1. Nominal Scale – ModeNominal Scale – Mode

2.2. Ordinal Scale – MedianOrdinal Scale – Median

3.3. Interval/Ratio Scales - MeanInterval/Ratio Scales - Mean Measures of DispersionMeasures of Dispersion

1.1. Nominal Scale – RangeNominal Scale – Range

2.2. Ordinal Scale – Absolute Deviation from the MedianOrdinal Scale – Absolute Deviation from the Median

3.3. Interval/Ratio Scales – Variance, Standard Error, Interval/Ratio Scales – Variance, Standard Error, Standard DeviationStandard Deviation

Graphical Representations of DataGraphical Representations of Data

1.1. Nominal/Ordinal Scales (Qualitative Data) – Bar Graph, Nominal/Ordinal Scales (Qualitative Data) – Bar Graph, Pie ChartPie Chart

2.2. Interval/Ratio Scales (Quantitative Data) – Line Graph, Interval/Ratio Scales (Quantitative Data) – Line Graph, Frequency Polygon, HistogramFrequency Polygon, Histogram

Inferential StatisticsInferential Statistics

Decide which test to use based on the TYPE of Decide which test to use based on the TYPE of data you have and the KIND of outcome you data you have and the KIND of outcome you

are looking forare looking for

Relationships (Correlations)Relationships (Correlations)1.1. Nominal Scale – Chi-Square Test of Nominal Scale – Chi-Square Test of

Independence, or PhiIndependence, or Phi2.2. Ordinal Scale – Kendall’s Tau or Spearman’s Ordinal Scale – Kendall’s Tau or Spearman’s

rr3.3. Interval/Ratio Scales – Pearson’s Interval/Ratio Scales – Pearson’s rr

DifferencesDifferences1.1. One Independent VariableOne Independent Variable

Between-Subjects manipulationBetween-Subjects manipulation 2 levels2 levels

Nonparametric – Chi-Square Goodness of Fit Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal)(nominal) and Mann Whitney U (ordinal)

Parametric – Independent means t-test or one-way Parametric – Independent means t-test or one-way ANOVAANOVA

3+ levels3+ levels Nonparametric – Chi-Square Goodness of Fit Nonparametric – Chi-Square Goodness of Fit

(nominal) and Kruskal Wallace (ordinal)(nominal) and Kruskal Wallace (ordinal) Parametric – One-way ANOVAParametric – One-way ANOVA

Within-Subjects manipulationWithin-Subjects manipulation 2 levels2 levels

Nonparametric – Chi-Square Goodness of Fit Nonparametric – Chi-Square Goodness of Fit (nominal) and Mann Whitney U (ordinal)(nominal) and Mann Whitney U (ordinal)

Parametric – Independent means t-test or one-way Parametric – Independent means t-test or one-way ANOVAANOVA

3+ levels3+ levels Nonparametric – no good testsNonparametric – no good tests Parametric – One way Repeated-Measures ANOVAParametric – One way Repeated-Measures ANOVA

Differences, continuedDifferences, continued2.2. Two Independent VariablesTwo Independent Variables

Between-Subjects manipulationBetween-Subjects manipulation Nonparametric – Wilcoxon-Wilcox Nonparametric – Wilcoxon-Wilcox

and Friedman testsand Friedman tests Parametric – Factorial ANOVAParametric – Factorial ANOVA

Within-Subjects manipulationWithin-Subjects manipulation Nonparametric – Friedman testNonparametric – Friedman test Parametric – Factorial Repeated-Parametric – Factorial Repeated-

Measures ANOVAMeasures ANOVA Mixed manipulation – Mixed Factorial Mixed manipulation – Mixed Factorial

ANOVAANOVA

Putting this all together…Putting this all together… To study behavior, we have to create To study behavior, we have to create

conditions that are controlled enough to be conditions that are controlled enough to be able to predict an outcome in the controlled able to predict an outcome in the controlled conditionsconditions

We have to think about how to study the We have to think about how to study the behavior of interest and how to make it behavior of interest and how to make it change in a predictable fashionchange in a predictable fashion

Experimental methodology allows researchers Experimental methodology allows researchers to control the sample and expose the to control the sample and expose the participants to changes that are predicted to participants to changes that are predicted to influence the outcomeinfluence the outcome

When you construct a particular experiment or When you construct a particular experiment or use a particular research method, a certain use a particular research method, a certain logic applies when choosing the “proper” logic applies when choosing the “proper” statistical method to analyze the statistical method to analyze the results/outcome results/outcome

Experiment ExampleExperiment Example This will be shown in classThis will be shown in class

Signal Detection ExperimentSignal Detection Experiment IV – Presence of TargetIV – Presence of Target Levels: Present or AbsentLevels: Present or Absent Manipulation: Within-SubjectsManipulation: Within-Subjects Measure/DV: Hits, False Alarms, Correct Measure/DV: Hits, False Alarms, Correct

Rejections, MissesRejections, Misses