how to critically analyse psychological research

5/22/2018 How to Critically Analyse Psychological Research

1/13

TheUniversity of Newcastle, Australia

How to Critically Analyse Psychological Research

Table of Contents

The Theory.....................................................................................................................................2

The Research Rationale.................................................................................................................2The Participants..............................................................................................................................2The Design and Procedure............................................................................................................2

1. Research method..............................................................................................................22. a! vs "ield research.........................................................................................................2#. Demand characteristics.....................................................................................................#$. %&perimenter !ias.............................................................................................................#'. (ocial desira!ility...............................................................................................................#). *alidity o" the e&perimental manipulation..........................................................................#+. (timulus sampling.............................................................................................................$. Relia!ility and validity o" measures o" the independent and-or dependent varia!les.......$1. /on"ounding varia!les in...................................................................................................$11. 0rder o" items-events........................................................................................................$

The (tatistical Analyses..................................................................................................................1. %&cluded participants........................................................................................................2. issing data......................................................................................................................#. *alidity and relia!ility o" dependent varia!les...................................................................$. (u""icient statistical power.................................................................................................. (tatistical assumptions......................................................................................................''. /orrect use o" in"erential statistics....................................................................................'). /orrect interpretation o" analyses.....................................................................................'

+. Alternative analyses..........................................................................................................'The Discussion...............................................................................................................................'

1. Alternative e&planations....................................................................................................'2. /ause3e""ect am!iguities...................................................................................................'#. Third varia!le.....................................................................................................................)$. ediators and moderators................................................................................................). Replication.........................................................................................................................)'. 4nteraction or main e""ect56................................................................................................)

Place the Research in the /onte&t o" (imilar Research................................................................+(uggestions "or 7uture Research...................................................................................................+4nappropriate /riticisms..................................................................................................................+

1. /ritici8ing the article rather than the research..................................................................+2. %thical criticisms................................................................................................................+#. 4ncomplete criticisms.........................................................................................................+$. /riticisms o" the relia!ility or e""ectiveness o" methodology that produced the predictedresults........................................................................................................................................... Random allocation o" participants to conditions...............................................................

9ow Not to Use this Document:...................................................................................................1(tructuring a /ritical Review........................................................................................................1Use"ul ;e!sites............................................................................................................................11


2/13


The Theory>ou may wish to critici8e the theory that the researchers are testing. 9ow does it

compare against competing theories in the area5 ;hat are its strengths and wea=nesses5 Ane&cellent resource "or thin=ing a!out theory construction is the (pecial 4ssue on ?Theoryconstruction in social personality psychology6 Personal e&periences and lesson learned@ in thePersonality and Social Psychology Review*ol. +B.

The Research Rationale4s there a "ault in the logic o" the theoretical rationale "or the research5 9ave the

researchers interpreted the theory that they are !asing their hypotheses on correctly5 Do thehypotheses "ollow logically "rom the theory5 Does the research design provide a satis"actorytest o" the research hypotheses5 Are all o" the necessary e&perimental and control conditionsincluded5 Are all o" the necessary varia!les measured5 (ee cCuire 2$B "or a gooddiscussion on these points.

The ParticipantsA!out ) percent o" psychology research is conducted using young, educated, white,

middle3class, ;estern, volunteer, psychology undergraduate students (herman et al., 1E;intre, North, F (ugar, 21B. 9ence, it is possi!le that ) o" psychology research cannot !egenerali8ed to the rest o" the worldGs population. 9owever, you should consider two points!e"ore ma=ing re"erence to this sample generalisation pro!lem. 7irst, the sample generalisationpro!lem is unli=ely to threaten the e&ternal validity o" research investigating !asic cognitive andperceptual processes such as vision, !ecause there is no reason to !elieve that psychologyundergraduates see di""erently to other types o" people (tanovich, 2), p. 112B. (econd, thesample generalisation pro!lem is widely recognised among psychologists (tanovich, 2), p.

11)B, and it does not need to !e stated e&plicitly unless the characteristics o" the sample pose aparticular pro!lem in relation to the speci"ic independent and dependent varia!les !einginvestigated.

The Design and Procedure1. Research method6 %very research method has its advantages and its disadvantages. Did

the researchers choose the most appropriate research method "or the particular researchHuestion that they were investigating5 Did they deal with the disadvantages o" that method54" not, how do you thin= that those disadvantages may have a""ected the results5 7ore&ample, did the researchers conduct their research on the internet, and i" so did they

address the limitations o" this particular methodology


3/13


#. Demand characteristics6 Demand characteristics are ?the totality o" cues which convey ane&perimental hypothesis to the su!Ject@ 0rne, 1'2, p. ))E see also the (pecial 4ssue inPrevention and Treatment, 22E (trohmet8, 2+B. The most common sources o" demandcharacteristics are the research setting, the implicit and e&plicit research instructions, andthe research procedure. Demand characteristics are a pro!lem !ecause, i" participants area!le to deduce the research hypotheses, then they may respond in a manner that they thin=

will con"irm the hypothesis in order to !e a ?good@ participant and not ?ruin@ the researche.g., Noren8ayan F (chwar8, 1B. ;hat demand characteristics do you thin= e&isted inthe research5 As (trohmet8 2+B noted, the impact o" demand characteristics depends onparticipantGs receptivity to to these characteristics and their motivation and a!ility to complywith them. Do you thin= that participants were a!le to guess the research hypotheses "romthese demand characteristics5 Do you thin= that participants were motivated to try to con"irmthese hypotheses5 9ow do you thin= that their attempts to con"irm the hypotheses will havea""ected the results5 ;as any deception and-or concealment used in the research in order toprevent demand characteristics "rom having an e""ect, and i" so, how e""ective do you thin=that this deception-concealment was5

$. %&perimenter !ias6 The e&perimenterGs nonver!al !ehaviour may give away clues a!out howthe participant is e&pected to respond Rosenthal F Rosnow, 1'B. As per demandcharacteristics see a!oveB, this nonver!al !ehaviour may then in"luence participantsGresponses and produce results that are caused !y arti"icial "actors that depend on theparticipantsG =nowledge that they are ta=ing part in an e&periment, rather than !y genuinepsychological processes that can !e generali8ed outside o" the e&perimental conte&t.Usually, e&perimenter !ias can !e avoided i" the e&perimenter is unaware o" the researchhypotheses or i" his-her nonver!al !ehaviour is una!le or unli=ely to in"luence theparticipantsG responses. ;as the e&perimenter !lind to the e&perimental conditions5 4" not,was there any way that his-her nonver!al !ehaviour could have systematically in"luenced

participantsG responses5

. Reactivity6 (ometimes the act o" measuring a thought or !ehaviour can change that thoughtor !ehaviour "or an overview, see 7rench F (utton, 211B. 7or e&ample, the act o"measuring the same attitude or !ehaviour at di""erent times during a research study maylead participants to assume that the researchers predict the attitude or !ehaviour to change"rom one measurement to the ne&t. 4" researchers ma=e multiple measurements o" the sameattitude or !ehaviour, have they addressed the potential reactivity o" this procedure5

'. (ocial desira!ility6 People want to present themselves in a good light when they ta=e part inresearch /rowne F arlowe, 1'$E Paulhus F Reid, 11B. They donGt want to !e seen as

?!ad@ or ?wrong@. To avoid these la!els, participants will o"ten downplay their sociallyundesira!le attitudes or !ehaviours. (o, "or e&ample, participants may descri!e themselvesas !eing less aggressive than they actually !elieve that they are in order to presentthemselves in a more positive light. ;as the research li=ely to have !een in"luenced !yparticipantsG desire to appear socially desira!le5 4" so, how might this motivation havea""ected the pattern o" results that the researchers "ound5 Researchers can use sa"eguardsagainst socially desira!le responding such as allowing participants to ma=e anonymousresponses or measuring individual di""erences in social desira!ility and then controlling "orthis varia!le in their statistical analyses. ;ere any o" these sa"eguards in place and, i" so,how e""ective do you thin= that they were5

). *alidity o" the e&perimental manipulation6 Did the e&perimental manipulation alter theindependent varia!le as predicted5 Did it alter any other varia!le as well as the independentvaria!le5 7or e&ample, researchers might attempt to manipulate sel"3esteem !y as=ing theirparticipants to watch either a happy or sad video.


4/13


instead o" sel"3esteem. 9ence, the e&perimental manipulation is invalid !ecause it is notmanipulating sel"3esteem, !ut mood instead. (ometimes, researchers may include amanipulation checkin their research. This is a measure that is intended to show that thee&perimental manipulation has had a signi"icant e""ect in manipulating the correct varia!le.;as a manipulation chec= included5 4" a manipulation chec= was included, did it show thatthe manipulation was e""ective5 4n other words, did it indicate signi"icant di""erences in the

independent varia!le !etween relevant e&perimental conditions5 Note that, even i" themanipulation chec= is success"ul, it remains possi!le that the e&perimental manipulationmanipulated more than Just the independent varia!le e.g., sel"3esteemB and that anadditional, con"ounding varia!le e.g., moodB was actually the one that was responsi!le "orthe signi"icant e""ects that were o!served.

+. (timulus sampling6 A related issue is that o" stimulus sampling ;ells F ;indschitl, 1B.;ere the o!served e""ects due to the independent varia!le or the particular stimuli that wereused to represent the independent varia!le5 7or e&ample, suppose a researcher tests thehypothesis that women li=e children more than men do. To test this hypothesis, theresearcher presents male and "emale participants with a single picture o" a child and as=sthem to rate how much they li=e that child. 4n this case, any gender e""ects may !e more todo with the speci"ic picture o" the child that the researcher has chosen to represent thegeneral category o" ?children@ e.g., perhaps the childGs own gender is having an e""ectB. 4norder to ensure the content validity o" this varia!le, the researcher should sample a variety o"di""erent pictures o" children stimuliB with di""erent gender, age, appearance, etc. in order torule out these potentially con"ounding varia!les "rom the research.

. Relia!ility and validity o" measures o" the independent and-or dependent varia!les6 9avemeasures o" the independent and dependent varia!les !een shown to !e a relia!le in thepresent research and in previous research e.g., test3retest relia!ility, internal relia!ilityB5

9as they !een shown to !e a valid measures in the present research and in previousresearch e.g., "ace validity, content validity, criterion validityB5 9ave the psychometric scales!een developed in an appropriate manner e.g., /lar= F ;atson, 1E 9aynes, Richard, FKu!any, 1B.

1./on"ounding varia!les in measures o" the independent and-or dependent varia!les6 Did themeasures o" the independent and-or dependent varia!les assess one or more additionalvaria!les to the one that the researchers were interested in5 7or e&ample, the items in ascale measuring aggressive !ehaviour might also tap sel"3esteem to some e&tent. 4n thiscase, perhaps the signi"icant e""ects that the researchers "ound represent di""erences in sel"3esteem rather than di""erences in aggression. Researchers can attempt to control "or

variation due to sel"3esteem !y including a sel"3esteem scale in their research and using thisas a covariate in their statistical analyses.

11. 0rder o" items-events6 The order in which researchers present items or events toparticipants can ma=e a !ig di""erence to the way in which participants interpret those itemsor events e.g.,


5/13


e""ects "or scales placed at the end o" the research. The researchers might claim that thesedi""erent e""ects are due to the content o" the scales. >ou might argue that the di""erente""ects are simply !ecause participants arenGt really attending to the items in the last scalei.e., a "atigue e""ectB. 0ne way researchers can deal with these sorts o" order e""ects is tocounter!alance the order in which the present things. Do they need to do this in the researchyou are loo=ing at5

The Statistical Analyses1. %&cluded participants6 ;ere any participants e&cluded "rom the analyses and i" so why5 Did

the researchers Justi"y any e&clusions appropriately5 7or a good discussion on the reasonsto e&clude outliers, see 0s!orne and 0ver!ay 2$B.

2. issing data6 4" participants leave Huestions or items !lan=, we end up with what we callmissing data. There are various di""erent methods o" dealing with missing data (cha"er FCraham, 22B. Did the researchers choose the most appropriate method5

#. *alidity and relia!ility o" dependent varia!les6 Did the researchers provide convincingevidence "or the validity o" each o" the dependent varia!les that they used includingpsychometric scalesB5 4n other words, did each dependent varia!le show signi"icant andappropriately si8ed correlations with the varia!les that it was supposed to !e related toconvergent validityB and, eHually importantly, wea= nonsigni"icant relationships with thevaria!les that it was not supposed to !e related to discriminant validityB5 Also, was theregood evidence o" the internal relia!ility o" the dependent varia!les5 7or e&ample, did eachpsychometric scale have a suita!le "actor structure and-or accepta!le /ron!ach alphacoe""icients L .)B5

$. (u""icient statistical power6 4" researchers "ind a signi"icant e""ect, then, ipso "acto, they musthave had su""icient statistical power to detect this e""ect. /onseHuently, it would !einappropriate to criticise the researchers "or have low statistical power due to small samplesi8e even i" the researchersG sample si8e is smaller than that used in previous research.9owever, i" the researchers "ound null "indings, then this can either !e interpreted asindicating that there is no e""ect present or that an e""ect is present !ut the researchers hadinsu""icient statistical power to detect this e""ect i.e., a Type 44 errorE see /ohen, 1++,12B. 9ence, statistical power is a critical concern when interpreting null "indings. ;heninterpreting a null "inding, consider whether the research contained enough participants todetect the e""ect. oo= !ac= at previous research that has "ound the e""ect in order to seehow many participants were used in that research. eta3analyses and other reviews are

good sources "or this in"ormation. Does the research use signi"icantly "ewer participants thanprevious success"ul research5 4" so, then the null "indings may !e due to a lac= o" statisticalpower. 7aul, %rd"elder, ang, and


6/13


larger i" aB cell si8es are uneHual, !B the AN0*A is larger e.g., 2 & # AN0*AB, or cB thereis more than one dependent varia!le.

. (tatistical assumptions6 Did the researchers meet all o" the assumptions that are associatedwith the particular statistical tests that they used e.g., eHual cell si8es, normal distri!ution,homogeneity o" varianceB.

'. /orrect use o" in"erential statistics6 All statistical techniHues have their limitations. Did theresearchers ta=e these limitations into account. 9ave a loo= at some general introductions tothe techniHues o" e&ploratory "actor analysis 7loyd F ;idaman, 1E Russell, 22B, pathanalysis (tage, Nora, F /arter, 2$B, or structural eHuation modelling and con"irmatory"actor analysis ac/allum F Austin, 2E (chrie!er, (tage, King, Nora, F


7/13


personalities to ma=e them aggressive. . 7or e&ample, a personGs weight mightcorrelate positively with their income, not !ecause there is any relationship !etween these

two varia!les, !ut !ecause they are !oth correlated with a third varia!le6 age. The olderpeople get, the more weight they put on andthe more income they earn


8/13


ou may also !e awarded mar=s i" you ma=e intelligent and speci"ic suggestions "or

"uture research. (o, !ased on your critical analysis o" the research, what would yoursuggestions !e "or a more appropriate piece o" research5 DonGt come up with hal"3!a=edG ideas6%.g., ?7uture research should loo= at and >@. 7ollow your ideas through and !e e&plicit6 %.g.,?7uture research should loo= at and >. 7or e&ample, "uture research should manipulate using the A using the ou will not gain mar=s "or ma=ing these types o"criticisms. >ou should "ocus your comments on the research, not the research article.

2. %thical criticisms6 Unless you are speci"ically instructed to do so, it is usually not appropriateto comment on the ethical aspects o" the research methodology that you are criticising. >oushould assume that the research has !een approved !y a human research ethics committeeand that, there"ore, ethical considerations have already !een dealt with.

#. 4ncomplete criticisms6 >ou need to !e as e&plicit, speci"ic, detailed, and comprehensive aspossi!le when ma=ing your criticisms. 4n general, each critical idea that you put "orwardshould contain6 aB a general introduction e.g., ?4t is possi!le that social desira!ility

+


9/13


in"luenced the results@B, !B a speci"ic ela!oration o" the criticism e.g., ?4n other words,participants may not have !een preJudiced !ecause they perceived this "orm o" !ehaviour to!e socially undesira!le@B, cB citations to theoretical and-or empirical wor= that supports yourassertions e.g., ?(mith and Iones 1+2B "ound that levels o" preJudice were reduced whenparticipants were aware that this "orm o" !ehaviour was the su!Ject o" the researchersGinvestigations@B, dB e&amples o" your criticism that are ta=en "rom the research e.g.,

?Participants were told that the current research was investigating preJudiceG@B, eB re"erenceto any evidence in the target research that supports your claim e.g., ?Poste&perimental"eed!ac= "rom participants did seem to show that they were concerned a!out the impressionthat their responses were ma=ing on others@B, "B a discussion o" the implications o" yourcriticism with respect to the research results and-or conclusions e.g., ?This pro!lem mayhave reduced the level o" preJudice that was "ound@B, gB suggestions "or "uture research!ased on your criticisms e.g., ?7uture research should attempt to conceal "rom participantsthe "act that preJudice is !eing measured@B. >ou will get very "ew mar=s i" you only includeincomplete criticisms in your research. 7or e&ample, you would not get many mar=s "orsimply saying ?4t is possi!le that social desira!ility in"luenced the results@ and leave it at that:

$. /riticisms o" the relia!ility or e""ectiveness o" methodology that produced the predictedresults6 4t is usually only appropriate to criticise the relia!ility or e""ectiveness o" a studyGsmethodology when that methodology has "ailed to produce the predicted results. 4t isinappropriateto criticise the relia!ility or e""ectiveness o" methodology when it has producedthe predicted results. (o, "or e&ample, researchers might use a sel"3esteem scale thatprevious research has "ound to !e e&tremely unrelia!le. 9owever, in their research, theresearchers "ind that the sel"3esteem scale showed signi"icant di""erences in the predicteddirections. 4n this case, it would !e inappropriate to criticise the sel"3esteem scale "or !eingunrelia!le !ecause the "act that it has revealed the predicted results means that it must have!een relia!le enough to do so. 9owever, it would !e appropriate to criticise the relia!ility o"

the researchersG sel"3esteem scale i" it produced une&pected null results. Note that, althoughyou should not criticise the relia!ility or e""ectiveness o" methodology when it has producedthe predicted results, you may still criticise the validityo" that methodology. This type o"criticism may lead to a more general criticism o" the conclusions that the researchersreached. 7or e&ample, you might argue that a sel"3esteem scale was an invalid measure o"sel"3esteem and that it really measured sel"3awareness. 4n this case, you would !e a!le tochallenge the researchersG conclusions and argue that they should signi"icant di""erences insel"3awareness rather than sel"3esteem.

. Random allocation o" participants to conditions 6 0ne o" the most pro!lematic criticisms thatstudents ma=e concerns the random allocation o" participants to conditions. 4n e&perimental

studies, participants should !e randomly assigned to e&perimental conditions. (o, "ore&ample, imagine that some researchers manipulate participantsG sel"3esteem !y giving themeither positive or negative "eed!ac= a!out their per"ormance on an intelligence test. Theythen measure participantsG aggressive !ehaviour in order to determine what e""ectdi""erences in sel"3esteem have on levels o" aggression. 7urther imagine that theresearchers "ind that participants who received negative "eed!ac= showed signi"icantly moreaggressive !ehaviour than people who received positive "eed!ac=. The researchers mightconclude that low sel"3esteem causes aggression. A student might attempt to critici8e thisconclusion !y arguing that ?i" there happened to !e a "ew e&tra aggressive people in thenegative "eed!ac= condition, then this could also e&plain the result@. Admittedly, it is possi!lethat a "ew e&tra aggressive people might have ended up in the negative "eed!ac= condition

!y pure chance alone. 9owever, the statistical tests that the researchers used in order todetermine whether or not there was a signi"icant di""erence !etween the positive andnegative conditions already ta=es this possi!ility into account. 4" the pvalue is less than .,we =now that there is a 1 in 2 chance that the studentGs e&planation or some other


10/13


e&planationB is correct. 9owever, as scientists, we have agreed to con"orm to the conventiono" accepting this 1 in 2 ris= as !eing low enough "or us to e""ectively ignore it. As a criticalanalyst, you should also con"orm to this universal scienti"ic convention and accept that,although it is possi!le that more aggressive people may have ended up in the negative"eed!ac= condition !y chance alone, it is not an accepta!le to criticise the research on this!asis !ecause the chances o" it having happened are relatively low given the statistical

results. Note that this whole argument rests on the assumption that the researchers haverandomly assigned participants to the positive and negative "eed!ac= conditions in theire&periment. The random allocation o" participants to conditions means that we can !erelativelycon"ident that the same types o" people are eHually represented within eachcondition. (o, "or e&ample, as well as having the same proportion o" aggressive andnonaggressive people in each condition, we will pro!a!ly have the same proportion o" menand women in each condition. This proportion may not necessarily !e eHual6 There couldonly !e # men in each condition.


11/13


2B Descri!e the wor= o" ou could repeat some o" this in"ormation whenyou get to the criticisms part, !ut this would !e a relatively ine""icient approach compared to theother two approaches. 9ence, 4 recommend either the second or third approaches.

"seful %ebsitesThe /ritical Thin=ing /ommunity6 http6--www.criticalthin=ing.org-inde&.c"m9ow to Avoid ogical 7allacies in arguments6

http6--www.geocities.com-anatheist21-su!s=epticism"allacies.htmPolson, D., Ng, /., Crant, ., F ah, D. 1+B. Atha!asca UniversityQs Psychology $$

%&perimental PsychologyB tutorial e&plaining nine sources o" threat to internal validity6http6--psych.atha!ascau.ca-html-*alidity-inde&.shtml

Iordan, /. 9., F Manna, . P. 1B. 9ow to read a Journal article in social psychology. 4n R. 7.


12/13


References


13/13


0rne, . 1'2B. 0n the social psychology o" the psychology e&periment6 with particularre"erence to demand characteristics and their implications.#merican Psychologist, 1,))'3)+#.

0s!orne, I. ;., F 0ver!ay, A. 2$B. The power o" outliers and why researchers shouldalways chec= "or themB. Practical #ssessment, Research 0 !valuation, . Retrieved 1)thIune 2 "rom http6--PAR%online.net-getvn.asp5vVFnV'

Paulhus, D. ., F Reid, D.

how to critically analyse psychological research

Documents