brainquake game-based learning and assessment applications

BrainQuake Game-Based Learning and Assessment Applications with Direct Representation of Mathematics:

A Feasibility Study KimkinyonaCullySaraAtienzaRyanBurkeBryanMatlen,Ph.D.

May 2018

WestEd — a nonpartisan, nonprofit research, development, and service agency — works with education and other communities throughout the United States and abroad to promote excellence, achieve equity, and improve learning for children, youth, and adults. WestEd has more than a dozen offices nationwide, from Massachusetts, Vermont and Georgia, to Illinois, Arizona and California, with headquarters in San Francisco. For more information about WestEd, visit WestEd.org; call (415) 565-3000 or, toll-free, (877) 4-WestEd; or write: WestEd / 730 Harrison Street / San Francisco, CA 94107-1242.

© 2018 WestEd. All rights reserved.

BrainQuake Game-Based Learning and Assessment Applications with Direct Representation of Mathematics: A Feasibility Study

OverviewofStudy..................................................................................................................2ResearchQuestions...................................................................................................................................................................3TheStudyIntervention...........................................................................................................................................................3Participants..................................................................................................................................................................................5Instruments..................................................................................................................................................................................9Analysis.......................................................................................................................................................................................13Findings.................................................................................................................................16RQ1:HowfeasibleistheBrainQuakesuiteforclassroomimplementation?..............................................16RQ2:DoestheBrainQuakesuiteshowpromiseforimprovingstudents’mathematicsachievementandstudents’attitudestowardmathematics?...........................................................................................................35RQ3:WhatistheimpactoftheBrainQuakepuzzlesuiteonteachers’pedagogicalcontentknowledgeforteaching?......................................................................................................................................................45ConclusionsandNextSteps...................................................................................................46GameModifications...............................................................................................................................................................47ClassroomImplementationModifications...................................................................................................................48References............................................................................................................................49

Appendix...............................................................................................................................50

BrainQuakeGame-BasedLearningandAssessment:AFeasibilityStudy | page

2

Overview of Study

AspartofthePhaseIISmallBusinessandInnovationResearchgrant(awardedbytheU.S.Institute

ofEducationSciences)BrainQuakedevelopedtwopuzzlesuites:1)theTilespuzzle,whichwas

developedtosupportalgebraicreasoning,and2)theTankspuzzle–whichwasdevelopedto

supportproportionalreasoning.WestEdwascontractedbyBrainQuaketoconductanindependent

evaluationofthenewpuzzles'impactonstudentoutcomes.Becausetheflagshippuzzle–Wuzzit

Trouble–hadalreadybeenshowntoexhibitpromiseforlearninginotherstudies(Killietal.2015;

Pope&Mangram,2015;Matlen,Atienza,&FoxCully,2015),thisstudy'sinterventionfocusedsolely

onexploringtheimpactandfeasibilityoftheTilesandTankspuzzles.Werefertothisintervention

comprisedofthesepuzzlesastheBrainQuakesuite,henceforth.

ToexplorethefeasibilityandusabilityoftheBrainQuakepuzzlesuite,aswellastoexplorethe

promiseoftheproductforachievingtheintendedoutcomes,WestEdconductedarandomizedtrial

inthefallof2017inmiddleschoolclassrooms(5thand6thgrades).Intherandomizedtrial,teachers

wererandomlyassignedtoeitherusetheBrainQuakepuzzlesuiteasasupplementtotheir

mathematicsinstruction(thetreatmentcondition),ortheyweretoconducttheirmathematics

instructionintheirnormalway(referredtoas,businessasusual;thecontrolcondition).This

reportdocumentsthefindingsfromthisstudyandsuggestionsforfutureresearchand

development.

The present work represents an important but early-stage step in understanding how the

BrainQuake puzzle suite is used in classroom contexts. The results are valuable in 1) providing

initialestimatesoftheBrainQuakepuzzlesuite’simpactonstudentoutcomes,2)characterizingits

use in a variety of classroom contexts, and 3) providing information about the feasibility and

fidelityoftheproduct.Thestudyisconductedwitharelativelysmallsampleandtheintervention

waslimitedtoarelativelyshortduration,thus,resultsshouldbeconsideredwithinthecontextof

thebroaderprogramofresearchoftheBrainQuakepuzzlesuite.Nevertheless,thisstudyprovidesa

usefulandnecessarystepinunderstandinghowtheBrainQuakepuzzlesuiteoperatesinclassroom

environments, the results of which can inform future development efforts as well as provide a

foundation for a largerprogramof research.Wepresent the resultswith these considerations in

mind.


3

Research Questions Thestudywasguidedbythefollowingresearchquestions:

1. HowfeasibleistheBrainQuakesuiteforclassroomimplementation?

2. DoesthefullsuiteofBrainQuakeproductsshowpromiseforimprovingstudents'

a. Mathematicsachievement,and

b. Students’attitudestowardmathematics?

c. IstheBrainQuakepuzzlesuitemoreeffectiveforcertainsubgroupsofstudents(e.g.,

lowervs.higherachieveingstudents)?

3. What is the impact of the BrainQuake puzzle suite on teachers' pedagogical content

knowledgeforteaching?

The Study Intervention Theinterventioninthepresentstudyinvolvedtheuseoftwopuzzles:TheTankspuzzleandthe

Tilespuzzle.

TheTilespuzzleaimstoengagestudentsinincreasinglycomplexalgebraicthinking.Players

manipulatearangeoftileswithvaryingfeaturesinordertofillatrayortrayssimultaneouslyby

arrangingthetilesinthetraysandthen“growing”themtofilltheavailablespace.Featuressuchas

UnGrow,seeyourpreviousfailedsolution,andsolveforeitherminimaltileuseorminimalgrow

movesprovideBrainQuake’ssignaturevariedsolutionpathoptions.

TheTankspuzzleisdesignedtofacilitatetheexplorationandunderstandingofincreasingly

complexpercent,decimal,fractionandnon-numericalproportionalrepresentations.Playersmust

allocatespaceinacircular,rectangular,squareorradialdeviceinordertodistributeagiven

numberofinputsintoproportionalmeasuresthatmatchtherequiredoutputdistributions.Unlike

theTilespuzzle,however,theTanks’proportionalreasoningpuzzleissignificantlylimitedwith

respecttoarichandvariedsolutionspace.Thislimitingfactorisduetothenatureofthecontent,


4

buttheinterfaceandpuzzlestestedasslightlymoreengagingthantheTilespuzzleinprior

usabilitywork.

Bothpuzzlesaredrivenbytheirownadaptiveengineandareconnectedtothemonitoringand

loggingsystemthatfeedstheBrainQuakescoringalgorithm.

Thepuzzlesaremeanttobeusedasasupplementtotraditionalinstructioninmathematicsfor

middleschoolstudents.Theyaredesignedtoengagestudentsinmathematicalthinkingwithoutthe

useoftraditionalmathematicalsymbols(e.g.,equations).Instead,thepuzzlesuseconcretephysical

representationstoinstantiatemathematicalconcepts–therepresentationscanbevirtually

manipulated(viatouchscreenorcomputer)andthereforeengagestudentsinmathematical

thinkingwithouttheneedforsymbolicrepresentationsofthemathematics,allowingthemto

exploreproblemsbeyondwhatwouldbetraditionallyconsidered'grade-level'.Teacherscan

supportstudentsinsolvingthepuzzles,butthepuzzlesaredesignedtobeplayedbythestudents.

ToexplorethefeasibilityandusabilityoftheBrainQuakepuzzlesuite,WestEdconducteda

randomizedtrialinthefallof2017inmiddleschoolclassrooms.Inthetrial,teacherswere

randomlyassignedtoeitherusetheBrainQuakepuzzlesuiteasasupplementtotheirmathematics

instruction,ortheyweretoconducttheirmathematicsinstructionintheirnormalway(referredto

as,businessasusual).

TreatmentteachersweretaskedwithusingtheBrainQuakepuzzlesasasupplementtotheir

mathematicsinstruction.Theywerespecificallyaskedtousethepuzzlesforatleastthreedaysa

week,foratleast10minuteseachtime(thoughtheywereallowedtousethepuzzlesmoreifthey

wished).Thewayinwhichteachersimplementedthepuzzles(ashomework,warm-upto

mathematicslesson,asreview,etc.),wasintentionallyleftopen,asthisallowedthestudytoexplore

thevariationinhowthepuzzlesareimplementedinnaturalisticsettings.Thoughtheway

BrainQuakepuzzleswereimplementedwasuptotheteachers'discretion,teacherswereadvisedto

usethepuzzlesinaninterspersedway–alternatingbetweenTilesandTanks–asthisalternating

formatwasbelievedtooptimallysupportstudents'mathematicskills.

Priortothestudy,treatmentteacherswereprovidedashortone-hourorientationofthepuzzle

suite.Teachersattendedtheorientationvirtuallyandthesessionwasrecordedforteacherswho

wereunabletoattendalivesession.Thegoaloftheorientationwastoprovideanoverviewofthe


5

puzzlesuite,demonstratehowthepuzzleswork,andprovideteacherswithanoppoortunitytoask

questions.BrainQuakestaffledtheorientation.

Controlgroupteacherswereinstructedtoconducttheirmathematicslessonsintheirstandardway,

withouttheuseoftheBrainQuakepuzzlesuite.

Allteacherswereprovidedabriefvideoofthestudythattheycouldwatchattheirleisure,which

outlinedtherationaleforthestudyandorientedteacherstothestudytasks.Teacherswere

randomlyassignedtoconditionsafterthisstudyoverview.

Thestudytookplaceoverthecourseofapproximately8weeksduringthefall2017semester–the

interventionspanningapproximately6weeks.Bothgroupsofteacherstookandadministered

surveysatthebeginningandendofthestudy.Measureswerealsocollectedtodocumentthe

implementationoftheBrainQuakepuzzlesduringthecourseofthestudy.Table1documentsthe

tasktimelineoverthelifeofthestudy.Thesurveysaredescribedinmoredetail,below.

Table1.Studytasktimeline.

Week1BeginningofStudy

Weeks2-7DuringStudy

Week8EndofStudy

ActivitiesconductedInClasstostudents

-AchievementAssessment-GearsPuzzle-MathAttitudesSurvey

-Teacherassignsinterspersedpuzzleintervention-ClassroomobservationsinsubsetTreatmentclasses(onesessionwhereBrainQuakegameisbeingused)-Studentfocusgroup(oneperclassroom)

-AchievementAssessment-GearsPuzzle-MathAttitudesSurvey-StudentPost-Survey

Activitiescompletedbyteacher

-TeacherBackgroundSurvey-TeacherMathPCKSurvey

-TeacherWeeklyLogs

-TeacherPost-Survey-TeacherMathPCKSurvey-TeacherInterview

Participants Thirty-four (34)5thand6thgrade teacherswererecruited toparticipate in thestudy. Eachhad

teachingexperiencerangingbetween2and7years,andcurrently teachesatapublicelementary


6

schoolinNorthernCalifornia.Onetothreeteacherstaughtateachschoolparticipatinginthestudy,

with74%ofteachersreportingincorporationoftechnologyintheircurrentclassroompracticesat

least3timesperweek.71%oftheteachersalsoreportedbeingverycomfortableusingtechnology.

Amongallofthestudyschools,researcherscollecteddatafrom4185thgradestudentsand3066th

grade students. Demographic information summarizing the student populations at each

participatingschoolappearsinTable2andinformationontheteachersparticipatingintheteacher

inteviewsaresummarizedinTable3.

Table2.ResearchStudyStudentDemographicSummarybySchool

Sch1

Sch2

Sch3

Sch4

Sch5

Sch6

Sch7

Sch8

Sch9

Sch10

Totalstudentsparticipating 30 91 93 28 33 NA 34 32 32 NA

Socioeconomicallydisadvantaged

NA 27% 32% 57% 73% NA 65% 25% NA NA

Englishlearners(EL) NA 3% 8% 11% 18% NA 18% 25% NA NA

Ethnicity NA NA NA NA

AmericanIndian 2% 0% 7% 0% 0% 31% Asian 1% 0% 0% 0% 0% 44% Black 1% 2% 0% 3% 6% 0% Hispanic 34% 29% 57% 79% 71% 0% Multi-racial 0% 0% 0% 0% 0% 3% White 18% 1% 11% 3% 18% 13% NotAvailable 100% 44% 68% 25% 15% 100% 5% 9% 100% 100%


7

Table2.ResearchStudyStudentDemographicSummarybySchool(Cont.)

Sch11

Sch12

Sch13

Sch14

Sch15

Sch16

Sch17

Sch18

Sch19

Sch20

School size (number ofstudents)

NA 58 32 57 84 NA 34 49 27 27


NA 28% NA 88% 45% NA 88% 67% 96% 93%

Englishlearners(EL) NA 5% NA 40% 14% NA 47% 29% 63% 56%

Ethnicity NA NA NA White 0% 0% 0% 0% 0% 0% 0%AmericanIndian 0% 0% 0% 0% 0% 0% 0%Asian 14% 7% 5% 0% 0% 0% 0%Black 0% 2% 0% 0% 0% 0% 0%Multi-racial 2% 5% 13% 0% 10% 4% 4%Hispanic 12% 81% 39% 97% 76% 96% 89%PacificIslander 0% 0% 1% 0% 0% 0% 0%White 60% 5% 6% 0% 10% 0% 0%NotAvailable 100% 12% 100% 0% 36% 100% 3% 4% 0% 7%

Sch21

Sch22

Sch23

Sch24

Schoolsize(numberofstudents) NA 31 56 30


NA NA NA NA

Englishlearners(EL) NA NA NA NA

Ethnicity NA NA NA NAWhite AmericanIndian Asian Black Hispanic White NotAvailable 100% 100% 100% 100%


8

Table3.Teacherpseudonymsandexperience

Pseudonym GradeTaught

YearsExp.

Teaching

HighestLevel

Education

#Students

LB10 5 7 MA 33LB09 5 7 MA 28

Sal30 5 6 MA 25

Sal28 5 5 BA 34


9

Instruments Thefollowinginstrumentswereusedtoanswerthestudy'sresearchquestions:

Studentpre/postmathematicsachievement

Themathachievementassessmentwasadministeredatbothpre-andpost-testandconsistedof16

multiple-choiceitemsand2open-endedquestionsaimedatassessingstudents’mathematical

ability.TheachievementassessmentwasdevelopedbytheNationalCenterforCognitionand

Mathematics(Davenportetal.2013).Threeoftheproblemsinvolvedunderstandingnumbersense

and15problemsinvolvedsolvingforanunknownquantityusingproportionalreasoning.This

assessmentwaschosenastheBrainQuakepuzzlesuiteinvolvedpracticeintheseskillareas.Items

werederivedfromConnectedMathematicsProject2materialsandstate,nationalandinternational

standardizedtests.Assessmentreliabilityinthepresentstudywas.64atpreand.66atpost.Open-

endeditemswerescoredbytrainedratersusingastandardizedholisticrubric.Researchers

computedweightedkappastomeasureinter-raterreliability,whichrangedfrom0.76to0.93.

Studentswereallowedtousescratchpapertocomputesolutionstothemathproblems.Thesurvey

wasestimatedtotakeapproximately30minutestocomplete.Theitemsfromthemathematics

achievementsurveyareincludedinTable4oftheAppendix.

Studentpre/postWuzzitTroubleassessment

WuzzitTrouble(referredtohenceforthastheGearsPuzzle),BrainQuake'sflagshippuzzlegame,

aimstosupportstudents'numbersense.Inthegame,mathematicalrepresentationsare

instantiatedthroughmechanicaldevicesthatcanbephysicallymanipulated.Theplayersolvesthe

puzzlebyrotatingsmallgearstoturnalargewheeltocollectitemshangingonthewheel.Fora

maximumscore,theitemsmustallbecollectedinthefewestnumberofmoves,followingthesame

sequenceoflogicalstepsrequiredtosolveadvancedalgebraicproblems.

Priorstudieshavesuggestedthatusingthesemoreintuitiverepresentations,studentscanengage

inmathematicalproblemsolvingonproblemstraditionallyconsideredwell-beyondtheirgrade-

level(e.g.,Killietal.2015;Pope&Mangram,2015).However,theGearsPuzzleusedasan

assessmenthasbeenlimited.Thus,inthepresentstudy,itwasusedasapre-andpost-assessment.

Duetosomeissuesextractingandlinkingdatatosubjectidentifiers,theanalysisoftheGearspuzzle

asanassessmentisnotdescribedinthepresentreport.


10

TheGearsPuzzlewasadministeredtothestudentsateachassessmentfor30minutes.Duringthis

time,thestudentscouldprogressthroughtheassessmentwhiletheadaptivealgorithmadvanced

thelevelstomeetthestudent'szoneofproximaldevelopment.Studentswereallowedtousethe

puzzleuntilthe30-minutetimeexpired.Eachstudent'sperformanceperlevelintermsofastar

score(acategoricalratingindicatingthenumberofmoves),timeelapsed,andlevelnumberwere

recorded.

Studentspre/postmathematicalattitudes

Thestudents’mathematicsattitudessurveyconsistedofasubsetofitemsfromthe2003

ProgrammeforInternationalStudentAssessment(PISA)studentquestionnaire(OECD,2005).

Researchersselecteditemsthatwereaimedatmeasuringfourconstructs:Viewstowardsmath,

confidenceinsolvingmathproblems,strategytowardsmath,andmathself-concept.Theinternal

consistencycoefficients(Cronbach’salphas)forsubscalesinthepresentstudyrangedfrom.74–

.87.

Themathematicsattitudesurveyconsistedof4-pointLikertitems,rangingfromStronglyAgreeto

StronglyDisagree.Theonlyexceptiontothisstructurewasquestion5,relatedtoconfidencewith

mathematicscalculations.This4-pointLikertscalequestionrangedfromVeryConfidenttoNotAt

AllConfident.Priorresearchhassuggestedthatparticipantsoverlyrelyonmidpointresponses

whentheyperceivethetopicasunimportant(Krosnick&Schuman,1988).Researcherstherefore

useda4-optionLikert-scaleforallitemssothatstudentswouldnotbetemptedtooverlyrelyona

neutralresponseforatopicinwhichtheymayhavehadlittlepersonalinvestment.Theoverall

structureofthestudentsurveyisshowninTable5oftheAppendix,withquestionsgrouped

accordingtoeachtheme.

Classroomobservations

Researchers developed an observation protocol that examined how teachers interacted with

students as they used the puzzle in class, aswell as how teachers linkedmathematical concepts

fromthegameintoinstruction.Thisprotocolfurtherexaminedwhethertherewereanyinstances

of student persistence, frustration, or methods of completing the suite of games. Trained

researchersconductedtheseclassroomobservationsinasubsetoftreatmentclassrooms.


11

Studentfocusgroups

Researchersaskedgroupsofroughlyfivetosevenstudentsabouttheirperspectivesonusingthe

BrainQuakesuiteofgamesintheclassroom.Thefocusgroupswereheldduringthemiddleofthe

implementationperiodinwhichstudentswereactivelyinteractingwiththeBrainQuakesuiteof

games.Thestudentswerespecificallyaskedtoreflectupontheirattitudesandanyfeedbackfor

improvementregardingthedifferentgamesintheBrainQuakesuite.Studentswerefurthermore

askedtoreflectupontheirabilitytopersistthroughchallengingordifficultpuzzles.Thesesessions

wereaudio-recorded.

Teacherinterviews

TeacherswereintervieweduponthecompletedimplementationoftheBrainQuakesuiteofgames.

TheinterviewprotocolaskedquestionsregardingthefeasibilityoftheBrainQuakesuiteintermsof

classroomuse,howteachersincorporatedtheproductintheclassroom,andtheirperceptionsof

theirstudent'sexperience,engagement,andreactionswheninteractingwiththegames.Trained

researchersconductedtheseinterviewswithinasubsetofthetreatmentteachers.Responsestothe

interviewquestionswereaudio-recorded.

Weeklyteacherlogs

Theteacherlogswereadministeredonceattheendofeachweek.Thelogsaskedteachersto

describethecourseformat,anyresourcesusedtoprepareinstruction,andfortreatmentteachers,

BrainQuakeimplementationdetails,includingtheamountoftimestudentswereassignedthesuite

ofgamesandthefrequencyofassignment.Thelogsalsoconsistedofopen-endedquestionsabout

whetherteachersbelievedtheirstudentslearnedfromBrainQuake,howengagedstudentswerein

playingthegame,andanytechissuesexperienced.Teacherscompletedthesurveyonline,andeach

logtookapproximately5minutestocomplete.

Teacherpre/postpedagogicalcontentknowledge

PedagogicalContentKnowledge(PCK)istheknowledgeofhowtoteachanacademicsubjectandis

engagedduringtaskssuchasplanningalesson,respondingtostudentquestionsinclass,and

gradingstudentcoursework.TheMathematicsKnowledgeforTeaching(MKT)measurewasused


12

tomeasureteacherpedagogicalcontentknowledge(Hill,Ball,&Schilling,2008).Teachers

completedportionsoftheassessmentdesignedtotestknowledgerelatedtoproportionalreasoning

ingrades4-8(LearningMathematicsforTeaching,2007).ThisstudyusedtheLMTtoassess

whetherteachers’PCKchangedacrosstheintervention.TheLMThasdemonstratedgood

reliability,rangingfrom0.71to0.84.

Teacherbackgroundquestionnaire

Theteacherbackgroundsurveymeasuredpriorexperienceinteaching,educationalbackground,

andexperienceusingtechnologytoteachmath.Thissurveyconsistedof13items.Thefirst6were

multiplechoicedemographicandeducationalbackgroundquestions.Otheritemsaskedteachers

askedabouttheiryearsofteachingexperience,studentgradelevelandnumberofstudentsintheir

targetclassroom.Anotheritemconsistedofa5-optionLikert-stylequestionrelatedtoproficiency

withoperatingsystemsandofficetools,andrangedfrom“NoExperience”to“Advanced”.Thelast

questionrelatedtotechnologyproficiencyandthefrequencyandcomfortofusingtechnologyinthe

classroom.This5-optionLikertscalerangedfromStronglyDisagreetoStronglyAgreeand

containedaNeitherAgree/Disagreeoption.

StudentandTeacherPost-SurveyoftheBrainQuakesuite

Thestudentpost-surveyconsistedof39itemswhichassessedstudents’opinionsregarding1)

engagement,2)usabilityofBrainQuakegames,3)feasibilityofcontinueduseofBrainQuake(e.g.,in

class,asalearningtool,etc.),4)perceivedabilityofBrainQuaketoimprovestudentlearningand

motivationand5)opinionsabouttheirexperiencewithpersistence,motivation,resilienceand

encouragementwithineachofthethreeBrainQuakepuzzletypes(Tanks,TilesandGears).Students

wereaskedtheextenttowhichtheyagreedordisagreedwithquestionstatements.Questions

relatedtousability,feasibility,improvedlearning,motivationandindividualpuzzlepersistenceand

resiliencewereona5-optionLikertscale,rangingfromStronglyDisagreetoStronglyAgreeand

containingaNotSureresponse.Overallgameperceptionquestionswerebasedona5-optionLikert

scale,rangingfromVeryHardtoVeryEasyandcontainedaNotTooHardorEasyresponse.

Individualpuzzletypequestionsrelatedtomotivationtotryagainwereona3-optionLikertscale,

rangingfromNotVeryMuchtoALotandcontainedaSomewhatoption.Individualpuzzletype

questionsrelatedtoencouragementwereprovidedas4-optionmultiplechoiceitems,rangingfrom

VeryDiscouragedtoMotivatedToFigureItOutandcontainedALittleDiscouragedandIWantedto

TryAgainoptions.Researchersagainuseda4-optionmultiplechoicescaleforallitemsrelatedto


13

encouragement(items27,33and39)sothatstudentswouldnotbetemptedtooverlyrelyona

neutralresponse.TheoverallstructureofthestudentsurveyisshowninTable6oftheAppendix,

withquestionsgroupedaccordingtotheme.

Thefeasibilitystudyteacherpost-surveysoughttocaptureteachers’perceptionsaboutBrainQuake

games.Likethestudentsurvey,theteachersurveyaskedteacherstoagreeordisagreewithitem

statementsconcerningthe1)usabilityofBrainQuakegames,2)feasibilityofcontinueduseof

BrainQuake(e.g.,inclass,asalearningtool,etc.),3)perceivedabilityofBrainQuaketoimprove

studentlearningandmotivationand4)opinionsabouttheirstudents'experiencewithtwoofthe

BrainQuakepuzzletypes(Tanksvs.Tiles).Teacherswereasked15questionsthatcorrespondedto

oneof10categories(seeTable7oftheAppendix).Forthefirst8questions,teacherswereprovided

5possiblechoices:StronglyAgree,Agree,NotSure,Disagree,andStronglyDisagree.Forthe7

remainingquestions(items8-15),teacherswereaskedtosupplytheirownanswerinanopen-

endedresponse.

Analysis Feasibilitydata

WestEdresearchersconductedfourfocusgroups,withapproximately5to7studentsineachgroup,

fourclassroomobservations,andthreeteacherinterviews.Tounderstandclassroom

implementationoftheBrainQuakegames,theirfeasibility,andtheuserexperienceofparticipants,

WestEdresearchersreviewedobservationanddebriefnotes,focusgrouptranscripts,interview

transcripts,andclassroomobservationnotestoidentifymajorthemesandpatterns.Guidedby

MilesandHuberman's(2014)qualitativedataanalysisprocess,researchersdevelopedcodesbased

onresearchquestionsandemergentthemesandthenutilizedthenVivosoftwaretoassistwiththe

analysisofthedata.

Studentoutcomedata

Teacherandstudentpost-surveyresponsesweresummarizedandgraphedforcomparison

betweengroupsandacrosspuzzletypes.Students’responsestothemathematicalassessmentpre-

andpost-testswerescoredandthegainscomputedbysubtractingthestudents’post-testscore

fromthepre-testscore.Tostatisticallyexaminestudentgains,pairedsamplet-testswere

conductedonstudents’pre-andpost-testscores,alongwithstandarddeviationsandCohen’sd


14

effectsizes.Thepairedt-testandeffectsizesallowedforassessment,onaverage,whethergains

differedfromzero.

AdditionalquantitativeanalyseswereconductedtodeterminetheimpactoftheBrainQuake

puzzlesonstudents'mathematicsachievement(asmeasuredbytheachievementassessment)and

students'attitudestowardsmathematics(asmeasuredbytheattitudesurvey).Toexaminethe

impactoftheBrainQuakepuzzlesonstudentoutcomes(achievementandattitudes),researchers

regressedtheoutcome(eitherachievementpost-testscoreortheattitudespost-score)onthe

conditionassignment(treatmentstatus),controllingforotherstudent-levelcharacteristics.These

regressionanalyseswereconductedusingtwo-level,hierarchicallinearmodels,witharandom

effectforteacherstoaccountforthenestedstructureofthedesign-studentswithinteachers.

Impactmodelingwasconductedwithallstudentswithvalidpre-andpost-testdata(n=515inthe

achievementanalysisandn=415intheattitudeanalysis).Foreachanalysis,threemodelswere

explored:aconditionmodel,acovariatemodel,andamoderatormodel.Becausethestudywas

underpowered,wewereconservativewithincludinglevel-twovariables,whichtakeawaydegrees

offreedomandthereforecanfurtherreducestatisticalpower.However,intheachievementsample,

weincludedgradeasaleveltwocovariate,asdescriptiveexplorationrevealedtherewereslightly

morestudentsin6thgradeinthecontrolgroup.Inallmodels,students’pre-testscore(onthe

achievementorattitudessurvey)wasincludedasalevel-1covariate.

ConditionalModel.Theconditionalmodel(model1)includedfixedeffectsintheconditionmodel

includedtheconditionvariableandstudents’pre-testscore.TheConditionModeltookthe

followingform:

γij=β00+β01TXj+β10Preij+ζ0J+ϵI0

Whereγijisthepost-testscoreforthei-thstudentofthej-thteacher,TXjisadichotomousvariable

indicatingassignmenttotreatmentPrerepresentsstudents'pre-testscore,andβ00isthegrand

meanofstudentscores.ϵI0isarandomlevel-1errortermandζ0Jistheteacherrandomeffect,the

variancecomponentofwhichcapturesthenestingofstudentswithininstructors.Importantly,the

maineffectoftreatmentassignmentiscapturedbyβ01.


15

CovariateModel.Inadditiontotheconditionandpre-testtermsintheconditionalmodel,the

covariatemodel(model2)includedstudent-levelfixedeffectsofstudents’gender(Female),

students’socioeconomicdisadvantagedstatus(SDS;providedbytheschooldistrict),students’

Englishlanguagelearnerstatus(ELL),andthestudents’scoresontheSmarter-BalancedMath

sectionfromthepreviousschoolyear(SBMath;thiswasnotincludedintheattitudeanalysisasit

wasnothighlycorrelatedwithscores).Covariateswereincludedbasedontheoreticalimportance

andpriorresearch(theyareknowntocorrelatewithSTEMoutcomes),correlationalstructure,and

therepresentativenesswithinthedataset(e.g.,specialeducationstatuswasnotincludedasa

covariatebecausetheywerenotstronglyrepresentedinthepresentsample).

ModeratorModel.Themoderatormodel(model3)includedallvariablesinthecovariatemodel,but

additionallyincludedcross-levelinteractionsbetweenthosevariablesandtreatmentstatus.This

lattermodelallowedustoexploremoderatoreffects-whethertheBrainQuakepuzzlesare

differentiallyimpactfulforsubgroupsofstudents(e.g.,malesvs.females,SDSvs.non-SDSstudents,

etc.).


16

Findings

RQ 1: How feasible is the BrainQuake suite for classroom implementation? ClassroomImplementation

Analysisofthequalitativedataprovidedinsightintothemethodsinwhichteachersimplemented

theBrainQuakegamesintheirclassroom.Themostprominentwaysinwhichteachers

implementedBrainQuakeincluded:usingthesuiteofgamesinconjunctionwiththeirmathlesson,

suchasawarm-uporcooldownactivity,andusingthetoolasanoptionalormotivationalactivity

(teacherswouldusetheBrainQuakesuiteofgamesasamotivationtofinishtheirclasswork).In

termsofusingBrainQuaketocomplimentthedesignatedmathperiod,oneteachercommented:

"[BrainQuakewas]usedoftenasabeginningofthedayactivitywhentheyfirstcameorwhendone

withtheirmustdo’sandusedasamaydo."

Whileteacherswouldfrequentlyusethesuiteofgamesinthemethodsdescribedabove,mostof

theteachersdidnothaveaconsistentwayofimplementingitonaweek-to-weekbasis.Teachers

demonstratedflexibilityinthewaytheyimplementedthesuite.Teacherssharedthattheycould

useitasawarmuponeweekandswitchthingsupthenextweekanduseitasamotivational

activity.Oneteacherspoketothisflexibility,"IusedBrainQuakeinthreedifferentways.Onewas

whenstudentsfinishedworkearly,theycanuseitforalittlebit.Othertimes,therewasdesignated

timeduringtheweek.Additionally,itwasusediftherewasfreetimeduringtheweek."

Furthermore,alargemajorityofteachersutilizedtheBrainQuakegamesasindividualstudent

work.Inparticularsituations,teachersusedBrainQuakeasgroup-workandstudentsappearedto

enjoyopportunitiestoengagewiththepuzzleswiththeirpeers.Theteacherexplainedthatthis

enjoymentwasfurthercomplementedwithpositivestudentattitudes,"[students]seemedmore

positiveabouttheirabilitytoprogressbecausetheyworkedtogether."Thismayspeaktothe

BrainQuakepuzzlesbeingmoredifficultforcertainclasses,andthatthetoolwasbeneficialingroup

settingsbecausethestudentscouldcollectivelyworktogethertofindsolutions.


17

DatafromtheweeklylogsexaminedthefrequencythattheBrainQuakesuiteofgameswas

implementedintheclassroomsetting.Baseduponthelogs,itwasobservedthatteachersinthe

treatmentconditionusedtheBrainQuakeappforroughlytwodaysaweekthroughoutthecourse

oftheimplementationperiod.Thiswasconsistentthroughoutmostthestudy(aslightdecrease

wasnoticedtowardsthefinalweekofthestudy,butthisislikelyduetoimplementationof

assessmentsforthestudy).

Figure 1: Treatment Teachers Used the BrainQuake app on Average 2 days per week

Teachers furthermore provided insight into the amount of time (in minutes) that they spent

preparing to use BrainQuake in the classroom. As detailed by the figure below, teachers spent

roughly 20minutes preparing for the implementation of the BrainQuake suite of games (with a

large increase to roughly 30 minutes in the second week of implementation). The teacher's 20

minutesofpreparationallowedthemtofitthesuiteofgamesintonearly450minutesofmathtime

intheclassroomeachweek.

Figure 2: Treatment Teachers Spent on Average 20 minutes in prep to use the BrainQuake games


18

Figure 3: Treatment Classrooms Spent on Average 450 minutes in math time each week

Per theweekly logs, teacherswithin the control condition reported the educational technology tools

that they used eachweek. Themost commonly utilized educational technology tools in the control

classroomswere Scootpad, Prodigy, ImagineMath, KhanAcademy, and SplashMath.While students

were using these educational tools, teachers reported that by in large, the class was overall very

engaged.Teachersreflectedthatamajorityoftheirstudentswereveryengagedwhenusingtechnology

intheclassroomandtypicallyonlyafewstudentsgotdistracted:"Moststudentsenjoyworkingonthe

computers and are actively engaged in themath practice and review. A handful of students are less

engagedandhaveahardtimestayingfocusedonwhattheyaresupposedtobedoing."

Figure 4: Control Teachers Utilized ‘Prodigy’ and ‘Imagine Math’ ed tech most commonly in their classrooms


19

Table8.WeeklyMathConcepts

Week Most Reported Math Concepts 15-Sep Ratios 22-Sep Decimal/Fraction Division 29-Sep Multiplying Multi-Digits 6-Oct Multiplication and Division 13-Oct Fractions (Add, Subtract, Multiply) 20-Oct Fractions (Add, Subtract, Multiply) 27-Oct Multiply and Divide Decimals 3-Nov Rates 10-Nov Ratios as Fractions/Percents 17-Nov Distance between Points 1-Dec Ratios 8-Dec Ratios

Duringtheimplementationperiod,bothtreatmentandcontrolteachersreportedintheweeklylogs

thespecificmathconceptsthattheirstudentswereworkingoneachweek.Table8abovedescribes

whichconceptwasmostreportedwithineachweeklylog.Duringtheimplementationperiodofthe

BrainQuakesuiteofgames,themostcommonconceptsthatwerecoveredinclassincludedworking

withratiosandmanipulatingfractionswithmultiplication,division,subtraction,andaddition.

AppropriatenessoftheMathContent

Mostteachersfeltthemathcontentwasappropriatefortheirstudents.However,thereweresome

differencesintheperceptionsof5thand6thgradeteachers.Some5thgradeteachersfeltthatthe

BrainQuakegamesmayhavebeentoochallengingfortheirstudents–oneexplained,

"Itwouldbeniceifthetopicscoveredinyourgameswouldbemorealignedwithwhatwe

coverin5thgrade.Thereisalotofworkwithpercentages,decimalsandfractions.The

beginningof5thgrademaynotbethetimetousethisgame...perhapsmoretowardtheend

oftheyear."

Additionally,anotherteacherwasabletouchuponthedeviationbetween5thgradecontent

knowledgeandtheabstractnatureoftheBrainQuakesuite.

"TheGearspuzzlewouldbeperfectformentalmathcalculation.Proportionsaretoo

difficultformost5thgradersaswehavenotevenreachedourunitonfractionsyet.Even


20

thoughtheyhavehadfractionworkinpreviousgrades,theyarestillstrugglingwith

understandingfractionalpartsonamoreabstractlevel."

Thisteacher'squotehighlightsthatstudentsmayhavesomeexperiencewiththecontentona

surfacelevel,buttheymaynotbeasequippedtoabstractlyinteractwiththesemathconcepts.

Overall,theBrainQuakesuitewasoutsideoftheabilityof5thgradestudents,butitisimportantto

note,asevidencedbythesequotes,thattheseteacherswerestillabletoforeseethepromisethat

thisplatformmaypossesswhenpresentedattheappropriatetimeintheircurriculum.Both

teachersemphasizethattheBrainQuakesuitecouldbeavaluabletoolwhenpairedwithaligned

content,andoneteacherfurthernotesthattheGearsPuzzle,whichtheyperceivetobeatan

appropriatelevel,isabeneficialresource.

Ontheotherhand,6thgradeteachersfeltthatthecontentoftheBrainQuakepuzzleswere

appropriatefortheirstudents.Theseteachersreflectedthattheirstudentswerecomfortablewith

themathconceptsexpressedintheBrainQuakepuzzles.One6thgradeteacherexplained,"Iliked

howitapproachedteachingmathconceptsinadifferentway,suchasthefractiongames.Kids

seemedtohavefunwhiletheyplayed…Thepuzzlesseemedtochallengemoststudentsbutthe

conceptswerefamiliar."Another6thgradeteacherhadasimilarperspectiveandreflectedthat,"I

likedthat[BrainQuake]feltlikeagameforthestudentsanditprovidedanengaging,visual

representationofthemathematics.[Itprovidesin]analternativefashionwhatwenormally

presenttostudents."Thefeedbackprovidedby6thgradeteachersspeakstotheenjoymentthat

bothstudentsandteachershadwheninteractingwiththeBrainQuakesuite.

TheBrainQuakesuitewasperceivedby6thgradeteachersasanappropriatechallengeandtoolfor

studentsbecauseitalignedwellwithconceptsinwhichthestudentswerealreadyfamiliar.

Additionally,itprovidedanavenueforstudentstolearnandinteractwithmathematicsfroma

differentperspectivethanthetraditionalclassroomapproach.Oneteacherremarkeduponthis

opportunity,byhighlightingthebenefitsofaccessingdifferentmethodsofteachingconcepts

throughtheintegrationoftheBrainQuakesuite.

"Withmymathinstruction,Itrydifferentwaystoaccessthemath;Thisprogramhelped

studentslearndifferentwaystounderstandratiosandpercentagesfrommynormal

instruction;bydoingsotheyareabletostrengthentheirunderstandingoftheconcepts;I


21

alwaystrytoteachwithdifferentformatstosupportlearning;Iusuallyteach3-5different

formats;Itallowsstudentstoseethesameconceptsindifferentways."

ThroughtheimplementationoftheBrainQuakesuite,thisteacherwasabletousethegameasa

tooltosuccessfullyintroduceanalreadyfamiliarcontentareainanewandabstractmanner,

strengtheningtheirworkingknowledgeofthesubject.Ultimately,thedifferencesobserved

betweentheexperiencesof5thand6thgradeclassrooms,theBrainQuakesuitewasmost

successfullyimplemented--fromtheperspectiveoftheteachers--intheclassroomsinwhich

studentswerealreadyfamiliarwiththecontent.

ClassroomFeasibility

Theteachertrainingwasholisticallyviewedas"agoodpreviewofwhattoexpect"anditwas

fundamentalforprovidinginformationaboutthebasicsofimplementingtheBrainQuakesuiteof

games.Reflectinguponthistrainingfurther,teachersfeltthatitcouldbeadditionallybeneficialif

thereweresimilarvideosavailablefortheteachertointroducethegametothestudents.Someof

theteachersfurtheridentifiedthatincludingdetailsonhoweachofthepuzzlescorrespondsto

statestandardswouldhelpthemincorporatethetoolintotheirdailycurriculum.

ThroughouttheimplementationoftheBrainQuakesuiteofgames,therewasarangeinthe

availabletechnologyacrossclassrooms.Byinlarge,theplatformwasimplementedusing

ChromebooksorPClaptops,butthereweresomeoutlyingclassroomsthataccessedtheplatform

throughIBMdesktops(threeclassrooms)inacomputerlaboriPads(twoclassrooms).Theextent

ofaccessibletechnologyfurthercorrespondedtotheteacher'sabilitytosetupandusethe

BrainQuakesuiteofgames.

Overall,amajorityofteachersreportedthattheBrainQuakesuitewasdifficulttosetupandusein

theclassroomduetothereoccurringtechnologychallenges.Oneteacherreflectedupontheir

experienceusingtheBrainQuakesuiteintheirclassroom,commenting,"Notbeingabletologin

easilyorthepuzzlesnotloadingquicklyenoughmadeusingthesepuzzlesfrustratingattimes.The

logisticsofgettingtheChromebooksoutofthestorageroomandcoordinatingusetimewiththe

otherteacherwasabitofahassle."Anadditionalteacherreportedontheantiquatedtechnology

availabletohisstudents,"Ourhardwarewasveryold.Wehavedesktops,notlaptops.Theywere

theoldIBMclones,pizzaboxstyledesktops.Clearingthecachedidnothelpfixproblemsonthese


22

machines."Theseglimpsesintotheteacher'saccesstotechnologyandsubsequentdifficulties

implementingtheplatformhighlightanimportanttrendrepresentedbyamajorityofteachers.

Classroomswithoutthelatesttechnologyoraccesstoimmediatetechnicalsupport,faced

challengesinaccessingandnavigatingthroughtheBrainQuakesuiteofgames.Asthisreflection

fromateachershowcased,thesetechnologicalissueswereoftentimescompoundedwithteachers

havingtologisticallyreorganizetimesinwhichtheycouldaccesstheschool'slimitedtechnology,

makingitmorechallengingtosetupandusetheBrainQuakesuiteofgames.

Itisimportanttonotethatahandfulofteachersthathadaccesstonewertechnologywereableto

integratetheBrainQuakesuitewithease.Oneteachercommented,"No[wedidnotexperienceany

technologicaldifficulties],ourschooldistrictisallcaughtupwithtechnology.Wehaveawholetech

teamandourinternetisreallygood.Wecouldplaywheneverwewanted."Onlyasmallsegmentof

theparticipatingteachersreportedthattheyexperiencedminimalissueswithaccessandset-up

anditseeminglyvariedasaresultoftheclass'savailabilitytotechnologyandimmediatesupport.

Overall,issueswithtechnologywerepersistentthroughoutthecourseoftheimplementation

period.Withinthetreatmentconditionoflogsthatprovidedaresponse,75%ofteachers

experiencedsomesortoftechnicaldifficultywhileimplementingthegameintheirclassroom

(62/(62+21)).Thisisincontrastto9%ofteacherswithinthecontrolconditionthatexperienced

notechnicaldifficultieswhenusingdifferenttechnologytoolsintheirclassroom(5/(5+49)).

Figure 5: 72% of Treatment Teachers Experienced Technical Difficulties While Implementing BrainQuake Resources


23

Figure 6: 72% of Control Teachers Experienced No Technical Difficulties While Implementing Technology Tools

Theoccurrencesoftechnological issues(asshownintheFigure7below)wererecordedoverthe

lengthofthestudyandwereapparenteveryweekoftheimplementation.Teachersprimarilyran

into difficulties accessing theWorldMap or Gears Puzzle game series, logging into the platform,

receiving app error messages, instances of the school district blocking access, and experiencing

gamefreezesorcrashesduringstudentplay.

Figure 7: The Most Commonly Experienced Technological Issues in Treatment Classrooms were App Error messages and Logging Into the Platform


24

Thecollectionofdifficultieswithintheclassroomwastroublingforteachersandasaresultmany

didnotfeelcomfortableimplementingtheBrainQuakegamesuitemostweeks.Someteachers

addedthatitmayhavetakenawayfromthestudent'senjoymentoftheproduct;"Twostudents

couldn'tlogin.Wegotthatresolvedbychangingtheirlogin.StudentsthatreachedtheWorldMap

reallyenjoyedit,butmostofmystudentswerestuckontheGears.Whenthecomputerwould

freeze,itwouldstartthemagain.Theywereveryfrustratedandreadytomoveon."Eventhough

studentswereperiodicallyconfrontedwithissueswhennavigatingthroughthesuiteofgames,this

quotealsotouchesuponthepositiveviewsfromstudentsthatwereabletoaccessthepuzzles.

WhentheBrainQuakesuitewasworking,theteacherobservedpositivestudentfeedbacktoward

theirexperienceinteractingwiththepuzzles.

StudentEngagement

StudentsweregenerallyabletosuccessfullyusetheBrainQuakesuiteofgames.Figure8shows

56%ofstudentsfoundtheproducteasytouse,indicatingthattheyagreedthat"whenusing

BrainQuakepuzzles,theyknewwhattodowithoutanyonetellingthem".Themajority(60%)of

post-surveystudentrespondentsthoughtthatmostkidswouldlearnhowtouseBrainQuake

puzzleseasily.However,somestudentsexperiencedchallengesduringgameplay:41%ofstudent

post-surveyrespondentsthoughtthatplayingBrainQuakepuzzleswasconfusing.


25

Figure 8: 56% of Student Post-Survey Respondents Found BrainQuake Easy to Use

Figure9indicatesthatwhilemoststudents(55%)foundexistingtutorialshelpful,58%reported

thatBrainQuakeshouldprovidemorehintsordirections.Overall,55%ofstudentrespondents

agreedthatBrainQuakepuzzlesfitwellintheirclassroomactivities.


26

Figure 9: 55% of Student Post-Survey Respondents Agreed that BrainQuake Puzzles Fit Well in Their Classroom

Researchers further observed student engagement strengths and challenges in the classroom.

When students could successfully interact with the BrainQuake puzzles, theywere engaged and

focused on working through and solving the different puzzles. Students commented that they

thoughttheBrainQuakesuitewashighlyengagingandenjoyable.Thiswasconsistentevenincases

wherethepuzzleswerebecominghardertosolve."Whenstudentswouldsolveapuzzletheywould

throw their armsup in excitement. Theywould also exclaim 'ahhhhh' [in a slightly disappointed

way]when theywouldget thequestionwrong."Students’persistenceat solvingpuzzles, even in

thefaceoffailure,continuestobeaconsistentcharacteristicoftheBrainQuakepuzzles.

Teachersfurtherobservedthatstudentswereveryexcitedtointeractwiththegames,particularly

at thebeginningof the implementation.One teachers’ studentsenjoyed theirexperienceworking

through theBrainQuake suite somuch that itbecamea tool for this teacher tomotivate them to

finishtheirotherclasswork,"itmotivatedthemtogettheirworkdone,becausetheywantedtoplay

thegame."Teachersadditionallycommentedonnotonlytheenjoymentsharedbytheirclass,but

theeaseinwhichtheywereabletofigureouthowtointeractwiththenewpuzzles."Onceallthe


27

loginissuesweresolved,thestudentswerehappytomovetotheTilesandTankspuzzles.Students

quicklyfiguredouthowtomanipulatebothpuzzlesandsolvedthefirstfewlevels."

Other teachers experienced this same level of enjoyment from their students, but as time

progressed throughout the study, there was a decrease in the level of student engagement that

teachersreported(asseeninFigure10).

Figure 10: Treatment Students Experienced a Larger Decrease in Engagement vs. Control Students After 7 Weeks

As depicted by Figure 10, students in the treatment condition experienced a larger decrease in

engagement as perceived by the teacher than the control group. Teachers within the control

condition experienced a steady rate of student engagementwhile their students interactedwith

differenttechnologyresources.Thedeclineexperiencedbythetreatmentconditioncouldbedueto

the increase in technical difficulties toward the end of the implementation period (including

students being unable to progress forward through the puzzles). The technical issues students

experienced when playing the games, was a point of frustration. "The students were on task;

however,theywerefrustrated/anxious/disappointedwhentheyranintosoftwareproblems."

OverallStudentandTeacherPerceptionsofBrainQuakeSuite

Studentand teacher respondentsdifferedon theirperceptionsof studentdifficulty regarding the

BrainQuakesuiteofgames.Figure11show40%ofstudentpost-surveyrespondents felt that the

gameswere'veryeasy'or'alittleeasy'. 42%hada“justright”response,describingthegamesas

'not too hard or easy'. While 18% described the games as 'a little hard', 0% of students

characterizedthegamesas'veryhard'.


28

Figure 11: 40% of Student Post-Survey Respondents felt that the BrainQuake Suite of Games Were ‘Very Easy’ or ‘A Little Easy’

In contrast, teacher post-survey responses indicated that teachers thought the games were

challenging for their students (see Figure12).No teachers responded that the gameswere 'very

easy'or 'a littleeasy' for thestudents. 20%described thegamesas 'not toohardoreasy'. 67%

describedthegamesas'alittlehard'and13%ofteacherscharacterizedthegamesas'veryhard'for

theirstudents.


29

Figure 12: 67% of Teacher Post-Survey Respondents described the BrainQuake Suite as ‘A Little Hard’ for Their Students

This contrast between student and teacher perceptions may be indicative of the two groups’

perceptionsofgameplay.Theteacherswereevaluatingthestudents’abilitytoperformthemath.

Thestudentswerelessfocusedonthemechanicsofmathematicsandinsteadwereengagedbythe

gamification of the platform. The students may have been immersed in the excitement of the

contest,andlessawareofthestruggletolearnthecorrespondingmathematics.

Figure 13 shows that 65% of the students reported enjoying the BrainQuake games, and 64%

indicatedthatthepuzzleswerefun.Manystudents(50%)feltthatotherstudentstheiragewould

liketheBrainQuakepuzzlesand55%didnotexperienceboredomwhileplayingthegames.


30

Figure 13: 65% of Student Post-Survey Respondents Reported Enjoying the BrainQuake Suite of Games

This perspective may be confirmed by the teacher’s perception that the students found the

engagement withmathematics in the BrainQuake games to be pleasant. Figure 14 displays that

73%of the teachersreported that they thought thestudentsenjoyed theBrainQuakegames,and

67% indicated that they thought the studentshad funusingBrainQuakepuzzles. Many teachers

(47%)reportedthatthestudentswantedtoplaytheBrainQuakegamesevenwithouttheteacher

telling them to; 33% reported that student were not bored, and 73% indicated students were

engagedwhileplayingthepuzzles.


31

Figure 14: 73% of Teacher Post-Survey Respondents Reported That Their Students Enjoyed the BrainQuake Suite of Games

When asked to contrast the three game types specifically, student post-survey respondents

indicatedanequaldesirability foreachgame,withonlyaslightpreferenceexpressed towardthe

Gearspuzzle. Whenaskedwouldtheykeepattemptingthepuzzleuntiltheygotitright,students

expressedagreementevenlyamongthethreepuzzles,Tanks,TilesandGears(80%,75%,and77%,

respectively). Positive response rates remained high when students were asked if they were

motivedtotryadifferentwayoriftheyfailedatfirst,didtheyfeeltherewerechancestoimprove

orlearn(68%,68%,71%,and63%,64%,65%,respectively).


32

Figure 15: 75%- 80% of Student Post-Survey Respondents Would Keep Attempting Each Type of BrainQuake Puzzle Until They Got It Right

Inthequalitativesectionofthesurvey,teacherswereaskedaboutobserveddifferencesinstudents’

experiences(e.g.,enjoyment,engagement,challenge,etc.).WhenplayingtheTilesgamecompared

totheTanksgame,themajorityofteacherpost-surveyrespondents(60%)indicatednodifference

instudentexperiencesbetweenthetwogames.20%ofteacherrespondentsindicatedthatthey

thoughtstudentsenjoyedorhadlesstechnicaldifficultieswiththeTanksgame.Oneteacherfelt,

"[Students]usedmorestrategywhenusingtheTilesgame.TheTanksgameallowedformoretrial

anderrorandwasdifficulttomaketheconnectiontothe[mathematical]values".13%ofteachers

hadnoresponseregardingdifferencesbetweenstudents'experienceswiththeTilesandTanks

games.

StudentandTeacherPerceptionsoftheTilesGame

StudentsoverallenjoyedtheTilesgame.Fromthediscussionsinthestudentfocusgroups,students

didagreethatthetilesgamewasdifficult,“Itgetssuperhard.Ithoughtitwaskindofhard,kindof

difficult.Itwashardtounderstandhowtoputthetilesright”buttheyalsofeltthattheTilesgame


33

wasanenjoyablechallenge.“Ilikedallofit.”“Itgetshardandyoulikestrugglewithit…thatwasa

goodthing.”Therewasacommonthemethatthesegamespresentedachallengingmathexperience

forstudents.

TeachershadfeedbackontheTilesgame,butoverallthoughtitwasuseful.Inateacherinterview

oneparticipantremarked,"Theycouldunderstanditmore.Theycouldvisuallyseethebarmoveor

increaseonthescreen."Thisteacherappreciatedthevisualaspectsofthegameandfoundthis

featuretobeanintegralmechanismforstudentunderstanding.Incontrast,adifferentteacher

proposedthatsomestudentsmayneedmorescaffoldingwhensolvingtheTilesgames.Thisteacher

respondedsaying:

"Tilesdidn’tgiveyouanyinformationonhowtodoit;notintuitivetousetiles;some

studentstooklongertounderstandhowtoplayandothersfigureditoutrightaway.Just

likemathconcepts,somelookedattheinstructions;otherswerenotabletoaccessthe

instructions.[Itwasa]mixedbag.Somethoughtitwaseasyandsomethoughtitwashard;

[Itwasthatwaywith]everystudent.Somethoughtitwashard,somethoughtitwaseasy."

Likelearningmathconcepts,studentswerelearninghowtoapplymathematicstothissuiteof

puzzleswhilelearninghowthepuzzlesthemselvesoperate.Thisteachertoucheduponthe

complexityofmeetingstudentsattheir"zoneofproximaldevelopment"andhowtheTilespuzzle

maybetoodifficultinitiallytomeetallstudent'sneeds.

StudentandTeacherPerceptionsoftheGearsGame

Bymajority,observedstudentswerethemostengagedandexpressedthemostenjoymentwhen

playingtheGearspuzzle.Duringaclassroomobservation,studentswerenotablyexcitedtospeak

aboutthedifferentcharacters(Wuzzits)thatwereintheGearspuzzle.Thefeatureofhavingmore

ofanobjective(solvingthepuzzletofreetheWuzzit)wassomethingthatgotthestudents’

attention.

Furthermore,studentsspoketothefactthatwithinthispuzzletheywereabletosolvethepuzzles

indifferentways.Onestudentexpressedthattheylikehavingnumerouswaystosolveananswer

because,"…thatishowwearelearningandwehavetolearndifferentwaysofmath,ofhowtodoit.

Causethere’sdifferentwaystogetdifferentanswers,well,thesameanswer."TheGearspuzzle

allowedstudentstocreativelynavigatethroughthelevelswhichwasseenasapositiveexperience


34

formanystudents.Thisstudentwasadditionallyabletomaketheconnectionafterplayingthe

Gearspuzzlesthattheyarelearningdifferentwaystodothesamemathwhentheycansolvethe

puzzleindifferentways.

TeachersagreedthatthestudentsenjoyedtheGearspuzzleanditwasperceivedtobeabeneficial

andenjoyablesuiteofpuzzlesfortheirstudents.Oneteacherremarkedthat,"Iwouldlovetosee

theGearspuzzlebecomearegularappthatstudentscoulduseandaccess."Thisteacher,likemany

others,thoughtthattheGearspuzzlewouldincorporatewellintotheirclassroomsasaregular

feature.Anotherteacherspokemoretothispointandfoundthatitwasappropriatefortheirclass.

Theyreflectedthat,"UsingGearswasfunandagoodlevelofproblemsolvingfor5thgraders."

StudentandTeacherPerceptionsoftheTanksGames

SimilartothestudentimpressionsoftheTilesgame,theTanksgamealsohadmostlypositive

reviewsfromstudents.StudentsenjoyedthechallengingaspectoftheTanksgameandremarked

thatthey,"PrefertheTanksgamebecausetheTanksgameisjustchallenging.Liketheysaid,it’s

morechallengingandtheTilesoneisjustalittletooeasy."

CollectivelystudentslikedtheaestheticsoftheTanksgameandthatthevisualsmadeitafun

experience.Onestudentgrouprespondedexcitedly,"OHthosearefun!Ilovethesparks,the

bubbles,andthefillbutton.Ilikethebackground!"Overall,thebackgroundandanimationofthe

Tankspuzzlesmadeitanentertainingplatformtosolvemathpuzzles.

TeachersevidencedamixofperceptionsoftheTanksgames.Someteachersexpressedthatitwas

toodifficultfortheirstudentsandthatitwastheirleastfavoriteofthepuzzleswhileotherteachers

reflectedthatitpresentedapositivechallenge.Oneteacherreflecteduponthebenefitstheyfelt

theirstudentsgaineduponinteractingwiththesepuzzles;"Tanksfeltlikemathandtheythoughtit

wasgood.[It]madethemfeelbetteraboutmathandtheirabilitytodomath.[They]feelmore

confidentaboutmathandnotasnegativeaboutmath."Thisquoteinawayreflectsthe

characteristicdesignfeatureinallofBrainQuakepuzzles–aninterfacethatallowsyoutoengagein

mathproblemsinachallengingbutsafeway(i.e.,withoutriskofnegativemotivational

consequencesoffailure).


35

RQ 2: Does the BrainQuake suite show promise for improving students’ mathematics achievement and students’ attitudes toward mathematics? Throughout this section,we report statistical analyses of content assessment data. 603 students

respondedtothepre-testassessmentand531studentscompletedthepost-testassessment.Ofthe

post-assessment respondents, 28 students had a corresponding pre-assessment, leaving 503

studentsthatcompletedbothapre-andpost-assessmentforcomparisonanalysis.

Weexamineoverallgainsfrompre-testtopost-

test,firstwithallstudentsthatcompletedapre-

and post-test, and thenwith low SES students.

There were 243 students represented in the

control group, and 260 students in the

treatment group. As displayed in Table 9

below, there are statistically significant

differences, at a 0.05 significance level, in pre-

test topost-testscores forboththecontroland

treatmentgroups. Resultsshowthatpost-tests

increaseforthecontrolgroup,aswellasforthe

treatment group after exposure to the

treatment.

Table 9. Quantitative Pre-test and Post-testScores (Means, Standard Deviations and Cohen's d Effect Size of the Difference) for bothconditiongroups

Condition N Pre-testMean,SD

Post-testMean,SD

Gain 95% CI forMeanDiff.

t-Statistic p-Value Effectsize

Control 243 5.13,2.72 5.57,2.98 0.44 -0.81,-

0.07 -2.37 0.02 0.15

Treatment 260 4.97,2.68 5.34,2.86 0.37 -0.70,-

0.04 -2.23 0.03 0.14

What do these statistics mean? Twoimportantstatisticsarep-valuesandeffectsizes.

P-valuesgiveinformationabouttheprecisionofthestatisticalanalyses.Lowerp-valuesarerepresentativeofhigherprecision,andwouldindicateahigherlikelihoodthatthedifferencesarestatisticallydifferentfromzero,iftherewastrulynodifference.Thestandardconventionineducationresearchistoconsiderp-valuessmallerthan.05tobestatisticallydifferentfromzero.

Effectsizes(reportedasCohen'sd)representthemagnitudeofthestatisticaleffects.Effectsizesrepresentthechangefrompre-topost-test,inunitsofstandarddeviations.Effectsizesbelow.25areconsideredsmall,butcouldbestilleducationallymeaningful.Thoseabove.25areconsideredsubstantiveforeducation(e.g.,WhatWorksClearinghouse,2017).


36

BaselineEquivalence

Prior to conducting the impact analysis,we examined the equivalence of the groupsprior to the

intervention(i.e.,referredtoas"atbaseline")todetermine if theanalyticsampleofstudentswas

similarpriortothestartofthestudy.Todothis,pre-interventionvariableswereregressedontothe

condition variable, accounting for the nested structure of the design. Table 10 reports the effect

sizes,coefficients,andp-valuesoftheseanalysesforeachpre-interventionvariable.

Table 10. Baseline equivalence statistics. Continuous variable effect sizes are reported inHedges' g whereas dichotomous variable effect sizes are reported in relation to the Coxindex.

Variable Achievement Sample Attitude Sample Effect Size Coef p Effect Size Coef p

PreTest 0.08 0.24 0.73 0.00 0.00 0.99 SB Math -0.01 -1.28 0.97 -0.05 -4.37 0.89

Female 0.24 1.49 0.07 0.05 1.08 0.76 ELL 0.07 1.12 0.86 0.08 1.15 0.84 SDS -0.64 0.35 0.22 -0.58 0.39 0.27

Effectsizescorrespondtostandarddeviationdifferencesinthetreatmentvs.thecontrolgroup

priortotheintervention.Forthemostpart,thegroupsdonotdifferstatisticallyonanybaseline

variablesacrossthesamples,andeffectsizesaremostlybelow.25andabove-.25.Theexceptionis

theSDS–therearemoreSDSstudentsinthecontrolrelativetothetreatment.Thus,weattemptto

controlforSDSbyincludingitasacovariateintheimpactanalyses,reportedbelow.

ImpactonAchievement

TheoutputfromthemodelsofstudentachievementarepresentedinTable11.Themodelsattempt

toaddressthequestion:Afterattemptingtocontrolforstudentpre-testscores(andinsomecases,

otherbaselinevariables),whatistheimpactoftheBrainQuakesuiteonstudents’mathematics

knowledge?Putanotherway,weareinterestedinwhatacontrolstudents’mathematics

achievementwouldbeifhis/herinstructor,inanalternativeuniverse,wereassignedtothe

treatmentgroup.Becausestudentscannotparticipatetobothconditionssimultaneously,our

randomizedtrialisaproxyforthiscounterfactualscenario.


37

Table11.Achievementmodelingoutput.

Variables Condition Model Covariate Model Moderator Model Coef SE p Coef SE p Coef SE p

Intercept 5.94 0.35 0.00 6.42 0.59 0.00 6.06 0.83 0.00 Condition 0.24 0.40 0.56 0.03 0.49 0.95 0.49 1.04 0.64

PreTest 0.48 0.04 0.00 0.29 0.05 0.00 0.16 0.09 0.07 Female -0.05 0.28 0.85 -0.63 0.44 0.15

sds 0.22 0.38 0.56 0.54 0.63 0.39 ELL -0.12 0.36 0.74 0.01 0.56 0.98

SB.Math 0.02 0.00 0.00 0.02 0.00 0.00 Grade 1.83 0.41 0.00 1.03 0.50 0.06 1.52 0.86 0.10

Condition* PreTest

0.20 0.11 0.07

Condition* Female

0.86 0.57 0.13

Condition* sds

-0.50 0.79 0.53

Condition* ELL

-0.21 0.74 0.78

Condition* SB.Math

-0.01 0.00 0.13

Condition*Grade

-0.75 1.09 0.50

Condition Effect Size (hedges g)

0.07 0.01 0.15

ThecentralvariableofinterestinthemodelsistheConditionvariable.Forthisvariable,positive

coefficientsindicatehigherpost-testscoresfortreatmentstudents.Theconditioncoefficientcanbe

interpretedastheaveragedifferenceinpost-testachievementscoresinthetreatmentrelativeto

thecontrolcondition,giventhatthegroupswereequivalentpriortotheintervention.

Inallmodels,theestimateofintheconditionvariableispositive–indicatinggreaterpost-test

achievementinthetreatmentrelativetothecontrolgroup–butnotatastatisticallysignificantrate

andatsmalleffectsizes.Forexample,intheConditionalModel,theeffectispositiveevenwhen

controllingforpre-testandgrade.24questionimprovementinthepost-score(however,the

treatmenteffectisnot"statisticallysignificant"i.e.,ap-valueof<.05).Theeffectsizeissmallat.07.


38

Thethirdmodelexploresmoderationeffects–thatis,aretheBrainQuakepuzzlesmoreimpactful

fordifferentsubgroupsofstudents?Thisquestionwasexploredbyincludinginteractionsinthe

modelsbetweenthetreatmentvariableandthebaselinevariableofinterest.Thevariablethat

producedamarginalmoderationeffectwasstudents'pre-testscore.Thecoefficientforthismodel

suggeststhatthetreatmentwasmorebeneficialforstudents'performinghigheronthepre-test

assessment(i.e.,forevery1pointimprovementonthepre-testscore,thereisa0.20-point

advantage,onaverage,forthetreatmentrelativetothecontrolgrouponthepost-test).

Analysisofthequalitativedatashowstudentswereabletoaccessmathinawaydifferentfrom

theirregularinstruction,"BrainQuakewasafunbreakfromdirectinstruction.Itallowedmy

studentstolearnmathconceptswithoutfearofgradedmaterialornegativefeedbackfroma

teacher".TeachersappreciatedhowBrainQuakeallowedstudentstoengagewithmathwithout

evenrealizingit.

Duetothetimingofthestudynotbeingalignedtotheproportionalityandratiosectionofthemath

curriculum,5thgradeteachershaddifficultyreportingthemathskillsstudentslearnedfromusing

BrainQuake.WhenaskeddidthestudentslearnmathfromusingBrainQuake,one5thgrade

teacherresponded,"Ihopeso.[We]willgaugelaterwhentheygettothoseconceptsinthemath

curriculum.I’llseeifthisclassisabletopickupthoseconceptsfasterthanotherclasses".

Incontrast,the6thgradeteachersreportedseveralmathrelatedbenefitsofthegamesonstudents'

mathematicalunderstandingofproportionsandfractionsandeaseandspeedofcompletingmath

problems.Belowisasampleofteacherresponsesaboutstudentlearning:

• "Theyseemtobeunderstandingproportionsbetter."

• "Ibelievetheyaregettingabetterunderstandingofhowfractionswork.Thevarietyof

questionsisaplus."

• "Ithelpedreinforcesomeoftheconceptswearelearninginclass.

• "Theythinksomemathiseasynow."

• "Thestudentsaregettingquickeratsolvingthepuzzles,especiallytheTanksgames.Afew

havestartedtoseeacorrelationbetweenthesizeofthemainTankandthevolumeofthe

filltubes."


39

Oneteacherreportedthegame-basedlearningenvironmentofBrainQuakedidnotofferher

studentsaneffectivelearningopportunity."Ithinktheylearnedverylittle.Ifeltlikeitwasmoreof

agame.Iftherewassomethingthatcouldofferhintsorfeedback,[like]helpvideosorshortmini-

lessons.Theydidn’timprovemuch.Therewasnothingthereteachingthemorshowingthemways

tosolvedifferentproblems."Thisquotemayreflecttheneedfordigitalmanipulatives,inorderto

bridgethepuzzleswithtraditionalmathematicalsymbols,whichareinevitablyhowstudentswill

beassessed.

Toimprovestudentlearning,severalteacherssuggestedtheBrainQuakesuiteofgamesbe

explicitlyalignedtoCommonCoreStateStandards,sothegamescanbereadilyincorporatedinto

theirmathcurriculum.ProvidingdirectconnectionsbetweentheBrainQuakesuiteofgamesand

themathcurriculumthroughdigitalmanipulativescanallowforbetteralignmentwithtraditional

mathassessmentsofstudentlearning.Overallteachersreportedthegreatestadvantageofutilizing

thegamesintheclassroomwasnotstudentgainsinmathematicsbuttheoverallexcitementabout

mathgeneratedbytheBrainQuakegames.

ImpactonStudentAttitudes

ToexamineimpactoftheBrainQuakepuzzlesonstudents'attitudestowardsmathematics,student

responsestotheattitudesurveywereconvertedtoanumber(with–2correspondingtoastrong

negativeresponse,2correspondingtoastrongpositiveresponse,and0representinganeutral

response).Students'responseswerethenaveragedacrossthesurvey.Thehierarchicalmodelsused

intheaboveimpactanalysisonachievementwerealsousedtoexamineimpactonattitudes–Table

12showstheresults.


40

Table12.Attitudesmodelingoutput.

Variables ConditionModel CovariateModel ModeratorModel Coef SE p Coef SE p Coef SE p

Intercept 0.51 0.04 0 0.52 0.08 0 0.38 0.11 0 Condition 0.05 0.06 0.39 0.07 0.07 0.36 0.27 0.14 0.05 PreTest 0.75 0.04 0 0.71 0.05 0 0.79 0.09 0 Female -0.11 0.06 0.06 -0.01 0.09 0.88

SDS 0.01 0.07 0.87 0.14 0.11 0.21 ELL -0.01 0.07 0.91 -0.05 0.12 0.67

Condition*PreTest

-0.13 0.11 0.24

Condition*Female

-0.16 0.12 0.18

Condition*SDS

-0.2 0.14 0.17

Condition*ELL

0.07 0.15 0.64

ConditionEffectSize (hedgesg)

0.08 0.12 0.45

Asintheachievementoutcomes,theimpactoftheBrainQuakepuzzlesonattitudesareconsistently

positive,infavorofthetreatmentinallmodels,indicatingthatthemathattitudesinthetreatment

conditionaremorepositiveatpost-testthanthoseofthecontrolcondition.Theestimatesarenot

statisticallysignificant,exceptinthemoderatormodel,wheretheeffecttrendstowardsstatistical

significance.Effectsizesareoverallsmallbutpositive-thesmalleffectsaretobeexpectedgiven

theshortdurationofthestudy.Inthemoderatorcase,theeffectisofasubstantialsize.

TofurtherexploreBrainQuake’simpactonstudents’mathattitudes,webrokedowntheanalysisof

theattitudesbythesurvey’sfoursubscales.Wethenmodeledthegaininthescores,predictedby

thetreatmentvariableandpre-testscore,witharandomeffectforteacher.Theresultsofthe

subscaleanalysesarepresentedinTable13.


41

Table13.Attitudesubscaleanalyses.

Subscale ControlGain

Additional Treatment

Gain

p-value

ViewsTowardsMath

-0.07 0.08 0.34

ConfidenceTowardsMath

0.03 0.11 0.28

MathSelfConcept 0.00 0.07 0.43 StrategiesTowards

Math 0.01 0.00 0.96

Onthreeofthefoursubscales,thegainforthetreatmentwashigherthanthecontrol–thiswas

particularlynoteworthyfortheconfidencetowardsmathematicssubscale–themodelsuggests

that,inthiscase,theadditionalgainforthetreatmentoverthecontrolisapproximately3.5times

thegainofthecontrol(i.e.,.11/.03=3.67)-thisisnotstatisticallysignificantbutofanoteworthy

sizetowarrantattentionandfurtherexploration.Inaddition,itisnoteworthythatfortwoother

subscales(mathself-conceptandviewstowardsmath),thegainsforthetreatmentarepositive

wherethegainforthecontroliseitherzeroornegative.

TheanalysisofsubscalessuggeststhattheimpactofBrainQuakepuzzlesonstudentattitudesis

promising,particularlyforimprovingtheirconfidencetowardsmathematics.

Becausethereappearedtobepositiveeffectsfortheattitudemeasureoverall,weexploredimpact

onattitudesbyitemtofurthercharacterizetheeffect.Ratherthanlookingatallitems,which

carrieshighriskoffalsepositivefindingsduetomultipletests,wereducedthesizeofouranalysis

byexaminingdifferentialgainsbetweenthetreatmentandcontrolgrouponthetopthreeitems

thatshowedthelargestdifferencebetweenthegroups,regardlessofwhethertheywerelargerfor

thecontrolortreatment.Thetopthreeitemswiththelargestdifferentialgainsarelisted,below.

(BelieveMathBestSubject)Ihavealwaysbelievedthatmathisoneofmybestsubjects

(StrategySearch)WhenIcannotunderstandsomethinginmathematics,Ialwayssearchfor

moreinformationtoclarifytheproblem.


42

(ConfidenceSolving)Howconfidentdoyoufeelabouthavingtodothefollowing

calculation?CalculatinghowmuchcheaperaTXwouldbeaftera30percentdiscount?

Consistentwithaboveeffects,gainsonallthreeoftheseitemswerepositive,andinallcases,larger

forthetreatmentrelativetothecontrolgroup.Inotherwords,itemswiththelargestdifferencesin

relationtopre-to-postgainsexclusivelyfavoredthetreatmentcondition.Inaddition,theseitemsare

inalignmentwiththeBrainQuakelogicmodel–forinstance,bydesign,studentscanarriveat

solutionstoBrainQuakepuzzlesinavarietyofdifferentways;thus,BrainQuakegamesare

expectedtosupportstudents’strategysearch.Moreover,oneofthepuzzles(Tanks)targeted

students’proportionalreasoning,whichisthesubjectofthequestionaboutconfidence.Finally,

becauseBrainQuakesupportsstudents’confidencetowardsmathematics,itisexpectedto

positivelyincreasetheirviewstowardsmathematics.

Tocharacterizethesizeoftheobservedgainsonthesetopthreeitems,weparsedstudentsinto

havingshownapositivegainornot(zeroornegativegain),andconductedageneralized

hierarchicallinearmodelforeachquestion,withconditionasapredictorandcontrollingforthe

pre-testscore,whilealsoincludingarandomeffectforteacher.Wethenexaminedtheoddsratio

fortheconditionvariablefromthesemodels.

Table14.Oddsratioanalyses.

Top3itemswithLargestGains

TreatmentOddsRatio

p-value

BelieveMathBestSubject

1.90 0.01

StrategySearch 1.89 0.007 ConfidenceSolving

(proportion) 1.41 0.15

AscanbeseenfromtheTable14,thetreatmentwasapproximately1.5-2timesmorelikelytogain

onthesethreequestionsrelativetothecontrolgroup.Intwoofthethreecases,thedifferenceinthe

gainsarestatisticallysignificant.Inallcases,theseresultsarepracticallysignificant.


43

StudentPersistence

Studentpersistencewasseenintwoways:1)accomplishingdifficultpuzzlelevels2)overcoming

technicalissueswithBrainQuakeapp.Studentpersistencewasreportedorganicallywhenstudents

patientlyworkedthroughdifficultpuzzles.Duringoneobservation,theresearchernotedastudent-

teacherinteraction,"Iwantyoutostrugglealittlebit.Ican'talwaysgiveyoutherightanswer."One

5thgradeteacherdescribedindetailhowheestablishedthenormofpersistencefromthe

beginningofthestudy.

“[Inthe]firstsession,thestudentssaidIdon’tunderstandhowtodoit.Iwouldgivehima

littlehintandfromthereon,theyworkedonitontheirown.Afterfirstsession,thestudents

wereabletopushthroughchallengingpuzzles.Theydiscovereditwasn’tashardasthey

originallythought,iftheysatandthoughtaboutitfirst.Thisisthetypicalformatofmy

class.”

Studentsdiscussedtheirdeterminationtocompletechallengingpuzzles,"Whenwedidn’tsolvethe

questionsrightaway,weweremoredeterminedtocompletethepuzzleandgetitright."

Studentpersistencewasalsoenforcedthroughclassroomimplementationrulessetbytheteachers.

Someteachersrequiredstudentstocompleteeachpuzzlewiththreestars,sostudentswereforced

toreplaypuzzles.Whenstudentswerenotaskedtoreplaylevelstothreestars,classroom

observationdatashowsstudentsselectedthenextbuttonbeforecountingthenumberofstars.

Duringthefocusgroups,thevastmajorityofstudentsreportedonlyfocusingonthediamondsand

notthenumberofstars.Onestudentmentioned,"ThethingIlikeisthatitatleastgivesyouone

star."

Thevastmajorityofstudentsreportedstrugglingtosolvepuzzlesispartofthevideogame

experience,"EvenwhenyouplayCallofDuty,yeah,it’sthewholepointofit.Becausetheywantto

challengeustoseehowsmartweare."Whenstudentswereaskediftheypreferredasingle

pathwayormultiplepathwaystosolveaproblem,analysisofthequalitativedatadidnotprovidea

clearpreference.Themajorityofstudentsfromoneclassroomreported,"Ilikewhenthereisonly

onewaytosolvetheproblem.GuessingistheonlywayIknow.Ifit’seasy,thenI’lljustdothe

work."Incontrast,anothergroupofstudentsfromadifferentclassroompreferredthespeed


44

providedfrommultiplepathways,"Ilikeitbetterbecauseyoucanmoveontoalevelfaster.Ittakes

youwaylongerwhenthereisonlyonesingleway[tosolvetheproblem]."

Studentpost-surveyindicated,asshowninFigure16,thatifstudentsdidn'tgetthepuzzlecorrect

onthefirsttry,(47%,51%,and53%,respectively)feltoptimisticabouttryingtosolvethepuzzle

again.

Figure 16: Student Post-Survey Respondents Indicated That If They Didn’t Get the Puzzle Correct on the First Try, 47%- 53% Felt Optimistic About Trying to Solve the Puzzle Again

Figure17showstudentresponsestothepost-surveyquestion,"IfIdidn'tgetoneofthe

Tanks/Tiles/Gearspuzzlesrightonthefirsttry,Iwasmotivatedtothinkofnewordifferentways

togettotherightanswer".TheGearspuzzle,byaslightmargin,seemedtoprovidethemost

motivationtoencouragethestudentstotryagainortryuntilthepuzzlewasfiguredout(74%,73%

and78%,respectively).


45

Figure 17: Student Post-Survey Respondents Indicated That If They Didn’t Get the Puzzle Correct on the First Try, 74%- 78% Felt Motivated to Think of New Ways to Get the Right Answer

RQ 3: What is the impact of the BrainQuake puzzle suite on teachers’ pedagogical content knowledge for teaching?

ImpactonTeachersPedagogicalContentKnowledge

TeacherswereprovidedanopportunitytocompletetheMathematicalKnowledgeforTeaching

(MKT)questionnairebothatthebeginningandendofthestudy,providingtheopportunityforusto

assesswhetherMKTchangesoverthecourseofthestudy.Onereasonwhyteachersmightincrease

intheirknowledgeforteachingmathematicsisbecauseoftheopportunitiesthatBrainQuake

puzzlesprovidesforsupportingstudents’problemsolving.Forexample,inthisandearlierwork,

wehaveobservedteacherswalkaroundtheroomandprovideassistancetostudents–teacher

assistanceofteninvolvesbackandforthdialogueaimedatsupportingthestudentthroughthe

mathematicalchallenge.Thus,thesescaffoldinginteractionsmayserveasopportunitiesfor


46

teacherstobothgaininsightabouttheirstudents’understandingofmathematics,aswellasprovide

practiceatsupportingstudentsinmathematicalproblemsolving.

Thenumberofteachersthatprovidedcompletepre-andpost-datafortheMKTwaslow(Ncontrol=

7,Ntreatment=9),thus,weviewtheseanalysesasexploratory.TheimpactoftheBrainQuakepuzzles

onteacherMKTwasexaminedviasingle-levelregressionmodel,regressingpost-MKTscoreonthe

conditionassignmentvariable,controllingforpre-MKTscore,yearsofexperienceteaching,grade

taught,andtheproficiencyofusingtechnologyforteaching(calculatedfromthebackground

questionnaireatpre-study).

Themaineffectofconditioninthismodelwasnotstatisticallysignificant,butwaspositiveandofa

notablesizeforeducationalimpact–specifically,theeffectofgoingfromthecontroltothe

treatmentconditiononpost-MKTscorewas2.67(SE=2.10,p=0.23).Theeffectsizeforthiseffect

was0.40,whichisasubstantialsizeforeducationandsuggeststheinterventionholdspromisefor

supportingteachers'mathematicalknowledgeforteaching.

Conclusions and Next Steps

ThestudyshowedlargelypositivefindingsandsupporttheconclusionthatBrainQuake’spuzzle

suiteisapromisingtoolthatcanbeusedtosupportmiddle-schoolteachingandlearningin

mathematics.Teachersandstudentsusedthepuzzlesinavarietyofways,withminimaltraining,

speakingtothepuzzlesuite'sabilitytobeflexiblyadaptedbyteachersindiverse

contexts.Thepuzzleswerealsolargelyperceivedpositivelybybothteachersandstudents,evenin

spiteofsometechnicalissuesexperienced.Oneconsistentfindingacrossthepilotstudy(aswellas

priorusability/feasibilitystudies)isthatthepuzzlegamesarehighlyengagingforstudents–

studentsdisplayedextraordinarypersistenceattryingtosolvethepuzzles,evenafterindicating

thattheyhadfailedmultipletimes.Thepuzzles’abilitytoincitepersistenceeveninthefaceof

failureisakeybenefitoftheBrainQuakepuzzles,andmayhavecontributedtotheconsistently

positiveimpactsweobservedinstudents’attitudestowardsmathematics.Thepotentialfor

BrainQuakepuzzlestoincreasestudentattitudestowardsmathisnon-trivial,asitcouldpotentially

resultinsignificantdownstreambenefitsthatcouldpositivelyinfluencestudents’mathematics

learningtrajectoriesinthelong-term.


47

ThoughwebelievetheseresultssupportBrainQuakes’puzzlessuitabilityforclassroomcontexts,

afterreviewingresultsfromthestudy,webelievethatafewmodificationstothepuzzlesuitewill

furtherenhanceBrainQuake’svalueandimpact.Below,weprovidesomepointsofdiscussionfor

modifyingthegamesgoingforward.Thesediscussionpointsarebasedonourunderstandingof

teachers'andstudents'feedback,aswellasourownobservations—withthegoaloffurther

improvingtheproductforclassroomuse.

Game Modifications Ø Furtherbuildoutthedigitalmanipulatives(DM)versionsforeachpuzzle.

BrainQuake'sinnovativedesignplatformallowsstudentstoengageinmathematicsina

positiveway–atthesametime,studentsareassessedbyteststhatinevitablyrequire

traditionalmathematicalnotationofformalschooling–DMsprovidethelinkbetween

these.WebelieveDMswillfurtherimprovetheproducts'impactonstudents'achievement

outcomes.

Ø Consider making Tiles and Tanks adaptive to students' game performance. Some

students reported the Tiles and Tanks were extremely difficult, whereas students in the

sameclassaskedtomakeTilesandTanksmoredifficultandaddatimedcomponenttothe

completionofthepuzzles.

Ø Teachers and students requested adding embedded hints and feedback to scaffold

the game play experience for students who are struggling to complete certain puzzles.

Because5thgradershadnotcoveredratiosandproportionalityinfallsemester,oneteacher

recommendedaddingintroductoryvideostopresentnewmathconcepts.

Ø Teachersalsoasked fora tutorialvideo to showstudentshow tonavigate the suiteof

games.

Ø Students and teachers suggested several game enhancements to improve student

engagement:

o StorylineintrotothesuiteofBrainQuakegames

o Bonuslevelforcompletingtenpuzzles

o HolidayversionsofWuzzits


48

o Twoplayerversion

Ø Studentsandteachersrequestedthegamesbeusedonatabletandincorporatethe

touch feature.Teachersandstudentsspecifically reportedmanipulating thegearson the

trackpad"hurttheirhands".

Classroom Implementation Modifications Ø Teachers suggested that having an explicit reference to state standardswould help

provide bothmore context for the teachers and allow them to integrate the BrainQuake

suiteofgamesappropriatelyintotheirclassroom.

Ø Teachers were adamant that the timing of the implementation would be more

appropriate if the students had access to the game when they were learning the

associatedmaterial.BaseduponthisfeedbackitwassuggestedthattheBrainQuakesuite

wouldbemostbeneficialforstudentswhenitcanalignwiththeircurrentcurriculum.

Ø TeachersrequestedthatPDFversionsoftheBrainQuaketrainingwouldbehelpfulto

have throughout their use of the suite of games. Providing teacherswith screenshots

fromeachofthegamesandassociatedinstructionsonhowstudentscanprogressthrough

thesepuzzleswouldbebeneficialfortheimplementationofthesuiteofgames.

Ø Studentsandteachersexpressedthatthegamemayhavebeenmoreaccessibleand

easiertonavigatethroughaniPad.Manystudentsandteachersreflectedthatphysically

manipulatingthepuzzlesthroughtheiPadwouldmakeforanenjoyableexperience.

Ø It was a common theme expressed by teachers that a teacher dashboard would be a

valuableadditiontotheBrainQuakesuiteofgames.Teacherscanuse thisasa tool to

measure both gaps in the students’ knowledge as well as a means to hold students

accountablefortheamountoftimeandpuzzlestheycomplete.

Ø Teachersfurthermoreidentifiedthathavingstructuredlessonsasanintroductiontothe

different games (Gears, Tiles, Tanks)would be helpful for students as they progress

throughthesuiteofgames.


49

References

Baker,R.S.,Corbett,A.T.,Koedinger,K.R.,Wagner,A.Z. (2004).Off-TaskBehavior in theCognitiveTutor Classroom: When Students "Game The System". Proceedings of ACM CHI 2004:Computer-HumanInteraction,383-390.

Davenport, J., Kao, Y. S., & Schneider, S. A. (2013). Integrating cognitive science principles toredesignamiddleschoolmathcurriculum.InProceedingsofthe35thannualconferenceofthecognitivesciencesociety.Austin,TX:CognitiveScienceSociety.

Hill, H. C., Ball, D. L., & Schilling, S. G. (2008). Unpacking pedagogical content knowledge:Conceptualizingandmeasuring teachers' topic-specificknowledgeof students. Journal forresearchinmathematicseducation,372-400.

Kiili, K., Devlin, K., Perttula, T., Tuomi, P., & Lindstedt, A. (2015). Using video games to combinelearningandassessmentinmathematicseducation.InternationalJournalofSeriousGames,2(4),37-55.

LearningMathematicsforTeaching(2007).Mathematicalknowledgeforteaching[Assessment].Retrievedfromhttp://www.umich.edu/~lmtweb/

Matlen,B.J.,Atienza,S.,&FoxCully,K.(2015).WuzzitTroubleFeasibilityReport.WestEdResearch

Report.

Miles,M.B.,Huberman,A.M.,&Saldaña,J.(2014).Qualitativedataanalysis:Amethodssourcebook(Thirdedition.).California:SAGE.

OECD.(2005).PISA2003TechnicalReport.Retrievedfromhttps://www.oecd.org/edu/school/programmeforinternationalstudentassessmentpisa/35188570.pdf Pope,H.,&Mangram,C. (2015).Wuzzit trouble:The influenceof adigitalmathgameonstudent

numbersense.InternationalJournalofSeriousGames,2(4).

What Works Clearinghouse. (2017). What Works Clearinghouse: Procedures and standardshandbook.InstituteofEducationSciences.Retrievedfromhttp://whatworks.ed.gov


50

Appendix

Thisappendixcontainsthetablesreferredtointhebodyofthisreport.Table4.StudentMathematicsAchievementAssessment

ItemTheme Item#

ItemStatement

NumberSense 1 Whichofthesenumbersisbetween0.07and0.08?

3 Inwhichlistarethenumbersorderedfromgreatesttoleast?

4 Whichofthefollowingisthedecimalrepresentationof28/36andcorrectlyidentifieswhetherthedecimalisterminatingorrepeating?

SolvingforanUnknownQuantity

2 Duringgymclass,Jamijumped4.5feet.Brendajumped3.72feet.HowmuchfartherdidJamijumpthanBrenda?

5 Apainterhad25Lofpaint.Heused2.5Lofpainteveryhour.Hefinishedthejobin5.5hours.Howmuchpaintdidhehaveleft?

6 Whichofthefollowingis70%of48? 7 Davidboughtabaseballcardfor$40.Sincethen,thecardhasincreased

invalueby25%.WhatisthevalueofDavid’scardnow? 8 Liamwantstobuyabicyclethatcosts$335.00.HisparentssayLiam

mustraise20%ofthemoneyhimself.HowmuchmoneymustLiamraise?

9 Theamadethetablebelowtoshowthenumberofmiddleschoolstudentswhoattendedthelastfootballgame.Ifthisdataweredisplayedinacirclegraph,whatpercentofthegraphwouldrepresenttheeighthgraderswhoattendedthegame? Grade NumberofStudent

inAttendance6 3757 2758 350

10 If50%ofanumberis20,whatis75%ofthenumber? 11 FourstoresarehavingasaleonDVDs.Whichstoreisofferingthebest

deal? 12 Whenanewhighwayisbuilt,theaveragetimeittakesabustotravel

fromonetowntoanotherisreducedfrom25minutesto20minutes.Whatisthepercentdecreaseinthetimetakentotravelbetweenthetwotowns?


51

13

Thereare900studentsenrolledinAdamsMiddleSchool.Accordingtothegraphabove,howmanyofthesestudentsparticipateinsports?

14 Ofthefollowing,whichistheclosestapproximationofa15percenttiponarestaurantcheckof$24.99?

15 Kateboughtabookfor$14.95,atoyfor$5.85,andagamefor$9.70.Ifthesalestaxontheseitemsis6percentandall3itemsaretaxable,whatisthetotalamountshemustpayforthe3items,includingtax?

16 Groundbeefcosts$2.59perpound.Whatisthecostof0.93poundsofgroundbeef?

Open-EndedQuestions

17a Leeworks22.5hourseachweekandearns$9.48perhour.WhatamountofmoneydoesLeeearneachweek?Showorexplainhow yougotyouranswer.

17b Eachweek,$25.60istakenoutofLee’searningsfortaxes.WhatpercentofLee’sweeklyearningsistakenoutfortaxes?Showorexplainhowyougotyouranswer.

17c Leewantstobuyacarthatcosts$3,000.Leeplanstosaveallearningsthatremainaftertaxesaretakenout.WhatistheminimumnumberofhoursthatLeemustworkinordertosave$3,000tobuythecar?Showorexplainhowyougotyouranswer.

18a Foreachofthefollowingproblems,estimatetheproductusingbenchmarkvaluesANDdescribeyourreasoning. 2.4X0.8=?

18b 5.21X0.4=? Table5.StudentAttitudeSurvey

ItemTheme Item#

ItemStatement

PersonalIdentification

1 Name

2 Teacher'sName


52

3 BrainQuakeIDViewsonMathematics

4a Ienjoyreadingaboutmathematics.

4b MakinganeffortinmathematicsisworthitbecauseitwillhelpmeintheworkIwanttodolateron.

4c Ilookforwardtomymathematicslessons. 4d IdomathematicsbecauseIenjoyit. 4e Learningmathematicsisworthwhileformebecauseitwillimprovemy

careerprospects. 4f IaminterestedinthethingsIlearninmathematics. 4g MathematicsisanimportantsubjectformebecauseIneeditforwhatI

wanttostudylateron. 4h Iwilllearnmanythingsinmathematicsthatwillhelpmegetajob.ConfidenceinMathematicsCalculations

5a Usingatrainschedule,howlongitwouldtaketogetfromCityAtoCityB.

5b CalculatinghowmuchcheaperaTVwouldbeaftera30percentdiscount.

5c Calculatinghowmanysquarefeetoftilesyouneedtocoverafloor. 5d Understandinggraphspresentedinnewspapers. 5e Solvinganequationlike3x+5=17. 5f Findingtheactualdistancebetweentwoplacesonamapwitha

1:10,000scale. 5g Solvinganequationlike2(x+3)=(x+3)(x-3) 5h Calculatingthefuelefficiency–milespergallonofgas(mpg)-ofacar.MathSelf-Concept 6a Ioftenworrythatitwillbedifficultformeinmathematicsclasses.

6b Iamjustnotgoodatmathematics. 6c IgetverytensewhenIhavetodomathematicshomework. 6d Igetgoodgradesinmathematics. 6e Igetverynervousdoingmathematicsproblems. 6f Ilearnmathematicsquickly. 6g Ihavealwaysbelievedthatmathematicsisoneofmybestsubjects. 6h Ifeelhelplesswhendoingamathematicsproblem. 6i Inmymathematicsclass,Iunderstandeventhemostdifficultwork. 6j IworrythatIwillgetpoorgradesinmathematics.StrategiesTowardsMathematics

7a WhenIstudyforamathematicstest,Itrytoworkoutwhatarethemostimportantpartstolearn.

7b WhenIamsolvingmathematicsproblems,Ioftenthinkofnewwaystogettheanswer.

7c WhenIstudymathematics,ImakemyselfchecktoseeifIremembertheworkIhavealreadydone.

7d WhenIstudymathematics,ItrytofigureoutwhichconceptsIstillhave


53

notunderstoodproperly. 7e IthinkhowthemathematicsIhavelearnedcanbeusedineveryday

life. 7f IgooversomeproblemsinmathematicssooftenthatIfeelasifIcould

solvetheminmysleep. 7g WhenIstudyformathematics,ImemorizeasmuchasIcan. 7h Itrytounderstandnewconceptsinmathematicsbyrelatingthemto

thingsIalreadyknow. 7i Inordertorememberthemethodforsolvingamathematicsproblem,I

gothroughexamplesagainandagain. 7j WhenIcannotunderstandsomethinginmathematics,Ialwayssearch

formoreinformationtoclarifytheproblem. 7k WhenIamsolvingamathematicsproblem,Ioftenthinkabouthowthe

solutionmightbeappliedtootherinterestingquestions. 7l WhenIstudymathematics,IstartbyworkingoutexactlywhatIneedto

learn. 7m Tolearnmathematics,Itrytoremembereverystepinaprocedure. 7n Whenlearningmathematics,ItrytorelatetheworktothingsIhave

learnedinothersubjects.Table6.StudentBrainQuakePostSurvey

ItemTheme Item#

ItemStatement


1 FirstName

2 LastName 3 BrainQuakeID 4 Teacher 5 School

Engagement 6 IlikedoingBrainQuakepuzzles

7 BrainQuakepuzzleswerefun 8 OtherstudentsmyagewouldlikeBrainQuakepuzzles 9 IgotboreddoingBrainQuakepuzzles 10 IfeltengagedwhenIplayedtheBrainQuakepuzzles

Usability/Feasibility

11 WhenIusedBrainQuakepuzzles,Iknewwhattodowithoutanyonetellingme.

12 IthinkmostkidswouldlearnhowtouseBrainQuakepuzzlesveryquickly.

13 PlayingBRAINQUAKEpuzzleswasconfusing. 14 TheBRAINQUAKEpuzzlesshouldprovidemorehintsordirections. 15 BrainQuakepuzzlesfitwellwithourclassroomactivities.

Improved 16 IfeellikeIlearnedaboutmathfromusingBrainQuakepuzzles.


54

LearningandMotivation

17 IwoulduseBrainQuakepuzzlesagaintolearnmoreaboutmath. 18 AfterusingBrainQuakepuzzles,IfeellikeI'mbetteratproblemsolving. 19 AfterusingBRAINQUAKEpuzzles,IfeellikeI'mbetteratfractionsand

decimals. 20 TheBRAINQUAKEpuzzleshelpedmeunderstandhowmathworks. 21 TheBrainQuakepuzzlestutorials/directionswerehelpful.

OverallGamePerception

22 TheBrainQuakepuzzleswere...

IndividualPuzzle(Tanks,Tiles,

Gears)Persistence Resilience

23,28,34

WhenIdidn'tunderstandoneoftheTanks/Tiles/Gearspuzzles,IwouldkeepattemptingituntilIgotitright.

24,29,35

IfIdidn'tgetoneoftheTanks/Tiles/Gearspuzzlesrightonthefirsttry,Iwasmotivatedtothinkofnewordifferentwaystogettotherightanswer.

25,30,36

TheTanks/Tiles/GearspuzzlehastaughtmethatevenifIfailatfirst,therearechancestoimproveandlearn.

MotivationtoTryAgain

26,32,38

OntheTanks/Tiles/Gearspuzzle,ifyoudidn'tgetthepuzzlerightonthefirsttry,howmuchdidyoufeelliketryingtosolveitagain?

31,37 PlayingtheTiles/GearspuzzlehastaughtmethatevenwhenIfacechallengingmathproblems,IcankeeptryinguntilIgetitright.

Encouragement 27,33,39

Ifyoudidn'tgettheTanks/Tiles/Gearspuzzlerightonthefirsttry,howdiditmakeyoufeel?

Table7.TeacherBrainQuakePostSurvey

ItemTheme Item#

ItemStatement


1 FirstName

2 LastName 3 School

4 District

StudentEngagement

5a MystudentsenjoyedusingBrainQuakepuzzles.

5b MystudentshadfunusingBrainQuakepuzzles.

5c MystudentswantedtoplayBrainQuakepuzzlesevenwithoutmetellingthemto.

5d MystudentsgotboredplayingBrainQuakepuzzles


55

5e MystudentswereengagedwhentheyplayedtheBrainQuakepuzzlesUsability/Feasibility

6a IthoughtbystudentsfoundBrainQuakepuzzleseasytouse.

6b UsingBRAINQUAKEpuzzlesintheclassroomishelpfulformystudents.

6c UsingBRAINQUAKEpuzzlesintheclassroomwaseasytodo.

6d UsingBRAINQUAKEpuzzleswasconfusingformystudents. 6e BrainQuakepuzzlesfitwellwithourclassroomactivities.

ImprovedLearning

7a IfeelmystudentslearnedaboutmathfromusingBrainQuakepuzzles.

7b IwoulduseBrainQuakepuzzlesagaintoteachmystudentsmoreaboutmath.

7c AfterusingBrainQuakepuzzles,Ifeelmystudentsarebetteratproblemsolving.

7d AfterusingBRAINQUAKEpuzzles,Ifeelmystudentsarebetteratproportionalreasoning.

7e TheBRAINQUAKEpuzzleshelpedmeunderstandhowmathworks.

7f TheBrainQuakepuzzlestutorials/directionswerehelpfulformystudents.

OverallGamePerception

8 Forthestudentsinmyclassroom,theBrainQuakepuzzleswere...

TeacherEngagement

9 WhatdidyoulikeaboutusingtheBRAINQUAKEpuzzlesinyourclassroom?

TechnicalChallenges

10 WerethereanyobstaclesassociatedwithusingBRAINQUAKEgamesinyourclassroom?

StudentDifficulty 11 WeretheBRAINQUAKEpuzzlesattherightlevelofdifficultyforyourstudents?Whyorwhynot?

FutureClassroomUsage

12 HowwouldyouusetheBRAINQUAKEpuzzlesinyourclassroomifyouweretodothisstudyalloveragain?

13 WouldyourecommendBRAINQUAKEpuzzlestoothercolleagues?Whyorwhynot?

IndividualPuzzleComparison(Tanks,Tiles)Enjoyment Engagement Challenge

14 Didyouobservedifferencesinstudents'experience(e.g.,enjoyment,engagement,challenge,etc.)whenplayingtheTilesgamescomparedtotheTanksgames?Orwerestudents'experiencessimilarfortheTilesandTanksgames?

15 Isthereanythingelseyouwouldliketoshareaboutyourexperience?

brainquake game-based learning and assessment applications

Documents