feedback design patterns for math online learning systems · 2018. 8. 29. · feedback design...

FeedbackDesignPatternsforMathOnlineLearningSystems PAULSALVADORINVENTADO,DepartmentofComputerScience,CaliforniaStateUniversityFullertonPETERSCUPELLI,SchoolofDesign,CarnegieMellonUniversityCRISTINAHEFFERNAN,MathematicalSciencesDepartment,WorcesterPolytechnicInstituteNEILHEFFERNAN,ComputerScienceDepartment,WorcesterPolytechnicInstitute

Increasingly,computer-based learningsystemsareusedbyeducatorsto facilitate learning.Evaluationsofseveralmath learningsystemsshowthat they result in significant student learning improvements. Feedback provision is one of the key features in math learning systems thatcontributetoitssuccess.Wehaverecentlybeenuncoveringfeedbackdesignpatternsaspartofalargerpatternlanguageformathproblemsandlearningsupportinonlinelearningsystems.Inthispaper,wepresentthreefeedbackdesignpatternsdevelopedfromtheapplicationofthedata-drivendesignpatternmethodologyona largeeducationaldatasetcollected fromactualstudentdata inamathonline learningsystem.Thesedesignpatternscanhelpteachers,learningdesigners,andotherstakeholdersconstructeffectivefeedbackforinteractivelearningactivitiesthatfacilitatestudentlearning.

Categories and Subject Descriptors: K.3.1 [Computers and Education]: Computer Users in Education—Distance learning;H.5.2 [InformationInterfacesandPresentation]:UserInterfaces—User-centereddesign

GeneralTerms:Design,HumanFactors

AdditionalKeyWordsandPhrases:designpatterns,onlinelearningsystems,problemsolving,feedback,datadrivendesignpatternproduction,ASSISTments

ACMReferenceFormat:

Paul Salvador Inventado, Peter Scupelli, Cristina Heffernan, and Neil Heffernan. 2017. Feedback Design Patterns for Math Online LearningSystems.EuroPLoP’17.(July12-162017),15pages.

1. INTRODUCTION

We define online learning systems as computer-based systems accessible over the internet that help instructorsmanage teaching resources, deliver content to their students, and facilitate student learning. Examples of suchsystems include learning management systems, intelligent tutoring systems, and massive open online courses(MOOCs). Such systems can be used to augment classwork and homework in traditional classroom settings, todelivercompletelyonlinecourses,tomanageflippedclassrooms,andsoforth.Researchinonlinelearningsystemssuggests incorporating interactive learningactivitieswithassociatedfeedbackto further improvestudent learning(Clark&Mayer, 2016). In fact, Koedinger and colleagues (2015) reported that students enrolled in a PsychologyMOOC learned about six times more when they additionally engaged in interactive learning activities withassociatedfeedback.

Pedagogicalfeedbackoftenreferstoprovidingstudents informationabouttheirperformance.Educatorsagreethat feedback is important, but it is difficult to design effective feedback. In the same way, designing effectivefeedbackforonlinelearningsystemsisnottrivialbecauseseveralfactorsneedtobeconsideredsuchasthelearningenvironment,thesubjecttaught,students’learninghistory,individualdifferences,andothers.Designpatternshaverecentlybeenconsideredtofacilitatetheselectionandapplicationofsolutionsthataddresseducationalchallenges

This material is based upon work supported by the National Science Foundation under DRL-1252297. Author's address: Paul Salvador Inventado,Department of Computer Science, California StateUniversity, Fullerton 800N. State CollegeBlvd.Fullerton,CA92834;email:[email protected];PeterScupelli,SchoolofDesign,CarnegieMellonUniversity,5000ForbesAve.,Pittsburgh,PA15213;email:[email protected];CristinaHeffernan,MathematicalSciencesDepartment,WorcesterPolytechnic Institute,100 InstituteRoad Worcester, MA 01609-2280; email: [email protected]; Neil Heffernan, Computer Science Department, Worcester Polytechnic Institute,WorcesterPolytechnicInstitute,100InstituteRoadWorcester,MA01609-2280;email:nth@wpi.eduPermissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationonthefirstpage.Copyrightsforcomponents of thiswork owned by others than the author(s)must be honored. Abstractingwith credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions [email protected]'17,July12—16,2017,Irsee,Germany©2017Copyrightisheldbytheowner/author(s).PublicationrightslicensedtoACM.ACMISBN978-1-4503-4848-5/17/07…$15.00https://doi.org/10.1145/3147704.3147738

FeedbackDesignPatternsforMathOnlineLearningSystems:Page-2

inonline-learningcontexts.Onesuchchallengethatthesepatternsaddressisthecreationoffeedback.Forexample,designpatternshavebeenwrittentomanageMOOCs,todesigne-learningcontent,andtoconstructfeedbackforinteractivelearningactivities(Warburton&Mor2015,Rusman,Lutgens&Ronteltap2005,Zimmerman,Herding&Bescherer2014).

This paper contributes three design patterns for constructing feedback to student responses in math onlinelearning systemsnamely: IncorrectExampleExplanation,Common-wrong-answerFeedback, and IncreasingHintSpecificity. Instructors, teachers, learningdesigners, andother stakeholders canuse thesepatterns indevelopingsuchsystems.The followingsectionsdescribepriorwork indesignpatterns foronline learningsystems, thedata-driven design pattern productionmethodology (3D2P) used to develop the patterns discussed in this paper, thepatternlanguagecontainingthesepatterns,thedesignpatterns,andfuturework.

2. RELATEDWORK

Severaldesignpatternlanguagesandcollectionshavebeendevelopedforonlinelearningsystems.Forexample,thee-Len project developed 42 e-learning design patterns, whichwere categorized into four special interest groupsnamely: SIG 1: Learning resources and learning management systems (LMS); SIG 2: Lifelong learning, SIG 3:Collaborativelearning;andSIG4:Adaptivelearning(Rusman,Lutgens&Ronteltap2005).AnotherexampleisMorandWarburton’s32MOOCdesignpatternsthataddressedvariousaspectsofMOOCdesignincludingparticipation,community, structure, learning, and orientation (Warburton &Mor 2015). Finally, Mor, Mellar,Warburton, andWinters (2014) compiled 29 design patterns for teaching and learning with technology, which covered learner-centereddesigns, learningcommunities,socialmediaandlearnerinteractioninsocialspaces,andassessmentandfeedback.

Amongthesepatternlanguagesandcollections,onlyafewpatternsdescribeddesignsforfeedbackstrategies.AnotableexampleistheHintonDemanddesignpattern,whichsuggestsgivingstudentstheoptiontorequesthintssothatknowledgeablestudentshavetheflexibilitytoanswerproblemsontheirown,whilestudentsstrugglingtoanswer a problem can get help (Zimmerman, Herding & Bescherer, 2014). Although feedback design patternsoutsidetheonline-learning-systemdomaincanresolveonlinelearningsystemissues,theymayneedadaptationtoaddress contextual differences. Some examples include Feedback Sandwich,Differentiated Feedback, and PeerFeedback,whicharepartofBerginetal.’s(2012)pedagogicaldesignpatterns.

3. DESIGNPATTERNMININGMETHODOLOGY

Figure 1 illustrates a pattern language we have been developing for online learning systems, which includefeedback-specificdesignpatterns.Auniquefeatureofthedesignpatternsinthispatternlanguageisthattheyweredevelopedusingthe3D2Pmethodology(Inventado&Scupelli2016b).Detailsaboutthe3D2Pmethodologyandthedesign patterns produced using the methodology can be found in Inventado and Scupelli (2015b, 2016a,b,c).Inventado&Scupellidefines3D2Pas:

... a four-step iterative process used to uncover design patterns from data collected in a particular domain.3D2Pstartsbyprospectingdatatofindinterestingrelationshipsinthedata.Theserelationshipsareinvestigatedfurther in the pattern-mining step to develop hypotheses based on recurring problems and high-qualitysolutions uncovered. Literature and experts in the field are consulted to test the validity of the hypotheses.Resultinghypothesesareusedtowriteproposedpatterns,whicharefurtherrefinedwiththehelpofthedesignpatterncommunitythroughmentoringandpatternwritingworkshops.Accepteddesignpatternsareevaluatedbyimplementingtheminexistingsystemsandevaluatingtheirperformance.Randomizedcontrolledtrialsareconductedtocomparetheresultingoutcomemeasures(e.g.,learninggain,timeontask)betweenapplyingthedesign pattern and not applying the design pattern. Results of the evaluation are used to further refine thedesignpatternasneeded.

Currently,ourpatternlanguagecontains19completedesignpatternsindicatedbythesolid-linedboxesinFigure

1.Tables1and2summarizesixdesignpatternsthatarecurrentlyindevelopmentandareindicatedbythebroken-lined boxes in the figure. There are five general design pattern themes namely Problems, Mastery Learning,Motivation, Personalized Learning, and Learning Feedback. Design patterns under the Problems theme addresschallenges related to the creation of problems in online learning systems. Design patterns under theMastery


Learning theme address challenges related to ensuring students’ mastery of a concept or skill. TheMotivationtheme contains design patterns that helpmaintain studentmotivationwhile learning. ThePersonalized Learningthemecontainsdesignpatternsthatenablesystemstoadapttostudents’skilllevels.TheLearningFeedbackthemecontains design patterns that address challenges in generating feedback for students learning through an onlinelearningsystem.Hints,Examples,andScaffoldingaredifferentstrategiestogeneratefeedback,whichfurthersplittheLearningFeedbacktheme.FeedbackContentisanothersubthemethatfocusesonthecontentofthefeedbackthat isutilizedbydifferent feedbackstrategies.Thethreedesignpatternsdescribed inthenextsectionfallunderthe Learning Feedback theme and are highlighted in blue in Figure 1. These patterns are Incorrect ExampleExplanation,Common-wrong-answerFeedback,andIncreasingHintSpecificity.

Fig.1.PatternLanguageforMathProblemsandLearningSupportinOnlineLearningSystems(ImagecourtesyoftheLearningEnvironmentsLab).

4. LIMITATIONS

Thedesignpatternspresentedinthispaperweredevelopedprimarilyformathbecausedatausedforthepatternprospectingandminingstepsof the3D2Pmethodologywere frommathproblemsets in theASSISTmentsonlinelearningsystem(Heffernan&Heffernan2014).Thedatausedintheanalysiswerecollectedbetween2012and2013which containedabout5.8million student interactionswith179,908problems. The selectedproblems coveredarangeof topicsmostlydesigned forgradessix througheightasdefinedby theCommonCoreStateStandards for


Mathematics (CCSSI 2010). We suspect that the design patterns presented in this paper may apply to otherdomains, but we opt to limit the scope of these patterns to math until we gather sufficient evidence of theireffectivenessinotherdomains.

5. DESIGNPATTERNS

Thepatternformatusedinthispaperseparateseachsectionwithaheadingmuchlikeotherpatternformats(c.f.,Carlsson 2004, Dearden & Finlay 2006). It contains the commonly used context, forces, problem, and solutionsections. Thebenefits section describes how the solution addresses the forces in the problem and the liabilitiessectionpresents issues thatmayarise from implementing the solution.Theevidence sectionprovides theoreticalfoundationsthatexplainwhytheproblemrecursandwhyasolutionmighteffectivelyresolve it.Forces,benefits,liabilities,andevidenceareorderedandalignedtofacilitatereadability.Forexample,force1isaddressedbybenefit1 but could result in liability 1, which is supported by evidence 1. The known uses section presents successfulapplications of the design pattern that validate its effectiveness. Finally, the related patterns section lists otherpatternsthatthedesignpatternreferencesorreferencesit.

Table1providessummariesofthethreedesignpatternsthatarepresentedinthefollowingsubsectionaswellaspatternsthatarecurrentlyunderdevelopment.Weanticipatethatthepatternsunderdevelopmentwillreferencethethreepatternsdiscussedinthispaper.Table2providessummariesofdesignpatternsthatarereferencedbythepatternsinthispaper.

TableI.FeedbackDesignPatternsforMathOnlineLearningSystems

DesignPattern Status Summary

IncorrectExampleExplanation PIP Askstudentstoexplainincorrectexamplestohelpthemunderstandandavoidcommonmistakesandmisconceptions.

Common-wrong-answerFeedback PIP Identifycommonwronganswersforagivenproblemandconstructfeedbacktoaddresstheunderlyingmisconception.

IncreasingHintSpecificity PIP Allowstudentstorequestprogressivelyelaboratehintsinwhichthelasthintcontainsthecorrectanswer.

MasteryLearningExerciseGenerator

UD Generateandassignproblemvariationsthattestaparticularskilltohelpstudentsmasterthatskill.

VideoHints UD Useavideotopresentfeedbackthathelpsstudentsvisualizetheproblem,capturetheirattention,andminimizetheirtendencytoskipfeedback.

RelateFeedbacktoAuthenticTasks

UD Useexamplesthatarebasedonreal-worldsettingstohelpstudentsunderstandthevalueoftheskilltaught.

DesirableFeedbackDifficulty UD Considerwhatstudentsalreadyknowtoprovidefeedbackthatwillchallengethem.

*Note:PIP-presentedinthispaper;UD-underdevelopment.

TableII.ReferencedDesignPatterns

DesignPattern Status Summary

PitfallDiagnosisandPrevention(Anthony1996)

P Payspecialattentiontovitalconceptsandemphasizethemwhenithasshownthatlasttimeyoutaughttheconceptstudentshadtroublewithit.

WorkedExamples(Inventado&Scupelli2015b)

P Providestudentswithanexamplesimilartotheproblemtheyareaskedtosolvesotheyunderstandhowtosolvetheproblemwithoutrevealingtheanswer.

ExplainWorkedSolutions(Inventado&Scupelli2016a)

P Providestudentswithclearlyexplainedworkedsolutionswhentheyareunabletoanswerproblemscorrectlydespitereceivingsupport.

Self-explainedAnswers UD Askstudentstoexplaintheiranswertoensuretheyunderstandit,tohelpreinforcetheirunderstanding,andtoencouragethemtomakegeneralizationsfromtheirsolutions.

ConciseFeedback UD Avoidextraneousandunnecessarilylongfeedbackexplanationssostudentscanfocusontheinformationtheyneedtosolvetheproblem.

ConversationalFeedback UD Useaconversationalstyleincommunicatingideaswithstudentstomakematerialmoreengaging.

*Note:P-published;UD-underdevelopment.


IncorrectExampleExplanation

Context:An instructorwants todeepenstudents’understandingof trickymathconceptsorskills.Suchtopicsaretrickybecausetheyarecomplexortheyinvolvespecialcasesthatchangehowaconceptisunderstoodorhowaskillis applied. The instructor uses an online learning system to assign exercises that can help students expand theirknowledgeandpracticetheirskills.

Problem:Studentscanunderstandandapplybasicconceptsandskills,buttheylackknowledgeandexperiencetoanswerunfamiliaroradvancedquestions.

Forces:

1. Studentsstruggletoidentifywhatmakestheiranswersincorrect.2. Studentsareunabletouncoverunderlyingmisconceptionsthatleadtoincorrectanswers.3. Manystudentsoftencommitthesamemistakesandsharethesamemisconceptionsontargetskills.

Solution:Providestudentswithseveralrelatedincorrectexamplesandaskthemtoexplainwhatandwhytheyarewrong.Beforecreatingincorrectexamples,ithelpsfirsttoidentifycommonwronganswerstoproblemsthatwillbeincludedinthemathexercisebecausestudentsarelikelytocommitthesamemistakes.Agoodsourceofincorrectexamples is previous students’ answers. Use the common wrong answers to construct incorrect examples andinterleavethemwithusualproblemsyouwouldincludeinanexercise.Askstudentstoidentifywhatiswrongwiththeincorrectexampleandtoexplainwhyitiswrongsotheycanidentifyfeaturesthatmakeitincorrectanduncoverthe underlying misconceptions that cause the error. Consider asking students to select from a list of possibleexplanations insteadofwritinga free-formexplanationtosimplify the taskespecially fornovice learners.Presentseveral incorrect examples, so students see different error variations and gain more experience. Feedback onstudents’self-explanationmayalsofacilitatelearning.

Benefits:

1. Byevaluating incorrectexamples, students learn to identify featuresofa solution thatmake it incorrect,whichwillhelpthemidentifyerrorsintheiranswers.

2. Studentscanbetteridentifyunderlyingmisconceptionsfromobservingseveralincorrectexamples.3. Different students can learn from the same incorrect examples, which may address their shared

misconceptionsandhelpthemavoidmakingthesamemistakesinthefuture.Liabilities:

1. High-performing students may find the self-explanation of incorrect examples too elaborate, time-consuming,ordistracting,whichmayhinderlearning.

2. Contentcreatorswillneedtodesignseveralincorrectexamplesforeachproblem.3. Incorrectexamplesalonemaynotdescribehowtosolvetheproblem.Correctexamplesarealsoneeded.

WorkedExamples(Inventado&Scupelli2015b),forexample,wouldcomplementthispatternwell.


Evidence:Ohlsson’s(1996)theoryoflearningfromerrorssuggeststhatindividualsneedtolearntodetectmisconceptions,toidentifythefeaturesthatcausedit,andtoexplainwhatadditionalconditionsorfeatureswillmakeitcorrect.

Using incorrectexamples,correctexamples,andself-explanationduringpracticecan improvestudent learning(Boothetal., 2013,Durkin&Rittle-Johnson2012,Hang, Liu&Shiu2008). Specifically, self-explanationof correctexamplesmayfacilitate learningbecause it forcesstudentstomaketheirknowledgeexplicit (Chi2000,Roy&Chi2005)andself-explanationofincorrectanswerscanhelpstudentsidentifyfeaturesthatmakethesolutionincorrectand recognize theirmisconceptions (Siegler 2002). Separate correct and incorrect examples or both correct andincorrectexamplestogethermaybeinterspersedwithpracticeproblems.Bothstrategiesareshowntobeeffective(Durkin&Rittle-Johnson2012,Booth,Lange,Koedinger,andNewton2013).

Students often make the same types of mistakes when they answer math problems, which are called errorpatterns,orcommonwronganswers thatdatesbackto theworkofRadatz (1979).Several researchstudieshavebeenconductedtoaddressstudents’commonerrors(Peng&Luo2009,Shulman1986).

KnownUses:IncorrectExampleExplanationshavebeenappliedindifferentlearningcontexts.InASSISTmentsforexample,itiseasy for content creators to construct and intersperse incorrect examples in a problem set. Figure 2 shows anexampleofan incorrectexampleexplanationproblem.Studentsareasked tochoose thestatement thatexplainswhy the answer was incorrect. The top left side of the image shows that the student has already answered apractice problem before answering the incorrect explanation problem. The design of this problem set is beingfinalizedbeforeitwillbedeployedtoASSISTmentsusers.Studentperformanceinansweringthisproblemsetwillbeevaluatedwhenenoughdataiscollected.

Fig. 2. Screenshot of an incorrect example explanation problem used in an ASSISTments problem set (Image courtesy of ASSISTmentshttps://assistments.org).

Thework of Booth, Lange, Koedinger, andNewton (2013) shows an example of incorrect examples and self-explanationthatwasdeployedintheAlgebra1CognitiveTutorsystem.Studentsusingthesystemansweredguidedpractice problems for two-step equations in Algebra with interspersed correct and incorrect examples. Theirmethodologyinvolvedaskingstudentstoself-explaincorrectandincorrectexamples.Unlikeusualself-explanationquestions,studentsusedmenuoptionstoselectwhattheythoughtwasdone intheexample(e.g.,add,subtract,multiply,divide),andwhythestepwascorrectorincorrect(e.g.,“Itwasillegalbecauseitcombinedtermsthatwerenot like terms”;“Itwas legalbutnothelpfulbecause itdidnot reduce thenumberof terms”).Practiceproblemsthatwereinterleavedwithinthesameexerciseinvolvedstudentssolvingproblemsthatwereautomaticallycheckedby the system. Students received correctness and explanatory feedbackwhen they submitted their answers andcould requesthints to gethelp.Boothet al. also conductedexperiments to compare studentperformancewhenexposed to (1)practiceonly, (2) correct exampleswith self-explanationandpractice, (3) incorrect exampleswithself-explanationandpractice, or (4) correct examples and incorrect exampleswith self-explanation, andpractice.


The results of the study showed that students exposed to practice, correct and incorrect examples, and self-explanation(i.e.,condition4)outperformedtheotherthreeconditions.

Huang,Liu,andShiu(2008)implementedacomputer-aidedsystemforlearningdecimalconcepts.Studentsusingthesystemanswereddecimalproblemexercisesthataskedthemtore-evaluatetheirincorrectanswertohelpthemfigureouttheirmistakesandunderstandconceptsmoreeffectively.Forexample,aftergivinganincorrectanswer,thesystemmayask“Doesthe“4”of“5.4”pancakesmeanthereare“four”pancakes?”sothatstudentscanfocusonthemeaningofthedecimalvalue.Huangetal.ranastudytocomparestudents’performanceinpre-,post-,anddelayed posttests when they either learned from incorrect examples through the computer-aided system(experimentalcondition)orfromansweringtest-sheetquestionsforpracticewithoutaccesstoincorrectexamples(controlcondition).Theresultsoftheexperimentshowedthatstudentsexposedtoincorrectexamplesperformedsignificantlybetterinimmediateanddelayedposttestscomparedtostudentswhoonlyengagedinpractice.

DurkinandRittle-Johnson(2012)introducedbothcorrectandincorrectexamplesandself-explanationquestionsin decimal-magnitude practice problems that students answered. In their case however, students used pen andpaperandnotalearningsystem.Nevertheless,resultsfromtheirexperimentalignedwithotherresearchshowingthevalueofincorrectexamples.Specifically,studentsinanexperimentalconditionweregivenpairsofcorrectandincorrectexamplesforthesameproblem.Foreachpair,theywereaskedtoexplainwhyanexamplewascorrectorincorrect,howthetwoexamplesweresimilarordifferent,andhowtheywouldteachanotherstudenttosolvetheproblem.Afterstudyingthreeexamplepairs,studentswereaskedtoanswerapracticeproblem.Thisprocesswasrepeated four times so that studentsworkedon12examplepairsand four interleavedpracticeproblems. In thecontrolcondition,studentswereonlyshowncorrectexamplesinsteadofanexamplepairbutwereexposedtothesamemethodologyofstudying12examplesandansweringfourinterleavedpracticeproblems.Studentsfrombothconditionsweregivenpre-,post-,anddelayedposttests,whichshowedthatstudentsexposedtobothcorrectandincorrectexamplesperformedsignificantlybetterthanstudentsexposedtoonlycorrectexamples.

RelatedPatterns: Commonwrong answers to a particular problem canbeused to construct incorrect examples,whichmayhelppreventstudentsfromperformingsimilarmistakes inthefuture. It isagoodideatoUseStudentSolutions (Köppe et al. 2015) to find incorrect examples because students are more likely to commit similarmistakes.SuchacaseimplementsPitfallDiagnosisandPrevention(Anthony1996).IncorrectExampleExplanationcanalsobeusedinconjunctionwithWorkedExamples(Inventado&Scupelli2015b)andSelf-explainedAnswerstosupportfurtherstudentlearning.


Common-wrong-answerFeedback

Context: A content creator designs a math exercise in an online learning system. It is a good idea to providestudentswithfeedback,sotheyunderstandconceptsbetterandlearnskillsproperly.Problem:Itisdifficulttocreatefeedbackthataddresseseachstudent’sincorrectanswertoamathproblem.Forces:

1. Studentswhostruggletoansweraproblemmaybeunabletosolveitunlesstheyreceivehelp.2. Students often share the same mistakes and underlying misconceptions that result in common wrong

answerstogivenproblems.3. Itusuallytakestimeandefforttoencodefeedbackforamathprobleminanonlinelearningsystem.

Solution:Constructfeedbackthataddressesstudents’commonwronganswerstoagivenproblem.Commonwronganswers need to be identified first before feedback can be created. Common wrong answers may be specificincorrectvaluesassociatedwithaparticularproblem,oran incorrectprocedure forsolvingaproblemsometimesreferred to as buggy rules (Brown&Burton 1978, Sleeman 1982). Experts can be consulted to identify commonwronganswersandbuggyrulesforagivenproblembasedontheirexperience.Alternatively,contentcreatorsmaycreateproblems sets in anonline learning system to collect initial studentdata thatwill identify commonwronganswers. Once common wrong answers are identified, static explanatory feedback can be designed forcorresponding incorrectanswers toaparticularmathproblem.Such feedbackcan thenbepresented tostudentswhen they submita specific incorrectanswer. In thecaseofbuggy rules,explanatory-feedback templates canbedesignedsothatspecific incorrectanswerscanbemergedwiththetemplateandpresentedtothestudentwhentheysubmitananswerthatviolatesabuggyrule.Consider,forexample,amathproblemthatasks“Imaginethat2outof three3ballsaregreen.What is thepercentageofgreenballs fromtheset?”Whenthestudentsubmitsaparticularanswerlike1.5,thatanswercanbecheckedagainstamappingofincorrectanswersandbuggyrules.Inthiscase,theassociatedbuggyrulemaybe:dividingthesecondvaluebythefirstvalue;afeedbacktemplatemaybe:“Areyousureyoushoulddivide<val2>by<val1>?”;andthemergedfeedbackpresentedmaybe:“Areyousureyoushoulddivide3by2?”Benefits:

1. Studentsreceivefeedbackbasedontheirincorrectanswerthatmayhelpthemsolvetheproblem.2. Common wrong answers can capture students’ common mistakes and misconceptions, which are

addressedbythefeedbackspecificallydesignedtohelpresolvethem.3. Contentcreatorsonlyneedtocreatefeedbackforcommonwronganswersandbuggyrulesinsteadofeach

possiblewronganswerforeachquestion.Liabilities:

1. Thesystemmaybeunabletoprovidefeedbackforuncommonwronganswersunlessdefault feedback isprovided,whichcanbelesseffective.

2. Contentcreatorsmayneedtoconsultexpertsconstantlytoidentifyandaddresscommonwronganswers.


3. Thelearningsystemwillneedtosupportfunctionalitiesthatallowtheprovisionofassociatedfeedbacktoincorrectanswers.Also,submittingthesameincorrectanswerwillprovidethesameexplanatoryfeedback,whichmightnotbehelpfulforthestudent.

Evidence:Studentswhostruggletounderstandaconceptoracquireaskillmaylearntodosowithappropriatefeedbackandguidance (Vygotsky1962).Humantutorsoftenprovide feedbackbasedonstudents’ incorrectanswers toaddresstheirunderlyingmisconceptions(Graesseretal.1999,Personetal.2003,Humeetal.1996).Studentsoftensharesimilarmisconceptionsthatleadthemtocommitsimilarmistakes(Brown&Burton1978,Sleeman1984).

The development of learning environments is often expensive and time-consuming (Murray et al. 2003).Students’ errors need to be analyzed to understand their underlying misconceptions and provide appropriatefeedback(Peng&Luo2009,Shulman1986).Learningsystemshaveusedbuggyrulestoassignappropriatefeedbacktocommonmisconceptions(Brown&Burton1978,Sleeman1984).KnownUses:Tutoringsystemsaredesignedtoprovideautomatedsupportforstudentslearninginvariedlearningsettings. It isdifficult to manually create feedback for every learning setting, so it makes sense to utilize mechanisms thatgeneralizeovercommonlyoccurringstudenterrors.

ASSISTments is an exampleof suchonline learning systems that allow teachers to easily construct and assignproblem sets to their students, to automatically evaluate students’ performance, and to generate reports thatidentifycommonmisconceptions,whichmayneedtobe furtherdiscussed inclass (Heffernan&Heffernan2014).ASSISTments collects data from students’ interactionswith problems,which allows it to identify commonwronganswersassociatedwiththeproblem.Teachersandcontentcreatorscanusethisdatatodesignbugmessagesthatareshowntostudentswhenevertheysubmitsuchcommonwronganswersforagivenproblem.Figure3showsascreenshotofastudentansweringamathprobleminASSISTmentsandFigure4showsascreenshotofASSISTments’authoring tool that displays the commonwrong answers associatedwith theproblem. Figure 4 shows threebugmessagesthatwillbeshowntostudentswhoanswer2,3,or4,whicharecommonwronganswersassociatedwiththe problem. The light blue box below the problem in Figure 4 shows the associated bugmessage shown to astudent who entered the commonwrong answer, 3. Although no specific experiments have been conducted toevaluate theeffectivenessofcommon-wrong-answer feedback inASSISTments, severalexperimentsshowthat itsdesign (including theuseof commonwrong answers) lead to significant learning gains in real classroom settings(Roschelleetal.2016).

Fig.3.Screenshotshowingastudent’sscreenwhenacommonwronganswerwassubmitted.Thelightblueboxisthebugmessageassociatedwiththecommonwronganswer,3(ImagecourtesyofASSISTmentshttps://assistments.org).


Fig. 4. Screenshotof theASSISTments authoring tool showing the commonwronganswers associatedwith aparticularproblemand thebugmessagesthatwillbeshownwhenaparticularanswerissubmittedbythestudent(ImagecourtesyofASSISTmentshttps://assistments.org).

GeometryTutor(Anderson,Boyle,&Yost1985)isatutoringsystemthatalsoutilizescommonwronganswerstoprovide automated feedback. Specifically, a set of ideal and buggy rules (IBR) was developed from theoreticalanalysisandempiricalobservationsofstudentbehaviortoprovidethesystemwithdomainknowledge.Theseruleswereusedbythesystemtoidentifystudents’misconceptionsonaparticulargeometryproblemandtoprovidethecorrespondingmessageassociatedwiththerule.Whenthestudentsubmittedananswerthatmatchedabuggyrulesuchas∠AMF≅∠BFE,forexample,thesystemwouldprovidethefollowingfeedback“No,itisnotusefultomakethatverticalinferencehere.Itisusefultomaketheverticalangleinferencewhentheanglesarecorrespondingpartsoftrianglesyouwanttoprovecongruent.Inthisproblem,whydon’tyoutrytomakeinvolvingthefactthatMisthemidpointofAB”.ExperimentsconductedonGeometryTutorshowedalargepositiveimpactonstudentlearningforusingthetutorinclass(Andersonetal.1995).

The Practical Algebra Tutor (PAT) is another tutoring system that utilized buggy rules to detect students’incorrectanswersandtoprovideassociatedfeedbackthataddressedtheirmisconceptions(Koedingeretal.1997).PATfocusedonhelpingstudentsapplybasicalgebraandreasoningskills inday-to-daysituationssuchascheckingthe amount of a paycheck, estimating the cost of a rental car for a trip, and choosing between long-distancetelephoneservices.Koedingeretal.’sexperiments showedsignificant learninggains in real classroomsettings forstudentswhousedthetutorcomparedtothosewhodidnot.Related Patterns: Image-enhanced Hints (Inventado & Scupelli 2016b), Concise Feedback, and ConversationalFeedbackcanbeusedtohelpconstructeffectiveCommon-wrong-answerFeedback.


IncreasingHintSpecificity

Context:Acontentcreatordesignshintsforaprobleminamathexercisepublishedonanonlinelearningsystem. Problem:Itisdifficulttoconstructappropriatehintsforstudentswithdifferentlevelsofbackgroundknowledge.Forces:

1. Studentsneedhintstorecallorclarifykeyconceptsthathinderthemfromansweringproblemscorrectly.2. Studentsdifferintheamountofinformationtheyneedtofigureouttheanswer.3. Providingtoolittleinformationmaynotbeenoughtohelpthestudentidentifytheanswer.4. Providingtoomuchinformationmayrevealtheanswertooquickly.

Solution:Providestudentswithasequenceofhintsthattheycanrequestprogressivelyandthatbeginswithgeneralhintsandincreasesinspecificity.Agoodstrategyfordesigningahintsequenceisstartingwiththesolutiontotheproblemandbreakingitintotheindividualstepsoftheprocess.ExplainWorkedSolutions(Inventado&Scupelli2016a),forexample,canguidetheconstructionofthesolutionbecauseitcontainsinformationneededtoexplaineachstep.Constructfeedbackforeachstepsothatitprovidesenoughinformationtoguidestudentstowardthenextstep,butdoesnotgiveawayanswersforthecurrentorsucceedingstep.ConciseFeedbackmayhelpclarifythecontentthatwillbeincludedinthefeedbackforeachstep.Itiscommonforthefinalsteptorevealtheanswertotheproblem,oftencalledthebottom-outhint,sothatstudentsdonotgetstuckansweringthesameproblemandareallowedtomoveontothenextproblemintheexercise.Benefits:

1. Readingthroughhintsinthesequenceallowsstudentstorecallandclarifykeyconceptstheyneedtosolvetheproblem.

2. Thesystemprovidesappropriatehelpforeachstepinthesequence,whichthestudentcanrequestprogressivelyasnecessary.

3. Studentscouldrequestmorehintsifthehintstheyreceivedwereunabletohelpthemsolvetheproblem.4. Hintspresentedearlierinthesequencereveallessinformationthatcouldgiveawaytheanswer;students

canonlyaccessmorespecificinformationwhentheyrequestit.Liabilities:

1. Studentsneedtogothrougheachhintinthesequencetofindrelevantinformation.However,thisprocesscanbetediousandcanpotentiallyboreorfrustratestudents.

2. Studentsdonotalwaysseekhelpeveniftheyneedassistanceandwhentheydoseekhelp,theymaynotuseiteffectively.

3. Contentcreatorsneedtoconstructappropriatehintsforeachstepinthesequence,whichisdifficultandtime-consuming.

4. Studentscan“game”thesystembyrevealingthebottom-outhinteveniftheydidnotusepriorhintsproductivelytohelpthemsolvetheproblem.


Evidence:1. AccordingtotheZoneofProximalDevelopment,expertguidancecanhelpstudentsachievedifficulttasks

thattheyarenotcapableofcompletingontheirown(Vygotsky1962).2. Humantutorsoftentailortheirfeedbackaccordingtostudents’prioractionsandtheirperceptionofwhat

thestudentknowsordoesnotknow(Personetal.2003).Thechallengeofhelp-seekingfacilitiesinonlinelearningsystemsisthatstudentswhoneedassistanceoftenfailtoseekhelpandthosewhodoseekhelpdonotusesuchfunctionalitieseffectively(Puustinen,1998,Ryanetal.,1998).

3. Studentsperformbetterwhentheyrequesthintspossiblybecausetheyreceivetimelyhelpcomparedtoproactivehelp,whichcouldbedistractingorannoying(Razzaq&Heffernan2010).

4. Tutorsoftenhelpstudentswhostruggletolearnaconcept,procedure,orskillbystartingwithhintsthatareasfarawayfromthesought-afteranswerandprovidemorespecifichelpuntiltheycanunderstandit(Humeetal.1996,Wood&Wood1999).Unfortunately,studentshavealsobeenshowntousehelpunproductivelyduetoseveralreasonsincludingdislikeofthesubjectmatter,thelearningenvironment,orcomputersingeneral,lackofeducationalself-drive,lowself-efficacy,andpoorlydesignedhelp(Bakeretal.2008).

KnownUses:MosttutoringsystemsemployIncreasingHintSpecificitybecauseitmimicsastrategycommonlyusedbyhumantutors.Forexample,systemdevelopers,learningdesigners,andteachersdesignandimplementhintsequencesforeachproblemintheASSISTmentsonlinelearningsystem(Heffernan&Heffernan2014).MostproblemsinASSISTmentsareformath,butitalsocontainsproblemsforotherdomainslikeEnglish,chemistry,andphysics.Thesystemdoesnotrequirecontentcreatorstoconstructhintsequencesinincreasingspecificity,butitisacommonpracticetheyfollowinASSISTments.StudentswhoanswerASSISTmentsproblemswithassignedhintsequencescanaccesshintsusinga“ShowHint”button.Everytimethebuttonisclickedassociatedhintsinthesequenceareprogressivelyrevealed.Allhintsareleftonthescreensostudentscaneasilyreviewthem.Figure5showsascreenshotofaprobleminASSISTmentswhereinthestudenthasalreadyrequestedtwooutofthreehintsinthesequence.Clickingonthebuttonagainwillrevealthecorrectanswertotheproblem,whichis60.9.

Fig. 5. Screenshot of an ASSISTments problem where a student has requested two out of the three hints available for the problem (Image courtesy of ASSISTments https://assistments.org).

AlgebraCognitiveTutorisanotherexampleofatutoringsystemthatallowsstudentstorequestmultiplelevelsofhintstohelpthemsolvealgebraproblems(Koedinger&Aleven2007).Studentscanprogressivelyrequesthints,whichprovidemorespecificadvicebasedonthesolutionstrategytheyused.Thisfeaturemeansthatthehintcontentisspecifictothecurrentstateofstudents’answers,whichmakehintsmoreappropriate.Unfortunately,thereislimitedinformationonhowcontentcreatorsconceptualizedhintsinthisresearch,butweknowtheyencodedthem.

GeometryCognitiveTutorisanothersystemdevelopedbylargelythesamegroupofresearcherswhodevelopedAlgebraCognitiveTutor.ItusesasimilarhintingstrategytosupportstudentswhoarelearningGeometry(Rolletal.


2011).ThereisactiveresearchintheCognitiveTutorfamilyoftutoringsystemstofurtherimprovestudentlearningsuchashelpingstudentsdevelopbetterhelp-seekingskills(Rolletal.2011).

Asofwritingthispaper,wewereunabletofindexperimentsthatevaluatedtheeffectivenessofusingprogressivehelpinanyofthesethreesystems.However,experimentsthatcomparedstudentperformancewithandwithoutusingthesystemhaveconsistentlyreportedimprovedlearninggains(Koedinger&Aleven2007,Rolletal.2011,Roschelleetal.2016).RelatedPatterns:HintonDemand(Zimmerman,Herding&Bescherer,2014)isoftenusedintandemwithIncreasingHintSpecificitybecauseitresultsinbetterstudentlearningcomparedtoproactivelyprovidinghints.ExplainWorkedSolutions(Inventado&Scupelli2016a)involvestheidentificationofsolutionstepsthatmayalsobeusedtocraftindividualhintsinthehintsequencetoprovideIncreasingHintSpecificity.Image-enhancedHints(Inventado&Scupelli2016b),ConciseFeedback,andConversationalFeedbackcanbeusedtoensurethequalityofeachhint.


6. SUMMARYANDNEXTSTEPS

ThepaperdiscussedthreedesignpatternsforconstructingfeedbackdesignpatternsformathonlinelearningsystemsnamelyIncorrectExampleExplanation,Common-wrong-answerFeedback,andIncreasingHintSpecificity.Onlinelearningsystemdevelopers,contentcreators,andteacherscanusethesepatternstoguidethecreationoffeedbackwhileensuringitseffectivenessinfacilitatingstudentlearning.

Weplantoapplyandtesttheeffectivenessofourdesignpatternsinotherdomainssuchasphysics,chemistry,computerprogramming,andsoforth.Similarly,itwouldbeinterestingtoevaluatehowwellthesepatternscantranslatetootherlearningenvironmentsliketraditionalclassrooms.Designpatternsthatarefoundtobeeffectiveinotherdomainsorlearningenvironmentsmaybegeneralized,andthosethatarenotmightleadtothedevelopmentofnewpatternsthatadapttothespecificconstraintsofthedomainorenvironment.

The3D2PmethodologyiscurrentlybeingusedondatacollectedfromtheASSISTmentsonlinelearningsystemtouncovermorepatternsthatwillbepartofthePatternLanguageforMathproblemsandLearningSupportinOnlineLearningSystems.Thedesignpatternsarebeingcompiledinanonlinedesignpatternrepository(http://learningenvironmentslab.org/openpatternrepository)andworkisbeingdonetofostercollaborationbetweendesignpatternauthors,domainexperts,anddesignpatternuserstocontinuewriting,evaluating,andrefiningdesignpatterns.

7. ACKNOWLEDGEMENTS

ThismaterialisbaseduponworksupportedbytheNationalScienceFoundationunderDRL-1252297.WewouldliketothankourshepherdElissavetaGourovaforherinvaluableadviceandfeedbackonthisworkandtheparticipantsofthewritingworkshopatEuroPLoP2017whohelpedusrefineandimproveourpaper.WealsothankSharrisFrancisco-InventadoforherhelpwiththevisualizationofthePatternLanguageforMathproblemsandLearningSupportinOnlineLearningSystems.

REFERENCESAnderson,J.R.,Boyle,C.F.,&Yost,G.1985.Thegeometrytutor.InIJCAI(pp.1-7).Anderson,J.R.,Corbett,A.T.,Koedinger,K.R.,&Pelletier,R.1995.Cognitivetutors:Lessonslearned.ThejournaloftheLearningSciences,4(2),

167-207.Anthony,D.L.1996.Patternsforclassroomeducation.InJohnVlissides,JamesCoplien,andNormanKerth(Eds.)PatternLanguagesofProgram

Design2.Addison-WesleyLongmanPublishingCo.,Inc.,Boston,MA,USA,391–406.Baker,R.,Walonoski,J.,Heffernan,N.,Roll,I.,Corbett,A.,&Koedinger,K.2008.Whystudentsengagein"gamingthesystem"behaviorin

interactivelearningenvironments.JournalofInteractiveLearningResearch,19(2),185.Bergin,J.,Sharp,H.,Chandler,J.,Sipos,M.,Eckstein,J.,Völter,M.,Manns,M.L.,Wallingford,E.,Marquardt,K.2012.Pedagogicalpatterns:

adviceforeducators.JosephBerginSoftwareTools;NewPaltz,NY.Booth,J.L.,Lange,K.E.,Koedinger,K.R.,&Newton,K.J.2013.Usingexampleproblemstoimprovestudentlearninginalgebra:Differentiating

betweencorrectandincorrectexamples.LearningandInstruction,25,24-34.Brown,J.S.,&Burton,R.R.1978.Diagnosticmodelsforproceduralbugsinbasicmathematicalskills.Cognitivescience,2(2),155-192.Carlsson,D.2004.AcategorizationofHCIpatterns.DepartmentofComputingScience,UmeaUniversity,Sweden.CommonCoreStateStandardsInitiative.2010.CommonCoreStateStandardsforMathematics.Washington,DC:NationalGovernorsAssociation

CenterforBestPracticesandtheCouncilofChiefStateSchoolOfficers.Chi,M.T.H.2000.Self-explainingexpositorytexts:Thedualprocessesofgeneratinginferencesandrepairingmentalmodels.InGlaser,R.(Ed.)

AdvancesinInstructionalPsychology,Mahwah,NJ:LawrenceErlbaumAssociates,pp.161-238Clark,R.C.,&Mayer,R.E.2016.E-learningandthescienceofinstruction:Provenguidelinesforconsumersanddesignersofmultimedialearning.

JohnWiley&Sons.Durkin,K.,&Rittle-Johnson,B.2012.Theeffectivenessofusingincorrectexamplestosupportlearningaboutdecimalmagnitude.Learningand

Instruction,22(3),206-214.Graesser,A.C.,Wiemer-Hastings,K.,Wiemer-Hastings,P.,Kreuz,R.,&TutoringResearchGroup.1999.AutoTutor:Asimulationofahumantutor.

CognitiveSystemsResearch,1(1),35-51.Heffernan,N.T.,&Heffernan,C.L.2014.TheASSISTmentsecosystem:Buildingaplatformthatbringsscientistsandteacherstogetherfor

minimallyinvasiveresearchonhumanlearningandteaching.InternationalJournalofArtificialIntelligenceinEducation,24(4),470-497.Huang,T.H.,Liu,Y.C.,&Shiu,C.Y.2008.Constructionofanonlinelearningsystemfordecimalnumbersthroughtheuseofcognitiveconflict

strategy.Computers&Education,50,61-76.Hume,G.,Michael,J.,Rovick,A.,&Evens,M.1996.Hintingasatacticinone-on-onetutoring.TheJournaloftheLearningSciences,5(1),23-47.Inventado,P.S.&Scupelli,P.2016a.DesignPatternsforMathProblemsandLearningSupportinOnlineLearningSystems.InProceedingsofthe

11thVikingConferenceonPatternLanguagesofPrograms(VikingPLoP2016),ACM,NewYork,NY,USA.


Inventado,P.S.&Scupelli,P.2016b.DesignPatternsforHelpingStudentstoLearntoVisualizeMathProblemsinOnlineLearningSystems.InProceedingsofthe21stEuropeanConferenceonPatternLanguagesofPrograms(EuroPLoP'16).

Inventado,P.S.&Scupelli,P.2016c.Learning-SupportDesignPatternsforStudentInteractionswithMathProblems.InProceedingsofthe23rdConferenceonPatternLanguagesofPrograms(PLoP’16).

Inventado,P.S.&Scupelli,P.2015a.Towardsanopen,collaborativerepositoryforonlinelearningsystemdesignpatterns.eLearningPapers,42(DesignPatternsforOpenOnlineTeaching):14-27.

Inventado,P.S.&Scupelli,P.2015b.AData-drivenMethodologyforProducingOnlineLearningSystemDesignPatterns.InProceedingsofthe22ndConferenceonPatternLanguagesofPrograms(PLoP2015).

Koedinger,K.R.,&Aleven,V.2007.Exploringtheassistancedilemmainexperimentswithcognitivetutors.EducationalPsychologyReview,19(3),239-264.

Koedinger,K.R.,Anderson,J.R.,Hadley,W.H.,&Mark,M.A.1997.IntelligentTutoringGoesToSchoolintheBigCity.InternationalJournalofArtificialIntelligenceinEducation,8,30-43.

Koedinger,K.R.,Kim,J.,Jia,J.Z.,McLaughlin,E.A.,&Bier,N.L.2015.Learningisnotaspectatorsport:DoingisbetterthanwatchingforlearningfromaMOOC.InProceedingsoftheSecond(2015)ACMConferenceonLearning@Scale(pp.111-120).ACM.

Köppe,C.,Niels,R.,Holwerda,R.,Tijsma,L.,VanDiepen,N.,VanTurnhout,K.,&Bakker,R.2015.Flippedclassroompatterns:designingvaluablein-classmeetings.InProceedingsofthe20thEuropeanConferenceonPatternLanguagesofPrograms(EuroPLoP2015).NewYork:ACM.

Mor,Y.,Mellar,H.,Warburton,S.,andWinters,N.(Eds.).2014.Practicaldesignpatternsforteachingandlearningwithtechnology.Rotterdam,Netherlands:SensePublishers.

Murray,T.,Blessing,S.,&Ainsworth,S.2003.Authoringtoolsforadvancedtechnologylearningenvironments:Towardcost-effectiveadaptive,interactiveandintelligenteducationalsoftware.SpringerScience&BusinessMedia.

Ohlsson,S.1996.Learningfromerrorandthedesignoftaskenvironments.InternationalJournalofEducationalResearch,25(5),419-448.Peng,A.,&Luo,Z.2009.Aframeworkforexaminingmathematicsteacherknowledgeasusedinerroranalysis.Forthelearningofmathematics,

29(3),22-25.Person,N.K.,Graesser,A.C.,Kreuz,R.J.,&Pomeroy,V.2003.SimulatinghumantutordialogmovesinAutoTutor.InternationalJournalof

ArtificialIntelligenceinEducation(IJAIED),12,23-39.Puustinen,M.1998.Help-seekingbehaviorinaproblem-solvingsituation:Developmentofself-regulation.EuropeanJournalofPsychologyof

education,13(2),271-282.Radatz,H.1979.Erroranalysisinmathematicseducation.JournalforResearchinMathematicsEducation,10(3),163-172.Razzaq,L.,&Heffernan,N.2010.Hints:isitbettertogiveorwaittobeasked?InIntelligentTutoringSystems(pp.349-358).Springer

Berlin/Heidelberg.Roll,I.,Aleven,V.,McLaren,B.M.,&Koedinger,K.R.2011.Improvingstudents’help-seekingskillsusingmetacognitivefeedbackinanintelligent

tutoringsystem.LearningandInstruction,21(2),267-280.Roschelle,J.,Feng,M.,Murphy,R.F.,&Mason,C.A.2016.Onlinemathematicshomeworkincreasesstudentachievement.AERAOpen,2(4),1-12.Roy,M.&Chi,M.T.H.2005.Self-explanationinamultimediacontext.InR.Mayer(Ed.),CambridgeHandbookofMultimediaLearning(pp.271-

286).CambridgePress.Rusman,E.,Lutgens,G.,andRonteltap,F.2005.Theproductionofe-learningdesignpatterns,andaresearchroadmapfore-learning.(Refnum:

101421-CY-2002-1-CY-MINERVA-MMP).RetrievedfromE-LENprojectwebsite:http://www2.tisip.no/E-LEN/documents/ELEN-Deliverables/Report_WP3_ELEN-Roadmap.pdf

Ryan,A.M.,Gheen,M.H.,&Midgley,C.1998.Whydosomestudentsavoidaskingforhelp?Anexaminationoftheinterplayamongstudents'academicefficacy,teachers'social–emotionalrole,andtheclassroomgoalstructure.JournalofEducationalPsychology,90(3),528.

Shulman,L.S.1986.Thosewhounderstand:Knowledgegrowthinteaching.Educationalresearcher,15(2),4-14.Siegler,R.S.2002.Microgeneticstudiesofself-explanations.InN.Granott&J.Parziale(Eds.),Microdevelopment:Transitionprocessesindevelopmentandlearning(pp.31-58).NewYork:CambridgeUniversitySleeman,D.1984.Anattempttounderstandstudents'understandingofbasicalgebra.CognitiveScience,8(4),387-412.Sleeman,D.1982.Assessingaspectsofcompetenceinbasicalgebra.InIntelligenttutoringsystems(pp.185-199).AcademicPress,NewYork.Vygotsky,L.S.1962.Languageandthought.MassachusettsInstituteofTechnologyPress,Ontario,Canada.Warburton,S.,andMor,Y.2015.AsetofpatternsforthestructureddesignofMOOCs.OpenLearning:TheJournalofOpen,Distanceande-

Learning,1-15.Wood,H.,&Wood,D.1999.Helpseeking,learningandcontingenttutoring.Computers&Education,33(2),153-169.Zimmerman,M.,Herding,D.,&Bescherer,C.2014.Pattern:HintonDemand.InMor,Y.,Mellar,H.,Warburton,S.,andWinters,N(Eds.)Practical

designpatternsforteachingandlearningwithtechnology.Rotterdam,Netherlands:SensePublishers.

feedback design patterns for math online learning systems · 2018. 8. 29. · feedback design...

Documents