Semester: 4th, Spring 2017

Title: Applying Cognitive Technologies and Emulating Interpersonal Communication in Affective Tutoring Systems

Project Period: 1/2/2017-30/6/2017 Semester Theme: Master Thesis Supervisor(s): Anders Henten

Project group no.: 4BUS 4.26 Members (do not write CPR.nr.): Lars Myrhøj Nielsen Pages: 91 Finished: 8/6/2017

Abstract: It is well-documented that Intelligent Tutoring Systems (ITS) have a positive effect on the learning outcome across all educational levels, compared to most traditional learning methods. ITS are able to provide individualized tutoring, based on individual traits of the learner. However, research shows that ITS are still inferior to both small-group tutoring and individual tutoring. This is mainly a result of the teachers’ ability to take the affective/emotional state of the individual student into consideration when communicating. This project investigates how cognitive and affective computing technologies can be used to emulate interpersonal communication, and thereby improve the effectiveness of the tutoring systems. For this purpose, a conceptual solution will be presented, based on theory on interpersonal communication and human emotion, and state of the art cognitive and affective technologies. The findings from the project proved that it is possible to emulate the interpersonal communication process in an ATS system, to a certain degree. State of the art cognitive technologies enable processing of complex multimodal signals, similar to how communication signals are processed in the human brain, but their overall capabilities do not yet compare to the capabilities of the human brain.

Aalborg University Copenhagen A.C. Meyers Vænge 15 2450 København SV Semester Coordinator: Henning Olesen Secretary: Maiken Keller

When uploading this document to Digital Exam each group member confirms that all have participated equally in the project work and that they collectively are responsible for the content of the project report. Furthermore each group member is liable for that there is no plagiarism in the report.

ApplyingCognitiveTechnologiesand

EmulatingInterpersonalCommunicationinAffectiveTutoringSystems

LarsMyrhøjNielsenMasterthesis

InnovativeCommunicationTechnologiesandEntrepreneurshipAalborgUniversityCopenhagen

June8th,2017

2

3

TableofContents

1.Introduction............................................................................................................................51.1Problemdefinition..........................................................................................................................6

Researchquestion....................................................................................................................................6Projectdelimitation..................................................................................................................................6

1.2Readingguide.................................................................................................................................7

2.Methodology...........................................................................................................................9

3.Backgroundresearch.............................................................................................................113.1Intelligentandaffectivetutoringsystems.....................................................................................11

BenefitsofusingITS...............................................................................................................................12AffectiveTutoringSystems.....................................................................................................................13

3.2Preliminaryinterviews..................................................................................................................15Purpose...................................................................................................................................................15Interviewguide.......................................................................................................................................15Evaluation...............................................................................................................................................16Keyfindings............................................................................................................................................18

3.3Chaptersummary.........................................................................................................................19

4.InterpersonalCommunicationandHumanEmotion..............................................................214.1Whydowecommunicate?...........................................................................................................214.2Principlesofcommunication.........................................................................................................234.3Thecommunicationprocess.........................................................................................................234.4Communicationmodels................................................................................................................24

Linearmodels.........................................................................................................................................24Interactionalmodels...............................................................................................................................26Transactionalmodels..............................................................................................................................27

4.5Modesofcommunication.............................................................................................................30Verbalcommunication...........................................................................................................................30Non-verbalcommunication....................................................................................................................31

4.6Humanemotionsininterpersonalcommunication.......................................................................34Definitionandrelevanceofemotion......................................................................................................34Perceptionofemotion............................................................................................................................35Emotionalcues.......................................................................................................................................37

4.7Theoriesonhumanemotion.........................................................................................................39Ekman’sbasicemotions.........................................................................................................................39Russell’sCircumplexModelofEmotions................................................................................................39Plutchik’sWheelofEmotions.................................................................................................................40Parrot’sclassificationofemotions.........................................................................................................41

4.8InterpersonalCommunicationModel...........................................................................................434.9Chaptersummary.........................................................................................................................43

4

5.CognitiveandAffectiveComputing........................................................................................455.1Aneweraofcomputing................................................................................................................455.2Whatiscognitivecomputing?.......................................................................................................455.3Deeplearningandneuralnetworks..............................................................................................46

Deeplearning.........................................................................................................................................46Neuralnetworks.....................................................................................................................................48

5.4Affectivecomputing.....................................................................................................................495.5Analysingandmodellingemotionsincomputers..........................................................................49

Facialexpressions...................................................................................................................................50Speechprocessing..................................................................................................................................51Emotion-miningintext...........................................................................................................................52

5.6Stateoftheartcognitiveandaffectivesystems............................................................................53DeepMind...............................................................................................................................................53IBMWatson............................................................................................................................................54Developmentplatforms..........................................................................................................................55Cognitiveassistants................................................................................................................................56Otherapplicationareas..........................................................................................................................57

5.7Chaptersummary.........................................................................................................................57

6.Analysis.................................................................................................................................596.1AffectiveTutoringSystemModel..................................................................................................59

Purposeandfunctionality.......................................................................................................................59Conceptualmodel...................................................................................................................................60

6.2Replacinghumantutoring............................................................................................................62Understandingthestudentasacommunicator.....................................................................................62Interpretingthestudent’smessage.......................................................................................................64Respondingappropriately......................................................................................................................66

6.3Implementationproposal.............................................................................................................68Systemoverview.....................................................................................................................................68Useofcognitiveservices........................................................................................................................69

6.4Follow-upinterviews....................................................................................................................73Purpose...................................................................................................................................................73Interviewguide.......................................................................................................................................73Evaluation...............................................................................................................................................74KeyFindings............................................................................................................................................76

6.5Chaptersummary.........................................................................................................................77

7.Discussion.............................................................................................................................79

8.Conclusion.............................................................................................................................81

9.Futurework...........................................................................................................................83

References................................................................................................................................85

5

1.IntroductionOver the past decade, the evolution of information technology has opened a new world ofpossibilitieswithin the technology sector.Alongwith vastly improved computer algorithms, thecomputationalpowerhasnowreachedalevelofsophistication,thatthecomputersnowappeartohavethecapabilityofactuallythinking.Theyareabletounderstandcomplexinformation,andlearn,reason, and act upon that information. Some systems can take contextual and environmentalfactorsintoaccount,andevenadaptandevolveovertime(Raghavanetal.,2016).Asaresultoftheseadvancementswearenowconsideredtobeenteringathirderaofcomputing,thecognitiveera (Kelly & Hamm, 2013). In 2011, IBM’s cognitive supercomputer,Watson, beat two humanchampions in the game show Jeopardy (Jackson, 2011). Five years later, in 2016, DeepBlue’sAlphaGodefeatedtheWorld’sbestplayerofthegameGo,which iswidelyconsideredthemostcomplexboardgameinventedbymankind(Hassabis,2016).Thesearejusttwoexamplesofwhatispossiblewithcognitivecomputing,but ithasthepotential torevolutionisetechnologyacrossallsectors(Pereiraetal.,2015).Whiletechnologyhasbeenusedtosupportlearningfordecades,adaptiveandinteractiveprogramshavenowbeendeveloped,whichcancanprovideadvancedindividualisedtutoring.Thesesystemsare known as Intelligent Tutoring Systems (ITS), and are capable ofmodelling and acting uponlearners’individualtraits,suchaspsychologicalstates,learningcharacteristics,needs,andpaceoflearning.Byadaptingtotheindividualstudent,ITSareabletoguidestudentsthroughdifferentstepsin problem solving, and offers a number of advantages over traditional classroom teaching.NumerousresearchershaveinvestigatedtheeffectITShasonthelearningoutcome,andfoundthatusingITSisgenerallyassociatedwithgreatlyincreasedlearningoutcomes,comparedtoteacher-ledlarge-groupinstructions,non-ITScomputer-basedinstruction,andtextbooksorworkbooks(Kulik&Fletcher,2015;Maetal.,2014;Steenbergen-Hu&Cooper,2014;VanLehn,2011).Thisresearchdid,however, also conclude that small-group and individual human tutoringwas themost effectivelearningmethod,thushighlightingtheimportantrolesofteachersandpedagogicinlearning.Thereasonfor thesuperiorityofhumantutoring, isa resultof theteachers’abilities to identifyandrespondtothestudent’saffectivestate(Mao&Li,2010).Astudent’semotionalstatehavegreatimpact on e.g. information processing, decision-making, and creative problem solving, andsubsequentlyaffectthelearningeffectiveness(Erez&Isen,2002).Thus, thekey toenhancing thecapabilitiesandeffectivenessof ITSs, itwidely considered tobeenablingthesystemstoadapttheircommunicationtothestudent’semotionalstate,inthesamewayhumanteachersdo(Sarrafzadehetal.,2008).SuchsystemsareknownasAffectiveTutoringSystems (ATS). By detecting emotional states, responding to these states, generating relevanttutoring strategies, and communicating with affect, ATS aim to help regulate the student’semotionalstate,andachievebetterlearningoutcomes.ATSemployssocalledaffectivecomputing,

6

asubsetofcognitivecomputing,whichreferstocomputingrelatedto,arisingfrom,orinfluencingemotions(Picard,1997).Theoverarchingend-goalofaffectivecomputingisforhumanstobeableto interact and interfacewithmachines as richly aswe interactwith eachother (Cowie, 2005).Affective computing utilises technologies such as visual recognition, speech recognition, andphysiologicalsignals,torecogniseandinterprethumanemotions.Theeducationalsector,however,hasnotyetbeenrevolutionisedbycognitiveandaffectivecomputingsolutions,butitisanareainwhichithasagreatpotentialuse.Thisprojectaimstoinvestigatehowcognitiveandaffectivecomputingcanbeusedtoimprovetheinterpersonal communication between teachers and students, and subsequently affect theeffectivenessofATSandthelearningoutcome.

1.1ProblemdefinitionWithinhumancommunication,cognitiveandaffectivetechnologies,andeducationallearning,thereareplentyofinterestingtopicstoresearch.Belowisapresentationofthefocusofthisproject,informoftheresearchquestion,andaprojectdelimitation.ResearchquestionBased on the preliminary research presented above, this project aims to answer the followingresearchquestion:

How can cognitive computing technologies be used in Affective Tutoring Systems to

emulateinterpersonalcommunicationbetweenstudentsandteachers?

ProjectdelimitationDuetotimeconstrainsandlimitedresources,therearecertainlimitationsastowhatthisprojectwill include.Theemphasisof thisproject ison interpersonalcommunicationandtheunderlyingcommunicationprocesses,andhowhumanscommunicate.TheaimistoinvestigatehowthiscanbeemulatedwithinanATS,however,theareaofHuman-ComputerInteraction(HCI)willonlybelightlytouchedupon,andnotinvestigatedthoroughly.Another relevant topic which is only discussed superficially, is that of learning processes andlearningtheories.Theseresearchareascouldprovebeneficialinlaterstagesorfutureworkoftheproject,buttheywillnotbeconsideredatthisstage.WithafocusontheresearchonhowcognitivecomputingcanbeusedwithinanATS,itisnotwithinthe scope of this project to develop a functional ATS. The focus here is on the conceptualdevelopment,however,cognitivetechnologieswhichcouldbeusedtobuildafunctionalATS,willbediscussed.

7

1.2ReadingguideInthisintroductorychapter,preliminaryresearchwithintheareasofcognitivecomputingandITSwas used to define and justify the research question. Chapter 2 presents the methodologicalapproachoftheresearch.Itdescribesthedifferentstagesoftheproject,themaincomponentsofthereport,andthetopicsthatwillbeaddressed,toproperlyevaluatetheresearchquestion.Inchapter3,additionalresearchwillbemadewithinthefieldsofITSandATS,bypresentingthearchitecturalcomponentsofsuchsystems,aswellasamoredetaileddescriptionofthebenefitsofusingtutoringsystems.Furthermore,interviewswillbemadewithanumberofuniversitystudents,tosupporttheresearchfindings.Inchapter4,thefirstgeneraltopicofinterestisaddressed,thatofinterpersonalcommunication.Here,themostinterestingaspectsofinterpersonalcommunicationwillbeinvestigated,withtheintentionofconstructingamodelwhichcanbeusedforfurtheranalysislaterintheproject.Chapter5 addresses the second major element of the project, cognitive and affective computing. Thisincludes amore thorough description of the two, and some of the underlying processeswhichenables cognitive computing. Furthermore, the current stateof the artwill be investigatedandpresented.Themainanalysiswillbepresentedinchapter6.AnATSmodelwillbepresented,whichwillbediscussedinrelationtotheinterpersonalcommunicationmodelconstructedinchapter4.Toaidtothisdiscussion,andaddamorepracticaldimension,aprototypeofanATSwillbedeveloped.Boththeseaspectsofthediscussionwillbeaddressedfromthestudent’sperspectiveaswell,withfollow-upinterviewswiththestudents.Chapter7willdiscussthemostevidentresultsfromtheanalysis,inrelationtotheviabilityhereof,andaddressanyshortcomingsthatmightsurface.Themainconclusionswillbepresentedinchapter8,includingasummaryofthefindingsfromtheproject,andananswertotheresearchquestion.Finally,chapter9willdiscussideasforfutureworkontheproject,basedontheconclusionsandtheprojectasawhole.

8

9

2.MethodologyTogainadetailedunderstandingofITSandATS,additionalresearchwasmadewithinthesetwofields.Thisresearchedaimedtoelaborateuponwhatsuchsystemsentail,howtheyarebuilt,whattheir purposes and benefits are, and how the outcome of using them actually compares totraditional forms of learning. Four students from Aalborg University Copenhagen were theninterviewed,togainafirsthandimpressionoftheirlearningexperiencesandpreferences.Itwasalsoofinteresttoinvestigatetherelationshipbetweenthestudentsandtheirteachers,evaluatetheir individual preferences for interpersonal interaction and feedback, and to explore whichspecific elements and functionalities to include in an ATS. The students were interviewedindividually,basedonasetofpre-definedopen-endedquestions.Before analysing how computers can emulate human communication, it is relevant to fullyunderstand the different aspects of interpersonal communication. Thus, topics such ascommunication elements and processes, verbal and nonverbal communication signals, anddifferent communication theories were investigated. Additional emphasis was then put on theeffectand functionalityofemotionswithin interpersonal communications, tobetterunderstandhowemotionsignalsandemotionalstatesaffectthecommunicationprocess.Basedonresearchwithintheseareas,aninterpersonalcommunicationmodelwasconstructed,illustratingthemostrelevantcomponents.Thesecondmajortopicofinterestinthisprojectwasthatofcognitivecomputing.Thistopicwasthoroughlyresearched,inregardstoboththeprocessesbehindcognitivecomputing,itscurrentandpotentialuse,andthecurrentstateoftheartofcognitivetechnologies.Duetotheeffectemotionshasonbothlearningandinterpersonalcommunication,affectivecomputingwasalsoinvestigated.BasedonthisextensiveresearchonbothITSandATS,interpersonalcommunicationandcapabilitiesof cognitive technologies, a conceptualmodel of an ATS were proposed. Thismodel took intoconsiderationallaspectsofthetheoreticalresearch.Basedonthisconcept,asecondmodelwasproposed, providing a concrete example of how existing cognitive services could be used toimplementasmall-scaleATS.TheATSmodelwerethenevaluatedfromtwodifferentperspectives.Firstofall, itwasanalysedaccording to the previously constructed interpersonal communication model, and discussed inregards to which extent the system can emulate individual human tutoring. Secondly, theconceptualmodelwas discussedwith the same four university students thatwere interviewedearlierintheprocess,togaintheirviewuponsuchasystem’sviability,valuecreation,andpotentialapplication in their daily learning environment. Similar to the first round of interviews, thediscussionswereindividual,andbasedonanumberofpre-definedquestions.Beforeconducting

10

theinterviews,however,thestudentswerepresentedtotheidea,withathoroughexplanationoftheconceptualmodel,whichwasthenusedasastartingpointforthediscussion.Basedonthefindingsfromtheanalysis,theprojectwasdiscussed,andtheresearchquestionwasevaluated.Themethodologicalapproachisillustratedinfigure1below.

Figure1:Illustrationofthemethodologicalapproach.

11

3.BackgroundresearchToproperly understand thedifferent elements of the researchquestion, additional backgroundresearchwillbemadewithinanumberofrelevantareas,consistingofbothprimaryandsecondaryresearch. This chapterwill investigate intelligent and affective tutoring systemsmore in depth,before discussing the general attitudes and needs of students, in regards to their learningexperiencesandcommunicationpreferences.

3.1IntelligentandaffectivetutoringsystemsFor decades, computers have been used to replace or support learning across a variety ofeducationalsituations.Twoof theearliestso-calledComputer-Assisted Instructionsystems(CAI)areSCHOLAR(1970)andBIP(1976).SCHOLARposed,answered,andevaluatedlearners’questionsandanswers,whileBIPwasusedtoassignprogrammingtaskstostudents,basedontheirindividuallearningneedsandcompetencies.Sincethen,learningtechnologieshavedevelopedintoadaptiveandinteractivecomputerprograms,whichcanprovideadvancedindividualisedtutoring(Maetal.,2014). This individualisation is created by modelling the learners’ individual traits, such aspsychologicalstates,learningcharacteristics,needs,andpaceoflearning.Today,thesesystemsareknownasIntelligentTutoringSystems(ITS).ITScanadapttotheindividualstudent,andguidethemthroughdifferentstepsinaproblemsolvingprocess,typicallyinformoffeedback,hints,orerrormessages (Steenbergen-Hu & Cooper, 2014). Using ITS has numerous benefits over traditionalclassroom teaching, in the sense that they are always available, non-judgemental, and providetailoredfeedback(Sarrafzadehetal.,2008).ITS incorporates numerous theories from different scientific fields, such as learning sciences,cognitivesciences,pedagogy,psychology,mathematics, linguistics,andartificial intelligence,andaredesignedtofollowthepracticesofexperthumantutors(Graesseretal.,2011).Throughouttheyears,ITShavebeendevelopedmainlywithinmathematicallygroundedacademicsubjectssuchasalgebra,physics,economicsandstatistics,buthavealsobeenappliedine.g.nursing,medicine,law,programming,andmilitarytraining(Irfan&Gudivada,2016;Kulik&Fletcher,2016;Maetal.,2014;Steenbergen-Hu&Cooper,2014).ITSareconsideredtoconsistoffourconceptualcomponents,whichmustbeimplementedwhendesigningsuchsystems,asstatedbySottilareetal.(2013).Firstofall,thesystemmustcontainaninterfacewhichthelearnercanusetocommunicate,andthroughwhichthetutoringsystemcanpresentandreceive information.Theother threecomponentsare threedifferentmodelswhichmustbeincorporatedinthebackboneoftheITS;adomainmodelrepresentingtheknowledgethatmust be learned, a student model representing the student’s knowledge, and a tutor modelrepresentinginstructionalstrategies.

12

Maetal.(2014)elaboratesonthefunctionalitiesofITSinrelationtotheseinstructionalstrategiesandthedifferenttutoringfunctionsthatthesystemcanhave:Presentinginformationtobelearned,askingand/oransweringquestions,assigningtasks,providingfeedbackorhints,orinotherwaysprovoking cognitive, motivational or metacognitive change. These functions must then beaddressedandcustomisedbasedonthestudent’spsychologicalstate,bytakingintoconsideratione.g.knowledge,motivations,andemotions.BenefitsofusingITSManydifferentresearchershavedocumentedthebenefitsthatITScanhaveonlearning.TwoofthemostcomprehensiveanalyseswereconductedbyMaetal.(2014)andSteenbergen-HuandCooper(2014),whichwillbepresentedinthissection.Maetal.(2014)conductedareviewof107differentITSresearchpapers,wheretheyinvestigatedthe learningoutcomeof ITS compared toother traditional formsof learning; teacher-led large-group instruction, small-group instruction, non-ITS computer-based instruction, textbooks orworkbooks,andindividualisedhumantutoring.TheyfoundthatusingITSwasgenerallyassociatedwithgreatlyincreasedlearningoutcomes,comparedtoteacher-ledlarge-groupinstructions,non-ITS computer-based instruction, and textbooks orworkbooks. However, the ITS did not have asignificant effect compared to neither individualised human tutoring nor small-group humantutoring.RegardlessofhowtheITSwasused,i.e.whetheritwasusedasaprimaryeducationaltool,supporting teacher-led instruction, supplementingafter-class instruction,oraidinghomework, itprovedtohavesignificantpositiveeffectsacrossalluses(Maetal.,2014).Theseresultswerefoundacrossalllevelsofeducation.AsimilarreviewwasconductedbySteenbergen-HuandCooper(2014),however,theirfocuswasexclusivelyonhighereducation.Theyanalysed theuseandeffectof22different ITSoncollegestudents,andtheirfindingscorrespondedwellwiththoseofMaetal.,(2014).Steenenbergen-HuandCooperalsofoundthatITSoverallhadapositiveonthestudents’learning.Furthermore,theyfoundthatITShadanincreasedeffectonlearninginallapplicationareas,besideshumantutoring,asalsoconcludedbyMaetal. (2014).AnotheraspectofSteenbergen-HuandCooper’sresearchdocumentedthatthelearningoutcomesdidnotdiffersignificantlyfordifferentsubjectdomains,butthatITSdidimprovelearningacrossallsubjects.Basedonpreviousresearchmade,theyarguedthatITShavealargereffectoncollegestudentscomparedtolowerlevelsofeducation.SimilarresearchwasmadebybothKulikandFletcher(2015)andVanLehn(2011),againwithsimilarfindings.However,bothmadeinterestingadditionalfindings,suchastherobustnessofITSacrosstime, place, and educational setting, and that the learning effects were reduced in poorlyimplementedITS.

13

TheanalysespresentedbyMaetal.(2014),Steenbergen-HuandCooper(2014),VanLehn(2011),and Kulik and Fletcher (2015) all provided uniform results. This is due to the adaptation andincreased interactivity that ITS provides, compared to other forms of traditional classroominstruction and learning.More specifically, the greater immediacy of feedback,more response-specificfeedback,greatercognitiveengagement,increasedopportunitiesforpracticeandfeedback,more learner control, and individualised task selection, all contribute to ITS improved learningoutcomes(Maetal.,2014).DespitethegenerallyprovenincreasedeffectofITS,allanalysesshowedthathumantutoringwasthemosteffectivelearningmethod,thushighlightingtheimportantrolesof teachers and pedagogic, and the interpersonal relationship between teachers and students.AccordingtoMaoandLi(2010),humantutors’superioritytoITS,isduetotheteachers’abilitytoidentifyandrespondtotheindividualstudent’saffectivestate.ThekeytoenhancingthecapabilitiesandeffectivenessofITS,itwidelyconsideredtobeenablingthesystemstoadapttothestudent’semotionalstate(Sarrafzadehetal.,2008).AffectiveTutoringSystemsIn recent years, the ITS have developed into systems capable of adapting to the cognitive andaffectivestateofthestudentsinthesamewaythathumantutorsdo(Alexander,Sarrafzadeh&Hill,2008).ThesetypesofITSareknownasAffectiveTutoringSystems(ATS).Someresearchershavereferred to such systems as Emotion-Sensitive Intelligent Tutoring Systems (EITS), but from thispointonward,thisprojectwillusethetermATS.ATScanhelpavoidsomeofthenegativeemotionalstatesinlearningenvironments,whichcanhavegreatimpactsonthestudents’cognitiveprocesses,suchasinformationprocessing,communicationprocessing,decision-makingprocesses,andcreativeproblem-solvingprocesses.Subsequently,thiswillaffecttheeffectivenessoftheirlearning(Erez&Isen,2002).Negativeemotionalstates,suchasanxiety,frustration,boredom,andlossofconcentration,arealsoexperiencedinITS,whichcanbea result of amismatch between the individual student’s character and learning needs, and thefunctionality and adaptability of the ITS (Malekzadeh, Mustafa & Lahsasna, 2015). Thus, it isimportanttohelpregulatingthestudent’semotionalstate,andensureincreasedengagementanda generally more positive attitude towards learning, thereby preventing or regulating negativestates,toachieveabetterlearningoutcome.Bydetectingemotionalstates,respondingtothesestates,generatingrelevanttutoringstrategies,andcommunicatingwithaffect,thisisexactlywhatATSaimtodo.Today,sensorsandinputdevicescapableofemotiondetectionaremoreaccessibleandavailablethaneverbefore,butaccordingtoBahreini,NadolskiandWestera(2016),therecentdevelopmentsinthesedevicesarestillunderexploited.Theystatethatsuchsensorsoffergreatopportunitiesformanye-learningapplications,includingATS.Besidesthedevicesalreadybeingavailabletostudents,theyaddthepossibilityofgatheringaffectiveuserdataunobtrusively.

14

Camerasareconsidered tobe tomostunderexploitedsensor,and it isbymanyconsidered thechannelwhichcanextractthemostextensiveandaccurateaffectinformation,duetorecognitionoffacialexpressions(Bahreini,Nadolski&Westera,2016).AccordingtoAlexander,SarrafzadehandHill(2008),therearetwomainmethodswhichcanbeusedto identifytheaffectivestateofusers.Thefirstmethodisdetectingemotionsbasedonphysicaleffects, by using e.g. physiological signals, heart-rate, gestures, or facial expressions. The othermethodisconcernedwithwhatactuallycausesapersontofeelacertainemotion,bytakingintoconsiderationfactorssuchaspersonality,goalsandpreviousinteractions.Themostoptimalwayofidentifyingstudents’emotions,however,istocombinethesetwomethods(Alexander,Sarrafzadeh& Hill, 2008).Malekzadeh,Mustafa and Lahsasna (2015) further elaborate, and state that it isequallyimportanttodeterminehowtoregulatetheemotionalstate,andrespondappropriately.ThefollowingsectionwilldescribehowthisisdoneinATS.ATSarchitecturalmodelBesidesthefourdifferentcomponentsoftraditionalITS(interface,domainmodel,studentmodel,andtutormodel),ATSsystemarchitecturesalsohaveanemotioninformationprocessingmodule,whichisusedinconjunctionwiththestudentmodel.Figure2illustratesthearchitectureofanATS,basedonMalekzadeh,MustafaandLahsasna(2015)andSarrafzadeh(2008).

Figure2:ArchitectureandcomponentsofanAffectiveTutoringSystem.

ThestudentinteractswiththeATS’sinterface,whichiswherequestionsareaskedbythestudentandrespondedtobythesystem,butitisalsointhismodulewhereinformationabouttheemotionalstateislogged,beforebeingsenttotheemotioninformationprocessingmodule.Theprocessingmodule then analyses this information and, based on the emotions detected from variousmodalities (e.g. physiological signals, facial expressions, and eye gaze), a weighing algorithmdeterminesanoverallemotional state.This state isoftendescribed inavalence-arousalmodel,whichisa2D-modelusedforemotionmodelling.Thestudentmodelalsoanalysesthestudent’sinputinrelationtothepedagogicalstate(informationsuchasknowledgelevelandlearningspeed),andevaluatesthemaccordingtotheemotionalstate.Beforeconstructinganappropriateresponsetothestudent,thetutoringmoduleselectsanappropriatetutoringstrategybasedonthestudentmodel.Thisstrategythendeterminestheresponse,bydrawingrelevantcontentandmaterialfrom

15

thedomainknowledgemodel.Thefeedbackisthenpresentedtothestudentintheinterface.Thisfeedbackcanbebothdomaindependent(relatedtocoursecontent)ordomainindependent(byshowingempathyorencouragement).

3.2PreliminaryinterviewsInthissection,agroupofuniversitystudentswillbeinterviewedinregardstotheirexperiencesandattitudestowardstheireducationalenvironment,withafocusontheinteraction,relationship,andcommunicationbetweenteacherandstudent,inbothclassroomteachingandprojectsupervision.PurposeThepurposeofthestudyistogatherprimarydataonboththestudents’educationalneedsandlearningexperiencesthroughouttheirstudies,andtoevaluatehowthisinformationcorrelateswiththefindingsfromsection2.1.Morespecifically,theinterviewsaimtoelaborateonthefollowingareasofinterest:

1) Thegeneralstudentexperiencesfromclass-roomteaching,groupwork,andsupervision.

2) Thenatureoftherelationshipbetweenstudentandteacher.3) The individualneeds in regards topersonal interactionand feedback,and towhatextent

theseneedsarefulfilled.

4) Technologiesandmethodscurrentlyusedbythestudentstosupportlearning.

Theaimistogainfurtherinsightsontheaspectofindividualandpersonalisedfeedback,inrelationtobothinteractionsandinterpersonalcommunicationbetweenteacherandstudents.InterviewguideThe interviews will be semi-structured, in the sense that the above mentioned areas will bediscussed,basedonanumberofpre-definedopen-endedquestions.FourstudentsfromAalborgUniversityCopenhagenwillbeinterviewedindividually,andtheinterviewprocessitselfwillbeanopenconversation.Basedonthefocusareas,thefollowingquestionsareconstructedandusedasastartingpointfortheinterview:

1) What are your general opinions and experiences from class-room teaching and project

supervision?

2) Howwouldyoudescribeyourrelationshipwithyourteachers,andhowdoesthisrelationshipaffectyourlearningexperience?

3) Howwelldoyoufeelliketheteachersknowyou,anddotheycommunicateappropriatelyto

youasanindividual?

4) Howdoyouprefertodiscussandreceivefeedbackinrelationtoreadingmaterial,lectures,

tasksandassignments?

16

5) Aretheresufficientresourcesforoptimalsupervisionandfeedback,ordoyoufeelthereisa

needforadditionaltutoring?

6) Howdoyouusetechnologytosupportthemandatorycoursecontent,projectwork,

andsupervision?

The following sections will describe the findings from the conducted interviews, as well ashighlightingsomeofthemostnoteworthyresponsesfromthestudents.EvaluationThissectionwillevaluatetheinterviewresponsesinregardstotheareasof interestsmentionedabove.Eachaspectwillbediscussedinageneralsense,andsupplementedwithrelevantquotations,beforepresentingthekeyfindingsoftheinterviews.Generalstudentexperiencesfromclass-roomteaching,groupwork,andsupervisionAllfourintervieweeswerestudyingatAalborgUniversityCopenhagen(AAU),andarethusworkingunder the AAU model, which revolves around project-oriented and problem-based learning(POPBL).Theirexperienceswiththismodelweremostlypositive,astherewasageneralconsensusabouttheincreasedbenefitsfromprojectwork.Especiallythefreedomtochooseyourownprojectfocus increased the engagement and possibilities of specialisation in topics of interests, werehighlightedbythestudents.Furthermore,theyallrespondedpositivelyabouttheuniversity’sfocusonworkinginprojectgroups,asitallowsforconstructiveandinspirationaldiscussions.ThenatureoftherelationshipbetweenstudentandteacherThereexistsanopenandfriendlyrelationshipbetweenthestudentsandtheirteachers,andtheteachersarealwaysapproachable.ExperiencesfromothereducationalprogrammesrevealedthatthisisratheruniqueforAAU,andstudentsfromotheruniversitiesandprogrammesdonothavethesamepossibilities.However,ascontinuouslymentionedbytheinterviewees,thestudentsarenottakingfulladvantageofthisopportunitytogetfeedbackfromtheteachers.Oneintervieweeattributedthistotheauthorityoftheteachers,whileanotherstatedthatitcanalsobearesultofnotwantingtoexposeone’slackofknowledge:“Ifthereisanexpectationthatyoushouldhaveacertainskillsetorknowledge,youdonotalwaysapproachtheteachers”,andelaboratedthat“…itcanbequiteembarrassingtorevealyouractualskilllevel”.Anotherintervieweeconfirmedthisandsaid that“Whensitting in frontofyourprofessor… I cannothelp thinking that it is somesortof

presentation.Youwanttoshowthatyoucanasktherightquestions,andhavetherightanswers.”

Oneofthereasonsfortheaccessibilityofhelpfromtheteachers,istherelativelysmallsizeoftheprogramme in which the interviewees were enrolled. The number of students enrolled in aprogramme has a huge impact on different aspects of their relationship to the teacher, andsubsequentlythelearningoutcome.Basedontheinterviewees’experiences,itbecameclearthat

17

programmesandcourseswith100+enrolledstudents,hadalmostnopersonalcommunicationwithandfeedbackfromtheteachers.Onestudentstatedthatquestionswereonlyansweredextremelysuperficially, and that teachers responding to e-mailswas a rare occurrence. In smaller groups,however,there ismoretimefor individualpresentations,discussionparticipation,andelaboratequestionanswering.Oneintervieweedescribedthisasa“quality-quantitytrade-off”.

Anotherbenefitofbeingasmallergroupofstudents,isthattheteachercanmoreeasilyadaptthecontentofthecoursetofitthesemesterprojects,needs,andinterestsofthestudents.Italsomakesheavycoursematerialmorerelatable,useful,anddigestible.Asaresults,theintervieweesfeltmoreengagedintheclass,andonestudentstatedthathewasmoremotivatedtoprovehimselftotheteacher, and that smaller groups “forced concentration, focus, responsibility and participation”.Anotherstudentstatedthat“theteachersaregreatatreadingthestudents.Thisisbecausewearesofew.Youjustconnectinadifferentway.”,highlightingthemoreintimaterelationshipthatexistbetweentheteacherandsmallergroupsstudents.TheindividualneedsinregardstopersonalinteractionandfeedbackTherewas a general consensus among the interviewees, that their options for supervision andfeedbackwasfulfilled.Thatbeingsaid,anumberofinterestingcommentsweremade.Firstofall,wastheimportanceofthegroup-orientedwork.Intheirgroups,thestudentshaveamuchmoreinformal relationshipcompared towithin theclass roomwitha teacherpresent.Thisallows forgettinganswerstosomeofthequestionsthattheywouldotherwisehesitatetoasktheteacher,asdiscussedinabove.Oneoftheinterviewedstudentsstatedthatgroupworkanddiscussionalsoletyouaskquestionswithoutsoundingstupidorworryingaboutwastingthetimeofallotherlectureattendants.Hecontinued,andsaidthatitalsomakesiteasierforshyornervousstudentstogetanswerstotheirquestions.Anotherintervieweeelaboratedonthis:“…anessentialpartofit[groupwork,ed.] istobeabletosparwitheachother.Youaremore inclinedtoaskquestions,andnot

afraidtosoundstupid,whichyoualsolearnalotfrom”.Whilesmallerandlessessentialquestionsarediscussedwithinprojectgroups,theteachersareusedmoreforgeneralsupervision.Asmentionedabove,smallerclassesaremoreconsideratethanlargerclasses,regardingadjustingthecontenttofittheindividualstudents.Onestudent,however,claimedthathestillthinksthat“theindividualneeds,knowledge,andlearningspeedisnotalwaysconsidered,especiallyinregards

totheeducationalbackgroundsofthestudents.”

Duringclass,thestudentsgenerallybenefitfromfeedbackfrombothfellowstudentsandteachers.However,thereseemedtobeslightlyopposingviews,assomementionedfeedbackfromfellowstudentsasoftenbeingsuperficialandunconstructive,whileothersfoundthefeedbackbeneficialandrelevant.

18

Inregardstopersonalassistanceandfeedback,differentinterestingissuesrosefromtheinterviews.Itbecameevidentthathavingone-on-oneindividualtutoring,washighlybeneficialforthestudents,whichtheyfoundverymotivatingandit“hadahugeimpactinregardstoimprovingandchallenging

yourself”.Thisinterpersonalinteractionwiththeteacherwasrarelypossibleduetothenumberofstudents in class, compared to the time requiredof the teacher to provide sufficient individualfeedback.Onestudentgaveanexamplehereof:“Inmorepracticalclasses,suchasprogramming,

youwouldbenefitfromhavinganadditionalteacher.Wewere30students,andourteacherwas

unabletosufficientlyhelpallofusduringclass.”TechnologiesandmethodscurrentlyusedbythestudentstosupportlearningIn addition to the support offered by teachers/supervisors and fellow students, as in the twoprevious sections, the interviewees all usedmodern technology to assist their learning. Searchengineswereoftenused to findanswers to simplerquestions,whileencyclopaediasandonlinedatabases were used for more scientific literature. One student specifically mentioned theusefulnessofonlineeducationalvideos,whereexpertselaborateonacertaintopic.Hearguedthatitsometimeshelpstohearsomeoneelserephrasingthecoursecontent,andthatitcanbebeneficialtolistentoexpertsexplainingaspecifictopicin-depth,whichwasonlysporadicallymentionedinthecourse.Anotherstudentexplainedhowtheuniversityestablishesonlinediscussionboards,andencourages students touse these if theyhaveanyquestions,however,he stated that studentsneverusedthisforum.KeyfindingsBased on the above mentioned responses, the following list shows the key findings from theinterview:

- HighsatisfactionwiththePOPBLmodel,andgreatbenefitsfromworkinginprojectgroups.- Mostuniversityprogrammesdonothaveanopenrelationshiptotheirprofessors,andtheir

resourcescanbelimitedduetothelargenumberofstudentsenrolledinacourse.- Smallerclassesenablemoretailoredcontent,whichmotivatesstudentparticipation,and

makescoursematerialmorerelatable.- Thereisatrade-offbetweenthequalityinpersonalfeedbackfromteachers,andthenumber

ofstudentsenrolledinacourse.- Interpersonalcommunicationandfeedbackfromtheteacherhadagreatimpactonstudent

motivationandlearningoutcomes,especiallyinmorepracticalclasses.- Individual needs, knowledge, and learning speed is not always considered within the

classroom.

19

- Despitehavingtheopportunitytoalwaysreachouttotheteachers,thestudentsfeelcertaininhibitionsrelatedtotheexpectationsfromtheteacher,andhaveafearofdemonstratingalackofknowledge.

- The informal relationship between students in project groups, allowed for discussion ofissuesandquestionswhichwouldnotbediscussedwiththeteacher.

- Differenttechnologies,suchassearchenginesandscientificdatabases,areusedbyallthestudentsassupplementarylearningresources.

Asbecameevidentfromthepreviouslydiscussedacademicresearch,thatusingATShasapositiveeffectonthelearningoutcomeregardlessoftheeducationallevel.TheresearchalsoshowedthatITSandATSaremosteffectivewhencomparedtolargerclasses,ratherthansmall-grouphumantutoring or individual tutoring. The interviews conducted confirmed these observations, which,despiteageneral satisfactionwith theeducationalenvironment,provedtheneed foradditionaltutoringinhighereducation.Personal feedback is considered an extremely important aspect of learning, andwhile personalfeedbackandpartiallyadaptedcoursecontentisappliedinsmallerclassesatAAU,thestatewithinlargerclassescannotbeconsideredpersonalatall,andisfarofftheoptimallearningenvironment.ByimplementingATS,thebarrieroffearingtoaskquestionsanddemonstratealackofknowledge,canbeeliminated.Thiswouldhelpmotivatingstudents, increase their focusandconcentration,assists their learning, and improve the overall effectiveness of their education. Furthermore, itwould assist studentswho are shy, nervous, or asocial, andmake themmore integrated in theclassroom.ATSscouldhelpbytakingintoconsiderationtheindividualstudent’sknowledge,learningspeed,andlearningneeds,andprovidequestionsandanswerstailoredtoeachindividual,nomatterthesizeoftheclass.However,itappearsthatATSwouldbeespeciallyhelpfulinprogrammesanduniversitieswheregroup-workdoesnotplayasprominentaroleasitdoesatAAU.

3.3ChaptersummaryThefirsttopicaddressedinthischapterwasthatofIntelligentTutoringSystems(ITS),i.e.tutoringsystemscapableofadaptingcontentandfeedback,bymodellingtheindividuallearner’straits,suchaspsychological states, learningcharacteristicsandneeds.Researchon ITSwas thenpresented,demonstratingthebenefitsofusingITSineducation.Untilrecently,ITShavebeenlimitedinregardstoproperlyanalyseandadapttothedeepercognitiveandemotionalstatesofthelearner.Today,tutoringsystemshavebecomemoreemotionallyintelligent,andsuchsystemsaretodayknownasAffectiveTutoringSystems(ATS).Itwasbrieflyinvestigatedhowtoobtainaffectinformation,beforedescribinganarchitecturalmodelofATS.

20

Thesecondpartofthechapterinvestigatedthelearningexperiencesandattitudesamonguniversitystudents.Thiswasdonebyconductingfoursemi-structuredinterviewswithstudentsfromAalborgUniversityCopenhagen.Thestudentswereaskedabouttheirgeneraleducationalexperiences,thenatureof therelationshipwiththeir teachers,andtheirneedsforpersonal interactionwithandfeedback from teachers. The responses showed a general satisfaction with the learningenvironment,however,allintervieweesalsomentionedproblemswithintheeducationalsysteminregards to personalised content and feedback. The responses also revealed that the studentssometimes feel certain inhibitions towards approaching their teachers. It became evident thattechnologicalsolutionssuchasATS,mustbefurtherinvestigated,astheyhavethepotentialtosolvesomeoftheexistingbarriersandproblemswithinlearningeffectivenessinhighereducation.

21

4.InterpersonalCommunicationandHumanEmotionAspreviouslymentioned, researchhas shown that theaffective tutoring systemsare inferior tosmall-groupandindividualhumantutoring,duetothesuperiorityofhumans’adaptationtoaffect.Toinvestigatewhatcommunicationelementsandapproachessuchaninteractionincorporates,itisrelevanttolookatinterpersonalcommunication.Thischapterwillpresentanumberofdifferentaspectsof interpersonalcommunication,startingwithargumentsforwhycommunication is important.Then,keyelementsofthecommunicationprocessareinvestigated,beforepresentingdifferenttheoriesoninterpersonalcommunication,andhowtheseelementsareusedwithinanumberofcommunicationmodels.Thechapteralsocoversthe role and importance of emotions in the communication process, to further investigate theunderstandingandcommunicationofaffect.Finally,basedonextensiveresearch,aninterpersonalcommunicationmodelwillbeconstructedforlateruseintheanalysis.

4.1Whydowecommunicate?Generally speaking, communication is the process of transmitting information and commonunderstandingfromonepersontoanother(Keyton,2011).Thistransferofinformationandmeaningcanbeeitherdeliberateoraccidental,butwheneverapersondoesorsayssomethingandothersobserveandattributemeaningtotheaction,communicationistakingplace(GambleandGamble,2011). Cheney (2011), however, states that if the exchange of information does not lead to acommonunderstanding,thereisnocommunication.Whendiscussingthedomainofcommunication,Lane(2008)dividesit intotwodifferentdistincttypes; intrapersonal and interpersonal communication (as depicted in figure 3). Intrapersonalcommunicationiswhattakesplacewithinanindividual.Examplesofthistypeofcommunicationareinternaldialog(aconversationwithinoneself)andself-talk(internaltalkthatisspecificallyaboutoneself).Theothertype,interpersonalcommunication,involvesatleasttwopeoplewhoestablisha communicative relationship. The focus area of this project will be on interpersonalcommunication.

22

Figure3:Differentcategoriesofcommunication(Lane,2008).

Interpersonal communication is considered to be an ever-changing processwhich occurswhenpeopleestablishandinteractinacommunicativeassociation(GambleandGamble,2011).Thus,anytypeof communicationwith twoormorepeople involved is considered interpersonal. Bochner(1989)statesthatforcommunicationtobeconsideredinterpersonal,threecriteriamustbefulfilled.Atleasttwocommunicatorsmusttakepartintheinteraction;bothmustactasbothsubjectandobject;andtheiractionsmustembodyeachother’sperspectivestowardsthemselvesandtheother.Hefurtherelaboratesthateachcommunicatorisconsideredtobe“bothaknowerandanobjectofknowledge,atacticianandatargetofofanother‘stactics,anattributerandanobjectofattribution,

acodifierandacodetobedeciphered”(Bochner,1989,p.336).Whenengaged in interpersonal communication,peoplehave thepower toaffecteachotherasinterconnectedpartners ina relationship, in termsofboth creating,maintaining, anddissolvingrelationships (Lane,2008).Thequalityofan interpersonal relationshipcanbedescribedalongacontinuum ranging from impersonal communication to intimate communication (Gamble andGamble,2011).Similarly,Lane(2008)alsodescribesinterpersonalcommunicationasacontinuumfromimpersonaltopersonalcommunication,eventhoughitcannotbederivedfromfigure3itself.Impersonalcommunicationisconsideredtobestereotypical,i.e.whenthereceiveristreatedasan“object” or imposed with a classic “role”, while personal communication occurs when theinteraction is basedon theuniqueness of theother person.As a result, themorepersonal theinteraction is, the more interpersonal the relationship is considered to be. When engaging ininterpersonalcommunication,thegoalistotreatandrespondtoeachotherasuniqueindividualswithadistinctrelationalculture(Miell,1984).Rubin,PerseandBarbato(1988)identifiedsixmotivesforinterpersonalcommunication;pleasure(fun),affection(caring),inclusion(sharingfeelings),escape(fillingtimetoavoidotherbehaviours),relaxation (unwinding), and control (power). According to Lane (2008), if interpersonalcommunicationfulfilsthesemotives,wearelikelytobesatisfiedwithourinteractions.However,

Communication

Intrapersonal

Internaldialogue

Self-talk

Interpersonal

Impersonal

Personal

23

howapersonapproachesandprocessesinterpersonalcommunicationisdeterminedbycontext,culture,gender,theenvironment,roles,needs,history,andindividualgoals(GambleandGamble,2011;Lane,2008).

4.2PrinciplesofcommunicationItisgenerallyunderstoodthattherearecertainprinciplesofcommunicationwhichareimportantto consider when communicatingwith others, despite their cultural background. The followingdescriptionsarebasedontheprinciplespresentedbyFujishin(2008)andLane(2008).Firstofall,communicationistransactional,whichmeansthatwhenparticipatinginface-to-faceconversation,the participants sends and receives messages simultaneously. Secondly, communication is

irreversible inthesensethatonceamessageisuttereditcannotbetakenbackorunsaid,andisthus always being interpreted by the receiver - a principle which is particularly applicable incomputer-mediatedcommunication,whereamessagecanbepermanentlystored.Furthermore,communicationisanongoingprocessasitisdifficulttodistinguishbetweenthebeginningandendof communication, as it can begin as either an intrapersonal process, or based on a previousinteraction with either the subject or someone/something else. The fourth principle is thatcommunicationisconstantanditisimpossiblenottocommunicate,asbehaviourisperceivedandinterpretedregardlessofthesender’sintentiontocommunicatethesesignals.Whetherapersonissilent or speaking, laughing or crying, or expressing anger or joy, he is still communicating. Inaddition,communicationis learned inthesensethatcommunicationpatternsandbehaviourareacquiredandevolvesovertime.Suchbehaviourcanbeattributedtodifferentfactorssuchasourvocabulary,gestures,touching,appearance,andhowwespeak.Finally,communicationiscreative.

Notonlyis itpossibletocommunicatecreativelywithart,musicandpoetry,butcommunicationcanalsobecreativeinrelationtohow,when,whatandtowhomwecommunicate.

4.3ThecommunicationprocessThecommunicationexchangeisoftendescribedasaprocessconsistingofanumberofdifferentelements,butcommonforallarethesenderandthereceiver(Lunenburg,2010).Thesenderistheinitiatorof thecommunicationand theoriginatorof themessage. It canbeapersonwritinganemail,apublicspeaker,anadvertisingcompany,oranyothercreatorofagivenmessagethatistobe transmitted. Before transmitting themessage, it is encoded by the sender. Encoding is theprocessbywhichthesendercomposesthemessageusingwords,symbols,orgestures.Themessageistheidea,thought,orfeeling,thatthesenderintendstocommunicatetothereceiver,whichcanbecommunicatedeitherverballyornonverbally.Themessageisthentransmittedviaachannel,i.e.themediumbywhichit iscommunicated,whichcanbeinanyforminterpretablebythehumansensorysystem,suchasface-to-faceconversation,aphonecall,anemail,orawrittenletter.Whenthemessage arrives at its destination, it is decoded by the receiver, i.e. the recipient(s) of themessage. The decoding is the mental process by which the receiver makes sense and extractmeaningfromtheinformationreceived.Whenthisinitialtransmissioniscompleteandthemessage

24

hasbeen interpreted, the receiver responds to themessage,aprocessknownas feedback.Thefeedbackcanbeverbal,nonverbal,orboth,anditisviathisprocessthatthesendercanestimatewhether or not the message has been received and understood as intended. Feedback incommunicationisdesirable,andonlyoccurintwo-waycommunication(Lunenburg,2010).Throughout the communication process, interfering forces, called noise, can distract from thecommunicationanddistortthemessage.Itcomesinanumberofdifferentforms,andcanbeeitherinternal (psychological or physiological factors) or external (environmental factors). Exampleshereof include interruptions, attitudes, and emotions. The quality of the communication isdeterminedbytheclarityinthedifferentelements,andnoiseinanyoftheelementscanreducetheeffectivenessofcommunication(Keyton,2011).Thefinalelementtoconsiderinthecommunicationprocess is thecontext,which refers to aspects suchas thephysical environment, time, culturalinfluences, gender, beliefs, values, and expectations. Communication is easier the more theparticipants’ contexts overlap,whereas the interaction becomes graduallymore difficult as thecontextsdiverge(Lane,2008).

4.4CommunicationmodelsTo simplify and visualise the relations of the elements mentioned in the previous section,communication models are used to illustrate the communication process, as well as theinterrelationsbetweendifferentcomponents.Suchmodelsareusefulinthesensethattheyenableanalysisofhowtheindividualcomponentsandtheirrelationsaffecttheoutcomeandprocessasawhole(Lane,2008).Fujishin(2008),however,arguesthatwhilecommunicationmodelsaregreatvisualisationtools,theyarenowherenearcomprehensiveenoughtodescribeeverythingaboutthecommunicationprocess.Thefollowingsectionswilldescribethreedifferenttypesofmodels;linear,interaction,andtransactionalmodels,andgiveexamplesofpopulartheoriesforeachtype.LinearmodelsThesimplestformofacommunicationmodelislinearofnature,meaningthatthecommunicationisunidirectionalfromthesendertothereceiver,andthusomitsthefeedbackorresponsefromthereceiver(GambleandGamble,2013).Anillustrationofalinearmodelcanbeseeninfigure4below.

Figure4:Illustrationofalinearcommunicationmodel,asperFujishin(2008).

25

Widelyconsideredoneofthesimplestwaysofillustratingacommunicationprocess,istheShannon-Weavermodelofcommunication,asillustratedinfigure5(ShannonandWeaver,1949).ThemodelwasdevelopedbyClaudeShannon,electricalengineeratBellTelecomCompany,in1948,andlaterpopularisedbyWarrenWeaver. Themodelwas initially developedas amathematicalmodel tomirror the functionality of radios and telephones, but has since been adopted in a number ofdifferentdomainsaswell,suchascommunicationsciences,psychology,andsocialsciences.TheShannon-Weaver model was the first to introduce the concept of noise, representing theinterferencetothefidelityofthemessage,asexperiencedinaphonecalloraradiotransmission(Fujishin,2008).

Figure5:TheShannon-Weavermodelofcommunication.

In 1960, David Berlo published his Sender-Message-Channel-Receiver (SMCR) model ofcommunication,basedontheShannon-Weavermodel(Berlo,1960).Berlointroducedanumberofdifferentfactorsaffectingthefourmainelementsofthemodel,inrelationtotheefficiencyofthecommunication.Thismodelcanbeseeninfigure6below.

Figure6:Berlo’sSMCRmodelofcommunication.

While the simplicity of the linear model is a great advantage in visualising a basic messagetransmission,andcontainsseveralofthepreviouslymentionedkeyelements,itisconsideredfartosimple todescribe thecomplexityof theactualprocesses. (GambleandGamble,2013;Fujishin,2008).

26

InteractionalmodelsFollowingthecriticismofthelinearmodels,differentbi-directionalmodelsweredevelopedovertheyears,whichbecameknownasinteractionalmodels(GambleandGamble,2013).Asthenameimplies,interactionalmodelsvisualisetheinterpersonalcommunicationprocessasaback-and-forthprocess.Figure7illustratestheelementsandinterrelationspresentinaninteractionalmodel.

Figure7:Illustrationofaninteractionalmodel,asperFujishin(2008).

An example of an interactional model, which breaks the classical one-way sender-receiverrelationship, is theSchrammmodel.ThismodelwasproposedbyWilburSchrammin1954,andtakes into consideration the interpersonal aspect, and illustrates the interaction as a circularcommunicationprocess(Schramm,1954).TheSchrammmodelcanbeseeninfigure8.

Figure8:TheSchrammmodelofinterpersonalcommunication.

Besides the two-way communication, interactional models have the strength of taking intoconsideration the feedback element. When this is taken into account, the sender adapts thecommunicationbasedonthefeedbackfromthereceiver, inordertoproperlycommunicatethemessage(Fujishin,2008).Eventhoughtheinteractionalmodelsareconsideredmoreaccuratethanlinearmodels,itstilldoesnotquitecapturethecomplexityandaccuratelyreflecttheprocessesofinterpersonalcommunication(GambleandGamble,2013).

27

TransactionalmodelsWhileinteractionalmodelsconsiderthecommunicationprocesssequentially,transactionalmodelsseecommunicationasasimultaneousprocess,asdepictedinfigure9.Thus,theparticipantsarereferred toascommunicators, rather than sender and receiver (Fujishin, 2008). In transactionalmodels, the communicators are viewed as interdependent, unlike the previously mentionedmodels,where theyare consideredas independentelements in the system.Another importantaspectoftransactionalmodels,istheinfluenceoftheindividualbackgroundsofthecommunicators,suchasculture,psychologicalandphysicalfactors,socialfactors,aswellasdemographicelementssuchasgender,andage(Lane,2008).

Figure9:Illustrationofatransactionalmodel,asperFujishin(2008).

By visualising communication as simultaneous, transactional models depicts the nature of aconversationinamuchmorerealisticmanner,comparedtobothlinearandinteractionalmodels(GambleandGamble,2013). Furthermore, themodels containall elements from theother twocategories, while also including the context of the communication, thus representing thecommunicationprocessinamuchmoreholisticway,asthecommunicatorscontinuallyinfluenceeachother(Fujishin,2008).AnexampleofatransactionalmodelwaspresentedbyStamp(1999),whichwillbefurtherelaboratedinthenextsection.Asummaryofthestrengthsandweaknessesofthethreetypesofcommunicationmodelscanbeseenintable1below.Model Communicationexamples Strengths WeaknessesLinear Telephone,television,radio,

email.Simplicity,applicabilitywithinelectroniccircuits.

Onlyconsidersone-waycommunication.Nocontextorfeedback.

Interactional Instantmessaging,individualspeeches,classpresentations.

Widerapplicability. Onlyconsidersone-waycommunication.Nocontext.

Transactional Face-to-faceconversations,andanyencounterinwhichmeaningisco-created.

Mostrealisticdepictionofinterpersonalcommunication.Includesbothcontextandfeedback.

Doesnotapplytoallformsofcommunication,e.g.broadcast,textingorposting.

Table1:Strengthsandweaknessesoflinear,interactional,andtransactionalcommunicationmodels.

28

Stamp’sGroundedTheoryModelofInterpersonalCommunicationAn interestingtransactionalmodelwaspresentedbyGlennStamp in1999,whichwasbasedongroundedtheoryexamining288researcharticlesoninterpersonalcommunicationoverthepast25years(Stamp,1999).Thiscomprehensiveandthoroughmodelisbasedonsevenmaincomponents,which were constructed on the basis of 17 sub-categories of interpersonal communication, asidentifiedfromtheresearch.Thecomponentsandtheirrelatedcategoriescanbeseenintable2,whereastherelationsbetweenthecomponentsareillustratedinfigure10.The model depicts an ongoing transactional communication process between two people, asconstructed by the seven different components. The first component, culture, is considered anoverarchingcomponent,asitaffectsallaspectsofthecommunication,bydefininge.g.identities,socialroles,norms,andthetypeofrelationshipsdeveloped.Thisculturalaspectisoneofthedriversbehindthesecondcomponent,theinternalstates.Theinternalstatesaretheinterrelatedaspectsofperception,personalityandcognition,whichdefineswhoweare,howwethink,andhowwemakesenseoftheworld(Stamp,1999).Thethreeprocessesofdeception,compliance-gaining,andself-disclosure, composes the thirdcomponentof themodel, interpersonal competencies,whichconcernsinterpersonalskillsandcompetencelevels.Besidestheseprocesses,thecomponentisalsofuelledbytheinternalstates.Thenextcomponentinthemodeliscommunicationapprehension,whichisbasedonthewillingnessorfeartocommunicate.Apprehensionaffectse.g.howwedeceiveorpersuadeanother,andisconsideredtobeamediatingcomponentbetweenthecompetenciesand themessage.The fifthcomponent,messagebehaviours, represents thebehavioursused toadvancethepreviouslymentionedcomponents,i.e.theactualverbalandnonverbalmessagessentinthecommunicationprocess.Thesixthcomponentisthecorecategory,thatisthephenomenonwhich all the other categories are related to. In this model it is considered to be theinteraction/relationshipitself.Duringaninteraction,conversationsandconflictstakesplace,anditis based on these actions that relationships are created, developed, and sustained. The finalcomponentofthemodelistheinterpersonaleffectthattheinteractionhasonbothcommunicators,and can thus be considered to be the feedback mechanism of the model. It encapsulates thereactionsandemotionsresultingfromtheinteraction,whichthenaffectstheinternalstateoftheparticipants,andsubsequentlyinfluencestheothercomponentsaswell.

29

Component Relatedcategories Descriptionandexamplesofrelatedvariables/researchtopics1.Culture Culture Howcultureaffectscommunication,andhowcommunicationdifferswithinvarious

cultures;Acculturation,informationacquisition,relationshiptypes,relationshipterms,socialidentity,intimacy,communicationproblems,communicationconstraints.

2.Internalstates Perception Thewaysinwhichthecommunicatorperceivestheworld;attitudes,attributionalprocesses,personperception,interpersonalaccounts,interpersonalexpectations.

Personality Enduringpersonalitycharacteristicsofthecommunicator,andtheireffectonthecommunicationprocess;Personalitydominance,self-esteem,gender,assertiveness,anomia,communicationstyle,individualdifferences,cognitivecomplexity,affectiveorientation,attachmentstyle.

Cognition Theinternalstateofthecommunicator;Informationprocessing,pre-encodingaspects,pre-articulatoryediting,memory,sequencingdecisions,socialcognition,motivation,knowledge,goals,cognitiveediting,plans,intent.

3.Interpersonalcompetencies

Competence Communicationskillsandcompetences;socialperspectivetaking,roletaking,socialcompetence,developmentskill,socialskilldeficits,competenceindifferentgroups.

Self-disclosure Differentfacetsofself-disclosurewithininterpersonalrelationships;level,valence,andanticipationofself-disclosure,therelationshipoftrust,affection,andreciprocitytoself-disclosure,disclosurestyles,behavioursassociatedwithdisclosure.

Deception Interpersonaldeception;deceptionasconstruct,detectionofdeception,deceptioninvariousrelationshiptypes,deceptivebehaviours,relationshipofdeceptiontootherpersonalitycharacteristics.

Compliancegaining Strategiestogaincompliancefromanotherperson,ortheneedtopersuadewithinaninterpersonalcontext;Compliance-gainingstrategiesindifferenttypesofrelationships,resistingcompliance-gainingstrategies,persuasion,selectingcompliance-gainingbehaviours,effectsofnoncompliance,sequencingofstrategies,verbalandnonverbalcompliance-gainingstrategies.

4.Communicationapprehension

Apprehension Theconstructcommunicationapprehension;effectsofcommunicationapprehension,expectationsofapprehensiveothers,non-verbalandverbalbehavioursassociatedwithapprehension,therelationshipbetweencommunicationapprehensionandself-esteem.

5.Messagebehaviours

Messagetype Differentclassesofmessagesthatareprimarilyverbalinorientation;control,affection,empathy,ridicule,narratives,harmfulspeech,concern.

Nonverbalcommunication

Aspectsassociatedwithnon-speechrelatedbehaviour.Decodingnonverbalcommunication,bodymovement,spaceviolations,illustrators,eyegaze,crowding,nonverbalencoding,motormimicry,speechrate,touch,nonverbalimmediacybehaviours.

6.Interaction/relationship(Corecategory)

Initialinteraction Communicationwithininitialinteractions;initialinteractionaloutcomes,attraction,homophily,dissimilarity,initialinteractionscripts,languagefeaturesduringinitialinteractions,interactionswithdifferenttypesofpersons.

Conversation Thenatureofconversation,conversationalcomponents,effectsonconversationsofparticulardevices;quantityofspeakingtime,conversationalgrammar,conversationalturntaking,conversationalstructure,interruptions,pauses,interactionmanagement.

Conflict Conflictswithininterpersonallife;modelsofconflict,verbalconflicttactics,managingconflict,conflictstrategies,mediation,argumentation,aggression,conflictindifferentcontexts.

Relationshipdevelopment

Communicationduringthedevelopmentofcloserelationships;socialpenetrationprocess,relationshipchange,turningpoints,idiomaticcommunicationindevelopingrelationships,relationshipescalation,impactofsocialnetworksonrelationshipdevelopment.

Closerelationships Communicationwithincloserelationships;instrumentalandexpressivecommunication,patternsofcommunication,understanding,contentthemes,relationshipsatisfaction,play.

7.Interpersonaleffect

Interpersonaleffect

Theinterpersonaleffectsofmessages,situations,orothervariables;outcomeofhelpingrequests,anxiety-arousingmessages,comfortingmessages,messageswithmultiplegoals,guiltmessages,stereotyping,consequencesofspeechstyles.

Table2:ThesevencomponentsofStamp’smodelandtheir17relatedcategories(Stamp,1999).

30

Figure10:Stamp’sgroundedtheorymodelofinterpersonalcommunication.

4.5ModesofcommunicationThis section will present the two different modes of communication, verbal and nonverbal.Furthermore,thedifferenttypesofcommunicationsignalswillbediscussedforbothmodes.VerbalcommunicationVerbalcommunicationreferstohumaninteractionthroughtheuseofwordsorgeneralmessagesin linguistic form,suchasspokenandwrittencommunication (ChandlerandMunday,2011a). Itconcernshowweusewordstocreateandconveymeaning,whichenablespeopletointeractandallowsthemto“defineandclassify;expressourbeliefs,attitudes,thoughts,andfeelings;organizeourperceptions;andtalkabouthypotheticaleventsthatoccurredinthepast,andeventsthatmay

occurinthefuture”(Lane,2008,p.126).Thisinteractionallowsfore.g.problemsolving,relationshipdevelopment,impressionmanagement,andfulfilmentofbasichumanneeds.

31

AsperOgdenandRichards(1923),thereexistsanarbitraryrelationshipbetweenwordsandthereferentsthattheyrepresent.Duetothenatureofthisrelationship,asinglewordcanbeassigneddifferentmeaningsdependingonthecontext,butdifferentwordscanalsobeusedtodescribethesameentity(Fujishin,2008).Toavoidmisunderstanding,languageissaidtoconformtothreetypesofcommunicationrules;syntactic,semantic,andpragmaticrules.Whilesyntacticrulesdefinethearrangementsofwords, semantic rules refer to themeaningof thewords.Pragmatic ruleshelpinterpretingverbalcommunicationinagivencontext,andtodeterminetheappropriateresponsewithinthatcontext.Thearbitraryrelationshipandthecommunicationrulesarecriticalcomponentsof verbal communication, as the meaning and ambiguity of words can greatly affect the(in)effectivenessofcommunication(Lane,2008).The aspect of meaning in verbal communication can be divided into two subcategories; thedenotative and connotative meaning. The denotative meaning of a word relates to the verydefinitionoftheword,i.e.its“dictionarymeaning”,whereastheconnotativemeaningreferstotheemotional or attitudinal response on an individual basis, as composed by feelings, images, andmemoriessurroundingthatword(Fujishin,2008).Thus,denotationandconnotationareextremelyimportanttoconsiderwhencommunicatingverbally.

Non-verbalcommunicationTheothermodeofcommunication,non-verbal,isdefinedasanyformofcommunicationotherthanverballanguage(ChandlerandMunday,2011b).Itshould,however,notbeconsideredsecondaryto verbal communication, as we communicate more through nonverbal communication thanthrough verbal language. Nonverbal communication is widely considered to be the dominantcommunicationmode,withsomeresearchersdemonstrating thatbetween65%and93%of themeaningfrommessagesiscommunicatedwithnonverbalcues(Ruggiero,2000).Thisisrelatedtothe fact that the majority of nonverbal communication is transmitted unconsciously orunintentionally (Chandler and Munday, 2011b). Forni (2002) even argues that when used inconjunctionwithverbalcommunication,itis12-13timesmorepowerfulthantheverbalmessageitaccompanies.Whilesomeaspectsofthetwomodesaresimilarorrelated,certaincharacteristicsdistinguishthenonverbal dimension of communication from verbal. Fujishin (2008) argues that nonverbalcommunicationismuchmorecomplexcomparedtoitsverbalcounterpart,asitcannotbebrokendown and examined in discrete units (such as words and sentences). It is considered to becontinuous and instantaneous, in the sense that e.g. surprises or disappointing messagesimmediately reflects physical and emotional response to information in e.g. face, posture, andbreathing.Fujishinalsostatesstatesthat“nonverbalcommunicationconveysemotionsandfeelings

muchmoreeffectivelythanwords”(Fujishin,2008,p.56).Theaspectofconveyingemotionswas

32

alsodenotedbytherenownedAmericanPsychologist,MichaelArgyle,asheidentifiedfourmainfunctionsofnonverbalcommunication(Argyle,1975):

1) Expressingemotion(affectdisplays).2) Communicateinterpersonalrelationshipsandattitudes.3) Supportverbalinteraction.4) Presentingpersonality.

Compared to verbal communication, nonverbal communication is considered more universallyunderstood, as it is still possible to derive some meaning from communication despite notunderstanding the spoken language. Certain specific nonverbal behaviours and expressions areconsideredtobeuniversal,asperPaulEkman(1978).However, the interpretationofnonverbalcuescanalsobeculturallydependent,andcommunicatedbothintentionallyandunintentionally,sotheinterpretationmustbedonecarefully(Lane,2008).Thefollowingsectionspresentsthefourdifferentcategoriesofnonverbalcommunication;kinesics,proxemics,haptics,andparalanguage.KinesicsThe first category of nonverbal communication, kinesics, revolves around the study of bodymovement andpositioning. It covers the areasof posture, gestures, facial expressions, andeyebehaviour,andisanextremelyimportantaspectofcommunicationandinteraction,duetotheoftenmultimodalaspecthereof.Whiletheposturereferstothepositioningandorientationofthebody,gesturesarethevisibleactsusedtocommunicatebythemeansofmovingoneormorebodyparts(Fujishin,2008).Therathersimpleactionsofstandingand/orgesturing,canconveyextremelystrongmessages,intentionallyorunintentionally.Whilegesturesareoftenconsideredtobeusedasawayofenhancingverbalcommunication,theycanalsobeusedasaprimarymethodforconveyingamessage,e.g.whenthereistoomuchnoisetouseverbalcommunication(Kendon,2007).Overtheyears,researchershaveprovengesturestobeauniversalandnaturalformofexpression,however,itisstillconsideredtobegovernedbycertainsocialconventionsandculturalcontexts(Fujishin,2008;Kendon,2007).Theothertwoareaswithinkinesics,facialexpressionsandeyebehaviour,arealsoconsideredtobecrucialcomponentsinournonverbalbehaviour.Itisnocoincidencethatthevastmajorityofstudieson human emotions have revolved around the visual cues given by facial expression and eyebehaviour,sincetheseareconsideredtobeextremelypotentsourcesofnonverbalcommunication,andfunctionsasourprimecommunicatorsofemotion(Lane,2008).Ourinternalemotionalstateislargely communicated through our facial expressions, and it is often done consistently andunintentionally (Fujishin, 2008). Facial expressions are constructed from the arrangement andactivationoffacialmuscles,whichisexplainedmoredetailedinsection4.6.

33

ProxemicsThis category concerns theuseofpersonal spaceanddistance,and the studyofproxemics canreveal many details about how we feel about ourselves and about others (Lane, 2008).AnthropologistEdwardHalldefinedfourdistancesofpersonalspaceinWesternculture,whichcanbeusedtoimplytherelationshipsbetweeninteractingpeople(Hall,1966).Thefourspacesare:

1. Intimatespace(0-0.45m):Reservedforintimateactivitiesandconversationofconfidentialinformation,andonlypeoplewithwhomweshareanintimaterelationshipareallowed.

2. Personalspace(0.45m-1.2m):Usedmainlyforinformalconversations3. Socialspace(1.2m-3.6m):Appropriateforbusinessdiscussionsthatareneitherpersonalnor

private.4. Publicspace(3.6m-7.6m):Oftenusedtocommunicateinclasslectures,publicspeeches,or

ceremonies.

Thefartherthedistanceisbetweentwopeople,themoreformalandimpersonaltheirrelationshipandconversation.Theexactdistancesmayvaryaccordingtothesetting,thepeopleinvolved,thecurrentemotionalstateofanindividual,aswellasanumberofculturalbackgroundfactors(Fujishin,2008).HapticsHapticsreferstocommunicationandinteractionviathesenseoftouch,asisconsideredtobethemost intimateformofnonverbalcommunicationbehaviour(Fujishin,2008).Theactualmeaningconveyedwithtouchdependsonanumberofdifferentfactorssuchasthebodypartbeingtouched,durationandmethodofthetouch,thephysicalcontext,aswellastheage,sex,andinterpersonalrelationship of the people involved. Research has shown that numerous emotions can becommunicatedsolelywithhaptics,asexemplifiedbyHertensteinandKeltner(2006),whofoundthatparticipantswereabletodecodeanger,fear,distrust,love,gratitude,andsympathyviatouch.ParalanguageThefinalareawithinnonverbalcommunicationisthatofparalanguage,sometimesalsoreferredtoas vocalics. It refers to how we say something, rather than what is being said (Lane, 2008).Paralanguagecoversanumberofdifferentprosodicelements,suchasvocalqualitiesandaccents.These qualities can strongly affect how people perceive each other, and how an utterance isunderstood.Itispossibletoradicallychangethemeaningofthespokenwordbyalteringone’svocalqualities,suchasthevolume,pitch,speed,andintonation(Fujishin,2008).Aslowlyspeaking,lowerpitchedvoicemightseemcalmandauthoritative,whereasapersonwithhighpitchedvoicewhospeaksatahigherratemightappearmoreexcited.An incredibleamountof informationcanbederivedfromparalanguage,as it ispossibleto identifyorapproximate individual factorssuchasethnicity,age,aswellasanumberofpersonalitycharacteristics,justbylisteningtohowaperson

34

speaks.Vocalqualitiesalsoenableustouseandinterpretspeechase.g.sarcasmormockery,andallowsformoresophisticated,sympatheticandemotionalcommunication(Lane,2008).Even though the two modes of verbal and nonverbal communication are defined as separateentities,theyareconsideredtobeextremelycloselyrelated.BothChandlerandMunday(2011a),Fujishin (2008), and Lane (2008), argue that due to the ambiguous nature of nonverbalcommunication,itispreferredtointerpretthetwoinconjunction,andshouldbeseennotasdistinctbutascomplimentarymodes.

4.6HumanemotionsininterpersonalcommunicationIf interpersonal communication is to be emulated in an ATS, the emotional signals in thecommunicationprocessisofcourseofhighestimportance.Thissectionwilldiscusstherelevanceofunderstandingandusingemotionsinacommunicationcontext.DefinitionandrelevanceofemotionWhen discussing interpersonal communication, emotions play an essential part due to thereciprocal relationship of emotion and communication. Emotions are decisive for relationshipsbuilding,whilesocialrelationshipsarethemostimportantsourceforhumanemotions,aspeopleare spurred toemotion from interpersonal interactions (Langlotz&Locher,2013;Qadar,2016).Many different definitions of emotion exist, but according to Lane “emotions are feelings we

experiencethatresultfromtheinteractionofphysiology,cognitions,andsocialexperienceandthat

they significantlyaffecthowwecommunicatewithothersand interpretothers’ communication”

(Lane,2008,p.98).ThelinkbetweenemotionalcommunicationandinterpersonalrelationshipiselaborateduponbyAndersenandGuerrero(1997),wholistssixapplyingprinciplesthatformsthislink;

1. Socially adaptive emotional communication is positively selected in the evolutionaryprocess.�

2. Socializationprocessesguidehowindividualsmanagetheircommunicationofemotion.3. Interpersonalschemata[scriptsof‘normalbehaviour’],includinggoals,needs,desires,and

expectationsaffecthowandwhenemotionisexperiencedandcommunicated.�4. Interpersonalcommunicationistheprimaryelicitorofmostemotions.5. An essential feature of the emotional experience is expression via interpersonal

communication.6. Emotionsgenerateotheremotionsininteractionchains.�

LanglotzandLocher(2013)dividestheseprinciplesintotwocategories;Thefirstthreearerelatedtothebasicbiological,socio-cultural,andcognitiveconnections;whilethelastthreearerelatedtotheactualrolethatemotionsplayininterpersonalcommunication.

35

Expressing emotions is also away of covering some of the basic human needs, e.g. expressingsadness resulting in social support and promoting personal bonds and intimacy (Mongrain &Vettese,2003).Feelingsandemotionsaffectswhen,how,andwhywecommunicate,andemotionisthusconsideredtobeakeyelementininterpersonalrelations(Lane,2008).Researchhasevenshownthathardlyanycommunicationtakesplacewithoutemotionalinvolvement,andsomeevensuggest that it is impossible to fully understand communicationwithout taking this aspect intoconsideration (Altrov, 2013). Furthermore, Qadar (2016) states that using, managing andcommunicatingemotionsisanessentialskillofhumandevelopment.ThisisbackedbyLanglotzandLocher,whostatethat“humansocialityisfundamentallygroundedinourabilitytoempathizeand

emotewithothers” (Langlotz&Locher,2013,p.92).Anotherkeyaspectof communicatingandexperiencingemotionsliesinourindividualpersonalities,asthreeoftheBigFivepersonalitytraitsstronglyinfluencesouremotions(extroversion,neuroticism,andagreeableness;Lane,2008).Emotional signals and how they are expressed can vary greatly depending on the person, theculture,andthespecificcontextinwhichthesignalsaresent(Hoey,2017).Asanexample,heartratemightincreasewhenapersonisfeelingangry,butthesameresponsecouldbeseenwhenapersonisperformingaphysicalactivity.Itcanbeextremelydifficulttodistinguishbetweenthesetwobehaviours,thusmakingthetaskofemotionrecognitionextremelydifficult.AccordingtoHoey(2017),emotionsignals canbedivided into twodistinct categories; signals thatareapparent toothers, and those less or not apparent to others. The former includes signals such as facialexpressions, gestures, body movement, and voice intonation. Less apparent signals are morephysiological of nature and include respiration, heart rate, blood pressure, and electro dermalresponse.PerceptionofemotionAcriticalaspecttoconsiderwhendiscussingtheuseofemotionsincommunication,ishowweeachindividually perceive each other and the world. By taking into consideration the individualdifferencesofthisperception,thegeneralcommunicationeffectivenesscanbeimproved.Perceptionistheprocessbywhichweselect,organise,andinterpretsensoryimpressionsinorderto givemeaning to the environment and interactions inwhichwe participate (Pipas& Jaradat,2014).Themeaningextractedfromthesesensoryinputscanthenevokecertainemotionsandevenchangetheemotionalstateofaperson(Maiocchi,2015).Asperceptionisessentiallyasubjectiveprocess, we all perceive the same stimuli differently, and thus communicate in amanner thatfacilitatesourownunderstandingoftheperceived.

36

ThePerceptionProcessAccording to Lane (2008), the perception process has three distinct stages that occur almostsimultaneously:selection,organisation,and,interpretation.Thefirststage,selection,relatestohowweselectfromthesurroundingstimuliofagivenscenario.Tosorttheinformation,allstimuliareevaluatedaccordingtotheirsalience(i.e.theinterest,use,andmeaning)andvividness(i.e.stimulithataremorenoticeable).Afterselectingwhichinformationtoprocess,thestimuliareorganisedonthebasisofe.g.schemas,closure,andproximityandsimilarity.Thefinalstageintheperceptionprocess is the interpretation stage, where we make meaning of the organised stimuli. Thisinterpretationisinfluencedbybothexpectancyandfamiliarity,i.e.whatweexpecttoperceive,andhowfamiliarwearewiththestimuli.PipasandJaradat(2014)statethattheperceptionisaffectedbyanumberofdifferentfactorsinboththeoneperceiving(theobserver),theobjectbeingperceived(thetarget),andthecontextinwhich the perception is made (the situation). Each observer has different attitudes, motives,interests,experiences,andexpectations,whichallcontributetotheperceptionofanobjectorasituation (Langton & Robbins, 2006). Such individual differences in the characteristics of theobserver can distort perceptions, and thus lead to misinterpretations of other’s behaviour.Furthermore,theculturalbackgroundoftheobserveralsoplaysasignificantroleintheperceptionprocess. The second factor affecting the perception is the characteristics of the target, such asloudness,novelty,movement,sound,size,orotherattributes.Inaddition,nearbyandsurroundingobjectswillalsoaffecthoweachindividualobjectisperceived,astheyareneverviewedasisolatedobjects but alwayswithin the situation inwhich they areperceived. The situational aspect alsoconcerns lighting conditions, the moment in time, location, temperature, noise, and othersituationalfactors.PerceptualerrorsEachofthethreementionedstagesareinfluencedbypersonalbias,whichcanleadtoperceptualerrors, such as selective perception, selective attention, or confusing factwith inference (Lane,2008). These perceptual errors are due to the influence of human physiology, social roles andstatuses,andindividualculturalbackground(PipasandJaradat,2014).Ourperceptionsareheavilyinfluencedbyphysiologyinregardstotheacuityofoursenses,asdifferingsensorycapacityalsomeans differences in perception. Furthermore, the biological rhythm of the body affectsmanydifferentprocesses, suchasbodytemperature,attention level,mood,andhormone levels.Asaresult,energyandstresslevelsalsotendtodiffer.Anotherinfluentialfactoristhatofsocialrolesandstatuses.Dependingontheroleandstatusofaperson,acertaintypeofbehaviourisexpectedincertainsocialsituations.Asthesamepersoncanplaymultiplesocialroles,italsoinfluencestheperceptionof thatperson.The thirdandmost influential factor is considered tobe theculturalbackgroundoftheobserver,asthevalues,norms,beliefs,symbols,andcustomsofaperson,isareflectionoftheorganisationalcultureandsubculturewhichheidentifieswith.Itinfluencesboth

37

thebehaviour andattitudesof theperson inquestion, but alsohow thatpersonperceives andinterpretsothers’behaviour.EmotionalcuesA number of different emotional cues and communication channels exist, i.e. emotions arecommunicatedbyarangeofverbalandnon-verbalcues.Bybeingabletounderstandanddecodetheseemotionalcues,interpersonalcommunicationbecomesmoreefficient,whichsubsequentlyleadstoagreatersatisfactionandmorewillingnesstocommunicate(Qadar,2016).Planalp(1998)presented fivedifferent classesofemotional cues, very similar to thecuespresented in section“Modesofcommunication”.AnoverviewofPlanalp’sclassescanbeseenintable3below.Class FormsofrealizationVerbalcues Language-specificemotionvocabularies.Metaphors.Speechacts.Emotional

discoursepractices,e.g.therapeuticdiscourse.Vocalcues Voicequality:low,loud,slow,fast,trembling,high-pitched,monotonous,animate

voice.Bodycues Animated,energeticmovement.Physicalactions:throwing,threatening,kissing,

caressing.Gait:walkingheavily,lightly,armswing,stride.Bodyposture:stiff/rigid,droopy,upright.Gestures:emblems,clenchinghandsorfists.

Physiologicalcues Blushing,pupildilation,heartrate,breathing,skintemperature.Facialcues Facialexpressionsthroughforeheadandeyebrows,eyesandeyelids,mouthand

lips.Table3:Emotionalcues,aspresentedbyPlanalp(1998).

Abarrier to correctly interpreting emotional cues, is thatof intercultural differences, as certainculturallydisplayrulesdeterminehowandwhentodisplayemotions,andcomesintheformofbothencodinganddecodingrules.Byadheringtodisplayrules,itispossibletoe.g.simulate,intensify,de-intensify,ormasktheactualemotion,andtherebycommunicateappropriatelyintheculturalcontext (Langlotz & Locher, 2013). On the other hand, violating such rules can lead tomiscommunication, hostility, or even communication breakdown (Gallois, 1993). Tannen (1986)even states that similar rules and misunderstanding hereof also exist in intraculturalcommunication, between people from different classes, regions, or gender. To avoid suchmisunderstandings,theexpressionofemotionsisadaptedbasedonthecontextandsurroundings,however, due to the sometimes ambiguous nature of emotions, different people may reactdifferentlytothesamedisplays(Langlotz&Locher,2013).Despitethesedifferences,someconsidershumanemotionalityuniversalduetothemeasurabilityinbiologicalprocesses,ascertainstimulicaninvokee.g.increasedheartrate,bloodpressure,etc.,which corresponds to a primary emotion (Langlotz & Locher, 2013). Additional factors in howemotionsareexperiencedand communicatedare individual context, physiology, and cognitions

38

(Lane,2008).Barriers tocommunicateemotionscanalsobebasedonconsciousorunconsciousevaluation of the context inwhich emotions are communicated; themotive can be consideredunethical, uncertainty of how the receiver will interpret or react to the message, the physicalenvironment in which communicate takes place, inappropriateness due to social roles, or forreasonsofprivacy(Lane,2008).TheimportanceofmultimodalityAsmentionedintheprevioussections,communicationsignalsaresentindifferentchannels,anddue to the complexityof the signals, they arepreferred tobe communicatedmultimodally, i.e.through more than one sensory channel. The phenomenon of multimodal communication isimportantinmanydifferentdisciplines,includingpsychology,perception,andcognition,andthusalsowhenstudyingthecontributionandimportanceofemotionsininterpersonalcommunication(Partan&Marler,2005).The multimodal aspect of emotional expression is especially important in interpersonalcommunication,sinceverbalcommunicationisalmostalwaysaccompaniedbyappropriategesturesandfacialexpressions,especiallyincaseofspontaneouslyoccurringemotionalexpression(Sherer&Elgring,2007).AccordingtoBänziger,GrandjeanandScherer(2009),anumberofprototypicalsetsofexpressionconfigurationscanbedefined,whichareusedtonotonlyunderstandsignalssentbyfamiliarpersons,butalsousedtounderstandtheexpressionsofcompletestrangerswithlittlecontextualinformationavailable.Theyconsidertheseprototypesascoreskillsusedininterpersonalcommunication,asobservedexpressionpatternsareusedtoevaluatee.g.theauthenticityofthemessages received. Discrepancies between different channel signals is often seen as a sign ofdeceptionfromthesender.Tobetterunderstandinterpersonalcommunication,relationships,andemotionalexpression,itisthusnecessarytoviewtheinteractionfromamultimodalpointofview,andtherebytaking intoconsiderationallelementsof thecomplexemotionalsignals (Langlotz&Locher,2013).Thecoreskillrelatedtounderstandingtheoverallemotionalmeaningofamessage,isconsideredtobetheabilitytoresolvediscrepanciesinmessages,thatistoevaluateandweightheindividualchannelsinamultimodalmessage(Gallois,1993).Analysing emotion expression unimodally, on the other hand, is considered to be highlycomplicated, and often results in misunderstandings, miscommunication, and misleading testresults.Thiscouldalsobearesultofmodulationorwithholdingofinformationinagivenchannel,which could havebeen interpretedmore accuratelywith information fromadditional channels.However, researchhasshownthatsomechannels (especially facialexpressions)aremoreeasilydecodedandunderstoodunimodallythanothers.PartanandMarler(2005)suggestthatmultimodalsignals should be studied by taking both the individual channels and the composite signal intoconsideration.

39

Throughouttheyears,anumberofstudieshavebeenmadeonmultimodalemotionrecognition.SomeofthemostimportanttestsdevelopincludeDANVA(Nowicki&Duke,1994),MERT(Bänziger,Grandjean&Scherer,2009),ERI(Scherer&Scherer,2011)andperhapsmostnotablythePONStest(Rosenthaletal.,1979).Whileallofthesetests includebothvisualandauditorycues,theexactapplicationhereofdiffers.Theresultsofthetests,however,allprovethatmultimodalemotionalexpressionsaremoreaccuratelyunderstoodthantheirunimodalcounterparts.

4.7TheoriesonhumanemotionThroughout the years, researchers from different academic fields have tried to conceptualisehuman emotions. This section will present some of themost renowned theories, such as PaulEkman’sbasicemotions,andRussell’scircumplexmodel.Ekman’sbasicemotionsMany different theories on human emotions exists, but one of the pioneerswithin the field isAmericanpsychologistPaulEkman.Basedoninterculturalstudies,Ekmanfoundsixbasicemotionsthatwereuniversal;anger,disgust,fear,happiness,sadness,andsurprise(Ekman,1978).Hestatesthat each of these emotions can be expressed using facial expressions, and are universallyunderstood.In1978,hepresentedtheFacialActionCodingSystem(FACS),whichisawidelyusedmeasurementsystemforfacialexpressions(Ekman,1978).ThesystemconsistsofnumerousActionUnits(AUs),whicharebasedonthemovementofthehumanfacialmuscles.WhileFACSitselfdoesnotindicateemotionalstates,itcanbeusedasabasisforfurtheranalysisusingFACScodesfromwhichemotionscanbeinferred.FACSwasupdatedin2002toaccommodateheadmovementsandeyepositions,whilealsorefiningsomeoftheAUcombinations(Ekman,FriesenandHager,2002).This research added to the list a number of additional basic emotions, however, these wereunrelatedtothefacialexpressions;amusement,contempt,embarrassment,excitement,guilt,pride

inachievement,relief,satisfaction,sensorypleasure,andshame.Inrecentyears,FACShavebeenusedinbothneuroscience,computervision,computergraphicsandanimation(Poriaetal.,2017).Russell’sCircumplexModelofEmotionsAnumberofdimensionalmodelsofemotionshavebeenproposedthroughouttheyears,andhavesincebeenwidelyadopted.OnesuchdimensionaltheoryisRussell’sCircumplexModelofEmotions,which distributes 28 emotions in a two-dimensional space consisting of arousal (vertical) andvalence(horizontal)(Russell,1980).Anillustrationofthemodelcanbeseeninfigure11below.Thismodelprovidedauniversalmethodofunderstandingandanalysingaffect,togreatinterestine.g.psychologyandlearning.Withinalearningcontext,thepreferredemotionalstatesarethosefoundinthefirstquadrant,i.e.thosepleasantwithhigharousal.Inotherwords,theseemotionalstateshavethehighestpositiveactivation,andthusthehighestlearningoutcome.

40

Figure11:Circumplexmodelof28emotions,accordingtovalenceandarousal.IllustrationbyNesbittetal.

(2015),basedonRussell(1980).

Plutchik’sWheelofEmotionsAnotherwidelyrenowneddimensionalmodelistheWheelofEmotions,asintroducedbyRobertPlutchikin1980(Plutchik,1980).Themodelisbasedoneightprimarybipolaremotions;joy-sadness,anger-fear, trust-disgust,and surprise-anticipation.Basedonthe intensityofeachofthesebasicprimaryemotions,moreadvancedemotionsarisesuchasannoyance,terror,boredom,andecstasy.Furthermore, Plutchik also theorised a number of primary, secondary and tertiary feelingscomposedoftwoemotions.Theseconstructsweredenoteddyads.Theverticaldimensioninthemodelrepresentsintensity,whiletheradialdimensionrepresentssimilaritiesamongtheadjacentemotions.Anillustrationofthecompletemodelcanbeseeninfigure12.

Figure12:Plutchik’sWheelofEmotions(Plutchik,1980).Thecolorsaturationindicatestheintensityofthe

primaryemotions,andprimaryfeelingsaredenotedbetweenthebranches.

41

Parrot’sclassificationofemotionsAnother, even more elaborate theory was presented by Parrot (2001). Like Plutchik, Parrotdescribesemotionsinthreedimensions,withthesixprimaryemotionsbeingliking,joy,surprise,anger, sadness, and fear. Each primary emotion then encompasses a number of secondaryemotions, which then again are broken down into tertiary emotions. The tree-structure of thetheory,andthefulllistofidentifiedemotionscanbeseenintable4.Primaryemotion

Secondaryemotion

Tertiaryemotions

Love Affection Adoration,affection,love,fondness,liking,attraction,caring,tenderness,

compassion,sentimentality

Lust Arousal,desire,lust,passion,infatuation

Longing Longing

Joy Cheerfulness Amusement,bliss,cheerfulness,gaiety,glee,jolliness,joviality,joy,delight,

enjoyment,gladness,happiness,jubilation,elation,satisfaction,ecstasy,euphoria

Zest Enthusiasm,zeal,zest,excitement,thrill,exhilaration

Contentment Contentment,pleasure

Pride Pride,triumph

Optimism Eagerness,hope,optimism

Enthrallment Enthrallment,rapture

Relief Relief

Surprise Surprise Amazement,surprise,astonishment

Anger Irritation Aggravation,irritation,agitation,annoyance,grouchiness,grumpiness

Exasperation Exasperation,frustration

Rage Anger,rage,outrage,fury,wrath,hostility,ferocity,bitterness,hate,loathing,scorn,

spite,vengefulness,dislike,resentment

Disgust Disgust,revulsion,contempt

Envy Envy,jealousy

Torment Torment

Sadness Suffering Agony,suffering,hurt,anguish

Sadness Depression,despair,hopelessness,gloom,glumness,sadness,unhappiness,grief,

sorrow,woe,misery,melancholy

Disappointment Dismay,disappointment,displeasure

Shame Guilt,shame,regret,remorse

Neglect Alienation,isolation,neglect,loneliness,rejection,homesickness,defeat,dejection,

insecurity,embarrassment,humiliation,insult

Sympathy Pity,sympathy

Fear Horror Alarm,shock,fear,fright,horror,terror,panic,hysteria,mortification

Nervousness Anxiety,nervousness,tenseness,uneasiness,apprehension,worry,distress,dread

Table4:Parrot’sclassificationofemotions(Parrot,2001)

42

AnotherapproachtomodellinghumanemotionwastakenbyHugoLövheim,aspresentedinhisCubeof Emotion (Lövheim, 2012). Lövheimdefines eight basic emotions in a three-dimensionalspace,accordingtotheirrelationtothesignalsubstancesdopamine,noradrenalineandserotonin,alsoknownasthemonoamineneurotransmitters.Asanexample,joyisconstitutedbyhighvaluesofdopamineandserotonin,butwithlowvaluesofnoradrenaline,asillustratedinfigure13.

Figure13:Lövheim’sCubeofEmotion(2012).

Anoverviewoftheemotiontheoriespresentedthroughoutthissection,canbeseenintable5.Theory PrimaryemotionsEkman(1978).Basicemotions. Anger,disgust,fear,happiness,sadness,surprise.Russel(1980).Circumplexmodelofemotions.

Exited,astonished,delighted,happy,pleased,content,serene,calm,tired,bored,depressed,miserable,frustrated,annoyed,angry,afraid,alarmed.

Plutchik(1980).Wheelofemotions. Joy,sadness,anger,fear,trust,disgust,surprise,anticipationParrot(2001).EmotionsinSocialPsychology.

Liking,joy,surprise,anger,sadness,fear

Lövheim(2012).CubeofEmotion. Shame/humiliation,Distress/anguish,Fear/terror,Anger/rage,Contempt/disgust,Surprise/startle,Enjoyment/joy,Interest/excitement.

Table5:Overviewofrenownedtheoriesofhumanemotion.

43

4.8InterpersonalCommunicationModelBased on the research presented throughout this chapter, a model illustrating the differentelementsofinterpersonalcommunicationiscreated.ThemodelisbasedontheoverallstructureofStamp’s Grounded TheoryModel of Interpersonal Communication, but includes elements frommanyoftheotherresearchareasinvestigatedaswell.Themodelisillustratedinfigure14below.

Figure14:Summarisedmodelofinterpersonalcommunication.

Thismodelwilllaterbeusedtodiscusstheuseofcognitiveandaffectivecomputingwithinaffectivetutoringsystems,inregardstohowitcanemulatehumanpersonaltutoring.

4.9ChaptersummaryAfterpresentinganumberofprinciplesofcommunication,thegeneralcommunicationprocesswasinvestigated,withafocusonthekeyelementswhichthisprocessisconsideredtoconsistof;sender,encoding,message,channel,decoding,receiver,feedback,context,andnoise.Thiswasfollowedbya description of different communication models incorporating these elements; linear,interactional,andtransactionalcommunicationmodels,includingexamplesofeachtypeofmodel.From the research, it became obvious that each of the mentioned components of thecommunication process must be taken into consideration when analysing interpersonalcommunication,inordertoseetheprocessholistically.Other importantaspectsof interpersonalcommunicationarethetwomodesofcommunication;verbalandnon-verbal.Whencommunicatingamessage,itisencodedandsentwithanumberofdifferentcues.Examplesofsuchcuesarewordsand linguistics (verbal),gestures,posture, facial

44

expressions, proxemics, and haptics (non-verbal). As each mode has both advantages anddisadvantages over the other, interpersonal communication often makes use of both modessimultaneously.Often,amessageissentwithmultimodalcues,whichincreasestheeffectivenessofcommunication.Then, the influence emotion has on interpersonal communication was investigated, as itsignificantly affects when, how, and why we communicate, and how we interpret others’communication. It iswidelyproventhatnocommunicationtakesplacewithoutsomeemotionalinvolvement,andthetwoareasareconsideredtobelinkedthroughbasicbiological,socio-cultural,and cognitive connections. The expression of emotions is based on the context of which thecommunication takes place, and how andwhen the emotions are expressed aremodulated bydifferentdisplayrules.Ontheotherhand,emotion-relatedstimuliareperceiveddifferentlyfromperson to person, as the perception process is subjective, based on each individual’s selection,organisation,and interpretationof thestimuli.Thisprocess isalsoaffectedbyphysiologicalandbiologicalfactors,aswellaspersonalbias,e.g.suchasselectiveperception,selectiveattention,andconfusing factwith inference,whichall lead toperceptualerrors.Researchalsohighlighted theimportanceofmultimodalitywhencommunicatingwithaffect,duetothecomplexityofemotionalsignals. In addition, a number of different theories on human emotion were investigated, aspresentedbyEkman(1978),Russel(1980),Plutchik(1980),Parrot(2001),andLövheim(2012).Finally, a model was constructed, describing the interpersonal communication process, bycombiningallinformationfromthepresentedresearch.Thismodelwillbeusedforfurtheranalysisinchapter6.

45

5.CognitiveandAffectiveComputingTheprevious chapterhighlighted the complexity and capacityof thehumanbrain in regards tocommunicationandemotionprocessing.Toproperlyemulate interpersonalcommunication, it isthusrelevanttolookathowtheneuralprocessescanberecreatedincomputers.ThischapterwillpresentanelaborateinvestigationofCognitiveandAffectiveComputing.Itwillbediscussedhowthetechnologiesareapplied,inregardstotheirunderlyingarchitecture,andthefunctionalityofthesystems.Furthermore,thecurrentstateoftheartofbothcognitiveandaffectivecomputingwillbeevaluated.

5.1AneweraofcomputingAccordingtoIBMSeniorVicePresident,JohnE.Kelly,theevolutionof informationtechnologyisapproachinganewandthirdera,thatisradicallydifferentfromthepreviouseras(Kelly&Hamm,2013). The first era,TheTabulatingEra,wasall aboutmechanical tabulatingmachines,used toorganisedataandperformcalculationsandcountingbasedonthisdata.Tabulatingmachineswereused in a number of different application areas, however, predominately in accounting. TheTabulatingErabeganinthenineteenthcentury,andcontinuedintothe1940’s,whereinformationtechnologyentereditssecondera:TheProgrammableEra.Focusnowshiftedtowardsstoringandprocessing informationdigitally, andprogramming the software toexecute logical sequencesofsteps, resulting inmuchmorepowerful systems (Kelly&Hamm,2013). In its early years, thesesystemsenabledcomputationprocessessuchasmessageencryptionanddecryption,whilemoreadvancedsystemshaveplayedakeypart inspaceexplorationinthesecondhalfofthecentury.Today, we are on the brink of the third era of computing: The Cognitive Era. Kelly states thatcognitivesystemswillevolvefromtheirinteractionswithbothdataandhumansovertime,fromwhichtheywillbeabletolearnandimprovetheircapabilitiesandaccuracy,therebyextendingwhateither humans or machines could do on their own (IBM, 2017a; Kelly & Hamm, 2013). Therelationshipsbetweenmanandmachinewillblur,andtheinteractionswillaugmentoursensesandlivesinwaysthatwerepreviouslyunimaginable(Roe,2014).

5.2Whatiscognitivecomputing?Beforedivingmore intotheareaofcognitivecomputing, it is importanttotakeastepbackandunderstandtheconceptofcognitionitself.Cognitionisatermdescribingdifferentconsciousmentalactivities and processes, such as the activities of thinking, understanding, learning, andremembering, as experienced through thought, experience and the senses (Merriam-Webster,2017;OxfordDictionaries,2017).Thus,itencompassesthecorepsychologicalprocessesofhumanthoughtandrealisation.Due to technological advancements in recent years, machines are now able to understandinformation,learn,reason,andactuponthatinformation.Thecomputationalpowerandalgorithms

46

havereachedsuchalevelofsophistication,thatthecomputersnowappeartohavethecapabilityof actually thinking (Cognitive Computing Forum, 2014). This has led to the notion of cognitivecomputing.Manydifferentdefinitionsofcognitivecomputingexist,butoneofthemostconcisedescriptionshasbeenmadebyRouse(2016),whostatesthatitisthesimulationofhumanthoughtprocessesinacomputerisedmodel.Mimickinghumancognitioninacomputer,however,iseasiersaidthandone.Alreadyinstageofrepresentingandstoringinformation,thecomputationalprocessbearslittleornoresemblancetothecorrespondingneurologicalprocessesofthehumanbrain(Raghavanetal.,2016).Earleyevenarguesthatthetermisamisnomer,sincecognition“describestheactofthinking,andcomputers

neitherthinknorunderstandintheliteralsense”(Earley,2015,p.12).Incognitivecomputing,themindisconsideredahighlyparallelinformationprocessor,inwhichvariousmodels,algorithms,andtechnologies are used to transform and reason upon information. Unlike traditional computingsystems, cognitive systemsareadaptive, learnandevolveover time,andeven takes contextualfactorsandtheirsurroundingenvironment intoconsideration,whencomputingandactinguponinformation(Raghavanetal.,2016).Enablingtechnologiesincludedatamining,patternrecognition,machine learning, computer vision, natural language processing, robotics, big data, and cloudcomputing (Cognitive Computing Forum, 2014; Raghavan et al., 2016; Rouse, 2016). Systemsutilisingmachinelearningcancontinuallyacquireknowledgefromnewdata,andareevenabletodealwithincomplete,unstructuredandambiguousinformation,thuseliminatingthepreviousbruteforceapproachestoartificial intelligence(Raghavanetal.,2016;Rouse,2016).PossiblythemostwidelyknowapplicationofcognitivetechnologytodayisIBM’sWatson,whoconfidentlybeattwopreviouswinnersofthegameshowJeopardy,atechnologywhichisfurtherelaboratedinsection5.6.

5.3DeeplearningandneuralnetworksOneofthemostsignificantreasonsbehindtheprogressofcognitiveandaffectivecomputing,isthere-emergence of deep learning. Deep learning is the foundation of applications across manydifferent cognitive computing subdomains, such as image, video, speech, and text analysis(Schmidhuber,2014).Thissectionwillpresenttheconceptofdeeplearning,withthemainfocusonthemostwidelyapplieddeeplearningmethodtoday,neuralnetworks.DeeplearningAccordingtoGuoetal.(2016),deeplearningis“asubfieldofmachinelearning,whichattemptsto

learnhigh-levelabstractionsindata,byutilisinghierarchicalarchitectures”(Guoetal.,2016,p.27).Similarly,DengandYu(2014)definesitas“Aclassofmachinelearningtechniquesthatexploitmany

layersofnon-linear informationprocessingforsupervisedorunsupervisedfeatureextractionand

transformation, and for pattern analysis and classification” (Deng and Yu, 2014, p. 199-200).Throughouttheyears,leadingdeeplearningresearchershavedemonstratedthesuccessofdeep

47

learning in both computer vision, phonetic recognition, voice search, hand-writing recognition,robotics,speechrecognition,andmanyotherdomains(DengandYu,2014).Someofthekeyfactorsfortheemergenceofdeeplearninginrecentyears,havebeentheincreasedprocessingpowerine.g.GPU’s,asignificantdecreaseofcomputinghardwareprices,theincreasedavailabilityoftrainingdata,andtheadvancesinmachinelearningalgorithms(Goodfellow,Bengio&Courville,2016;Guoetal.,2016).Initsearlydays,thedomainofdeeplearningwasintendedtobecomputationalmodelsofhowthebrainlearns.Today,theneuralaspectofmachinelearningstillplaysalargepartinengineeringdeeplearningsystems,however,somedeeplearningresearchersarenotconcernedwithneuroscienceat all, but mainly focuses on fields such as information theory, linear algebra, and probability(Goodfellow,Bengio&Courville,2016).Asaresult,todaythenotionofdeeplearningappealstoamoregeneralprincipleoflearningmultiplelevelsofcomposition.Thenotionofdeeplearningrelatestothebiologicalprocessesofthehumanbrainwhichisorganisedinaso-calleddeeparchitecture.Thismeansthatstimuliorinputsarerepresentedatmultiplelevelsof abstractions,where each level corresponds to a different area of the brain’s cortex (Bengio,2009).Inaddition,informationisprocessedinanumberofdifferentstagesofvaryingabstractionlevels, asexemplified in thevisual system.Here, stimuli are sequentiallyprocessed fromsimpleprocesses such as detection of edges and identifying primitive shapes, to understanding morecomplexvisualshapes,beforefinallymakingsenseofthevisualimage.Thehumanbrainconsistsofapproximately86billionneurons,alsoknownasnervecellsorbraincells,arebasic functionalunitsof thenervoussystem,whichcarryand transmit informationviaelectricalnerveimpulsesoverso-calledsynapses(Azvedoetal.,2009;Martin,2010a).Onaverage,eachneuronhasabout7,000synapticconnectionswithotherneurons,withsomehavingup to15,000synapses(Martin,2010b).Whenanerveimpulsereachesasynapse,aneurotransmitterisreleased,whichtriggersanelectrical impulse inthenextneuron.This istheprocesswhichdeeplearningresearchershavebeentryingtoemulatefordecades,anditiseasytoseethesimilaritiesindeeplearningsystems.Thefollowingsectionsdescribehowsuchprocessesareconstructedindeeplearningsystems.Asthenameimplies,ArtificialNeuralNetworks,oftensimplyreferredtoasNeuralNetworks,areunmistakeably related to theprocessesandactivitiesof thebrain.The ideaofbuildingcomplexintelligentsystemsfromalargenumberofmoresimplecomputationalunitsfirstsurfacedinthe1980’s,andhasnowbeensuccessfullyappliedinpractice(Goodfellow,Bengio&Courville,2016).Overthepastdecade,thefocusofdeeplearningresearchhaschangedfromunsupervisedlearningtowards supervised learning,due to the increasedperformanceof the latter. Today, supervisedlearning is themost common formof deep learning, andmachine learning in general. (LeCun,

48

Bengio&Hinton,2015).Anexampleofasupervisedlearningmethodistheuseofneuralnetworks,whichisthemostwidelyusedmethodfordeeplearningtoday.NeuralnetworksAspreviouslymentioned,thebrainconsistsofbillionsofinterconnectedneurons.NeuralNetworks(NN)arebuilt ina similarmanner,as seen in theconceptual illustration in figure15below.NNconsistofmanydifferentconnected,small,andsimpleprocessorscallednodes,orneurons.EventhoughaNNcancontainmanydifferentarchitecturallayersofnodes,theyaredividedintothreemainlayers;theinput,hiddenlayers,andtheoutputofthenetwork(Bengio,2009).Whiletheinputandoutputbothconsistofasinglelayerofnodes,thenumberofintermediatelayersofnodeswhichareneither inputsnoroutputs,areall consideredtobepartof thehidden layer.Until2006,nosystemexistedwhichcouldcomputemorethanacoupleoflayers,therebyrestrictingthenumberofhiddenlayerstooneortwo,andconsequentlythelimitingthecomplexityoftheneuralnetworks.TheworkonDeepBeliefNetworks,publishedbyHinton,OsinderoandTehin2006,wasconsideredground-breaking, in the sense that their research enabled building deeper multi-layer neuralnetworks(Bengio,2009).TheDeepBeliefNetworkintroducedtheideaoftrainingonelayeratatime,usinganunsupervisedlearningalgorithmoneachlayer,analgorithmknownasaRestrictedBoltzmannMachine(Hinton,Osindero&Teh,2006).Sincethen,thealgorithmsandneuralnetworkshavedevelopedevenfurtherwithgreaterresults,buttheconceptualidearemainsthesame.

Figure15:IllustrationofaNeuralNetworkwiththeinputlayerx,threehiddenlayersh1-h3,andasingle

outputnodeh4.

InaNN,informationcanbesentfromanumberofdifferentinputnodesinthefirsthierarchicallayer (indicatedbyx), to thenodes in the firsthidden layerof thenetwork (h1),wherebythesenodesareactivated.Basedontheinformationreceivedfromtheinputnodes,andapre-definedweightedconnection,eachnodeinh1evaluatestheinput,processestheinformation,beforefinally

49

passingitsprocesseddataontoanodeinthenexthierarchicallayer,h2(Schmidthuber,2014).Thisprocessrepeatsthroughall layers,until thefinaloutputnode(s)hasbeencomputed.Theactualnumberofnodesand layers in aNNvaries greatly,but some systems consistof severalmillionnodes,andbillionsofconnections(Pesenti,2014).TheuniqueandcrucialelementofNNsliesinthedetermining how much weight a given node should have in the network. These weights aredeterminedbyaprocesscalledbackpropagation.

5.4AffectivecomputingAnareaof cognitive computingwhichhas receiveda lotof attention in recent years, is thatofemotionalintelligenceincomputers,alsoknownasaffectivecomputing.Inher1997paper,RosalindPicard, professor atMITMedia Laboratory for Perceptual Computing, first presented the term,whichshedefinesas“computingthatrelatesto,arisesfrom,orinfluencesemotions”(Picard,1997,p.1).Asisthecasewithcognitivecomputing,affectivecomputingisaninterdisciplinaryfield,whichspansacrossbothcomputerscience,cognitivescience,socialscience,andpsychology.Sometimesalso referred to as emotional-oriented computing, or simply emotional computing, affectivecomputinghasbecomeanemergingfieldofresearch,whichaimstoenablecomputerstorecognise,feel,inferandinterprethumanemotions(Poriaetal.,2017).Theoverarchingend-goalofaffectivecomputing,isforhumanstobeabletointeractandinterfacewithmachinesasrichlyasweinteractwitheachother(Cowie,2005).Emotionsarekeycomponentswhendiscussingessentialcognitiveprocesses,andrelatesnotonlytohumancreativityandintelligence,butalsoplaysahugepartinrational thinking and decision-making. Thus, to achieve a natural human-computer interaction,computers need the ability to recognise and express affect (Picard, 1997). Two of the mostimportantaspectsofaffectivecomputingarethoseofsentimentanalysisandemotionrecognition,astheyarekeycomponentsofhumancognitionandcommunication(Cambria,2016;Poriaetal.,2017).

5.5AnalysingandmodellingemotionsincomputersAsmentioned,emotionrecognitionandpolaritydetectionarethetwomostimportantaspectsofsentiment analysis, and affective computing in general. The two are highly interrelated andinterdependentconcepts,withtheformeroftenunderstoodandprocessedwithemotionlabels,whilethelatterconsistsofbinaryclassificationtasks,suchaspositive/negativeorlike/dislike.Theinterdependency often manifests itself in polarity detection being inferred from emotionrecognition, and hence, emotion recognition is sometimes considered a subtask of polaritydetection (Cambria, 2016). Despite its recent uprise, technologists have largely ignored takingemotion into consideration in systems design throughout the years, which has often lead tofrustratingandunsatisfyinguserexperiences.Thiscomplexityisduetothemultifacetedaspectofemotions,suchascognition,conation,interaction,personality,andculture(Cowie,2005).

50

LeeandNorman(2016)categorisesthecurrentcomputationalmodelsformodellingemotionsintofive overarching areas; dimensional, anatomical, rational, communicational, and appraisal.Dimensionalmodelsfocusonhigh-levelcoreaffects(suchaspositive,negative,ormood)anddonotcategoriseemotionsintodiscretestates(suchasfearorhappiness).Theanatomicalmodelsarebuiltwithastartingpointinbraintheoriesandneuralscience,andtiesemotionstocertainneuralandbiologicalprocesses.Thethirdcategory,rationalmodels,concernsthefunctionsorrolesthatare facilitated by certain emotional states,whereas communicationmodels aims to endow thebehaviourof social intelligenceby sharingordisplayingemotional states.Themostwidelyusedtheory formodellingemotions today,areappraisalmodels.Suchmodelsarecomponentbased,meaningthatthemodelitselfconsistsofseveralsub-models.Appraisalmodelsarebasedontheappraisaltheory,inwhichitisbelievedthatanygivenemotionalstateisbasedonanindividual’sevaluationofthesurroundings,situation,orcontextualcues(Lee&Norman,2016).Basedontheemotional state, certain physical and cognitive behaviour will manifest in the person. As thesituationalcontextchanges,maybeasaresultoftheindividual’sexpressedbehaviour,thesituationisre-evaluated,leadingtoarecursivefeedbackloopwhichcanenabledifferentemotionalstates.The following sections will discuss how emotions can be modelled in a computer. Specifically,emotionsderivedfromfacialrecognition,speechprocessing,textanalysis.FacialexpressionsFacial recognition and expressions are extremely important mediums in interpersonalcommunication,whichallowsustoexpressemotionsorcommunicateconversationalcues,eitherconsciouslyorunconsciously (Tao&Tan,2005).Theymake itpossible to forman impressionofanotherperson’ssentimentandemotionalstate(Poriaetal.,2017).Aprerequisitefordetectingemotionsfromfacesinimagesorvideo,istheactionofdetectingthefaceitself.Facialrecognitionhasbeenanimportantresearchareaforcomputerscientistsformanyyears,anditisakeyelementinanumberofdifferentface-relatedapplications,includingemotionrecognition.Facialrecognitionresearchhasseenimmenseimprovementsinrecentyears.In2001,thefirstground-breakingreal-timefacialdetectionsystemwasdevelopedbyPaulViolaandMichaelJones,andsincethenthedetectionratehasgreatlyincreased(Viola&Jones,2001;Tao&Tan,2005;Sun,Wu&Hoi,2017).Throughouttheyears,anumberofdifferentmethodshavebeenusedforfacialexpressionrecognition,suchasfeatureextraction,SupportVectorMachines,Gaborwavelets,andHiddenMarkovModels(Tao&Tan,2005).Inrecentyears,however,deeplearninganddeepconvolutionalneuralnetworkshasachievedremarkablesuccessincomputervision.Deeplearningmethodshaveagreatadvantageovertraditionalcomputervisionmethods,sinceitgreatlyreducesmanualengineeringandhand-crafteddesigns(Sun,Wu&Hoi,2017).

51

Sincetheemergenceofthesetechniques,differentneuralnetworkapplicationshavedominatedobject detection benchmark evaluations, including the ImageNet Visual Recognition Challenge(ILSVRC).Sun,Wu,andHoi(2017)extendedtheFasterregion-basedconvolutionalneuralnetworkbycombiningseveraldifferentstrategiestoachievestate-of-the-artresultsonthewidelyusedFaceDetection Dataset and Benchmark (FDDB). An example of a specific application using neuralnetworksisIBMWatson’sFacialRecognition,whichisdescribedinsection5.6.SpeechprocessingThespokenlanguageisusedtoconveymeaning,withouthaveanyphysicalmanifestationperse,unlike e.g. imageprocessingwhere anobjectwithin the image canbe categorisedbasedon itsfeatures,makingspeechanalysisextremelydifficult(Manning&Socher,2017).Thegoalofnaturallanguageprocessing(NLP),istoallowcomputerstounderstandnaturallanguage,fromwhichtheyareabletoperformagiventask.Theyrangefromrelativelyeasytasks,suchasspellcheckingorkeywordsearch,toharderandmorecomplextaskslikesemanticanalysisandquestionanswering(Manning&Socher,2017).Parameterssuchasaccents,pronunciation,articulation,pitch,andnoise,makesspeechrecognitionahighlycomplexprocess.Previously,speechrecognitionmodelsweremostlybuiltonHiddenMarkovModels,buttheaccuracyofsuchmodelsdonotcomparetorecentresults.The capabilitiesof speech recognition systemshaveachievedbreakthrough results this year,byachievingalowerworderrorrate(WER)thanhumansarecapableof.WERisanindicatoroftheinaccuracy of recognising spoken words. Research has shown that the WER of humans isapproximately 5.9%, but inMay 2017, Google CEO Sundar Pichai announced that their speechrecognitiontechnologyachievedarecordWERof4.9%(Protalinski,2017).Incomparison,Microsoftachieveda6.3%WERinSeptember,2016,whereasIBMmanagedaWERof5.5%inMarch,2017(Saon,2017;Tung,2016).PichaistatesthatGoogle’sground-breakingachievementisaresultoftheadventofneuralnetworkswhenprocessingthespeechsignal(Protalinski,2017).Analysingemotionsfromspeechisnotonlyamatterofunderstandingtheverbalcontent,i.e.whatisbeingsaid,buthow it issaidcanplayanevenbiggerpartindeterminingtheactualemotionalstate.Featuressuchaspitch,loudness,speakingrate,pauses,rhythm,andthecorrelationsbetweenthem, all contributewith emotional information (Gangamohan, Kadiri& Yegnanarayana, 2016).Paralinguisticfeaturescarryalotofaffectinformation,andifthesearenotconsidered,importantinformationcanbelost,whichcanultimatelyleadtoacompletemisunderstandingoftheutterance(Pantic&Rothkrantz,2003).The process of emotion recognition in speech happens relatively effortlessly in interpersonalcommunication,despitebeingahighlycomplexphenomenonwhichisnotwellunderstood.Asaresult, it is very difficult to recreate this process in a computer (Gangamohan, Kadiri &

52

Yegnanarayana, 2016). In general, the process of emotion detection in speech consists of sixcomponents;speechinput,speechnormalisation,featureextraction,featureselection,aclassifiersystem,andemotiondetection(Sudhkar&Anil,2016).Themostimportantpartistoextractandselect the relevant features from the speech input. The features are usually divided into threecategories:prosody,voicequality,andspectral features (Gangamohan,Kadiri&Yegnanarayana,2016).Anexampleofusingprosodicfeatures isSudhkarandAnil (2016),whodescribedifferentbasicemotionsbasedonparameterssuchaspitch,intensity,speakingrate,andvariance.Anotherhighlyimportantelementoftheemotionrecognitionprocess,istocreateanaccurateclassifier,butinordertodoso,agoodandcomprehensivedatabaseofannotatedemotionalspeechisneeded.Collecting a sufficiently extensive database is highly problematic, since many features areconsidered to be speaker and sound-specific, which makes classification rather difficult, asdescribedbyGangamohan,KadiriandYegnanarayana(2016).Theyevengoasfarasstatingthatemotionrecognitionbyamachineisan“elusive”goal.Emotion-miningintextDetecting and analysing affect from text is another widely researched field within affectivecomputing,knownasemotion-mining.Yadollahi,ShahrakiandZaiane(2017)definesfourdifferenttypes of emotion-mining tasks: emotion detection, emotion polarity classification, emotionclassification,andemotioncausedetection.Emotion-miningcanbeappliedinanumberofdifferentareas.Itcanbeusedtoderivelevelsofsatisfactionincustomerservice;allowcontentfilteringbasedonemotionsexpressed;authorprofiling;predictingdepression,stress,orsuicidalthoughts;andofcourse to determine emotional states in tutoring systems (Yadollahi, Shahraki & Zaiane, 2017).Emotion-miningcanbeperformedone.g.personalnotes,emails,newsarticles,blogs,books,orchatmessages.TheemergenceofsocialmediasuchasTwitterandFacebook,hasopenedthedoorfor new possibilitieswithin emotion classification and socialmedia analytics, as exemplified byFarías,PattiandRosso(2016),whoprovedthataffectiveinformationcanbeusedtodetectironyintweets.Classificationofemotionscanbedoneoneitherdocumentorsentencelevel.Whenanalysingondocumentlevel,anentiredocumentoftextistheunitofinput,andthustheemotionaltoneofthewholedocumentisderived.Asthenameimplies,sentencelevelclassificationdeterminesthepolarityandtoneofasinglesentence(Yadollahi,Shahraki&Zaiane,2017).Cambria(2016)dividestheexistingapproachestoemotionrecognitionandsentimentanalysisintothreemaincategories;knowledgebasedtechniques,statisticalmethods,andhybridapproaches.Inknowledgebasedtechniques,textisclassifiedintocategoriesofaffect,basedonthepresenceofwordsthatareratherunambiguous,suchas“happy”or“sad”.However,thesetechniqueshaveonemajor weakness, that is poor recognition in more complex linguistic constructs. The secondcategory,statisticalmethods,worksbyfeedingamachinelearningalgorithmalargetrainingdata

53

setwithaffectivelyannotatedtexts,fromwhichthecomputerlearnse.g.theaffectivevalenceoftheaffectkeywords,aswell lexicalaffinityandwordco-occurrences (Cambria,2016).Thethirdcategory,hybridapproaches,harnessandcombinesthepoweroftheformertwo,fromwhichitispossible to recognise emotion and detect polarity from unimodal or multimodal data. Hybridapproachesareconsideredmoreaccurate,especiallywhenclassifierstrainedon largedatasets,analyses large text inputs.Mostacademic research incorporatesdata setsand sourcesofaffectwordssuchastheAffectiveLexicon,SentiWordNet,andSenticNet.Anumberofdifferentmethodsforemotion-miningexists, suchasSupportVectorMachines, rulebasedalgorithms,andHiddenMarkovModels. Inaddition,deep learningandneuralnetworkshavebeenusedwithsuccessfulresults,astheyareabletograspmorecomplexconceptualrules,andshifttheapproachfrombeingsyntax-basedtowardsmoresemantics-awareframeworks(Cambria,2016).

5.6StateoftheartcognitiveandaffectivesystemsThis sectionwillpresent someof themostadvancedcognitiveandaffective systemsdevelopedtoday.Thisincludesstateofthearttechnologicalsolutions,differentdevelopmentplatforms,andcognitiveassistants.DeepMindAcquiredbyGooglein2014for$400million,DeepMindTechnologiesisanindustryleaderandaglobalinnovatorwithincognitivecomputing.Overthepastdecade,thecompanyhasachievedgreatsuccessinbuildingneuralnetsfordeeplearning(Pereiraetal.,2015).DeepMindhasdevelopedacomputerprogramwiththecapabilitiesoflearningtoplayAtarivideogamesfromscratch,byusingareinforcementlearningsystembasedonrecurrentneuralnetworks.Using a trial-and-error approach without human intervention and guidance, the autonomoussystemlearnedanddevelopedovertime,reachingandevensurpassinghuman-levelperformance(Goodfellow,Bengio,andCourville,2016;Pereiraetal.,2015).Usingcomputervisioncapabilities,thesystemiscapableofprocessingandinterpretingbothnaturallanguageandvisualdatastreams,andtolearnandactbasedontheperceiveddata.WhileAtarigamesare rathersimpleofnature,DeepMinddemonstrated the truecapabilitiesofcognitivecomputing,whentheirprogramAlphaGodefeatedtheWorld’sbestplayerofthegameGo,LeeSedol(Hassabis,2016).Goiswidelyconsideredthemostcomplexboardgameinventedbymankind,andthuswinningfouroutofafivegames-seriescameasahugesurprisetothegeneralpublic.Thevictorywasatrueeye-openerformany,asAlphaGomademoveswhichonlyhada1in10,000chanceofbeingplayedbyahuman.Duetothecomplexityofthegame,itwouldbeavirtuallyimpossibletasktocodeallpossiblemoveswithabrute-forceapproach,furtheremphasisingtheground-breakingprogresscurrentlybeingmadewithinthefieldofcognitivecomputing.Accordingto CEO and co-founder of DeepMind, Demis Hassabis, AlphaGo’s ability to find solutions that

54

humansdonotlearnorconsider,hashugepotentialinotherapplicationareasanddomainsaswell(Hassabis, 2016). Furthermore, Hassabis states that even though thematch demonstrated howhumansandartificialintelligencecanpusheachothertowardsnewideasandsolutions,thereisstillalongwayformachineslikeDeepMindtobefullycapableoftheintellectualtasksofhumans.IBMWatsonFormanyyears,IBMhavebeenconsideredthefrontrunnerwithinthefieldofcognitivecomputing,andtheircognitivetechnology,Watson,isatechnologythathasthecapabilitiesofthinkinglikeahuman (IBM, 2017b). It helpedmaking the capabilities and possibilities of cognitive computersbecomemainstreamafteritfamouslywonthegameshowJeopardyin2011,afterbeatingtwoofthebesthumancontestants(Jackson,2011).Afterdecadesofresearchandmorethanfiveyearsofdevelopment,Watsonwasabletounderstand,analyse,andanswerthequestionsaskedbythehost(Kelly&Hamm,2013).During the show,Watsondidnothaveaccess to the internet,but couldaccessfourterabytesofencyclopaediadata(200millionpagesofcontent)storedlocally.Sincethen,thecapabilitiesofWatsonhavevastly improved. Ithasevolvedfrombeingquestion-and-answercomputer, to a supercomputer which can analyse and interpret all kinds of structured andunstructured data, such as text, images, audio and video. Watson can understand a user’spersonality, tone, and emotion, which can then be used to e.g. provide personalisedrecommendations,or tocreateemotionally intelligentvirtualagents (IBM,2017b). In2016, IBMmadeitpossiblefordeveloperstoharnessthepowerofWatson,bymakingAPIsavailablealongwiththelaunchoftheBluemixplatform,whichisfurtherelaboratedinsection6.3,Bluemix.Italsopowersoneofthemostpowerfuldataanalyticsplatformsonthemarkettoday,WatsonAnalytics

(IBM,2017c).Case:WatsonforOncologyToday,Watsonisusedinavastamountofdifferentapplicationsacrossallsectors,withtheWatsonforOncologybeingaprimeexampleofitscapabilitiesandpotential.Basedonexperttrainingfromphysicians at Memorial Sloan Kettering Cancer Center (MSK), Watson provides clinicians withevidence-based treatment options for cancer patients (IBM, 2017d). When applied to a newdomain,itisessentialthatWatsonlearnsthelanguageandtermsusedinthatspecificdomain.Thisisdoneby feedingWatsonhugeamountsof structuredandunstructureddata, in this case thisincludesphysicians’notes,patient information, literatureof thebestmedical sciencewithin thefieldofoncology,aswellasmedicalreportsofpatientswithasimilardiagnosis(IBM,2014a).Thisdata is then indexed,andWatson is trainedwithquestion/answerpairswhichservesasgroundtruth,andeventhoughitdoesnotgiveexplicitanswerstoallpossiblequestions,ithelpsWatsonunderstandthelinguisticpattersinoncology.Basedonthistraining,Watsonisabletounderstandinquiries from thedoctors, fromwhich it builds andevaluateshypothesis.Using statistical datamodellingandweightedevidencescores,Watsonestimatesitsconfidenceofahypothesisbasedonthisevaluation.Thisabilitytoinstantaneouslyrunanalyticsonavastbodyofdata,andevaluate

55

each unique patient against clinical evidence, allows the oncologists tomake better andmoreinformeddecisions(IBM,2017d;IBM,2014a).DevelopmentplatformsInrecentyears,techcompanieshavesoughttochangeorexpandtheirbusinessmodel,byallowingdevelopersaccess to theircognitive technologies (Pereiraetal.,2015).Market leadersandtechgiantsIBM,Microsoft,QualcommandHewlettPackardareexamplesofsuchcompanies,eachwiththeirowndevelopmentplatforms(Google,2017;HewlettPackardEnterprises,2017;IBM,2017b;Microsoft,2016;Zaki,2015).Theseplatformsallowdevelopersandbusinessesofanysizetousesomeofthemostadvancedcognitivetechnologiesthatexisttoday.Accesstotheseservicescanbecome powerful catalysts across all industries (Pereira et al., 2015). Visual recognition, facedetection, speech analysis, and language translation are some of the services available. Thefollowing sections presents two of the leading cognitive development platforms in themarkettoday,IBM’sBluemixandMicrosoft’sCognitiveServices.IBMBluemixAfter18monthsofdevelopment,IBMreleaseditsBluemixplatforminFebruary,2014(IBM,2014b).Bluemixallowsdeveloperstobuildwebapplications,usinganumberofdifferentservices intheareas of data and analytics, storage, network, and commerce. Some of the most noteworthyservices on the platform are those provided and powered by Watson, IBMs cognitive supercomputer. These services include speech-to-text and text-to-speech conversion, conversationalservices, customisablevisual recognition,aswellas languageclassificationand translation (IBM,2017e).Furthermore,servicessuchasTradeoffAnalytics,helpspeoplemakemoreinformedandbetterchoices,bytakingintoaccountmultipleandsometimesconflictinggoals(IBM,2017f).Thisisagreatexampleofanemulationofhowhumansthink,behave,andmakepurchasingdecisions,where multiple criteria are usually taken into consideration. In real estate, for example,compromisesmustoftenbemadeinoneormorefactorssuchasprice,mortgage,location,andsize.Tradeoffanalyticscanhelppeoplemakingsuchadecision.Inrelationtoaffectivecomputing,twoof themost interestingservicesprovidedbyWatsonarePersonality InsightsandToneAnalyser,bothofwhichwillbedescribedinmoredetailinsection6.3,alongwithanumberofotherWatsonservices.MicrosoftCognitiveServicesPreviouslyknowasProjectOxford,Microsoft released theirCognitiveServicesplatform in2016.Microsoft categorises their cognitive services into four different categories: vision, speech,language, knowledge, and search. The vision services extract rich information from images andvideo,andprovidesAPIsfore.g.facialdetection,intelligentvideoprocessing,contentmoderation,andemotionanalysis.AllthespeechprocessingandlanguageservicesarepoweredbyMicrosoft’sLanguageUnderstanding Intelligent Service (LUIS),which allows for speech, speaker, and intent

56

detection,aswellasconversionsbackandforthbetweentextandaudio.Otherservices,suchaslinguisticanalysisarealsoavailable.Usingactivemachinelearning,theprecisionandaccuracyofLUISimprovesovertime,therebycontinuouslyenhancingthecapabilitiesoftheservice(Microsoft,2017a).Inadditiontovisionandspeechcapabilities,Microsoftalsoprovidesknowledgeservices,whichcanbeusedtoe.g.understandanddistinguishdifferentmeaningsofaworddependingonthecontextinwhichitisused.Evenwithinthesameparagraph,thesocalledEntityLinkingIntelligenceService,canidentifythedifferententities.Furthermore,anumberofintelligentsearchAPIsareavailable,applicable inboth image,news,video,andwebsearch.Microsoftarecontinuouslymakingnewservicesavailable,tofurtherincreasethealreadyimpressiveAPIcatalogue(Foley,2017).CognitiveassistantsToday,Cognitive Assistants (CAs), sometimes also referred to as virtual assistants or intelligentassistants, arean integralpartofhowpeople interactwith theirphones, computers, andotherelectronicdevices.AccordingtoHamidMotahari,researchgroupleadonIBMCognitiveServices,CAsoffer“computationalintelligencecapabilitiestypicallybasedonnaturallanguageprocessing,

machine learning and reasoning, and provide cognition powers that augment and scale human

intelligence”(Motahari,2013).TheemergenceofCAshavemadeitpossibletoperformtaskssuchasorderingataxi,schedulingmeetings,performingsearches,andassistinginnavigation,simplybyissuingavoicecommandoraskingaquestion.CAsaredesignedtounderstandnatural languageinputs, interpret and process the request, and respond with an appropriate answer in naturalhumanlanguage(Waters,2016).Today,CAsareonlypartiallybasedonitsuser,withpersonaldatasuchascalendarsandaddresses,butinthefuture,theCAsmightevenknowmoreaboutapersonthan the person himself, and the technologywill develop from being separate from its owner,towardsthetwobeingembodiedandunifiedasone(Bauer,2014).Leadingtechnologycompanieshave developed their own IAs, with Google Assistant, Apple’s Siri, Microsoft’s Cortana, andAmazon’sAlexabeingthemostprominentandwidelyusedCAs.Asanexampleofthepopularityofthese services, Siri alone received more than two billion requests per week in October 2016,accordingtoAppleCEO,TimCook(Clover,2016).CAisanareainwhichthetechgiantsarecurrentlyfightingabattleofacquiringnewtechnology,andmaking theirproductmoresophisticatedandsuperior to that of their competitors, as there is money to be made in automating everydayprocesses(Waters,2016;Corbyn,2016).OrenEtzioni,headofAllenInstituteforArtificialIntelligence,statesthateventhoughCAshavehadamajorbreakthrough,today’sdevicesdonotcomparetothekindofcapabilitiesthatiswantedandenvisioned by the technology sector (Corbyn, 2016). He sees the futuremarket leader, as thecompanythatcanmaketheCAactmorelikeahotelconcierge,withsophisticateddialogueandtheability to handle far more complex tasks, better and faster than possible today in the above

57

mentionedtechnologies.Atechnologythataimstoovercomethis,isViv.AcquiredbySamsungin2016,Vivisanopenartificialintelligenceplatformwhosegoalistoprovideconsumerswithasingleintelligentconversationalinterfacetointeractingwithdevicesandserviceseverywhere,byallowingdeveloperstointegratetheirservicesandcapabilitiestotheplatform(Viv,2017;Samsung,2016).Instead of hardcoding all possible outcomes to a request, Viv detects the user’s intent, andgeneratesthecodedynamicallyusingpatentedDynamicProgramGeneration(Kittlaus,2016).OtherapplicationareasInthemarketingdomain,cognitivecomputinghasalsoseenaprominentriseandiswidelyappliedthroughouttheindustry.Agreatexampleofacognitivetechnology,isEquals3’sLucy.PoweredbyIBM’sWatson,Lucyisasoftwarewhichcanunderstandnaturallanguagequestionsandinquiries,and based on big data sets of structured and unstructured data, Lucy can analyse and presentcorrespondingresults inavisuallyappealing interface (Equals3,2016). Itcanhelpmarketersbybuilding audience insights, and perform market and segmentation analyses, and automaticallyvisualiseanddescribethemostrelevantfindingsfromthedata.Anotherapplicationareainwhichaffectivecomputinghaslongbeenresearchediswithinthefieldof robotics. Throughout the years, many different emotionally intelligent robots have beendeveloped,eachwiththeirownindividualpurposeandcharacteristic,buttheyallhavethecommontraitofunderstandingand/orexpressingemotions.Themostwidelyknownemotionallyintelligentrobottoday,isthehumanoidrobot,Pepper,developedbySoftbankandAldebaranRobotics.Usingdirectionalmicrophonesandhigh-definitioncameras,Peppercan identifyemotionsbyanalysingspeech (e.g. tones and lexical use), facial expressions, and posture, and communicate in acorresponding tone (SoftBank Robotics, 2017). Today, Pepper is used in a number of differentapplicationareas,rangingfromin-storecustomerservice,toteachingassistanceatuniversities.

5.7ChaptersummaryThischapterinvestigatedcognitivecomputing,i.e.thesimulationofhumanthoughtprocessesinacomputerisedmodel.Cognitivesystemsarecapableof interpretingcomplex information,reasonand act upon that information, and even develop and learn over time. The recent progress incognitive computing, is a result of developments within a number of fields, such as patternrecognition, machine learning, big data, and cloud computing. To clarify the functionality ofcognitivecomputingsystems,deeplearningandneuralnetworkswerealsopresented.To properly research technologies related to affective tutoring systems, the field of affectivecomputingwasalsoinvestigated.Affectivecomputingisanareaofcognitivecomputingconcerningcomputing that relates to, arises from, or influences emotion. Different aspects of affectivecomputing were discussed, including emotion recognition from facial expressions and speechsignals,andemotion-miningintext.Finally,differentstateoftheartcognitiveandaffectivesystems

58

and technologies were discussed, such as DeepMind and IBM Watson, different cognitivedevelopmentplatforms,andanumberofcognitiveassistants.

59

6.AnalysisAfter investigating relevant research on the topics of interpersonal communication, humanemotions,andcognitiveandaffectivecomputing,itispossibletoevaluatethecapabilitiesofanATSbuilt to emulate human tutoring. This chapterwill first and foremost present a conceptual ATSmodel,beforediscussingtheextenttowhichitcanactuallyreplicatetheinteractiontakingplacewithininterpersonalcommunicationbetweenastudentandateacher.Furthermore,asecondarymodelwillbepresented,illustratinghowdifferentcognitivetechnologiescanbeutilisedinasmall-scaleATSimplementation.Thepurposeandfunctionalityofthesemodelsarethendiscussedwiththefouruniversitystudents,whomwherealsointerviewedinsection3.2,togaintheirviewuponsuch a system’s viability, value creation, and potential application in their daily learningenvironment.

6.1AffectiveTutoringSystemModelThis sectionwill present the conceptual ATSmodel, in terms of both the requirements for thesystem, itsoverall structure,each separate component, and the individual functionalitiesof thecomponents.PurposeandfunctionalityThemainpurposeof theATS is to communicatewith the student in amannerwhichemulatesinterpersonalcommunication.Inordertoappropriatelydoso,theATSmustconsideranumberoffactorsinrelationtoboththestudent,theverbalandnonverbalmessagesreceived,andtakeallthese factors into considerationwhen constructing the response.More specifically the tutoringassistantmust:

1. Understand the student: Understand the student as a person, as a learner, and as acommunicator.Asdescribedintheframework,thisincludesawidevarietyoffactorssuchaspersonality,socialandculturalcontexts,psychologicalandphysiologicalfactors,andtheinternalstatesofthestudent.

2. Understandthemessage:Interpretandunderstandthemessagethatiscommunicatedbythe student. Themain concern here is to decode themessage encoded by the student,mostlythroughverbalandnonverbalcues.

3. Respond appropriately: Act appropriately during the interaction process, construct aresponse,andcommunicateaccordingtothemessageandemotionalstateofthestudent.

Thefollowingsectionwillpresenttheconceptualidea,beforegoingmorein-depthwiththesethreeaspects,andpresentdifferentcognitiveserviceswhichcanbeusedtobuildanaffectivetutoringsystem.

60

ConceptualmodelBasedontheresearchfromchapters3,4,and5,aswellas thesystemrequirementspresentedabove,aconceptualideamodelisformed,describingthearchitectureofanATS.Anillustrationofthemodelcanbeseeninfigure16.Inthissection,eachofthedifferentmodelcomponentsandtheirfunctionalitywillbedescribed.Thefivemaincomponentsare:thestudentasacommunicator,theinterfaceinwhichthemessagesaresent,backgroundinformationonthestudent,theemotionprocessor,andapedagogicalunit.StudentThe first component of the model is the student himself. In this component, the student isconsideredacommunicator,i.e.simultaneouslysenderandreceiver.Themostimportantelementsarehowheinteractswiththesystem,andhowthefeedbackfromthesystemisreceived.Aspectssuchasindividualcharacteristicsandpersonalityisnotconsideredatthisstageinthemodel.

Figure16:TheconceptualATSmodel.

61

InterfaceThesecondcomponent,theinterface,iswheretheactualmessagesaretransmitted.Thisrelatestoboth verbal and nonverbal communication, from both the student and the ATS. While muchinformationcanbedrawnfromverbalcommunication,nonverbalcommunicationsignalsareofextremely high importance when discussing communication with affect, and must also beincorporatedinthemodel.Itisalsointheinterface,thattheadaptedfeedbackconstructedbytheATSissenttothestudent.BackgroundinformationThefirstpartofwhatisconsideredthecoreATS,isthebackgroundinformationcomponent,whichcontainsvaluableinformationaboutthestudent.Someofthemostrelevantinformationincludesdemographicaldata,andsocialandculturalbackground.Suchrathersimpleinformationcanbeofgreatvalueinthebigpictureofcommunicatinginapersonalmanner.Thistypeofdatacanbeusedinlaterstagesofthecommunicationprocess,suchaswheninterpretingamessageandconstructingtheresponse.Anotherimportantaspecttoconsideristhestudent’spersonality.Individualneeds,values, and behavioural traits, are key elements in understanding the individual student. Asmentioned in section 5.4, this information is also importantwhen determining the pedagogicalstrategyinthetutoringmodel.Finally,bothphysiologicalandpsychologicalcharacteristicsshouldalsobeincluded.EmotionprocessingThe second part of the ATS, and the fourth component of the overall model, is the emotionprocessing component. As the name implies, this is where affect information captured in theinterface isanalysed.Throughanalysisofverbalandnonverbalcues, themessageandemotionsignalsareevaluated.Theresultsfromthisevaluationservestwopurposes.First,theemotiontonesaresenttotheemotioninformationprocessor,andsecondly,thestudent’sintentwiththemessageisevaluated.What separates ATS from ITS, is the inclusion of affect data, and subsequently an emotioninformationprocessormodule.ThisisanessentialpartoftheATS,asthisiswherealltheaffectinformationisevaluated.Besidestheemotionaltonesderivedfromverbalandnonverbalcues,thestudent informationmustalsobe considered,as individual student characteristics carrieshighlyvaluable information. Theoverall emotional stateof the student, is determinedusing a customalgorithmthatweighsalltheinputs,andcomputesasingleoutput,whichcanthenbeusedinthetutoringmodel.

62

PedagogicalunitThefifthandfinalcomponentofthemodelisthepedagogicalunit.Inthiscomponent,thelearningstrategy is evaluated, and the feedback to the student is constructed. The first element in thiscomponent is the domainmodel. As described in section 3.1, the domainmodel contains therelevant information and knowledge within a certain learning domain. By comparing studentquestionsorresponsestothis information, it ispossible toevaluatetherelevanceandaccuracyhereof, before using it as an input to the tutoring model. The tutoring model process thisinformationalongwiththestudentinformationandtheemotionalstatecomputedintheemotioninformationprocessormodule,anddeterminesanappropriatepedagogicalstrategytouse.Thethirdelementwithinthepedagogicalunitistheinteractionengine.Asdescribedinsection4.4,the interaction isconsideredthecorecomponentof interpersonalcommunication,which isalsoreflected in its complexitywithin the system. This is the component inwhich all information iscombined,andthefinalresponseisconstructedandsenttothestudentviatheinterface.

6.2ReplacinghumantutoringThissectionevaluatestheindividualelementswithintheATSmodel,andtheextenttowhichtheproposedsolutioncanemulatepersonaltutoring,aspresentedintheinterpersonalcommunicationmodel.ThispartoftheanalysiswillbemadewithastartingpointinthethreecomponentsoftheATS,aspresentedintheprevioussection,andthusconcernsthefollowingthreeareas;informationabout the student, the processing of emotional display, and constructing the response in thepedagogical unit. In other words, understanding the student, interpreting the message, andrespondingappropriately.UnderstandingthestudentasacommunicatorWhethercommunicatingwithateacherorinteractingwithanATS,theindividualcharacteristicsofthestudenthimselfdonotchange.Thesecharacteristics,however,remainjustasimportantwhenit comes to the ATS interpreting the messages from the student, as it is in interpersonalcommunication.Thestudentremainsthesameunique individual, inregardstoallofthefactorsmentioned in the communication model: psychological and physiological factors; perception,cognition,andpersonality;interpersonalskillsandcompetencies;aswellastheculturalcontextanddemography.Sinceeachoftheseaspectshelpdefinethestudentasacommunicator,theymustallbe consideredand implementedas thoroughlyaspossible in theATS.Due to thenatureof theinterpersonalcommunication,theindividualcharacteristicsmustbetakingintoconsiderationwhenboth decoding the messages sent from the student, and when constructing a message to thestudent. In the ATS model, the background information module stores most of the studentinformationmentionedinthissection.Thefollowingsubsectionswilldiscussvariouspsychological,physiological,andsocialfactorstoconsiderwithintheATS.

63

Perception,cognition,andpersonalityOtheraspectsrelatedtothepsychologyofthestudent,arehisperceptionandcognitiveprocesses,aswellasthecharacteristicsofhispersonality.Theseaspectsareallcomplexmentalstatesandprocesses,andarethusveryhardtoassessinastraightforwardmannerbytheATS,althoughitispossiblebyanalysingandbuildingamentalprofileofeachindividualstudent.Thiscouldaddresstraitssuchasneedsandvalues,informationandmemoryprocessing,socialcognition,aswellastheattitudes, motivation, goals and intents of the student. Such information would be importantadditionstothestudentinformationmodule,andthesubsequentlyrelatedprocessesofemotionprocessing,evaluating intent,anddecidingon the tutoring strategyand response.Furthermore,personalitytraitssuchasaffectiveorientation,assertiveness,self-esteem,andcommunicationstylecould also provide valuable informationwithin the ATS. A number of tools and tests exists forevaluationofbothperception,cognitionandpersonalitytraits,andbyimplementingsuchanalysisresults,thelearningeffectivenesscouldimprove.Apracticalexampleofapersonalityanalysistoolisdescribedinsection6.3.SocialandInterpersonalskillsIninterpersonalcommunication,socialandinterpersonalskillsandcompetenciesplayahugeroleinhowweinteractandcommunicatewithothers.Factorssuchassocialperspectiveandroletaking,trust,andsocialcompetences,definesactionsandrolesdependingonthesocialcontextsoftheinteraction.Thisespeciallyappliestoself-disclosurewithin interpersonalrelationships,deceptivebehavioursanddetectionofdeception,theuseofcompliancegainingstrategies,anddetectionandresistancetootherpeople’scompliancegainingstrategies.Despite their importance in interpersonal communication, it can be argued that several of thementionedfactorsdonotdirectlyapplywithinhuman-computerinteraction.Tomaximiselearningeffectiveness, however, the ATS should consider e.g. deceptive behaviour and behavioursassociatedwithself-disclosure,inordertofullycomprehendtheactualskilllevelandlearningspeedofthestudent,aswellasanalysingthestudent’saffectdisplaycorrectly.Detectingthesesignals,isoneaspectwherehumantutoringisoftensuperiortotheATS,andthusitmustbeimplementedwithinthesystemtooptimisetheeffectivenesshereof.Theresearchconductedthroughoutthisproject did not reveal any existing ATS with such capabilities, highlighting the potentialimprovementsthatlieswithinthisarea.CulturalcontextanddemographyIn transactional communication models, and thus interpersonal communication, culture isconsidered an overarching component, affecting all elements of the communication process(Stamp,1999).IntheATSmodel,cultureandculturalcontextaremainlyconcernedwiththestudentandthemessagecomposedbythestudent.Culturalaspectsareimportantconsiderationsinbothunderstanding the student as an individual, and in responding with appropriate feedback.

64

Communicationvaries,andsomesignalsareperceiveddifferentlywithindifferentcultures,thus,thestudent’sculturalbackgroundmustbeproperlyunderstoodinordertodecodethemessagescorrectly.Inotherwords,twostudentswithdifferentculturalbackground,mightexpressdifferentmessageswiththesameintent,orinterpretthesamemessageindifferentways.Amongstothers,the cultural context concerns information acquisition, relationship types and terms, andcommunicationproblemsandconstraints.Wherease.g.relationshiptypesandintimacylevelsaredifficulttoapplytoanATS,demographicalinformationsuchasage,gender,educationallevel,socialstatus,andevenethnicity,iseasilyimplementedandcouldprovidevaluableinformationaboutthestudent.Duetoethicalconcerns,however,thiscouldbeconsideredpersonaldata,andthusnotapplicablewithinthesystem.WhilemanyoftheseaspectsandtraitsmentionedareverydifficulttoevaluatebytheATS,theyarealso extremely difficult for people to analyse and take into consideration in a communicationcontext.Ithasbeenproventhatthesuperiorityofindividualhumantutoringisaduetotheabilityofaffectrecognitionfrommorecomplexsignalsandcues,aswellastheirpersonalknowledgeaboutthestudentandhisneeds(Erez&Isen,2002;Mao&Li,2010).Somecharacteristics,however,aremorestraightforwardandeasierevaluatedthanothers.Asanexample,demographicinformationcanbeaccuratelydescribedwithintheATSassimplekey-valuepairs(e.g.“age=21”),whereassocialidentity,personality,andculturalcontextaremuchmorecomplicatedelementstohandlewithinanATS,duetotheirmulti-facetednature.Interpretingthestudent’smessageThesecondmainfunctionalityoftheATSistointerpretthecommunicationsignalsfromthestudent.Inthecommunicationmodel,fourdifferentelementsexistwithinthemessagecomponent:verbalcues;nonverbalcues;apprehensionandencoding;andthemessagetype.Tosomeextent,thesearealladdressedintheproposedATSmodel,aswillbediscussedinthefollowingparagraphs.AnalysingverbalandnonverbalcuesByanalysingbothverbalandnonverbalcuesfromthestudent,muchmorecomplexinformationcanbederived,comparedtoonlyusingoneortheother,andasaresult, theaccuracyofaffectrecognition improves drastically (Fujishin, 2008; Lane, 2008). Verbal cues can be analysedwithnaturallanguageprocessing,fromwhichthedenotativeandconnotativemeaningcanbederived,basedonthesyntax,semantics,andpragmaticrules.Today’sspeechrecognitiontechnologiesareextremelyadeptinregardstotranscriptionandunderstandingwhatisspoken,andsomesystemsevenlearnsfromthespeakerandimprovesovertime.Whenitcomestotheactualmeaningoftheverbal content, additional factorsmust be taken into consideration, such as the accompanyingnonverbal cues and contextual information. By analysing different modalities, a wide range ofnonverbalcuescanbeevaluatedwithintheATS.Visualsareconsideredtobetheoneofthemostdominantinputinaffectrecognition,andespeciallyfacialexpressionscarryagreatamountofaffect

65

information (Lane, 2008). Along with gesture and posture recognition, facial expressions areanalysed using image processing and advanced machine learning algorithms. These cues areimportant parts of determining both the emotional tone, and the intents of themessage, alsoreferredtoasthemessagetypeandsubsequentmessagebehaviour.Whiletechnologicalsolutionscapableofdetectingemotionaltonesexists(seesection6.3),itisextremelydifficultforcomputersto detectmotiveswithin amessagewith the intention of e.g. ridicule or sarcasm, especially insystemsusingfewmodalities.Thisisoneareainwhichthetechnologyiscurrentlynotascapableasitshumancounterpart.Similarsentencescanbeconstructed,butdependingonthetonewithwhichthemessage is communicated, the intentioncangreatlydiffer.Whileparalinguistic cues canbederivedfromspeech,theresultsfromanalysingprosodicfeaturesarestillsomewayoffofhumancapabilities(Gangamohan,Kadiri&Yegnanarayana,2016).EvaluatingapprehensionandencodingAnother complex part of understanding the student’s message, is to gain insights into hisapprehension and message encoding. The encoding is a result of behaviours associated withapprehension,i.e.howthestudentperceivesthereceiver’sexpectationsandapprehension,aswellashowdifferenttypesofmessageencodingareperceivedbythereceiver.Toproperunderstandthis thought process, a deeper understanding of the student’s personality and interpersonalcompetencesisrequired,aswellasinformationaboutthestudent’sculturalbackground,andotherindividualtraitsisrelevant.Thus,theaccuracyindecodingthemessagedependspartiallyonthelevelofdetailwithinthestudentinformationcomponent,aspresentedintheprevioussection.DeterminingtheoverallemotionalstateBesidestheirrolesinunderstandingtheintentsofthestudent,analysingtheverbalandnonverbalcuesarekeyelementsindeterminingtheemotionalstate.Ininterpersonalcommunication,thisislargelyasubconsciousprocess,butinaffectivecomputingsystems,thismustbeimplementedusingadvancedalgorithms.Theresultsfromeachindividualmodalityisweightedandevaluatedaccordingto all other affect information. This also includesusingbackgroundandpersonality data,whichserves as a ground truth, from which the emotional deviations can be calculated, and thedominatingemotionalstateisdetermined.IntheATSmodel,thisevaluationhappensintheemotioninformationprocessormodule.

66

RespondingappropriatelyAfteranalysingthestudent’smessageandemotionalstate,theATSmustrespondaccordingtotheaffectdisplayandthestudent’sintent,anddoingsorequiresanumberofdifferentprocessestobecarriedout,asdescribedinthefollowingparagraphs.FunctionalityofthetutoringmodelIntheATSmodel,theresponseisdecidedinthetutoringmodel,whichdrawsoninformationfromtheemotioninformationprocessingmodule,theidentifiedintent,andthedomainmodel.Itcanbearguedthatthedomainmodelactsasacontextualidentifier,whichisthenusedtodeterminetherelevant contextual response. Together with the domainmodel, the intent and content of themessagedetermineswhatelementstheresponseshouldcontain,i.e.whattheATSresponds.Theequally importantaspectofhow itresponds, isdeterminedfromtheevaluationofthestudent’semotionalstate,basedontheoutputfromtheemotioninformationprocessingmodule.Thus,thecorrespondingdecisionmakingprocessininterpersonalcommunication,wheretheteacherdecideswhatandhowtorespondinordertomotivateandchallengethestudent,isimplementedwithinthisunitintheATS.CommunicatingtheresponseThenext step is then to actually communicate the constructed response.Whereas humans arelimited mainly by their physiological capabilities, an ATS can only communicate through thechannels enabledby the system implementation.While thedifferent outputmodalities arenotexplicitlydescribed in theconceptualmodel, theiruse isdeterminedbythetutoringmodel,butenabledby the interactionengine.A typicalATS responds to the studentusing text, speech,oranimation,butitvariesdependingontheexactpurposeandfunctionalityofagivensystem.Itcanbedifficulttoaddressspecificallywhichcuestouse,butthissystemisbuiltaroundanaturalspeechinterface,forasseamlessandnaturalaninteractionaspossible.Byreplacingtheteacherwithacomputersystem,thecharacteristicsofthesecondcommunicatorchangesradicallycomparedtointerpersonalcommunication.Justtonameafewofthechanges:thepersonality isno longershapedbyneedsandvalues, therearenoaffectingphysiologicalorsocialfactors,anddemographyandculturalbackgroundarenon-existentfactors.ByutilisingdeeplearningandenablingtheATStolearnfromitsinteractions,itcanbearguedthatfactorssuchasapprehension,compliancegaining,andinterpersonalskillsdevelopovertime,andasaresult,thedyadicrelationshipbetweenthestudentandtheATSisestablishedandevolves.Thedevelopmentoftherelationshipisextremelyimportantinregardstoe.g.affectrecognitionandcommunicationstyle.Furthermore,thecompatibilityofcommunicationstyles,andpersonalitieshasahugeeffectonthestudent’smotivationandengagement,andthusthelearningoutcome.

67

Anotherimportantaspecttoconsiderwhenresponding,istheinteractionitself.Asdescribedinthecommunicationmodel,thenatureoftheinteractionchangesovertime.Fromtheinitialinteraction,a dyadic relationship is formed between the communicators, which changes as conversationsunfold,andconflictsaremanagedandresolved.Therelationshipevolvesovertime,intoacloserrelationshipwithcertainpatternsofcommunication,andmoreexpressiveconversations.ThistypeofchangemightalsobeseenwheninteractingwiththeATS,ifthesystemiscapableoftakingintoaccountpreviousinteractionswiththestudent.Thiscanactuallybeconsideredanessentialpartinrelationtothelearningprocess,asthestudent’sknowledgeandskilliscontinuouslymonitored.Byenablingthesystemtocommunicatewithemotionandunderstandtheemotionsexpressed,thestudentmightdevelopamorepersonalrelationshipwiththeATS,similartothatofinterpersonalinteraction.ThechangingroleoffeedbackThe final component from the communication model is the feedback. When the ATS haverespondedtoaquestionorrequestfromthestudent,itcanevaluatethefeedbackfromthestudent,by continuously analysing the communication signals. This enables the system to evaluate howmessagesarereceived,andadaptiftheresponsedoesnotcorrespondtotheintendedreaction.Byutilisingthefeedbackfromtheinteraction,theATScanmoreaccuratelyadapttoeachindividualstudent profile, and as a result craftmore appropriate and personalised responses. This use offeedbackresemblesthatofitscounterpartwithininterpersonalcommunication,however,withintheATS, this process is only one-way. The student does not get the same immediate feedbacksignals from the ATS, as he would from a human tutor, where additional nonverbal cues areavailable,suchaskinesics,hapticsandproxemics.Thediscussioninthissectionhashighlightedsomeofthedifferencesandsimilaritiesbetweentheelements in theATSmodel and the interpersonal communicationprocess. For amorepracticalevaluationofhowtheATScanapplycognitivetechnologies,aprototype implementationwillbedescribedinthefollowingsection.

68

6.3ImplementationproposalTo evaluate a more concrete example of the proposed ATS, and small-scale prototype will beproposed. This section discusses how such a prototype can be constructed, and how existingcognitiveservicescanbeusedintheimplementation.SystemoverviewInsection6.1,acompleteATSarchitecturalmodelwaspresented,whichisusedasthefoundationfor this proposed prototype implementation. By simplifying and adjusting some of the existingcomponents,aswellasaddingnewfunctionalcomponents,anewmodeliscreated,whichcanbeseen in figure 17. Thus, this model does not represent a complete system architecture.Consequently,someaspectshavebeensimplifiedorleftoutcomparedtotheATSmodel.The two elementsanalysing verbal cuesandanalysing nonverbal cues in the conceptualmodelcontainedavarietyofdifferentcueswhichcouldbeusedforemotionprocessing.Inthiscase,onlytwocuesareused;facialexpressionsandspeech,whichreplacesthecuecomponentsinthemodel.Facialexpressionscanbeanalysedthroughimageprocessing,asenabledbyMicrosoft’sEmotionAPI.Ontheotherhand,thespeechsignalmustbeconvertedbeforebeingfurtherprocessed.Thisisillustratedwithtwoseparatecomponents,speechconversionandanalysingverbalcontent,astwo different cognitive services are used for this purpose; Text to speech, and Tone Analyser,respectively.Furthermore,theStudentInformationcomponentissplitintotwodifferentelements,BackgroundinformationandStudentpersonality.TheWatsonservicePersonalityInsightsanalysespersonality traits of the student, thus removing the related elements in the initial backgroundinformationcomponent.Thecontentofthedomainmodelandthetutoringstrategieswillnowbeimplemented in the interaction engine, powered by Watson’s Conversation service. Thiscomponenttakesboththeemotionalstateandtheconvertedspeechasaninput,fromwhichitcandetecttheintentandrelatedentities,thuseliminatingtheintentelementintheemotionprocessingcomponent.Finally,aspeechsynthesiselementisincluded,whichsynthesestheoutputfromtheinteraction engine, thus completing the speech interface between the student and the ATS.Elaboratedescriptionsoftheseservicesaremadeinthefollowingsection.

69

Figure17:Illustrationoftheinterrelationsbetweenthecomponentsandcognitiveservicesusedinthe

prototypemodel.

UseofcognitiveservicesThis sectionwill describe how the different cognitive services are used in the of the prototypemodel,coveringtheEmotionAPI,SpeechtoText,ToneAnalyser,PersonalityInsights,Conversation,

andTexttoSpeechservices.MicrosoftEmotionAPIAs repeatedly mentioned throughout chapter 4, nonverbal cues are extremely importantcomponentsof interpersonalcommunication,andespecially facialexpressionscarrygreataffectinformation. Through their Cognitive Services platform, Microsoft provides a facial recognitionservicecalledEmotionAPI,whichcandetectemotionsofpeopleinimagesandvideo.Ittakesanimageorvideocontainingoneormorefacesasaninput,andreturnsaconfidencefortheemotions

70

expressedbyeachidentifiedfacesintheimage(Microsoft,2017b).Thedominantemotiondetectedin the face, is considered to be the one with the highest confidence score within the definedcategories;anger,contempt,disgust,fear,happiness,neutral,sadness,andsurprise.Inthisproject,theserviceservesdifferentfunctions.Firstandforemost,itenhancesthetutoringassistant’sabilitytocorrectlyunderstandwhatisbeingcommunicated,byimprovingtheemotionalaspectofthecommunicationandinteraction.Bymeasuringtheemotionovertime,however,newrichinformationcanbeextractedfromtheinteraction.Thisrelatestobothachangeofemotionalstateovertime,butcanalsobeusedtotakefeedbackintoconsideration.WatsonSpeechtoTextOne of the ideas behind using a speech to text conversion in the system, is to allow thecommunicatortointeractwiththeassistantasnaturallyaspossible,i.e.viaaspeechinterface.Forthecomputertobeabletoprocessandevaluatethespeechinput,itmustfirstbeconvertedtotext.ThiscanbedoneusingtheSpeechtoTextservicepoweredbyWatson,whichisoneofmanyservicesenabling speech transcription capabilities. The input signal is processedbasedon grammar andlanguage structures, and is continuously re-evaluated, which means that the transcription isupdatedandcorrectedasmorespeechisheard(IBM,2017g).Theserviceallowsforprocessingofanaudio-stream,therebyenablingreal-timecapabilitiesforthesystem.WatsonToneAnalyserAsnotedinsection4.5,interpretingaspokenmessagenotonlyconcernswhatiscommunicated,butalsohowitiscommunicated.Tosomeextent,thiscanbeevaluatedbyWatson’sToneAnalyserservice. The emotional tone analysis is not performed on paralinguistic features, but insteadconductedontheverbalcuesfromthewrittentext.Usingcognitivelinguisticanalysis,Watsoncanidentifyavarietyoftonesatbothsentenceanddocumentlevel,fromwhichtheserviceevaluatesemotions,socialtendencies,andwritingstyleoftheauthor(IBM,2017h).Theemotionaltonesarethedifferenttypesofemotionsandfeelingsthatareexpressedintheinput.Theemotionsofjoy,fear,sadness,disgust,andangerareevaluatedaccordingtothelikelihoodofthem being perceived in the content. Similar to the personality analysis, the social tones areevaluatedusingtheBigFivepersonalitytraits,whichmeasurestheperson’ssocialtendencies.Thisisanextremelyinterestingfunctionality,asitenablesacomparisonbetweenthedefinedbaselineofthecommunicator,andthecurrenttendencyexpressedintheutterance.Finally,thelanguagetones describe the perceived writing style, in regards to analytical attitude and reasoning,confidenceandcertainty,andtentativenessinthetone(IBM,2017i).Inadditiontothethreedifferenttypesoftones,theservicealsoincludesaCustomerEngagementmodel,whichallows for conversationmonitoring in regards to sadness, frustration, satisfaction,

71

excitement,politeness,impoliteness,andsympathy(IBM,2017i).Theintendeduseofthismodeliswithinthecontextofcustomerserviceandsupport,butcaneasilybeapplied inamoregeneralconversational context. The insights from theToneAnalyser canbeused to refineand improvecommunications, and craft appropriate, considerate and affective response messages to thecommunicator.Assuch,itisveryapplicablewithinanATS.WatsonPersonalityInsightsToderiveadetailedoverviewof thestudent’s individualpersonality traits,Watson’sPersonalityInsightsservicecanbeused.PersonalityInsightsisoneofthemostinterestingcognitiveservicesprovidedbyWatson. Basedon a number of psychometric analyses,Watson analyses a bodyofwrittentext,andreturnsacomprehensivepersonalityprofilefortheauthoroftheinput,inthiscasethestudent(IBM,2017j).Theanalysisisperformedwithinthreedifferentareas;BigFive,Needs,andValues,andbasedontheseresults,consumptionpreferencesofthepersonareinferred.BigFivedescribeshowapersongenerallyinteractsandengageswiththeworld,isoneofthemostrenowned models for evaluating personality characteristics. As the name implies, the modelconsistsoffiveprimarydimensions,eachwithanumberofadditionalfacets(IBM,2017k):

1) Agreeableness:Describestendenciestobecompassionateandcooperativetowardothers,bytakingintoconsiderationaltruism,corporation,modesty,morality,sympathy,andtrust.

2) Conscientiousness:Thetendencytoact inanorganisedorthoughtfulway.Facetsincludeachievementstriving,orderliness,andself-discipline.

3) Extraversion:Describesaperson’stendenciestoseekstimulationinthecompanyofothers,relatingtoe.g.howenergetic,outgoing,andsociablethepersonis.

4) Emotional range:Howemotions are sensitive to the individual’s environment. Considersfacetssuchasanger,anxiety,andself-consciousness.

5) Openness:Theextenttowhichaperson isopentoexperiencewithindifferentactivities.Facetsincludeadventurousness,artisticinterests,imagination,andintellect.

Theseconddimensionofthepersonalityanalysisisconcernedwiththeperson’sneeds.Atotalof12differentneedsareanalysed:Excitement,harmony,curiosity,ideal,closeness,self-expression,liberty,love,practicality,stability,challenge,andstructure.Thethirdelementdescribesvalues,i.e.themotivatingfactorswhichinfluencetheauthor’sdecision-making.Thevaluesconsideredbytheserviceare:self-transcendence,conservation,hedonism,self-enhancement,andexcitement(IBM,2017k).Theinsightsinferredfromtheanalysiscanhelpunderstandhowapersoninteractswiththeworld,howheresonateswithhissurroundings,andthemotivatingfactorsinfluencinghisdecision-making(IBM,2017j).Itcanhelpdeterminebehaviouraltraitsandintentsfortheanalysedperson.

72

WatsonConversationDependingonthecharacteristicsofthestudentandtheanalysedmessage,theresponsemustbeappropriate and relevant, and communicated through a natural speech interface. The mostcomprehensiveanduser-friendlyserviceavailabletoday,capableofhandlingthecomplexityoftheinteractionbetweenastudentandanATS,istheIBMWatsonConversationservice.Itactsasthebrainofaninteraction,andallowsforcreationofpowerfulcognitiveconversationalapplications.The service understands natural-language text inputs and, using a variety of machine learningtechniques,returnsanappropriateresponse(IBM,2017l).Thefirststepofusingtheconversationservice,istocreateaworkspacecontainingthedialogflowandtrainingdata.Thedialogisdefinedbasedonso-calledintentsandentities.Intentsrepresentthepurposeoftheuser’sinput,suchasaquestionaboutthelearningmaterial.Byprovidingadditionalexamplekey-wordsforagivenintent,theaccuracyofintentdetectionincreases(IBM,2017l).Basedon the detected intent, a logic tree is used to navigate the dialog,which is constructed by thedeveloper,whodefinesboththeunderlyinglogicandthecorrespondinganswerstoeachintent.Entities,ontheotherhand,representsatermoranobjectthatisrelevantandrelatedtotheintents,andprovidesaspecificcontextforanintent.Anexampleofsuchanentitycouldbeaspecificcourse,book,ortopic.

Whatreallysetsthisconversationserviceapartfromitscompetitors,isthefactthatitallowsforlogicbasedonadditionalparameters,suchasemotionalstatesorpersonaldata.Thismeansthate.g. theemotional statedetectedby the facial expressionanalysis or the toneanalyser, canbeincorporatedintothelogic,fromwhichitispossibletoconstructdifferentresponses,accordingtotheemotiondisplayedbythestudent.Theresponsecouldalsobemodifiedbasedonthegenderofthestudent,orevenonthebasisofphysiologicalsignals.WatsonTexttoSpeechTocomplete the interactionandmake itasnaturalandhuman-likeaspossible, theconstructedresponsemustbecommunicatedinspeech,justastheinputsfromthestudentareinspokenwords.Aspreviouslymentioned,manydifferentconversiontoolsandservicesexists,suchastheTexttoSpeechservicepoweredbyWatson.UsingIBM’sspeech-synthesiscapabilities,theserviceconvertswrittentexttonatural-soundingspeechwithminimaldelay(IBM,2017m).Theoutputissynthesisedbasedonanumberofdifferentmodels,suchasacousticmodelsandprosodicfeatures.Usingdeepneural networks, it is possible to generate values for prosodic aspects of the speech, such asdurationandintonation.Thisisdonebasedonlinguisticattributessuchaspartofspeech,lexicalstress,positional features,andword-levelprominence (IBM,2017n).Beforegenerating the finaloutput,thetextistransformedtoalanguagedrivenbycertainlinguisticrules,suchasabbreviationexpansion,beforegeneratingthefinalpronunciation.

73

6.4Follow-upinterviewsThis section will present the thoughts behind and the results of four conducted interviews,conductedon the same four studentswhowere initially interviewedaspartof thebackgroundresearch.TheywillbepresentedtotheATSmodel,fromwhichtheconceptwillbediscussed.PurposeThepurposeoftheseinterviewsistoevaluatetheviabilityoftheproposedconcept,andtodiscussthe value creation of this solution. More specifically, the interviews aim to elaborate on thefollowingkeyareas:

1) The students’ experiences with and attitudes towards cognitive systems andpersonalisedservices.

2) Theviability,valuecreation,andpotentialuseoftheproposedATS.3) Theattitudestowardsincorporatingaffectdataandusingofbackgroundinformation

Asinthepreliminaryinterview,theseinterviewswillbesemi-structured,basedonanumberofpre-definedopen-endedquestions,andtheinterviewprocessitselfwillbeanopenconversation.Basedontheresponsesfromtheinterviews,ageneraldiscussionofthesolutionwillbemade.InterviewguideBeforedoingtheactualinterview,thestudentswillbepresentedtotheconceptualideaandmodel,togainaproperunderstandingofthefunctionalityoftheATS.Tofullycoverthefocusareaslistedabove,thefollowingquestionsareconstructedandusedasastartingpointfortheinterview:

1) WhatareyourexperienceswithusingpersonalisedsystemsorservicespoweredbyAIorcognitivecomputing?

2) InwhichareasdoyouseethelargestapplicabilityofanATSwithinalearningcontext?3) HowdoyouseetheATSbeingusedtosupportlearningandansweringquestions?4) Inyouropinion,whatwouldbethemostimportantaspectsofthesystem,inregardstothe

personalisationofthefeedback?5) CouldyouseeyourselfusingandbenefittingfromhavingaccesstoanATS?Why/whynot?6) InwhichareasdoyouthinktheATShasanadvantage/disadvantageoverhumantutoring?7) DoyouhaveanysuggestionsforchangesorimprovementsoftheATS?

Thefollowingsectionwilldescribethefindingsfromtheconductedinterviews,whilehighlightingsomeofthemostnoteworthyandrelevantresponsesfromthestudents.

74

EvaluationInthissection,theresponsesfromtheinterviewedstudentsarepresented,andevaluatedaccordingtothethreekeyareasmentionedabove.ExperienceswithandattitudestowardscognitivesystemsandpersonalisedservicesThe interviews revealed that all students have had previous encounterswith cognitive systemsthroughmobilevirtualassistants,suchasApple’sSiri,GoogleAssistants,andSamsung’sBixby.Therewasageneralconsensus,thatinteractingwiththesesystemsleftthestudentsdissatisfied,andaftertheinitialinteractions,noneofthemusetheserviceonaregularbasis.Thisdissatisfactionwasaresult of different factors, such as; the language barrier; unnatural interactions; and limitedfunctionalityandcapabilitiesofthevirtualassistant,evenwithsimplerequests.Twoofthestudentsevenstatedtheusewas“moreofagimmick,ratherthanameaningfulinteraction”.Moreover,itwasmentionedthattheylackspecificusecasesandclearpurposesofinteractingwiththesystem,somethingwhichmightmotivatethestudentstousethesystemonacontinuousbasisinthefuture.Despite the rather striking criticism, therewere very positive thoughts on the future of virtualassistants. It became clear that the students see a large potential and applicability for virtualassistants in thenear future,and that it isagreatwayofapplyingand takingadvantageof thetechnologicalcapabilitiesineverydaylife.Viability,valuecreation,andpotentialuseoftheproposedATS.The preliminary interview in section 3.2 proved that the relationship between the student andteacher is somewhat professional and restricted. This has lead to the students in some casesrefrainingfromaskingquestionsaboutcontentorlecturesthatarenotproperlyunderstood.Inthisinterview,onestudentdiscussedtheeffectanATScouldhaveonthisrelationship:“Youact inacertainway[withateacher,ed.]becauseyouwanttodisplayknowledgeandcompetence,andfear

askingtoomanyquestions.ByinteractionwithanATSinstead,suchinhibitionsareoutoftheway.

YoucanapproachtheATSmorehonestly,asyoudonotfeelaconstantjudgement,oraneedto

prove yourself, as youwould with a teacher. I see the ATS as a tool which can help solve this

problem”.TheremainingthreestudentshadsimilarattitudestowardshowtheATScouldassist,support,orevenreplacethestudent-teacherinteraction.Basedontheirownexperiences,twoofthestudentsnotedthatafterfinishingalecturewheretheteacherhasonlytoucheduponcertaintopics superfluously, the accessibility of the ATS could help them find relevant answers to anyquestions thatmightarise.Thishighlights the importanceof thedomainmodel,whichmustbecarefullyimplemented,tomaketheanswersasconciseandrelevantaspossible.All of the interviewed students responded positively towards whether or not they could sethemselvesbenefittingfromhavinganATSavailable.Theydid,however,alsomentionanumberoffactorswhicharecrucial for theviabilityof theconcept.Onesuch factor, is theaccuracyof theemotionalanalysis.“Theresponsemustmakesense,andnotbepulledoutofthinair”,onestudent

75

said, and continued “… the nonverbal signalsmust not bemisinterpreted, nor should irrelevant

signals should be considered.”An inaccurate analysis or erroneous processing of the emotionalsignals,wouldresultinlessappropriateanswersfromtheATS.Subsequently,theeffectoftheATSwilldeteriorate.Onadifferentnote,thestudentsalsorequestedthattheATSmustbeaccessible,andhaveastraightforwardandnaturalinterface,tomaketheinteractionasseamlessaspossible.Itwasrepeatedlysuggestedthattheinterfaceshouldbehands-free.AnotherfactoristhequalityoftheresponsefromtheATS.Byqualityitisunderstoodthattheresponsemustcorrelatewiththestudent’semotionalstate,bewell-formulated,andpresentedinthesamelanguageasspokenbythestudent.Besidesthepreviouslydiscussedapplicationareas,thestudentsindicatedthattheyweremostlikelytousetheATSwhenpreparingforalecture,orstudyingforanexam.TheATS’simmediateresponsetoanysurfacingquestionwouldbeverywelcome,especiallywhenundercertaintimeconstraints,andwhentheneedforananswerorfeedbackisurgent.Onestudentsuggestedthatanalysingone’scommunicationwhenpreparingapresentation,wouldbeanothergreatuseoftheATS,withthepurposeof“…gettingfeedbackonyourappearance,withoutbeing judgedbyteachersorfellowstudents.”.Anotherstudentsawthepotentialofgatheringinteractiondata,anduseittoevaluateorplanthecourses,bylookingforpatternsacrossthequestionsbeingaskedbydifferentstudents.AttitudestowardsincorporatingaffectdataanduseofbackgroundinformationGenerallyspeaking,thestudentsembracedtheideaoftakingnonverbalcommunicationsignalsintoconsiderationwhendeterminingthetutoringstrategyandsubsequentresponse.“Iliketheideaofit[theATS,ed.]readingmynonverbalsignals,todecidewhetherIaminamentalstateoffrustration

andneed tobe calmeddown,or if I ammore focused,and ready tobegivenanelaborateand

thoroughresponsetomyquestion”,onestudentsaid.Anotherstatedthatmakinguseofnonverbalsignals“…definitelymakessense,toconstructmoreaccurateandpedagogicallycorrectresponses

(…) as made possible by the added affect information”. In regards to using physiological data,however, some concern was uttered throughout the interviews. One stated that “if the extrainformationcangivebetterfeedback,thenthatisperfectlyfine”,butontheoneconditionthatthedata must be captured in an unobtrusive way, since “extensive use of equipment would be

disturbing, and definitely be a barrier of using the system”. Another student agreed with thisstatement,andsaidthat“theaddedvalueofphysiologicalsignalsisnotworthit,ifitrequiresusingacomplicatedsetupwithequipmentandsensors”.Theideaofusingacameraorother“passive”monitoringequipment,whichthestudentisnotconsciouslyawareof,wasmorepositivelyreceivedamongtheinterviewees.Twostudentswerescepticalofwhetherthecuessentwheninteractingwithacomputer,areactuallycomparabletotheircounterparts in interpersonalcommunication,e.g.with a teacherduring class.Oneof them stated that his appearance ismoreneutralwheninteractingwith a computer, compared towhenhe communicates in person, thus identifying apotentialinaccuracywithintheemotionalreadings.

76

Theuseofbackgroundinformationofthestudent,invokedsomeinterestingdiscussionsduringtheinterview.Firstofall,all intervieweescouldseegreatpotential inusingthis typeof informationwhen determining the tutoring strategy. One interviewee shared his thoughts onwhat kind ofinformationhesawthismodulebenefittingfrom:“Itcouldbeeverythingfromcoldfacts,suchas

ageandgender,butalsowhetheryouareintrovertorextrovertasaperson,orwhetheryouliketo

see things inwriting,orasdrawingsand illustrations. (…) I think that this could yieldextremely

interestingresults”.Theaspectofincorporatingcommunicationandlearningpreferences,wasalsomentionedbyasecondstudent,whosaidthathewasa“visuallearner”,soresponseswithvideosorgraphswouldbealotmorevaluabletohim,comparedtoaregularwrittenorspokenresponse.“Itcouldalsobeusedinrelationtohowstudyingmaterial ispresentedtothestudentinthefirst

place,byincorporatingindividualisedcontent,basedonknowledge,learningstyle,andpersonality”,athirdstudentsaid.Furthermore,itwasalsoaddedthatacommunicationinterfacethatusesthestudentsmothertonguewouldbeagreatadditiontothesystem.Ingeneral,thestudentswereopentoincorporatingpersonalinformationtoacertaindegree,buthadhugeconcernsaboutprivacyandprotectionofsuchpersonaldata.Onestudentstatedthat“Iamnotafraidtogiveuppersonalinformation,aslongasIbenefitfromit,andreceiveabetterand

morepersonalservice”,however,theotherintervieweeswheremorereluctanttowardstheidea.“Personaldatashouldonlybeusediftheyareproventogreatlyimproveperformanceofthesystem”,

onesaid.Anotherclaimedthat“Themorethesystemknowsaboutthesituationalcontextandthe

student,themoreinterestingtheinteractioncanbe.Butwhohasaccesstothisinformation,and

whatelsemightthedatabeusedfor?”.Theseconcernswererecurrentamongtwooftheotherstudentsaswell.KeyFindingsBasedontheabovementionedresponses,thefollowinglistshowsthekeyfindingsfromtheinterview:

- Despitepreviousexperienceswithvirtualassistants,noneofthestudentsusessuchservicesregularly,duetodissatisfactionwiththelanguageunderstanding,unnaturalinteractions,andlimitedcapabilitiesofthesystems.

- Theuseandapplicationofvirtualassistantshaveahighpotentialinthenearfuture,butduetolackofusecases,thestudentshavenotyetadoptedthetechnology.

- ATScouldbeaviablesolutiontobridgethegapwithininterpersonalcommunicationbetweenstudentandteacher,inregardstothestudents’inhibitionstoaskquestions.

- ThereisapositiveattitudetowardsusingATSasaneducationaltool,butitmustbeensuredthattheinteractionisasseamlessandnaturalaspossible

- Accuracyofemotionsignalprocessingisextremelyimportant,andtherelevanceandqualityoftheresponsemustbehigh,forthestudentstousetheATSonacontinuousbasis.

77

- Theuseofaffectdatafromnonverbalsignalswaspositivelyreceived,butitmustnotbeattheexpenseoftheuserexperience.

- Thereisaslightscepticismtowardswhetherthesamenonverbalsignalsaresentwheninteractingwithacomputersystemcomparedtocommunicatinginterpersonally.

- Extensivebackgroundinformationcouldprovidevaluableinformation,especiallyinregardstoindividuallearningstylesandpreferences.

- Thereisaconcernabouttheprivacyanddataprotectionofpersonaldatainregardstothestudentinformationmodule.

ThelearningsformtheseinterviewsshowedthatthegeneralconsensusamongtheinterviewedstudentswerethattheydefinitelyseeapotentialinusingATSinthefuture,ifthesystemsareaccurate,effective,andeasytouse.Takingthepossibilitiesofcognitivecomputingintoconsideration,theaccuracyandeffectivenessofsystemssuchasATSwillmostlikelyonlyimprovefromthispointonwards.

6.5ChaptersummaryBased on the research conducted in chapters 2-5, a conceptualmodel of an Affective TutoringSystems was presented. The model expanded upon the generic ATS architecture presented insection3.1,bytakingintoconsiderationelementsfromtheinterpersonalcommunicationprocess.The similarities and differences between this ATSmodel and the interpersonal communicationmodelfromsection4.8werethendiscussed,accordingtothethreeaspectsofunderstandingthestudent,interpretingthemessage,andrespondingappropriately.Furthermore,asecondmodelwaspresented,illustratinganddescribinghowdifferentcognitiveservicescouldbeusedtoimplementasmall-scaleATS.Finally, follow-up interviews were conducted on the same four university students from thepreliminaryinterviews,toevaluatetheviabilityofthesolution.Thisinterviewrevealedthatslightconcernsabouttheaccuracyofemotionanalysis,userexperience,anddataprotection,butthatthegeneralattitudestowardsATSwerepositive.

78

79

7.DiscussionFrom the preliminary research on intelligent and affective tutoring systems, and technologicaldevelopmentswithincognitivecomputing,thefollowingresearchquestionwasformed:

How can cognitive computing technologies be used in Affective Tutoring Systems to

emulateinterpersonalcommunicationbetweenstudentsandteachers?

ThefindingsprovedthattheinterpersonalcommunicationprocesstosomeextentcanbeemulatedinanATSsystem,byimplementingcognitiveservices.Whilethemethodologicalapproachmadeitpossible toanswer the researchquestion, itwouldbebeneficial for theproject toevaluate theprojectandproposedconceptwithexperts in the fieldof cognitiveandaffective computing,orlearningtechnologiessuchas ITSandATS.Suchevaluationcouldprovideadditional insights intoareas such as: the technological developments of cognitive services, the potential of cognitivesystems, insights into market tendencies of cognitive technologies, and the general businesspotentialofATS.ItcanbearguedthattheproposedATSmodelfromsection6.1hasbothstrengthsandweaknesses,whenitcomestothethedifferentcomponentswithinthemodel.ThestartingpointofthemodelwasbasedonthegenericarchitecturalcomponentsofATS,aspresentedbyMalekzadeh,MustafaandLahsasna(2015)andSarrafzadeh(2008),however,slightadaptationsweremadetoconformwith the interpersonal communication process. Since the focus of the project was mainly onincorporatingaffectiveinformationinasimilarmannertointerpersonalcommunication,theaffect-relatedcomponentswerenaturallydiscussedmorein-depthcomparedtocomponentslessaffectedbyemotions.Someofthecomponentswhichwereonlybrieflytouchedupon,weretheemotioninformationprocessormodule,thedomainmodel,andthetutoringmodel.TheyareallessentialcomponentsforthecorefunctionalityoftheATS,andmustbefurtherinvestigatedtofullyevaluatethepotentialofATS.Determiningtheemotionalstateoftheindividualstudentisacrucialfactorindetermininghowtorespond,highlightingtheimportanceoftheemotionprocessormodule.Someofthetheoriespresentedonhumanemotionscanbeusedasaframeworkforprocessingtheoverallemotional state, where especially Russell’s (1980) circumplex model of emotion is of interest.Similarly, the appropriate tutoring strategy should be incorporated to optimise the learningoutcome,however,theoryonlearningprocessesanddetailedpedagogicalstrategiesisnotwithinthescopeofthisprojectandhasthusnotbeenconsidered,butcouldbeconsideredinfuturework.Theprototypemodel,describingthepotential implementationofasystemprototype,illustratedhowcognitiveservicescouldbeusedinanATS.Whileasystemcouldbedevelopedaccordingtothismodel,thisisbynomeanstheonlypossiblesolution.Whereastheproposedmodelusedfacialexpression analysis and emotion-mining from text to derive affect information, it could be

80

interesting to explore the possibilities that comes with using several additional modalities.Additionalspeechcapabilities,andphysiologicaldatacouldbeincorporatedwithpresumablelargeeffects.Emotionalanalysisfromspeechisoneoftheareasinwhichthecurrenttechnologyisstillfarfromhumancapabilities,asdescribedinsection5.5.EmotionrecognitioninspeechsignalscouldbeextremelyinterestingtoincorporateintoATS,duetoamountofaffectinformationthesesignalscarryininterpersonalcommunication.Inthemodel,Microsoft’sEmotionAPIisproposedasafacialexpression analysis tool, and despite its impressive capabilities, more detailed emotionaldescriptionswould be desirable. Furthermore,manyATS use animated agents to communicateresponsesandfeedbacktothestudent,andthisisanaspectthatcouldbeinterestingtoexploreaswell.The interviews in this project gave insights into the needs and attitudes towards ATS from thestudents’perspectives,however, itcouldbeofgreat interesttoapproachtheproblemfromtheteachers’perspectivesaswell.Thiswouldprovideadditionalinsightsintoboththestudent-teacherrelationship,andtheneedforATStosupportlearninginbothsmallerandlargerclasses.Anotherwayofexpandingupontheempiricaldatagatheredintheproject,wouldbetointerviewstudentswithdifferenteducationalbackgroundsanddifferenteducationallevels.Anotherapproachcouldbe togatheradditionalquantitativedata,which could support thequalitative findings from theinterviews.Intheconductedinterviewsinsection6.4,theattitudesofstudentstowardsATSwereevaluated,however, this was based on a discussion of the conceptual models. To further support theseinterview findings, it could bebeneficial to conduct an actual user test of an implementedATSprototype.Intheinterviews,severalstudentsvoicedtheirconcernfortheusabilityofamultimodalATS,especiallyinrelationtoincorporationofphysiologicalsignals.Theirconcernwasthatgalvanicskin responsesensorsorheartbeat sensorswouldbe tooobtrusive,and thusdecrease theuserexperienceoftheATS,whichshouldbeasnaturalandseamlessaspossible.Thestudentswerealsosceptic towards whether or not the same signals are actually expressed in human-computerinteractionasininterpersonalcommunication.Apracticalimplementationwouldhelpaddressthisissueaswell.Anothermajorconcernwhichbecameevidentfromtheinterviews,wastheissueofusingpersonaldatainthebackgroundinformationmodule.Whilethestudentssawalargepotentialin using this data to personalise feedback, theywould also be reluctant to give access to suchpersonal information,withoutaclear ideaaboutwhoownsandaccessthedata.This isanissuewhichmustbeaddressedinafutureimplementation,asthepersonaldatacouldprovideextremelyvaluableinformationtoboththeemotioninformationprocessormodule,andthetutoringmodel.AnothersolutiontothisissuewouldbetoapplymachinelearningtointeractionaldatafromtheATS, which could enable profiling of the students based on their interactions instead of abackgroundprofile,thusreducingtheneedforcomprehensivepersonaldata,assuchdatamightbeinferredfromstudent-specificpatternrecognitionalgorithmsinstead.Atthispoint,however,suchanimplementationismerelyspeculation.

81

8.ConclusionToday, Intelligent Tutoring Systems (ITS), i.e. interactive systems capable of adapting to theindividual learner,canbeusedtoprovide individualisedtutoring.UsingITShasshownimprovedlearningoutcomescomparedtomosttraditionalformsofeducation,butstillfallsshortofindividualand small-group human tutoring, due to human teachers’ ability to detect and respond to theaffective state of the students. This has lead to thenotionofAffective Tutoring Systems (ATS),which,asthenameimplies,alsotakesaffectinformationaboutthestudentintoconsiderationwhendecidingonatutoringstrategyfortheindividual.BackgroundresearchwasmadewithinthefieldsofITSandATStoproperlyunderstandthecapabilitiesoftutoringsystems.Interviews were conducted with four university students, which resulted in several interestingfindings.Firstofall,itbecameevidentthatinterpersonalcommunicationwithandfeedbackfromteachershadapositiveimpactonthestudents’motivationandlearningoutcomes.Secondly,theneeds,knowledge,andlearningspeedoftheindividualstudentisrarelyconsidered,especiallyinlarger classes. Finally, the students feel certain inhibitions related to the expectations from theteacher,andhaveafearofdemonstratingalackofknowledgeandexposingthemselves,byaskingquestionstocoursematerialwhichhasnotbeenproperlyunderstood.TheremainderofthisprojectaimedtoinvestigatehowcognitivecomputingtechnologiescouldbeusedinanAffectiveTutoringSystem(ATS)toemulatetheinterpersonalcommunicationbetweenstudentsandteachers.Basedoncommunicationtheoryandresearchoncognitiveandaffectivecomputing,threemodelswerecreated:onedepictingtheinterpersonalcommunicationprocess,anotherdescribinganATSconcept,and finallyamodeldescribinga small-scaleATSprototype.The firstmodel (figure14),illustrating the interpersonal communication process, was built based on extensive theoreticalresearchwithinthefieldsofinterpersonalcommunicationandhumanemotions.Thesecondmodel,theATS(figure16),extendedtheATSarchitecturepresentedbyMalekzadeh,MustafaandLahsasna(2015),byelaboratingonthedifferentcomponentsof themodel,basedoncommunicationandemotion theory. Using the interpersonal communication model as a reference point, it wasdiscussedtowhichextenttheATSconceptemulatedtheinterpersonalcommunicationprocess,inregardstounderstandingthestudent,interpretingthestudent’smessage,andcommunicatinganappropriateresponse.Itbecameevidentthatmanyaspectsofthecommunicationprocessarealsoapplicable in the ATS system, such as: considering individual background information wheninterpretingthestudent’smessages;analysingbothverbalandnonverbalcommunicationsignalsinmultiple channels; andadapting the responseaccording to thedetectedemotional stateof thestudent. That being said, many of the aspects and processes taking place in interpersonalcommunication are extremely difficult to represent, analyse, and replicate in a computer.Understanding and incorporating cultural context, apprehension, self-compliance, andmessageencoding,areexamplesofsuchcomplexprocesses.

82

The thirdmodel (figure 17)was created to illustrate how cognitive services could be used andcombined inanATS.Byreviewingdifferentcognitivedevelopmentplatforms, itbecameevidentthat a number of cognitive serviceswere available for developers,which enabled e.g. emotionrecognitionfromfacialexpressions,emotionaltoneanalysis,andelaboratepersonalityanalysis.Thefunctionalityof these servicesaredirectly relatable to someof theaffect recognitionprocessesoccurringininterpersonalcommunication.Increased sets of trainingdata, new technological developments, reducedhardwareprices, andresearchandinnovationwithincomputerscience,psychology,cognitivescienceandsocialsciences,arejustsomeofthefactorsthatwill increasethecapabilitiesofcognitivecomputinginthenearfuture. Powered by deep learning and neural networks, cognitive systems have already provencapableofoutperforminghumans inanumberoftasksthatwerepreviouslyunimaginable,thusmakingitdifficulttoarguewhycognitivecomputingshouldnotrevolutionisetheeducationalsectoraswell.Thefindingsfromtheprojectprovedthatitispossibletoemulatetheinterpersonalcommunicationprocess in an ATS system, to a certain degree. State of the art cognitive technologies enableprocessingofcomplexmultimodalsignals,similartohowcommunicationsignalsareprocessedinthehumanbrain,buttheiroverallcapabilitiesdonotyetcomparetothecapabilitiesofthehumanbrain. While processing power will only increase in the future, forecasting the upper limit ofinformationprocessingcapabilitieswithinthesystemisextremelydifficult.

83

9.FutureworkDespitegeneratinganumberofinterestingfindings,therearestillnumerousareaswhichcouldbeinvestigated,togeneratefurtherresultsandinsightsintotheareaofcognitivecomputinginATS.ThenextsteptowardsamoretangibleprojectwouldbetoimplementtheATSproposedinfigure17.Implementingandevaluatinganactualprototypewouldallowformoreelaborateusertesting,but it would also require additional knowledge on e.g. learning processes and pedagogicalstrategies, to proper implement the tutoringmodel of the ATS. It is also necessary to create amethodoralgorithmtohandlethedifferentAPIresponses,andweighandcomputetheemotionalscores.Thiscouldbedonebyevaluatingtheemotionsinanarousal-valencespace,aspresentedinRussell (1980). Another relevant addition to the current projectwould be to include additionalstakeholders,suchasteachers,amorediversegroupofstudents,orexpertsoncognitivetechnologyandbusinessdevelopment.

84

85

References- Alexander,S.,Sarrafzadeh,A.,&Hill,S.(2008).Foundationofanaffectivetutoringsystem:

Learninghowhumantutorsadapttostudentemotion.Internationaljournalofintelligentsystemstechnologiesandapplications,4(3-4),355-367.

- Altrov,R.(2013).Aspectsofculturalcommunicationinrecognizingemotions.Trames,(2),159-174.

- Andersen,P.A.,&Guerrero,L.K.(Eds.).(1997).Handbookofcommunicationandemotion:

Research,theory,applications,andcontexts.AcademicPress.- Argyle,M.(1975).BodilyCommunication.NewYork:InternationalUniversitiesPress.- Azevedo,F.A.,Carvalho,L.R.,Grinberg,L.T.,Farfel,J.M.,Ferretti,R.E.,Leite,R.E.,...&

Herculano-Houzel,S.(2009).Equalnumbersofneuronalandnonneuronalcellsmakethehumanbrainanisometricallyscaled-upprimatebrain.JournalofComparativeNeurology,513(5),532-541.

- Bahreini,K.,Nadolski,R.,&Westera,W.(2016).Towardsmultimodalemotionrecognitionine-learningenvironments.InteractiveLearningEnvironments,24(3),590-605.DOI:10.1080/10494820.2014.908927

- Bauer,C.(2014,October27).BeyondSiri,GoogleNowandCortana:Whatisthefutureof

virtualassistants?Accessed16/4/2017.Retrievedfromhttps://thenextweb.com/insider/2014/10/27/beyond-siri-google-now-cortana-future-virtual-assistants/

- Bänziger,T.,Grandjean,D.,&Scherer,K.R.(2009).Emotionrecognitionfromexpressionsinface,voice,andbody:TheMultimodalEmotionRecognitionTest(MERT).Emotion,9(5),691.

- Bengio,Y.(2009).LearningdeeparchitecturesforAI.Foundationsandtrends®inMachine

Learning,2(1),1-127.- Berlo,D.K.(1960).Theprocessofcommunication:Anintroductiontotheoryandpractice.

NewYork,NY:Holt,RinehartandWinston.- Bochner,A.P.(1989).Interpersonalcommunication.InE.Barnouw,G.Gerbner,W.Schramm,

T.L.Worth,&L.Gross(Eds.),Internationalencyclopediaofcommunications(pp.336-340).NewYorkOxfordUniversityPress.

- Chandler,D.,&Munday,R.(2011a).Verbalcommunication.Adictionaryofmediaandcommunication.OUPOxford.

- Chandler,D.,&Munday,R.(2011b).Nonverbalcommunication.Adictionaryofmediaandcommunication.OUPOxford.

- Cheney,G.(2011).Organizationalcommunicationinanageofglobalization:Issues,

reflections,practices.LongGrove,IL:WavelandPress.�- Clover,J.(2016,October25).TimCook:'WeDon'tBuy'theNeedtoGiveUpPrivacyforAI.

Accessed16/4/2017.Retrievedfromhttps://www.macrumors.com/2016/10/25/apple-artificial-intelligence-privacy/

86

- CognitiveComputingForum(2014).ComputingForum2014.Accessed16/4/2017.Retrievedfromhttp://www.cognitivecomputingforum.com

- Corbyn,Z.(2016,January31).MeetViv:theAIthatwantstoreadyourmindandrunyourlife.Accessed16/4/2017.Retrievedfromhttps://www.theguardian.com/technology/2016/jan/31/viv-artificial-intelligence-wants-to-run-your-life-siri-personal-assistants

- Cowie,R.(2005).Emotion-orientedcomputing:Stateoftheartandkeychallenges.HumaineNetworkofExcellence.

- Deng,L.,&Yu,D.(2014).Deeplearning:methodsandapplications.FoundationsandTrends®inSignalProcessing,7(3–4),197-387.

- Earley,S.,2015.Cognitivecomputing,analyticsandpersonalization.ITProfessional17(4),pp.12-18.IEEE.DOI:10.1109/MITP.2015.55

- Ekman,P.(1978).Facialactioncodingsystem.PaloAlto,CA:ConsultingPsychologistsPress.- Ekman,P.,Friesen,W.V.,&Hager,J.C.(2002).Facialactioncodingsystem.SaltLakeCity,UT:

AHumanFace.- Erez,A.,&Isen,A.M.(2002).TheInfluenceofpositiveaffectonthecomponentsof

expectancymotivation.JournalofAppliedPsychology,87(6),1055–1067.doi:10.1037/0021-9010.87.6.1055.

- Equals3(2016).HowdoesLucywork.Accessed30/4/2017.Retrievedfromhttp://www.equals3.ai/howdoeslucywork

- Farías,D.I.H.,Patti,V.,&Rosso,P.(2016).IronydetectioninTwitter:Theroleofaffectivecontent.ACMTransactionsonInternetTechnology(TOIT),16(3),19.

- Foley,M.J.(2017,February7).MicrosofttomakemoreCognitiveServicestoolsavailableto

developers.Accessed6/6/2017.Retrievedfromhttp://www.zdnet.com/article/microsoft-to-make-more-cognitive-services-tools-availabile-to-developers/

- Fujishin,R.(2008).Creatingcommunication:Exploringandexpandingyourfundamental

communicationskills.Rowman&LittlefieldPublishers.- Gallois,C.(1993).Thelanguageandcommunicationofemotion:universal,interpersonal,or

intergroup?.AmericanBehavioralScientist,36(3),309-338.- Gamble,T.K.,&Gamble,M.W.(2013).Interpersonalcommunication:Buildingconnections

together.SagePublications.- Gangamohan,P.,Kadiri,S.R.,&Yegnanarayana,B.(2016).AnalysisofEmotionalSpeech-A

Review.TowardRoboticSociallyBelievableBehavingSystems,1,205-238.SpringerInternationalPublishing.

- Goodfellow,I.,Bengio,Y.,&Courville,A.(2016).Deeplearning.MITPress.- Google(2017).GoogleCloudComputing,HostingServices&APIs.Accessed6/6/2017.

Retrievedfromhttps://cloud.google.com- Graesser,A.C.,Conley,M.,&Olney,A.(2011).Intelligenttutoringsystems.InK.R.Harris,S.

Graham,&T.Urdan(Eds.),APAeducationalpsychologyhandbook:Vol.3.Applicationstolearningandteaching(pp.451–473).Washington,DC:AmericanPsychologicalAssociation.

87

- Guo,Y.,Liu,Y.,Oerlemans,A.,Lao,S.,Wu,S.,&Lew,M.S.(2016).Deeplearningforvisualunderstanding:Areview.Neurocomputing,187,27-48.

- Hall,E.T.(1966).TheHiddenDimension.NewYork:Doubleday.- Hassabis,D.(2016,March16).WhatwelearnedinSeoulwithAlphaGo.Accessed30/4/2017.

Retrievedfromhttps://blog.google/topics/machine-learning/what-we-learned-in-seoul-with-alphago/

- Hertenstein,M.J.,Keltner,D.,App,B.,Bulleit,B.A.,&Jaskolka,A.R.(2006).Touchcommunicatesdistinctemotions.Emotion,6(3),528-533.

- HewlettPackardEnterprises(2017).HavenOnDemand.Accessed16/4/2017.Retrievedfromhttps://www.havenondemand.com

- Hinton,G.E.,Osindero,S.,&Teh,Y.W.(2006).Afastlearningalgorithmfordeepbeliefnets.Neuralcomputation,18(7),1527-1554

- Hoey,J.(2017).Lecture2-OverviewofEmotionsandAffectiveComputing.Presentation,January9,2017.SchoolofComputerScience.UniversityofWaterloo.

- IBM(2014a,October7).IBMWatson:Howitworks.Accessed13/2/2017.Retrievedfromhttps://www.youtube.com/watch?v=_Xcmh1LQB9I.

- IBM(2014b,February24).IBMInvests$1BtoDeliverUniquePlatform-as-a-Service

CapabilitiestoConnectEnterpriseDataandApplicationstotheCloud.Accessed6/6/2017.Retrievedfromhttp://www-03.ibm.com/press/us/en/pressrelease/43257.wss

- IBM(2017a).CognitiveComputing.Accessed16/4/2017.Retrievedfromhttp://www-03.ibm.com/software/products/en/category/SW77A

- IBM(2017b).IBMWatson–BuildyourcognitivebusinesswithIBM.Accessed16/4/2017.Retrievedfromhttps://www.ibm.com/watson/

- IBM(2017c).IBMWatsonAnalytics.Accessed16/4/2017.Retrievedfromhttps://www.ibm.com/analytics/watson-analytics/us-en/

- IBM(2017d).IBMWatsonforOncology.Accessed13/2/2017.Retrievedfromhttps://www.ibm.com/watson/health/oncology/

- IBM(2017e).Catalog–IBMBluemix.Accessed6/6/2017.Retrievedfromhttps://console.ng.bluemix.net/catalog/

- IBM(2017f).TradeoffAnalytics.Accessed23/3/2017.Retrievedfromhttps://console.ng.bluemix.net/catalog/services/tradeoff-analytics/

- IBM(2017g).AboutSpeechtoText.Accessed28/4/2017.Retrievedfromhttps://www.ibm.com/watson/developercloud/doc/speech-to-text/index.html

- IBM(2017h).OverviewoftheToneAnalyserservice.Accessed28/4/2017.Retrievedfromhttps://www.ibm.com/watson/developercloud/doc/tone-analyzer/index.html

- IBM(2017i).Understandyourtonescore.Accessed28/4/2017.Retrievedfromhttps://www.ibm.com/watson/developercloud/doc/tone-analyzer/understand-tone.html

- IBM(2017j)PersonalityModels.Accessed28/4/2017.Retrievedfromhttps://www.ibm.com/watson/developercloud/doc/personality-insights/user-overview.html

88

- IBM(2017k).PersonalityModels.Accessed28/4/2017.Retrievedfromhttps://www.ibm.com/watson/developercloud/doc/personality-insights/models.html

- IBM(2017l).Conversation|Gettingstarted|IBMDeveloperCloud.Accessed30/4/2017.Retreivedfromhttps://www.ibm.com/watson/developercloud/doc/conversation/getting-started.html

- IBM(2017m).TexttoSpeech|AboutTexttoSpeech.Accessed30/4/2017.Retrievedfromhttps://www.ibm.com/watson/developercloud/doc/text-to-speech/index.html

- IBM(2017n).TexttoSpeech|Thesciencebehindtheservice.Accessed30/4/2017.Retrievedfromhttps://www.ibm.com/watson/developercloud/doc/text-to-speech/science.html

- Irfan,M.T.,&Gudivada,V.N.(2016).CognitiveComputingApplicationsinEducationandLearning.HandbookofStatistics,35,283-300.

- Jackson,J.(2011,February16).IBMWatsonVanquishesHumanJeopardyFoes.Accessed16/4/2017.Retrievedfromhttp://www.pcworld.com/article/219893/ibm_watson_vanquishes_human_jeopardy_foes.html

- Kelly,J.E.,&Hamm,S.(2013).Smartmachines:IBM'sWatsonandtheeraofcognitive

computing.NewYork:ColumbiaBusinessSchoolPublishing.- Kendon,A.(2007).SomeTopicsinGestureStudies.InA.Esposito(Ed.),Fundamentalsof

VerbalandNonverbalCommunicationandtheBiometricIssue,18.IOSPress.- Keyton,J.(2011).Communicationandorganizationalculture:Akeytounderstandingwork

experience.ThousandOaks,CA:Sage.�- Kittlaus,D.(2016,May9).BeyondSiri:TheWorldPremiereofVivwithDagKittlaus.Accessed

16/4/2017.Retrievedfromhttps://www.youtube.com/watch?v=Rblb3sptgpQ- Kulik,J.A.,&Fletcher,J.D.(2016).Effectivenessofintelligenttutoringsystems:ameta-

analyticreview.ReviewofEducationalResearch,86(1),42-78.- LangtonN.,&RobbinsS.P.(2006).FundamentalsofOrganizationalBehavior.Ontario:

Pearson.- Lane,S.D.(2008).Interpersonalcommunication:Competenceandcontexts.Pearson.- Langlotz,A.,&Locher,M.A.(2013).Theroleofemotionsinrelationalwork.Journalof

Pragmatics,58,87-107.- LeCun,Y.,Bengio,Y.,&Hinton,G.(2015).Deeplearning.Nature,521(7553),436-444.- Lövheim,H.(2012).Anewthree-dimensionalmodelforemotionsandmonoamine

neurotransmitters.Medicalhypotheses,78(2),341-348.doi:10.1016/j.mehy.2011.11.016.PMID22153577.

- Lunenburg,F.C.(2010).Communication:Theprocess,barriers,andimprovingeffectiveness.Schooling,1(1),1-11.

- Ma,W.,Adesope,O.O.,Nesbit,J.C.,&Liu,Q.(2014).Intelligenttutoringsystemsandlearningoutcomes:Ameta-analysis.JournalofEducationalPsychology106(4),901-918.AmericanPsychologicalAssociation

89

- Maiocchi,M.(2015).PerceptionandEmotions.TheNeuroscientificBasisofSuccessfulDesign,25-40.SpringerInternationalPublishing.

- Malekzadeh,M.,Mustafa,M.B.,&Lahsasna,A.(2015).AReviewofEmotionRegulationinIntelligentTutoringSystems.EducationalTechnology&Society,18(4),435-445.

- Manning,C.&Socher,R.(2017).CS224n:NaturalLanguageProcessingwithDeepLearning.StanfordUniversity.

- Martin,E.A.(2010a).neuron.InConciseMedicalDictionary,Eighthedition.- Martin,E.A.(2010b).synapse.InConciseMedicalDictionary,Eighthedition.- Merriam-Webster(2017).Cognition.Accessed16/4/2017.Retrievedfrom

https://www.merriam-webster.com/dictionary/cognition- Microsoft(2016).CognitiveservicesAPIs.Accessed16/4/2017.Retrievedfrom

https://www.microsoft.com/cognitive-services/en-us/- Microsoft(2017a).CognitiveServices.Accessed6/6/2017.Retrievedfrom

https://azure.microsoft.com/da-dk/services/cognitive-services/- Microsoft(2017b).MicrosoftCognitiveservices-EmotionAPI.Accessed28/4/2017.Retrieved

fromhttps://www.microsoft.com/cognitive-services/en-us/emotion-api- Miell,D.E.(1984).Cognitiveandcommunicativestrategiesindevelopingrelationships.

Doctoraldissertation,UniversityofLancaster.- Mongrain,M.,&Vettese,L.C.(2003).Conflictoveremotionalexpression:Implicationsfor

interpersonalcommunication.PersonalityandSocialPsychologyBulletin,29(4),545-555.- Motahari,H.(2014,March19).CognitiveWorkAssistants-VisionandOpenChallenges.

[PowerPointPresentation].Accessed16/4/2017.Retrievedfrom

https://www.slideshare.net/hrmn/cognitive-assistants-vision-and-open-challenges- Nesbitt,K.,Blackmore,K.,Hookham,G.,Kay-Lambkin,F.,&Walla,P.(2015).Usingthestartle

eye-blinktomeasureaffectinplayers.InSeriousGamesAnalytics,401-434.SpringerInternationalPublishing

- Nowicki,S.,&Duke,M.P.(1994).Individualdifferencesinthenonverbalcommunicationofaffect:Thediagnosticanalysisofnonverbalaccuracy.JournalofNonverbalBehavior,18,9–35.

- Ogden,C.K.,&Richards,I.A.(1923).Themeaningofmeaning:Astudyoftheinfluenceof

thoughtandofthescienceofsymbolism.NewYork:Harcourt,Brace&World,Inc.- OxfordDictionaries(2017).Cognition.Accessed16/4/2017.Retrievedfrom

https://en.oxforddictionaries.com/definition/cognition- Pantic,M.,&Rothkrantz,L.J.M.(2003).TowardanAffect-SensitiveMultimodalHuman-

ComputerInteraction.ProceedingsoftheIEEE,91(9).IEEE.- Parrott,W.(2001).EmotionsinSocialPsychology.KeyReadingsinSocialPsychology.

Philadelphia:PsychologyPress.ISBN978-0863776830.- Partan,S.R.,&Marler,P.(2005).Issuesintheclassificationofmultimodalcommunication

signals.TheAmericanNaturalist,166(2),231-245.- Pereira,D.T.,Schatsky,D.,Sallomi,P.&Dalton,R.(2015,December15).Cognitive

technologiesinthetechnologysector.Fromsciencefictionvisiontoreal-worldvalue.Accessed

90

16/4/2017.Retrievedfromhttps://dupress.deloitte.com/dup-us-en/focus/cognitive-technologies/artificial-intelligence-in-technology-sector-tmt.html

- Pesenti,J.(2014,December4).CognitiveComputing.Accessed6/6/2017.Retrievedfromhttps://www.youtube.com/watch?v=8zYp4yH4PoQ

- Pipas,M.D.,&Jaradat,M.(2014).Perceptionofemotionininterpersonalcommunication.Calitatea,15(S3),96-101.

- Planalp,S.(1998).Communicatingemotionineverydaylife:Cues,channels,andprocesses.Handbookofcommunicationandemotion:Research,theory,applications,andcontexts,29-48.

- Plutchik,Robert(1980),Emotion:Theory,research,andexperience:Vol.1.Theoriesofemotion,1,NewYork:Academic.

- Protalinski,E.(2017).Google’sspeechrecognitiontechnologynowhasa4.9%worderrorrate.Accessed30/5/2017.Retrievedfromhttps://venturebeat.com/2017/05/17/googles-speech-recognition-technology-now-has-a-4-9-word-error-rate/

- Qadar,F.(2016).Self-Talk:EffectsonEmotioninInterpersonalCommunicationContext.TheUniversityofArizona.

- Raghavanetal.,(2016).CognitiveComputing:Theoryandapplications.Elsevier.- Roe,C.(2014,May8).ABriefHistoryofCognitiveComputing.Accessed16/4/2017.Retrieved

fromhttp://www.dataversity.net/brief-history-cognitive-computing/- Rosenthal,R.,Hall,J.A.,DiMatteo,M.R.,Rogers,P.L.,&Archer,D.(1979).Sensitivityto

nonverbalcommunication:ThePONStest.Baltimore:JohnHopkinsUniversityPress.- Rouse,M.(2016).CognitiveComputing.Accessed16/4/2017.Retrievedfrom

http://whatis.techtarget.com/definition/cognitive-computing- Rubin,R.B.,Perse,E.M.,&Barbato,C.A.(1988).Conceptualizationandmeasurementof

interpersonalcommunicationmotives.HumanCommunicationResearch,14(4),602-628.- Russell,J.A.(1980).Acircumplexmodelofaffect.JournalofPersonalityandSocial

Psychology,39(6),1161-1178.doi:10.1037/h0077714- Samsung(2016,October6).SamsungtoAcquireViv,theNextGenerationArtificialIntelligence

Platform.Accessed16/4/2017.Retrievedfromhttps://news.samsung.com/global/samsung-to-acquire-viv-the-next-generation-artificial-intelligence-platform.

- Saon,G.(2017,March).Reachingnewrecordsinspeechrecognition.Accessed30/5/2017.Retrievedfromhttps://www.ibm.com/blogs/watson/2017/03/reaching-new-records-in-speech-recognition/

- Sarrafzadeh,A.,Alexander,S.,Dadgostar,F.,Fan,C.,&Bigdeli,A.(2008).“HowdoyouknowthatIdon’tunderstand?”Alookatthefutureofintelligenttutoringsystems.Computersin

HumanBehavior,24(4),1342-1363.- Scherer,K.R.,&Ellgring,H.(2007).Multimodalexpressionofemotion:Affectprogramsor

componentialappraisalpatterns?.Emotion,7(1),158.

91

- Scherer,K.R.,&Scherer,U.(2011).Assessingtheabilitytorecognizefacialandvocalexpressionsofemotion:ConstructionandvalidationoftheEmotionRecognitionIndex.JournalofNonverbalBehavior,35(4),305.

- Schmidhuber,J.(2015).Deeplearninginneuralnetworks:Anoverview.Neuralnetworks,61,85-117.

- Schramm,W.(1954).Howcommunicationworks.InW.Schramm(Ed.),Theprocessandeffectsofcommunication,3-26.Urbana,Illinois:UniversityofIllinoisPress.

- Shannon,C.E.,&Weaver,W.(1949).Themathematicaltheoryofcommunication.UniversityofIllinoisPress.

- SoftBankRobotics(2017).Pepperrobot:characteristics.Accessed30/4/2017.Retrievedfromhttps://www.ald.softbankrobotics.com/en/cool-robots/pepper/find-out-more-about-pepper

- Sottilare,R.,Graesser,A.,Hu,X.,&Holden,H.(Eds.).(2013).Designrecommendationsfor

IntelligentTutoringSystems.Orlando,FL:U.S.ArmyResearchLaboratory.- Stamp,G.H.(1999).AQualitativelyConstructedInterpersonalCommunicationModel:A

GroundedTheoryAnalysis.HumanCommunicationResearch,25(4),531-547.- Steenbergen-Hu,S.,&Cooper,H.(2014).Ameta-analysisoftheeffectivenessofIntelligent

TutoringSystems(ITS)oncollegestudents’academiclearning.JournalofEducationalPsychology,106,331–347.doi:10.1037/a0034752

- Sudhkar,R.S.,&Anil,M.C.(2016).EmotionDetectionofSpeechSignalswithAnalysisofSalientAspectPitchContour.Emotion,3(10).

- SunX.,Wu,P.,&Hoi,S.C.(2017).FaceDetectionusingDeepLearning:AnImprovedFaster

RCNNApproach.Accessed8/6/2017.Retrievedfromhttps://arxiv.org/pdf/1701.08289.pdf- Tannen,D.(1986).That’snotwhatImeant!.NewYork:Ballantine.- Tao,J.,&Tan,T.(2005).AffectiveComputing:AReview.AffectiveComputingandIntelligent

InteractionLectureNotesinComputerScience,981-995.doi:10.1007/11573548_125- Tung,L.(2016,September).Microsoft'snewestmilestone?World'slowesterrorrateinspeech

recognition.Accessed30/5/2017.Retrievedfromhttp://www.zdnet.com/article/microsofts-newest-milestone-worlds-lowest-error-rate-in-speech-recognition/

- VanLehn,K.(2011).Therelativeeffectivenessofhumantutoring,IntelligentTutoringSystems,andothertutoringsystems.EducationalPsychologist,46,197–221.doi:10.1080/00461520.2011.611369

- Viola,P.,&Jones,M.(2001).Robustreal-timefacedetection.ProceedingsEighthIEEEInternationalConferenceonComputerVision.ICCV2001.doi:10.1109/iccv.2001.937709

- Viv(2017).Viv.Accessed16/4/2017.Retrievedfromhttp://viv.ai- Waters,R.(2016,September26).Siri,Alexa,Cortanaandtheunstoppableriseofthedigital

assistant.Accessed16/4/2017.Retrievedfromhttp://www.afr.com/technology/apps/business/siri-alexa-cortana-and-the-unstoppable-rise-of-the-digital-assistant-20160925-grnxvj

- Yadollahi,A.,Shahraki,A.G.,&Zaiane,O.R.(2017).CurrentStateofTextSentimentAnalysisfromOpiniontoEmotionMining.ACMComputingSurveys(CSUR),50(2),25.