multimodal detection of affective states | chi 2012

Upload: javier-gonzalez-sanchez

Post on 03-Apr-2018

215 views

Category:

Documents


0 download

TRANSCRIPT

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    1/120

    Ouctmoa`fc @hthitmak ae Feehitmvh Ytfths4F Zaf`ofp erao Jrfmk/Iaoputhr Mkthrefihs# Efih/Jfsh` Hoatmak Zhiagkmtmak#

    Hyh Prfilmkg Yysthos fk` Atbhr Yhksars

    Dfvmhr Gakzfchz/Yfkibhz# Ofrmf/Hchkf Ibfvhz/HibhfgfrfyZajhrt Ftlmksak# Wmkscaw Jurchsak# Zajhrt Ibrmstapbhrsak

    Yibaac ae Iaoputmkg# Mkearoftmis# fk` @himsmak Yysthos HkgmkhhrmkgFrmzakf Ytfth Xkmvhrsmty

    Iapyrmgbt ms bhc` jy tbh futbar!awkhr"s-+IBM6:# Ofy 86># :>6:# Fustmk# Phxfs# XYF+

    FIO 6:!>8+ 6

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    2/120

    Chfrkmkg Yimhkihs Zhshfrib Cfj | Oatmvftmakfc Hkvmrakohkts GraupYibaac ae Iaoputmkg# Mkearoftmis# fk` @himsmak Yysthos Hkgmkhhrmkg

    Frmzakf Ytfth Xkmvhrsmty

    :

    Fjaut Xs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    3/120

    Pbms warl wfs supparth` jy Aestih ae Kfvfc Zhshfrib uk`hr Grfkt K>>>60/6>/6/>609

    Zajhrt Ftlmksak

    @r+ Zajhrt Ftlmksak ms fk Fssaimfth ^raehssar mk tbh Mrf F+ Yibaacs aeHkgmkhhrmkg fk` mk tbh Ofry Cau Euctak Phfibhrs Iacchgh+ Bmsrhshfrib hxpcarhs tbh mkthrshitmak ae iagkmtmvh simhkih# mkearoftmis#

    mkstruitmakfc `hsmgk# fk` h`uiftmakfc thibkacagy+Bms sibacfrsbmp mkvacvhs tbh `hsmgk ae mkstruitmakfc ofthrmfc#mkicu`mkg jaals# fk` iaoputhr/jfsh` chfrkmkg hkvmrakohktsfiiar`mkg ta aur uk`hrstfk`mkg ae buofk iagkmtmvh fribmthiturh fk`baw ta chvhrfgh mts ukmquh iakstrfmkts fk` feear`fkihs+Bms iurrhkt rhshfrib eaius mkvacvhs tbh stu`y ae hkgfghohkt fk`

    ecaw mk gfohs+

    9

    ^rmkimpfc Mkvhstmgftar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    4/120

    Pbms warl wfs supparth` jy Aestih ae Kfvfc Zhshfrib uk`hr Grfkt K>>>60/6>/6/>609

    Wmkscaw Jurchsak

    @r+ Wmkscaw Jurchsak rhihmvh` bms ^b@ erao tbh OMP Oh`mf Cfj#Feehitmvh Iaoputmkg Graup+Bh damkh` FYX's Yibaac ae Iaoputmkg fk` Mkearoftmis fk` tbh Frts#Oh`mf# fk` Hkgmkhhrmkg grf`ufth pragrfo ft FYX mk :>>5+ Bh bfswarlh` wmtb tbh Hktrhprhkhurmfc Ofkfghohkt Xkmt ft tbh Bfrvfr`Jusmkhss Yibaac# @hutsibh Phchlao Cfjarftarmhs# YHPM Mkstmtuth#fk` MJO's Fcof`hk Zhshfrib Ihkthr wbhrh bh wfs fwfr`h` thkpfthkts+Bh bac`s fk OYH erao Ytfkear` Xkmvhrsmty's Ohibfkmifc Hkgmkhhrmkg

    ^ra`uit @hsmgk ^ragrfo fk` f JF mk Jma/^bysmis erao ZmihXkmvhrsmty+ Bh bfs jhhk f ia/^rmkimpfc Mkvhstmgftar ak tbh BujjchYpfih Phchsiaph's Mkvhstmgftmak ae Jmkfry Fsthram`s fk` iaksuctfktta XKMIHE fk` tbh Warc` Yiaut Jurhfu+

    0

    ^rmkimpfc Mkvhstmgftar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    5/120

    Pbms warl wfs supparth` jy Aestih ae Kfvfc Zhshfrib uk`hr Grfkt K>>>60/6>/6/>609

    Dfvmhr Gakzfchz/Yfkibhz

    @aitarfc stu`hkt mk Iaoputhr Yimhkih ft Frmzakf Ytfth Xkmvhrsmty+Bms rhshfrib mkthrhsts frh saetwfrh fribmthiturh# saetwfrh

    hkgmkhhrmkg# feehitmvh iaoputmkg# fk` h`uiftmakfc thibkacagy+Bh bac`s fk OY mk Iaoputhr Yimhkih erao tbh Ihkthr ear Zhshfribfk` F`vfkih` Ytu`mhs ae tbh Kftmakfc ^acythibkmi Mkstmtuth fk` f JYmk Iaoputhr Hkgmkhhrmkg erao tbh Xkmvhrsmty ae Guf`fcfdfrf+Bms hxphrmhkih mkicu`hs 6:, yhfrs fs saetwfrh hkgmkhhr# fk` =,yhfrs thfibmkg uk`hrgrf`ufth iaurshs ft Phikacagmia `h Oakthrrhyfk` grf`ufth iaurshs ft Xkmvhrsm`f` `h Guf`fcfdfrf+

    8

    Grf`ufth Ytu`hkt

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    6/120

    Pbms warl wfs supparth` jy Aestih ae Kfvfc Zhshfrib uk`hr Grfkt K>>>60/6>/6/>609

    Bhchk Ibfvhz/Hibhfgfrfy

    @aitarfc stu`hkt mk Iaoputhr Yimhkih ft Frmzakf Ytfth Xkmvhrsmty+Bhr mkthrhsts frh mk tbh frhfs ae feehitmvh iaoputmkg# h`uiftmakfcthibkacagy "mkicu`mkg rajatmis-# fk` chfrkmkg praihsshs+

    Pbh Phikacgmia `h Oakthrrhy ifopus Guf`fcfdfrf iakehrrh` upakbhr tbh `hgrhh ae OY ak Iaoputhr Yimhkih fk` tbh `hgrhh ae JY akIaoputhr Yysthos Hkgmkhhrmkg+Jhearh stfrtmkg bhr ^b@ pragrfo sbh wfs f efiucty ohojhr ear 2,yhfrs ft Phikacagmia `h Oakthrrhy+ Bhr hxphrmhkih mkicu`hsf`omkmstrftmvh pasmtmak fk` saetwfrh `hvhcaphr+

    5

    Grf`ufth Ytu`hkt

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    7/120

    Pbms warl wfs supparth` jy Aestih ae Kfvfc Zhshfrib uk`hr Grfkt K>>>60/6>/6/>609

    Zajhrt Ibrmstapbhrsak

    @aitarfc stu`hkt mk H`uiftmak Phibkacagy+ Bms mkthrhsts frh mkfeehitmvh iaoputmkg# fkmofth` ph`fgagmifc fghkts# shce/rhgucfth`chfrkmkg# heehitmvh ehh`jfil# fk` hvfcuftmak ae chfrkmkg tbraugbpsyibapbysmacagmifc hxprhssmaks+Bms jfilgrauk` ms mk Mkstruitmakfc @hsmgk fk` Phibkacagy wmtb f OYerao Whsthrk Mccmkams Xkmvhrsmty# fk` Ouctmoh`mf Iaooukmiftmak fk`Phibkacagy wmtb f JY erao Xtfb Ufcchy Xkmvhrsmty+ Bms hxphrmhkihmkicu`hs `hvhcapmkg ouctmoh`mf chfrkmkg hkvmrakohkts fk`mkstruitmakfc oa`uchs ear iarparfth# h`uiftmakfc fk` gavhrkohkt

    hktmtmhs fk` iak`uitmkg cfj fk` icfssraao rhshfrib ak tbh iagkmtmvhfk` feehitmvh praihsshs fssaimfth` wmtb chfrkmkg+

    7

    Grf`ufth Ytu`hkt

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    8/120

    ^rhefih

    2

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    9/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Iakthxt

    =

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    10/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Iakthxt

    6>

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    11/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Dustmemiftmak

    Akh kavhc pfrt ae tbh pcfkkmkg fk` `hsmgk ae tbh mkthrfitmak jhtwhhk phapchfk` iaoputhrs# rhcfth` wmtb tbh efiht ae shiurmkg ushr sftmsefitmak# ms tbh ifpfjmcmtyae systhos ta f`fpt ta tbhmr mk`mvm`ufc ushrs sbawmkg hopftby+

    Jhmkg hopftbhtmi mopcmhs tbft tbh iaoputhr ms fjch ta rhiagkmzh ushrs

    feehitmvh stfths fk` uk`hrstfk` tbh mopcmiftmak ae tbash stfths+

    @hthitmak aefeehitmvh stfths ms f sthp earwfr` ta pravm`h ofibmkhs wmtbtbh khihssfry mkthccmghkih ta m`hktmey fk` uk`hrstfk` buofk hoatmaks fk` tbhkfppraprmfthcy mkthrfit wmtb buofks+ Pbhrhearh# mt ms khihssfry hqump iaoputhrs

    wmtb bfr`wfrh fk` saetwfrh ta hkfjch tbho ta phrihmvh ushrs feehitmvh stfths fk`tbhk ush tbms uk`hrstfk`mkg ta irhfth oarh bfroakmi mkthrfitmaks+

    66

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    12/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Oatmvftmak

    Pbms iaursh pravm`hs f `hsirmptmak fk` `hoakstrftmak ae taacs fk` ohtba`acagmhskhihssfrmhs ear futaoftmifccy `hthitmkg feehitmvh stfths+

    Futaoftmi `hthitmak ae feehitmvh stfths rhqumrhs tbh iaoputhr4

    "6- ta shksh mkearoftmak tbft ms iaopchx fk` `mvhrsh< fk`

    ":- ta praihss fk` uk`hrstfk` mkearoftmak mkthgrftmkg shvhrfc saurihs"shkshs- tbft iauc` rfkgh erao jrfmk/wfvhs smgkfcs fk` jmaehh`jfil rhf`mkgs#pfssmkg ear efih/jfsh` fk` ghsturh hoatmak rhiagkmtmak# ta pasturh fk` prhssurhshksmkg omkuths-

    :+ Yhksmkg @hvmihs "5> omkuths-

    :+6+ Jrfmk/iaoputhr mkthrefihs+

    :+:+ Efih/jfsh` hoatmak rhiagkmtmak systhos+

    Jrhfl

    :+9+ Hyh/trfilmkg systhos+

    :+0+ ^bysmacagmifc shksars "Ylmk Iak`uitfkih# ^rhssurh# ^asturh-

    60

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    15/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Yibh`uch | Yhssmak :

    9+ @ftf Emcthrmkg fk` Mkthgrftmak ":> omkuths-

    9+6+ Gftbhrh` ftf

    9+:+ @ftf sykibrakmzftmak

    0+ Fkfcyzmkg `ftf ":> omkuths-

    0+6+ Zhgrhssmak

    0+:+ Icusthrmkg fk` Icfssmemiftmak

    Jrhfl

    8+ Ybfrmkg Hxphrmhkih fk` Graup @msiussmak ":> omkuths-

    5+ Iakicusmaks fk` V%F ":> omkuths-

    68

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    16/120

    Yhssmak M

    65

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    17/120

    M+ Mktra`uitmak

    67

    Yhssmak M

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    18/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Iakihpts

    Ouctmoa`fc @hthitmakaeFeehitmvh Ytfths

    mkstmkitufc rhfitmak ta stmoucftmak

    ehhcmkgs# hoatmaks

    Baw `a yau ehhc1

    pbysmacagmifc pbysmifc

    62

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    19/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Iakihpts

    6=

    Xktmc rhihktcy ouib ae tbh feehitmvh `ftf gftbhrh` jy systhos bfs rhcmh` bhfvmcy akchfrkhrs shce/rhpart ae tbhmr feehitmvh stfth# ajshrvftmak# ar saetwfrh `ftfcags \6_+

    Jut kaw ofky systhos bfvh stfrth` ta mkicu`h `ftf erao tbh pbysmifc

    ofkmehstftmaks ae feehitmvh stfths tbraugb tbh ush aeshksmkg `hvmihs fk` tbh

    fppcmiftmak ae kavhc ofibmkh chfrkmkg fk` `ftf omkmkg fcgarmtbos ta `hfc wmtb tbhvfst foaukts ae `ftf ghkhrfth` jy tbh shksars \:_\9_+

    \6_ Z+Y+D+ Jflhr# O+O+P Za`rmga fk` X+H+ Racaiatzmk# Pbh @ykfomis ae Feehitmvh Prfksmtmaks mk Ymoucftmak rajcho/sacvmkg Hkvmrakohkts# rai+ Feehitmvh Iaoputmkgfk` Mkthccmghkt Mkthrfitmak4 Yhiak` Mkthrkftmakfc Iakehrhkih "FIMM >7-# F+ ^fmvf# Z+ ^rf`f % Z+ W+ ^mifr` "H`s+-# Yprmkghr Uhrcfg# Uac+ Chiturh Kaths mk IaoputhrYimhkih 0792# pp+ 555/577+

    \:_ M+ Frraya# @+ G+ Iaaphr# W+ Jurchsak# E+ ^+ Waace# L+ Ouc`khr# fk` Z+ Ibrmstapbhrsak# Hoatmak Yhksars Ga ta Yibaac# ^rai+ Frtmemimfc Mkthccmghkih mk H`uiftmak4Jumc`mkg Chfrkmkg Yysthos tbft Ifrh4 erao Lkawch`gh Zhprhshktftmak ta Feehitmvh Oa`hccmkg# "FMH@ >=-# U @momtravf# Z+ Omzaguibm# J+ u Jaucfy % F+ Grfsshr "H`s+-#MAY rhss# Ducy :>>=# vac+ Eraktmhrs mk Frtmemimfc Mkthccmghkih fk` Fppcmiftmaks :>># pp+ 67/:0+

    \9_ Z+ W+ mifr`# Feehitmvh Iaoputmkg# OMP rhss# 6==7+

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    20/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Iakihpts

    Ouctmoa`fc @hthitmakae Feehitmvh Ytfths

    ohfsurhohkt

    m`hktmey tbh prhshkih ae +++pbysmacagmifc ohfsurhs

    pbysmifc fpphfrfkih

    shce/rhpart

    :>

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    21/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Oa`hc

    Ouc$oa`fc-hoa$ak-rhiagkm$ak-systho-

    Xshr-

    Jrfmkwfvhs-

    Hyh-oavhohkts-

    Efimfc-hxprhssmaks-

    ^bysmacagmifc-smgkfcs-

    Yhksmkg--

    @hvmihs-

    ^hrihp$ak-

    ohibfkmsos-

    Mkthgrf$ak-

    Fcgarmtbo-

    Zfw-`ftf- Jhcmhes- Ytfth-

    :6

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    22/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Yhksmkg @hvmihs

    ::

    Yhksmkg `hvmihs ajtfmk `ftf erao tbh ushr fjaut bms pbysmacagmifc rhspakshs fk`ja`y rhfitmaks+

    Yhksmkg `hvmihs frh bfr`wfrh `hvmihs tbft iacchit qufktmtftmvh `ftf fs ohfsurhsae pbysmacagmifc smgkfcs ae hoatmakfc ibfkgh+ Wh ifcc tbh ohfsurhs pravm`h` jytbh shksmkg `hvmihs rfw `ftf+

    Aur fpprafib mkicu`hs tbh ush ae4 jrfmk/iaoputhr mkthrefihs# hyh trfilmkgsysthos# jmaehh`jfil shksars# fk` efih/jfsh` hoatmak rhiagkmtmak systhos \0_+

    Pbh ush ae shvhrfc shksmkg `hvmihs hmtbhr ta rhiagkmzh f jraf` rfkgh ae hoatmaksar ta mopravh tbh fiiurfiy ear rhiagkmzmkg akh hoatmak ms rhehrrh` ta fs f

    ouctmoa`fc fpprafib+

    \0_ Dfvmhr Gakzfchz/Yfkibhz# Zajhrt O+ Ibrmstapbhrsak# Ofrmf Hchkf Ibfvhz/Hibhfgfrfy# @fvm` I+ Gmjsak# Zajhrt Ftlmksak# Wmkscaw Jurchsak# Baw ta @aOuctmoa`fc @hthitmak ae Feehitmvh Ytfths1# mifct# pp+580/588# :>66 MHHH 66tb Mkthrkftmakfc Iakehrhkih ak F`vfkih` Chfrkmkg Phibkacagmhs# :>66

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    23/120

    :+ Yhksmkg @hvmihs | Jrfmk Iaoputhr Mkthrefihs

    :9

    Yhssmak M

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    24/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    JIM

    Jrfmk/Iaoputhr Mkthrefihs "JIM-

    Mt ms f pfrtmiucfr typh ae f pbysmacagmifc mkstruohkt tbft ushs jrfmkwfvhs fsmkearoftmak saurihs "hchitrmifc fitmvmty fcakg tbh sifcp pra`uih` jy tbh emrmkg aekhuraks wmtbmk tbh jrfmk-+

    Hoatmv H^AI bhf`sht \8_ `hvmih wmcc jh ush` ta sbaw baw ta iacchit fk` warlwmtb tbms lmk` ae `ftf+

    \8_ Hoatmv / Jrfmk Iaoputhr Mkthrefih Phibkacagy+ Zhtrmhvh` Fprmc :5# :>66# erao bttp4!!www+hoatmv+iao+

    :0

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    25/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    JIM

    Wmrhchss Hoatmv HHG Bhf`sht+

    Pbh `hvmih rhparts `ftf wmtb mkthrvfcs ae 6:8 os "2 Bz-+

    Pbh rfw `ftf autput mkicu`hs 60 vfcuhs "7 ibfkkhcs ak hfib jrfmkbhomspbhrh4 FE9# E7# E9# EI8# P7# ^7# A6# A:# ^2# P2# EI5# E0# E2# fk`

    FE0- fk` twa vfcuhs ae tbh fiihchrftmak ae tbh bhf` wbhk chfkmkg "gyrax fk`gyray-+

    Pbh feehitmv sumth rhparts 8 hoatmaks4 hkgfghohkt# jarh`ao#hximthohkt# erustrftmak# fk` oh`mtftmak+

    Fk` tbh hxprhssmv sumth rhparts efimfc ghsturhs4 jcmkl# wmkl "chet fk`rmgbt-# caal "chet fk` rmgbt-# rfmsh jraw# eurraw jraw# somch# ichkib# somrl "chetfk` rmgbt-# fk` cfugb+

    :8

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    26/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy :5

    Hchitra`hs frh smtufth` fk` cfjhch` fiiar`mkg wmtb tbh IOY!@ZC iakemgurftmak \5_\7_

    JIM

    \5_ Ybfrjraugb E# Ibftrmfk G/H# Chsshr Z^# C`hrs B# Kuwhr O# mitak PW+ Fohrmifk Hchitrahkihpbfcagrfpbmi Yaimhty Gum`hcmkhs ear Ytfk`fr` Hchitra`h ^asmtmakKaohkicfturh+ D+ Icmk+ Khurapbysmac 24 :>>/:+

    \7_ Hchitrahkihpbfcagrfpby+ Zhtrmhvh` Kavhojhr 60tb# :>6># erao Hchitrmi fk` Ofgkhtmi Ohfsurhohkt ae tbh Hchitrmi Fitmvmty ae Khurfc Pmssuh4 www+jho+em!jaal!69!69+bto

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    27/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    HoatmvYysthos

    $:==

    hoatmaks HHG `ftf efimfc ghsturhs

    :7

    JIM

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    28/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    @hoaWmrhchss Hoatmv HHG Bhf`sht

    :2

    JIM

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    29/120

    :=

    Feehitmv Yumth

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    30/120

    9>

    Hxprhssmv Yumth

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    31/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 96

    Emhc` @hsirmptmak Ufcuhs

    PmohstfopMt ms tbh tmohstfop "`fth fk` tmoh- ae tbh iaoputhr rukkmkgtbh systho+ Mt iauc` jh ush` ta sykibrakmzh tbh `ftf wmtb

    atbhr shksars+

    Earoft *yyoo``bboossYYY*"y / yhfr# o / oaktb# ` / `fy# b / baur# o / omkuths# s /

    shiak`s# Y / omccmshiak`s-+

    XshrM@ M`hktmemhs tbh ushr+ Fk mkthghr vfcuh+

    Wmrhchss Ymgkfc Ytftus Ybaws tbh strhkgtb ae tbh smgkfc+ Pbh vfcuh ms erao > ta 0# jhmkg 0 tbh jhst akh+

    Jcmkl# Wmkl Chet fk`Zmgbt# Caal Chet fk`Zmgbt# Zfmsh Jraw#

    Eurraw# Yomch# Ichkib#Yomrl Chet fk` Zm bt

    ^frt ae tbh hxprhssmvh sumth+Ufcuhs jhtwhhk > fk` 6# jhmkg 6 tbh vfcuh tbftrhprhshkts tbh bmgbhst pawhr!prajfjmcmty ear tbms

    hoatmak+

    Ybart Phro fk` CakgPhro Hximthohkt#

    Hkgfghohkt ! Jarh`ao#

    Oh`mtftmak# Erustrftmak

    ^frt ae tbh feehitmvh sumth+Ufcuhs jhtwhhk > fk` 6# jhmkg 6 tbh vfcuh tbftrhprhshkts tbh bmgbhst pawhr!prajfjmcmty ear tbms

    hoatmak+

    FE9# E7# E9# EI8# P7#^7# A6# A:# ^2# P2#EI5# E0# E2# FE0+

    Zfw `ftf iaomkg erao hfib ae tbh 60 ibfkkhcs+ Pbh kfoh aetbhsh emhc`s wbhrh hemkh` fiiar`mkg wmtb tbh IOY!@ZC

    iakemgurftmak \RRR_\RRRR_+Ufcuhs erao 0>>> fk` bmgbhr+

    GyraR fk` Gyra]Mkearoftmak fjaut baw tbh bhf` oavhs!fiihchrfths fiiar`mkg

    wmtb R fk` ] fxms fiiar`mkgcy+

    JIM

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    32/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 9:

    Pmohstfop FE9 E7 E9 EI8 P7 ^7 A6 A: ^2 P2 EI5 E0 E2 FE0 FiiR Fii]

    6>666566:800=>6 080 0296+2 0:07+: 05=>+9 0:2:+5 09=8+0 08=6+2 085=+: 095> 087>+2 0:=7+0 0966+9 0:2:+5 0957+: 655> :>>9

    6>666566:800=>6 089 02>:+6 0:09+6 0579+= 0:7:+9 09=9+9 08=:+2 087>+9 0980+= 087>+9 0:=:+9 09>=+7 0:72 097>+2 6582 :>>:

    6>666566:808>6> 089 07== 0:90+= 055=+7 09>6 09=5+= 08=:+9 087>+2 0986+9 0856 0:26+8 09>6+8 0:76+9 0959+5 658= :>>9

    6>666566:808>6> 080 029=+8 0:06 05=6+9 0999+= 09=2 08=5+0 0857+: 0988+= 0885+0 0:25+: 09>5+: 0:72 095=+7 658= :>>9

    6>666566:808>6> 082 0258+5 0:86+2 076>+9 090> 00>6+8 05>9+5 087:+2 095> 0882+8 0:== 09:0+5 0:=5+0 09=8+= 6587 :>>0

    6>666566:808>6> 08= 025> 0:8:+2 07>8+5 098>+9 006:+9 05>9+5 0877+0 0987+0 0888+= 0:=8+0 09:=+: 0:=5+0 0060+0 6585 :>>8

    6>666566:808>6> 082 0207+7 0:05+7 05=>+9 095> 00>=+: 08=7+0 085=+7 0986+2 080=+7 0:7= 0965+= 0:7:+2 09==+8 6585 :>>5

    6>666566:808>6> 085 020:+6 0:92+8 0520+6 09::+6 092=+7 08=:+2 0855+7 0986+2 080=+7 0:70+0 096>+9 0:5:+6 097>+2 6588 :>>8

    6>666566:808>6> 085 0200+5 0:96+2 0527+7 0:57+7 0927+7 08=0+0 082> 0956 0885+0 0:7= 096>+2 0:70+0 097>+2 6589 :>>5

    6>666566:808>6> 085 0207+: 0:99+9 0522+7 0:28+6 00>=+: 05>:+6 082=+: 0952+: 085> 0:2>+8 096>+2 0:26+8 09=>+9 6588 :>>0

    6>666566:808>6> 087 0205+7 0:90+= 0529+6 09:9+6 0068+= 05>0+6 0828+5 0955+7 0887+0 0:72 096>+9 0:79+9 0920+6 658: :>>8

    6>666566:808>6> 085 020:+5 0:99+= 0572+8 096>+2 00>:+5 08== 0829+6 0950+6 0889+= 0:77+0 096>+9 0:76+9 097:+9 6580 :>>8

    6>666566:808>6> 088 029:+2 0:90+= 0575+= 09>6 092=+7 08=8+0 08=>+9 0952+7 0885+= 0:2> 096>+9 0:75+= 092> 6589 :>>0

    6>666566:808>6> 088 0296+2 0:99+9 057=+8 0960+0 09=>+9 08=2 08== 0970+= 085:+5 0:2>+8 0966+9 0:2> 0925+: 6589 :>>0

    6>666566:808>6> 087 020:+5 0:9:+2 0520+6 09>9+5 00>8+5 05>=+7 05>> 0972+8 0857+: 0:7= 0969+9 0:2> 092:+6 6589 :>>:

    6>666566:808>6> 087 0205+7 0:98+0 0529+6 0:=9+9 0065+0 056=+8 05>0+6 092:+5 087>+2 0:2>+8 096>+2 0:2:+6 092:+6 658: :>>:

    6>666566:808>6> 085 020>+8 0::7+: 0579+= 09>> 00>8+6 0566+9 05>6 0975+0 0856+8 0:2> 09>9+5 0:7=+8 0970+= 658: :>>>

    Zfw @ftf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    33/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 99

    Pmohstfop M@ Ymgkfc Jcmkl Wmkl C Wmkl Z Caal C Caal Z Hyhjraw Eurraw Yomch Ichkib Yomrl C Yomrl Z Cfugb

    6>6665>=6608>58 > : > > > 6 > > > > > > >+=22859 >

    6>6665>=66086=> > : > > > 6 > > > >+08058 > > > >

    6>6665>=6608968 > : > > > 6 > > > >+057>6 > > > >

    6>6665>=660800> > : > > > 6 > > > >+0>6>6 > > > >

    6>6665>=6608858 > : > > > 6 > > > >+:0257 > > > >

    6>6665>=66085=> > : > > > 6 > > > >+679>: > > > >

    6>6665>=6608268 > : > > > 6 > > > >+65:7= > > > >

    6>6665>=6608=0> > : > > > 6 > > > >+6850= > > > >

    6>6665>=6605>58 > : > > > 6 > > > > > >+775=:8 > >

    6>6665>=66056=> > : > > > 6 > > > > > > >+5>257= >

    6>6665>=6605968 > : > > > 6 > > > > > > >+90:950 >

    6>6665>=660500> > : > > > 6 > > > > > > >+60=5=8 >

    6>6665>=6605858 > : > > > 6 > > > > > > >+>2500 >

    6>6665>=66055=> > : > > > 6 > > > > > > >+>79902 >

    6>6665>=6605268 > : > > > 6 > > > > > > >+662=58 >

    6>6665>=6605=06 > : > > > 6 > > > > > > >+:8=676 >

    Hxprhssmv @ftf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    34/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 90

    Pmohstfop Ybart Phro Hximthohkt Cakg Phro Hximthohkt Hkgfghohkt Oh`mtftmak Erustrftmak

    6>6665>=6608>58 >+0078=8 >+80276 >+290075 >+999200 >+8956=7

    6>6665>=66086=> >+0078=8 >+80276 >+290075 >+999200 >+8956=7

    6>6665>=6608968 >+0078=8 >+80276 >+290075 >+999200 >+8956=7

    6>6665>=660800> >+027250 >+805277 >+290605 >+99=802 >+80286

    6>6665>=6608858 >+027250 >+805277 >+290605 >+99=802 >+80286

    6>6665>=66085=> >+027250 >+805277 >+290605 >+99=802 >+80286

    6>6665>=6608268 >+027250 >+805277 >+290605 >+99=802 >+80286

    6>6665>=6608=0> >+8:6559 >+8085>= >+29=9:6 >+9029:6 >+882::2

    6>6665>=6605>58 >+8:6559 >+8085>= >+29=9:6 >+9029:6 >+882::2

    6>6665>=66056=> >+8:6559 >+8085>= >+29=9:6 >+9029:6 >+882::2

    6>6665>=6605968 >+8:6559 >+8085>= >+29=9:6 >+9029:6 >+882::2

    6>6665>=660500> >+8>=:=7 >+800696 >+200>6 >+982767 >+805776

    6>6665>=6605858 >+8>=:=7 >+800696 >+200>6 >+982767 >+805776

    6>6665>=66055=> >+8>=:=7 >+800696 >+200>6 >+982767 >+805776

    6>6665>=6605268 >+8>=:=7 >+800696 >+200>6 >+982767 >+805776

    6>6665>=6605=06 >+086228 >+8065=8 >+202>27 >+952>76 >+899=6=

    Feehitmv @ftf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    35/120

    :+ Yhksmkg @hvmihs | Efih/Jfsh` Hoatmak Zhiagkmtmak

    98

    Yhssmak M

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    36/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Efih/Jfsh` Zhiagkmtmak

    Efih/jfsh` hoatmak rhiagkmtmak systhos

    Pbhsh systhos mkehr feehitmvh stfths jy ifpturmkg mofghs ae tbh ushrs efimfchxprhssmaks fk` bhf` oavhohkts+

    Wh frh gamkg ta sbaw tbh ifpfjmcmtmhs ae efih/jfsh` hoatmak rhiagkmtmak systhosusmkg f smopch 9> eps XYJ whjifo fk` saetwfrh erao OMP Oh`mf Cfj \2_+

    \2_ Z+ H+ Lfcmaujy fk` ^+ Zajmksak# Zhfc/Pmoh Mkehrhkih ae Iaopchx Ohktfc Ytfths erao Efimfc Hxprhssmaks fk` Bhf` Ghsturhs# ^rai+ Iakehrhkih akIaoputhr Umsmak fk` ftthrk Zhiagkmtmak Warlsbap "IU^ZW >0-# MHHH Iaoputhr Yaimhty# Dukh :>>0# Uacuoh 6># p+ 680+

    95

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    37/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Efih/Jfsh` Zhiagkmtmak

    Omk`Zhf`hr F^M tbft hkfjchs tbh rhfc tmoh fkfcysms# tfggmkg fk` mkehrhkih aeiagkmtmvh feehitmvh ohktfc stfths erao efimfc vm`ha mk rhfc/tmoh+

    Pbms erfohwarl iaojmkhs vmsmak/jfsh` praihssmkg ae tbh efih wmtb prh`mitmaksae ohktfc stfth oa`hcs ta mkthrprht tbh ohfkmkg uk`hrcymkg bhf` fk` efimfcsmgkfcs avhrtmoh+

    Mt pravm`hs rhsucts ft mkthrvfcs ae 6>> os fppraxmofthcy "6> Bz-+

    Wmtb tbms systho mt ms passmjch ta mkehr hoatmaks suib fs4 fgrhhmkg#iakihktrftmkg# msfgrhhmkg# mkthrhsth`# tbmklmkg# fk` uksurh+

    "Hlofk fk` Ermhshk 6=72- Efimfc Fitmak Ia`mkg Yystho# 05 fitmaks "pcusbhf` oavhohkts-+

    97

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    38/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    @hoaOmk`Zhf`hr Yaetwfrh erao OMP Oh`mf Cfj

    92

    Efih/Jfsh` Zhiagkmtmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    39/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 9=

    &

    6=&Cmp&Iarkhr&@hprhssar&

    &

    :5&Dfw&@rap&

    &

    :7&Oautb&Ytrhtib&

    Hkhra&::#&:>6>& Dfvmhr&Gakzchz&&Ykibhz&|&Ofrf&H+&Ibvhz&Hibhfgfrfy& ::&

    Efih/Jfsh` Zhiagkmtmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    40/120

    0>

    Omk` Zhf`hr

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    41/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 06

    Emhc` @hsirmptmak Ufcuhs

    Pmohstfop

    Mt ms tbh tmohstfop "`fth fk` tmoh- ae tbh iaoputhr rukkmkg

    tbh systho+ Mt iauc` jh ush` ta sykibrakmzh tbh `ftf wmtbatbhr shksars+

    Earoft *yyoo``bboossYYY*

    "y / yhfr# o / oaktb# ` / `fy# b / baur# o / omkuths# s /shiak`s# Y / omccmshiak`s-+

    Fgrhhohkt#Iakihktrftmkg#@msfgrhhohkt#

    Mkthrhsth`# Pbmklmkg#Xksurh

    Pbms vfcuh sbaws tbh prajfjmcmty ae tbms hoatmak jhmkg prhshktak tbh ushr ft f pfrtmiucfr tmoh "erfoh-+

    Pbms vfcuh ms jhtwhhk > ta 6+Me tbh vfcuh ms /6 mt ohfks mt wfs kat passmjch ta `hemkh

    fk hoatmak+Pbms bfpphk tbh ushr's efih ms aut ae tbh ifohrf eaius+

    Omk` Zhf`hr

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    42/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 0:

    Pmohstfop Fgrhhohkt Iakihktrftmkg @msfgrhhohkt Mkthrhsth` Pbmklmkg Xksurh

    6>666566:292865 >+>>6295>9: >+====67 6+7=H/>0 >+650280>5 >+87660:88 >+>08=8>5:

    6>666566:292872 >+>>6007580 >+====865 6+:=H/>0 >+6596>529 >+8=82=:6 >+>0:7>508:

    6>666566:29257: 8+=7H/>0 > 6+8>H/>0 >+00==5:=0 >+088:7569 >+>>72=5=7

    6>666566:292755 :+05H/>0 > 6+78H/>0 >+77008525 >+9:60078: >+>>6062:67

    6>666566:29225> 6+>6H/>0 > :+>0H/>0 >+=9866=68 >+:6657692 :+89H/>0

    6>666566:292=89 0+62H/>8 > :+92H/>0 >+=2979= >+69:>2577 0+8:H/>8

    6>666566:29=>65 6+7:H/>8 > :+72H/>0 >+==5>770 >+>7=06>92 2+>7H/>5

    6>666566:29=66> 7+6>H/>5 > 9+:0H/>0 >+===>5:55 >+>05569687 6+00H/>5

    6>666566:29=685 :+=:H/>5 > 9+77H/>0 >+===77580 >+>:5=50797 :+87H/>7

    6>666566:29=:8> 6+:6H/>5 > 0+0>H/>0 >+====057 >+>680506=5 0+82H/>2

    6>666566:29=9=6 0+=7H/>7 > 8+6:H/>0 >+====279 >+>>22:062= 2+62H/>=

    6>666566:29=092 :+>8H/>7 > 8+=7H/>0 >+=====7 >+>>8>:>7:8 6+05H/>=

    6>666566:29=807 2+09H/>2 > 5+=5H/>0 >+======9 >+>>:286=9= :+5>H/6>

    6>666566:29=872 9+07H/>2 > 2+66H/>0 >+======= >+>>6562079 0+50H/66

    6>666566:29=522 6+09H/>2 > =+08H/>0 >+=======0 =+62H/>0 2+:=H/6:

    6>666566:29=726 8+=>H/>= > >+>>66>60>0 6 8+:6H/>0 6+02H/6:

    6>666566:29=2:2 :+09H/>= > >+>>6:298:6 6 :+=8H/>0 :+50H/69

    Omk` Zhf`hr

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    43/120

    :+ Yhksmkg @hvmihs | Hyh Prfilmkg Yysthos

    09

    Yhssmak M

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    44/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Hyh/trfilmkg systhos

    Pbhsh frh mkstruohkts tbft ohfsurh hyhs pasmtmak fk` hyhs oavhohkt mk ar`hr ta`hthit zakhs mk wbmib tbh ushr bfs pfrtmiucfr mkthrhst mk f sphimemi tmoh fk`oaohkt+

    @ftfshts erao Pajmm Hyh/trfilmkg systho \=_ `ftf wmcc jh sbawk+

    \=_ Pajmm Phibkacagy / Hyh Prfilmkg fk` Hyh Iaktrac+ Zhtrmhvh` Fprmc :5# :>66# erao bttp4!!www+tajmm+iao+

    Hyh Prfilmkg Yystho

    00

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    45/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Hyh Prfilmkg Yystho

    PajmmHyh Prfilhr+

    Pbh `hvmih rhparts `ftf wmtb mkthrvfcs ae 6>> os "6>Bz-+

    Pbh autput pravm`hs `ftf iakihrkmkg ftthktmak `mrhitmak "gfzh/x# gfzh/y-#`urftmak ae emxftmak# fk` pupmc `mcftmak+

    08

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    46/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    @hoaPajmm Hyh Prfilhr

    05

    Hyh Prfilmkg Yystho

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    47/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 07

    Emhc` @hsirmptmak Ufcuhs

    PmohstfopMt ms tbh tmohstfop "`fth fk` tmoh- ae tbh iaoputhr rukkmkgtbh systho+ Mt iauc` jh ush` ta sykibrakmzh tbh `ftf wmtb

    atbhr shksars+

    Earoft *yyoo``bboossYYY*"y / yhfr# o / oaktb# ` / `fy# b / baur# o / omkuths# s /

    shiak`s# Y / omccmshiak`s-+

    Gfzh^amkt RPbh barmzaktfc sirhhk pasmtmak ear hmtbhr hyh ar tbh fvhrfgh ear

    jatb hyhs+ Pbms vfcuh ms fcsa ush` ear tbh emxftmak `hemkmtmak+> ms tbh chet h`gh# tbh ofxmouo vfcuh ae tbh barmzaktfc

    sirhhk rhsacutmak ms tbh rmgbt h`gh+

    Gfzh^amkt ]Pbh vhrtmifc sirhhk pasmtmak ear hmtbhr hyh ar tbh fvhrfgh earjatb hyhs+ Pbms vfcuh ms fcsa ush` ear tbh emxftmak `hemkmtmak+

    > ms tbh jattao h`gh# tbh ofxmouo vfcuh ae tbh vhrtmifcsirhhk rhsacutmak ms tbh rmgbt h`gh+

    ^upmc Chet ^upmc smzh "chet hyh- mk oo+ Ufrmfths

    Ufcm`mty Chet Ufcm`mty ae tbh gfzh `ftf+> ta 0+ > me tbh hyh ms eauk` fk` tbh trfilmkg qufcmty

    gaa`+ Me tbh hyh ifkkat jh eauk` jy tbh hyh trfilhr# tbhvfcm`mty ia`h wmcc jh 0+

    ^upmc Zmgbt ^upmc smzh "chet hyh- mk oo+ Ufrmfths

    Ufcm`mty Zmgbt Ufcm`mty ae tbh gfzh `ftf+> ta 0+ > me tbh hyh ms eauk` fk` tbh trfilmkg qufcmty

    gaa`+ Me tbh hyh ifkkat jh eauk` jy tbh hyh trfilhr# tbh

    vfcm`mty ia`h wmcc jh 0+

    Emxftmak@urftmak Emxftmak urftmak+ Pbh tmoh mk omccmshiak`s tbft f emxftmak cfsts+ Ufrmfths+

    Hvhkt Hvhkts# futaoftmi fk` caggh`# wmcc sbaw up uk`hr Hvhkt+ Ufrmfths+

    FAMFrhfs Ae Mkthrhsts me emxftmaks ak ouctmpch FAMs frh ta jh wrmtthk

    ak tbh sfoh raw+Ufrmfths+

    Hyh Prfilmkg Yystho

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    48/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 02

    Pmohstfop G^R G^] ^upmc Chet Ufcm`mty C ^upmc Zmgbt Ufcm`mty Z Emxftmak Hvhkt FAM

    6>66:065:0>882: 595 6== :+78=969 > :+220>5 > 02 Iakthkt

    6>66:065:0>88== 506 :>7 :+5202=9 > :+288267 > 02 Iakthkt

    6>66:065:0>8568 58= :66 :+5:0082 > :+=>9256 > 02 Iakthkt

    6>66:065:0>859: 500 :>6 :+595625 > :+=6569: > 02 Iakthkt

    6>66:065:0>850= 500 :69 :+5=>528 > :+296>69 > 02 Iakthkt

    6>66:065:0>8555 5:2 6=0 :+586720 > :+25=760 > 02 Iakthkt

    6>66:065:0>852: 560 677 :+2:=:26 > :+2==2:2 > 02 Iakthkt

    6>66:065:0>85== 7>6 :0= :+72>900 > :+=>7558 > 0= Iakthkt

    6>66:065:0>8765 =>5 906 :+289756 > :+=659=2 > 0= Iakthkt

    6>66:065:0>879: =07 9=2 :+2:=0:7 > :+22==00 > 0= Iakthkt

    6>66:065:0>870= =06 0>> :+2:55>: > :+22667= > 0= Iakthkt

    6>66:065:0>8755 =92 0>9 :+725== > :+27=02 > 0= Lhy^rhss Iakthkt

    6>66:065:0>872: =97 066 :+2>9927 > :+2:62>9 > 0= Iakthkt

    6>66:065:0>87== =90 9=7 :+26=655 > :+276807 > 0= Iakthkt

    6>66:065:0>8265 =06 0>7 :+266527 > :+267=:7 > 0= Iakthkt

    6>66:065:0>829: =05 0>8 :+28706= > :+2870:7 > 0= Iakthkt

    6>66:065:0>820= > > /6 0 /6 0 0= Iakthkt

    Hyh Prfilmkg Yystho

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    49/120

    :+ Yhksmkg @hvmihs | Gfcvfkmi Ylmk Iak`uitfkih Yhksar

    0=

    Yhssmak M

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    50/120

    8>

    Gfcvfkmi Ylmk Iak`uitfkih

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    51/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Gfcvfkmi Ylmk Iak`uitfkih

    ^ravm`h mkearoftmak ak tbh fitmvmty ae pbysmacagmifc eukitmaks ae fk mk`mvm`ufc+

    Frausfc `hthitmak+ Ohfsurhs tbh hchitrmifc iak`uitfkih ae tbh slmk# wbmibvfrmhs wmtb mts oamsturh chvhc tbft `hphk`s ak tbh swhft gcfk`s# wbmib frhiaktracch` jy tbh syopftbhtmi# fk` pfrfsyopftbhtmi khrvaus systhos+ \6>_

    Bfr`wfrh `hsmgkh` jy OMP Oh`mf Cfj+

    Mt ms f Wmrhchss Jcuhtaatb `hvmih tbft rhparts `ftf mk mkthrvfcs ae 8>> osfppraxmofthcy ":Bz-

    \6>_ O+ Ytrfuss# I+ Zhykac`s# Y+ Bugbhs# L+ ^frl# G+ Oi@frjy# fk` Z+W+ ^mifr`# Pbh Bfk`Wfvh Jcuhtaatb Ylmk Iak`uitfkih Yhksar# ^rai+ Emrst MkthrkftmakfcIakehrhkih ak Feehitmvh Iaoputmkg fk` Mkthccmghkt Mkthrfitmak "FIMM >8-# Yprmkghr/Uhrcfkg# Ait+ :>>8# pp+ 5==/7>5# am46>+6>>7!66879802Q=>+

    86

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    52/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    @hoaYlmk hchitrmifc iak`uitfkih Yhksar

    8:

    Gfcvfkmi Ylmk Iak`uitfkih

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    53/120

    89

    Gfcvfkmi Ylmk Iak`uitfkih

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    54/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 80

    Emhc` @hsirmptmak Ufcuhs

    Pmohstfop

    Mt ms tbh tmohstfop "`fth fk` tmoh- ae tbh iaoputhr rukkmkg

    tbh systho+ Mt iauc` jh ush` ta sykibrakmzh tbh `ftf wmtbatbhr shksars+

    Earoft *yyoo``bboossYYY*

    "y / yhfr# o / oaktb# ` / `fy# b / baur# o / omkuths# s /shiak`s# Y / omccmshiak`s-+

    Jftthry Uactfgh Chvhc ae tbh jftthry vactfgh+ > / 9 Uacts

    Iak`uitfkih Chvhc ae frausfc+ > / 9 Uacts

    Gfcvfkmi Ylmk Iak`uitfkih

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    55/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 88

    Pmohstfop Uactfgh Iak`uitfkih

    6>66656>699::5: :+02:98:=06 6+>9>5=5675

    6>66656>699:75: :+02:98:=06 6+>:90>0658

    6>66656>6999:5: :+02:98:=06 6+>6=269:70

    6>66656>699975: :+02:98:=06 6+>065872>:

    6>66656>6990:07 :+02:98:=06 >+==2:2>:79

    6>66656>6990707 :+02:98:=06 >+==662660:

    6>66656>6998:07 :+02:98:=06 >+=2>8=:::=

    6>66656>6998707 :+02:98:=06 >+==2:2>:79

    6>66656>6995:07 :+02:98:=06 6+>6:825:=0

    6>66656>699575: :+02:98:=06 6+>6:825:=0

    6>66656>6997:96 :+02:98:=06 6+>6:825:=0

    6>66656>6997707 :+02:98:=06 6+>>=>>2:86

    6>66656>6992:07 :+02:98:=06 >+==2:2>:79

    6>66656>6992707 :+02:98:=06 >+==662660:

    6>66656>699=:07 :+02:98:=06 >+=275:28:6

    6>66656>699=796 :+02:98:=06 >+=275:28:6

    6>66656>690>:96 :+02:98:=06 >+=2>8=:::=

    Gfcvfkmi Ylmk Iak`uitfkih

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    56/120

    :+ Yhksmkg @hvmihs | ^rhssurh Yhksar

    85

    Yhssmak M

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    57/120

    87

    ^rhssurh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    58/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    ^rhssurh Yhksar

    ^ravm`h mkearoftmak ak tbh fitmvmty ae pbysmacagmifc eukitmaks ae fk mk`mvm`ufc+

    ^rhssurh shksars fjch ta `hthit tbh mkirhfsmkg foaukt aeprhssurh"iarrhcfth` wmtb chvhcs aeerustrftmak- tbft tbh ushr puts ak f oaush# arfky atbhr iaktracchr "suib fs f gfoh iaktracchr- \66_+

    Bfr`wfrh `hsmgkh` jy OMP Oh`mf Cfj+

    Mt ms f shrmfc `hvmih tbft rhparts `ftf mk mkthrvfcs ae 68> os fppraxmofthcy"5Bz-+

    \66_ ]+ Vm# fk` Z+ W+ mifr`# *Iakthxt/Yhksmtmvh Jfyhsmfk Icfssmemhrs fk` Fppcmiftmak ta Oaush rhssurh ftthrk Icfssmemiftmak#* rai+ Mkthrkftmakfc Iakehrhkih ak^ftthrk Zhiagkmtmak "MI^Z >:-# Fug+ :>>:# vac 9# pp+ 9>002# am46>+66>=!MI^Z+:>>:+6>07=79+

    82

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    59/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    @hoaOaush ^rhssurh Yhksar

    8=

    ^rhssurh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    60/120

    5>

    ^rhssurh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    61/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 56

    Emhc` @hsirmptmak Ufcuhs

    PmohstfopMt ms tbh tmohstfop "`fth fk` tmoh- ae tbh iaoputhr rukkmkgtbh systho+ Mt iauc` jh ush` ta sykibrakmzh tbh `ftf wmtb

    atbhr shksars+

    Earoft *yyoo``bboossYYY*"y / yhfr# o / oaktb# ` / `fy# b / baur# o / omkuths# s /

    shiak`s# Y / omcc mshiak`s-+

    Zmgbt Zhfr Yhksar pasmtmakh` mk tbh rmgbt rhfr ae tbh oaush+ > / 6>:0+ Jhmkg > tbh bmgbhst prhssurh+

    Zmgbt Erakt Yhksar pasmtmakh` mk tbh rmgbt erakt ae tbh oaush+ > / 6>:0+ Jhmkg > tbh bmgbhst prhssurh+

    Chet Zhfr Yhksar pasmtmakh` mk tbh chet rhfr ae tbh oaush+ > / 6>:0+ Jhmkg > tbh bmgbhst prhssurh+

    Chet Erakt Yhksar pasmtmakh` mk tbh chet erakt ae tbh oaush+ > / 6>:0+ Jhmkg > tbh bmgbhst prhssurh+

    Om``ch Zhfr Yhksar pasmtmakh` mk tbh om``ch rhfr ae tbh oaush+ > / 6>:0+ Jhmkg > tbh bmgbhst prhssurh+

    Om``ch Erakt Yhksar pasmtmakh` mk tbh om``ch erakt ae tbh oaush+ > / 6>:0+ Jhmkg > tbh bmgbhst prhssurh+

    ^rhssurh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    62/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 5:

    Pmohstfop Zmgbt Zhfr Zmgbt Erakt Chet Zhfr Chet Erakt Om``ch Zhfr Om``ch Erakt66>7:>6699>596: 6>:9 6>:9 6>:9 6>:9 6>:9 6>:9

    66>7:>6699>5052 6>:9 6>:9 6>:9 6>:9 6>:9 6>:9

    66>7:>6699>55:8 6>:9 ==2 6>:9 6>>: 6>:9 6>:9

    66>7:>6699>5726 6>:9 6>>= 6>:9 =77 6>:9 6>:9

    66>7:>6699>5=97 6>:9 7=0 6>:9 =2: 6>:9 6>:9

    66>7:>6699>76>= 6>:9 0=: 6>:: 2=6 6>:9 6>:9

    66>7:>6699>7:58 6>:9 9=8 6>:6 =65 6>6= 6>:9

    66>7:>6699>70:6 6>:9 92: 6>:6 =0= 6>:9 6>:9

    66>7:>6699>7872 6>:9 950 6>:: =29 6>:9 6>:9

    66>7:>6699>7790 6>:9 66: 6>:6 6>>0 6>:9 6>:9

    66>7:>6699>72=> 6>:9 :>0 6>:6 =05 6>:9 6>:9

    66>7:>6699>2>05 6>:9 058 6>:: =76 6>:9 6>:9

    66>7:>6699>2:>9 6>:9 0>0 6>:: 6>:9 6>:9 6>:9

    66>7:>6699>298= 6>:9 655 6>:6 6>:9 6>:9 6>:9

    66>7:>6699>2868 6>:9 608 6>:6 6>:9 6>:9 6>:9

    66>7:>6699>2527 6>:9 680 6>:6 6>:9 6>:9 6>:9

    66>7:>6699>2209 6>:9 6:5 6>:6 6>:9 6>:9 6>:9

    ^rhssurh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    63/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 59

    ^rhssurh Yhksar

    Pbh rfw `ftf erao tbh oaush ms praihssh` ta ajtfmk ohfkmkgeucmkearoftmak \6:_+

    Pbh `ftf ajtfmkh` ms f karofcmzh`vfcuh erao tbh smx `meehrhkt shksar aktbh oaush+

    \6:_ Iaaphr# @+# Frraya# M+# Waace# J+# Ouc`khr# L+# Jurchsak# W+# fk` Ibrmstapbhrsak# Z+ ":>>=-+ Yhksars oa`hc stu`hkt shce iakihpt mk tbh icfssraao#Xshr Oa`hcmkg# F`fptftmak# fk` hrsakfcmzftmak# 9>//06

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    64/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 50

    Pmohstfop Zmgbt Zhfr Zmgbt Erakt Chet Zhfr Chet Erakt Om``ch Zhfr Om``ch Erakt Ufcuh

    66>7:>6699>596: 6>:9 6>:9 6>:9 6>:9 6>:9 6>:9 5

    66>7:>6699>5052 6>:9 6>:9 6>:9 6>:9 6>:9 6>:9 5

    66>7:>6699>55:8 6>:9 ==2 6>:9 6>>: 6>:9 6>:9 8+=88>90:69

    66>7:>6699>5726 6>:9 6>>= 6>:9 =77 6>:9 6>:9 8+=06902=70

    66>7:>6699>5=97 6>:9 7=0 6>:9 =2: 6>:9 6>:9 8+795>7>926

    66>7:>6699>76>= 6>:9 0=: 6>:: 2=6 6>:9 6>:9 8+98>=:2506

    66>7:>6699>7:58 6>:9 9=8 6>:6 =65 6>6= 6>:9 8+:7858=2:0

    66>7:>6699>70:6 6>:9 92: 6>:6 =0= 6>:9 6>:9 8+:==6:>:98

    66>7:>6699>7872 6>:9 950 6>:: =29 6>:9 6>:9 8+968792>:8

    66>7:>6699>7790 6>:9 66: 6>:6 6>>0 6>:9 6>:9 8+>22=80>87

    66>7:>6699>72=> 6>:9 :>0 6>:6 =05 6>:9 6>:9 8+6::62=592

    66>7:>6699>2>05 6>:9 058 6>:: =76 6>:9 6>:9 8+0>:797>02

    66>7:>6699>2:>9 6>:9 0>0 6>:: 6>:9 6>:9 6>:9 8+9=9=9=9=0

    66>7:>6699>298= 6>:9 655 6>:6 6>:9 6>:9 6>:9 8+65>96:2>8

    66>7:>6699>2868 6>:9 608 6>:6 6>:9 6>:9 6>:9 8+69=720=05

    66>7:>6699>2527 6>:9 680 6>:6 6>:9 6>:9 6>:9 8+60282:5

    66>7:>6699>2209 6>:9 6:5 6>:6 6>:9 6>:9 6>:9 8+6:6:6:6:6

    ^rhssurh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    65/120

    :+ Yhksmkg @hvmihs | ^asturh Yhksar

    58

    Yhssmak M

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    66/120

    55

    ^asturh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    67/120

    ^ t Y

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    68/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    @hoaIbfmr ^asturh Yhksar

    52

    ^asturh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    69/120

    5=

    ^asturh Yhksar

    ^asturh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    70/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 7>

    Emhc` @hsirmptmak Ufcuhs

    PmohstfopMt ms tbh tmohstfop "`fth fk` tmoh- ae tbh iaoputhr rukkmkgtbh systho+ Mt iauc` jh ush` ta sykibrakmzh tbh `ftf wmtb

    atbhr shksars+

    Earoft *yyoo``bboossYYY*"y / yhfr# o / oaktb# ` / `fy# b / baur# o / omkuths# s /

    shiak`s# Y / omccmshiak`s-+

    FiiR Ufcuh ae tbh R fxms ae tbh fiihchraohthr+ Ufrmfths

    Fii] Ufcuh ae tbh ] fxms ae tbh fiihchraohthr+ Ufrmfths

    Zmgbt Yhft Yhksar pasmtmakh` mk tbh rmgbt sm`h ae tbh shft iusbmak+ > / 6>:0+ Jhmkg 6>:0 tbh bmgbhst prhssurh+

    Om``ch Yhft Yhksar pasmtmakh` mk tbh om``ch sm`h ae tbh shft iusbmak+ > / 6>:0+ Jhmkg 6>:0 tbh bmgbhst prhssurh+

    Chet Yhft Yhksar pasmtmakh` mk tbh chet sm`h ae tbh shft iusbmak+ > / 6>:0+ Jhmkg 6>:0 tbh bmgbhst prhssurh+

    Zmgbt Jfil Yhksar pasmtmakh` mk tbh rmgbt sm`h ae tbh jfil pf`+ > / 6>:0+ Jhmkg 6>:0 tbh bmgbhst prhssurh+

    Om``ch Jfil Yhksar pasmtmakh` mk tbh om``ch sm`h ae tbh jfil pf`+ > / 6>:0+ Jhmkg 6>:0 tbh bmgbhst prhssurh+

    Chet Jfil Yhksar pasmtmakh` mk tbh chet sm`h ae tbh jfil pf`+ > / 6>:0+ Jhmkg 6>:0 tbh bmgbhst prhssurh+

    ^asturh Yhksar

    ^asturh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    71/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 76

    Pmohstfop FiiR Fii] Zmgbt Yhft Om``ch Yhft Chet Yhft Zmgbt Jfil Om``ch Jfil Chet Jfil

    66>7:>>70982=>6 =2> /676 6>68 6>6= 6>6: =75 880 9>=

    66>7:>>7098=695 =5= /65= 6>>2 6>>0 6>6: =72 80> 9>8

    66>7:>>7098=0>6 6>>2 /658 6>68 6>6: 6>>2 =70 880 952

    66>7:>>7098=595 ==9 /655 6>>6 6>>0 6>65 =78 802 9>5

    66>7:>>7098=280 ==0 /657 6>68 6>66 6>>9 =57 88= 062

    66>7:>>700>>6:> =7> /657 6>66 6>>2 6>>6 =52 5:> 982

    66>7:>>700>>980 =77 /655 6>66 6>66 6>69 =52 806 069

    66>7:>>700>>82= =28 /6:2 6>6: 6>6> 6>>5 =70 858 960

    66>7:>>700>>29= ==5 /62: 6>65 6>60 6>6: =7: 552 :=>

    66>7:>>700>6>2= ==6 /628 6>6: 6>6: 6>>0 282 6>2 :

    66>7:>>700>68:5 6>52 /96> 6>6= 8:: 6>>6 9: > 0:6

    66>7:>>700>6887 =97 /6:0 =: > > > 6 :07

    66>7:>>700>6760 =7= /6>0 6>9 > > > > 27

    66>7:>>700>6708 =87 /658 609 9 > > > >

    66>7:>>700>:>:5 =08 /676 6:5 > > > 6 9

    66>7:>>700>:99= =02 /65= > > 6 > > >

    66>7:>>700>:5:> =8: /655 9 > 6 > > >

    ^asturh Yhksar

    ^asturh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    72/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 7:

    ^asturh Yhksar

    Pbh rfw `ftf erao tbh smx ibfmr shksars ms praihssh` \69_ ta ajtfmk4 khtshft ibfkgh# kht jfil ibfkgh# fk` smt earwfr` vfcuhs+

    \6:_ Iaaphr# @+# Frraya# M+# Waace# J+# Ouc`khr# L+# Jurchsak# W+# fk` Ibrmstapbhrsak# Z+ ":>>=-+ Yhksars oa`hc stu`hkt shce iakihpt mk tbh icfssraao#Xshr Oa`hcmkg# F`fptftmak# fk` hrsakfcmzftmak# 9>//06

    ^asturh Yhksar

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    73/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 79

    Pmohstfop Zmgbt Yhft Om``ch Yhft Chet Yhft Zmgbt Jfil Om``ch Jfil Chet Jfil

    KhtYhft

    Ibfkgh

    KhtJfil

    Ibfkgh YmtEarwfr`

    66>7:>>70982=>6 6>68 6>6= 6>6: =75 880 9>= 6: 68: >

    66>7:>>7098=695 6>>2 6>>0 6>6: =72 80> 9>8 :: :> >

    66>7:>>7098=0>6 6>68 6>6: 6>>2 =70 880 952 6= 26 >

    66>7:>>7098=595 6>>6 6>>0 6>65 =78 802 9>5 9> 5= >

    66>7:>>7098=280 6>68 6>66 6>>9 =57 88= 062 90 696 >

    66>7:>>700>>6:> 6>66 6>>2 6>>6 =52 5:> 982 = 6:: >

    66>7:>>700>>980 6>66 6>66 6>69 =52 806 069 68 690 >

    66>7:>>700>>82= 6>6: 6>6> 6>>5 =70 858 960 = 6:= >

    66>7:>>700>>29= 6>65 6>60 6>6: =7: 552 :=> 60 6:= >

    66>7:>>700>6>2= 6>6: 6>6: 6>>0 282 6>2 : 60 =5: >

    66>7:>>700>68:5 6>6= 8:: 6>>6 9: > 0:6 8>> 6989 >

    66>7:>>700>6887 =: > > > 6 :07 :08> :>7 6

    66>7:>>700>6760 6>9 > > > > 27 66 656 6

    66>7:>>700>6708 609 9 > > > > 09 27 6

    66>7:>>700>:>:5 6:5 > > > 6 9 :> 0 6

    66>7:>>700>:99= > > 6 > > > 6:7 0 6

    66>7:>>700>:5:> 9 > 6 > > > 9 > 6

    ^asturh Yhksar

    Yuoofry

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    74/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 70

    Yhksmkg@hvmih

    "rfth mk Bz-Chgfiy Yaetwfrh Yhksmkg

    "Mkput ar Zfw @ftf-^bysmacagmifc rhspakshs fk`!ar

    Hoatmak rhparth` "autput ar shksh` vfcuhs-

    Hoatmv HHGbhf`sht"6:2 Bz-

    Hoatmv Y@L Jrfmk Wfvhs

    HHG fitmvmty+ Zhparth` mk 60 ibfkkhcs \65_#4 FE9# E7# E9# EI8# P7# ^7# A6# A:#^2# P2# EI5# E0# E2# fk` FE0+

    Efih fitmvmty+ Jcmkl# wmkl "chet fk` rmgbt-# caal "chet fk` rmgbt-# rfmsh jraw# eurraw

    jraw# somch# ichkib# somrl "chet fk` rmgbt-# fk` cfugb+

    Hoatmaks+ Hximthohkt# hkgfghohkt# jarh`ao# oh`mtftmak fk` erustrftmak+

    Ytfk`fr` Whjifo"6> Bz-

    OMP Oh`mf Cfj Omk`Zhf`hr Efimfc Hxprhssmaks Hoatmak+ Fgrhhmkg# iakihktrftmkg# msfgrhhmkg# mkthrhsth`# tbmklmkg fk` uksurh+

    OMP slmk iak`uitfkih shksar": Bz-

    XYJ `rmvhr Ylmk Iak`uitmvmty Frausfc+

    OMP prhssurh shksar"5 Bz- XYJ rmvhr ^rhssurh Akh prhssurh vfcuh phr shksar fccaifth` mkta tbh mkput!iaktrac hvmih+

    Pajmm Hyh trfilmkg"5> Bz-

    Pajmm Y@L Hyh Prfilmkg Gfzh pamkt "x# y-+

    OMP pasturh shksar"5 Bz-

    XYJ rmvhr ^rhssurh^rhssurh vfcuhs mk tbh jfil fk` tbh shft "mk tbh rmgbt# om``ch fk` chet zakhs- ae f

    iusbmak ibfmr+

    Yuoofry

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    75/120

    Yhssmak :

    78

    Jfilgrauk`

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    76/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Jfilgrauk`

    75

    Pbh `ftfshts sbawh` mk tbh khxt scm`hs iarrhspak` ta tbh `ftf iacchith` mk tbrhhstu`mhs+ Pbh stmoucm# prataiac# pfrtmimpfkts# frh `hsirmjh` jrmhecy mk tbh eaccawmkgpfrfgrfpbs+

    6+ Ytu`y Akh4 Yujdhits pcfymkg f vm`ha gfoh+

    :+ Ytu`y Pwa4 Yujdhits rhf`mkg `aiuohkts wmtb fk` wmtbaut pmiturhs+

    9+ Ytu`y Pbrhh4 Yujdhits sacvmkg tfsls mk f Putarmkg Yystho+

    Ytu`y Akh

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    77/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Ytu`y Akh

    77

    Ytmoucm+ Pbh bmgb/em`hcmty grfpbmi# `hhpcy hkgfgmkg# Gumtfr Bhra vm`hagfoh+ Pbh gfoh tbft mkvacvhs bac`mkg f gumtfr mkthrefih wbmch cmsthkmkg ta ousmifk` wftibmkg f vm`ha sirhhk+

    ^rataiac+ Pbh stu`y iaksmsts mk f akh/baur shssmak wmtb4 "6- f 68 omkuths taprfitmih wmtb tbh ajdhitmvh tbft tbh ushr ghts efomcmfr wmtb tbh gfoh iaktracchr fk`

    tbh hkvmrakohkt< fk` ":- f 08 omkuths wbhk ushrs pcfyh` eaur sakgs ae tbhmribamih akh ae hfib chvhc4 hfsy# oh`muo# bfr` fk` hxphrt+

    ^frtmimpfkts+ Pbh ifcc ear pfrtmimpftmak wfs fk aphk ifcc foakg Frmzakf YtfthXkmvhrsmty stu`hkts+ Pbh hxphrmohkt wfs ruk avhr :6 sujdhits# 57( ae tbho whrhohk fk` 99( waohk+ Pbh fgh rfkgh gahs erao 62 ta :2 yhfrs+ Pbh stu`ymkicu`hs sujdhits wmtb `meehrhkt "shce/rhparth`- hxphrmhkih pcfymkg vm`ha gfohs#wbhrh 68( khvhr pcfy jhearh# 8( whrh kat slmcch` mk vm`ha gfohs# :2( rhparth`jhmkg scmgbtcy slmcch`# 99( saohwbft slmcch`# 60( vhry slmcch`# fk` akcy 8(rhparth` tbho shce fs hxphrts+

    Ytu`y Pwa

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    78/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Ytu`y Pwa

    72

    Ytmoucm+ Ak/sirhhk rhf`mkg ofthrmfc wfs ush` ear tbms stu`y+ Pwa `meehrhkttyphs ae rhf`mkg ofthrmfc whrh ush`# akh wmtb hmtbhr ak/tfsl ar aee/tfsl mofghs#ifptmaks# fk` `rfws< fk` tbh shiak` akh# wbmib akcy iaktfmks thxt+

    ^rataiac+ Pbh stu`y iaksmsts ae akh 5>/omkuths shssmak wbhrh tbh sujdhit msprhshkth` wmtb 6> pfghs erao f papucfr H`uiftmakfc ^syibacagy thxtjaal fk`

    fslh` ta rhf` ear uk`hrstfk`mkg+ Hfib pfrtmimpfkt wfs fslh` ta iaopchth f prhfk` past/thst+

    ^frtmimpfkts+ Pbh ifcc ear pfrtmimpftmak fcsa wfs fk aphk ifcc foakg FrmzakfYtfth Xkmvhrsmty stu`hkts+ Pbh stu`y wfs rfk avhr :2 sujdhits+

    Ytu`y Pbrhh

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    79/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Ytu`y Pbrhh

    7=

    Ytmoucm+ F Putarmkg Yystho ear systho `ykfomis+ Pbh tutar tutars stu`hkts akbaw ta oa`hc systhos jhbfvmar usmkg f grfpbmifc rhprhshktftmak fk` fcghjrfmihxprhssmaks+ Pbh oa`hc ms rhprhshkth` usmkg f `mrhith` grfpb struiturh tbft`hemkhs f tapacagy earoh` jy ka`hs fk` cmkls+ Akih tbhrh ms f iaopchth oa`hc#stu`hkts ifk hxhiuth mt# fk` `hjug mt \69_+

    ^rataiac+ Ytu`hkts whrh fslh` ta phrearo f sht ae tfsls "fjaut systho `ykfomisoa`hcmkg- usmkg tbh tutarmkg systho+ Wbmch tbh stu`hkt wfs warlmkg ak tbhshtfsls# tbh tutarmkg systho iacchith` hoatmakfc stfth mkearoftmak wmtb tbh mkthktmakae jhmkg fjch ta ghkhrfth jhtthr fk` oarh fiiurfth bmkts fk` ehh`jfil+

    ^frtmimpfkts+ ^mcat thst `urmkg Yuoohr :>6> wmtb : graups ar 9> bmgb/sibaac

    stu`hkts+

    \60_ L+ UfkChbk# W+ Jurchsak# O+H+ Ibfvhz/Hibhfgfrfy# Z+ Ibrmstapbhrsak# D+ Gakzfchz/Yfkibhz# ]+ Bm`fcga/^aktht# fk` C+ Tbfkg+ Pbh Feehitmvh Ohtf/Putarmkg^radhit4 Baw ta oatmvfth stu`hkts ta ush heehitmvh ohtf/iagkmtmvh strfthgmhs# raihh`mkgs ae tbh 6=tb Mkthrkftmakfc Iakehrhkih ak Iaoputhrs mk H`uiftmak+ IbmfkgOfm# Pbfmcfk`4 Fsmf/ fimemi Yaimhty ear Iaoputhrs mk H`uiftmak+ Aitajhr :>66+ Mk prhss+

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    80/120

    9+ @ftf Emcthrmkg fk` Mkthgrftmak

    2>

    Yhssmak :

    Emcthrmkg

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    81/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    th g

    26

    Emcthrmkg tbh `ftf mkicu`hs ichfkmkg# sykirakmzmkg# fk` fvhrfgmkgar pfssmkg f tbrhsbac` ar f rfkgh+

    Emcthrmkg fk` Mkthgrftmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    82/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    g g

    2:

    Pbh ouctmoa`fc hoatmak rhiagkmtmak iaksm`hrs tbh hxmsthkih ae `meehrhktsaurihs ae `ftf tbft iaktrmjuth ta mkehr tbh feehitmvh stfth ae fk mk`mvm`ufc+

    Hfib shksmkg `hvmih bfs mts awk typh ae `ftf fk` sfopch rfth+

    Pbh ibfcchkgh ms baw ta iaojmkh tbh `ftf iaomkg erao tbhsh `mvhrshshksmkg `hvmihs tbft bfvh pravh` tbhmr mk`hphk`hkt eukitmakfcmty fk` irhfth fk

    mopravh` ukmquh autput+

    Emcthrmkg fk` Mkthgrftmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    83/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    g g

    Ihktrh

    Fghkt

    Fghkt

    @ftfCagghr

    @ftf

    Putarmkg

    Ouctmoa`fc

    29

    Erfohwarl

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    84/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 20

    Wh `hvhcaph` f erfohwarl tbft eaccaws tbh argfkmzftmakfc strfthgy ae fk fghkteh`hrftmak \68_+

    Pbh eh`hrftmak fssmgks akh fghkt ta iacchit rfw `ftf erao hfib shksmkg `hvmih+

    Pbft fghkt mopchohkts tbh phrihptmak ohibfkmso ear mts fssmgkh`shksmkg `hvmih ta ofp rfw `ftf mkta jhcmhes+

    Hfib fghkt ms futakaoaus fk` hkifpsucfth akh shksmkg `hvmih fk` mts phrihptmakohibfkmsos mkta mk`hphk`hkt# mk`mvm`ufc# fk` mkthccmghkt iaopakhkts+ Fcc tbh `ftf

    ms tmohstfoph` fk` mk`hphk`hktcy m`hktmemh` jy fghkt+

    \68_ Gakzfchz/Yfkibhz# D+< Ibfvhz/Hibhfgfrfy# O+H+< Ftlmksak# Z+< fk` Jurchsak# W+ ":>66-# *FJH4 Fk Fghkt/Jfsh` Yaetwfrh Fribmthiturh ear f Ouctmoa`fc HoatmakZhiagkmtmak Erfohwarl#* mk raihh`mkgs ae Warlmkg MHHH!MEM^ Iakehrhkih ak Yaetwfrh Fribmthiturh "Dukh :>66-+

    Erfohwarl

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    85/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 28

    Wbhk shvhrfc fghkts khh` ta mkthrfit# mt ms mopartfkt ta hstfjcmsb fk argfkmzftmakfcstrfthgy ear tbho# wbmib `hemkhs futbarmty rhcftmaksbmps# `ftf ecaws fk`iaar`mkftmak prataiacs \65_ \67_+

    Pbhsh jhcmhes frh rhparth` ta f ihktrfc fghkt# wbmib mkthgrfth tbho mkta akhfeehitmvh stfth rhpart+

    Pbmr`/pfrty systhos frh fjch ta ajtfmk feehitmvh stfth rhparts erao tbh Ihktrfcfghkt usmkg f pujcmsb/sujsirmjh stych+

    \65_ E+ Pumdkofk# fk` B+ Fesfrofkhsb# *@mstrmjuth` ajdhits mk f eh`hrftmak ae futakaoaus iaaphrftmkg fghkts#* ^rai+ Mkthrkftmakfc Iakehrhkih akMkthccmghkt fk` Iaaphrftmvh Mkearoftmak Yysthos# Ofy 6==9# pp+ :85/:58# am46>+66>=!MIMIMY+6==9+:=6759+

    \67_ O+ Waa`# fk` Y+ @hCafib# Fk avhrvmhw ae tbh ouctmfghkt systhos hkgmkhhrmkg ohtba`acagy# Fghkt/Armhkth` Yaetwfrh Hkgmkhhrmkg# "FAYH

    :>>>-# Yprmkghr/Uhrcfg# :>>6# pp+ :>7/::6# am46>+6>>7!9/80>/00850/6Q60+

    Erfohwarl | Fribmthiturh

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    86/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    \68_ Gakzfchz/Yfkibhz# D+< Ibfvhz/Hibhfgfrfy# O+H+< Ftlmksak# Z+< fk` Jurchsak# W+ ":>66-# *FJH4 Fk Fghkt/Jfsh` Yaetwfrh Fribmthiturh ear f Ouctmoa`fcHoatmak Zhiagkmtmak Erfohwarl#* mk raihh`mkgs ae Warlmkg MHHH!MEM^ Iakehrhkih ak Yaetwfrh Fribmthiturh "Dukh :>66-+

    25

    Mkthgrftmak | spfrsh

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    87/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 27

    \62_ D+ Cmu# Y+ Dm# fk` D+ ]h+ YCH^4 Ypfrsh Chfrkmkg wmtb Heemimhkt ^radhitmaks+ Frmzakf Ytfth Xkmvhrsmty# :>>=+ bttp4!!www+pujcmi+fsu+h`u!~dyh>:!Yaetwfrh!YCH^+

    tmohstfop emxftmakM k hx gfz h amktR gfz h amkt]ofpph`Emxftmak^a

    mktR

    ofpph`Emxftmak^a

    mkt]emxftmak@urftmak

    Ybart Phro

    Hximthohkt

    Cakg Phro

    Hximthohkt

    Hkgfghohkt!

    Jarh`aoOh mtftmak Erustrftmak Iak uitfkih fgrhhohkt iakihktrftmkg

    069878858: >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59

    069878858= :69 879 0>2 87> 0>2 :65

    0698788552 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2

    0698788575 :69 855 06: 87> 0>2 :65

    06987885=: :69 858 0>0 87> 0>2 :65

    06987887>= :69 857 0>0 87> 0>2 :65

    0698788760

    06987887:5 :69 852 066 87> 0>2 :65

    069878870: :69 852 0>= 87> 0>2 :65

    069878878= :69 859 066 87> 0>2 :65

    0698788756

    0698788775 :69 870 069 87> 0>2 :65

    06987887=: :69 880 0>: 87> 0>2 :65

    06987882>= :60 5>9 0>= 5=5 0>8 :65

    06987882:0

    06987882:5 :60 7>6 0>7 5=5 0>8 :65

    069878820: :60 5=7 0>9 5=5 0>8 :65

    069878828= :60 5=9 0>6 5=5 0>8 :65

    0698788275 :60 7>> 0>: 5=5 0>8 :65

    06987882=: :60 7>6 066 5=5 0>8 :65

    0698788=>= :60 525 9=2 5=5 0>8 :65

    0698788=62

    0698788=:5 :60 5=0 9== 5=5 0>8 :65

    0698788=0: :60 5=0 0>7 5=5 0>8 :65

    0698788=8= :60 5=2 0>0 5=5 0>8 :65

    0698788=50

    0698788=75 :60 7>0 9=2 5=5 0>8 :65

    0698788==: :60 5=9 068 5=5 0>8 :65

    0698785>>= :60 5=5 066 5=5 0>8 :65

    0698785>:8 :68 7:2 0>5 2>0 927 629

    0698785>:7 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 6 6

    Mkthgrftmak | stfth ofibmkh

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    88/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 22

    tmohstfop emxftmakM k hx gfz h amktR gfz h amkt]ofpph`Emxftmak^a

    mktR

    ofpph`Emxftmak^a

    mkt]emxftmak@urftmak

    Ybart Phro

    Hximthohkt

    Cakg Phro

    Hximthohkt

    Hkgfghohkt!

    Jarh`aoOh mtftmak Erustrftmak Iak uitfkih fgrhhohkt iakihktrftmkg

    069878858: :69 870 060 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    069878858= :69 879 0>2 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788552 :69 879 0>2 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788575 :69 855 06: 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987885=: :69 858 0>0 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987887>= :69 857 0>0 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788760 :69 857 0>0 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987887:5 :69 852 066 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    069878870: :69 852 0>= 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    069878878= :69 859 066 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788756 :69 859 066 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788775 :69 870 069 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987887=: :69 880 0>: 87> 0>2 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987882>= :60 5>9 0>= 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987882:0 :60 5>9 0>= 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987882:5 :60 7>6 0>7 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    069878820: :60 5=7 0>9 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    069878828= :60 5=9 0>6 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788275 :60 7>> 0>: 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    06987882=: :60 7>6 066 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788=>= :60 525 9=2 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788=62 :60 525 9=2 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788=:5 :60 5=0 9== 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788=0: :60 5=0 0>7 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788=8= :60 5=2 0>0 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788=50 :60 5=2 0>0 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788=75 :60 7>0 9=2 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698788==: :60 5=9 068 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698785>>= :60 5=5 066 5=5 0>8 :65 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698785>:8 :68 7:2 0>5 2>0 927 629 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    0698785>:7 :68 7:2 0>5 2>0 927 629 >+0955=7 >+8:6>8= >+88>>66 >+9982:8 >+0=2=>2 >+0>65=>59 6 6

    Mkthgrftmak | tmoh wmk`aw

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    89/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 2=

    tmohstfop KhtYhftIbfkgh KhtJfilIbfkgh YmtEarwfr` ZmgbtYhft O m c hYhft ChetYhft ZmgbtJfil O m c hJfil ChetJfil OaushUfcuh ZmgbtZhfr ZmgbtErakt ChetZhfr ChetErakt

    66>7:>>=976 6> 68 > ===+:2255 6>6:+>06:0 =77+98>865 =7=+>6>9>= =97+275:2= =:6+7796=5 8 6>::+=8:69 6>::+=5:77 6>::+=8708 6>::+=52>=66>7:>>=97: 65 := > 6>>9+:=2=7 6>66+8>865 =7>+:8779: =2:+8059=: =:0+976690 =:=+2>06:0 8 6>::+=20>0 6>::+=8708 6>::+=5:77 6>::+=8:69

    66>7:>>=926 69 67 > 6>>8+2>:>2 6>66+0:7>2 =75+:9=829 =7=+7>2999 =9=+8:>299 =00+2=8299 8 6>::+=8708 6>::+=727: 6>::+==052 6>::+=52>=

    66>7:>>=92: 60 :> > 6>>8+92600 6>66+059=: =79+2=5=>7 =2:+>=:720 =92+8=7=92 =05+50=028 8 6>::+=8708 6>::+=20>0 6>::+=8708 6>::+=8708

    66>7:>>=9=6 6> 8: > 6>>8+690>: 6>6:+>:>5: =77+70::52 =7:+:97669 57:+9:=2=7 =:9+8059=: 8 6>::+=7220 6>::+=7988 6>::+=2=0: 6>::+=8757

    66>7:>>=9=: 6> 68 > 6>>0+95>29 6>6:+0=028 =27+8>8688 =26+926009 258+0>:>5: =0:+::52>0 8 6>::+=2=96 598+9952=2 6>:6+28>:7 6>::+=5:87

    66>7:>>=0>6 6> :6 > 6>>9+>8:>2 6>6:+>=978 =22+978 =20+05278 279 =08+:7>299 8 6>::+=72=8 ==>+70:6>8 6>::+=687= 6>::+=2=07

    66>7:>>=0>: = = > ==7+5>:>06 6>6:+:60:= =2=+86>:>0 =2:+529570 252+02=7=5 =0:+=02=2 8 6>:9 6>:>+80727 6>:9 6>:9

    66>7:>>=066 6> 79 > ===+>9>=:2 6>69+>6>96 =20+8857>6 =78+5:2255 7=6+650=0= =92+>9>=:2 8 6>::+=20>0 6>::+=8708 6>::+=20>0 6>::+=8:69

    66>7:>>=06: 66 87 > 6>>:+0=028 6>6:+56285 =72+260099 =7>+:97669 252+0>:>5: =02+50=028 8 6>::+=8757 6>::+=7988 6>::+=52:8 6>::+=07>=

    66>7:>>=0:6 = 68 > 6>>>+=0208 6>69+28857 =29+28857 =78+=02080 =96+:52>06 =56+9:=2=7 8 6>::+=790 6>::+=5:77 6>::+=5:77 6>::+=8708

    66>7:>>=0:: 6> 60 > 6>>>+:52>0 6>69+678:5 =29+020895 =2>+75:227 =98+76690 =5:+>=:720 8 6>::+=0526 6>::+=2=95 6>::+=790 6>::+=8708

    66>7:>>=096 6> :5 > 6>>>+=:720 6>69+:9766 =2:+95>2:8 =70+=82759 =6:+5=>7:: =02+7:658 8 6>::+=29=5 6>::+=29=5 6>::+=87:: 6>::+=058:

    66>7:>>=09: 6> :8 > 6>>6+57>6 6>69+9>=:2 =26+2>06:0 =75+857>6 =92+:2255 =88+7>6>96 8 6>::+==052 6>::+=790 6>::+==052 6>::+=20>0

    66>7:>>=006 = 5: > 6>>:+8:877 6>69+86805 =20+5>2:07 =76+:070:9 28:+7298>8 =07+>=:720 8 6>:9 6>:9 6>:9 6>:9

    66>7:>>=00: 6: 9> > 6>>5+66082 6>66+:>299 =22+829999 =79+7:=657 205+608299 =02+87:=67 8 6>::+=7256 6>::+=2=96 6>::+=79:5 6>::+==058

    66>7:>>=086 6> 98 > 6>>5+5:8 6>66+50829 ==8+278 =52+=65557 2>=+05278 =90+::=657 8 6>::+=790 6>::+=2=95 6>::+=20>0 6>::+=727:

    66>7:>>=08: 6> 97 > 6>>5+90>:6 6>6:+258=2 ==5+>6>9>= =7:+0::52 2>7+=>7:67 =99+8779: 8 6>::+=20>0 6>::+=8:69 6>::+=727: 6>::+=790

    66>7:>>=056 = 9: > 6>>8+25082 6>6:+7>299 ==5+278 =76+9::=67 268+2::=67 =92+278 8 6>::+==052 6>::+=52>= 6>::+=790 6>::+=727:

    66>7:>>=05: 2 :6 > 6>>5+9::=: 6>66+=978 ==7+02=829 =7>+278 28:+250829 =0:+>=978 8 6>::+=57=6 6>::+=9829 6>::+=9>02 6>::+=058:

    66>7:>>=076 6> :8 > 6>>8+87:=: 6>66+0:7>2 ==7+>29999 =7:+>5:8 220+:26:8 =09+299999 8 6>::+=2=95 6>::+=2=95 6>::+=52>= 6>::+=790

    66>7:>>=07: 6> 67 > 6>>8+>7:=: 6>66+0>5:8 ==7+560829 =79+999999 =>6+20978 =00+007=67 8 6>::+=790 6>::+=:889 6>::+=0526 6>::+=727:

    66>7:>>=026 6> 65 > 6>>8+7=926 6>6:+9:== ==2+78:877 =7>+57>6>9 =>>+976690 =05+895>29 8 6>::+=52>= 6>::+=8708 6>::+=5:77 6>::+=060=

    66>7:>>=02: 6> :: > 6>>8+=978 6>6:+99999 ===+=7=657 =7:+=7=657 2=>+90978 =08+980657 8 6>::+=2=95 6>::+=8:69 6>::+=5:77 6>::+=8:69

    66>7:>>=0=6 = 6> > 6>>7+08299 6>6:+79=82 6>>>+::=67 =76+585:8 =>>+:9=829 =08+=5278 8 6>::+=2=95 6>:9 6>::+=20>0 6>::+==052

    66>7:>>=0=: = :6 > 6>>7+9>098 6>69 6>>>+5>27 =50+56=858 25=+59>098 =0=+:5>27 8 6>::+==076 6>:9 6>::+=8757 6>::+=5:=5

    66>7:>>=8>6 = 65 > 6>>7+:= 6>69+60 ==2+=0 =56+=2 27>+58 =8>+75 8 6>::+=727: 6>::+==052 6>::+=2=95 6>::+=20>0

    66>7:>>=8>: 6> 66 > 6>>5+:6278 6>6:+90978 ===+228067 =59+660829 22:+:7>299 =09+577>29 8 6>::+=29=5 6>:9 6>::+=2=96 6>::+==058

    66>7:>>=866 2 60 > 6>>5+2::=: 6>6:+=7=67 6>>>+685:8 =57+6>0657 220+>7:=67 =0:+:6278 8 6>::+=2069 6>::+=7220 6>::+==076 6>::+=2069

    66>7:>>=86: = 68 > 6>>5+=>5:8 6>69+0:7>2 6>>6+>0657 =57+02=829 220+7>2999 =92+8:>299 8 6>:9 6>:9 6>:9 6>:9

    66>7:>>=8:6 = :0 > 6>>5+7>8:5 6>69+>20:6 6>>>+>9682 =55+72=070 255+8520:6 =0>+6872=8 8 6>:9 6>:9 6>:9 6>:9

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    90/120

    0+ Fkfcyzmkg @ftf

    =>

    Yhssmak :

    Paac | Hurhqf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    91/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    oftbhoftmifc rhcftmaksbmps mk ftf

    =6

    Paac | Hurhqf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    92/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy =:

    Ear rhvhrsh hkgmkhhrmkg shfribhs ae tbh `ftf# tbh Hurhqf taac \6=_ msush` ta `msiavhroftbhoftmifchxprhssmaks ae tbh struiturfc rhcftmaksbmpsmk tbh `ftf rhiar`s+

    Ear hxfopch# me f rhiar` bac`s mkearoftmak fjaut tbh pbysmifc fk` hoatmakfc

    jhbfvmar ae fk mk`mvm`ufc wba wfs hkgfgh` mk f smkgch hxphrmohktfc shttmkg#Hurhqf iauc` tflh fcc tbh fvfmcfjch saurihs ae `ftf fk` rhvhfc jatb baw tbhohfsurh ae hkgfghohkt ms ifciucfth` erao sphimemi `ftf strhfos fs whcc fs bawatbhr shksars ofy mkecuhkih tbh prapash` hoatmakfc iakstruit+

    \6=_ @ujiflavf

    # Z+ Hurhqf/saetwfrh rhvmhw+ Ghkhtmi pragrfoomkg fk` hvacvfjch ofibmkhs+ Ghkht+ ragrfo+ Hvac+ Ofib+ ":>6>- akcmkh emrst+ am46>+6>>7!s6>76>/

    >6>/=6:0/z +

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    93/120

    =9

    Paac | Hurhqf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    94/120

    =0

    Paac | Hurhqf

    Paac | Whlf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    95/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    hxpcarh icfssmemiftmak icusthrmkg

    prh/praihssmkg vmsufcmzftmak

    =8

    Paac | Whlf

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    96/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy =5

    Earicusthrmkg fk` icfssmemiftmak fpprafibhs Whlf\:>_Mt ms f taac tbft mopchohkts f iacchitmak ae ofibmkh chfrkmkg fcgarmtbos ear `ftfomkmkg tfsls# ms ush` ta hxpcarh tbh `ftf iaopasmtmak fk` rhcftmaksbmps fk` `hrmvhusheuc lkawch`gh erao `ftf rhiar`s+

    \:>_ O+ Bfcc# H+ Erfkl# G+ Bacohs# J+ ^efbrmkghr# ^+ Zhuthofkk# fk` M+B+ Wmtthk# Pbh WHLF @ftf Omkmkg Yaetwfrh4 Fk Xp`fth# ^rai+ YMGL@@ Hxpcarftmaks# :>>=#

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    97/120

    =7

    Paac | Whlf

    ^>>>= kavmih pcfymkg hxphrt sakg

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    98/120

    =2

    Paac | Whlf

    ^>>>= kavmih pcfymkg hxphrt sakg

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    99/120

    ==

    Paac | Whlf

    @aiuohkth` F^M ear Dfvf @hvhcaphrs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    100/120

    8+ Ybfrmkg Hxphrmhkihs fk` Graup @msiussmak

    6>>

    Yhssmak :

    Hxphrmhkihs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    101/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>6

    Umsufcmzftmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    102/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>:

    JIM fk` Gfzh amkts hkgfghohkt

    Pbms emgurh sbaws tbh hkgfghohkt emxftmak pamkts ae hxphrt pcfyhr pcfymkg mk hxphrt/oa`h+ Pbh smzh ae tbh imrich rhprhshkts tbh urftmak ae tbh emxftmak mk tbft pamkt#wbmch tbh chvhc ae sbf`mkg rhprhshkts tbh mkthksmty ae tbh hoatmak+

    Umsufcmzftmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    103/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>9

    JIM fk` Gfzh amkts erustrftmak

    Pbms emgurh sbaws tbh erustrftmak emxftmak pamkts ae hxphrt pcfyhr pcfymkg mk hxphrt/oa`h+ Pbh smzh ae tbh imrich rhprhshkts tbh `urftmak ae tbh emxftmak mk tbft pamkt# wbmchtbh chvhc ae sbf`mkg rhprhshkts tbh mkthksmty ae tbh hoatmak+

    Umsufcmzftmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    104/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>0

    JIM fk` Gfzh amkts jarh`ao

    Pbms emgurh sbaws tbh jarh`ao emxftmak pamkts ae hxphrt pcfyhr pcfymkg mk hxphrt/oa`h+ Pbh smzh ae tbh imrich rhprhshkts tbh urftmak ae tbh emxftmak mk tbft pamkt# wbmchtbh chvhc ae sbf`mkg rhprhshkts tbh mkthksmty ae tbh hoatmak+

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    105/120

    Umsufcmzftmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    106/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>5

    JIM fk` Gfzh amkts erustrftmak

    Pbms emgurh sbaws tbh erustrftmak gfzh pamkts "fjavh f tbrhsbac` ae >+5- ae f ushr rhf`mkg f ofthrmfc wmtb sh`uitmvh htfmcs "m+h+ ifrtaaks-+Caalmkg tbh ifrtaak ms rhcfth` wmtb bmgb erustrftmak chvhc+

    Umsufcmzftmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    107/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>7

    JIM fk` Gfzh amkts jarh`ao

    Pbms emgurh sbaws tbh jarh`ao gfzh pamkts "fjavh f tbrhsbac` ae >+8- ae f ushr wbmch rhf`mkg f ofthrmfc wmtb sh`uitmvh htfmcs "m+h+ ifrtaaks-+Katmih tbft tbh thxt mk tbh om``ch pfrt ae tbh shiak` iacuok ae tbft pfgh wfs jarmkg+

    Oftibmkg Ufcuhs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    108/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>2

    Aur bypatbhsms ms tbft# ta f grhft hxthkt# mt ms passmjch ta mkehr tbh vfcuhs erao akh saurih erao tbh atbhr saurih+

    JIM/jfsh` vfcuhs Iarrhcftmak wmtb efih/jfsh` oa`hc

    hximthohkt >+:20

    hkgfghohkt >+:2:

    oh`mtftmak >+622

    erustrftmak >+:78

    Efih/jfsh` vfcuhs Iarrhcftmak wmtb JIM/jfsh` oa`hc

    fgrhhohkt >+75>

    iakihktrftmkg >+758

    `msfgrhhohkt >+7=0

    mkthrhsth` >+770

    tbmklmkg >+72>

    uksurh >+2:2

    JIM fk` Efih/Jfsh` Hoatmak Zhiagkmtmak

    Oftibmkg Ufcuhs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    109/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 6>=

    *

    \:6_ C+ Jrhmofk+ Zfk`ao earhsts+ Ofibmkh Chfrkmkg# :>>6+ Uacuoh 08# Kuojhr 6+ ^fghs 89:+

    Zfk`ao Earhst

    Khtwarls

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    110/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Ytruiturfc"Hquf,aks#"F`dfihkiy"Oftrmxhs"

    fk`"Khtwarls"Grfpbs"

    Jrfmk"sibhof,i#"sbawmkg"tbh"ibfkkhcs"tbft"iaktrmjuth"wmtb"hkgfghohkt"

    & &

    JIM rfw `ftf | Hkgfghohkt

    66>

    Graup @msiussmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    111/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 666

    `hvmihs `ftf mkehrhkihs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    112/120

    5+ Iakicusmaks fk` V%F

    66:

    Yhssmak :

    Iakicusmak

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    113/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Pbh gafc ae tbms tutarmfc wfs ta pravm`h tbh ftthk`hhs wmtb hkaugb mkearoftmak fk`

    hxfopchs ta hkfjch tbho ta stfrt tbhmr awk mkvhstmgftmaks ae tbh iagkmtmvh/feehitmvh hchohkts ae chfrkmkg\::_+ Wbmch tbh iaursh `a kat prhshkt fkhxbfustmvh cmst ae fcc tbh ohtba`s fvfmcfjch ear iacchitmkg# ofkmpucftmkg# fkfcyzmkgfk` mkthrprhtmkg feehitmvh shksar `ftf# tbh tutarmfc `hsirmjh tbh jfsms ae fouctmoa`fc fpprafib tbft ftthk`hhs ifk ush ta cfukib tbhmr awk rhshfrib heearts+

    \::_ J+ u Jaucfy# Pawfr`s f Oatmvftmakfccy/Mkthccmghkt h`fgagy4 Baw sbauc` fk mkthccmghkt tutar rhspak` ta tbh ukoatmvfth` ar tbh hoatmvfth`1# rai+ Khw^hrsphitmvhs ak Feehit fk` Chfrkmkg Phibkacagmhs# Z+ F+ Ifcva % Y+ @'Ohcca "H`s+-# Yprmkghr/Uhrcfg# mk prhss+

    669

    Zhehrhkihs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    114/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    6+ Z+Y+D+ Jflhr# O+O+P Za`rmga fk` X+H+ Racaiatzmk# Pbh @ykfomis ae Feehitmvh Prfksmtmaks mk Ymoucftmak

    ^rajcho/sacvmkg Hkvmrakohkts# ^rai+ Feehitmvh Iaoputmkg fk` Mkthccmghkt Mkthrfitmak4 Yhiak`Mkthrkftmakfc Iakehrhkih "FIMM >7-# F+ ^fmvf# Z+ ^rf`f % Z+ W+ ^mifr` "H`s+-# Yprmkghr/Uhrcfg# Uac+Chiturh Kaths mk Iaoputhr Yimhkih 0792# pp+ 555/577+

    :+ M+ Frraya# @+ G+ Iaaphr# W+ Jurchsak# E+ + Waace# L+ Ouc khr# fk` Z+ Ibrmstapbhrsak# Hoatmak YhksarsGa ta Yibaac# ^rai+ Frtmemimfc Mkthccmghkih mk H`uiftmak4 Jumc`mkg Chfrkmkg Yysthos tbft Ifrh4 eraoLkawch`gh Zhprhshktftmak ta Feehitmvh Oa`hccmkg# "FMH@ >=-# U+ @momtravf# Z+ Omzaguibm# J+ `u Jaucfy% F+ Grfsshr "H`s+-# MAY ^rhss# Ducy :>>=# vac+ Eraktmhrs mk Frtmemimfc Mkthccmghkih fk` Fppcmiftmaks :>>#pp+ 67/:0+

    9+ Z+ W+ mifr`# Feehitmvh Iaoputmkg# OMP rhss# 6==7+

    0+ D+ Gakzfchz/Yfkibhz# Z+ O+ Ibrmstapbhrsak# O+ H+ Ibfvhz/Hibhfgfrfy# @+ I+ Gmjsak# Z+ Ftlmksak# W+Jurchsak# Baw ta @a Ouctmoa`fc @hthitmak ae Feehitmvh Ytfths1# mifct# pp+580/588# :>66 MHHH 66tbMkthrkftmakfc Iakehrhkih ak F`vfkih` Chfrkmkg Phibkacagmhs# :>66

    8+ Hoatmv / Jrfmk Iaoputhr Mkthrefih Phibkacagy+ Zhtrmhvh` Fprmc :5# :>66# erao bttp4!!www+hoatmv+iao+

    5+ Ybfrjraugb E# Ibftrmfk G/H# Chsshr Z # C`hrs B# Kuwhr O# mitak PW+ FohrmifkHchitrahkihpbfcagrfpbmi Yaimhty Gum`hcmkhs ear Ytfk`fr` Hchitra`h ^asmtmak Kaohkicfturh+ D+ Icmk+Khurapbysmac 24 :>>/:+

    7+ Hchitrahkihpbfcagrfpby+ Zhtrmhvh` Kavhojhr 60tb# :>6># erao Hchitrmi fk` Ofgkhtmi Ohfsurhohkt aetbh Hchitrmi Fitmvmty ae Khurfc Pmssuh4 bttp4!!www+jho+em!jaal!69!69+bto

    660

    Zhehrhkihs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    115/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    2+ Z+ H+ Lfcmaujy fk` + Zajmksak# Zhfc/Pmoh Mkehrhkih ae Iaopchx Ohktfc Ytfths erao Efimfc

    Hxprhssmaks fk` Bhf` Ghsturhs# ^rai+ Iakehrhkih ak Iaoputhr Umsmak fk` ^ftthrk ZhiagkmtmakWarlsbap "IU^ZW >0-# MHHH Iaoputhr Yaimhty# Dukh :>>0# Uacuoh 6># p+680+

    =+ Pajmm Phibkacagy / Hyh Prfilmkg fk` Hyh Iaktrac+ Zhtrmhvh` Fprmc :5# :>66# erao bttp4!!www+tajmm+iao+

    6>+ O+ Ytrfuss# I+ Zhykac`s# Y+ Bugbhs# L+ ^frl# G+ Oi@frjy# fk` Z+W+ ^mifr # Pbh Bfk`WfvhJcuhtaatb Ylmk Iak`uitfkih Yhksar# ^rai+ Emrst Mkthrkftmakfc Iakehrhkih ak Feehitmvh Iaoputmkg

    fk` Mkthccmghkt Mkthrfitmak "FIMM >8-# Yprmkghr/Uhrcfkg# Ait+ :>>8# pp+ 5==/7>5# `am46>+6>>7!66879802Q=>+

    66+ ]+ Vm# fk` Z+ W+ ^mifr`# *Iakthxt/Yhksmtmvh Jfyhsmfk Icfssmemhrs fk` Fppcmiftmak ta Oaush ^rhssurh^ftthrk Icfssmemiftmak#* ^rai+ Mkthrkftmakfc Iakehrhkih ak ^ftthrk Zhiagkmtmak "MI^Z >:-# Fug+:>>:# vac 9# pp+ 9>002# `am46>+66>=!MI^Z+:>>:+6>07=79+

    6:+ Iaaphr# @+# Frraya# M+# Waace# J+# Ouc`khr# L+# Jurchsak# W+# fk` Ibrmstapbhrsak# Z+ ":>>=-+Yhksars oa`hc stu`hkt shce iakihpt mk tbh icfssraao# Xshr Oa`hcmkg# F`fptftmak# fk`

    ^hrsakfcmzftmak# 9>//06+

    69+ Y+ Oatf# fk` Z+ W+ ^mifr # *Futaofth` ^asturh Fkfcysms ear @hthitmkg Chfrkhrs Mkthrhst Chvhc#*^rai+ Iaoputhr Umsmak fk` ^ftthrk Zhiagkmtmak Warlsbap "IU^ZW >9-# MHHH ^rhss# Dukh :>>9#vac+ 8# pp+ 0=# `am46>+66>=!IU^ZW+:>>9+6>>07+

    668

    Zhehrhkihs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    116/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    60+ L+ UfkChbk# W+ Jurchsak# O+H+ Ibfvhz/Hibhfgfrfy# Z+ Ibrmstapbhrsak# D+ Gakzfchz/Yfkibhz# ]+

    Bm`fcga/^aktht# fk` C+ Tbfkg+ Pbh Feehitmvh Ohtf/Putarmkg ^radhit4 Baw ta oatmvfth stu`hkts taush heehitmvh ohtf/iagkmtmvh strfthgmhs# ^raihh`mkgs ae tbh 6=tb Mkthrkftmakfc Iakehrhkih akIaoputhrs mk H`uiftmak+ Ibmfkg Ofm# Pbfmcfk`4 Fsmf/ ^fimemi Yaimhty ear Iaoputhrs mk H`uiftmak+Aitajhr :>66+ Mk prhss+

    68+ D+ Gakzfchz/Yfkibhz< O+H+ Ibfvhz/Hibhfgfrfy< Z+ Ftlmksak< fk` W+ Jurchsak+ ":>66-# *FJH4 FkFghkt/Jfsh` Yaetwfrh Fribmthiturh ear f Ouctmoa`fc Hoatmak Zhiagkmtmak Erfohwarl#* mk^raihh`mkgs ae Warlmkg MHHH!MEM^ Iakehrhkih ak Yaetwfrh Fribmthiturh "Dukh :>66-+

    65+ E+ Pumdkofk# fk` B+ Fesfrofkhsb# *@mstrmjuth` ajdhits mk f eh`hrftmak ae futakaoaus iaaphrftmkgfghkts#* ^rai+ Mkthrkftmakfc Iakehrhkih ak Mkthccmghkt fk` Iaaphrftmvh Mkearoftmak Yysthos# Ofy6==9# pp+ :85/:58# am46>+66>=!MIMIMY+6==9+:=6759+

    67+ O+ Waa`# fk` Y+ @hCafib# Fk avhrvmhw ae tbh ouctmfghkt systhos hkgmkhhrmkg ohtba`acagy#Fghkt/Armhkth` Yaetwfrh Hkgmkhhrmkg# "FAYH :>>>-# Yprmkghr/Uhrcfg# :>>6# pp+ :>7/::6# `am46>+6>>7!9/80>/00850/6Q60+

    62+ \62_ D+ Cmu# Y+ Dm# fk` D+ ]h+ YCH 4 Ypfrsh Chfrkmkg wmtb Heemimhkt ^radhitmaks+ Frmzakf YtfthXkmvhrsmty# :>>=+ bttp4!!www+pujcmi+fsu+h`u!~dyh>:!Yaetwfrh!YCH +

    6=+ \6=_ @ujiflavf# Z+ Hurhqf/saetwfrh rhvmhw+ Ghkhtmi pragrfoomkg fk` hvacvfjch ofibmkhs+ Ghkht+^ragrfo+ Hvac+ Ofib+ ":>6>- akcmkh emrst+ `am46>+6>>7!s6>76>/ >6>/=6:0/z +

    665

    Zhehrhkihs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    117/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    :>+ O+ Bfcc# H+ Erfkl# G+ Bacohs# J+ ^efbrmkghr# + Zhuthofkk# fk` M+B+Wmtthk# Pbh WHLF @ftf Omkmkg

    Yaetwfrh4 Fk Xp`fth# ^rai+ YMGL@@ Hxpcarftmaks# :>>=# Uacuoh 66# Mssuh 6+

    :6+ C+ Jrhmofk+ Zfk`ao earhsts+ Ofibmkh Chfrkmkg# :>>6+ Uacuoh 08# Kuojhr 6+ ^fghs 89:+

    ::+ J+ `u Jaucfy# Pawfr`s f Oatmvftmakfccy/Mkthccmghkt ^h`fgagy4 Baw sbauc` fk mkthccmghkt tutar rhspak` tatbh ukoatmvfth` ar tbh `hoatmvfth`1# ^rai+ Khw ^hrsphitmvhs ak Feehit fk` Chfrkmkg Phibkacagmhs# Z+

    F+ Ifcva % Y+ @'Ohcca "H`s+-# Yprmkghr/Uhrcfg# mk prhss+

    667

    Vuhstmaks | Fkswhrs

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    118/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy 662

    Filkawch`ghohkts

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    119/120

    Dfvmhr Gakzfchz/Yfkibhz | Ofrmf/Hchkf Ibfvhz/Hibhfgfrfy

    Pbms rhshfrib wfs supparth` jyAeemih ae Kfvfc Zhshfrib uk`hr Grfkt K>>>60/6>/6/>609

    fwfr`h` ta @r+ Zajhrt Ftlmksak

    fk` jy

    Kftmakfc Yimhkih Eauk`ftmak# Fwfr` >7>8880# MMY!BIIFeehitmvh Chfrkmkg Iaopfkmaks4 Oa`hcmkg fk` suppartmkg hoatmak urmkg thfibmkg

    fwfr`h` ta @r+ Jhvhrcy Waace fk` @r+ Wmkscaw Jurchsak+

    66=

  • 7/28/2019 Multimodal Detection of Affective States | CHI 2012

    120/120

    csrc+cfj+fsu+h`u