entrainment on backchannel‐ preceding cuesrlevitan/papers/acl_talk.pdf · • does the way one...

EntrainmentonBackchannel‐PrecedingCues

RivkaLevitan Agus;nGravano JuliaHirschberg

1

Entrainment

'Aretheirheadsoff?'shoutedtheQueen.

'Theirheadsaregone, if itpleaseyourMajesty!'thesoldiersshoutedinreply.

'That's right!' shouted theQueen. 'Can youplaycroquet?’

'Yes!'shoutedAlice.'Come on, then!' roared the Queen, and Alice

joined the procession, wondering verymuchwhatwouldhappennext.

‐‐Alice’sAdventuresinWonderland

2

Entrainment

'Aretheirheadsoff?'shoutedtheQueen.

'Theirheadsaregone, if itpleaseyourMajesty!'thesoldiersshoutedinreply.

'That's right!' shouted theQueen. 'Can youplaycroquet?’

'Yes!'shoutedAlice.'Come on, then!' roared the Queen, and Alice

joined the procession, wondering verymuchwhatwouldhappennext.

‐‐Alice’sAdventuresinWonderland

3

Backchannels

Speaker A: All right so I have a- a a nail on top Speaker B: okay Speaker A: with an owl in the lower left.

Backchannel:ashortsegmentofspeechuWeredbyalistenertoconveythatsheispayingaWenXon.

4

A.Gravano&J.Hirschberg,2009Backchannel‐invi,ngcuesintask‐orienteddialogue

T1: speech <pause> speech <pause> speech T2: <backchannel>

5

A.Gravano&J.Hirschberg,2009Backchannel‐invi,ngcuesintask‐orienteddialogue

T1: IPU <pause> IPU <pause> IPU T2: <backchannel>

6

Entrainmentonbackchannel‐precedingcues

•  DifferentspeakersdisplaydifferentcombinaXonsofcues.

•  Differentspeakersmayrealizeacueindifferentways.

7

Entrainmentonbackchannel‐precedingcues

•  DifferentspeakersdisplaydifferentcombinaXonsofcues.

•  Differentspeakersmayrealizeacueindifferentways.

•  Dospeakerpairsentrainonbackchannel‐precedingcues?

8

Entrainmentonbackchannel‐precedingcues

•  Threemeasuresofbackchannelcueentrainment:– Measure1:Cuesets– Measure2:CuerealizaXon– Measure3:LocalcuerealizaXon

9

10

TheColumbiaGamesCorpus• Twelvespontaneoustask‐orientedconversaXons• IneachconversaXon,twopeople,separatedbyacurtain,playedcomputergames.• ThirteenpeopleparXcipated;elevenparXcipatedintwosessionseach.• AllfeaturesextractedautomaXcallyinPraat.

Measure1:Similarcuesets

•  Dospeakersusesimilarsetsofcues?

Speaker1RisingintonaXonHigherpitch

LongerduraXon

Speaker2RisingintonaXonHigherintensityLongerduraXon

Changeinvoicequality____________________________________

2cuesincommon

11

Measure1:Similarcuesets

•  Dospeakersusesimilarsetsofcues?

Speaker1RisingintonaXonHigherpitch

LongerduraXon

Speaker2RisingintonaXonHigherintensityLongerduraXon

Changeinvoicequality____________________________________

2cuesincommon

12

Measure1:Similarcuesets

•  Dospeakersusesimilarsetsofcues?

Yes(pairedt‐test,p<0.05)

Speaker1RisingintonaXonHigherpitch

LongerduraXon

Speaker2RisingintonaXonHigherintensityLongerduraXon

Changeinvoicequality____________________________________

2cuesincommon

13

Measure2:CuerealizaXon

•  Dospeakersrealizetheircuesinsimilarways?

Foranyfeaturefmodelingcuec:

thissessionexhibitsentrainmentonc.

14

€

d1,2f < min(d1,*

f , d*,2f )

Measure2:CuerealizaXon

15

Measure2:CuerealizaXon

16

Measure2:CuerealizaXon

17

Measure2:CuerealizaXon

18

Measure2:CuerealizaXon

19

Measure2:CuerealizaXon

20

Measure2:CuerealizaXon

•  Pitch,intensityandvoicequalityshowsignificantlysmallerdifferencesbetweenpartnersthanbetweennon‐partneredpairs.

21

Measure2:CuerealizaXon

•  Pitch,intensityandvoicequalityshowsignificantlysmallerdifferencesbetweenpartnersthanbetweennon‐partneredpairs.

•  Convergence:ThedifferencesbetweenpartnersinintensityandpitcharesmallerinthelasthalfofaconversaXonthanthefirst.

22

Measure3:Localentrainment

•  Doesthewayonespeakerinvitesabackchannelaffecthowherpartnerinviteshisnextbackchannel?

23

IPU backchannel IPU backchannel

IPU backchannel IPU backchannel

IPU backchannel IPU backchannel

Measure3:Localentrainment

•  Doesthewayonespeakerinvitesabackchannelaffecthowherpartnerinviteshisnextbackchannel?

24

IPU

IPU

IPU

Pearson’scorrelaXon

IPU

IPU

IPU

Measure3:Localentrainment

•  Doesthewayonespeakerinvitesabackchannelaffecthowherpartnerinviteshisnextbackchannel?

25

IPU

IPU

IPU

Pearson’scorrelaXon

IPU

IPU

IPU

 SignificantcorrelaXonsformeanpitchandintensity(r=0.3).

Entrainmentonbackchannel‐precedingcues

•  Threemeasuresofbackchannelcueentrainment:– Measure1:Cuesets– Measure2:CuerealizaXon– Measure3:LocalcuerealizaXon

26

Entrainmentonbackchannel‐precedingcues

•  Threemeasuresofbackchannelcueentrainment:– Measure1:Cuesets✓– Measure2:CuerealizaXon– Measure3:LocalcuerealizaXon

27

Entrainmentonbackchannel‐precedingcues

•  Threemeasuresofbackchannelcueentrainment:– Measure1:Cuesets✓– Measure2:CuerealizaXon✓– Measure3:LocalcuerealizaXon

28

Entrainmentonbackchannel‐precedingcues

•  Threemeasuresofbackchannelcueentrainment:– Measure1:Cuesets✓– Measure2:CuerealizaXon✓– Measure3:LocalcuerealizaXon✓

29

AssociaXonwithdialoguesuccess

•  Threemeasuresofdialoguesuccess:– Score– ProporXonofinterrupXonsS1: <unfinished speech> S2: <speech>

– MeanbackchannellatencyS1: <speech> S2: <pause> <backchannel>

30

AssociaXonwithdialoguesuccess

31

AssociaXonwithdialoguesuccess

32

AssociaXonwithdialoguesuccess

33

Conclusions

  Speakersusesimilarsetsofcues.  Theyrealizethecuesinsimilarways.

  Thisentrainmentislocalaswellasglobal.

  ThisentrainmentisposiXvelyassociatedwithtasksuccessandnegaXvelyassociatedwithpoorcoordinaXon.

34

FutureWork

•  Whoentrainstowhom?•  WhatistherelaXonshipbetweendifferenttypesofentrainment?

•  Isthisentrainmentassociatedwithmoreperceptualmeasuresoftasksuccess?

35

Thankyou!