in-car speech user interfaces and their effects on driver cognitive load

In-Car Speech User Interfaces and their Effects on Driver Cognitive Load

Andrew Kun

Students and staff @ UNH

Jon Oppelaar

Oskar PalinkoZeljko Medenica Nemanja Memarovic

Alex Shyrokov Puneet Lakhanpal

Outline

• Ubicomp and cars• Studies:

– Police radio– SUI characteristics– Glove PTT– Navigation

• Conclusion

Navigation

Music, phones

Police cars

• Toys vs. tools• Vanguard of in-car

ubicomp

Project54

Studies

• Goal: – Evaluation tool– Improved HCI

Cognitive load

• Measures:– Performance

• Driving• Visual attention

– Physiological• Pupil diameter• Heart rate• Skin conductance

– Subjective • NASA TLX

Outline

• Ubicomp and cars• Studies:

– Police radio– SUI characteristics– Glove PTT– Navigation

• Conclusion

DriveSafety driving simulator• tinyurl.com/p54sim

Driving performance

• Variance:– Lane position– Steering wheel angle– Velocity

• Collisions

Seeingmachines eye tracker• tinyurl.com/eyetracker

Visual attention

• Fixations (> 100 ms): number, timing, etc.• Percent dwell time: looking at road

Outline

• Ubicomp and cars• Studies:

– Police radio– SUI characteristics– Glove PTT– Navigation

• Conclusion

Police radio study: SUI vs. hardware

Outline

• Ubicomp and cars• Studies:

– Police radio– SUI characteristics– Glove PTT– Navigation

• Conclusion

SUI characteristics study

• Secondary task: speech control of radio• 2 x 2 x 2 design:

– SR accuracy: high/low– PTT button: yes/no – ambient recognition– Dialog repair strategy: mis-/non-understanding

Rec. accuracy influences steering

Recognition AccuracyHighLow

Mea

n va

rianc

e of

ste

erin

g w

heel

ang

le

[deg

2]

500

400

300

200

100

0

Error bars: +/- 1 SD

Low acc. + PTT influences lane position

PTTYesNo

Mea

n la

ne p

ositi

on v

aria

nce

[m2]

0.3

0.2

0.1

0.0

Error bars: +/- 1 SD

Outline

• Ubicomp and cars• Studies:

– Police radio– SUI characteristics– Glove PTT– Navigation

• Conclusion

Glove PTT study• Motivation: PTT vs. driving performance• Secondary task: speech control of radio• 2 x 3 x 3 design:

– SR accuracy: high/low– PTT activation: push-hold-release/push-release/no push– PTT button: ambient/fixed/glove

Push-hold-release Push-release No-push

Ambient Fixed Glove Ambient Fixed Glove Ambient Fixed Glove

High

Low

Glove PTT study

Fixed PTT operation

Fixed PTT – glances!

Glove PTT operation

Glove PTT – few glances

Glove PTT transcription

Hand position on wheel [degrees]

Outline

• Ubicomp and cars• Studies:

– Police radio– SUI characteristics– Glove PTT– Navigation

• Conclusion

Navigation study

• Problems:– In-car devices vs.

driving performance and visual attention– Driving performance/visual attention vs.

probability of accident

Navigation study

• With vs. without map:– Time looking at the

road?– Effect on driving

performance?• Voice instruction

for both• 8 male subjects

Driving simulator

1

2

2

3 3

1

Driving scenario

Visual attention

• Fixations (> 100 ms): number, timing, etc.• Percent dwell time: looking at road

Driving performance

• Variance:– Lane position– Steering wheel angle– Velocity

• Collisions

Effect of visual attention?

• Average variances (13 segments)• Cross-correlation peaks:

PDT decrease → variance increase?

Eye tracker camera Rear camera

Camera setup

Front camera

Standard PND

Voice-only PND

Questions

• Influence on visual attention?• If yes, effect on driving performance?

PDT on the outside world

Series186%

88%

90%

92%

94%

96%

98%

90.4%

96.90%

Standard PND Spoken output only PND

PDT

[%]

PDT at world vs. distance

60-80 80-100 100-120 120-140 140-16080

85

90

95

100standard spoken only

distance from previous intersection [m]

PDT

on o

utsi

de w

orld

[%]

PDT at screen vs. distance

60-80 80-100 100-120 120-140 140-160

-5

0

5

10

15

20

distance from previous intersection [m]

PDT

on s

tand

ard

PND

[%]

Questions

• Influence on visual attention?• If yes, effect on driving performance?

Cross-correlation

0 1 2 3 4 5 6 7 80

0.001

0.002

0.003

0.004

0.005

0.006

0.007Lane position

standard

p = 0.05

spoken only

lag [seconds]

Rlp

[met

ers^

2 ]

0 1 2 3 4 5 6 7 80

5

10

15

20

25

30

35Steering wheel angle

standard

p = 0.05

spoken only

lag [seconds]

Rstw

[deg

rees

^2 ]

Cross-correlation, gazes > 200 ms

0 1 2 3 4 5 6 7 80

0.001

0.002

0.003

0.004

0.005

0.006

0.007

Lane position

standard

p = 0.05

spoken only

lag [seconds]

Rlp

[met

ers^

2 ]

0 1 2 3 4 5 6 7 80

5

10

15

20

25

30

35

Steering wheel angle

lag [seconds]

Rstw

[deg

rees

^2 ]

Fixations at standard PND screen

0.1-0.2 0.2-0.3 0.3-0.4 0.4-0.5 0.5-0.6 0.6-0.7 0.7-0.8 0.8-0.90

10

20

30

40

50

60

70

duration [sec]

num

ber o

f fixa

tions

[#]

Navigation study conclusions

• Display: (visual attention)↘

• (visual attention)↘ → (variances↗)

• (long gazes at PND) → (variances↗)• Display viewing: 60% of gazes last >200ms• Collision risk: need testing• But…

I prefer to have a PND screen…

Strongly agree Agree Undecided Disagree Strongly disagree

0

6

1 1

0

Outline

• Ubicomp and cars• Studies:

– Police radio– SUI characteristics– Glove PTT– Navigation

• Conclusion

Design implications

• SUI may improve performance.• SUI details matter (SR accuracy, PTT).• Must earn users trust.• Complicated displays may reduce visual

attention even more.• Small screen? Voice-only may be just fine.

Next?

• “You’re on the right track.”• Complex environments?• Other possibilities e.g. augmented reality.• Safety?

Acknowledgements

• US DOJ (NIJ, COPS)• NSF• Microsoft Research• TellMe