1

HumanHuman--Computer InteractionComputer Interactionin a Smart Home Environmentin a Smart Home Environment

Klaus FellbaumKlaus FellbaumMaik HampickeMaik Hampicke

Communication EngineeringBrandenburg University of Technology, Cottbus

2

Content

• Smart Home

• Learning Home

• User Interfaces for Smart Homes

• Speech and Speaker Recognition

• Investigations and Results

• Summary and Conclusions

3

Smart Home

4

Smart Home Definition

Smart Home and Smart Home Environment

– Home (including the narrow environment) withsmart technology and services.

Aim...

Integrating, automating and improving

– safety & security,

– communication,

– comfort and,

– energy saving.

Prerequisites:

– Networking,

– adaptation to the user’s needs

5

Smart Home Definition

6

Smart Home Example

7

Learning Home I

Learning Home: Simulation environment for the Sentha project

Aim

- Realisation,- demonstration,- test and evaluation of smart home

functions,- test bed for elderly persons.

Components

- Network,- control tools, sensors and actuators- user interfaces,- speech processing tools

(speech and speaker recognition).

8

Sentha’s Learning Home II

room 2

door

1

door

2

window 2

lamp 1

lamp 3

speechcontrolled dooropener

dawn switch

valve controller

doorcontact

lamp 2

doorcontact

windowcontact

firedetector

room temperaturecontroller

push buttonlamp 3 push button

lamp 2

push buttonlamp 1

win

dow

1

windowmotor

windowcontact

push buttonwindow 1

motiondetector

distributor

wall socket forhousehold device

scenario elementlogic component

wall socketlamp 3

display /user interface

time switch

room 1

9

User Interfaces for Smart Homes

user interfaces

single remotecontrol device

networked andprogramed switches

touch pad,tablets, PDA’s

TV set withremote control

personaland portable

computer

stationaryand mobiletelephon

recognition systems

smartclothes

mixedsystems

- speech, speaker- gestures,- ...

10

Speech Recognition and Speech Interface

Human(information processing,

and control)

Computer (decisionsmade on the basis of input

information and internalprograms)

Hum

an-C

ompu

ter-

Inte

rfac

e

Speech/Speaker

Recognition

Input processing,movement

interpretation(mouse)

Speech

Hearing Speech SpeechSynthesis

Operate, input,mouse moving

Informationoutput

(display,avatar,...)

receivesinformation with

sens organs

- speech dialogstructure,

- user discipline,- error handling,- feedback control,- background noise,

non-speech effects,- system activation,- speech vocabulary,- training period,- hardware design.

Influence parametersInfluence parameters forforspeech recognition systemsspeech recognition systems

11

Speech Recognition (Applications)

Voice Control of…

• telephone (acoustic dialing)

• tv and hifi devices

• alarm systems

• household appliances

• environment control systems (light, heating, airconditioner,...)

• door/window opener

• robots

12

Speech Recognition (State of the art)

Isolated word recognition(speaker adaptive)

• many systems on the market

• very different recognition accuracy (70...95%)

• mostly as add-on to PCs (software solution), very fewstand-alone systems

• low price (range $10...$100)

• (mostly) serious adaptation/integration problems

• (mostly) awkward training procedures

• noise sensitivity

13

Speech Recognition (State of the art)

Continuous speech recognition

• few systems on the market

• very different recognition accuracy (60...80%)

• mostly as add-on to PCs (software solution), very fewstand-alone systems

• expensive (> $100)

• (mostly) serious adaptation / integration problems

• (mostly) awkward training procedures

• noise sensitivity...need for smart home applications ???

14

Speaker Recognition

15

Principle of speaker recognition

Speech-sample

Feature VectorFV

ClaimedIdentity

FVMemory

Decision

Speech-Output

Pre-processing

FVComparison

Desired Transaction

Answer

Acceptor

Reject

16

start of thedoor opener procedure

pushes the intercombutton

three times

guest resident

push the intercombutton once(standard)

standard doorbellrings

speech dialogactivation

resident decides

to openthe door

toremain

the doorclosed

Pleasespeak!

User speaks keysentence:

„..xxxxxx..“

systemconfirms

systemdemands

repetition

systemrefuses

decision based onspeaker recognition

openthe door

automatical

enddialog

user can start a newdoor opener procedure

activation ofthe

training phase

speech dialogactivation

Pleasespeak!

trainingphase

end of training

burglar

differentstrategies

to fool the speakerrecognition system

Userspeakstraining

sentence(s)

system actionsuser actions

Speech Dialog:Speech Dialog:

Example:Example:oopenpeningingofof thethemain doormain door

17

Investigations and Results

18

Evaluation of young and elderly persons about pro’s andcont’s of a speech recognition system

Results:

- Speech recognition (voice control) is the top solution, if …- the recognition accuracy is

high enough (>95 %)- the user has a certain experience- it is used for simple command

and control functions

- For control of more complex functions a mixed interface(graphic, audio,…) is preferable à better for feedback and errorhandling.

Investigations and Results I

19

Investigations and Results II

- System training was too long (50 sentences were necessary)à decrease of concentration and speaking discipline.

- User became impatient, made comments and turned their faceaway from the microphone.

- A little insight into the mechanism of speech recognitionimproved the results.

- An acoustic or visual signal as request to start speaking wasfound to be very useful.

20

Summary and Conclusions

21

Summary and Conclusions

- Speech components have been proved as very important in a smarthome environment.

- Concerning the successful use of speech and speaker recognition,there was no significant difference between younger and olderusers, but experience and training played a key role.

- A careful design and integration of the speech interface is of crucialimportance.

- Speech components have to be carefully embedded into a generalhuman computer interaction concept. The key problem is theoptimal mixture of acoustic and visual information in both, the inputand the output interface. The mixture differs individually and mustbe adapted individually.

- Human computer interaction must not restrict the personal freedomto do things ‘by hand’, the feeling to be dependend on a technicalsystem has to be avoided.