speechless speech libian_camerafinalproject

Speechless Speech

MIT Media Lab Li Bian | www.chakuli.com

Hey Mary, a bunch of us are going to Legal Seafood tonight.

Would you like to come along?

Jeremy and David are coming too.

We are thinking of leaving at around 9PM.

Audio Visual Speech Recognition

------- a technique that uses image processing capabilities in lip reading to aid speech recognition systems in recognizing undeterministic phones or giving preponderance among near probability decisions.

IBM (ViaVoice)

Affective Computing Group, Media Lab (facial feature -> emocons)

DARPA (Advanced Speech Encoding thru tuned electromagnetic

resonator)

Haar classifier Flash light + Luminance Equ. Compression

Y = 0.2126 R + 0.7152 G + 0.0722 B Luminance (relative):

Haar classifier_frontal face

Haar classifier_mouth

Lossy Compression

BMW car

BMW car

Original transcript:

Hello, hi, my name is Mary, and I am calling from BMW regarding your

car washing services.

Could you please come to the factory to pick up your car this Saturday

afternoon?

Thank you.

BMW car

Person 1 (male, native English speaker)

calling about…card

Person 2 (male, native English speaker)

Hello my name is Mary, calling from BMW, if you can give me a call… this factory..

Person 3 (male, non-native English speaker)

pick up...

BMW car

Person 1 (male, native English speaker)

Hello my name is Mary, I am calling about....factory...pick up the car

Person 2 (male, native English speaker)

Hello my name is Mary, I am calling from BMW...pick up your car Saturday morning...

Person 3 (male, non-native English speaker)

I am doing...to pick up...

BMW car

Person 1 (male, native English speaker)

Hello Mary

Person 2 (male, native English speaker) Mary...calling from BMW...

Person 3 (male, non-native English speaker)

I am ..calling from BMW...call me back....

Dinner Call

Dinner Call

Original transcript:

Hi Lizzie, would you like to go for dinner tonight at 7PM? um, can we

meet at Kendall Square?

Alright, give me a call back if you get this message.

Thank you.

Dinner Call

Person 1 (male, native English speaker)

...Please call back

Person 2 (male, native English speaker)

...call back..

Person 3 (male, non-native English speaker)

...

Dinner Call

Person 1 (male, native English speaker)

Hi, would you like to go...?...Please call back, thanks

Person 2 (male, native English speaker)

give me a call back...

Person 3 (male, non-native English speaker)

Let's meet at...call me back for...

Dinner Call

Person 1 (male, native English speaker)

Hi, would you like to go...?7PM...Please call back. Thank you, bye

Person 2 (male, native English speaker)

calling to ask you about dinner, at 7PM...thank you, bye

Person 3 (male, non-native English speaker)

...Let's meet at...call me soon...

Homework Call

Homework Call

Original transcript:

Hi John, um just calling to see if you have finished the homework # 4

that's due tonight.

If you are still working on it, can we work on it together later tonight?

Alright, give me a call back if you get this message.

Thank you. Bye.

Homework Call

Person 1 (male, native English speaker)

...bye

Person 2 (male, native English speaker)

...bye

Person 3 (male, non-native English speaker)

...

Homework Call

Person 1 (male, native English speaker)

...bye

Person 2 (male, native English speaker)

...thank you, bye

Person 3 (male, non-native English speaker)

Hi…I am from...phone number...bye

Homework Call

Person 1 (male, native English speaker)

Hello John...work on....thank you, bye

Person 2 (male, native English speaker)

Hello John....please give me a call when you get this message...ok, bye

Person 3 (male, non-native English speaker)

Hello John...text me, bye

Legal Seafood

Legal Seafood

Original transcript:

Hey Mary, a bunch of us are going to Legal Seafood tonight.

Would you like to come along? Jeremy and David are coming too.

We are thinking of leaving at around 9PM.

Legal Seafood

Person 1 (male, native English speaker)

...

Person 2 (male, native English speaker)

Mary...terminal

Person 3 (male, non-native English speaker)

Larry....I...

Legal Seafood

Person 1 (male, native English speaker)

...Mary....I'm going to...would you like to come, too?

Person 2 (male, native English speaker)

Mary...coming too...around 9PM...

Person 3 (male, non-native English speaker)

..come along...come too

Legal Seafood

Person 1 (male, native English speaker)

...Would you like to come, too?

Person 2 (male, native English speaker)

we are meeting to go to Legal Seafood.....is coming too, would you like

to come along?

Person 3 (male, non-native English speaker)

...Would you mind to come along...

Visual Audio enhance

Visual Audio

Speech Recognition Computer Vision

enhance

train

Visual Audio enhance

Speech Recognition Computer Vision train

Facial features Text Messages translate

Thank you!