the traveler's companion

The Traveler's The Traveler's CompanionCompanion

*This image is for illustration only

What is it ?What is it ?Why would I want it ?Why would I want it ?

The traveler’s companion is a simple compact digital camera that can translate foreign languages on signs and posters.

• Park your car without worrying if this is a tow zone.Park your car without worrying if this is a tow zone.

• Find out what that “50% off” sale sign is referring to.Find out what that “50% off” sale sign is referring to.

• Read the names on the street and road signs.Read the names on the street and road signs.

For example:For example:

You find a great parking spot You find a great parking spot downtown, but you see downtown, but you see thisthis sign sign looming right above you... looming right above you...

and you can’t readand you can’t read

Hebrew or Arabic.Hebrew or Arabic.

Would you hesitate ?Would you hesitate ?

Point - Click - Select and Point - Click - Select and readread

With the Traveler’s companion, all you With the Traveler’s companion, all you would need to do is…would need to do is…

• Point the viewfinder at the sign, then Point the viewfinder at the sign, then Click. Click. (just like an ordinary camera)(just like an ordinary camera)

• Select the area of interest.Select the area of interest.

...and then read the translation that appears ...and then read the translation that appears on the back panel screen.on the back panel screen.

This is what you would see:This is what you would see:

BACKBACK

parking place regulated byparking place regulated by

parking meter or parking parking meter or parking cardcard

from 08:00 until 18:00from 08:00 until 18:00

the parking for three hours.the parking for three hours.

SENSITIVITYSENSITIVITY

The User InterfaceThe User Interface

• Image acquisitionImage acquisition - works just like an - works just like an automatic compact camera.automatic compact camera.

• SelectionSelection - the user simply taps the - the user simply taps the area of interest on the screen.area of interest on the screen.

• Result displayResult display - printed in 4 lines on - printed in 4 lines on the LCD screen and has a scroll wheel the LCD screen and has a scroll wheel placed immediately to its right.placed immediately to its right.

A look under the hood- A look under the hood- The hardware components:The hardware components:

• A good autofocus/zoom system A good autofocus/zoom system

• A moderate-speed A moderate-speed (~100 MHz)(~100 MHz) fixed-point DSP fixed-point DSP [e.g.TMS320VC549][e.g.TMS320VC549]

• A cheap color CCD chip (~1 Mpixels)A cheap color CCD chip (~1 Mpixels)

• EPROM (~2MB) containing the built in algorithms.EPROM (~2MB) containing the built in algorithms.

• ROM card (~1MB) containing the dictionary (~5000 words), ROM card (~1MB) containing the dictionary (~5000 words), basic grammar rules and local fonts (~20 alphabets).basic grammar rules and local fonts (~20 alphabets).

• Large touch-LCD display (~4 sqr. inches)Large touch-LCD display (~4 sqr. inches)

• Rugged rain-proof plastic/rubber casing.Rugged rain-proof plastic/rubber casing.

Power consumption:Power consumption:

• DSP - 800 mWDSP - 800 mW

• LCD panel - 250 mWLCD panel - 250 mW Total: 1.2 WTotal: 1.2 W

• CCD - 150 mWCCD - 150 mW

Power available from 2xAA battery pack: 7.2 WHPower available from 2xAA battery pack: 7.2 WH

Battery-life with continuous use: 6 hoursBattery-life with continuous use: 6 hours

Battery-life with anticipated use: over 1000 snapshotsBattery-life with anticipated use: over 1000 snapshots

(assuming ~20 seconds per average snapshot)(assuming ~20 seconds per average snapshot)

The software The software components:components:

1. The border location algorithm 1. The border location algorithm - finds “keystone” areas in the image, given a manually selected sensitivity.

2. Sign parsing2. Sign parsing - divides the user selected area into individual lines, words and letters.

3. OCR/spellchecker3. OCR/spellchecker - performs a simple OCR of the text, with wildcard dictionary lookup.

A look under the hood- A look under the hood- 1. The border location algorithm:1. The border location algorithm:

raise thecontrast

edgedetectio

n

“keystone”filter

#1

#2

A look under the hood- A look under the hood- 2. Sign Parsing:2. Sign Parsing:The run-lengths minimization algorithm:The run-lengths minimization algorithm:

A look under the hood- A look under the hood- 3. OCR/spellchecker :3. OCR/spellchecker :

?קום חניה מוסרר באמצעו??קום חניה מוסרר באמצעו?

Parsing:Parsing:

OCR:OCR:

Spellchecker:Spellchecker: מקום חניה מוסדר באמצעותמקום חניה מוסדר באמצעות

TranslationTranslation*:: parking place parking place regulated byregulated by

* * unrecognized words will be transliterated

DSP operation count breakdown:DSP operation count breakdown:

• Boarder location-Boarder location-

contrast [O(N)] contrast [O(N)] 10^7 ops. 10^7 ops.

edge detection [O(N)] edge detection [O(N)] 10^7 ops. 10^7 ops.

“ “keystone” filter [O(sqrt(N))] keystone” filter [O(sqrt(N))] 10^5 ops. 10^5 ops.

Total: ~0.2 secTotal: ~0.2 sec

• Sign parsing-Sign parsing-

horizontal scanning [O(N)] horizontal scanning [O(N)] 10^7 ops. 10^7 ops.

vertical sub-scanning [O(N)] vertical sub-scanning [O(N)] 10^7 ops. 10^7 ops.

Total: ~0.2 secTotal: ~0.2 sec

DSP operation count breakdown: DSP operation count breakdown:

• OCR/spellchecker-OCR/spellchecker-

OCR: [#alphabets] x [alphabet len.] x [font size] x [offset alignments] =OCR: [#alphabets] x [alphabet len.] x [font size] x [offset alignments] =

= 20 x 26 x 16 x 64 = 5 x 10^5 ops./character= 20 x 26 x 16 x 64 = 5 x 10^5 ops./character

lookup: [#words in sign] x [sign word len.] x [dic. size] x [dic. word lookup: [#words in sign] x [sign word len.] x [dic. size] x [dic. word len.] =len.] =

= 20 x 4 x 5000 x 5 = 2 x 10^6 ops.= 20 x 4 x 5000 x 5 = 2 x 10^6 ops.

spell check: [#lookup ops.] x [error fraction] x [best-guess retries] =spell check: [#lookup ops.] x [error fraction] x [best-guess retries] =

= 10^7 x 0.15 x 6 = 10^7 ops.= 10^7 x 0.15 x 6 = 10^7 ops.

Total: ~0.7 secTotal: ~0.7 sec

Some other examples:Some other examples:

Images with problematic borders:Images with problematic borders:

Non-keystoneNon-keystone

False bordersFalse borders

Multiple overlapping signsMultiple overlapping signs

Images with problematic text:Images with problematic text:

Non-horizontal textNon-horizontal text Hand writingHand writing

Possible implementation Possible implementation approaches:approaches:

• The shortcut:The shortcut: Software within an existing PDA, with a camera added-on.

• Collaboration:Collaboration: Add software to an existing digital camera, using its DSP.

• Independent:Independent: Fabricate the whole product, including dedicated optics.