Veggie Vision:A Produce Recognition

SystemR.M. Bolle J.H. Connell N. Haas R.

Mohan G. TaubinIBM T.J. Watson Resarch Center

Veggie Vision:A Produce Recognition

SystemR.M. Bolle J.H. Connell N. Haas R.

Mohan G. TaubinIBM T.J. Watson Resarch Center

Presented by

Chris McClendon

Presented by

Chris McClendon

What is it?What is it?

Veggie vision in an automated produce ID system

Veggie vision in an automated produce ID system

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

HardwareHardware

A scale A polarized light

A camera A PIII 200 MHz A method

A scale A polarized light

A camera A PIII 200 MHz A method

ChallengesChallenges

The segmentation problem Foregroud/background differentiation

Packaging Background variation

The segmentation problem Foregroud/background differentiation

Packaging Background variation

ChallengesChallenges

Color Constancy One element of recognition is based on color profiles.

The lighting in a grocery store is subject to large variation

Color Constancy One element of recognition is based on color profiles.

The lighting in a grocery store is subject to large variation

ChallengesChallenges

Speed of Recognition The system should integrate with the time scale for other checkout operations

The agreed time parameter should be around 1 second

Speed of Recognition The system should integrate with the time scale for other checkout operations

The agreed time parameter should be around 1 second

ChallengesChallenges

Performance Ideally equal to barcode scanning (100%)

Realistic expectations of performance should be at least as good as that of the average checker (~80%)

Performance Ideally equal to barcode scanning (100%)

Realistic expectations of performance should be at least as good as that of the average checker (~80%)

ChallengesChallenges

Ease of Use Integrated into existing barcode reader housing

Minimal operator training

Ease of Use Integrated into existing barcode reader housing

Minimal operator training

ChallengesChallenges

System Training Gradual adaption to variations in season, supplier, and ripeness/freshness

System Training Gradual adaption to variations in season, supplier, and ripeness/freshness

ChallengesChallenges

Database size Seasonal Varies by store, region, and harvest

Database size Seasonal Varies by store, region, and harvest

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

SolutionsSolutions Parallel vs. perpendicular polarization for filtering out glare from the light source

Parallel vs. perpendicular polarization for filtering out glare from the light source

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

SolutionsSolutions

2 images used for segmentation, one light and one dark

Brightness variation greater than the threshold (T∆) are considered foreground

2 images used for segmentation, one light and one dark

Brightness variation greater than the threshold (T∆) are considered foreground

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

SolutionsSolutions

Notice the plastic bag is also illuminated

Another threshold (Tdark) is used to identify the transparent bags

Notice the plastic bag is also illuminated

Another threshold (Tdark) is used to identify the transparent bags

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Question Session #1Question Session #1

Any other illumination-related problems in this scenario?

Any other illumination-related problems in this scenario?

Question Session #1Question Session #1

Any other illumination-related problems in this scenario?

Dark produce

Waxed or shiny produce

Any other illumination-related problems in this scenario?

Dark produce

Waxed or shiny produce

Segmentation ResultsSegmentation Results

Necessary Conditions Stationary

Scale assistance ensures stable produce

Necessary Conditions Stationary

Scale assistance ensures stable produce

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Feature SelectionFeature Selection

Histograms used extensively for feature representation Much smaller than actual images Well researched method for representation of visual cues

Training issues

Histograms used extensively for feature representation Much smaller than actual images Well researched method for representation of visual cues

Training issues

Color FeaturesColor Features

HSI (HSL) color space

HSI (HSL) color space

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Apples vs OrangesApples vs Oranges

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Texture FeaturesTexture Features

More difficult to describe computationally Textel--artichokes or pineapples Random variation--parsley

More difficult to describe computationally Textel--artichokes or pineapples Random variation--parsley

Texture Detection Methods

Texture Detection Methods

Measure A Convolution of crossed bar masks

[ -1 2 -1][-1 -1 2 2 -1 -1]

Measure A Convolution of crossed bar masks

[ -1 2 -1][-1 -1 2 2 -1 -1]

€

Ch (x,y)

Cv (x,y)

M(x,y) = Ch (x,y)2 +Cv (x,y)

2

Texture Detection Methods

Texture Detection Methods

Method B Deviation of image intensity from its nearest neighbors

Performed on reduced images for speed

Method B Deviation of image intensity from its nearest neighbors

Performed on reduced images for speed

Question Session #2Question Session #2

Which method would you use? Which method would you use?

Question Session #2Question Session #2 Results Results

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Classification & Training

Classification & Training

Symbols Pi, i = 1,2,….,N prototype histograms

Q, a produce histogram to be identified

Each histogram has 4 components F = {hue, saturation, intensity, texture}

Normalized

Symbols Pi, i = 1,2,….,N prototype histograms

Q, a produce histogram to be identified

Each histogram has 4 components F = {hue, saturation, intensity, texture}

Normalized

Classification & Training

Classification & Training

Symbols Each prototype histogram Pi is associated with an identifier I(Pi)

How to compute distance Manhattan style

Symbols Each prototype histogram Pi is associated with an identifier I(Pi)

How to compute distance Manhattan style

Classification & Training

Classification & Training

Distance equation Distance equation

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Classification & Training

Classification & Training

Now that the distance is calculated Decision Rule 1

Now that the distance is calculated Decision Rule 1

€

d j < T, j =1...n

€

I(P j ) = I(P1), j = 2...nQuickTime™ and a

TIFF (Uncompressed) decompressorare needed to see this picture.

Classification & Training

Classification & Training

For acceptable threshold but multiple identifiers Decision Rule 2

For acceptable threshold but multiple identifiers Decision Rule 2

€

d j < T, j =1...n

€

I(P j ) ≠ I(P1), j = 2...n

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Classification & Training

Classification & Training

Out of bounds Decision Rule 3

Out of bounds Decision Rule 3

€

d1 > T QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Acting on ResultsActing on Results

Sure Directly accepted by the register

Okay & Uncertain Sorted nearest matches displayed

Sure Directly accepted by the register

Okay & Uncertain Sorted nearest matches displayed

Acting on ResultsActing on Results

If none of the options are chosen, a new prototype for Pchosen is added

The correctly identified Q’s prototype is judged for accuracy

All non-chosen prototypes of the P class are ‘aged’

If none of the options are chosen, a new prototype for Pchosen is added

The correctly identified Q’s prototype is judged for accuracy

All non-chosen prototypes of the P class are ‘aged’

Question Session #3Question Session #3

How do you think it did?How do you think it did?

Overall ResultsOverall Results

Feature Success trial 1 Feature Success trial 1

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Overall ResultsOverall Results

Feature Success trial 2 Feature Success trial 2

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Overall ResultsOverall Results

Training technique analysis Training technique analysis

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Overall ResultsOverall Results

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Super Straigtforward Results

Super Straigtforward Results

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.