CSE 803 Fall 2008 Stockman 1

Veggie Vision by IBM

Ideas about a practical systemto make more efficient the selling

and inventory of produce in a grocery store.

CSE 803 Fall 2008 Stockman 2

Problem is recognizing produce

• properly charge customer

• do inventory

• save customer and checker time

CSE 803 Fall 2008 Stockman 3

15+ years of R&D now

This information was shared by IBM researchers. Since that time, the system has been tested in small markets and has been modified according to that experience.

CSE 803 Fall 2008 Stockman 4

Up to 400 produce types

CSE 803 Fall 2008 Stockman 5

CSE 803 Fall 2008 Stockman 6

Practical problems of application environment

CSE 803 Fall 2008 Stockman 7

Engineering the solution

CSE 803 Fall 2008 Stockman 8

System to operate inside the usual checkout station

• together with bar code scanner

• together with scale

• together with accounting

• together with inventory

• together with employee

• within typical store environment

* figure shows system asking for help from the cashier in making final decision on touch screen

CSE 803 Fall 2008 Stockman 9

Modifying the scale

CSE 803 Fall 2008 Stockman 10

Need careful lighting engineering

CSE 803 Fall 2008 Stockman 11

Need to segment product from background, even through plastic

CSE 803 Fall 2008 Stockman 12

Previously published thresholding decision

CSE 803 Fall 2008 Stockman 13

Quality segmented image obtained

CSE 803 Fall 2008 Stockman 14

Design of pattern recognition paradigm (from 1997)

FEATURES are: color, texture, shape, and size all represented uniformly by HISTOGRAMS

Histograms capture statistical properties of regions – any number of regions.

CSE 803 Fall 2008 Stockman 15

Matching procedure Sample product represented by

concatenated histograms: about 400 D 350 produce items x 10 samples = 3500

feature vectors of 400D each Have about 2 seconds to compare an

unknown sample to 3500 stored samples (3500 dot products)

Analyze the k nearest: if closest 2 are from one class, recognize that class (sure)

CSE 803 Fall 2008 Stockman 16

HSI for pixel color: 6 bits for hue, 5 for saturation and intensity

For each pixel

quantify H

HIST[H]++

same for S&I

CSE 803 Fall 2008 Stockman 17

Histograms of 2 limes versus 3 lemons

Distribution or population concept adds robustness:

• to size of objects

• to number of objects

• to small variations of color (texture, shape, size)

CSE 803 Fall 2008 Stockman 18

Texture: histogram results of LOG filter[s] on produce pixels

Leafy produce A

Leafy produce B

CSE 803 Fall 2008 Stockman 19

Shape: histogram of curvature of boundary of produce

CSE 803 Fall 2008 Stockman 20

Banana versus lemon or cucumber versus lime

Large range of curvatures indicates complex object

Small range of curvatures indicates roundish object

CSE 803 Fall 2008 Stockman 21

Size is also represented by a histogram

CSE 803 Fall 2008 Stockman 22

Each pixel gets a “size” as the minimum distance to boundary

Purple grapes Chinese eggplants

CSE 803 Fall 2008 Stockman 23

Learning and adaptation System “easy” to train: show it

produce samples and tell it the labels. During service: age out oldest

sample; replace last used sample with newly identified one.

When multiple labeled samples match the unknown, system asks cashier to select from the possible choices.

CSE 803 Fall 2008 Stockman 24

Where is Veggie Vision today?

http://www.internetnews.com/xSP/article.php/3642386

System uses almost all color features Installed in few places: many stores have

self-checkout, putting work on customer. IBM has a “shopping research” unit

http://www.usatoday.com/tech/news/techinnovations/2003-09-26-future-grocery-shop_x.htm

Customers will tolerate a higher human error than a machine error