Camera Phone Color Appearance UtilityFinding a Way to Identify Color

Phillip Lachman

Robert Prakash

Elston Tochip

Outline

Motivation Goal Methodology

Image Scaling via Edge Detection Color Identification Color Selection & Differentiation

Results Lessons Learned Future Work

Motivation

Phones becoming the portal able digital platform for variety of imaging applications i.e. pictures, video, organizers etc.

Approximately 10 million blind people within the U.S. 55,200 legally blind children 5.5 million elderly individuals

http://www.afb.org/Section.asp?sectionid=15#num

Color blind people within the U.S. ~ 8% of males ~ 0.4% - 2% females

http://www.otal.umd.edu/UUPractice/color/

Goal

To develop a software application that will be able to accomplish the following:1) Receives a camera quality image2) Identifies the predominant color(s)

regions within the image3) Estimates the color name for the

predominant region4) Audibly transmits the predominant color

to the user

Locating the Target

General Guidelines and Suggestions Use a White Card

Provides a white color to baseline lighting conditions Required for computing color of target Suggested by fellow classmates and Bob Dougherty

Detecting the White Card Use an Edge Detection Algorithm

Many Image Processing Edge detection methods available Identify edges by computing changes in gradient around pixels.

Chose Canny Edge Detection algorithm Fundamentally easy to understand and implement

Edge Detection

Finding the Target: Discrimination

How does the algorithm discriminate the target being photographed?

Background clutter and scenery complicate the image Discrimination Solution

The White Card “Scope” Use the rectangular white card with a square target hole to allow

object color through Use Edge Detection image processing algorithms to find the white

card Find the white card, find the target!

Target Hole

Card

Background

Target Color

Finding the Target: Discrimination Cont

The White Card Problem White backgrounds or light color backgrounds cause

edge detection problems in Canny Algorithm

Original Image After Edge Detection

Where is the card??

Finding the Target: Discrimination Cont

The White Card Problem Cont. Adding a Black Outline to the card edges and target

hole greatly improve detection!

Original Image After Edge Detection

There’s the card!!

Finding the Target: Aiming the Camera How does a blind person AIM the camera to take a

picture of the target? Photographs may NOT include target Photographing target too close may not allow enough lighting to

determine color Aiming Solution- White Card Holder

Use a phone attachment which holds the white card AND attaches to phone camera

Guarantees white card and target in the camera Field of View Guarantees camera is not directly on top of the object, providing

ample lighting for color detection

Camera White Card

Mount

Hinge Assembly to allow for folding

Baseboard

Finding the Target: Aiming the Camera

Additional Benefits of Card Holding Device Fixes Orientation of the card

Chose to have card positioned vertically with edges parallel to photo edges

Simplifies algorithm detection, increasing speed

Removes excess background scenery Device maintains a fixed 6-8 inches between camera

and white card Scene is dominated by white card and maximizes number of

pixels covering the target

Finding the Target: Examples with and without device

Phase 1: Blurring and Sharpening Edges Preprocess Images to blur and eliminate noisy pixels Apply a 3x3 Laplacian Matrix Kernel to resulting image

Kernel is an approximation of the second derivative, highlighting changes in intensity

Matrix =

Adding results of Gaussian Blurring and Laplace yields image with cleaner and more distinct lines

Finding the Target: Edge Detection Algorithm

-1 -1 -1

-1 8 -1

-1 -1 -1

Edge Detection: Blurring and Sharpening

Applying Laplace removesNoise and Smoothes lines

Original Image

After Smoothing and Laplace

Finding the Target: Edge Detection Algorithm

Phase 2: Apply Canny Edge Detection Algorithm

Step 1: Applies Gaussian smoothing in 2 dimensions to the image via convolution Size of the mask = 20x20 with a sigma = 5

Step 2: Compute the resulting gradient of the intensities in the image

Step 3: Threshold the norm of the gradient image to isolate edge pixels

Edge Detection: Applying Canny Edge Detection Algorithm

Original Image

After Canny and Threshold

Finding the Target: Edge Detection Algorithm

Phase 3: Finding the edges in the photo Step 1: Recursively search, row by row, from the

outer left/right edges of the image towards the center. Search for 1 quarter length from right/left side

Target Hole

Card

Left Fourth

Finding the Target: Edge Detection Algorithm

Phase 3 Cont: Step 2: Bin and compute outer edges based on

which values are closest to the center Find left/right edges based on bin having AT LEAST 10% of

the total pixels available on the each side

Step 3: Compute Top and Bottom Edge Compute the average row at which both the left and right

edges begin/end Value gives rough estimate of top/bottom of white card

Right Side Left Side

First bin From CenterExceeding 10%

First bin From CenterExceeding 10%

Finding the Target: Target Hole Detection Algorithm

Outer Target White Card edges Located Proceed to Locate INNER Target Hole edges

Phase 4: Identifying Inner Target Hole Edges Step 1: Crop Canny Thresholded image to dimensions

obtained for outer edge Step 2: Perform recursive row by row outside to inside

search until a high threshold is found on both sides. Step 3: Bin and compute left/right edges as before Step 4: Compute Top and Bottom edges as before

Finding the Target: Output to Color Detection

Phase 5: Compute overall Target Hole Position in ORIGINAL image

Sum up inner and outer edge values computed previously Crop the original image to these dimensions and output to

Color Detection model

Color Detection: Original Idea

Keep it simple:

Use brightest point on white card as white point.

Normalize R,G and B separately. Good results, slight reddish tinge.

20 40 60 80 100 120 140 160 180 200 220

10

20

30

40

50

60

70

80

90

100

20 40 60 80 100 120 140 160 180 200 220

10

20

30

40

50

60

70

80

90

100

Color Detection: “Gray World”

Use “Gray world” theory:

Normalize means of RGB to 128

Results slightly better in low lighting conditions, but less effective under good lighting.

20 40 60 80 100 120 140 160 180 200 220

10

20

30

40

50

60

70

80

90

100

20 40 60 80 100 120 140 160 180 200 220

10

20

30

40

50

60

70

80

90

100

Color Identification: Original Idea

Keep it simple: Bin using RGB Problem:

No clear grouping Small changes in one value, changes color

dramatically Solution:

Attempt to identify groups by using max of R, G and B values.

Still contained overlaps

Color Identification: CIE- L*a*b

Convert RGB to CIELab Benefits:

Device independent.

Problems: Conversion formulas complicated and

processor intensive. Light source information is required.

Solution: use HSV

Color Identification: Use HSV Convert RGB to HSV

The HSV (Hue, Saturation, Value) model is a simple transformation from RGB.

Hue, the color type (like red, blue, etc) ranges from 0-360

Saturation, the "vibrancy" of the color: Ranges from 0-100%

Value, the brightness of the color: Ranges from 0-100%

Color Identification: RGB to HSV Equations used for conversion

Color Selection & Differentiation

Currently code identifies 24 colors based on HSV color system.

Color identification is acceptable, but starts to fail in low lighting conditions.

Results

White

Light Green

White

Indian Red

Lessons Learned Edge Detection: Think about the Big

Picture User Feasibility is Critical

If a soldier cannot aim a gun, how accurate is his shot?

Simplicity is Essential Presetting card orientation led to efficiency and

shortcuts for edge detection Slanted Orientation requires much more

processing time and development Original code variants tried to and failed to account

for all orientations

Lessons Learned

Color Detection: HSV is a compromise between simply

binning on RGB values and conversion to L*a*b.

Normalization using the white point more effective than “gray world”.

Minimum level of lighting in required, since camera is low quality.

References http://robotics.eecs.berkeley.edu/~mayi/imgproc/cacode.html

http://homepages.inf.ed.ac.uk/rbf/HIPR2/canny.htm

http://www.aquaphoenix.com/lecture/matlab10/page3.html

http://en.wikipedia.org/wiki/Canny

http://www.afb.org

http://www.otal.umd.edu/UUPractice/color/

Class notes on Color and jpeg tutorials

Future Work Implementing the processing onto an

actual camera phone Decreasing the processing time to audibly

deliver the color to the user Increasing color library Refining overall algorithm to distinguish

more detailed backgrounds. Patches Patterns Color Designs