Eyes-Free Barcode Localization and Decoding for Visually ImpairedMobile Phone Users

Vladimir Kulyukin Aliasgar Kutiyanawala∗

Computer Science Assistive Technology LaboratoryDepartment of Computer Science

Utah State UniversityLogan, UT, 84322

vladimir.kulyukin@aggiemail.usu.edu, aliasgar.k@aggiemail.usu.edu

AbstractAn eyes-free barcode localization and decodingmethod is presented that enables visually im-paired (VI) mobile phone users to decode MSI(Modified Plessy) barcodes on shelves and UPCbarcodes on individual boxes, cans, and bottles.Simple and efficient barcode localization and de-coding techniques are augmented with an interac-tive haptic feedback loop that allows the VI userto align the phone’s camera with a fixed surfacein the pitch and yaw planes. The method is im-plemented on a Google Nexus One smart phonerunning Android 2.1. A laboratory study is pre-sented in which the method was evaluated by oneVI and four blindfolded sighted participants.

Keywords: Accessible Shopping, Eyes-Free Barcode Lo-calization & Decoding, Assistive Technology 1

1 IntroductionSupermarkets are one of the most functionally challengingenvironments for VI individuals. A modern supermarkethas a median area of 4,529 square meters and stocks an av-erage of 45,000 products [1]. VI people typically rely onfamily members or friends who take them grocery shop-ping. When these individuals are unavailable, VI shoppersreschedule their shopping trips or rely on store staffers as-signed to them when they come to the store. Some staffersare unfamiliar with the store layout, others become irritatedwith long searches, and still others do not have adequate

∗Contact Author1Submitted to IPCV 2010

English skills to read the products’ ingredients [2]. Thesedifficulties cause VI shoppers to abandon searching for de-sirable products or settle for distant substitutes. PeaPod orsimilar home delivery services provide grocery shopping al-ternatives. However, such services are not universally avail-able and, when available, require shoppers to schedule andwait for deliveries, thereby reducing personal independenceand making spontaneous shopping impossible. To over-come these access barriers, accessible shopping systems areneeded that increase personal independence, enable sponta-neous shopping, and do not require that supermarkets un-dergo extensive technological adjustments. Such systemswill make VI individuals independent of external assistanceand fundamentally improve their quality of life.

In 2006, we began our work on ShopTalk [3], a wear-able system for independent blind supermarket shopping.ShopTalk consists of a small computer device (OQO model01), a Belkin numeric keypad, a Hand Held ProductsIT4600 SR wireless barcode scanner and its base station,and a USB hub that connects all components. To help carrythe equipment, the user wears a small CamelBak backpack.The numeric keypad is attached by velcro to one of thebackpack’s shoulder straps. To ensure adequate air circula-tion, the OQO is placed in a wire frame attached to the out-side of the backpack. The remaining components - the bar-code scanner’s base station, the USB hub, and all connect-ing cables - are placed inside the backpack’s pouch. Sincethe system gives speech instructions to the user, the user hasa small headphone. The system was our first attempt (andthe first attempt reported in the accessible shopping litera-ture) to use shelf barcodes (MSI, not UPC) as topologicalpoints for locating products through verbal directions.

In 2007 - 2008, after two single subject studies at Lee’s Mar-ket Place, an independently owned supermarket in Logan,

UT, and one single subject study at Sweet Peas, a small in-dependent natural foods store in Logan, UT, ten VI partic-ipants were recruited for a longitudinal formal study. Theproduct database was extended to include 4,297 products.The experiment, performed during regular business hours,had each participant shop for the same set of three randomlychosen products five times. The product retrieval rate was100%. All ten participants found all three products in everyrun (see [2] for detailed ANOVA analysis and experimentdesign). A key finding of ShopTalk is that all participants inthe sample could execute verbal route directions in super-markets and product search instructions triggered by bar-code scans with 100% accuracy.

In 2008 - 2009, ShopTalk was ported onto the Nokia E70smartphone running Symbian OS 9.1. The smartphone isconnected to a small Bluetooth barcode scanner (BaracodaPencil2) [4]. The port, called ShopMobile, was tested ina laboratory study with six sighted blindfolded participants.The experiment had eighty shelf barcodes obtained from theUSU bookstore placed on shelves assembled into 3 aisleswhere aisles 1 and 3 had only one side while aisle 2 hadtwo sides. Each participant successfully retrieved a set offour products.

We are currently developing ShopMobile II, the nextversion of ShopMobile. Like its direct predecessors,ShopTalk [5] and ShopMobile [4], ShopMobile II uses twodata structures to represent the shopping environment. Thefirst data structure is a topological map, which is a di-rected graph whose nodes are decision points: the storeentrance, aisle entrances, and cashier lane entrances. Theedges are labeled with directions. Due to the regularity ofmodern supermarkets, we found it sufficient to have threedirectional labels: left, right, and forward. The topologi-cal map is built at installation time by walking through thestore, noting decision points of interest, and then represent-ing them in the map. The second data structure, called theBarcode Connectivity Matrix (BCM), is designed to takeadvantage of the inventory control systems used by manygrocery stores. These inventory systems place barcodes onthe shelves immediately beneath every product type. Shelfbarcodes assist the store personnel in managing the productinventory. Once the locations of all shelf barcodes in thestore are known, this information is used to guide the shop-per through to the target shelf section, because products arelocated directly above their corresponding shelf barcodes.More information on how the BCM can be computed fromthe inventory control databases is available in [2].

Unlike its predecessors, ShopMobile II no longer requires abarcode scanner, because it relies exclusively on computer

vision techniques to recognize MSI barcodes on shelves andUPC barcodes on individual products. Vision-based bar-code localization and decoding is a well-known researchproblem. Algorithms have been proposed that look forline sets with mono-oriented gradients to locate the bar-code region [6, 7]. Other algorithms extract connectedregions with mono-oriented texture to perform local bar-code searches in cluttered 3-D scenes [8, 9, 10]. Houghtransformation-based methods extract barcode lines by as-suming robustness against orientation and size [11]. Var-ious morphological filters and self-learning networks havealso been tried [12]. One common disadvantage of these al-gorithms is that they are very time-consuming and requireexternal servers for image processing. Recently several bar-code decoding solutions for camera phones have been pro-posed [13, 14]. However, these solutions target sightedusers in that they require that the phone camera is carefullyaligned and centered on a barcode for the decoding to work.An accessible barcode localization and decoding method isproposed in [15]. However, unlike the method presented inthis paper, it has not been implemented on a mobile phoneyet. We wholeheartedly endorse and commend these R&Defforts. Our approach is based on the hypothesis that so-phisticated vision techniques may not be needed to makebarcode localization and decoding accessible to VI mobilephone users, because simple and efficient vision techniquescan be augmented with interactive user interfaces that en-sure that images have certain properties.

In this paper, we present an interactive haptic feedback loopthat allows the VI mobile phone user to align the camerawith a fixed surface in the pitch and yaw planes, whichguarantees that the captured image is taken from the mobilephone parallel to the surface. The remainder of our paper isorganized as follows. In Section 2, we describe our barcodelocalization and decoding method. In Section 3, we presentan interactive haptic camera alignment loop that enables theVI mobile phone user to align the phone’s camera with fixedsurfaces in the pitch and yaw planes. In Section 4, we givebrief descriptions of the target user and the target use casetoward which we are working. We also present a laboratorystudy with one VI individual and four sighted blindfoldedparticipants. In Section 5, we offer our conclusions.

2 Eyes-Free Barcode Localizationand Decoding

A barcode can be viewed as a homogeneous region consist-ing of alternate black and white lines condensed in a smallregion. In ShopMobile II, barcodes are characterized as

2

image regions that have the following two properties. Wecall the first property alternating frequency and define it asthe number of black to white and white to black transitionsalong the x-axis. We call the second property vertical con-tinuity and define it as the continuity of black and whitelines along the y-axis. As an example, consider two one-pixel wide lines A and B placed on the barcode as shownin the left part of Figure 1. These lines can be encoded asbit strings where black pixels map to zeros and white pixelsmap to ones. The alternating frequency is the number of0-1 and 1-0 transitions in the bit string. Vertical continuityis measured as the longest common subsequence of the twobit strings. Other string matching techniques can certainlybe used provided that there is no impact on the overall effi-ciency.

Figure 1: One Pixel Wide Lines on the Barcode.

Figure 2: Line Detection Filter Kernel.

For efficiency, the 1024 by 768 image is reduced to 320by 240 pixels. Each pixel, a packed integer containingthe pixel’s alpha and RGB components, is mapped to agrayscale value Y between 0 and 255, where Y = 0.3 ×R + 0.59×G+ 0.11×B. Since the Google Nexus One’scamera has a provision to take a greyscale image, the abovestep reduced to setting Y = G for efficiency. In our previ-ous work [16], we used a Gabor filter to localize barcodes

in images. In the current implementation, the Gabor filter isreplaced with a much more efficient line detection filter thatconvolutes the kernel shown in Figure 2 through the entireimage. This kernel is inspired by the line vertical detectionkernel based on a 3 × 3 matrix discussed in [17]. We ex-perimentally extended the kernel to a 13 × 3 matrix withdecreasing coefficients along the y-axis to make it less sen-sitive to the noise in the image. Given an m×n matrix, ourfilter can be generated as follows:

f [i][j] =

{(m+1

2 − |i|)× n2−14 if j = 0

−(m+12 − |i|)× (n+1

2 − |j|) if j 6= 0,

where −m/2 ≤ i ≤ m/2 and −n/2 ≤ j ≤ n/2.

Since barcode lines pass through this filter along with someother vertical lines generated by graphics and text that maybe present in the image, the filtered image is searched in arasterized pattern with two one pixel wide lines in order toisolate areas with high alternating frequency and high ver-tical continuity. Fast histogram analysis is performed onall pairs of lines with sufficiently high coefficients for thealternating frequency vertical continuity. After a small setof candidate regions is obtained, each candidate region ismapped to the original image, because, while the reducedresolution is sufficient for fast and adequate barcode local-ization, it turned out not to be adequate for ZXing [13],which we used for barcode decoding. ZXing is an opensource library for decoding several 1D and 2D barcodetypes, including UPC, but does not support the MSI bar-code type. We extended ZXing to decode MSI barcodes andextended the API to accept a set of barcode regions for de-coding. Thus, the localized candidate regions are mappedback to the original image scale (1024 by 768 pixels) andcropped from it so that ZXing can process them. Futurereferences to ZXing in the paper will refer to this extendedversion of ZXing.

We designed an experiment to measure the contribution ofour barcode localization algorithm to ZXing. A total of 187images of real products were captured with a HTC TouchPro smartphone. In the first run, ZXing alone decoded 91images. In the second run, the images were first processedwith our barcode localization algorithm and then ZXing ranon the cropped candidate regions. This time barcodes werecorrectly decoded in 133 images, which amounts to an in-crease of 42 images or 46.15% over the number of imagesdecoded by ZXing alone. The software for this experimentwas implemented using Java SE and Netbeans and ran on aDell Optiplex 960 Core2Duo machine running at 3GHz.

3

3 Interactive Camera Alignment

The shopper always takes a picture of a fixed surface afterthe phone’s camera is aligned with the surface in the pitchand yaw planes. The alignment is carried out through an it-erative haptic control loop. If the product is a box, the useris instructed to find the top of the box, i.e. the side withan opening tab, by touch and then align the camera withthe bottom, i.e. the opposite side. This is because on mostboxes sold in the U.S. the UPC barcode is on the bottom.Once the bottom is found, the shopper aligns the camerawith an edge of the side and clicks a button on the touchscreen to start the alignment loop. The current readings ofthe phone’s orientation sensor are taken to define the abso-lute pitch and yaw planes. The shopper then slowly movesthe camera away from the surface. If the camera is mis-aligned in the pitch or yaw planes, specific haptic signals(vibrations) are issued to re-align it. The signals persist untilthe camera is aligned. The user is instructed to stop movingthe camera away from the surface when she thinks that thecamera is approximately 10cm from the surface. Then thepicture is taken by a click of a button on the touch screen. Ifthe product is a can, the shopper aligns the camera with theedge of the top or bottom circle. If it is a bottle, the camerais aligned with the edge of the bottom circle.

When the shopper reaches an aisle, she aligns the phonecamera with a shelf and captures an image. If a barcode isdecoded successfully and happens to be a shelf barcode, thesystem checks if it is the target barcode. If it is, the systemverbally informs the shopper about it. If it is not, the systemgenerates verbal directions to the target product (e.g. “Twoshelves up, three barcodes left.”). If the barcode cannot bedecoded in the image, the system divides the image intosub-images that are likely to contain barcodes. Each sub-image is processed with a barcode decoder. If the barcode isdecoded, the system goes into the verbal instruction mode.If the barcode is not decoded, the system assumes that thebarcode is rotated by 90 degrees. This assumption is en-forced by the interactive haptic alignment loop. The imageis rotated by 90 degrees, and the entire processes is repeatedagain. If part of a barcode is detected, the system gives theuser haptic feedback to move the camera slightly left, right,up, or down, depending on where the partial barcode is de-tected. When the system is unable to find the barcode in theimage, it notifies the user through audio feedback (currentlya ping) to take another image. The overall flow of controlof the barcode localization and decoding method is given inFigure 3. If necessary, user feedback messages can be de-livered in a variety of formats: haptic, audio, speech, or acombination thereof.

Downscale image

Apply line detection filter

Rasterized search

Extract barcode from original image

a b

cd

e

Figure 4: Accessible Barcode Localization and DecodingMethod.

4 A Laboratory StudyBefore we describe our laboratory studies, we will brieflydescribe the target user and the target use case toward whichwe are working. We will call our target user Alice (namechanged to protect identity) who was a participant in one ofour longitudinal accessible shopping experiments in Lee’sMarket Place. She is in her mid-twenties, a mother oftwo children, and a seasoned mobile phone user. Both sheand her husband are completely blind and cannot help eachother with grocery shopping. Alice has excellent orientationand mobility (O&M) skills, takes independent walks aroundher neighborhood, uses the Cache Valley transit system, andhas a full-time job. Alice is a resident of Logan, Utah, andgoes to Lee’s Market Place several times a week. ShopMo-bile II is a mobile solution for individuals like Alice.

We are working toward the following use case. When Al-ice enters the supermarket, she picks up a shopping basket.Alice also has adequate cane skills to pull a shopping cartif she has to. Alice takes out her mobile phone and, if hershopping list is prepared, selects a product by menu brows-ing. The system tells Alice where the product is located,and if requested, generates a template-based route descrip-tion. When Alice reaches the aisle, she aligns her phonecamera with a shelf and takes a picture. If the camera is notvertically aligned to the shelf, the system uses the phone’sorientation sensor to give Alice haptic (vibratory) feedbackto align the camera. Alice then takes an image. If part of abarcode is detected, another vibratory signal requests Alice

4

Get imagefrom cameraDecode barcode Barcode decoded?

Barcode decoded? Decode barcode

LocalizebarcodeDecode barcode Localizebarcode

Rotateimage

Barcode decoded?Is it a shelfbarcode?Is it a UPCbarcode?

Ask user totry againAsk user toverify product

Barcodecorrespondsto target product?Barcodecorrespondsto target product?

Confirm productto the userNotify user thatselected product isnot target product Provide directionsto target product

No No

NoNo

No

No

Yes

Yes Yes

Yes

Yes

Yes

Ask user totry againNo

Yes

Figure 3: Barcode Localization and Decoding.

to move the phone to the left (right). When the shelf barcodeis recognized, the system uses its database to give Alice ver-bal directions to the target product, e.g. two shelves up, fivebarcodes right or this is aisle 5, target product is in aisle 2.When the target shelf barcode is recognized, Alice reachesabove the barcode and takes a product from the shelf. If theproduct’s identity cannot be determined manually, e.g. bytouch, smell, shake, etc., they system allows Alice to inter-actively recognize its UPC barcode to verify the product’sidentity.

We approximated the target use case with a laboratorystudy. We ported all software to a Google Nexus One smartphone equipped with a five megapixel camera. The phoneruns on Android 2.1 on a 1 GHz processor with 512 MBof RAM. A grocery store environment was simulated byassembling a shelf section in our laboratory that consistedof five shelves. Twelve empty boxes of real products wereplaced on three shelves.

We recruited one VI individual and four sighted individu-als to test the system. The sighted individuals were blind-

Participant 1 Participant 2 Participant 3 Participant 4 Participant 50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Finding a Product Verifying a Product

Number of Sca

ns

Figure 5: Mean Values for Number of Scans to Retrieve andVerify Products for Each Participant.

folded. Each participant was given a ten minute trainingsession in which the system was shown to him/her and thenasked to retrieve and verify ten randomly selected products.For each participant, the system logged the image associ-ated with each scan, the result of the scan and the method

5

Participant 1 Participant 2 Participant 3 Participant 4 Participant 50

0.5

1

1.5

2

2.5

3

3.5

Finding a Product Verifying a ProductNumbe

r of Scans

Figure 6: Median Values for Number of Scans to Retrieveand Verify Products for Each Participant.

by which the barcode was decoded.

All participants were able to retrieve all products success-fully. Figure 5 shows the average number of scans for prod-uct retrieval and verification. It took an average of 2.72scans for a participant to retrieve a product and an averageof 3.32 scans to verify a product. The STD values are 4.4for product retrieval and 5.6 for product verification. Themedian values for product retrieval and product verificationare shown in Figure 6.

In general, the median values for product retrieval are lowerthan the median values for product verification. For partic-ipant 5 (a sighted individual), the median value for productretrieval is 0. The system is designed so that that the shop-per has the option of picking up a product from the shelfwithout having to scan the MSI shelf barcode below it, be-cause some shoppers can locate the target product by count-ing shelf barcodes by touch.

Figure 7 shows a distribution of the result of the scans forall participants. ZXing alone decoded about 13% of all bar-code scans. When used in conjunction with our barcodelocalization algorithm it decoded another 16% of all bar-code scans. Rotating the image before localizing and de-coding the barcode results in about 4% more barcode scansto be decoded. Barcode localization causes the number ofbarcodes decoded to increase from 38 to 97, an increase ofover 155%.

In the informal post-experiment interviews, all participantssaid that our touch screen user interface was very simpleto use but too basic. The interface consisted of a singlebutton encompassing the entire touch screen of the GoogleNexus One phone. The barcode decoding procedure wasinitiated either by finger tapping on any part of the screen

13%16%

4% 68%

ZXingLocalizationRotation and LocalizationNot Decoded

Figure 7: Results of Barcode Scans.

or by tapping on the phone’s trackball, a small joystick-likehardware component below the touch screen. All partici-pants preferred tapping the screen to tapping the trackball,which resulted in many accidental images with no barcodespresent in them, which negatively affected the barcode lo-calization and decoding performance. In our future work,we will design a more sophisticated approach (e.g. fingergestures) to eliminate accidental images.

5 ConclusionWe presented an eyes-free vision-based barcode localiza-tion and decoding method that enables visually impaired(VI) mobile phone users to decode MSI (Modified Plessy)barcodes on shelves and UPC barcodes on individual boxes,cans, and bottles. We showed how simple and efficient bar-code localization and decoding techniques were augmentedwith an interactive haptic feedback loop that allows the VIuser to align the phone’s camera with a fixed surface in thepitch and yaw planes. Our method is implemented on aGoogle Nexus One smart phone running Android 2.1. Wedescribed a laboratory study in which the method was eval-uated by one VI and four blindfolded sighted participants.All participants were able to retrieve and verify all the prod-ucts successfully. Our touch screen user interface should beimproved to eliminate accidental images. Experiments arebeing planned with VI individuals in a real grocery store.

AcknowledgmentsThis research has been supported, in part, through NSFgrant IIS-0346880.

6

References

[1] Food Marketing Institute Research. The Food RetailingIndustry Speaks 2006. Food Marketing Institute 2006.

[2] Nicholson J., Kulyukin V., Coster D. ShopTalk: Inde-pendent Blind Shopping Through Verbal Route Direc-tions and Barcode Scans The Open Rehabilitation Jour-nal, ISSN: 1874-9437 Volume 2, 2009, pp. 11-23, DOI10.2174/1874943700902010011.

[3] Nicholson J. and Kulyukin V. ShopTalk: IndependentBlind Shopping = Verbal Route Directions + BarcodeScans. Proceedings of the 30-th Annual Conference ofthe Rehabilitation Engineering and Assistive Technol-ogy Society of North America (RESNA 2007), June2007, Phoenix, Arizona, Avail. on-line and on CD-ROM.

[4] Janaswami, K. (2010). ShopMobile: A Mobile Shop-ping Aid for Visually Impaired Individuals. M.S. Re-port, Department of Computer Science, Utah State Uni-versity, Logan, UT, to appear.

[5] Kulyukin V., Nicholson J., and Coster D. ShopTalk: To-ward Independent Shopping by People with Visual Im-pairments. Proceedings of the 8th ACM Conference onComputers and Accessiblity (ASSETS 2008), Halifax,Canada.

[6] Normand N. and Viard-Gaudin C. A Two-DimensionalBar Code Reader. Pattern Recognition, Vol. 3, pp. 201-203, 1994.

[7] Viard-Gaudin C., Normand N., and Barba D. AlgorithmUsing a Two-Dimensional Approach. In Proceedingsof the Second Int. Conf. on Document Analysis andRecognition, No. 20-22, pp. 45-48, October 1993.

[8] Ando S., and Hontanj H., Automatic Visual Searchingand Reading of Barcodes in 3-D Scene, In Proceedingsof the IEEE Int. Conf. on Vehicle Electronics. No. 25-28, pp. 49-54, September 2001.

[9] Ando S. Image Field Categorization and Edge/CornerDetection from Gradient Covariance. IEEE Transac-tions on Pattern Analysis and Machine Intelligence,Vol. 22, No. 2, pp. 179-190, February 2000.

[10] Ando S., Consistent Gradient Operators. IEEE Trans-actions on Pattern Analysis and Machine Intelligence,Vol. 22, No. 3, pp. 252-265, March 2000.

[11] Muniz R., Junco L., and Otero A., A Robust SoftwareBarcode Reader using the Hough Transform., In Pro-ceedings of International Conference on InformationIntelligence and Systems, No. 31, pp. 313-319, Novem-ber 1999.

[12] Arnould S., Awcock G. J., and Thomas R., RemoteBar-code Localization Using Mathematical Morphol-ogy., Image Processing and its Applications, Vol. 2, No.465, pp. 642-646, 1999.

[13] The Zebra Crossing Barcode Decoding Libraryhttp://code.google.com/p/zxing/

[14] Occipital, LLC. RedLaser http://redlaser.com/

[15] Tekin, E. and Coughlan, J.M., An algorithm enablingblind users to find and read barcodes WACV09, 2009

[16] Kutiyanawala A., Qi X., Tian J. A Simple and Effi-cient Approach to Barcode Localization. 7th Interna-tional Conference on Information, Communication andSignal Processing, Macau 2009

[17] Fisher R., Perkins S., WalkerA. and Wolfart E. Line Detection.http://homepages.inf.ed.ac.uk/rbf/HIPR2/linedet.htm

7