© devi parikh 2008 localization and segmentation of 2d high capacity color barcodes gavin jancke...
TRANSCRIPT
![Page 1: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/1.jpg)
© Devi Parikh 2008
Localization and Segmentation of
2D High Capacity Color Barcodes
Gavin Jancke
Microsoft Research, Redmond
Devi Parikh
Carnegie Mellon University
![Page 2: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/2.jpg)
© Devi Parikh 2008
Motivation
UPC Barcode
QR Code Datamatrix
![Page 3: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/3.jpg)
© Devi Parikh 2008
HCCB
Microsoft’s High Capacity Color Barcode
![Page 4: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/4.jpg)
© Devi Parikh 2008
Application Uniquely identifying commercial audiovisual
works such as motion pictures, video games, broadcasts, digital video recordings and other media
![Page 5: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/5.jpg)
© Devi Parikh 2008
GoalLocate and Segment the barcode from consumer
images
![Page 6: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/6.jpg)
© Devi Parikh 2008
Overview
Design specifications of Microsoft’s HCCB
Approach
Localization
Segmentation
Progressive Strategy
Results
Conclusions
![Page 7: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/7.jpg)
© Devi Parikh 2008
Microsoft’s HCCB
4 or 8 colors
Triangles
String of colors
palette
![Page 8: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/8.jpg)
© Devi Parikh 2008
Microsoft’s HCCB
![Page 9: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/9.jpg)
© Devi Parikh 2008
Microsoft’s HCCB
![Page 10: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/10.jpg)
© Devi Parikh 2008
Microsoft’s HCCB
![Page 11: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/11.jpg)
© Devi Parikh 2008
Microsoft’s HCCB
R rows
S symbols per rowS = (r+1)*R
Aspect ratio: r
![Page 12: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/12.jpg)
© Devi Parikh 2008
Approach
Thresholding
Orientation prediction
Corner localization
Row localization
Symbol localization
Color assignments
Barcode localization
Barcode segmentation
point inside the barcode is known
![Page 13: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/13.jpg)
© Devi Parikh 2008
Localization: Thresholding
Identify thick white band and row separators
Normalization
Adaptive
![Page 14: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/14.jpg)
© Devi Parikh 2008
Localization: Orientation
orientation orientation
dist
ance
-90 900sum
mat
ion
![Page 15: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/15.jpg)
© Devi Parikh 2008
Localization: Corners Rough estimates
whiteness mask non-texture mask combined mask
![Page 16: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/16.jpg)
© Devi Parikh 2008
Localization: Corners Gradient based refinement
![Page 17: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/17.jpg)
© Devi Parikh 2008
Localization: Corners Line based refinement
![Page 18: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/18.jpg)
© Devi Parikh 2008
Segmentation: Rows
Summation
Flip?
![Page 19: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/19.jpg)
© Devi Parikh 2008
Segmentation: Symbols
S E
Local search
Number of symbols per row
q(S,E) = q(samples|S,E)
![Page 20: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/20.jpg)
© Devi Parikh 2008
Segmentation: Colors
Palette
![Page 21: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/21.jpg)
© Devi Parikh 2008
Segmentation results given accurate localization Satisfactory
Corner localization Unsatisfactory
No one strategy works well on all images However (1) Errors of different strategies are
complementary (2) Results are verifiable with decoder in the
loop!
Observations
![Page 22: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/22.jpg)
© Devi Parikh 2008
Progressive strategy Hence – progressive strategy!
Similar to ensemble of weak classifiers Or hypothesize-and-test
Multiple strategies: Rough + gradient + line, or rough + line, or
rough + gradient, or rough alone Different values of threshold during rough
corner detection Total 12
Order of strategies
![Page 23: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/23.jpg)
© Devi Parikh 2008
Results
Dataset of 500 images
Performance metric: % barcodes successfully decoded
Decoder model: Barcode successfully decoded if 80% of symbols are correctly identified
![Page 24: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/24.jpg)
© Devi Parikh 2008
Results
Allows for explicit trade-off between accuracy and computational time
![Page 25: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/25.jpg)
© Devi Parikh 2008
Results
![Page 26: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/26.jpg)
© Devi Parikh 2008
Results
![Page 27: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/27.jpg)
© Devi Parikh 2008
Results
![Page 28: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/28.jpg)
© Devi Parikh 2008
Results
![Page 29: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/29.jpg)
© Devi Parikh 2008
Results
![Page 30: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/30.jpg)
© Devi Parikh 2008
Results
![Page 31: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/31.jpg)
© Devi Parikh 2008
Results
![Page 32: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/32.jpg)
© Devi Parikh 2008
Results
![Page 33: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/33.jpg)
© Devi Parikh 2008
Results
![Page 34: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/34.jpg)
© Devi Parikh 2008
Results
![Page 35: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/35.jpg)
© Devi Parikh 2008
Conclusions 2D High Capacity Color Barcode (HCCB)
Successful localization and segmentation of HCCB from consumer images
Varying densities, aspect ratios, lighting, color balance, image quality, etc.
Simple computer vision and image processing techniques
Progressive strategy
![Page 36: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/36.jpg)
© Devi Parikh 2008
Acknowledgements
Microsoft Research Larry Zitnick Andy Wilson Zhengyou Zhang
Carnegie Mellon University Advisor: Tsuhan Chen
![Page 37: © Devi Parikh 2008 Localization and Segmentation of 2D High Capacity Color Barcodes Gavin Jancke Microsoft Research, Redmond Devi Parikh Carnegie Mellon](https://reader030.vdocument.in/reader030/viewer/2022032414/56649eec5503460f94bfe298/html5/thumbnails/37.jpg)
© Devi Parikh 2008
Thank you!