moodstocks - mobile image recognition - paris tech talks #6
DESCRIPTION
An overview of Mobile Image Recognition, a presentation of Moodstocks' approach and a focus on Moodstocks SDK core library for Paris Tech Talks Meetup #6. * https://moodstocks.com/ * http://www.meetup.com/Paris-Tech-Talks/events/165819522/ * https://github.com/Moodstocks/jpecTRANSCRIPT
![Page 1: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/1.jpg)
Supersonic image recognition for your mobile apps
Cédric @deltheil
Paris Tech Talks #6
Moodstocks
Tech co-founder
![Page 2: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/2.jpg)
Mobile Image Recognition
Your Mobile App
Reference Images + ID-s
Image Recognition Platform
query match ID = 1234
index
user
developer
ID = 1234
![Page 3: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/3.jpg)
The old way: thin client
Mobile App
Image Recognition API
client
server
reference images
send JPEG image via HTTP
match ID or nil
![Page 4: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/4.jpg)
Drawback: network latencies!
![Page 5: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/5.jpg)
Our approach: thick clientMobile App
Moodstocks API
client
server
reference images up to millions images
Moodstocks SDK
image signatures up to 10k images
query match ID or nil
server-side search for large DB
sync
![Page 6: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/6.jpg)
Benefit: speed
a typical search runs in a few ms
![Page 7: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/7.jpg)
Moodstocks SDK
Mobile App
Wrapper
Core Library
Objective-C Java
C
![Page 8: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/8.jpg)
Moodstocks SDK
Mobile App
Wrapper
Core Library
we will now focus on this part
![Page 9: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/9.jpg)
Design Goal #1
Portability• core library written in C99
• +99% code in common between iOS / Android
• e.g supporting Intel CPU-s on Android was… no effort!
![Page 10: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/10.jpg)
Design Goal #2
Efficiency• in-memory data structures for maximum speed
• leverage NEON capabilities (SIMD) where applicable
• a typical search runs in ~ 30 ms / 10k images (iPhone 5S)
![Page 11: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/11.jpg)
Design Goal #3
Low footprint
• 100% home made, lightweight implementation
• implement & embed only what we need!
• around 800 kB for the iOS ARMv7s release build (-Os)
![Page 12: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/12.jpg)
in 1/2 day
Our choices pay off
![Page 13: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/13.jpg)
Core Library Architecture
Public Header File
Internal Modules
3rd Party Libraries
Image Recognition
Pipeline
![Page 14: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/14.jpg)
Public Header File
Internal Modules
3rd Party Libraries
Image Recognition
Pipeline
![Page 15: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/15.jpg)
Image Recognition Pipeline
• full-stack: the whole recognition occurs on-device!
• robust to camera noises (blur, lighting, …)
• supports 1D and 2D barcodes too
Image Recognition Pipeline
![Page 16: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/16.jpg)
Public Header File
Internal Modules
3rd Party Libraries
Image Recognition
Pipeline
![Page 17: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/17.jpg)
Internal Modules (1/4)
• master-slave sync engine for image signatures
• efficient: works with diff-s & data bundles (less requests)
• fail-safe: retries + the client always stays consistent
Sync StorageLogs API client
![Page 18: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/18.jpg)
Internal Modules (2/4)
Sync StorageLogs API client
• collects search results & metadata (OS, etc) along time
• a long-running thread posts batches every 30s
• tolerant to network errors -> HTTP 409 (Already Exists)
![Page 19: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/19.jpg)
Internal Modules (3/4)
• persists image signatures on-disk
• persists search results events on-disk
• used as a temp on-disk buffer at sync time
Sync StorageLogs API client
![Page 20: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/20.jpg)
Internal Modules (4/4)
• used for server-side recognition (send image via HTTP)
• includes a pool to re-use connections
• also used to post logs batches to the server (HTTP)
Sync StorageLogs API client
![Page 21: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/21.jpg)
Public Header File
Internal Modules
3rd Party Libraries
Image Recognition
Pipeline
![Page 22: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/22.jpg)
3rd Party Libraries (1/4)
libcurl Tokyo Cabinet jpec msgpack
• the +500M users file transfer library
• multi API -> multiple transfers in the same thread
• we use it as the backend of our sync engine + HTTP reqs.
![Page 23: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/23.jpg)
3rd Party Libraries (2/4)
libcurl Tokyo Cabinet jpec msgpack
• key / value embedded database (hash / table / B+tree)
• … and also a fantastic C library (see tcutil.h)
• we use it as the backend of our storage engine
![Page 24: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/24.jpg)
3rd Party Libraries (3/4)
libcurl Tokyo Cabinet jpec msgpack
• our home-made, tiny JPEG encoder (600 LOC-s)
• we open sourced it -> github.com/Moodstocks/jpec
• we use it for server-side requests (send image)
![Page 25: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/25.jpg)
3rd Party Libraries (4/4)
libcurl Tokyo Cabinet jpec msgpack
• efficient binary serialization format
• we use it to pack the data we synchronize
![Page 26: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/26.jpg)
Thanks!
Questions? Comments? cedric AT moodstocks DOT com | @deltheil
![Page 27: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/27.jpg)
Building
• we use simple Makefile-s and shell scripts
• we cross-compile w/ the iOS and Android toolchains
• we combine all the arch-s into a fat static lib (armv7, …)
![Page 28: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/28.jpg)
Testing
• we write unit tests with ct (Easy Unit Testing) by @krarick
• we run them on device! e.g jailbreaked iPhone + openssh
• we use a mock to test the sync engine
![Page 29: Moodstocks - Mobile Image Recognition - Paris Tech Talks #6](https://reader034.vdocument.in/reader034/viewer/2022042815/5577c84dd8b42ae0418b4705/html5/thumbnails/29.jpg)
Quality
• no warnings: we compile with -Wall -Werror
• static analysis with Clang Static Analyzer
• memory leak checks with Valgrind