using deep neural networks for fashion applications
TRANSCRIPT
Using Deep Neural Networksfor Fashion Applications
Ahmad Qamar
motivation
challenges
related work
data collection
Thread Genius
applications
future work
demosthreadgenius.co
set of terms used interchangeably
● features, fingerprints, representations, latent factors, vectors
● attributes, labels, concepts
● product , SKU, in-store
● street, wild-type, UGC
threadgenius.co
threadgenius.co
motivation
threadgenius.co
2000 today→ web content increasingly image-heavy→ more participation and engagement+ 1.8B+ photos uploaded daily (2014)
- only 15% contain relevant metadata
→ millennials are increasingly brand agnostic
- logo detection fails in extracting signalthreadgenius.co
threadgenius.co
millennials prefer branded content that is ...
● social is saturated with content● attention is limited commodity
authentic and information dense
threadgenius.co
challenges
+ attribute extraction works well- limited and generic taxonomy of attributes- word attributes don’t fully capture image
threadgenius.co
visual understanding is becoming ubiquitous
- visual search suffers from poor results- lack of focus on specific domain
threadgenius.co
vs.
query out-of-the-box model fashion domain model
threadgenius.co
threadgenius.co
threadgenius.co
collaborative filtering not fit for fashion
“people who buy a also buy b”
- no information about content
- half-life of fashion products is ~1.5 months
- higher quantity (vs. movies), lower volume (vs. music)
threadgenius.co
image models require lots of data to prevent overfitting
- limited public datasets for fashion
- taxonomy requires domain consideration
- collecting training data is painstaking
- ambiguity and variability present in fashion products and photos
threadgenius.co
threadgenius.co
related work
threadgenius.co
Convnets 101: convnets learn features and the classifier simultaneously
images from Sander Dieleman
image
[Kiapour et al] Where to Buy It: Matching Street Clothing Photos in Online Shops
+ learn a NN similarity function between query and candidate to outputs match-score; works well
- querying is expensive since score must be computed for all candidates
- learn separate functions for each category (tops, footwear, …)
threadgenius.co
[Liu et al] DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations
+ provide 800K fashion images for academic community
+ perform landmark detection for localization
- questionable data quality: taxonomy contains irrelevant words (eg. brooklyn, kurt), images are mislabeled, bounding boxes too tight
threadgenius.co
images for “brooklyn”
[Bell et al] Learning visual similarity for product design with convolutional neural networks
threadgenius.co
+ learn image embeddings that place similar products close
- requires cropped images for querying
[Bell et al] Learning visual similarity for product design with convolutional neural networks
threadgenius.co
train on two tasks
● metric learning● category prediction
+
-
[Ren et al] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
threadgenius.coobject detection model
[Ren et al] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
threadgenius.co
train a mini-network that classifies objectness and regresses bounding boxes
threadgenius.co
data collection
fashion products exhibit different types of structure● category: top, footwear, ...● product: sweater, pumps, ...● detail: shawl collar, square toe, ...● color: yellow, anthracite, ...● pattern: paisley, colorblock, ...● aesthetic: preppy, scandalous, …
- degrees of freedom make similarity search subjective
threadgenius.co
compiled a taxonomy of ~1000 fashion attributes
threadgenius.co
for each attribute, images are sourced from● retail websites● social media networks (Pinterest, The Hunt)
● and fashion resale networks (Poshmark, Vinted)
threadgenius.co
Long sleeve silk chiffon shirt-style dress featuring graphic pattern in navy, burgundy, and green. Vented crewneck collar. Gathering at front and back yokes. Detachable self-tie fastening at waist. Two-button barrel cuffs. Detachable viscose chiffon slip lining in black. Tonal stitching. Body: 100% silk. Lining: 100% viscose. Imported.
+ image-attribute pairs help model learn fashion feature detectors- image-attribute pairs are not enough: attribute classifiers simply compute histograms over visual features
threadgenius.co
≈
+ image-image pairs allow for unsupervised learning of similarity+ captures invariances
threadgenius.co
+ -
images require cleaning
threadgenius.coimages from [Kiapour et al]
3M+ images annotated
threadgenius.co
threadgenius.co
Thread Genius
images/UGC
unified space of images, products, and metadata
products
OPENING CEREMONY
$495
ZARA$125metadata
varsity jacket, color black,
standup collar threadgenius.co
threadgenius.co
object detector
RPN model trained on bounding box labels
feature extractor
alternate training on attribute classification and metric learning
indexing
Tech StackResearch pipeline
Deployment pipeline
Training data
...
Data annotation
Lasagne Annoy
Model training + experimentation
+ validation
Compute + Storage + Server
TG GPU Server
Product Inventory
Lookbooks
...
fashion blogs ...
Workflow manager
TG API
threadgenius.co
threadgenius.co
applications
retail
alternative products to: sold out inventory, pricey items
shop.threadgenius.co
visual marketing
making Instagram shoppable
+ 2-3x lift in conversion
threadgenius.co
audience generation
build custom audiences for specific products
threadgenius.co
audience generation
+ 3x increase in CTR
threadgenius.co
threadgenius.co
future work
threadgenius.co
experimental model [beta]
word2vec on collections of images and text
● words represented by embedding / lookup table● images represented by convnet
Long sleeve silk chiffon shirt-style dress featuring graphic pattern in navy, burgundy, and green. Vented crewneck collar. Gathering at front and back yokes. Detachable self-tie fastening at waist. Two-button barrel cuffs. Detachable viscose chiffon slip lining in black. Tonal stitching. Body: 100% silk. Lining: 100% viscose. Imported.
document
threadgenius.co
other
● combined model that directly extracts features of component apparel items (convnet+RNN)
● refine training of experimental image-text model
threadgenius.co
demos
threadgenius.co
experimental model: semantic arithmetic with images and text
threadgenius.co
robo-Bill Cunningham
Questions?
we’re hiring for backend and ML roles
