using deep neural networks for fashion applications

Using Deep Neural Networksfor Fashion Applications

Ahmad Qamar

ahmad@threadgenius.co

motivation

challenges

related work

data collection

Thread Genius

applications

future work

demosthreadgenius.co

set of terms used interchangeably

● features, fingerprints, representations, latent factors, vectors

● attributes, labels, concepts

● product , SKU, in-store

● street, wild-type, UGC

threadgenius.co

threadgenius.co

motivation

threadgenius.co

2000 today→ web content increasingly image-heavy→ more participation and engagement+ 1.8B+ photos uploaded daily (2014)

- only 15% contain relevant metadata

→ millennials are increasingly brand agnostic

- logo detection fails in extracting signalthreadgenius.co

threadgenius.co

millennials prefer branded content that is ...

● social is saturated with content● attention is limited commodity

authentic and information dense

threadgenius.co

challenges

+ attribute extraction works well- limited and generic taxonomy of attributes- word attributes don’t fully capture image

threadgenius.co

visual understanding is becoming ubiquitous

- visual search suffers from poor results- lack of focus on specific domain

threadgenius.co

vs.

query out-of-the-box model fashion domain model

threadgenius.co

threadgenius.co

threadgenius.co

collaborative filtering not fit for fashion

“people who buy a also buy b”

- no information about content

- half-life of fashion products is ~1.5 months

- higher quantity (vs. movies), lower volume (vs. music)

threadgenius.co

image models require lots of data to prevent overfitting

- limited public datasets for fashion

- taxonomy requires domain consideration

- collecting training data is painstaking

- ambiguity and variability present in fashion products and photos

threadgenius.co

threadgenius.co

related work

threadgenius.co

Convnets 101: convnets learn features and the classifier simultaneously

images from Sander Dieleman

image

[Kiapour et al] Where to Buy It: Matching Street Clothing Photos in Online Shops

+ learn a NN similarity function between query and candidate to outputs match-score; works well

- querying is expensive since score must be computed for all candidates

- learn separate functions for each category (tops, footwear, …)

threadgenius.co

[Liu et al] DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations

+ provide 800K fashion images for academic community

+ perform landmark detection for localization

- questionable data quality: taxonomy contains irrelevant words (eg. brooklyn, kurt), images are mislabeled, bounding boxes too tight

threadgenius.co

images for “brooklyn”

[Bell et al] Learning visual similarity for product design with convolutional neural networks

threadgenius.co

+ learn image embeddings that place similar products close

- requires cropped images for querying

[Bell et al] Learning visual similarity for product design with convolutional neural networks

threadgenius.co

train on two tasks

● metric learning● category prediction

+

-

[Ren et al] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

threadgenius.coobject detection model

[Ren et al] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

threadgenius.co

train a mini-network that classifies objectness and regresses bounding boxes

threadgenius.co

data collection

fashion products exhibit different types of structure● category: top, footwear, ...● product: sweater, pumps, ...● detail: shawl collar, square toe, ...● color: yellow, anthracite, ...● pattern: paisley, colorblock, ...● aesthetic: preppy, scandalous, …

- degrees of freedom make similarity search subjective

threadgenius.co

compiled a taxonomy of ~1000 fashion attributes

threadgenius.co

for each attribute, images are sourced from● retail websites● social media networks (Pinterest, The Hunt)

● and fashion resale networks (Poshmark, Vinted)

threadgenius.co

Long sleeve silk chiffon shirt-style dress featuring graphic pattern in navy, burgundy, and green. Vented crewneck collar. Gathering at front and back yokes. Detachable self-tie fastening at waist. Two-button barrel cuffs. Detachable viscose chiffon slip lining in black. Tonal stitching. Body: 100% silk. Lining: 100% viscose. Imported.

+ image-attribute pairs help model learn fashion feature detectors- image-attribute pairs are not enough: attribute classifiers simply compute histograms over visual features

threadgenius.co

≈

+ image-image pairs allow for unsupervised learning of similarity+ captures invariances

threadgenius.co

+ -

images require cleaning

threadgenius.coimages from [Kiapour et al]

3M+ images annotated

threadgenius.co

threadgenius.co

Thread Genius

images/UGC

unified space of images, products, and metadata

products

OPENING CEREMONY

$495

ZARA$125metadata

varsity jacket, color black,

standup collar threadgenius.co

threadgenius.co

object detector

RPN model trained on bounding box labels

feature extractor

alternate training on attribute classification and metric learning

indexing

Tech StackResearch pipeline

Deployment pipeline

Training data

...

Data annotation

Lasagne Annoy

Model training + experimentation

+ validation

Compute + Storage + Server

TG GPU Server

Product Inventory

Lookbooks

...

fashion blogs ...

Workflow manager

TG API

threadgenius.co

threadgenius.co

applications

retail

alternative products to: sold out inventory, pricey items

shop.threadgenius.co

visual marketing

making Instagram shoppable

+ 2-3x lift in conversion

threadgenius.co

audience generation

build custom audiences for specific products

threadgenius.co

audience generation

+ 3x increase in CTR

threadgenius.co

threadgenius.co

future work

threadgenius.co

experimental model [beta]

word2vec on collections of images and text

● words represented by embedding / lookup table● images represented by convnet

Long sleeve silk chiffon shirt-style dress featuring graphic pattern in navy, burgundy, and green. Vented crewneck collar. Gathering at front and back yokes. Detachable self-tie fastening at waist. Two-button barrel cuffs. Detachable viscose chiffon slip lining in black. Tonal stitching. Body: 100% silk. Lining: 100% viscose. Imported.

document

threadgenius.co

other

● combined model that directly extracts features of component apparel items (convnet+RNN)

● refine training of experimental image-text model

threadgenius.co

demos

threadgenius.co

experimental model: semantic arithmetic with images and text

threadgenius.co

robo-Bill Cunningham

Questions?

we’re hiring for backend and ML roles

we’re hiring for backend and ML roles

ahmad@threadgenius.co