xiaofeng ren at ai frontiers: the quest for video understanding
TRANSCRIPT
The Quest for Video Understanding
Xiaofeng Ren
Institute of Data Science and Technologies (iDST), Alibaba Group
Alibaba Ecosystem
3CLOUD COMPUTING OPERATING SYSTEM
DATATECHNOLOGY TEAM
Online and Mobile Commerce Platform
$BUYERSSELLERS
Marketing
Financing & Payment Solution
ProfessionalService Providers
DigitalEntertainment
Logistics
Mobile Browser
Location-BasedSocial MediaWeibo
Alibaba Mission
4CLOUD COMPUTING OPERATING SYSTEM
DATATECHNOLOGY TEAM
Online and Mobile Commerce Platform
$BUYERSSELLERS
Marketing
Financing & Payment Solution
ProfessionalService Providers
DigitalEntertainment
Logistics
Mobile Browser
Location-BasedSocial MediaWeibo
make it easy to do business anywhere
Taobao Search: Text Text
9
Title:茜舞2017春装新品韩版修身公
主裙红色连衣裙无袖打底背心裙春秋
Q:无袖蓬蓬裙连衣裙夏 a型
Title:美特斯邦威卫衣女士春装字母拼色针织套头衫
Q:百搭灰色套头无帽卫衣女
Taobao Search: Text Text
10
Title:茜舞2017春装新品韩版修身公
主裙红色连衣裙无袖打底背心裙春秋
Q:无袖蓬蓬裙连衣裙夏 a型
Title:美特斯邦威卫衣女士春装字母拼色针织套头衫
Q:百搭灰色套头无帽卫衣女
Taobao Search: Text Image
11
Title:茜舞2017春装新品韩版修身公
主裙红色连衣裙无袖打底背心裙春秋
Q:无袖蓬蓬裙连衣裙夏 a型
• Joint text and image embedding for description• Leverage user click data to learn query-description similarity
Taobao Search: Image Image
12
3 billion images over 100 million products, 12 million daily active users
Videos on Taobao
14
• Promotional video• Video guide on how to use product• Documentary on how a product is made
… …
Live Streaming on Taobao
15
• Many top-selling stores have live streaming • Allow user interaction (chat, FAQ)• Conversion rate much higher than normal
Videos on Taobao
16
• Short videos are abundant on Xianyu (second-hand trading platform)• Demonstrations of products; sometimes people showing skills• Much richer information than text descriptions
Videos on Taobao
17
• Authoring tools• Video enhancements• Content analysis and tagging• Video-to-text matching
… …
Youku & Tudou: “Youtube-like”
19
Founded 2006
Founded 2005
IPO 2010
IPO 2011
Merged
2012Joined Alibaba
2016
Sample Video Applications
30
DigitalEntertainment
Logistics
CLOUD COMPUTING
Videos Everywhere
31CLOUD COMPUTING OPERATING SYSTEM
DATATECHNOLOGY TEAM
Online and Mobile Commerce Platform
$BUYERSSELLERS
Marketing
Financing & Payment Solution
ProfessionalService Providers
DigitalEntertainment
Logistics
Mobile Browser
Location-BasedSocial MediaWeibo