Upload File

recognizing human activities at scale - microsoft.com filerecognizing human actions learning actions...

  • Home
  • Documents
  • Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling
26

Upload: lamnga

Post on 05-Feb-2018

220 views

Category:

Documents


3 download

Report

TRANSCRIPT

Page 1: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling
Page 2: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Recognizing Human Activities At Scale

Juan Carlos Niebleswww.niebles.net

@@jcniebles

Page 3: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 3

An image is worth one thousand wordsA video is worth one million words

Source: YouTube

“a tiger attacking a person on a grass field”

“a man packing a suitcase in a store”

“someone unlocking a combination lock”

“the tiger is being playful” “the man is unpacking the suitcase”

“the person is attempting to pick the lock”

Image:

Video:

Page 4: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Video Generation and Consumption is Huge

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 4

Netflix: 100 million hours watched per day

YouTube: 400 hours uploaded per minute

Cisco: ~1 million minutes of video per second by 2020

~200 peta-pixels/second

Page 5: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Videos are about People

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 5Source: YouTube

Page 6: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 6

Recognizing Human Actions

ActionDetection

Page 7: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Action Recognition in Videos

Short, pre-trimmed videos, only containing one action

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 7

KTH dataset [Schuldt, 2004] HOHA dataset [Laptev, 2008] UCF101 dataset [Soomro, 2012]

Page 8: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Action Recognition in Videos

Traditional action classification pipeline

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 8

Input Video(pre-trimmed)

Feature extraction(handcrafted/learne

d)

Featureencoding

YES

NO

Classifier

Page 9: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Source: YouTube

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 9

Temporal Detection of Actions

Page 10: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

“Polishing shoes”Classifier

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 10

Temporal Detection of Actions

• Apply complex classifier at each temporal location frame• Exhaustive search• Repeat for all actions we want to detect• Questionable scalability

Long input video

… …

Page 11: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Generic ActivityProposal

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 11

Fast Activity Proposals for Action DetectionLong input video

… …

[Caba Heilbron, Niebles & Ghanem. CVPR 2016][Escorcia, Caba Heilbron, Niebles & Ghamen, ECCV 2016]

• Runs very quickly (>130 fps)• Find all temporal intervals that contain “any activity”• Retrieve action intervals with high recall

Page 12: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Generic ActivityProposal

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 12

Long input video

… …

Generic ActivityProposal

Generic ActivityProposal

“Polishing shoes”Classifier

“Polishing shoes”Classifier

“Polishing shoes”Classifier

[Caba Heilbron, Niebles & Ghanem. CVPR 2016][Escorcia, Caba Heilbron, Niebles & Ghamen, ECCV 2016]

Fast Activity Proposals for Action Detection

Generic ActivityProposal

Page 13: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 13

Fast Activity Proposals for Action Detection

[Caba Heilbron, Niebles & Ghanem. CVPR 2016][Escorcia, Caba Heilbron, Niebles & Ghamen, ECCV 2016]

Page 14: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 14

Recognizing Human Actions

Learning ActionsWithWeak

Supervision

ActionDetection

Page 15: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 15

TestVideo

LearnedModel

PredictionPan Fry Put to Plate

Page 16: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Fully Supervised Learning

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 16

TrainVideo

Model

LabelPan Fry Put to Plate

• Many training videos with per frame action labels• Costly to annotate!

Page 17: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Weakly-Supervised Learning

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 17

TrainVideo

Model

? ? ? ?Label ? ?

• Only use action ordering• Disambiguate by aggregating across with many

training videos

Pan Fry then Put to Plate

[Huang, Fei-Fei & Niebles, ECCV 2016]

Page 18: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Extended Connectionist Temporal Classification

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 18

• Extends the CTC framework• Explores space of frame-to-labels assignments efficiently• Incorporates pairwise frame similarities

[Huang, Fei-Fei & Niebles, ECCV 2016]

InputVideo

Pan Fry

Cut Bun

Put Plate

Page 19: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Evolution of Training Frame-to-Label correspondence

GroundTruth

ep1ep2ep3ep4ep5ep8ep11ep14ep17ep20Ours (weak)

Our approach starts without label correspondences for the training

videos and iteratively improves the localization of the actions.

Page 20: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 20

Weakly Supervised Activity Segmentation Results

[Huang, Fei-Fei & Niebles, ECCV 2016]

Page 21: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 21

Weakly Supervised Action Detection Results

0

1

Dri

ve C

ar

[Huang, Fei-Fei & Niebles, ECCV 2016]

Page 22: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 22

0

1

[Huang, Fei-Fei & Niebles, ECCV 2016]

Dri

ve C

arWeakly Supervised Action Detection Results

Page 23: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Hierarchical Modeling of Composable Activities

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 23

[Lillo, Soto & Niebles, CVPR 2014]

[Lillo, Niebles & Soto, CVPR 2016]

Page 24: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

Large-scaleBenchmark

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 24

ActionDetection

Learning ActionsWithWeak

Supervision

Recognizing Human Actions

Page 25: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

ActivityNet – www.activity-net.org

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 25

[Caba Heilbron, Escorcia, Ghanem & Niebles, CVPR 2015]

Page 26: Recognizing Human Activities At Scale - microsoft.com fileRecognizing Human Actions Learning Actions With Weak Supervision Action Detection. Temporal Action Labeling

7/13/16 Juan Carlos Niebles - Recognizing Human Actions at Scale 26

Thank you!Juan Carlos Niebles

www.niebles.net

@@jcniebles

ActionDetection

Learning Actions

WithWeak

Supervision

Large-scaleBenchmark


VERBS. Action Verbs –a word that expresses ACTION!!! There are two types of action verbs: physical actions and mental / emotional actions

TNI Corrective Action Committee Report - NELAC Institute · implementing appropriate corrective actions and actions to prevent recurrence, documenting all corrective actions, and

2018 – 2019 DIGITAL TERRITORY ACTION PLAN · 2018 – 2019 DIGITAL TERRITORY ACTION PLAN EXPLORE 34.4% DELIVER 27.9% DESIGN 37.7% ACTIONS BY PHASE ACTIONS BY DIRECTION Connecting

Action D.1: Communication and dissemination actions › images › english › ActionD1 fin.pdf · Action D.1: Communication and dissemination actions The action started on October

Digital Education Action Plan: priorities, actions and timeframe · 2018-09-26 · Digital Education Action Plan: priorities, actions and timeframe Georgi Dimitrov & Fabrizia Benini,

Crimes, federal, United Nations criminal action, Federal Government illegal actions,

CQC Inspection Action Plan - Advisory Actions · CQC Inspection Action Plan – Advisory Actions : Status . For discussion . History : CQC Compliance Action Plan was presented to

Creating and Using Action Links and Actions - Copy

Truth & Reconciliation Calls to Action Actions for EGBC

Action guide: the actions you can take so Everyone Wins

Eurocode 1 Actions on Structures Part 1-7 General Action-Accidental Action

Crisis Action Team Actions (as of131400 Sep)

The Psychological Actions and Entrepreneurial · PDF fileThe Psychological Actions and Entrepreneurial Success: An Action Theory Approach1 ... over, action orientation implies thatmappingoughttobeparsimonious,

Responses to irrational actions in action observation and

2020 SustainabilityPlan Malaysia · Malaysia 17 Actions Ethicsand Transparency People Safe operation Environment Climate change Innovation and technology 1 Action 6 Actions 4 Actions

UCC Athena SWAN Institutional Action Plan Priority actions ... · UCC Athena SWAN Institutional Action Plan Priority actions are highlighted. ... Data shows attrition of female students

FAA Call to Action on Runway Safety Short-term Actions

ACTIONS AGENDA ITEM/ACTION ASSIGNMENT PODCAST

Bonn Call for Action - International Atomic Energy Agency · The Bonn Call-for-Action highlights ten main actions, and related sub-actions, that were identified as being essential

The Economics of Class Actions and Class Action Waivers

Actions speak louder with words: The roles of action and ......Actions speak louder with words: The roles of action and pedagogical language for grounding mathematical proof Mitchell

Green City Action Plan: A Framework for GrEEEn Actions Melaka

UML Activities & Actions - Inria · Charles ANDRE - UNSA Actions & Activities †An action execution corresponds to the execution of a particular action. † Similarly, an activity

Strategy Potential Actions Key Performance Indicators Short-...Strategy Potential Actions Key Performance Indicators Short-term Action (1 – 3 years) Mid-Term Action (3 – 5 years)

Action Program: Enabling Regional Actions

Community Facilities Network Plan Action Plan...2016/09/05  · Community Facilities Network Plan Action Plan (2019) V.4 Area-based actions – network priority actions Incorporating

Climate Action Revenue Incentive Program (CARIP) Public ......Re-promote toilet recycling rebate for residents Action Type Other Actions Actions Taken this Year Hosted a 3 -part environmental

FINAL REPORT ACTION A.1: Preparatory actions for work with

BEPS Actions – where legislative changes/action may be needed

Insight to Action Using Wave Analytics and Custom Actions

Humanitarian Assessment Provider Handbook€¦ · Humanitarian Action Guide 2016, page 369.) Key actions – action you take to achieve the standards ... implement the key actions,

Update on the actions from the UK Governments 2016 ‘Action

MITIGATION AND ADAPTATION ACTION PLAN, PILOT ACTIONS …

THE NATIONAL STRATEGY TO SECURE CYBERSPACE · action, versus other alternative actions, versus non-action, taking into account any existing or future private solutions. Federal actions

Maryland Environmental Literacy Action Plan 1.TRAINING & PROFESSIONAL DEVELOPMENT 2.STUDENT-FOCUSED ACTIONS 3.POLICY & RELATED ACTIONS