generating annotations for how-to videos using...
TRANSCRIPT
![Page 1: Generating Annotations for How-to Videos Using …juhokim.com/files/CHI2013-wip-crowdsourcing-poster.pdfGenerating Annotations for How-to Videos Using Crowdsourcing Phu Nguyen, Juho](https://reader035.vdocument.in/reader035/viewer/2022080720/5f79909712b03d63c706b378/html5/thumbnails/1.jpg)
Motivation• How-tovideosarevaluableforteaching
• Searchingforhow-tovideosisdifficult.Usersoften
havetoskimmanyvideostofindtherightone
• Havingvideoannotationswouldimprovesearching,
browsing,andindexingvideos
ProblemGenerating video annotations is difficult
• Manuallyannotatingeachvideoiscumbersome
• Usingcomputervisionrequirestrainingdata
Research QuestionCan a crowdsourcing method be efficient and scalable enough to generate annotations for how-to videos?
• UsingPhotoshophow-tovideosasourexample
domain
• TestingourproposedmethodonAmazon
MechanicalTurk
Online Experiment
Future Work• Improveanalytictoolsfortheworkflow
•• Pipelinesubtaskstogetherautomatically
• Reducenumberofcrowdsourcingtasks/video
Acknowledgements• FundingfromQuantaComputer,Inc
• EECSSuperUROPprogram
Annotation Accuracy from a Mechanical Turk Experiment
• Thesizeofthevideoplayermadeithardforworkerstoseewhatisgoingon
• Workersstillhadtroubleknowingwheninstructorclicksontool
• Allvideosarelabeledcorrectlywithmajorityvoting
• Workershadtroubleselectingthecorrectlabelifmultiplelabelsseemsimilar(“Gradient”vs.“GradientMap”)
• Mostoftheincorrectbeforesandafterswereduetobadvideoclips(choosingbeforeimageofanincorrecttoolinclip)
• Someerrorswereduetodialogobstructions
Capturing Time Marks Labeling Tools Used Capturing Before/After Images
Procedure that
generates annotations
Task 2: Label Tools Used
Theworkerlabelsthetoolusedina20s
clipofahow-tovideowithdropdown
menus.
Task 3: Capture Before/After Images
Theworkerwatchesa20sclipofahow-
tovideoandcapturesthebeforeandafter
imagesforatooluse.
Generating Annotations for How-to Videos Using CrowdsourcingPhu Nguyen, Juho Kim, Rob Miller - MIT CSAIL
How-to Video Annotation Generator Annotations Interface
Task 1: Capture Time Marks
Theworkerwatchesa60sclipofahow-to
videoandclicksthe“ToolClicked”button
whenevertheinstructorclicksonatool.