Arbitary Product Detection From Advertisement Video By Using Object Independent Featyres

Tomohiko TAKAHASHL Masaru SUGANO, Keiichiro HOASHL and Sei NAITOInternational Conference on Multimedia and Expo 2011

Arbitrary Product Detection from Advertisement Video by UsingObject Independent Features

OutlineIntroductionStructureImplementPreliminary ExperimentExperiment ResultConclusionsIntroductionautomatic annotation of TV content is essential for TV viewers, in order to enable efficient search from such large scaled TV content data.Hundreds of objects appear in typical TV programs, but it is unrealistic and inefficient to annotate all appearing objects in TV content. The extraction of important object from TV content is indispensable for the practical application.This paper focus on extracting the advertised product as the important object, from TV advertisement video.Introduction1) Movie producer intentionally emphasizes the product.there is large difference between the product and its neighboring area.2) The target product is filmed with sharp focus.The product area shows more detailed visual feature compared with the neighboring area.StructureRepresentative frames are extracted from an input video every 10th frame.The method is developed mainly for MPEG-2 compressed video.

Face and Telop DetectionUsing the method in [2].Haar-like features-based face extractor.

[2] M. Naito, et al. "High-level feature extraction experiments for TRECVID 2001", Proc of TRECVID 2007

Feature Point Density

DCT AC Energy Acquisition

Edge DetectionThe edges are detected by Canny's filter from representative frames.Then, approximate rectangles are extracted from each frame by the Douglas-Peucker method [7].[7] D. Douglas and T. Peucker, "Algorithms for the reduction of the number of points required to represent a digitized line or its caricature", The Canadian Cartographer 10(2), 112 122 (1973)

Color Histogram AcquisitionThe color histogram is used to evaluate the importance of each product candidate area.Area SegmentationWhen high density of the feature point area is shown as a shaded grid, by grouping each neighboring high density feature point, the candidate areas are detected.Regarding DCT AC energy, candidate areas are also detected in the same manner.

Important Object Evaluation(1/2)The feature point density difference difffp_all is:

The DCT AC energy difference diffAC_all is:

Important Object Evaluation(2/2)Color difference diffcolor:

Difference between the candidate area and its marginal areas is calculated as:

Score WeightingThe important object is shot in proper location and size within the frame, we adopted the one-third rule [8];The most important frames are the full shot [8].The object importance value is cumulated as per the duration of object appearance.

[8] D. Arijon, "Grammar of the film language", Focal Press Ltd. 1976Preliminary Experiment30 frames including 30 different types of the products from the advertisement movies (Test Set A).Another 30 frames including the actor/actress(Test Set B).30 frames including telops (Test Set C), and 30 frames including non-important object (Test Set D)

difffp_all : diffDCT_all : diffcolor is approximately 4: 1 :8, thus to normalize the each parameter, we defmed t1=2.0, t2=8.0, and t3=1.0.

Experimental Result (1/3)132 advertisement videos, 57 advertisements are from Japanese terrestrial channels and the others are free content.Each is 15-30 seconds long and its resolution is 720x540.Set ground truth:

Products are in wide variety such as handbag, sweets, coffee, jewelry, medicine, bicycle, cosmetics, TV set, and soon.If one advertisement video includes two products, we selected the top two objects by the proposed method.

Experimental Result (2/3)If the size difference of the detected area and that of the ground truth is smaller than 50% of the ground truth, we counted as the correct detection.This work achieved almost the equivalent accuracy with [6].

[6] T. Takahashi, M. Sugano, S. Sakazawa, "Automatic Thumbnail Extraction for DVR Based on Product Technique Estimation" IEEE Trans on CE, Vol. 56, No.2, May 2010

Experimental Result (3/3)More than 40% of the false detections were the false-positive detections. More accurate human's body detection should be applied.when a neighboring object contains many feature points, multiple objects may form one large candidate area. One of the possible solutions to this issue is to introduce graph cut algorithm.

ConclusionsEvaluated a novel method to extract important objects from TV content. Adopted the proposed method for the product detection from advertisement video. In the experiment, our method achieved F-measure of : 79.4.