Video Special Effects

Wen-Hung Liao10/3/2006

Outline

Hardware-based video special effects Software-based video special effects Video content analysis

Hardware-based VFX

Matrox RT.X2 http://www.matrox.com/video/ct/home.cfm

Real-time multi-layer workflows in HD and SD: Designed primarily for real-time native HDV and DV editing, Matrox RT.X2 also provides a high-quality MPEG-2 4:2:2 I-frame codec so you can capture other HD and SD formats using RT.X2's analog inputs, and mix all types of footage on the timeline in real time.

Where to purchase?

http://www.voxelvision.com.tw/ http://www.avideo.com.tw/

Real-time CPU Effects

Realtime primary color correction Realtime secondary color correction Realtime chroma and luma keying Realtime speed changes Realtime transitions Realtime track matte Realtime move & scale Realtime SD clip upscaling in an HD timeline Realtime HD clip downscaling in an SD timeline Native Adobe Premiere Pro effects

Real-time GPU Effects Realtime Adobe Motion effect Realtime advanced 2D/3D DVE Realtime shadow Realtime blur/glow/soft focus Realtime page curl Realtime surface finish Realtime pan & scan Realtime mask Realtime mask blur Realtime mask mosaic Realtime four-corner pin Accelerated shine Native Adobe Premiere Pro transitions Realtime crystallize Realtime lens flare Realtime old movie effect

Graphics Processing Unit (GPU)

A Graphics Processing Unit or GPU (also occasionally called Visual Processing Unit or VPU) is a dedicated graphics rendering device for a personal computer, workstation, or game console.

Modern GPUs are very efficient at manipulating and displaying computer graphics, and their highly parallel structure makes them more effective than typical CPUs for a range of complex algorithms.

GPU Operations

A GPU implements a number of graphics primitive operations in a way that makes running them much faster than drawing directly to the screen with the host CPU.

The most common operations for early 2D computer graphics include the BitBLT operation (combine two bitmap patterns using a RasterOp), usually in special hardware called a "blitter", and operations for drawing rectangles, triangles, circles, and arcs.

Modern GPUs also have support for 3D computer graphics, and typically include digital video-related functions as well.

Applications: Example 1

OpenVIDIA : GPU accelerated Computer Vision Library, http://openvidia.sourceforge.net/

The OpenVIDIA project implements computer vision algorithms on computer graphics hardware, using OpenGL and Cg.

The project provides useful example programs which run real time computer vision algorithms on single or parallel graphics processing units.

Applications: Example 2

Real-time stereo using GPU “... Since the GPU is built to process images it is

particularly well suited to perform some computer vision and image processing algorithms very efficiently.  We developed a real-time stereo algorithm that runs on the GPU and is several times faster than most CPU-based implementations.”

Software-based Video Special Effects

Examples: EffectTV: http://effectv.sourceforge.net/ FreeFrame: http://

freeframe.sourceforge.net/gallery.html

RGB/YUV Conversion

http://www.fourcc.org/index.php?http%3A//www.fourcc.org/intro.php

RGB to YUV Conversion Y = (0.257 * R) + (0.504 * G) + (0.098 * B) + 16 Cr = V = (0.439 * R) - (0.368 * G) - (0.071 * B) + 128 Cb = U = -(0.148 * R) - (0.291 * G) + (0.439 * B) + 128

YUV to RGB Conversion B = 1.164(Y - 16) + 2.018(U - 128) G = 1.164(Y - 16) - 0.813(V - 128) - 0.391(U - 128) R = 1.164(Y - 16) + 1.596(V - 128)

Types of Special Effects

Applying to the whole image frame Applying to part of the image (edges, moving

pixels,…) Applying to a collection of frames (frame-

buffer) Applying to detected areas Overlaying virtual objects:

at pre-determined locations in response to user’s position

Compressed-Domain Processing

Video special effects editing in MPEG-2 compressed videoFernando, W.A.C.; Canagarajah, C.N.; Bull, D.R

Fade, dissolve and wipe production in MPEG-2 compressed videoFernando, W.A.C.; Canagarajah, C.N.; Bull, D.R.;

Video Content Analysis

Event detection For indexing/searching To obtain high-level semantic description of

the content.

Image Databases Problem: accessing and searching large databases

of images, videos and music Traditional solutions: file IDs, keywords, associated t

ext. Problems:

can’t query based on visual or musical properties depends on the particular vocabulary used doesn’t provide queries by example time consuming

Solution: content-based retrieval using automatic analysis tools (see http://wwwqbic.almaden.ibm.com)

Retrieval of images by similarity Components:

Extraction of features or image signatures and efficient representation and storage

A set of similarity measures A user interface for efficient and ordered representation of

retrieved images and to support relevance feedback

Considerations Many definitions of similarity are possible User interface plays a crucial role Visual content-based retrieval is best utilized when

combined with traditional search

Image features for similarity definition

Color similarity Similarity: e.g., “distance” between color histograms Should use perceptually meaningful color spaces (HSV, La

b...) Should be relatively independent of illumination (color cons

tancy) Locality:“find a red object such as this one

Texture similarity Texture feature extraction (statistical models) Texture qualities: directionality, roughness, granularity...

Shape Similarity Must distinguish between similarity between actual geometrical

2-D shapes in the image and underlying 3-D shape Shape features: circularity, eccentricity, principal axis

orientation...Spatial similarity Assumes images have been (automatically or manually)

segmented into meaningful objects (symbolic image) Considers the spatial layout of the objects in the scene

Object presence analysis Is this particular object in the image?

Main components of retrieval system

Database population: images and videos are processed to extract features (color, texture, shape, camera and object motion)

Database query: user composes query via graphic user interface. Features are generated from graphical query and input to matching engine

Relevance feedback: automatically adjusts existing query using information fed back by user about relevance of previously retrieved objects

Video parsing and representation

Interaction with video using conventional VCR-like manipulation is difficult - need to introduce structural video analysis

Video parsing Temporal segmentation into elemental units Compact representation of elemental unit

Temporal segmentation

Fundamental unit of video manipulation: video shots Types of transition between shots:

Abrupt shot change Fades: slow change in brightness Dissolve Wipe: pixels from second shots replace those of previous

shot in regular patterns

Other factors of image change: Motion, including camera motion and object motion Luminosity changes and noise

Representation of Video Video database population has three major components:

Shot detection Representative frame creation for each shot Derivation of layered representation of coherently moving structu

res/objects

A representative frame (R-frame) is used for: population: R-frame is treated as a still image for representation query: R-frames are basic units initially returned in video query

Choice of R-frame: first - middle - last frame in video shot sprite built by seamless mosaicing all frames in a shot

Video soundtrack analysis Image/sound relationships are critical to the perception and unde

rstanding of video content. Possibilities: Speech, music and Foley sound, detection and representation Locutor identification and retrieval Word spotting and labeling (speech recognition) A possible query could be: “find the next time this locutor is again

present in this soundtrack”

Video scene analysis

500-1000 shots per hours in typical movies One level above shot: sequence or scene (a series of consecutiv

e shots constituting a unit from the narrative point of view)