shih-fu chang and open issues concept, technologies, video …b89035/pullpull/video... · 2005. 3....
TRANSCRIPT
1
Video Adaptation: Concept, Technologies,
and Open Issues
SHIH-FU CHANG
Presented by Jun-Cheng Chen
03/17/2005
2
Outline
Introduction
Unified Conceptual Framework and Technology Taxonomy
Active Research Areas
Open Issues
Support of Adaptation in International Standards
Conclusion
3
Introduction(1/3)
Video adaptation:Emerging field in pervasive media applications (such as PC, TV, PDA, or cellular phone).Transform the input video to an output in video or augmented multimedia.utilize manipulations at multiple levels
Signal, structural, semantic.
constrained optimizationIts objective is to maximize the utility of final presentation while satisfying various constraints (such as bandwidth).
4
Introduction(2/3)
Video adaptation differs from video coding in its scope and intended application locations.
signal level vs structural level vs semantic level, tanscoding vs selection vs summarization, bandwidth vs power vs time-constrained.
Often In the intermediate location, such as proxy between server and client.
Video adaptation is still a relatively less defined field.No coherent set of concepts, terminologies, or issues defined over well-formulated problems
5
Introduction(3/3)
6
7
Unified Conceptual Framework and Technology Taxonomy
8
Unified Conceptual Framework and Technology Taxonomy
Entity:Defined to refer to the basic unit of video that undergoes the adaptation process.
Different levels, such pixel, object, frame, shot, scene, syntactic components, and semantic components
Each entity is associated with certain resource requirements and utility values.
9
Unified Conceptual Framework and Technology Taxonomy
Utility:It represents the quality or users’ satisfaction of the video content (such as PSNR).
Adaptations space:The space of feasible adaptation for a given video entity.
Different adaptation operators can be defined for different types of entities. (ex: a video frame can be reduced in resolution, spatial quality, or skipped to reduce bandwidth cost.)
10
Unified Conceptual Framework and Technology Taxonomy
Systematic Procedure for Designing Video Adaptation Technologies
Video Adaptation TaxonomyFormat transcoding
Selection/Reduction
Replacement
Synthesis
11
Systematic Procedure for Designing Video
Adaptation Technologies (1/3)Identify the adequate entities for adaptation.
Identify the feasible adaptation operators.
Develop models for measuring and estimating the resource and utility values associated with video entities undergoing identified operators.
12
Systematic Procedure for Designing Video Adaptation Technologies(2/3)
Given user preferences and constraints on resource or utility, develop strategies to find the optimal adaptation operator(s) satisfying the constraints.
Problem formulation:Given a content entity E, user preferences, and resource constraints Cr, find the optimal adaptation operations Aopt within the feasible adaptation region so that the utility of the adapted entity e’ is maximized.
13
Systematic Procedure for Designing Video Adaptation Technologies(3/3)
[10].Y. Wang, J.-G. Kim, and S.-F. Chang, “Content-based utility function prediction for real-time MPEG-4 transcoding,” presented at theIEEE Int. Conf. Image Processing, Barcelona, Spain, 2003.
14
Video Adaptation Taxonomy
Format transcoding:To transcode video from one format to another, in order to make the video compatible with the new usage environment.
Selection/Reduction:Select some components of the entity and reduce them for saving resources.
Example: We can change the bit rate, frame rate or resolution for shots and frames in a video clip,
15
Video Adaptation Taxonomy
Replacement:Replace selected elements in a video entity with less expensive counterparts, while aiming at preserving the overall perceived utility.
Example: a video sequence may be replaced with key frames.
Synthesis:Synthesize new content presentations based on analysis results.
The goal is to provide a more comprehensive experience or a more efficient tool for navigation.
16
Video Adaptation Taxonomy
17
Active Research Areas
Semantic Event-Based Adaptation
Structural-Level Adaptation
Transcoding
Rapid Fast-Forward Drastic Temporal Condensation
18
Semantic Event-Based Adaptation
Doing video analysis for events and boundaries detection.
By using the information of video content, such as “the scoring points in sports video”, and “the breaking news in broadcast programs”.
Results of video event analysis can be utilized to produce different forms of adaptation.
19
Semantic Event-Based Adaptation
In this way, we can save bandwidth or the total viewing duration.
20
Semantic Event-Based Adaptation
Example: The percentage of important segments in the whole stream (such as sports broadcast).
They found non-important segments occupy more than 50% of duration.
Their system which focuses on sports can reach higher than 90% accuracy
[6]S.-F. Chang, D. Zhong, and R. Kumar, “Real-time content-based adaptive streaming of sports video,” presented at the IEEE Workshop Content-Based Access to Video/Image Library, IEEE CVPR Conf., Honolulu, Hawaii, Dec. 2001.
21
Structural-Level Adaptation
The structures in video are caused by event occurrence orders, camera control patterns, and the final editing process.Exploration of relations of structural elements provides great potential for video adaptation.Example:
Key frame extractionMosaicing
22
Structural-Level Adaptation
23
Transcoding
Signal level adaptationInvolving various manipulations of coded representations and issues of bit allocationManipulation of video signals:
Spatial: change spatial resolution, i.e., frame size.Precision: change the bit plane depth, color depth, or the step size for quantizing the transform coefficients.Temporal: change the frame rateObject: transmit a subset of objects
24
25
Rapid Fast-Forward DrasticTemporal Condensation
Video skimming
Bad ways:Increase the frame rate of the player.
Make the audio track unrecognizable.
Uniformly sample the frames in the original sequence.Important video frames may be skipped and audio content may be unrecognizable.
Extract keyframes to form shorter image sequence.Lose the synchronization between video and the associated audio track.
26
Rapid Fast-Forward DrasticTemporal Condensation
[14].H. Sundaram, L. Xie, and S.-F. Chang, “A utility framework for the automatic generation of audio-visual skims,”presented at the ACMMultimedia Conf., Juan Les Pins, France, 2002.
Adaptation entities: video shots.
Adaptation operations: length trimming or dropping of individual shots.
The problem was formulated as constrained optimization.
Constraints: viewing time, dialogs, key phrases, key audio, etc.
27
Open Issues
Define Utility Measures and User Preferences
Resolve Ambiguity in Specifying Adaptation Operation
Relations Among Adaptation, Utility, and Resource
Search Optimal Solutions in Large Spaces
Design End-to-End Integrated Systems
28
Define Utility Measures and User Preferences
It is difficult to define a universal measure for different levels or dimensions.
Levels include Perceptual, semantic, and comprehensiveness.
Signal-level measures are often inadequate mmany high-level operations such as shot removal, modality replacement, etc.
These operations also cause complex changes to content at other levels.
Users preferences often vary with content, task, and usage environment.
29
Define Utility Measures and User Preferences
Some possible alternativesInfer user preferences based on the usage history.
Correlate subjective preferences with content characteristics.
30
Resolve Ambiguity in Specifying Adaptation Operation
Some adaptation operations are not unambiguously defined.
“remove the second half of each shot”
“drop 10 % of transform coefficients”
Some possible ways Restrict adaptation operation only on unambiguous representation formats, such as JPEG 2000 and MPEG-4 fine grained scalable schemes.
Estimate the bound of variations in resource and utility.
31
Relation Among Adaptation, Utility, and Resource
Relations among adaptation, resource, and utility are often complex.
The complexity is especially high when the dimensionality of each space is high.
Potential approachesSample the adaptation space and store the corresponding resource and utility values.
Decompose the adaptation space into low-dimensional spaces and sample each subspace separately.
These schemes may lose the chance of exploring correlations among different dimensions.
32
Search Optimal Solution in Large Spaces
Exploration of the above multi-space relations often leads to formulation of constrained optimization.
Analytical solutions may exist for some cases.
example: rate-distortion model (low dimensional cases)Adaptation space: quantization
Resource space: bit rate
Utility space: SNR
In general, each space may have high dimensionality and the relations across spaces may be complex.
33
Design End-to-End Integrated Systems
DifficultiesRequire joint consider joint consideration of the adaptation subsystem with other subsystems.
Inconsistent and imperfect content analysis subsystem
Rights managementContent owners impose many restrictions on video content altering.
34
Design End-to-End Integrated Systems
Possible solutions:Adopt modular designs of subsystems and provide well-defined abstraction of requirements and performance of each subsystem.
Follow the international standard which are needed for describing information related to media rights management.
35
Support of Adaptation in International Standards
Mpeg-7 Content Descriptions
Mpeg-21 Digital Item Adaptation
Standardized Adaptation Framework
36
Mpeg-7 Content Descriptions
Descriptors (Ds) & Description schemes (DSs)XMLUsage history DSUserPreferences DS (creators, time periods, locations, etc.)Summary descriptionsVariation descriptions
Transcoding hintsMotion hints (for guiding motion-based transcodingmethods)Semantic importance hints (for guiding rate control) Etc…
37
Mpeg-21 Digital Item Adaptation
Digital Item Adaptation(MPEG21 part7):Address an extended scope of issues related to adaptation of digital multimedia content.
Usage environment descriptions (UEDs)Used to describe a wide array of user, terminal capabilities, network, and natural environment characteristics.
Universal constraints description (UCD) toolsimilar to UEDsmore explicit
AdaptationQos toolRelations between constraintsFeasible adaptation operations and associated utilities
38
Standardized Adaptation Framework
39
Conclusion
Despite the burgeoning activities and advances, this field is in need of an analytical foundation and solutions to many challenging open issues.It is worthwhile to note that solutions to most of the above identified open issues require joint consideration of adaptation with several other closely related issues, such as analysis of video content, rights management of digital content, etc.