![Page 1: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/1.jpg)
© 2006 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice
Compressed-Domain Video Processing and Transcoding
Susie Wee, John ApostolopoulosMobile & Media Systems LabHP Labs
Stanford EE392J Lecture
![Page 2: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/2.jpg)
2 28 February 2007
Compressed Domain Processing and Transcoding
SmartMedia Router
Full resolutionDecode and Display
Low resolutionDecode and Display
Low-bandwidthwireless link
High-bandwidthLAN
Capture andEncode
TranscoderProcessor
Transcoding: Media streaming over packet networksProcessing: Media processing in the compressed domain
![Page 3: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/3.jpg)
3 28 February 2007
High-quality video processing with low computational complexity and
low memory requirements.
010001101001101101100110101011100010 Process
MPEG stream MPEG stream
Goal:
Efficiently process MPEG video streams
Problem Statement
![Page 4: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/4.jpg)
4 28 February 2007
Outline
Background•
Manipulating Temporal Dependencies
•
Compressed-Domain Splicing
•
Compressed-Domain Reverse Play
•
MPEG2-to-H.263 Transcoding
•
Overview and Demo
![Page 5: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/5.jpg)
5 28 February 2007
P
GOP Layer Picture Layer
MacroblockLayer
BlockLayer
8x8 DCT4 8x8 DCT
1 MV
MPEG Structures
Slice Layer
BB
PB
BI
P B B I B B P B B P B B I
![Page 6: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/6.jpg)
6 28 February 2007
MPEG Structures
•
GOPs*•
Pictures*: Intraframe
(I frame), Forward predicted (P frame), Bidirectionally
predicted (B frame)
•
Slices*: (16x16n pixel blocks)•
Macroblocks
(16x16 pixel blocks):−
1 MV and 4 DCT blocks per MB (plus chrominance)•
I frame: Intra
MBs•
P frame: Intra or Forward
MBs•
B frame: Intra, Forward, Backward, or Bidirectional
MBs
•
Blocks
(8x8 pixel blocks): 8x8 DCT(*
start codes)
P B B I B B P B B P B B I
![Page 7: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/7.jpg)
7 28 February 2007
MPEG Coding Order
•
Preceding and following I or P frames (anchors) are used to predict each B frame.
•
“Anchor”
frames must be decoded before B data. (Note: In coding order, all arrows point forward.)
I0 B1 B2 P3Displayorder B4 B5 P6 B7 B8 I9
I0 B1 B2P3 B4 B5P6 B7 B8I9Codingorder
![Page 8: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/8.jpg)
8 28 February 2007
MPEG Bitstream
•
GOP header•
Picture header•
Picture data•
Pictures in coding order•
Start codes
(seq, GOP, pic, and slice start codes; seq
end code)
−
Unique byte-aligned 32-bit patterns−
0x000001nm 23 zeros, 1 one, 1-byte identifier−
Enables random access
10110010
I B B P B B P B B I
![Page 9: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/9.jpg)
9 28 February 2007
720 x 1280 pixels, 60 fps~ 1 Gbps
480 x 720 pixels, 30 fps~ 250 Mbps
Encode 100110101011100010
20 Gbytes / 2 hour movie~ 20 Mbps
5 Gbytes / 2 hour movie~ 5 Mbps
HDTV quality
Broadcast quality
Raw video MPEG stream
Video Compression
![Page 10: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/10.jpg)
10 28 February 2007
Encode 100110101011100010
Raw video1 Gbps
MPEG stream20 Mbps
Decode100110101011100010
Raw video1 Gbps
MPEG stream20 Mbps
20,000 MOPS
2,000 MOPS
MPEG Video Compression
![Page 11: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/11.jpg)
11 28 February 2007
ProcessDecode Encode
How do we process compressed video streams?
1 Gbps 1 Gbps 20 Mbps20 Mbps
2,000 MOPS 20,000 MOPS
Problem statement
•
Difficulties:−
Computational requirements: 22,000 MOP overhead from decode and encode operations (plus additional processing)
−
Bandwidth requirements: Need to process uncompressed data at 1 Gbps
−
Quality issues: Even without processing, the decode/encode cycle causes quality degradation.
![Page 12: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/12.jpg)
12 28 February 2007
010001101001101101100110101011100010 TranscodeMPEG stream MPEG stream
Naive Solution:
ProcessDecode Encode
How do we process compressed video streams?Use efficient compressed-domain processing (CDP) algorithms.
Ideal Solution:
1 Gbps 1 Gbps 20 Mbps20 Mbps
2,000 MOPS 20,000 MOPS
MPEG CDP
![Page 13: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/13.jpg)
13 28 February 2007
SmartMedia Server
TranscoderTranscoderProcessorProcessorTr
ansc
oder
Tran
scod
erPr
oces
sor
Proc
esso
r
Develop algorithms to perform equivalent spatial-domain video processing tasksusing fast algorithms that operate directly on the compressed-domain data.
Compressed-Domain Processing
OUR APPROACH
Decode Encode
2,000 MOPS > 20,000 MOPSConventional Approach
SPATIAL-DOMAINPROCESSING
MV+DCT-DOMAINPROCESSING
HuffmanDecode
HuffmanEncode
200 MOPS 200 MOPS
BITSTREAMPROCESSING
![Page 14: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/14.jpg)
14 28 February 2007
Splicing
Reverse Play
Frame-Level Video Processing
![Page 15: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/15.jpg)
15 28 February 2007
Difficulties and Solutions
•
High compression causes temporal dependencies−Frame conversion
•
Coding order vs. display order−Data reordering
•
Rate control / Buffer limitations−Requantization, Frame conversion
•
MPEG header information (e.g. time stamps, vbv_delay)−Rewrite header bits
![Page 16: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/16.jpg)
16 28 February 2007
Outline
•
BackgroundManipulating Temporal Dependencies
•
Compressed-Domain Splicing
•
Compressed-Domain Reverse Play
•
MPEG2-to-H.263 Transcoding
•
Overview and Demo
![Page 17: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/17.jpg)
17 28 February 2007
Manipulating Temporal Dependencies in Compressed Video
•
Video compression uses temporal prediction→ Prediction dependencies
in coded data
•
Many applications require changes in the dependencies
•
Manipulating (modifying) temporal dependencies in the compressed video [Wee]−Adding−Removing−Changing
![Page 18: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/18.jpg)
18 28 February 2007
Temporal Prediction
Original Video Frames:
Coded Video Frames:
Each frame Fi
is coded with two components1. Prediction
Anchor frames
Side information
2. Residual
Coded ResidualReconstructed Frame
{ }n21 F̂,...,F̂,F̂ ˆ =F
( )ii S,ˆP iA
iS{ }1-i21 F̂,...,F̂,F̂ ˆ ⊆iA
{ }n21 F,...,F,F=F
( )iiii S,ˆP - FR iA=
( ) iiii R̂S,ˆP F̂ += iAiR̂
![Page 19: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/19.jpg)
19 28 February 2007
Prediction Dependencies
Each coded frame is dependent
upon its anchor frames.
Prediction Depth
( ) iiii R̂S,ˆP F̂ += iA
F1
F2 F3
F4
F5 F6
F7
F10
Level 0
Level 1
Level 2
Level 3
F8 F9Level 4
![Page 20: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/20.jpg)
20 28 February 2007
Manipulating Dependencies
Fewer levels of dependence
Remove dependence between frames
F1
F2 F3
F4
F5 F6
F7Level 0
Level 1
F10
F8 F9
F1
F2 F3
F4
F5
F6
F7
F10
Level 0
Level 1
Level 2 F8 F9
![Page 21: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/21.jpg)
21 28 February 2007
Problem Formulation
Compression algorithm has a set of prediction rules.
Choose prediction modes during encoding.
Each choice yields:different compressed representation of frame Fi ,
different distribution between P & R components,
different set of prediction dependencies.
iS,ˆiA
( ) iiii R̂S,ˆP F̂ += iA
'iS,ˆ '
iA( ) '
i'i
'i
'i R̂S,ˆP F̂ += '
iA
![Page 22: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/22.jpg)
22 28 February 2007
Manipulating Dependencies
Given a compressed representation with dependencieshow do we create a new compressed representation with
dependencies ?
•
Reconstruct frames
•
Compressed-domain approximation
•
Calculate new prediction and residual
( ) iiii R̂S,ˆP F̂ += iA
( )'i'ii
'i S,ˆPF R '
iA−=
iS,ˆiA
'iS,ˆ '
iA
ii F F̂ ≈
( ) ( )'i'iiii
'i S,ˆPS,ˆPR̂ R '
ii AA −+≈
![Page 23: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/23.jpg)
23 28 February 2007
P Bb Bb I B B P
P B B I B B P
P B B I B B I
P Bf Bf I B B P
P to I
B to Bfor
B to Bback
Original
Frame Conversions: Remove Dependencies
![Page 24: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/24.jpg)
24 28 February 2007
P B B I B B P
P B B P B B PI to P
Original
Frame Conversion: Add Dependencies
1. Find and code new MVs.
MV Resampling
2. Calculate new prediction.
3. Calculate and code new residual.
![Page 25: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/25.jpg)
25 28 February 2007
Compressed-Domain Processing
MPEG standard addresses prediction rules, buffer requirements, coding order, and bitstream syntax.
MPEG CDP steps•
Determine and perform appropriate frame conversions (i.e. manipulate dependencies).
•
Reorder data.•
Perform rate matching.•
Update header information.
001011011010011001 110001011010110100CDP
![Page 26: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/26.jpg)
26 28 February 2007
Outline
•
Background•
Manipulating Temporal DependenciesCompressed-Domain Splicing
•
Compressed-Domain Reverse Play
•
MPEG2-to-H.263 Transcoding
•
Overview and Demo
![Page 27: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/27.jpg)
27 28 February 2007
100110101011100010 001011011010011001 110001011010110100
Decode Decode Encode
Cut & Paste
Transcode
Compressed-Domain Splicing
Application: Ad Insertion for DTV
![Page 28: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/28.jpg)
28 28 February 2007
Splicing: SMPTE standard
•
SMPTE formed a committee on splicing.•
Disadvantages:−User must predefined splice points during
encoding⇒ Complicated encoders.
−Splice points can only occur on I frames•
Not frame accurate.
•
Advantages:−Simple cut-and-paste
operation.
![Page 29: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/29.jpg)
29 28 February 2007
The TV Newsroom
IEEE Spectrum
![Page 30: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/30.jpg)
30 28 February 2007
100110101011100010 001011011010011001 110001011010110100
Decode Decode Encode
Cut & Paste
Transcode
2,000MOPS
2,000MOPS
20,000MOPS
< 2,000 MOPSfor one splice point every sec
Compressed-Domain Splicing
![Page 31: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/31.jpg)
31 28 February 2007
Splicing: Conventional approach
Decode
Decode
Cut &Paste Encode
00101101101001
00101101101001
110001011010112,000 MOPS
20,000 MOPS
2,000 MOPS
1 Gbps20 Mbps
20 Mbps
20 Mbps
1 Gbps
1 Gbps
Splice00101101101001
10010110011001
11000101101011
> 24,000 MOPS for one splice point every sec
![Page 32: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/32.jpg)
32 28 February 2007
Compressed-Domain Processing
Can we do better by exploiting properties of1) the MPEG compression algorithm
and2) the splicing operation ?
![Page 33: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/33.jpg)
33 28 February 2007
Splicing: Our approach
ProcessHead
ProcessTail
Match &Merge
00101101101001
10010110011001
11000101101011
CD Splice00101101101001
10010110011001
11000101101011
< 2,000 MOPS for one splice point every sec
![Page 34: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/34.jpg)
34 28 February 2007
Splicing Algorithm (simplified overview)
Only process the GOPS affected by the splice.Step 1: Process the head stream.•
Remove (backward) dependencies on dropped frames.
Step 2: Process the tail stream.•
Remove (forward) dependencies on dropped frames.
Step 3: Match & Merge the head and tail data.•
Perform rate matching.•
Reorder data appropriately.•
Update header information.
![Page 35: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/35.jpg)
35 28 February 2007
P Bb Bb I Bb Bb P
P B B I B B P
P B B I B B I
P Bf Bf I Bf Bf P
P to I
B to Bfor
B to Bback
Original
Frame Conversion: Remove Dependencies
1. Eliminate temporal dependencies.
2. Calculate new prediction (DCT-domain).
3. Calculate and code new residual.
![Page 36: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/36.jpg)
36 28 February 2007
Splicing
GOP with splice-out frame
Head Input Stream
X X
GOP with splice-in frame
Tail Input Stream
X
Spliced Output Stream
Hn-2 Hn-1 Hn
T0 T1 T2 T3
T1 T2Hn-2 Hn-1 F(Hn,T0)
![Page 37: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/37.jpg)
37 28 February 2007
ResultsSplicing between two sequenceswith one splice every 20 frames
![Page 38: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/38.jpg)
38 28 February 2007
Splicing: Remarks
•
Proposed Algorithm−Frame-accurate splice points−Only uses MPEG stream (Encoder is not affected.)−Frame conversions in MV+sparse DCT domain.−Rate control by requantization
and frame
conversion. (Do not insert synthetic frames.)−Quality only affected near splice points.−Computational scalability: Video quality can be
improved if extra computing power is available.
![Page 39: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/39.jpg)
39 28 February 2007
Outline
•
Background•
Manipulating Temporal Dependencies
•
Compressed-Domain Splicing
Compressed-Domain Reverse Play
•
MPEG2-to-H.263 Transcoding
•
Overview and Demo
![Page 40: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/40.jpg)
40 28 February 2007
Reverse-Play Transcoding
Develop computation-
and memory-efficient transcoding algorithms for reverse play
of a
given forward-play MPEG video stream.
001011011010011001 110001011010110100
Encode
Store &Reorder
Transcode
Decode
![Page 41: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/41.jpg)
41 28 February 2007
Reverse-Play Architecture #1
•
High memory requirements−Frame buffer must store 15 frames
•
High computational requirements−Motion estimation dominates computational needs
Decode(15)
Encode(15)
Reorder(15)
FrameBuffer
MotionEstimation
I, P, BI, P, B
#1
GOP 15
![Page 42: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/42.jpg)
42 28 February 2007
Compressed-Domain Processing
Can we do better by exploiting properties of1) the MPEG compression algorithm
and2) the reverse-play operation ?
![Page 43: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/43.jpg)
43 28 February 2007
161514
13
1211
109
87
5
432
16
B-Frame Symmetry: Swap MVs
161514
13
1211
109
87
5
432
1
1615
1413
1211
109
87
5
43
21
161514
13
121110
9
87
5
4321
66 6
MV1 MV2
1615
1413
1211
109
87
5
43
21
161514
13
121110
9
87
5
4321
6 6MV2 MV1
ForwardPlay
ReversePlay
Anchor Frame 1 B Frame Anchor Frame 2
![Page 44: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/44.jpg)
44 28 February 2007
Reverse-Play Architecture #2
•
Reduced memory requirements−
Frame buffer must store 5 frames
•
Reduced computational requirements−
ME still dominates computational needs
Decode(5)
Encode(5)
Reorder(5)
FrameBuffer
MotionEstimation
HuffmanDecode
HuffmanEncode
SwapMVs
I, P
B
I, P
B
#2
Exploit symmetry of B frames.
![Page 45: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/45.jpg)
45 28 February 2007
Reverse-Play Architecture #3
•
Reduced memory requirements•
Reduced computational requirements
Decode(5)
Reorder(5)
FrameBuffer
ReverseMVs
HuffmanDecode
HuffmanEncode
SwapMVs
I, P
B
I, P
B
Encode(5)
#3
Exploit forward MVF in coded data.
![Page 46: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/46.jpg)
46 28 February 2007
Search Area: No correspondenceForwardsearcharea
Reversesearcharea
Time T0
Time T1
Time T0
Time T1
Interpret MVs as specifying a match between blocks.Develop motion vector resampling methods.
![Page 47: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/47.jpg)
47 28 February 2007
In-place Reversal Method
Forward MV
Reverse MVIn-placereversal
![Page 48: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/48.jpg)
48 28 February 2007
Maximum Overlap Method
Overlap weights
W(-1,-1)
W(0,0)W(1,0)
Local neighborhood
MV(-1,-1)W(-1,-1)
MV(0,-1)W(0,-1)
MV(1,-1)W(1,-1)
MV(-1,0)W(-1,0)
MV(0,0)W(0,0)
MV(1,0)W(1,0)
MV(-1,1)W(-1,1)
MV(0,1)W(0,1)
MV(1,1)W(1,1)
Other MV resampling methods exist.
![Page 49: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/49.jpg)
49 28 February 2007
Decode(15)
Encode(15)
Reorder(15)
FrameBuffer
MotionEstimation
I, P, BI, P, B#1GOP 15
Decode(5)
Encode(5)
Reorder(5)
FrameBuffer
MotionEstimation
HuffmanDecode
HuffmanEncode
SwapMVs
I, P
B
I, P
B
#2
Decode(5)
Reorder(5)
FrameBuffer
ReverseMVs
HuffmanDecode
HuffmanEncode
SwapMVs
I, P
B
I, P
B
Encode(5)
#3
Computational Requirements
#1: 4200 McyclesLog Search ME
#2: 1030 McyclesLog Search ME
#3a: 420 McyclesMaximum Overlap
#3b: 330 McyclesIn-place Reverse
![Page 50: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/50.jpg)
50 28 February 2007
Experimental Results
Girl
BusCarousel
Football
0.65 dB
0.2 dB
![Page 51: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/51.jpg)
51 28 February 2007
Observations•
Girl sequence showed largest PSNR loss (.65 dB) due to high detail and texture in source.−MV accuracy is important!
•
Bus sequence benefits from maximum overlap because of large motions in source.
•
Carousel has little performance loss because block MC does not match motion in source.
•
Football has little performance loss due to blurred source.−When MV accuracy is not important, fast
approximate methods do not siginificantly sacrifice quality.
![Page 52: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/52.jpg)
52 28 February 2007
Reverse Play Summary•
Developed several compressed-domain reverse-play transcoding algorithms.
•
CDP: Exploit properties of compression algorithm and reverse-play operation−Exploit symmetry of B frames.−Exploit MVF information given in forward stream.
•
Order of magnitude reduction in computational requirements.
•
Worst case 0.65 dB loss in PSNR quality over baseline transcoding.
![Page 53: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/53.jpg)
53 28 February 2007
Outline
•
Background•
Manipulating Temporal Dependencies
•
Compressed-Domain Splicing
•
Compressed-Domain Reverse Play
MPEG2-to-H.263 Transcoding
•
Overview and Demo
![Page 54: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/54.jpg)
54 28 February 2007
Motivation
•
Video communication
requires the seamless delivery of video content to users with a broad range of bandwidth and resource constraints.
•
However, the source signal, communication channel, and client device
may be incompatible.
•
Therefore, efficient transcoding algorithms
must be designed
001011011010011001 110001011010110100TranscoderStandard-compliant
input bitstreamStandard-compliant
output bitstream
![Page 55: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/55.jpg)
55 28 February 2007
MPEG2-to-H.263 Transcoding
MPEG-to-H.263 Transcoder• Transcode MPEG video streams to lower-rate H.263 video streams.
• Interlace-to-progressive conversion.• Order of magnitude reduction in computational requirements.
Stream DVD movies to portable
multimedia devices.
MediaServer
Low-bandwidthwireless linkTranscoder
TranscoderStream DTV program
material over the internet.
Media ServerDTV or DVD content
![Page 56: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/56.jpg)
56 28 February 2007
Problem Statement
Develop a fast transcoding algorithm that adapts the bitrate, frame rate, spatial resolution, scanning format, and/or coding
standard while preserving video quality
Important features:−
Interlaced-to-progressive
transcoder.−
Inter-standard
transcoder, e.g. MPEG-2 to H.263 (or MPEG-4)
Contributors: Wee, Apostolopoulos, Feamster
(MIT)
MPEG-2 input• High bitrates• DVD, Digital TV• >1.5 Mbps, 30 fps• Interlaced and progressive
H.263 output• Low bitrates• Wireless, internet• <500 kbps, 10fps• Progressive
![Page 57: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/57.jpg)
57 28 February 2007
Difficulties
•
High cost of conventional transcoding
•
Differences in video compression standards•
Interlaced vs. progressive formats
MPEG-2Decode
ProcessFrames
H.263Encode
001011011010011001 110001011010110100
H.263 bitstreamMPEG-2 bitstream
![Page 58: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/58.jpg)
58 28 February 2007
Issue: Standard Differences
MPEG-2 H.263 Video Formats
Progressive and Interlaced
Progressive Only
I frames More (enable random access)
Fewer (compression)
Frame Coding Types
I, P, B frames I, P, Optional PB frames
Prediction Modes
Field, Frame, 16x8
Frame Only
Motion Vectors
Inside Picture Only
Can point outside picture
![Page 59: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/59.jpg)
59 28 February 2007
MPEGDecode
TemporalProcessing
SpatialProcessing
H.263Encode
TemporalProcessing
MPEGDecode
SpatialProcessing
TemporalProcessing
MPEGDecode
SpatialProcessing
H.263Encode
MotionEstimation
H.263EncodeMotion
Estimation
CalculateMVs
Conventional
Proposed Approach: Also exploit input coded data!
Improved Approach: Exploit bitstream syntax!
Development of Proposed Approach
![Page 60: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/60.jpg)
60 28 February 2007
Block Diagram
DropB frames
MPEG IPDecoder
SpatialDownsample
H.263 IPEncoder
EstimateMV
RefineSearchMPEG MVs &
Coding ModesH.263 MVs &Coding Modes
![Page 61: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/61.jpg)
61 28 February 2007
Issue: Match Prediction Modes•
Match MPEG-2 Fields to H.263 Frames
•
Vertical downsampling
=> discard bottom field•
Horizontal downsampling
=> downsample
top field•
Temporal downsampling
=> drop B frames•
Match MPEG-2 IPPPPIPPPP to H.263 IPPPPPPPPP−
Problems•
Convert MPEG-2 P fields to H.263 P frames•
Convert MPEG-2 I fields to H.263 P frames
I(0,t)P(0,b)
B(1,t)B(1,b)
B(2,t)B(2,b)
P(3,t)P(3,b)
B(4,t)B(4,b)
B(5,t)B(5,b)
i(0) p(1) p(2)
I(6,t)P(6,b)
B(7,t)B(7,b)
MPEG-2 Fields
H.263 Frames
![Page 62: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/62.jpg)
62 28 February 2007
Outline
•
Background•
Manipulating Temporal Dependencies
•
Compressed-Domain Splicing
•
Compressed-Domain Reverse Play
•
MPEG2-to-H.263 Transcoding
Overview and Demo
![Page 63: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/63.jpg)
63 28 February 2007
1001101010111 0010110110100 1100010110101
Decode Decode Encode
Cut&Paste
Transcode
2,000MOPS
2,000MOPS
20,000MOPS
< 2,000 MOPSfor one splice point every sec
Compressed-Domain Splicing
SMPTE splicing solution• User must define splice points during encoding
Specialized complex encoders. • Restricted splice points
Enables Ad
Insertion for DTV
HP splicing solution• Works with any MPEG stream.• Frame-accurate splice points
![Page 64: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/64.jpg)
64 28 February 2007
Reverse-Play Transcoding
00101101101001 11000101101011
Encode
Store &Reorder
Transcode
Decode 2,000MOPS
20,000MOPS
< 1,000 MOPS
HP reverse-play solution• Frame-by-frame reverse play.• Works with any MPEG stream.• Order of magnitude reduction in computational requirements.
VCR functionality for DTV and
Set Tops
![Page 65: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/65.jpg)
65 28 February 2007
MPEG2-to-H.263 Transcoding
MPEG-to-H.263 Transcoder• Transcode MPEG video streams to lower-rate H.263 video streams.
• Interlace-to-progressive conversion.• Order of magnitude reduction in computational requirements.
Stream DVD movies to portable
multimedia devices.
MediaServer
Low-bandwidthwireless linkTranscoder
TranscoderStream DTV program
material over the internet.
Media ServerDTV or DVD content
![Page 66: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/66.jpg)
66 28 February 2007
ReferencesReferences:•
“Manipulating Temporal Dependencies in Compressed Video Data with
Applications to Compressed-Domain Processing of MPEG Video”, S. Wee, ICASSP 1999.
•
“Splicing MPEG Video Streams in the Compressed Domain”, S. Wee, B. Vasudev, IEEE Workshop on Multimedia Signal Processing, 1997.
•
“Compressed-Domain Reverse Play of MPEG Video Streams”, S. Wee, B. Vasudev, SPIE Inter. Sym. On Voice, Video, and Data Communications, 1998.
•
“Reversing Motion Vector Fields, S.J. Wee, IEEE International Conference on Image Processing, October 1998.
•
“Field-to-frame Transcoding with Spatial and Temporal Downsampling”, S. Wee, J. Apostolopoulos, N. Feamster, ICIP 1999.
•
“Compressed-Domain Video Processing”, S. Wee, B. Shen, J. Apostolopoulos, HP Labs Technical Report, 2002, CRC Handbook of Video Databases, 2004.
http://www.hpl.hp.com/techreports/2002/HPL-2002-282.html
![Page 67: Compressed-Domain Video Processing and Transcoding](https://reader035.vdocument.in/reader035/viewer/2022070222/613d6cf4736caf36b75d2f4c/html5/thumbnails/67.jpg)