evaluation of bandwidth performance for interactive spherical video
Post on 28-Nov-2014
1.977 Views
Preview:
DESCRIPTION
TRANSCRIPT
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
Evaluation of Bandwidth Performance for Interactive Spherical VideoPatrice Rondao Alface, Jean-François Macq, Nico Verzijp
Bell Labs, Alcatel-LucentICME’s WoMAN’11, Barcelona, July 2011
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
2
AcknowledgmentFP7 FascinatE project
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
3
AGENDA
1. Introduction
2. Interactive Spherical Video Transmission
3. Transmission Optimization
4. Experimental Results
5. Conclusions
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
4
IntroductionInteractive Spherical Video
Heymann et al., “Representation, Coding & Interactive Rendering of High-Resolution Panoramic Images and Video using MPEG-4”, in Proc. Panoramic Photogrammetry Workshop (PPW), Berlin, Germany, Feb. 2005.
• Capture devices
6 to 12 HD views
• Panorama/Texture mapping on mesh sphere
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
5
IntroductionInteractive Spherical Video - Applications
• Immersive Media Cultural heritage…
• Entertainment – live events 360º Media Coverage
Music concerts
Sports events…
• Video-surveillance & Security
• Geographical Information Systems (GIS) Real Estate
Google Street View…
Image source: www.immersivemedia.com
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
6
Interactive Spherical Video TransmissionState of the Art
• Compression of the panorama as a video and implicit mapping on a sphere at the client’s renderer side
Panorama coding with uniform quality
MPEG-4 Visual (Smolic et al. 2003)
Flash/PaperVision3D
A. Smolic et al.,”Efficient Representation & Coding of Omni-directional Video using MPEG-4”, Proc. WIAMIS 2003, 4th European Workshop on Image Analysis for Multimedia Interactive Services, London, UK, April 9.-11. 2003.
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
7
Interactive Spherical Video Transmission
• Observations:
Field-of-View (fov) for a given user-selected orientation is limited to 45 - 60 degrees (vertical axis, aspect ratio gives the horizontal fov width)
More than the half of the bandwidth is wasted on parts of the panorama or views that are not viewed.
• Contribution:
Leveraging from the 2D case (Mavlankar et al. 2009), namely a tiling approach,
we propose to adapt the encoding for Interactive Random Access we propose to adapt the encoding for Interactive Random Access Viewing in live video transmission/adaptation scenariosViewing in live video transmission/adaptation scenariosA. Mavlankar, B. Girod, Background extraction and long-term memory motion-compensated
prediction for spatial-random-access-enabled video coding, Picture Coding Symposium 2009, pp.1-4, 6-8 May 2009
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
8
Interactive Spherical Video TransmissionSpherical Video Random Access
• Tiling on panorama:
the requested ROI does not map as in 2D to an equal number of tiles.
Visibility mask
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
9
Interactive Spherical Video TransmissionTiling
• Tile compression with adaptive quality
Tile Intra compression using JPEG2000 (quality constant on a tile)
Tiles coded independently at different rates (qualities) depending on their estimated visibility
Fixed bandwidth budget for the panorama
Whole panorama Visible tiles only (RoI) Possible impairment (interaction delay)
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
10
Transmission OptimizationTile-Based Quality Adaptation
• Study of four tile-based adaptation techniques with respect to transmission delay
• The delay induces an uncertainty about the actual requested ROI
• For a given bandwidth budget, bitrate is adapted on tiles based on following approaches:
• Motivation: when delay increases, a probabilistic approach should behave better
1. Constant rate2. Direct ROI encoding3. ROI with uniform quality background encoding4. ROI with probability-based adaptive quality background
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
11
Transmission OptimizationProposed probability-based approach
• Problem of optimizing the quality of the tiles in function of the user requests is modeled as an integer linear program
Given an offered bandwidth BW, the objective is to maximize, for each Given an offered bandwidth BW, the objective is to maximize, for each frame, the total utility of the selected tiles, hence minimizing their frame, the total utility of the selected tiles, hence minimizing their global MSE.global MSE.
Tile marginal utility
Tile at position i, quality qTile bitrate
Tile MSE
Tile marginal cost
Tile inserted at quality ≥ q
Probability of tile visibility
,
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
12
Transmission OptimizationProposed probability-based approach
• Relaxing the problem by considering the MSE of each tile as a convex function of its cost-bitrate ratio, and dropping the integrality constraint, this fractional knapsack can be solved by a greedy heuristic:
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
13
Transmission OptimizationProposed probabilistic Model of the Tile Visibility
Optimization of the tile selection under 2 classes of test scenarios
1. Ideal case:
• No delay
• Probability of visibility is known
• Knapsack optimization can directly use the visibility information
2. User interaction delay case
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
14
Results (case 1: no delay)
Tiling: client viewport quality vs. bandwidthRate-distortion for client viewport
30
35
40
45
50
55
0 200 400 600 800 1000 1200 1400
Client - Bandwidth budget (kB/frame)
Client viewport - PSNR (dB)
1x12x24x48x816x1632x32
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
15
Transmission OptimizationProbabilistic Model of the Tile Visibility
Optimization of the tile selection under 2 classes of test scenarios
1. Ideal case
2. User interaction delay case
• Delay assumed to be constant in the system
• If delay increases, the uncertainty around requested viewport position also increases
• The uncertainty around the visibility mask for the panorama tiles is characterized by a Gaussian with a standard deviation set proportional to the product of the navigation speed and the delay: σ v . d
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
16
Low delay: 40ms
Quality is higher close to the estimated viewport
High delay: 1280ms
Low probability for any viewport position; quality tends to be constant on the whole panorama
15dBbetterthanconstantbitrate
7dBbetterthanconstantbitrate
Results (case 2: user interaction delay)
75kB for 4Kx2K panorama
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
17
Results (case 2: user interaction delay)
PSNR vs. Delay for a fixed bandwidth 75KBs
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
18
Results (case 2: user interaction delay)
Discrepancy at the client side - delay of 1280ms
Discrepancy for a delay of 1280 ms
15,00
20,00
25,00
30,00
35,00
40,00
45,00
50,00
75000 150000 300000 600000 1200000
bandwidth (Bps)
PS
NR
DirectRoI
RoIandconstantqualitybackground
RoIwithadaptivequalitybackground
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
19
Conclusions
• Analysis of interactive spherical video delivery techniques.
• Received quality optimization under bw constraints and interaction delay.
• Proposed probability-based tile quality optimization method behaves better than state-of-the-art.
• Future work: enabling this adaptation for different live delivery architectures.
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
20
http://belllabs.be/internships
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
21
Transmission OptimizationExperimental Setting
• Point Grey Ladybug3 (6 HD cameras)
• Panorama: 4096x2048
• Viewport: 1280x960
• FOV: 45 degrees vertically
• Pan-Tilt interactions only
• Static camera setting
• Live Constant bitrate transmission with smooth user interactivity
• Live coding: JPEG2000 intra-picture coding-only for low delay
• Panorama partitioned into JPEG2000 independently coded tiles
• Analysis in terms of Interaction speed v and transmission delay d
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
22
Interactive Spherical Video TransmissionState of the Art
• Dodecahedron-based spherical patches optimized sampling for view coding using MPEG-2 (multi-texture approach)
Chi-Wing Fu et al., "The Rhombic Dodecahedron Map: An Efficient Scheme for Encoding Panoramic Video," Multimedia, IEEE Transactions on , vol.11, no.4, pp.634-644, June 2009
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
23
Interactive Spherical Video Transmission2D Random Access
• Example: Random Access for H.264
Interactive Regions of Interest (ROIs) are usually set at the encoder side by applying a regular grid of tiles on the video frames.
Content-based approaches (e.g. background/foreground) with FMO slices
Content-basedArbitrary
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
24
Results (case 2: user interaction delay)
Temporal analysis
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
25
Results (case 2: user interaction delay)
Delay 1280ms
Delay 40ms
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
26
top related