hvs-dbs: human visual system-aware dynamic luminance backlight scaling … fine... ·...
TRANSCRIPT
HVS-DBS:Human Visual System-aware Dynamic Luminance Backlight Scaling for Video
Streaming Applications
Andrea Bartolini, University of Bologna, DEIS
2
Outline
• Introduction
• Dynamic Backlight Scaling (DBS)
• Related work
• HVS-DBS - Idea
• HVS-DBS - Framework Implementation
• Conclusions
3
IntroductionEvery modern portable handheld device is equipped with a color LCD display:
• Increasing panel size
• Power consumption proportional to the panel area
• LCD power consumption is one of the main limiters to battery lifetime
Proc
LCD
RF
Proc
LCD
RF
RF
CPU
LCD
(Voice only dev) (Smartphone) (Gaming dev)
Proc
LCD
RF
Mobile device power breakdown from a legacy mobile device to a Smartphone/multimedia mobile device
to a gaming targeted mobile device. [1]
[1] “POWER MANAGEMENT IN MOBILE DEVICES”, Findlay Shearer, ELSEVIER
Video System Power
Breakdown [1]
4
Dynamic Backlight ScalingLCD emitted light is function
of 2 parameters:
• Backlight intensity
↓ power saving ↓ darker image
• LCD pixels trasmittance
↑ lighter image
LCD
Power Backlight
Pixel
trasmittance
final user
Power
Saving
Same
image
Ideally we can achieve the sameperception at human eyes with lowerbacklight intensity and higher LCD tramittance, while saving power.
In practice due to display system non-idealities image degradation.
Distortion
!!!
How to control & keep it constant for all the rendered images ??
Overall quality loss depends on :• pixel trasformation• pixel value• final backlight intensity level• LCD display electrical and optical properties• human visual system (HVS) features
5
How to control QoS degradation
Two main approaches in state-of-the-art DBS tecniques:• Frame Dependent: [1] [2] [3] [4]
• It computes on-line the amount of backlight scaling on every frame using a simpleimage distortion metric as constraint.
• The common approach is to on-line create the frame luminance histogram to keep constant the number of pixel that will saturate inside the increased luminance frame
• The simple performance metric adopted doesnot account for both HVS peculiarities and display system non-idealities weak relationship between the adopted metric and real perceived degradation
[1] “DLS: dynamic backlight luminance scaling of liquid crystal display.”, Chang N. Et al, IEEE Trans. VLSI Systems 2004
[2] “A Backlight Power Management Framework for Battery-Operated Multimedia Systems”, Shim H. et al, IEEE Des. Test 2004[3] “Quality Adapted Backlight Scaling (QABS) for Video Streaming to Mobile Handheld Devices, Cheng L. et al
[4] “Perception-guided power minimization for color sequential displays”, Cheng W et al, ACM Symposium on VLSI. 2006
Goal
Distortion
Input ImageDistortion
LevelProcessing
(not HVS based)
LuminaceCompensation
BacklightScaling
LCD
Backlight
# of saturated pixels
6
How to control QoS degradation
Two main approaches in state-of-the-art DBS tecniques:• Frame Independent: [1] [2] [3] [4]
• Very complex HVS aware frame processing algorithm it can not be evaluated on-line.
• The relationship between image distortion and BL scaling is statistically analysed off-line.• Uses a set of benchmark images to calculate an image-independent empirical function
which relates the image degradation levels to the transformation parameter applied to the image itself.
• Image degradation dependent only on the applied BL scaling.
• This approach does not consider the perceived image distortion as a function of the image itself.
[1] “Dynamic tone mapping for backlight scaling.”, Iranli A. and Pedram M., DAC 2005
[2] “HEBS: Histogram Equalization for Backlight Scaling”, Iranli A. et al, DATE 2005[3] “Backlight dimming in power aware mobile displays, Iranli A. et al, DAC 2006
[4] “HVS-Aware Dynamic Backlight Scaling in TFT-LCDs”, Iranli A. et al, IEEE Trans. VLSI Systems 2006
7
Our approach & main contributions
We are focused on:• Video streaming applications
• QoS-Power saving tread-off
• Frame dependent processing algorithm
• Overcome the QoS limitations of today DBS techniques
• Low power overhead
• Real implementation
• Freescale i.MX31 eval. board + 3.5” SHARP TFT LCD Display
Our technique is:
• Human Visual System – and Display non-ideality aware
• Capable of keeping final QoS under control
• No impact on video performance and low power overhead
• We exploit in a smart and efficient way existing hardware facilities
• no extra HW or display modification are required
We introduced a new way to predict on-line the final distortion for backlight scaling techniques
8
HVS-DBS – ideaOur approach is • Frame dependent
• Uses a HVS-aware on-line distortion metric
• Based on a try-and-test approach to find the minimum backlight level that guaranties the goal QoS performance
Target
QoS
check
Input
frame
DBS
trasformation
α, backlight
No !Update
α, backlight
HVS-aware
QoS evaluation
Yes !
Apply to LCD
α, backlight
Block Diagram1. The incoming frame
pixels luminance is increased with a starting DBS parameter set
2. HVS-aware QoS evaluation: distortion between the original frame and the DBS transformed one
3. Is the target QoS we want ?
a. No !!
• Update α, backlight
• Re-iterate 1,2,3
a. No !!
b. Yes !!
• Apply α, backlight to the LCD
Due to real time constraints all these steps must be executed before
next frame has been decoded !!
This bound the number of iterations and (α, backlight ) possible
value !!
Iteractive section
9
• The i.MX31 represents the next step in high performance application processors
• Based on an ARM11 core
• Multi-Media and Floating-Point HW Acceleration supporting:• MPEG4 real-time encode of up to VGA at 30 fps
• 3D Graphics and other applications acceleration
• Image Processing Unit (IPU)
• IPU is designed to:
• interface to video/still image sensors and displays
• IPU main functions:
• Capturing image
• Preprocessing & Postprocessing:
– Color space format conversion
– Combining video and graphics planes
• Post-filtering
• Displaying video and graphics on:
– Synchronous Displays
– Asynchronous Displays
– TV
• The IPU main functionality is present in every new multimedia processor:
• the HVS-DBS framework can be easily ported on other different platforms.
Freescale i.MX31
10
Framework implementation
Video decoder
Input
Video
Stream
MemBuffer
Post-processing
Color Space Conversionand image resizing
LCD Display
Video chain original
1. Input Frame coming from
video decoder is memorized
in a memory buffer
2. The Video driver send it to
the IPU post-processing to
adapt its color format and
size to the LCD Display
3. Then the frame is displayed
on the LCD pannel
11
Video chain HVS-DBS
Framework implementation
Video decoder
Input
Video
Stream
Proc. rate
MemBuffer
Down-sampling
QoS Embedded
Framework
search engine
Target QoSreached?
Target QoS
Post-processing
Color Compensation on Original Frame
and
LCD Backlight Scaling
Pre-processing
DBS Trasnformationon DownSampled
Frame
α0
Backlight 0
New values
Backlighti+1 & αi+1
NO!
Backlight i+1
αi+1
LCD Display
Apply
BacklightBEST & αBEST
YES!
IDS
IPREP
QoSi
Iteractive
section
1. YES! Select the BEST α and Backlight pair that
match target QoS and send it to Post-process stage
2. The original frame (Full Dimension) pixels are
luminance increased of α BEST factor with IPU HW
3. Send the Enhanced frame to the LCD display and
Backlight BEST value
Software routine
executed by the CPU !!
Executed in HW
by the IPU !!1. Each N frames the frame is
downsampled to reducing it dimension
2. The downsampled frame is luminance
enhanced using the IPU HW CSC block
3. The QoS Embedded Framework
computes the QoS score between the
Downsampled and the pre-processed
frames
4. Is this QoS score the one we want?
5. NO ! Try with another α and Backlight
pair
12
QoSi
IDS
IPREP
Framework implementation
Video decoder
Input
Video
Stream
Proc. rate
MemBuffer
Down-sampling
QoS Embedded
Framework
search engine
Target QoSreached?
Target QoS
Post-processing
Color Compensation on Original Frame
and
LCD Backlight Scaling
Pre-processing
DBS Trasnformationon DownSampled
Frame
α0
Backlight 0
New values
Backlighti+1 & αi+1
NO!
Backlight i+1
αi+1
LCD Display
Apply
BacklightBEST & αBEST
YES!
Post-process and pre-process routine
• Increase the input image pixels luminosity
• Require to multiply all the image pixels for a matrix
)(
00
00
00
)( RGBpixelRGBpixel
B
G
R
ENHANCED
too expensive in
CPU !!!
ENERGY EFFICIENCY
REAL-TIME PERFORMANCE
REQUIREMENT
Executed by Image Processing Unit HW
Color Space Conversion module
Low performance and
power overhead !!!
13
IDS
IPREP
QoSi
Framework implementation
Video decoder
Input
Video
Stream
Proc. rate
MemBuffer
Down-sampling
QoS Embedded
Framework
search engine
Target QoSreached?
Target QoS
Post-processing
Color Compensation on Original Frame
and
LCD Backlight Scaling
Pre-processing
DBS Trasnformationon DownSampled
Frame
α0
Backlight 0
New values
Backlighti+1 & αi+1
NO!
Backlight i+1
αi+1
LCD Display
Apply
BacklightBEST & αBEST
YES!
14
QoS embedded framework
QoS
Embedded Framework
LCD Model
RGB2IPT
LCD Model
RGB2IPT
SSIM index
It implements an embedded version of the
Visual QoS framework presented in [1].
2. RGB2IPT
• It transforms LCD Model output images from RGB to IPT colour space
• IPT accounts for human eyes different sensibilities for different colours
3. SSIM index routine
• models the HVS peculiarities and evaluates the differences between the two images from the LCD model block in IPT color space
• based on the assumption that the human visual system is efficient in extracting structural information from the viewing field
• It has been demonstrated to be able to quantify the HVS perceived differences between two images.
Basic Blocks:1.LCD model
• models the behaviour of a real embedded LCD panel and its non-ideality
• Input: (RGB image & backlight level)
• Output: image that evaluate the final rendering on the LCD display
[1] “Visual quality analysis for dynamic backlight scaling in LCD systems”, Bartolini A. Ruggiero M. and Benini L., DATE 2009
IDS IPREP
QoSi
15
LCD Model and RGB2IPT
QoS
Embedded Framework
SSIM index
What inside the LCD Model and RGB2IPT conversion ?
• LCD gamma & pixels saturation model
• Display image rendering estimation
• LMS cones responce color space rapresentation,
describe human eyes cone sensitivity suitable for detecting color adaptation artifact
• IPT color space rappresentation
• Enhanced version of CIELAB
• Directly related to percieved lightness, hue
and saturation
BACKLIGHT LEVEL
43.0|| R
G
B
43.0||
43.0||
I
P
T
Not linear !! Linear
X
Y
Z
L
M
S
LCD Model
RGB2IPT
LCD Model
RGB2IPT
too expensive with
only CPU ALU
operations !!!
Input frame format:
• RGB565
•5 bit red & blue
•6 bit green
Set of Look-up-Table (LuT)
• Accessed with the RGB565 pixel value
• Contains the I,P,T value (8bit each)
• One for each backlight level allowed 9
levels
16bit
We can
compute
SSIM only
over I
LuT need to
memorize
only I
Saving
memory
Improove
performance
IDS IPREP
QoSi
16
SSIM index
QoS
Embedded Framework
What inside the SSIM index ?
• Too high computational cost !!
• Cost dominated by the 2D filter
routine input size
• Input image down-sampled
• Gaussian window must be
scaled down too to preserve the
same locality information
• Fixed point aritmetic
• Error map must be averaged
• Downsample it before
reducing the average and
filters cost
LCD Model
RGB2IPT
LCD Model
RGB2IPT
SSIM index
2D Filter
2D Filter
2D Filter
2D Filter
2D Filter
-+-+ -+
ImO ImDLS
ImO2 ImDLS
2
µDLS
µDLS2µOµDLS
µO
µO2
σO2 σDLS
2σOσDLS
22
2
DLSO
DLSO
22
2
DLSO
DLSO
avg SSIM
Error Map
IDS IPREP
QoSi
17
IDS
IPREP
QoSi
Framework implementation
Video decoder
Input
Video
Stream
Proc. rate
MemBuffer
Down-sampling
QoS Embedded
Framework
search engine
Target QoSreached?
Target QoS
Post-processing
Color Compensation on Original Frame
and
LCD Backlight Scaling
Pre-processing
DBS Trasnformationon DownSampled
Frame
α0
Backlight 0
New values
Backlighti+1 & αi+1
NO!
Backlight i+1
αi+1
LCD Display
Apply
BacklightBEST & αBEST
YES!
Search engine
• Input : current SSIM index score
• It checks the conditions : SSIMi > SSIMTARGET
• SSIM index monotonically decreases on α
• It searches on a binary tree the future (αi, Backlighti )
• Binary search tree dept limited to real time constraints and performance
trade-off => 3 iteration => 9 Backlight levels
18
Experimental results
Framework implementation performance :
• Input test videoproperties:
• Different frame contents :
• Indoor scene darker pixels
• Outdoor scene lighter pixels
• HVS-DBS Power overhead
< Frame time budget !!
CPU 11%
Peripheral 1%
Main memory 13%
Δ T Frame-processing 17,3 ms
19
Histogram based DBS performance
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
20% 8% 4%
% o
f v
ide
o f
ram
es
fo
r a
Qo
S
reg
ion
% of saturated pixels
Very low Low Medium High
QoS regions
Terminator3 frames for QoS regions Frame dependent (not HVS)
Luminance histogram technique
keeps the saturated pixels on
each frames below a specific
threshold
Bad frame even with
conservative settings !!!
There is always a noticeable
image distortion !!!
0%
5%
10%
15%
20%
25%
30%
20% 8% 4%
Po
wer
savin
g
% of saturatedpixels
Terminator 3 system power
Total video Indoor section Outdoor section
Entire system power
saving
• CPU
• Memory
• Peripheral• LCD
Power saving depends
on the frame content !!
Saturated pixels
histogram smalloverhead up 25%
power saving
20
Entire system power
saving
• CPU
• Memory
• Peripheral• LCD
HVS-DBS performance
0%
5%
10%
15%
20%
25%
Low Medium High Best
Po
wer
savin
g
Target SSIM
Terminator 3 system power
Total video Indoor section Outdoor section
QoS regions
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Low Medium High Best% o
f v
ide
o f
ram
es
fo
r a
Qo
Sre
gio
n
Target SSIM
Terminator3 frames for QoS regions
Very low Low Medium High
Absence of bad frames in
Medium – High – Best
settings
Fine grain tuning of the
final QoS !!!
HVS-DBS
Up 5% of system power saving
with no Image distortion
Up to 20% of system powersavings with low quality mode
allowed
21
Conclusion
• We proposed a novel dynamic backlight scaling approach to overcome state-of-the-art DBS techniques limitations
• Our solution is based on a on-line HVS-aware metric to findthe optimal backlight level for a specified QoS
• We provide a real implementation with reduced complexity in a multimedia embedded processor
• No video performance impact
• Significant system power savings - 5% to 20%
• Our HVS-DBS is robust and finds the optimal trade-offbetween QoS and power savings
22
Pubblications
• A. Marongiu, L. Benini, A. Acquaviva, A. Bartolini. Analysis of Power Management Strategies for a Large-Scale SoC Platform in 65nm Technology. In DSD 2008
• M. Ruggiero, A. Bartolini, L. Benini. DBS4video: Dynamic Luminance Backlight Scaling based on Multi-Histogram Frame Characterization for Video Streaming Application. In EMSOFT 2008
• A. Bartolini, M. Ruggiero, L. Benini. Visual quality analysis for dynamic backlight scaling in LCD systems. In DATE 2009
• A. Bartolini, M. Ruggiero, L. Benini. HVS-DBS: Human Visual System-aware Dynamic Luminance Backlight Scaling for Video Streaming Applications. In EMSOFT 2009