ph.d. defense analysis and design of vector error diffusion...

1

Analysis and Design ofAnalysis and Design ofVector Error Diffusion Systems forVector Error Diffusion Systems for

Image Image HalftoningHalftoning

Embedded Signal Processing Laboratory

The University of Texas at Austin

Austin TX 78712-1084

Niranjan DameraNiranjan Damera--VenkataVenkata

Committee MembersProf. Ross Baldick

Prof. Alan C. Bovik

Prof. Gustavo de Veciana

Prof. Brian L. Evans (advisor)

Prof. Wilson S. Geisler

Prof. Joydeep Ghosh

Ph.D. Defense

2

OutlineOutline

• Digital halftoning– Grayscale error diffusion halftoning

– Color error diffusion halftoning

• Contribution #1: Matrix gain model for color error diffusion

• Contribution #2: Design of color error diffusion systems

• Contribution #3: Block error diffusion – Clustered-dot error diffusion halftoning

– Embedded multiresolution halftoning

• Contributions

3

Digital Digital HalftoningHalftoning

Clutered Dot DitherAM Halftoning

Blue-noise MaskFM Halftoning 1993

Dispersed Dot DitherFM Halftoning

Green-noise HalftoningAM-FM Halftoning 1992

Error DiffusionFM Halftoning 1975

Direct Binary SearchFM Halftoning 1992

4

Grayscale Error DiffusionGrayscale Error Diffusion

• Shape quantization noise into high frequencies

• Two-dimensional sigma-delta modulation

• Design of error filter is key to high quality

current pixel

weights

3/16

7/16

5/16 1/16

+ _

_+

e(m)

b(m)x(m)

difference threshold

compute error

shape error

u(m)

)(mh

Error Diffusion

Spectrum

5

Modeling Grayscale Error DiffusionModeling Grayscale Error Diffusion

• Sharpening is caused by a correlated error image [Knox, 1992]

Floyd-Steinberg

Jarvis

Error images Halftones

6

Modeling Grayscale Error DiffusionModeling Grayscale Error Diffusion

• Apply sigma-delta modulation analysis to two dimensions– Linear gain model for quantizer in 1-D [Ardalan and Paulos, 1988]

– Linear gain model for grayscale image [Kite, Evans, Bovik, 2000]

– Signal transfer function (STF) and noise transfer function (NTF)

– 1 – H(z) is highpass so H(z) is lowpass

Ks

us(m)

Signal Path

Ks us(m)

Kn

un(m)

+

n(m)

Kn un(m) + n(m)

Noise Path, Kn=1

( ))(1

)(z

zz

HN

BNTF n −==

( )( ) ( )zz

zHK

K

X

BSTF

s

ss

11)( −+==

Q(.)u(m) b(m) {

7

Vector Color Error DiffusionVector Color Error Diffusion

• Error filter has matrix-valued coefficients

• Algorithm for adapting matrix coefficients[Akarun, Yardimci, Cetin 1997]

( ) ( ){ ( )43421

(

vectormatrix

kmekhmtk

−= ∑℘∈

+ _

_+

e(m)

b(m)x(m)


compute error

shape error

u(m)

)(mh(

t(m)

8

Color Error DiffusionColor Error Diffusion

• Open issues– Modeling of color error diffusion in the frequency domain

– Designing robust fixed matrix-valued error filters

– Efficient implementation

– Linear model for the human visual system for color images

• Contributions– “Matrix gain model” for linearizing color error diffusion

– Model-based error filter design

– Parallel implementation of the error filter as a filter bank

9

Contribution #1:Contribution #1:

The Matrix Gain ModelThe Matrix Gain Model

• Replace scalar gain with a matrix

– Noise is uncorrelated with signal component of quantizer input

– Convolution becomes matrix–vector multiplication in frequency domain

( ) ( ) 12minarg −=

−= uubu

ACCmuAmbK((((

(Es

IK((

=n

( ) ( )( ) ( )zNzHIzB ((

−=n

( ) ( )( )( ) ( )zXIKzHIKzB1−

−+=(((((

s

Noise component of output

Signal component of output

u(m) quantizer inputb(m) quantizer output

10

Contribution #1: Matrix Gain ModelContribution #1: Matrix Gain Model

How to Construct an Undistorted HalftoneHow to Construct an Undistorted Halftone

• Pre-filter with inverse of signal transfer function to obtain

undistorted halftone

• Pre-filtering is equivalent to the following when

( ) ( )( )[ ] 1−−+= KIKzHIzG((((((

IKL(((

−= −1

Modified error diffusion

+ _

_+

e(m)

b(m)x(m)u(m)

)(mh(

t(m)

L

11


Validation #1 by Constructing Undistorted HalftoneValidation #1 by Constructing Undistorted Halftone

• Generate linearly undistorted halftone

• Subtract original image from halftone

• Since halftone should be “undistorted”, the residual should

be uncorrelated with the original

=

0043.00024.00044.0

0073.00114.00013.0

0020.00097.00006.0

rxC(

Correlation matrix of residual image (undistorted halftone minus input image) with the input image

12


Validation #2 by Knox’s ConjectureValidation #2 by Knox’s Conjecture

=

1816.01839.02173.0

1817.02348.02661.0

1778.00019.03664.0

exC(

Correlation matrix for anerror image and input imagefor an error diffused halftone

=

0247.00004.00160.0

0229.00550.00153.0

0082.00839.00485.0

exC(

Correlation matrix for anerror image and input imagefor an undistorted halftone

( ) 0=zEs

( ) ( )zNzE =n

13


Validation #3 by Distorting Original ImageValidation #3 by Distorting Original Image

• Validation by constructing a

linearly distorted original

– Pass original image through

error diffusion with matrix

gain substituted for quantizer

– Subtract resulting color

image from color halftone

– Residual should be shaped

uncorrelated noise

Correlation matrix of residual image (halftone minus distorted

input image) with the input image

=

0056.00101.00015.0

0126.00057.00007.0

0004.00140.00012.0

rxC(

14


Validation #4 by Noise ShapingValidation #4 by Noise Shaping

• Noise process is error image for an undistorted halftone

• Use model noise transfer function to compute noise spectrum

• Subtract original image from modeled halftone and compute

actual noise spectrum

15

Contribution #2Contribution #2

Designing of the Error FilterDesigning of the Error Filter

• Eliminate linear distortion filtering before error diffusion

• Optimize error filter h(m) for noise shaping

Subject to diffusion constraints

where

( )[ ] ( ) ( )( ) ( )

∗−∗=

22min mnmhImvmb

(((EE n

( ) 11mhm

=

∑(

( )mv( linear model of human visual system

* matrix-valued convolution

16

Contribution #2: Error Filter DesignContribution #2: Error Filter Design

Generalized Optimum SolutionGeneralized Optimum Solution

• Differentiate scalar objective function for visual noise shaping with respect to matrix-valued coefficients

• Write the norm as a trace andthen differentiate the trace usingidentities from linear algebra

( )[ ]{ }( ) ℘∈∀= i0ih

mb

d

Ed n

2

( )xxx ′= Tr

( ){ }A

XXA ′=

((

((

d

Trd

( ){ }BA

XBXA ′′=

(((

(((

d

Trd

( ){ }BXABXA

XBXAX ′′+=

′ (((((((

((((

d

Trd

( ) ( )ABBA((((

TrTr =

17


Generalized Optimum Solution (cont.)Generalized Optimum Solution (cont.)

• Differentiating and using linearity of expectation operator give a generalization of the Yule-Walker equations

where

• Assuming white noise injection

)()()()()()( qpsirphqvsvkirkv nnp q s

ank

++−−′=−−′ ∑∑∑∑ ((((((

)()()( mnmvma ∗= (

[ ] ( )kkmnmnkrnn δ≈+′= )( )()( E

[ ] ( )kvkmnmakran −≈+′= ()( )()( E

18


Generalized Optimum Solution (cont.)Generalized Optimum Solution (cont.)

• Optimum solution obtained via steepest descent algorithm

( ) ( )( ) )()()()()()( qpsirphqvsvkirkv nnp q s

ank

ih ++−−′+−−′−=∇ ∑∑∑∑ (((((((J

( ) ( ) ( ) ( ){ }( )))((1 )()( ihihih θαθθ (

((J∇−Ρ=+

( )( ) ( ) 1Imfififm

−

℘−=Ρ ∑

℘∈

(((()(

3

1

α - convergence rate parameter

Ρ - projection operator

θ - iteration number

19


Linear Color Vision ModelLinear Color Vision Model

• Pattern-Color separable model [Poirson and Wandell, 1993]

– Forms the basis for S-CIELab [Zhang and Wandell, 1996]

– Pixel-based color transformation

B-W

R-G

B-Y

Opponentrepresentation

Spatialfiltering

•E

20


Linear Color Vision ModelLinear Color Vision Model

• Undo gamma correction on RGB image

• Color separation– Measure power spectral distribution of RGB phosphor

excitations

– Measure absorption rates of long, medium, short (LMS) cones

– Device dependent transformation C from RGB to LMS space

– Transform LMS to opponent representation using O

– Color separation may be expressed as T = OC

• Spatial filtering is incorporated using matrix filter

• Linear color vision model

( ) Tmdmv(((

)(= where )(md(

is a diagonal matrix

)(md(

21

Floyd-Steinberg Optimum Filter

22

Contribution #3Contribution #3

Block Error DiffusionBlock Error Diffusion

• Input grayscale image is “blocked”

• Error filter uses all samples from neighboring blocks and diffuses an error block

( ) ( ) ( )kmekhmtk

−= ∑∈S

(

+ _

_+

e(m)

b(m)x(m)


compute error

shape error

u(m)

)(mh(

t(m)

23

Contribution #3: Block Error DiffusionContribution #3: Block Error Diffusion

Block Interpretation of Vector Error DiffusionBlock Interpretation of Vector Error Diffusion

)(mh(

pixel block mask

• Four linear combinations of the 36 pixels are required to

compute the output pixel block

24


Block FM Block FM HalftoningHalftoning

• Why not “block” standard error diffusion output?– Spatial aliasing problem

– Blurred appearance due to prefiltering

• Solution– Control dot shape using block error diffusion

– Extend conventional error diffusion in a natural way

• Extensions to block error diffusion– AM-FM halftoning

– Sharpness control

– Multiresolution halftone embedding

– Fast parallel implementation

25


Block FMBlock FM HalftoningHalftoning Error Filter DesignError Filter Design

• Start with conventional error filter prototype

• Form block error filter as Kronecker product

• Satisfies “lossless” diffusion constraint

• Diffusion matrix satisfies

=

16

7

16

3

16

5

16

1ã

Dã((

⊗=Γ D(

diffusion matrix

11 =Γ(

0((

≥Γ

11D =(

0D((

≥

26


Block FM Block FM Halftoning Halftoning Error Filter DesignError Filter Design

• FM nature of algorithm controlled by scalar filter prototype

• Diffusion matrix decides distribution of error within a block

• In-block diffusions are constant for all blocks to preserve isotropy

D(

3/16

7/16

5/16 1/16

ã

27


Block FM Block FM HalftoningHalftoning ResultsResults

• Vector error diffusion with diffusion matrix

[ ]1D((

2

1

N= N is the block size

Pixel replication Floyd-Steinberg Jarvis

28


Block FM Block FM HalftoningHalftoning with Arbitrary Shapeswith Arbitrary Shapes

Plus dots Cross dots

29


Embedded Embedded Multiresolution HalftoningMultiresolution Halftoning

• Only involves designing the diffusion matrix– FM Halftones when downsampled are also FM halftones

• Error at a pixel is diffused to the pixels of the same color

Halftone pixels at Low, Medium and High resolutions

HHHH

HMHHMH

HHHH

HMHHLMH

30


Embedded Embedded HalftoningHalftoning ResultsResults

High resolution halftone

Medium resolution halftone

Low resolution halftone

Simple down-

sampling

31

ContributionsContributions

• Matrix gain model for vector color error diffusion– Eliminated linear distortion by pre-filtering

– Validated model in three other ways

• Model based error filter design for a calibrated device

• Block error diffusion– FM halftoning

– AM-FM halftoning (not presented)

– Embedded multiresolution halftoning

• Efficient parallel implementation (not presented)

32

Published Published HalftoningHalftoning Work Not in DissertationWork Not in Dissertation

N. Damera-Venkata and B. L. Evans, ``Adaptive Threshold Modulation for Error Diffusion Halftoning,'' IEEE Transactions on Image Processing, January 2001, to appear.

T. D. Kite, N. Damera-Venkata, B. L. Evans and A. C. Bovik, "A Fast, High Quality Inverse Halftoning Algorithm for Error Diffused Halftoned images," IEEE Transactions on Image Processing,, vol. 9, no. 9, pp. 1583-1593, September 2000.

N. Damera-Venkata, T. D. Kite , W. S. Geisler, B. L. Evans and A. C. Bovik ,``Image Quality Assessment Based on a Degradation Model'' IEEE Transactions on Image Processing, vol. 9, no. 4, pp. 636-651, April 2000.

N. Damera-Venkata, T. D. Kite , M. Venkataraman, B. L. Evans,``Fast Blind Inverse Halftoning'' IEEE Int. Conf. on Image Processing, vol. 2, pp. 64-68, Oct. 4-7, 1998.

T. D. Kite, N. Damera-Venkata, B. L. Evans and A. C. Bovik, "A High Quality, Fast Inverse Halftoning Algorithm for Error Diffused Halftoned images," IEEE Int. Conf. on Image Processing, vol. 2, pp. 64-68, Oct. 4-7, 1998.

33

Submitted Submitted HalftoningHalftoning Work in DissertationWork in Dissertation

N. Damera-Venkata and B. L. Evans, ``Matrix Gain Model for Vector Color Error Diffusion,'' IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing, June 3-5, 2001, to appear.

N. Damera-Venkata and B. L. Evans, ``Design and Analysis of Vector Color Error Diffusion Systems,'' IEEE Transactions on Image Processing, submitted.

N. Damera-Venkata and B. L. Evans, ``Clustered-dot FM Halftoning Via Block Error Diffusion,'' IEEE Transactions on Image Processing, submitted.

34

Types of Types of Halftoning Halftoning AlgorithmsAlgorithms

• AM halftoning

– Vary dot size according to underlying graylevel

– Clustered dot dither is a typical example (laserjet printers)

• FM halftoning

– Vary dot frequency according to underlying graylevel

– Error diffusion is typical example (inkjet printers)

• AM-FM halftoning

– Vary dot size and frequency

– Typical example is Levien’s “green-noise” algorithm [Levien 1993]

35

Designing Error Filter in Scalar Error DiffusionDesigning Error Filter in Scalar Error Diffusion

• Floyd-Steinberg error filter [Floyd and Steinberg, 1975]

• Optimize weighted error

– Assume error image is white noise [Kolpatzik and Bouman, 1992]

– Use statistics of error image [ Wong and Allebach, 1997]

• Adaptive methods– Adapt error filter coefficients to minimize local weighted mean

squared error [Wong, 1996]

36


FM FM HalftoningHalftoning with Arbitrary Dot Shapewith Arbitrary Dot Shape

input pixelblock

minorityPixel block?Quantize as usual Quantize with

dot shape

Diffuse error withblock error diffusion

No Yes

37

Contribution #3: Block Error Diffusion Contribution #3: Block Error Diffusion

AMAM--FM FM HalftoningHalftoning with Userwith User--controlled Dot Shape controlled Dot Shape

input pixelblock

minorityPixel block?Quantize as usual Quantize with

dither matrix

Diffuse error withblock error diffusion

No Yes

38


AMAM--FM FM HalftoningHalftoning with Userwith User--controlled Dot Size controlled Dot Size

+ _

_+

e(m)

x(m)+

+

u(m)

)(mh(

t(m)

++

+

b(m)

Block green noise error diffusion

)(mg(

h(

• Promotes pixel-block clustering into super-pixel blocks

39


AMAM--FM FM HalftoningHalftoning ResultsResults

Clustered dot dithermodulation

Output dependent feedback

40


Block FM Block FM HalftoningHalftoning with Sharpness Controlwith Sharpness Control

• The above block diagram is equivalent to prefiltering with

( ) ( ){ }[ ] ILzHIzHIzG(((((((

+−+= )(s

+ _

_+

e(m)

x(m)+

+

u(m)

)(mh(

t(m)

L

++

+

b(m)

Modified error diffusion

41


Block FM Block FM HalftoningHalftoning with Sharpness Controlwith Sharpness Control

=

100

010

001

6.0L(

=

100

010

001

2.0L(

42


Diffusion Matrix for EmbeddingDiffusion Matrix for Embedding

( )′= 0000000000000001l

′

= 00000

3

10

3

100000

3

100m

′

=

12

1

12

1

12

1

12

1

12

10

12

10

12

1

12

1

12

1

12

1

12

10

12

10h

( )′= hhhhhmhmhhhhhmhlD(

43

Floyd-Steinberg

Optimum Filter

44


Implementation of Vector Color Error DiffusionImplementation of Vector Color Error Diffusion

( )

=)()()(

)()()(

)()()(

zzz

zzz

zzz

zH

bbbgbr

gbgggr

rbrgrr

HHH

HHH

HHH(

Hgr

Hgg

Hgb

+

b

g

r

×

×g

45


Implementation of Block Error DiffusionImplementation of Block Error Diffusion

H11

H12

H14

H13

+

2

2

2

2

z2

z1z2

z1

z2-1

z1-1

z1-1 z2

-1

2

ph.d. defense analysis and design of vector error diffusion...

Documents