Fl.- HEWLETT.:~ PACKARD

Inverse Halftoning and KernelEstimation for Error Diffusion

Ping Wah WongMedia Technology LaboratoryHPL-93-92October, 1993

halftoning, inversehalftoning, errordiffusion, imagereconstruction,statistical filtering,projection, kernelestimation

Two different approaches in the inverse halftoning oferror diffused images are considered. The firstapproach uses linear filtering and statisticalsmoothing that reconstructs a gray scale image from agiven error diffused image. The second approach canbe viewed as a projection operation, where oneassumes the error diffusion kernel is known, and findsa gray scale image that will be halftoned into the samebinary image. Two projection algorithms, viz.,minimum mean square error (MMSE) projection andmaximum a posteriori probability (MAP) projection,that differ on the wayan inverse quantization step isperformed, are developed. Among the filtering and thetwo projection algorithms, MAP projection providesthe best performance for inverse halftoning. Usingtechniques from adaptive signal processing, wesuggest a method for estimating the error diffusionkernel from the given halftone. This means that theprojection algorithms can be applied in the inversehalftoning of any error diffused image withoutrequiring any a priori information on the errordiffusion kernel. It is shown that the kernelestimation algorithm combined with MAP projectionprovide the same performance in inverse halftoningcompared to the case where the error diffusion kernelis known.

© CopyrightHewlett-Packard Company 1993

Internal Accession Date Only

1 Introduction

The problem of inverse halftoning has received relatively little attention compared to digitalhalftoning [1]. Assuming a gray scale image Xm,n is halftoned into a binary image bmn, thegoal of inverse halftoning is to reconstruct a gray scale image xm,n from bm,n so that £m,n iseither as close to Xm,n as possible, or xm,n is a candidate (halftoning is a many to one mapping)which when error diffused, bm,n would result. In addition to being an interesting theoreticalproblem, inverse halftoning is also of great practical interest because it can be applied toa wide variety of problems such as scaling, tone correction, interchanging between halftonemethods (rehalftone), facsimile image processing, and many others. Inverse halftoning hasalso been applied recently in the lossy compression of binary images [2].

Popular halftoning algorithms can in general be classified into two categories, viz.J ordereddither and error diffusion. They differ both in their processing structures and in the char­acteristics of the resulting binary images. For example, the spectra of dithered halftoneimages exhibit strong discrete components due to the periodicity of the dithering matrices.The spectra of error diffused images, however, do not have strong periodic components, andthe local energy content in the spectra increases gradually as one moves from low to highfrequencies [3].

Inverse halftoning for dithered halftone images has been reported in [4-6]. In these cases,combinations of filtering and statistical processing in the spatial domain are applied to re­construct gray scale images from their dithered halftones. In [7], the error diffusion processis modified so that inverse halftoning of error diffused images is posed as a quadratic pro­gramming problem where a mean square error criterion is minimized. Because the size ofthe quadratic programming problem is very large, it is partitioned into smaller subprob­lems where an approximate solution is found. Recently, a statistical technique based onthe Bayesian approach has also been applied to reconstruct continuous tone images fromboth dithered and error diffused halftones [8]. In [2], a lossy compression algorithm for errordiffused images is reported where a binary image is first inverse halftoned, and the resultinggray scale image is then encoded and finally rehalftoned. Inverse halftoning of error diffusedimages is performed in [2] using a look-up table trained in similar spirit as in vector quantizerdesign.

It is well known that any halftoning process can be interpreted as a source that introducesnoise into continuous tone images so that the output images can take on binary valuesonly. Since the noise due to error diffusion is in general concentrated in high frequencies, areasonable approach for inverse halftoning of error diffused images would be to design a lowpass filter for removing these high frequency components. Low pass filtering alone, however,does not work well in general. If the cutoff frequency of the filter is set so low so that mostof the errors are removed, the resulting image is blurry because a significant amount of thehigh frequency information in the original gray scale image is also removed. On the otherhand, if the cutoff frequency is high, most of the error from the halftoning process will stillbe present in the output image, and the reconstruction will also not be of high quality.

In this paper, we consider two different methods for inverse halftoning. The first one is basedon a combination of linear filtering and statistical smoothing, which does not require anyknowledge about the error diffusion kernel. The second method, which can be interpretedas a projection approach, assumes the error diffusion kernel is known a priori, and finds agray scale image which will be halftoned into the given binary image. The complexity of thisprojection algorithm is comparable to that of error diffusion itself and hence this projection

1

Xm,n + Um,nQ(.)

bmn+

,

-+-

+em,n

hk,l

Figure 1: A generic error diffusion system.

approach is very efficient. Two projection algorithms, uiz., minimum mean square error(MMSE) projection and maximum a posteriori probability (MAP) projection, that differ ina specific choice of certain parameters, will be developed and compared.

One crucial weakness of the projection algorithms both in this paper and in [7] is thatthe error diffusion kernel is assumed to be known a priori. This is perhaps an unrealisticassumption in practical situations because one usually does not have any information on theerror diffusion kernel when one is given an error diffused image. To remove this assumption,we suggest a method based on techniques in adaptive signal processing for estimating theerror diffusion kernel. This kernel estimation method is then combined with the projectionalgorithms to provide an iterative method for the inverse halftoning of any error diffusedimages. The performance of this kernel estimation algorithm in conjunction with projectionin inverse halftoning will be investigated.

In Section 2, error diffusion and the inverse problem are described. The filtering and smooth­ing approach for inverse halftoning is considered in Section 3. In Section 4, we describe theprojection approach, where both MMSE and MAP projection are developed and investi­gated. An algorithm for error diffusion kernel estimation is developed in Section 5. Itsapplication in inverse halftoning is also considered. Section 6 summarizes the results in thispaper.

2 Error Diffusion and Its Inverse

Let Xm,n be a gray scale image with dynamic range A, i.e., xm,n E [0, A] for all m and n.Also let bm,n E {a, I} be a halftoned version of xm,n' In the case of error diffusion, bm,n isobtained from Xm,n using a system as shown in Fig. 1. In this paper, we consider a typicalcase of error diffusion where the processing is performed in a scanned fashion, i. e., in a rowby row fashion from the top to the bottom, and for each row from the left to the right. Theinput-output relationship of error diffusion is described by

Um,n = Xm,n - L hk,lem-k,n-l,(k,I)ES

em,n = Abm,n - um,n = AQ(Um,n) - um,n,

if Um,n ~ A/2otherwise,

2

(1)

(2)

(3)

(0,0) '\

* 7/16

3/16 5/16 1/16

Figure 2: The Floyd-Steinberg kernel for error diffusion.

where um,n is the state variable of the system, hk,1 is the error diffusion kernel, S is theregion of support for hk,1, and em,n is the quantizer error. The region of support of the errordiffusion kernel is always strictly "causal" with respect to the scanned direction so that onlythe "past" errors are diffused to the current location. In particular, we have (0,0) ¢ S. Formost practical error diffusion systems [1,9-11], the coefficients of the error diffusion kernelhk,1 satisfy

hk,1 2: 0; L: hk,1 = 1. (4)(k,I)ES

An example of a kernel with four coefficients is the Floyd-Steinberg kernel [9] as shownin Fig. 2. To actually perform the error diffusion process using (1)-(3), it is necessary todefine the boundary condition of either the state variable or the quantizer error. We assumeas usual that the quantizer error em,n when (m, n) falls outside the range of the image isidentically zero.

The problem of error diffusion is very well known, as many kernels and variations of scanningmethods have been reported [1,9-12]. Once a kernel and the boundary conditions have beendecided on, the error diffusion process is performed by iterating (1)-(3) in a straightforwardmanner. The problem of inverse halftoning for error diffused images, however, is moredifficult even when the kernel is known. From a signal processing viewpoint, the inverseproblem can be thought of as an estimation problem. There are two different ways to specifythe goal of inverse halftoning, and they in general would lead to different approaches andresults.

First, an inverse halftoning procedure can be thought of as one that attempts to produce agray scale image Ym,n from a halftone bm,n so that they appear similarly to a human observerwhen observed at a certain distance away. Such a formulation suggests a filtering approachwhereby the error signal due to the halftoning process is removed. It is well known that errordiffusion is a two-dimensional analog of one-dimensional sigma-delta modulation [13,14].Despite some recent works in the development of alternative algorithms for decoding a sigma­delta modulated signal [15,16], low pass filtering remains a popular decoder for sigma-deltamodulation. This is partially because the oversampling ratios in typical applications ofsigma-delta modulation are high, and hence low pass filtering can provide good performance[14,17-19]. In imaging systems, however, the oversampling ratio is usually limited. Inother words, the sampling frequency in digital images typically are not higher than theimage bandwidth by an order of magnitude. Consequently, simple low pass filtering alonecannot provide good enough performance for inverse halftoning. We shall describe in thenext section an inverse halftoning algorithm using multiple passes of low pass filtering andstatistical smoothing, so that good image quality can be obtained. Although we are focusedon the inverse halftoning of error diffused images in this paper, such a filtering and smoothing

3

approach is also applicable for the inverse halftoning of ordered dither halftones [4-6].

An alternative way to describe the goal of inverse halftoning for error diffused images is thefollowing: Given a binary image bm,n, one wants to find a gray scale image Ym,n so that whenYm,n is error diffused, bm,n would result. Such a formulation clearly poses the task exactlyas an inverse problem, i.e., we want to find an inverse of bm n, rather than just finding anapproximation to the inverse as in the filtering approach. This formulation also implicitlysuggests that the error diffusion kernel is known beforehand. This approach, which can beviewed as a projection operation, is similar to a recently developed decoder for signals froma sigma-delta modulator [16]. Indeed, we shall develop two projection algorithms (minimummean square error projection and maximum a posteriori projection) for finding Ym,n assuminga known error diffusion kernel.

The assumption of a known error diffusion kernel is equivalent to assuming a known struc­ture (e.g.) single-loop versus two-stage) for the case of sigma-delta modulation. While thisassumption is perhaps not overly restrictive for sigma-delta modulation decoding, it severelylimits the applicability of the projection approach in inverse halftoning because the errordiffusion kernel is generally unknown when one is faced with an error diffused halftone. Toovercome this difficulty, we develop an algorithm for error diffusion kernel estimation usingideas from adaptive signal processing. (See, for example, [20].) This kernel estimation al­gorithm is then used in conjunction with the projection algorithm so that the projectionapproach can be applied to the inverse halftoning of any error diffused image.

3 Filtering and Smoothing

In this section, we develop an algorithm for inverse halftoning of error diffused images basedon filtering and smoothing. Since error diffusion injects noise into the image predominantlyin the high frequency range, a low pass filter can remove much of the noise and producean approximation of the original gray scale image. Indeed if the frequency content of theimage is very low compared to the sampling frequency, or equivalently if the oversamplingratio of the original gray scale image is high, one can use a low pass filter with a sufficientlylow cut-off frequency and obtains good results. For typical error diffused images of naturalscenes, however, low pass filtering alone cannot reconstruct gray level images of good enoughquality.

To illustrate this, we let fs be the sampling frequency of the gray scale image, i.e., I, = 1/~where ~ is the grid size. We then filter a typical error diffused image using a low-pass filterwith a cut-off frequency at fs/4, i.e., one half of the maximum possible frequency. A filterwith such a pass band is often referred to as a half-band filter in signal processing. Thispreassumes that the frequency content of the original gray scale image is primarily within±fs/4 in both the horizontal and vertical directions, which appears to be a reasonableassumption for many test images. Note also that frequency components above fs/4 are notwell represented by the halftone anyway because it is generally difficult, if not impossible,to distinguish with certainty whether any particular high frequency component found in anerror diffused image is originated from the original gray scale image or from the halftoneprocess itself. Fig. 3 shows an error diffused image obtained using the Floyd-Steinbergkernel, and the result of low pass filtering using a half-band filter. It is evident that whilethe low pass filtered image appears to be much smoother than a binary image, a substantialamount of residual noise due to the halftoning process still remains. This is well reflected inthe local variation of gray levels in areas that should have been smooth. It is true that the

4

Halftone Low Pass Filtered

Figure 3: An error diffused image and the output after being filtered using a half-band filter.

output image would have been smoother if we had used a low pass filter with a lower cut-offfrequency. Such an output image, however, is also blurred to a larger extent when comparedto the one in Fig. 3.

A close observation of Fig. 3 reveals that the binary patterns in the error diffused image arespread out spatially, and the resulting image consists of small clusters of "peaks and valleys"over the gray scale background. Since these spatial clusters are generally of several pixelswide, the major frequency components are below the half-band frequency 18/4. This meansthat they cannot be removed by further low pass filtering because a low pass filter with alower cut-off frequency will cause excessive blurring to the resulting image. Note that medianfiltering also does not work well in this situation because the residual noise here consists ofsmall clusters rather than being "impulsive."

In order to remove these residual errors and reconstruct high quality gray scale images, wesuggest in this section applying a pipeline as shown in Fig. 4 that uses multiple passes of lowpass filtering and nonlinear statistical smoothing. The reason of inserting a low pass filterat every stage is to remove the potentially unwanted high frequency components generatedby the nonlinear smoothing process. These low pass filters can, for example, be all halfbandfilters to avoid overly blurring the reconstructed image.

The nonlinear statistical smoothing process can be described as follows: For each pixel loca­tion (m, n), take a small neighborhood 'Rm,n around the location (m, n). We then computethe sample mean !-lm,n of the pixels in 'Rm,n, as well as a measure vm,n (an example to be givenlater) that characterizes locally the amount of variation among the pixels within the neigh­borhood. We then compare the value of the center pixel with the thresholds !Lm,n + ,Vm,n

5

Um,n low statistical T(um ,pass

smoothingfilter

v

T(·)gray scale

elow imag----0 T(·) pass

filter

halftone I ~Image-----.I. T (.)

v

k times

Figure 4: Pipeline for inverse halftoning of error diffused images using the filtering approach.

and I-lm,n - ,Vm,n, where, 2:: 0 is a parameter to be adjusted to obtain good performance.The value of the center pixel is modified according to

{

I-lm,n + ,Vm,nnew

Xm,n = I-lm,n - ,Vm,n

x oldm,n

if oldI Xm,n > I-lm,n + ,Vm,nif old1 Xm,n < I-lm,n - ,Vm,n

otherwise.(5)

The superscripts "old" and "new" refer to the value of the pixel before and after, respectively,each statistical smoothing process along the pipeline of Fig. 4. This operation is repeatedover the entire image where the window is slid across the image one pixel at a time.

An example of Vm,n for measuring the local variation is the r t h power norm centered at thesample mean. Specifically, we have

(6)

where1 ~ xC!l~

I-lm,n = L...JII Rm,n II (i,j)ERm,n ',)

and II A II denotes the number of elements in the set A. In the special case where r = 2,Vm,n is identical to the sample standard deviation of the pixels in the region Rm,n.

The idea of (5) is that a pixel value is changed only when it is statistically far away fromthe sample mean of its neighbors. Note that both I-lm,n and Vm,n vary with the position ofthe window position, and hence this statistical smoothing procedure is strictly performedaccording to local statistics. When the window falls in a location that contains an edgein an image, the sample variance of the local block tends to be larger than the case wherethe window does not contain an edge. This means that the value of a pixel is less likely tobe adjusted when it is near an edge. As a result, such an algorithm provides a smoothingoperation without overly blurring an image.

6

32 33IT'

,

.... - .... -+-,

31 ,... -32

,,Lena/, IT'Lena I

/, I30 .. I / peppers

PSNR , ,PSNR 31 ~.. ,

/(dB) (dB) ,,/29

/2830

27 290 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8

k k

(a) (b)

Figure 5: Performance of inverse halftoning using linear filtering and statistical smoothing. ThePSNR's in (a) and (b) were computed with respect to the original unprocessed gray scale imagesand half-band filtered versions of the original gray scale images respectively.

It is evident from (5) that a smaller value of , implies it is more likely for the value ofthe center pixel to exceed either one of the thresholds, and hence adjustments to the pixelvalues will occur more frequently throughout the image. Consequently, we expect a smootheroutput image for a smaller ,. The maximum amount of smoothing using (5) is obtainedwhen, = 0, for which (5) simply replaces at every step the value of the center pixel with thesample mean of the pixels in the window. The statistical smoothing in this case becomes asimple low pass filtering using a filter with a sine frequency response. In the general case,however, (5) is a nonlinear operation.

Since, controls the extent of smoothing, it makes sense to use a small value of, in the initialstages of Fig. 4 where the estimated gray scale image is very noisy. As the result propagatesdown the pipeline, the gray scale image gradually becomes less noisy and hence, can beincreased so that blurring of the reconstructed image is avoided as much as possible. In ourexperiments using several standard test images, we found that the best performance wereobtained by setting the initial and final values of, to 0.25 and 1 respectively. That is, fora particular choice of k, , = 0.25 at the first stage and linearly increases along the stages to, = 1 at the kt h stage. This is similar to a recent report on the inverse halftoning of ordereddither images [5], where gradually increasing the cut-off frequency of a low pass filter alongthe stages of processing produces output images of better quality. Empirical results with theconfiguration in Fig. 4, however, show that increasing the cut-off frequency of the low passfilter along the stages do not improve the reconstructed images.

We tested this inverse halftoning approach using several standard images halftoned using theFloyd-Steinberg error diffusion kernel. The sample standard deviation is chosen as the mea­sure of variation of pixels within the window, i.e., we let r = 2 in (6). We found that the bestresults are obtained when Rm,n is a 3 by 3 square window centered at (m, n). The output

7

k=l k=2

k=8

Figure 6: The pepper image reconstructed from its Floyd-Steinberg error diffused version using lowpass filtering and statistical smoothing.

8

images when using larger windows, e.g., 3 by 5 or 5 by 5, are typically oversmoothed and ap­pear to be blurred. The results for the Lena and the pepper images using 3 by 3 windows areshown in Fig. 5. The peak signal-to-noise ratio (PSNR) figures are computed by comparingthe reconstructed images with both the original gray scale image and the original gray scaleimage filtered using a half-band low pass filter. Fig. 6 shows the images reconstructed fromits Floyd-Steinberg error diffused version using the filtering approach. The reconstructedimage shows that low pass filtering in conjunction with statistical smoothing is quite effec­tive in reconstructing visually pleasing images. We shall develop in the next section anotherapproach that reconstructs images of higher quality than those in this section.

Comparing the performance of this algorithm with the table based decoder in [2], whereonly the results for the Lena image is reported, we find that this approach outperforms thatin [2] by about 0.7 dB for the 3 by 3 mask size, and the advantage over those in [2] with a4 by 4 mask size is about 0.3 dB. We should point out that the table based approach in [2]has an advantage in terms of complexity. The inverse halftoning process there is very simpleand efficient assuming the look-up tables are available, although the design of these tablesrequires a training using several test images in similar spirit as in vector quantizer design. Itis somewhat more difficult to compare the results here with those in [7] because the PSNR'sthere are computed with respect to a singular value decomposition (SVD) band limitedimage. The error diffusion process considered in [7] is also different from the traditionalerror diffusion. '

Inverse halftoning is a problem, like many other inverse problems, that does not have a uniquesolution. In other words, when given a halftone image, we cannot in general hope to recoverthe exact original image. For example, it is very unlikely if not impossible for any inversehalftoning procedure to reconstruct the high frequency information (those near 18/2) in theoriginal image; the reason being one cannot distinguish without ambiguity whether certainhigh frequency components are originated from the original image or from the halftoning

.process itself, or both. Therefore, the comparison figure of the inverse halftoned image withthe original gray level image in Fig. 5 is aimed for indication purposes, and can serve as abasis for comparison between different inverse halftoning algorithms.

4 Projection in Inverse Error Diffusion

We assume for the rest of this Section that the error diffusion kernel is known, and describea projection algorithm for inverse halftoning that outperforms the filtering approach of theprevious Section. The assumption on known error diffusion kernels will be removed in thenext Section where an estimation algorithm for the error diffusion kernel will be introduced.

In the projection approach, the inverse halftoning problem is cast as the following inverseproblem: Given a binary image bm,n and an error diffusion kernel hk,l, we want to find a grayscale image xm,n so that when xm,n is error diffused, we would obtain bm,n' The mathematicalrelationship governing xm,n, hk,l and bm,n are given in (1)-(3) where Xm,n in (1) is replaced byxm,n. Observe that because of the quantization process in (3), the problem of determiningxm,n given bm,n and hk,l does not have a unique solution. This is also evident from a simplecounting argument. For example, if we restrict xm,n to be digital images of size N by N at 8bits per pixel, there are 256N 2 possible gray scale images. On the other hand, there is onlya total of 2N 2 possible binary images of the same size. Consequently, there are multiple grayscale images that can be halftoned into the same binary image, and hence the solution of

9

the inverse problem cannot be unique. Here our goal is to find a candidate xm,n that wouldgenerate the given halftone.

Given the huge number of possible gray scale images, it is computationally infeasible toperform the inverse problem by an exhaustive search. In [7], the error diffusion process ismodified so that the inverse problem is formulated as a quadratic programming problemwhere a certain mean square error criterion is minimized. For computational reasons, thequadratic programming problem is partitioned into many subproblems of smaller size, andan approximation solution is then found. We introduce here a different projection algorithmthat is more closely related to a decoding method for reconstructing an analog signal from thebinary output of a sigma-delta modulator [16]. It is shown in [16] that this method producesbetter results in si~ma-delta modulation decoding than those obtained from the traditionalfiltering approach l14, 17,18]. We therefore expect this projection algorithm to outperformthe filtering and smoothing approach in the previous section. As we shall see shortly, theprojection algorithm to be introduced is essentially error diffusion with an additional inversequantization step, and hence its complexity is quite low.

To begin, we first obtain xm,n, a crude approximation of Xm,n, by low pass filtering bm,n' Wecan then obtain a better approximate to xm,n using the following algorithm:

Algorithm 1 Projection algorithm for obtaining a gray scale image xm,n given an errordiffused halftone bm,n, an error diffusion kernel hk,f, and xm,n (a crude approximation toxm,n)'

1. Set (m, n) to the first location in the scan direction of error diffusion.

2. If already passed the last pixel of the image, stop.

3. Perform one step of error diffusion using xm,n to obtain bm,n'

4. If bm,n = bm,n, let xm,n = xm,n' Increment (m, n) to the next pixel location and go tostep 2.

5. For the case bm,n =J bm,n, adjust xm,n so that the corresponding state variable um,nwould be on the side of the quantizer threshold to give Q(um,n) = bm,n' In this paper,we chose xm,n so that

if bm,n = 1if bm,n = 0

(7)

where a :2: A/2 and f3 ~ A/2 are two predetermined constants. (More detail on thislater.}

6. Increment (m, n) to the next pixel location and goto step 2. •The philosophy of the projection algorithm can be summarized as follows: Perform the errordiffusion process using the known kernel and the gray scale image xm,n' At any step if theoutput binary pixel is not equal to the given halftone bm,n, we adjust the input xm,n so that

10

18000 r-------,---------,

3>../2>../2um,n

OL.-.---""----------"-------"-----'->../2

6000airplane --1--+-1

Lena (solid line),peppers ---+-

(dashed line)

frequency

12000

Figure 7: Empirical distribution (histogram) of the state variable in Floyd-Steinberg error diffusion.

the generated output binary pixel equals bm,n' The adjustment is reflected in Step 5 of thealgorithm where the conditions 0: ~ A/2 and 13 < A/2 are sufficient to guarantee that thecorrect binary output image will be produced. Note that um,n is a continuous amplitudevariable. Hence if we assume it has a probability density function, the probability of Um,n

being exactly equal to A/2 is zero. As a result, we have written 13 ::; A/2 in the Algorithminstead of a strict inequality 13 < A/2.

We mentioned that the solution to the inverse error diffusion problem is non-unique. It isreflected in Step 5 of Algorithm 1 where 0: and 13 can be arbitrarily chosen as long as 0: ~ Aand 13 ::; A/2. To be specific in choosing these two constants, we first state the following factconcerning error diffusion.

Fact 1 Consider an error diffusion system described by (1)-(3). If we assume that thequantization error ek,l is zero for (k, I) outside the support range of the image) then

Um,n E [-A/2,3A/2] and em,n E [-A/2, A/2]

for all (m, n) within the support range of the image. I

A proof of a similar property concerning single-loop sigma-delta modulation can be foundin [21]. A simple proof of this fact is given in Appendix A. In view of Fact 1, we can limitour choices of 0: and 13 to

-A/2 < 13 < A/2, and A/2 ::; 0: ::; 3A/2.

To further narrow down the choices of 0: and 13, we can analyze the probability distributionof the state variable Um,n' An exact analytical analysis of this distribution appears to be

11

48000 ,...---------r----------,

3>../2

~ Jarvis, Judicer" - and Ninke

>../2um,n

o'-----"'---'-----'-----~-----'->../2

32000

16000

frequency

Figure 8: Empirical distribution (histogram) of the state variable in error diffusion for severaldifferent kernels.

very difficult because of the nonlinearity involved in the quantizer. Because of this, weexperimentally obtained histograms of Um,n using several different error diffusion kernels andtest images. Fig. 7 shows typical distributions of Um,n for several different images with theFloyd-Steinberg kernel, and Fig. 8 shows the distribution of um,n for several error diffusionkernels using the Lena image. These plots show a remarkable similarity in the distributionof Um,n for different images and different error diffusion kernels. All of these curves indicatethe distributions of Um,n tend to cluster around the value >"'/2, which is the threshold of thequantizer in (3). The most likely value of Um,n, for practical purposes, can be considered tobe >.../2.

In light of the distribution of Um,n, we can formulate two different ways for choosing a andf3 in Algorithm 1. One strategy is that we choose

um,n = a = E[um,nlbm,n = 1] for bm,n = 1, (8)

andUm,n = f3 = E[um,nlbm,n = 0] for bm,n = O. (9)

It is well known that the optimum zeroth order predictor (a constant) for a random variablethat minimizes the mean square error is the mean value of the distribution. Hence (8) and (9)toe;ether minimize the square error between the estimated and the true state variable, viz.,El(um,n - um,n)2]. Algorithm 1 combined with this strategy for obtaining xm,n through (7)can therefore be interpreted as a minimum mean square error (MMSE) projection algorithm.

On the other hand, we can also choose

a = (3 = >.../2 (10)

in Algorithm 1. This means that we always use the most likely value of um,n for um,nwhenever an adjustment to xm,n is necessary. Because of this, we can refer to Algorithm 1 in

12

halftonehmn,

estimate ofgray scale

Image

1T(·)

am,nprojection

bmn,better

estimate ofgray scale

Image

v

P(·)

alee

I 1 gray sc

low ImagP(·) c--- - - - ---0 P(·) pass

filter

halftoneImage

...k times

Figure 9: Pipeline for inverse halftoning of error diffused images using the projection approach.The block T(·) is as shown in Figure 4.

conjunction with (10) as a maximum a posteriori probability (MAP) projection algorithm.Observe that when a and f3 are chosen as in (10), the resulting adjustment to the statevariable, and hence also to xm,n, is the minimum amount possible in the sense that um,n ismoved just across to the correct side of the quantizer threshold so that the correct binaryoutput will result. This approach can therefore also be interpreted as a minimum adjustmentprojection algorithm. We shall, however, refer to this as MAP projection in the rest of thispaper.

As we observe from empirical results, xm,n is in general a better approximation of Xm,n thanxm,n' Since the solution to inverse error diffusion is non-unique, about the only statement wecan make concerning either the MMSE or the MAP projection algorithm is that xm,n wouldbe halftoned to bm,n' There is no guarantee, however, that the values of xm,n as obtainedare legal, i.e., xm,n may be outside of the range [0, -X]. In practice, we can "clip" the valuesof xm,n resulting from the projection algorithms to make sure that they always fall in thedesired range.

Because of the "clipping" process required, it is in general necessary to perform multiplepasses of projection so that a good estimate is obtained. A pipeline of the implementationis shown in Fig. 9. As argued in Section 3, the cut-off frequency of the low pass filters areset to a quarter of the sampling frequency, i.e., one half of the highest possible bandwidthin the image.

The performance in PSNR of the two projection algorithms is shown in Fig. 10, and somereconstructed images are shown in Fig. 11. It is evident that MAP projection generally (ex­cept for a single stage in the pipeline, i. e.) k = 1) outperforms MMSE projection, where thegap increases as k increases. Comparing the results in Figures 5 and 10, one can concludethat the MAP projection algorithm clearly provides the best performance, while the differ­ence in performance between MMSE projection and the filtering approach is quite small.

13

32 35Lena , .. - L~na

AP projection...... ' ena LenaMAP projection

31 ... 34 ..- ... -,MMSE projection, ... ", ..., ...,

30 33 ...PSNR A PSNR(dB) (dB)

29 32 ..... ---~ ...... peppersMMSE projection

28 31

27 300 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8

k k

(a) (b)

Figure 10: Performance of minimum mean square error (MMSE) projection and maximum a poste­riori probability (MAP) projection in inverse halftoning. The PSNR's in (a) and (b) were computedwith respect to the original unprocessed gray scale images and half-band filtered versions of theoriginal gray scale images respectively.

If one compares the reconstructed images in Figures 6 and 11, one can also conclude thatthe projection algorithm tends to reconstruct sharper images than those from the filteringapproach, which is perhaps not surprising given the nature of the filtering approach. Judgingby a subjective evaluation of the reconstructed images, the output image from the projectionapproach for k = 3 is perhaps sufficient for practical purposes, although the reconstructedimages with more passes (higher k) do appear to be slightly smoother.

The fact that the projection approach can provide better performance is perhaps not surpris­ing in that both the MMSE and the MAP projection algorithms use information from theerror diffusion kernel to find an approximation to the original gray scale image. It appearsthat the price we need to pay for the extra performance is in the assumption that the errordiffusion kernel is known. This is indeed a high price because such an assumption may notbe realistic in practical situations where all we have is an error diffused image to be inversehalftoned. We shall see in the next Section that this assumption is actually not necessary,i. e. J it is possible to use a kernel estimation algorithm in conjunction with the projectionalgorithm to obtain essentially the same performance in inverse halftoning.

5 Error Diffusion Kernel Estimation and Inverse Halfton-.mg

We have seen that if we know the coefficients of an error diffusion kernel, we can achievebetter performance in inverse halftoning by using the projection algorithms in Section 4. Tomake this approach practical for general inverse halftoning of any error diffused image, we

14

k=1 k=2

k=4 k=8

Figure 11: The pepper image reconstructed from its Floyd-Steinberg error diffused version usingthe maximum a posteriori probability projection approach.

15

•

discuss here a method to estimate the error diffusion kernel given a gray scale image andits error diffused halftone. With such an error diffusion kernel estimation algorithm (to bedescribed later), our strategy for inverse halftoning when given an arbitrary error diffusedimage is as follows:

Algorithm 2 Inverse error diffusion using projection (Algorithm 1) and error diffusionkernel estimation (Algorithm 3).

1. Given an error diffused image bm,n, obtain a gray scale imaqe x~~n usmg low passfiltering.

2. Set i = 1.

3. Use the kernel estimation algorithm, estimate an error diffusion kernel h~i{ from x~-nl), ,and bm,n'

4. Use the projection algorithm, find a gray scale image x~n from ht{ and bm,n.

5. If x~n is satisfactory, set the output image Ym,n = x~n and stop.

6. Increment i by 1, and go to step 3.

Such an approach therefore uses the kernel estimation algorithm and the projection algorithmalternatively to estimate the kernel and the gray scale image until a satisfactory outputimage is obtained. We do not at this point have a proof of convergence for this algorithm.Experimental results using several images, however, confirm that we do have convergence inthree to four passes through the loop of the algorithm, and that the estimated kernels arequite accurate (details to be provided later).

To describe the error diffusion kernel estimation algorithm, we restate the problem as follows:Given a gray scale image Xm,n and the corresponding halftone bm,n error diffused using thekernel hk,l, we want to find gk,l (an estimate of hk,l) so that if Xm,n is error diffused usinggk,/' the output would be as close to bm,n as possible. This can be interpreted as a systemidentification problem, where we shall apply techniques in adaptive signal processing toobtain a solution.

Consider a system as shown in Fig. 12, where gk,l is an estimate of hk,l' It is evident bycomparing Figures 1 and 12 that if the two kernels are identical, i. e., if gk,l = hk,l for all(k,l) E 5, then Ym,n = bm,n for all m and n. Using the usual terminology of adaptive signalprocessing [20], we want to find gk,l to minimize a metric of the error between the desiredoutput bm,n and the actual output Ym,n' Such an error metric, however, is difficult to workwith because both bm,n and Ym,n are binary valued. In adaptive signal processing algorithmssuch as the least mean square (LMS) algorithm, we need to have a relatively smooth errorsurface so that the error can be gradually steered toward a minimum by adjusting the systemparameters [20]. Any error metric defined on bm,n - Ym,n would not have the desired propertybecause there are only a maximum of four possible values for bm,n - Ym,n'

It is obvious from the structure of the error diffusion system that the state variable Um,n,

which is the input to the quantizer, uniquely determines the output bm,n' Therefore, min­imization of the output of the error diffusion system can be achieved via minimizing an

16

dut

I putactua outXm,n + Vm,n

Q(.) Ym,n+

nput -- +t bm.desi~e

gk,l outp

Figure 12: Error diffusion kernel estimation formulated as a system identification problem. Thecoefficients gk,l are to be estimated.

appropriately defined error measure between the state variables Um,n and Vm,n, provided wemake sure to take the quantizer outputs into consideration. To do so, we first observe thatwe are in a sense minimizing the difference between the output of two quantizers by adjust­ing vm,n through gk,l. In view of that, we define an error criterion between the two statevariables by

{Vmn - Um nt - , ,m,n - 0

if bm,n =I Ym,nif bm n = Ym n', ,

(11)

In such case, one can apply any optimization procedure to to minimize, e.g., E[€~,n] over allpossible kernels.

A close examination of (11) reveals that the minimization problem as such is not well posed.The reason being that Um,n cannot be determined because the true kernel hk,l is not known.To overcome this difficulty, we can define an alternative error using a prediction of Um,n frombm,n = Q(um,n) similarly as in the development of the projection algorithm. It then becomesclear that we can choose Um,n using either the MMSE approach or the MAP approach as inSection 4. To be specific, we use the MAP approach in the rest of this paper. That is, wedefine an error criterion as

£ _ { Vm,n - >../2m,n - 0

if bm,n =I Ym,nif bm,n = Ym,n'

(12)

Several comments are in order here. First, maximum a posteriori probability here refers tothe choice of >../2 for Um,n in the definition of the error £m,n, as opposed to choosing, forexample, um,n = E[um,nlbm,n]' The adaptive algorithm to be described for steering gk,l willstill minimize the square error E[£';,n]' Second, £m,n depends only on Vm,n, Ym,n and bm,n,all of which are known and hence £m,n can be computed. Third, as in the MAP projectionalgorithm, (12) implies that when the outputs bm,n and Ym,n are not equal, we adjust Vm,nthrough gk,l towards the correct side of the quantizer threshold so that the output Ym,n canbe reversed.

From Fig. 12, we can write

Vm n = Xm n - '"""gkl(bm-k n-1 - Vm-kn-l)., ,L...J" ,k,l

17

Therefore, the instantaneous square error is

£~,n = [xm,n - Lgk,l(bm-k,n-l - Vm-k,n-l) - >-./2] 2 I{Ym,n#bm•n}k,l

where 10 is the indicator function. The kernel gk,l that minimizes the instantaneous squareerror can be found by using many adaptive signal processing algorithms. For specificity, wechoose to use the LMS algorithm [20] in this paper. At any step where Ym,n =J bm,n, we let

(13)

where

a£~,n _ ( [""' (i) ]--;(if - -2 bm-k,n-l - Vm-k,n-l) xm,n - L..Jgk,l(bm-k,n-l - Vm-k,n-l) - >-./2ugk,l k,l

I-" is a constant to control the rate of convergence, and i denotes the number of times ad­justments to the kernel has been made. The procedure of estimating the kernel can besummarized as

Algorithm 3 Estimation of Error Diffusion Kernel.

1. Set i = I, and initialize gki~. For example, one can choose,

(1) 1gk,l=m all (k, l) E S.

2. Set (m, n) to the first location in the scan direction of error diffusion.

3. If already passed the last pixel of the image, stop.

4. Perform 1 step of error diffusion.

5. If bm,n = Ym,n, increment (m, n) to the next pixel location and go to step 3.

6. Adjust 9k:~ according to (13) to give 9k:t1

) .

7. Increment i by 1, increment (m, n) to the next pixel location, and go to step 3. I

To test the performance of the kernel estimation algorithm, we first isolate this from theinverse halftoning problem. In other words, we use an original gray scale image and its errordiffused version as the inputs for estimating the error diffusion kernel. Table 1 shows theresult of the estimation for both the pepper and Lena images where the Floyd-Steinbergkernel was the true kernel used. It can be observed from Table 1 that convergence wasessentially achieved after two passes through the image. Furthermore, the differences between

18

Table 1: Results of error diffusion kernel estimation using the original gray scale images and theirerror diffused version. The true kernel used was the Floyd-Steinberg kernel.

number of passes estimated kernel coefficientsImage through 90,1 91,1 91,0 91,-1

the image

1 0.4643 0.03827 0.3094 0.1881peppers 2 0.4533 0.04502 0.3129 0.1888

3 0.4545 0.04402 0.3130 0.1885

1 0.4499 0.03446 0.3251 0.1905Lena 2 0.4394 0.03673 0.3330 0.1909

3 0.4378 0.03791 0.3334 0.1909

True Kernel________1 0.4375 I 0.0625 I 0.3125 I 0.1875 I

Table 2: Results of error diffusion kernel estimation in the inverse error diffusion problem, i.e.,the kernels are estimated using a given halftone image and the estimated gray scale image inAlgorithm 2. The true kernel used was the Floyd-Steinberg kernel.

number of passes estimated kernel coefficientsImage through 90,1 91,1 91,0 91,-1

the image

1 0.4925 0.02844 0.2587 0.22042 0.4484 0.06945 0.2999 0.1823

peppers 3 0.4109 0.09071 0.3194 0.17914 0.4157 0.09083 0.3128 0.18075 0.4192 0.09152 0.3054 0.1839

1 0.5228 -0.07849 0.2673 0.28832 0.4569 0.01786 0.2923 0.2329

Lena 3 0.4758 0.00667 0.2953 0.22224 0.4721 0.02526 0.2695 0.23315 0.4362 0.02989 0.3052 0.2287

True Kernel________1 0.4375 1 0.0625 1 0.3125 I 0.1875 I

19

1210

------

8

----------

known kernel -

6k

estimated kernel - - -

4

35

34

33PSNR(dB)

32

31

300 2

Figure 13: Comparison of the maximum a posteriori probability projection algorithm in inversehalftoning with and without the assumption of a known error diffusion kernel. The PSNR's werecomputed with respect to half-band filtered versions of the original gray scale images.

the coefficients of the first and the second pass are relatively small. Although the errordiffusion process itself is highly nonlinear, the convergence for the kernel estimation is goodwhere the coefficients of the estimated kernel are within 0.03 of the true one.

When the kernel estimation algorithm is used in the inverse halftoning process, i.e., thekernel estimation is performed using the given halftone image and an estimated gray scaleimage, we still obtain convergence of the kernel. Table 2 shows the estimated kernels for thepepper and Lena images versus the number of passes through the algorithm, i. e., numberof times Steps 3 and 4 of Algorithm 2 are each executed. Despite the fact that we onlyused an estimated gray scale image in the kernel estimation process, Table 2 shows thataccuracy of the estimated coefficients are still quite accurate. The fact that it takes morepasses (processing) for convergence when compared to the results in Table 1 is perhaps notunexpected because we only use an estimated gray scale image in this case. Experimentalresults with many test images so far indicate that convergence can always be achieved. Thisis perhaps a beauty of the well known LMS algorithm. At this point, we do not know of aproof for the necessary and sufficient conditions required for convergence. A rigorous proofof convergence can certainly be a very interesting future research problem.

Finally, we show in Fig. 13 a comparison of performance for the projection algorithm withand without kernel estimation, i.e., with and without the assumption of a known kernel. Wecan observe that there is virtually no difference between the two cases for practical purposes.This means that the projection algorithm can be employed in general inverse halftoning ofany error diffused images.

20

!

6 Discussion

We have suggested and investigated two different approaches for inverse halftoning of errordiffused images. The first approach uses multiple passes of linear filtering and statisticalsmoothing to obtain a gray scale image from an error diffused halftone. Experimental resultsshow that it can reconstruct images of high quality from its error diffused version. Thesecond approach, on the other hand, attempts to find a gray scale image using projectionfrom which the given error diffused images can be produced. In this sense, this projectionapproach can be considered as an exact inverse of error diffusion. Two projection algorithms,viz., maximum a posteriori probability (MAP) projection and minimum mean square error(MMSE) projection, are suggested and tested. It is shown that the MAP projection providesbetter performance than both MMSE projection and the filtering approach by a considerablemargin.

An important assumption in the projection algorithms is that the error diffusion kernel isknown, which can severely limit the applicability of this approach. We devise a method toestimate the error diffusion kernel based on the LMS algorithm in adaptive signal processing.As a result, the kernel estimation method combined with the projection algorithm providesa general approach for the inverse halftoning of any arbitrarily error diffused images. Exper­imental results show that images obtained with and without using a known error diffusionkernel are virtually indistinguishable.

Finally, we remark that inverse halftoning can provide a basis for many binary image process­ing procedures. For example, operations such as binary image scaling, rotation and intensityequalization can be difficult when directly performed in the binary domain. They can bemore easily accomplished, for example, by first converting the binary image to a continuoustone image. The intended image processing procedures are then performed, and finally theresults are rehalftoned. Another example is in the conversion from one type of halftone toanother type, such as from error diffusion to ordered dither, where inverse halftoning is alsoan integral step.

Appendix

A Proof of Fact 1

It is easy to deduce from (2) and (3) that

Um,n E [-,\/2,3,\/2] implies em,n E [-'\/2, ,\/2]

for any (m, n). It is therefore only necessary to proof the bound for Um,n'

Recall the boundary condition that the quantization error ek,l are zero when (k, l) is outsideof the support range of the image. Since the error diffusion process is performed in a rasterscan fashion, we proceed by induction as follows:

First, we haveUO,o = xO,o E [0,'\] c [- ,\/2,3,\/2].

For the processing at the first row, we assume as in standard induction procedure thatUO,k E [-'\/2,3'\/2J for k < n, i.e., eO,n E [-'\/2,'\/2] for k < n. Using (4) and the boundary

21

condition, we have

UO,n = XO,n - L: ho,leo,n-1 E [XO,n - >"/2, XO,n + >../2] C [->../2,3>../2].(O,I)ES

We assume for the general case that Uk,1 E [->../2,3>../2] for (k,l) < (m,n), where a point(k,l) is said to be less than (m,n) if according to the scanning process, the pixel at position(k,l) is processed before the one at (m, n). For the mth row, we have for n = 0 that

Um,n = Xm,n - L: hk,lem-k,n-l.

(k,I)ES

Since S is a set with causal support according to the scan ordering, (m - k, n -l) < (m, n)for any (k, l) E S. Therefore,

Um,n E [xm,n - >"/2, xm,n +>../2] C [->../2,3>../2].

References[1] R. A. Ulichney, Digital Halftoning. Cambridge MA: MIT Press, 1987.

[2] M. Y. Ting and E. A. Riskin, "Error diffused image compression using a halftone-to­grayscale decoder and predictive pruned tree-structured vector quantization," submit­ted to IEEE Transactions on Image Processing, 1992.

[3] Q. Lin, "Halftone image quality analysis based on a human vision model," in Proceedingsof IS&T/SPIE Symposium on Electronic Imaging, (San Jose, CA), February 1993.

[4] C. M. Miceli and K. J. Parker, "Inverse halftoning," Journal of Electronic Imaging,vol. 1, pp. 143-151, April 1992.

[5] M. Analoui and J. Allebach, "New results on reconstruction of continuous-tone fromhalftone," in Proceedings of ICASSP, (San Francisco CA), pp. III 313-316, March 1992.

[6] Z. Fan, "Retrieval of gray images from digital halftones," in Proceedings of ISCAS, (SanDiego, CA), pp. 2477-2480, May 1992.

[7] S. Hein and A. Zakhor, "Reconstruction of continuous tone images from their error­diffused halftone version," in Proceedings of IS&T/SPIE Symposium on Electronic Imag­ing, (San Jose, CA), February 1993.

[8] S. M. Schweizer and R. L. Stevenson, "A Bayesian approach to palette restoration," inProceedings of IS&T/SPIE Symposium on Electronic Imaging, (San Jose, CA), February1993.

[9] R. Floyd and L. Steinberg, "An adaptive algorithm for spatial grey scale," in SIDInternational Symposium) Digest of Technical Papers, pp. 36-37, 1975.

[10] J. F. Jarvis, C. N. Judice, and W. H. Ninke, "A survey of techniques for the display ofcontinuous tone pictures on bilevel displays," Computer Graphics and Image Processing,pp. 13-40, 1976.

22

[16]

[15]

[11] P. Stucki, "MECCA-a multiple-error correcting computation algorithm for bilevel im­age hardcopy reproduction," Tech. Rep. RZ1060, IBM Research Laboratory, Zurich,Switzerland, 198!.

[12] B. W. Kolpatzik and C. A. Bouman, "Optimized error diffusion for image display,"Journal of Electronic Imaging, vol. 1, pp. 277-292, July 1992.

[13] H. Inose, Y. Yasuda, and J. Murakami, "A telemetering system by code modulation­Ll-~ modulation," IRE Transactions on Space Electronics and Telemetry, vol. SET-8,pp. 204-209, September 1962.

[14] J. C. Candy, "A use of limit cycle oscillations to obtain robust analog-to-digital con­verters," IEEE Transactions on Communications, vol. COM-22, pp. 298-305, March1974.

S. Hein and A. Zakhor, "Optimal decoding for sigma delta modulators," Tech. Rep.UCB/ERL M90/51, University of California, Berkeley CA, June 1990.

N. T. Thao and M. Vetterli, "Optimal MSE signal reconstruction in oversampled A/Dconversion using convexity," in Proceedings of ICASSP, (San Francisco CA), pp. IV165-168, March 1992.

[17] R. M. Gray, "Spectral analysis of quantization noise in a single-loop sigma-delta modula­tor with dc input," IEEE Transactions on Communications, vol. COM-37, pp. 588-599,June 1989.

[18] P. W. Wong and R. M. Gray, "Two-stage sigma-delta modulation," IEEE Transactionson Acoustics, Speech, and Signal Processing, vol. ASSP-38, pp. 1937-1952, November1990.

[19] W. Chou, P. W. Wong, and R. M. Gray, "Multi-stage sigma-delta modulation," IEEETransactions on Information Theory, vol. IT-35, pp. 784-796, July 1989.

[20] B. Widrow and S. D. Stearns, Adaptive Signal Processing. Englewood Cliffs, NJ:Prentice-Hall, Inc., 1985.

[21] R. M. Gray, W. Chou, and P. W. Wong, "Quantization noise in single-loop sigma-deltamodulation with sinusoidal inputs," IEEE Transactions on Communications, vol. COM­37, pp. 956-968, September 1989.

23