Paper fingerprinting using alpha-masked image matching

Tuan Q. Pham† Stuart W. Perry Peter A. FletcherCanon Information Systems Research Australia (CiSRA)1 Thomas Holt drive, North Ryde, NSW 2113, Australia.†corresponding author: tuan.pham@cisra.canon.com.au

Abstract

In this paper, we examine the problem of authenticatingpaper media using the unique fibre structure of each pieceof paper (the so-called ”paper fingerprint”). In particular,we look at methods to authenticate paper media when texthas been printed over the authentication zone. We show howalpha-masked correlation [8] can be applied to this problemand develop a modification to alpha-masked correlationthat is more closely matched to the requirements of thisproblem and produces an improvement in performance. Wealso investigate two methods of pixel inpainting to removeprinted text or marks from the authentication zone and allowordinary correlation to be performed. We show that thesemethods can perform as well as alpha-masked correlation.Finally two methods of improving the robustness to forgeryare investigated.

1. Introduction

When a reasonable facsimile of an object can be createdat a cost less than the value of the object, the objectmay be become a target for forgery. The concept of anobject’s ”value” can be defined in many different ways.Paper currency is an example of an object with a cost tomanufacture that is often small compared to the value societyplaces on the object. Even plain paper documents whosecost is almost negligible to produce can have a very largevalue due to the importance of the information printed on thedocument. In recent times the availability of cheap printingtechnology has further reduced the cost of manufacturefor paper objects, increasing the temptation of forgery andimproving the quality of forgeries.

There have been many different strategies employed toprotect objects from forgery. One strategy is to make theequipment required for manufacture of the authentic objectso expensive that forgery is discouraged. For example, theaddition of holograms, special paper and/or inks or finelyprinted details.

Another approach that has gained interest recently issecurity based on the inherent randomness present in manyobjects. An ordinary piece of office paper when viewed atthe micrometer scale is a highly random arrangement offibres and filler material. Every sheet of paper has a different

arrangement of fibres and no two regions on the same sheetof paper have the same arrangement. The randomness of thisarrangement can be exploited to create a unique signature forthe sheet of paper. A recent article described the creation ofunique signatures for a variety of common objects such aspaper, coated cardboard packaging and matt-finish plasticcards using the diffuse reflection from a laser focussedonto a small region of the object [3]. A unique signaturecould be obtained even when the object had been soakedin water and then dried, baked in an oven, crumpled andcreased, or written over with a ballpoint pen or thick marker.An object could be authenticated by cross-correlating thesignature obtained from the object with those in a databaseof signatures of authentic objects.

The basis of this idea has been around for a while. It canbe traced back as least as far as US patents filed in 1981[9] and 1984 [10]. In these patents the inventor disclosesthe idea of using information about inherent irreproduciblerandomness within an object imaged by an optical system toform a unique signature for the object. More recently it hasbeen noted that a coherent light source is not required fora unique signature to be computed for paper [16]. Indeed,the earlier work [9], [10] does not assume a coherent lightsource. By simply imaging the paper surface illuminatedby incoherent light at high resolution, sufficient informationabout the random structure of the paper surface can beobtained to create a highly unique signature for the paper[16]. The authors have found that the requirements forimaging a piece of common office paper to obtain a highlyunique signature are easily met by many common consumerdocument scanners. In addition, experiments we have per-formed indicate that the signatures obtained are robust todamage to the paper such as light staining, wrinkling, andwetting followed by drying. The use of a desktop scannerfor this application provides a number of benefits includingremoving the requirement for specialised equipment to readthe signatures. In this work the signature we use is simply agreyscale 8-bit 256 by 256 pixel image of the paper surfacecaptured by the scanner at 600dpi. In [3] and [16], cross-correlation was used to match an obtained signature with apreviously stored signature, and this approach can be used tomatch the above image signatures from a desktop scanner.

However, simply cross-correlating an image of the surfaceof the paper does not provide a signature authentication

system that is robust to marks, writing or printing acrossthe signature zone. Such marks can obscure the signature ofinterest in the cross-correlation operation and increase thelikelihood of an authentic object being declared unauthentic.There are many applications of this technology where itwould be useful to allow the user to print additional in-formation onto the document after the signature is collectedand still have the object be recognised as authentic. In [2], amethod was described where the image of a signature zonewas filtered before comparing the signature with that in adatabase using cross-correlation. The filtering was intendedto remove the effect of any marks within the signaturezone, such as damage or printed text. This method has adisadvantage that the filtered pixels are still included in thecross-correlation calculations and hence the filtered pixelsdegrade the accuracy of the signature matching. In fact themethod presented in [2] would only be effective when thearea of damage or printed text was small compared to thesize of the signature zone. Unfortunately, [2] contains almostno implementation information about the method presentedand no actual numerical results. For this reason, a directcomparison between this method and the presented approachis not possible.

In this paper we present signature matching algorithmsthat are robust to marks, handwriting or text printed over thesignature region either before the initial reference signatureis collected, or subsequently. We do this by using severalmodified signature matching algorithms which are based oncross-correlation, but are tailored to exclude pixels deter-mined to correspond to printed text from the calculation.

In section 2 we review the alpha-masked correlation ofFitch, et al. [8], and present a modification to the alpha-masked correlation algorithm to better account for back-ground illumination variation. Experiments are performedto show how alpha-masked correlation can be applied to theproblem of paper authentication. In section 3, alternativealgorithms to solve this problem using pixel inpainting arepresented and experiments performed. Section 4 presentstwo methods to improve the robustness to forgery of theproposed methods. Section 5 presents the conclusions.

2. Correlation-based alpha-masked imagematching

Alpha-masked image matching is a technique to matchsub-regions of two images. The Regions Of Interest (ROI),defined by alpha masks α1 and α2, allow registration ofnon-rectangular objects. The alpha mask 0 < α(x, y) < 1can be seen as a weighting map, where a weight of zeroat (x, y) means that the pixel at (x, y) should be removedfrom consideration. When pixels have differing α-values, itmeans that some are considered more important than others.

2.1. Fitch’s alpha-masked correlation

Suppose the two ROIs from images f1 and f2 are relatedby a translation (x0, y0) and unknown noise n:

f2(x − x0, y − y0) = f1(x, y) + n(x, y),

Fitch et al. [8] define a weighted error function:

E(x0, y0) =∑x,y

(n(x, y))2α1(x, y)α2(x − x0, y − y0)

=∑x,y

(f1(x, y) − f2(x − x0, y − y0)

)2

α1(x, y)α2(x − x0, y − y0) (1)

which can be written as a summation of three correlations:

E(x0, y0) = α1f22 ⊗ α2 − 2α1f1 ⊗ α2f2 + α1 ⊗ α2f

22

where (f ⊗ g)(x0, y0) =∑

x,y f(x, y)g(x − x0, y − y0)denotes the correlation between f and g. Once the errorfunction is calculated, the translation between the two im-ages can be determined by finding the coordinates (x0, y0)at which E(x0, y0) is minimum.

To further simplify the representation of the error func-tion, let us make the following definition: Cmn = α1f

m1 ⊗

α2fn2 . With this notation, the error function can be expressed

as:E = C20 − 2C11 + C02

A problem with the above error function E is that the sumin (1) contains a different number of terms for each (x0, y0),depending on how often α1(x, y) and α2(x−x0, y−y0) arezero at the same time. Thus, [8] normalizes the error functionby the area of overlap between α1 and α2:

Eα =1

C00(C20 − 2C11 + C02) (2)

2.2. Correlation vs matching: peaks vs troughs

Image matching as described above is presented as aleast squares estimation problem, in which the best matchbetween two images is regarded as the position at whichthe squared, normalized differences between two images isat a minimum, with a perfect match achieving a value ofzero. This is at variance with the view of image matchingby correlation, in which the best match between two imagesis regarded as the position with the maximum correlationvalue. In the case of image matching, finding the best matchis equivalent to finding the minimum value in an image. Inthe case of correlation, finding the best match is equivalentto finding the maximum value in an image.

If one attempts to estimate the statistical significance ofa match, then a maximum correlation value is easier tointerpret than a least squares error, as the magnitude ofthe correlation peak can be compared against an assumed

distribution of non-matching correlation values, allowingstatistical inferences to be drawn from the data. For example,a normal distribution could be used, as the correlation imagecontains both positive and negative values, unlike the matchimage, which contains only positive values.

A simplistic method is used to convert the least-squareserror value returned from image matching to somethingsimilar to a magnitude value from a correlation process.The least-squares image is simply negated, normalized toremove any DC offset, and divided by the root-mean-squarevalue of the image. This results in an matching image witha standard deviation of 1, and as long as the width ofthe correlation peak is small compared to the size of thecorrelation image, the peak strength can be analyzed in termsof the number of standard deviations away from the meanvalue. The correlation image around the peak may also beprocessed using quadratic interpolation to estimate a sub-pixel accurate peak position and peak magnitude.

2.3. Normalized correlation of background sub-tracted images

Image matching using the alpha-masked correlation for-mula (2) is not robust to different gains and offsets betweenthe images being matched due to the mean squared differ-ence in (1). To make the algorithm robust to different gainsand offsets, the input images should be normalized to havezero mean and unit standard deviation over the masked areabefore correlation. This is easily done for images of paperbecause blank paper usually has a flat intensity distribution.The offset is removed first by subtracting from f1 and f2

their background intensities (average intensity of the non-printed area). The gain is then normalized by the standarddeviation over the same background area.

For more complicated images that do not have a flatbackground, a background image f can be estimated usingnormalized convolution [15]:

f =(αf) ∗ k

α ∗ k(3)

where k is a blur kernel with large support such as Gaussianblur of σ=5 pixels and ∗ is the convolution operator.

After background subtraction, the terms C20/C00 andC02/C00 in (2) are simply variance estimates of the pa-per background. Although these estimates do vary due todifferent areas of overlap C00, the variation is small ifthe overlapping area of the two masks does not changesubstantially over all x0, y0 offsets. As a result, C20/C00

and C02/C00 can safely be removed from the error function(2), leaving a single term in the normalized correlation errorfunction:

EN =α1f1 ⊗ α2f2

α1 ⊗ α2=

C11

C00. (4)

Compared to Eα in (2), EN requires only half the numberof correlations to compute. EN also produces a correlation

3.33 (3.46)

2.02 (6.22)

3.89 (2.91)

4.08 (3.36)

(a) paper 1 before printing (b) paper 2 before printing

(c) paper 1 after printing (d) paper 2 after printing

23.16 (3.06) 25.64 (3.31)

Figure 1. Paper matching using alpha-masked correla-tion versus (cross correlation).

peak at the matching offset rather than a trough like Eα.As a result, a strong peak in EN signifies a good matchbetween the two input images.

2.4. Paper matching using alpha-masked correla-tion

To demonstrate that a paper can be uniquely identifiedeven after substantial marking, we perform a simple papermatching experiment. The experiment involves two differentpieces of paper scanned before and after text printing. The600dpi paper scans are subjected to alpha-masked corre-lation and cross-correlation. We show that while differentsheets of paper with the same printed text could be mis-classified as matching by standard correlation, alpha-maskedcorrelation correctly classifies them as non-matching.

Figure 1a and 1b show two 256×256 scans of two blanksheets of paper. The printed versions of these sheets ofpaper are in Figure 1c and 1d. The intensities of these 8-bitimages are linear stretched between [200 255] so that thepaper texture is more visible in Figure 1. There is a smalltranslation between images 1a and 1c and 1b and 1d dueto the feeding mechanism of the scanner. Also due to thesame positioning error, the text in Figure 1c and 1d appearslightly shifted, even though the same text was printed. Thealpha masks are constructed by thresholding the imagesat intensity 160 followed by two iterations of 8-connectedbinary erosion.

We performed alpha-masked correlation and cross-correlation between every image pair and the correlationpeak strengths defined in section 2.2 are displayed in be-tween the paper images in Figure 1 (the cross-correlation

peak strengths are displayed within brackets). Zoomed-inversions of the correlation images are also displayed belowthe peak strength numbers. The contrast of these correlationimages has been enhanced to visualize the correlation peaksmore effectively.

The most striking difference between alpha-masked cor-relation and cross-correlation occurs between image pairsfrom the same paper before and after printing. While alpha-masked correlation produces high peaks around 25 formatching papers, cross-correlation peaks remain low around3. The correlation images below these numbers (in bold inFigure 1) also confirm very sharp peaks when using alpha-masked correlation. The cross-correlation peaks, on the otherhand, are barely visible. Cross-correlation peak values of3.06 and 3.31 between the matching pairs are no better thanthe peak values of non-matching pairs.

Another interesting result is the correlation between thetwo different pieces of paper after printing. While alpha-masked correlation correctly gives a small correlation of2.02, cross-correlation produces a visible peak at an appre-ciable strength of 6.22 (in italic in Figure 1). This visiblepeak is a result of correlation of the printed text, not thetexture of the papers. Cross-correlation is therefore notrobust to printing.

3. Alpha-masked image matching using in-painting

Image inpainting is a technique to fill missing pixels ofan image with plausible intensities. The missing pixels maybe lost during transmission or they may cover unwantedobjects to be erased from the image. In a typical solution[1], the image is decomposed into texture and structurecomponents. Texture synthesis is then used to fill in thetexture component, whereas anisotropic diffusion is usedto transport intensities along iso-contours into the missingregions of the structure component.

This paper does not introduce a new inpainting solution.Rather, a smooth infilling of masked images followed bycross-correlation is presented as an alternative to alpha-masked correlation. Laplacian-based smooth inpainting [5]minimizes sudden intensity changes at mask boundaries.Spectrum distortions are therefore minimized. The inpaintedimages can then be matched reasonably well even when themasking area is large.

3.1. Laplacian pyramid

A Laplacian pyramid [4] is an over-complete representa-tion of an image using a set of band-pass images at succes-sively reduced dimensions. The decomposition of an inputimage into its Laplacian pyramid is illustrated in Figure 2a.The input image f0 is decomposed into a high-pass image h0

and a low-pass image l0. The high-pass image h0, a.k.a. the

Laplacian image, is stored as the base level of the pyramid,while the low-pass image l0 is losslessly downsampled(requires band-limitedness). The downsampled image f1 isagain subjected to low- and high-pass decomposition. Thisprocess is repeated recursively until a desired number ofpyramid levels are obtained (3 levels as shown in Figure 2).

Because the Laplacian pyramid is generated using losslessoperations, the original image can be reconstructed perfectlyfrom the Laplacian pyramid as shown in Figure 2b. Althoughthe Laplacian pyramid requires 33% more storage than a sin-gle image, it allows separate manipulation of different band-pass images. This is especially useful in texture generationor infilling, where the texture is different at different scales.

3.2. Alpha-masked Laplacian pyramid

The concept of Laplacian pyramid can be extended toimages with uncertain data. The data uncertainty is specifiedby an alpha mask 0 ≤ α ≤ 1, where zero corresponds toinvalid data and one for valid data. Rather than the low-pass operator BLUR in Figure 2a, Normalised Convolution(NC) [15] is used to smooth out an alpha-masked image bya kernel k:

l0 =(α0f0)∗k

α0 ∗ kh0 = α0(f0−l0) α1 ={α0 ∗ k}↓ (5)

where f0 and α0 are the input image and its alpha-mask,l0 and h0 are the low- and high-pass decompositions, α1 isa generated alpha-mask for the downsampled image f1, ∗and ↓ denote a convolution and a downsampling operatorrespectively.

As can be seen in Figure 3, the normalised convolutionoperation imports valid intensities into the masked areas. Itdoes so in a similar fashion to image morphology where themasked regions are successively eroded and filled with localmean intensities. Note that in (5) the high-pass image hi

has to be multiplied with the mask αi of the same pyramidlevel to invalidate the influence of the masked pixels. TheLaplacian pyramid generated this way is compatible with anormal Laplacian pyramid. The difference happens in thereconstructed image, where invalid pixels are automaticallyfilled with mean intensities from surrounding valid pixels.

3.2.1. Implementation details. To avoid the need for alossless subsample operation during pyramid construction,the high-pass image hi is computed slightly differently from(5):

hi = αi(fi − {fi+1} ↑) (6)

where ↑ denotes an upsampling operator. In this paper,we use a upsampling/downsampling factor of two. Theblur kernel is a separable Burt-Adelson [4] 5-tap filter[0.05 0.25 0.4 0.25 0.05] (which approximates a Gaussianfilter of σ = 1). The Laplacian images are successivelygenerated until the alpha mask αi no longer contains anyzero pixels.

(a) Laplacian pyramid construction from an image (b) Image reconstruction from a Laplacian pyramid

Figure 2. Laplacian pyramid as an over-complete representation of an image.

SUBSAMPLE NC NC

BLUR

_ +

_ +

SUBSAMPLE

f0 l0

h0

h1

f1 l1

α0 α1

f2 SUBSAMPLE

Alpha-masked Laplacian pyramid

Figure 3. α-masked Laplacian pyramid construction.

3.2.2. Smooth inpainting results. The results of smoothimage inpainting for two natural images can be seen inFigure 4. Compared to the images with average intensityinfilling on the left, the inpainted images on the right lookmuch more pleasing. The text masks are almost invisible.Because image inpainting transports neighboring intensitiesinto the masked areas, low-frequency details are recon-structed very well. The missing of information is onlydetectable at textured areas like the fur of Lena’s hat orthe stripe pattern of Barbara’s clothes.

3.3. Alpha-masked image matching using inpaint-ing

The alpha-masked images can be inpainted before match-ing to avoid extra correlations in the alpha-masked andnormalized correlation. By matching two scans of the samepaper before and after text printing, we show that inpainted

(a) Lena with mask (b) Inpainted result of (a)

(c) Barbara with mask (d) Inpainted result of (c)

Figure 4. Smooth inpainting of natural images.

correlation sometimes produces better peak detection thanalpha-masked correlation. Infilling using average local in-tensity performs equally well as normalized correlation withbackground subtraction. This is understandable since bothinfilling and background subtraction essentially fills themasks with local average intensities.

3.3.1. Inpainted paper matching. Two inpainting resultsfor a scan of a printed paper using mean intensity infill-ing and smooth inpainting are shown in Figure 5. The256×256 central region of Figure 5a is matched against theblank paper signatures in Figure 1a-b. Due to optical dotgain [11], the intensities around printed texts are slightlydarker than the background. Smooth inpainting in Figure5b transports these dark intensities into the masked area,whereas mean infilling in Figure 5a uses the correct average

(a) mean intensity infilling (b) smooth Laplacian inpainting

Figure 5. Paper 1 with printed text after inpainting.

alpha-masked normalized mean-filled inpaintingmatch 23.16 22.32 22.31 14.17non-match 4.08 2.92 2.94 2.68ratio 5.67 7.63 7.59 5.28

Table 1. Matching a printed paper (Fig. 1c) with twofingerprints (match = Fig. 1a, non-match = Fig. 1b).

background intensity. This gives rise to a lower fingerprintmatch strength for the smooth inpainted image compared tothat of the mean-filled image (first row of Table 1). However,both infilling methods produce substantially higher matchstrengths than the correlation peak strengths of non-matchingpapers. Infilling is therefore an effective yet inexpensivealpha-masked image matching method.

3.3.2. Robustness against mask fill factor. One big advan-tage of alpha-masked image matching over other signatureverification techniques such as [2] is the ability to handle alarge area of alteration from the original signature. In Figure6, we investigate the performance of alpha-masked imagematching at different levels of the fill factor of the mask.The masks in Figure 6a are successively eroded to reducetheir fill factors (defined as ratio of the valid area in whiteover the area of the whole mask). This mask is used withthe printed image in Figure 1c to match against two blanksignatures in Figure 1a-b. The resulting match strengths areplotted against fill factors in Figure 6b as continuous linesfor matching pair and dashed lines for non-matching pair.

It is interesting to see that alpha-masked image matchingstill produces a high matching strength after more than 80%of the original image has been masked out (continuous linesin Figure 6). Normalized correlation and mean-filled correla-tion performs equally well, both are much better than alpha-masked correlation and inpainted correlation. The perfor-mance of alpha-masked correlation, while being preferablefor large fill factors (as shown in Table 1), degrades quicklyas the fill factor decreases. All four methods in Figure6b also produces consistently low correlation between non-matching paper pairs (dashed lines in Figure 6).

An additional experiment was performed to test theproposed method’s robustness to mask fill factor. A page

…

fill = 0.77

fill = 0.58

fill = 0.45

fill = 0.05

(a)mask

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.80

5

10

15

20

25

30

35

40

45

50

mask fill factor

mat

ch s

tren

gth

alpha−masked correlation [8]normalized correlation (section 2.3)mean−filled correlation (section 3.3)inpainting correlation (section 3.3)correlation of non−matching pairs

(b) match strength versus mask fill factor

Figure 6. Correlation results under fill-factor stress test.

containing Japanese test was scanned twice on an EPSON10000XL scanner. Japanese text was chosen for this experi-ment because compared to English text, the different spatialfrequency characteristics of the three Japanese character sets(kanji, hiragana, and katakana) would be likely to producea wider spread of mask fill factors on which to test theproposed method. To mitigate the effects of fixed patternscanner noise on the result, each scan was performed at adifferent position on the scanner platen. Both scans werecollected as 8bit greyscale images with a resolution of600dpi. The scans were roughly aligned by eye; howeverno attempt at electronic correction of residual rotationalmisalignment between the images was made. Each scan wasdivided into 240 256×256 pixel image patches and eachpatch in the first scan was matched against the correspondingpatch in the second scan using alpha-mask correlation [8].The match strength and mask fill factor for each patch isshown in Figure 7 below. It should be noted that each pointon Figure 7 represents a different part of the document, andtwo iterations of 8-connected binary erosion of the printedcontent was used to generate different fill-factors.

This experiment was performed using a C implementationof the normalised alpha-masked correlation method on acomputer running Microsoft Windows XP, Service Pack 3.The computer had two Intel Xeon 5060 Dual Core CPUsrunning at 3.20GHz, with 3.25 GBytes of RAM. The 240match operations took a total of 161.48 seconds, givingan average time of 0.67 seconds per match operation. Nospecial optimisation of the code to take advantage of themultiple CPUs or multiple cores was performed.

4. Improving the robustness of paper finger-printing

Being a pure image-based method, the paper verificationtechnique using correlation is susceptible to attack if a forger

…

fill = 0.94

fill = 0.86

fill = 0.73

fill = 0.16

(a)ROI

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

5

10

15

20

25

mask fill factor

mat

ch s

tren

gth

alpha−masked correlation [8]

(b) match strength versus mask fill factor

Figure 7. Paper matching on a Japanese document.

manages to print a desired paper signature onto a fakemedium. While we believe that it is difficult to reproducepaper textures at different resolutions, we want to improvethe robustness of our paper fingerprinting technique. Thissection presents two such methods to address the robustnessto forgery problem.

4.1. Multiple orientation paper fingerprinting

The reflection of light from most turbid materials consistsof two major components: the surface and the subsurfacereflectance. Surface reflection, a.k.a. specular reflection, isthe mirror-like reflection of light at the paper surface. Due tothe roughness of paper, this surface reflection is more diffusethan the specular reflection from a smooth surface (seeFigure 8f). Subsurface reflection, a.k.a. diffuse reflection,is the reflection of light from within the paper after lightpenetrates the surface and scatters within the substrate (seeFigure 8e).

The diffuse and specular reflectance components can beseparated from an image of a paper using photometric stere-ography [14]. As illustrated in Figure 8a-b, two scans fromopposite orientations (i.e. 180◦ difference in illuminationangle) obtained by the same scanner setting are required.Because shadows in one image correspond to specular reflec-tion in the other image, the specular reflectance is roughlycancelled when the two aligned images are added together.The diffuse reflectance is derived in Figure 8c as the sum ofFigure 8a and 8b. The specular reflectance in Figure 8d isthe difference of Figure 8a and 8b. As expected, the specularimage looks shiny, whereas the diffuse image looks dull andcontains pores from the filler material within the paper. Avery similar diffuse reflectance image is obtained from twoaligned scans at 90◦ and 270◦ orientation.

Different from [6], which bundles the diffuse and specularreflectance into a feature vector for matching purposes, we

_ + + +

(a) paper scanned at 0º (b) paper scanned at 180º

(c) diffuse reflectance (d) specular reflectance

(f) (e)

Figure 8. Diffuse and specular components of paper.

propose to match the two reflectance components sepa-rately to improve robustness against forgery. The specularreflectance image could also be turned into a surface depthmap using shape from shading [13]. While it is possibleto forge a 3D surface using molds or very fine topologycreation tools, the forgery needs to have a matching diffusereflectance at precisely the same alignment with the 3Dtopology. This makes it more difficult for a document to beforged at a cost of one extra scan of the candidate document.

4.2. Double-sided paper fingerprinting

Another way to improve the robustness of paper finger-printing is to verify its signatures on both sides. During aregistering phase, the signatures for both sides of a documentare collected. These signatures are stored in a database to-gether with the displacement between the signatures. Whenan object is presented for authentication, the signatures onboth sides of the document are scanned. These candidatesignatures must match the signatures in the database andthe alignment of the signatures in the database must alsobe the same before a match is declared. This uses the factthat the rear side fingerprint can be detected when the frontside of the paper is scanned and visa-versa (see Figure 9).This allows accurate alignment determination between thefront and back signatures. With the double-side approach,a forger is required to match signatures on both sides ofthe document and match the relative positions betweenthe signatures on both sides. This greatly complicates theforgery process while still using only an inexpensive scannerduring verification.

5. Conclusion

In this paper we have looked at the problem of matchingpaper media in the presence of printed text, or other mark-

(a) front side at 0◦ (b) back side at 0◦ (c) correlation of a&b

Figure 9. Matching different sides of paper at 1200dpi.

ings, on either one or both of the paper media to be matched.We examined Fitch’s formulation of alpha-masked correla-tion [8], and adapted this method to the problem of interest.In particular, Fitch’s method is modified to better handlethe problem non-uniform background illumination and is re-normalised. An experiment was performed that showed thatalpha-mask correlation is effective at matching paper withprinted text present, whilst ignoring correlations amongst theprinted text that might otherwise create false matches. Ourmodified version of alpha-mask correlation was shown toout-perform the basic alpha-mask correlation in some cases.As alternatives to the alpha-masked correlation approach, weinvestigate two methods of pixel inpainting; mean intensityin-filling of pixels identified as text and an approach basedon the Laplacian pyramid decomposition of the paper image.After in-painting text or markings on the paper image,ordinary cross-correlation may then be used to compute amatch strength for the paper media. We performed a numberof experiments and found that Laplacian pyramid inpaintedimages followed by ordinary cross-correlation perform noworse than alpha-masked correlation in this problem. Infact, the Laplacian pyramid inpainted augmented methodperformed better in some cases. The mean-filled pixel in-painting method performed even better than the Laplacianmethod, equalling the performance of our modified versionof alpha-masked correlation.

In addition we investigated methods to further improve thesecurity of paper authentication. The first method involvedscanning the paper from two different directions and de-composing the scans into a specular and diffuse component.The specular component is mostly a function of the 3Dsurface structure of the paper, while the diffuse componentis mostly a function of the internal structure of the paper.By separating these components, two pieces of paper canbe matched on the internal and surface structure separately,improving the robustness to forgery. The second methodmade use of the fact that for many ordinary office papers,the fingerprint collected on one side of the paper can bedetected in the image collected on the other side of the paper.By collecting signatures from both sides of the paper, andmeasuring the through paper displacement of the signatures,a greater robustness to forgery can be achieved.

Aspects of this work are the subject of a number of

published and unpublished patent applications [7], [12].

References

[1] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. Simultaneousstructure and texture image inpainting. IEEE Trans. on IP,12(8):882-889, 2003.

[2] Y. Boutant, D. Labelle, and H. Seux. Method for recognitionand tracking of fibrous media and applications of such amethod, particularly in the computer field. US patent applica-tion 2005/0257064, filed in Apr. 2005.

[3] J. D. R. Buchanan, R. P. Cowburn, A.-V. Jausovec, D. Petit,P. Seem, G. Xiong, D. Atkinson, K. Fenton, D. A. Allwood,and M. T. Bryan. Forgery: ’fingerprinting’ documents andpackaging. Nature, 436:475, 2005.

[4] P. J. Burt and E. H. Adelson. The Laplacian pyramid asa compact image code. IEEE Trans. on Communications,32(4):532-540, 1983.

[5] P. J. Burt and E. H. Adelson. Method for compensating forvoid-defects in images. US patent 4698843, RCA Corporation,Oct. 1987.

[6] W. Clarkson, T. Weyrich, A. Finkelstein, N. Heninger, J. A.Halderman, and E.W. Felten. Fingerprinting blank paper usingcommodity scanners. In Proc. IEEE Symposium on Securityand Privacy, 2009.

[7] S. E. Ecob, S. W. Perry, and S. A. Rudkin. Authenticatingpartially transparent medium. Australian patent applicationAU2007254655, Canon Kabushiki Kaisha, filed in Dec. 2007.

[8] A. J. Fitch, A. Kadyrov, W. J. Christmas, and J. Kittler. Fastrobust correlation. IEEE Trans. on IP, 14(8):1063-1073, 2005.

[9] R. N. Goldman. Non-counterfeitable document system. USpatent 4423415, Light Signatures Inc., Jun. 1981.

[10] R. N. Goldman. Verification system for document substanceand content. US patent 4568936, Light Signatures, Dec. 1984.

[11] S. Gustavson. Dot gain in color halftones. Ph.D. disertationno. 492, Linkoping University, Sweden, 1997.

[12] S. J. Hardy, K. G. Larkin, S. Farrar, P. A. Fletcher, and T. Q.Pham. Alpha-masked image matching. US patent application2009/0003601, Canon Kabushiki Kaisha, filed in Jun. 2008.

[13] B. K. P. Horn and M. J. Brooks (eds). Shape from shading.MIT press, 1989.

[14] P.-A. Johansson and P. Hansson. Method of determining anilluminated surface. US patent 6757065, STFI, 2004.

[15] H. Knutsson and C.-F. Westin. Normalized and differentialconvolution: methods for interpolation and filtering of incom-plete and uncertain data. In Proc. of CVPR, 1993.

[16] H. Sugino, K. Ito, and T. Shimizu. Document verificationsystem. US patent 6948657, Fuji Xerox Co. Ltd., Apr. 2003.