takao baker shi

8/14/2019 Takao Baker Shi

1/8

Steady-State Feedback Analysis of Tele-Grafti

Naoya Takao Simon Baker Jianbo Shi

Matsushita Research The Robotics Institute The GRASP LaboratoryMatsushita Corporation Carnegie Mellon University University of Pensylvania

Osaka, Japan Pittsburgh, PA 15213 Philadelphia, PA 19104

We analyze the feedback loop in Tele-Grafti, a camera-projector based remote sketching system which we recentlydeveloped. We derive the gain through the feedback loopand the nal images that will be viewed by the users of thesystem. We then derive the optimal gain as the gain thatresults in the nal viewed images being as close as possibleto the sum of the images actually drawn on the paper. Wealso propose a way of using the projector to compensate forinsufcient ambient light and describe how our analysis isaffected by: (1) automatic gain control (AGC) and (2) colorimagery. We end by proposing an algorithm to decomposethe nal viewed images into estimates of the drawn images.

1 IntroductionSystems combining video cameras and LCD projectors arebecoming more and more prevalent. The rst such systemwas Pierre Wellners Xerox DigitalDesk [Wellner, 1993 ].Other such systems include the University of North Car-olinas Ofce of the Future [Raskar et al. , 1998 ], INRIAGrenobles MagicBoard [Hall et al. , 1999 ], and Yoichi

Satos Augmented Desk [Sato et al. , 2000 ]. Recently,there has been a growing body of work in the computer vi-sion literature aimed at using video cameras to improve theuse of projectors as display devices [Raskar and Beardsley,2001, Sukthankar et al. , 2001a, Sukthankar et al. , 2001b ].

Recently we have developed Tele-Grafti, a camera-projector based remote sketching system [Takao et al. ,2003 ]. Tele-Grafti allows two people to communicate re-motely via hand-drawn sketches. What one person drawsat one site is captured using a video camera, transmitted tothe other site, and displayed there using an LCD projector.See Figure 1(a) for a schematic diagram of two Tele-Graftisites connect by a network and Figure 1(b) for an image of

a real Tele-Grafti system. See the accompanying movietelegrafti-uidemo.mpg for an example of two people in-teracting using Tele-Grafti. Also, see [Takao et al. , 2002 ]for larger versions of the gures in this paper.

The combination of projectors and cameras in the samesystem can either be benecial or a hindrance, dependingon what you want to do. In hand-tracking systems for desk-top user-interfaces, the light projected by the projector canmake simple background subtraction algorithms inapplica-

ble. Instead, several groups have used infra-red cameras forhand-tracking, a modality unaffected by the light radiatedby the projector [Sato et al. , 2000, Leibe et al. , 2000 ]. Onthe other hand, the projector can be used to project struc-tured light, thereby allowing the estimation of 3D shape us-ing the laser range nding principal. By carefully syn-chronizing the camera and the projector, the structured lightcan even be time-multiplexed with the projected image tomake it imperceptible to humans [Raskar et al. , 1998 ].

A particular problem in certain applications is feedback .If the projector radiates light that is then imaged by the cam-era, and the resulting image then passed back to the projec-tor, potentially after being processed in one way or another,there is the potential for feedback. It is possible that the projected image may get brighter and brighter until the camerasaturates resulting in an unusable system. Alternatively, thesystem might oscillate between two states. Finally, there isthe possibility of visual echoing [Takao et al. , 2002 ]. Onthe other hand, feedback can be used to help the system, asis done in [Sukthankar et al. , 2001a ] for shadow removal.

Camera-projector based communication systems are all

susceptible to feedback. When we rst gave a demonstra-tion of Tele-Grafti at ICCV [Takao et al. , 2001 ], the sys-tem initially suffered from severe feedback, quickly satu-rated, and became unusable. At the time we added extralights as a quick x. In this paper we study the cause of the feedback and propose a principled solution to it.

In particular we present a steady-state analysis of thefeedback in Tele-Grafti. Much of the analysis is applicableto other camera-projector systems. Starting with photomet-ric models of the projector, the camera, and the paper usedto sketch on, we derive the gain of the system, the nal statethat the system converges to, and the nal image that willbe viewed by the users. Based on this result, we derive the

optimal gain as the gain that results in the nal viewedimage being as close as possible to the sum of the two orig-inal, non-feedback, images. Another common problem withLCD projector-based desktop applications is that the ambi-ent light is sometimes very weak compared to the projector.(This was the problem at the ICCV demo.) We propose away of using the projector to augment the ambient light inthe scene, and derive appropriate settings for this scheme.We also briey discuss how our analysis is affected by au-

1


2/8

TeleGraffiti Site 1

Camera 1 Camera 2

Projector 1 Projector 2

PC 1

TeleGraffiti Site 2

PC 2

Mirror 2Mirror 1

Paper 1 Paper 2

Real Network Connection

Virtual Connections

Mirror

LCD

Camera

Projector

Paper

PC

(a) Schematic Diagram of a Two Site Tele-Graf ti System (b) Image of a Real Tele-Graf ti Site

Figure 1: (a) A schematic diagram of two Tele-Graf ti sites connected by a computer network [Takao et al. , 2003 ]. (b) One site in areal Tele-Graf ti system. The cameras at both sites continuously capture video of what is written on the paper. The video is digitized,compressed, and transmitted to the other site where it is projected onto the paper. Since the pieces of paper can be moved, they have to betracked in real time at both sites. We can then warp the video appropriately to correctly align the projected video with the paper. See theaccompanying movie telegraf ti-uidemo.mpgfor an example of two people interacting using Tele-Graf ti.

tomatic gain control (AGC) and color imagery.The nal steady state images viewed by the users of

Tele-Grafti are a weighted combination of the images that

would have been imaged with Tele-Grafti switched off. Inmany scenarios it is desirable to estimate what was actuallydrawn on the paper by the system users. We end this paperby proposing an algorithm to decompose the steady stateimages into estimates of what was drawn on the paper.

2 Image Formation ModelWe illustrate our image formation model in Figure 2. Lightis projected by the projector, reected by the mirror ontothe paper where it is combined with the ambient light in thescene, and reected again to be captured by the camera.

2.1 Projector ModelWe assume that if a pixel in the LCD projector is set toproject intensity

, the radiance of the lighttransmitted by the LCD image plane is:

" !$ # & %'

(1)

where#

is the gain of the projector; i.e. we assume thatthe response of the projector is linear. The derivation of the relationship between the radiance of the light transmit-ted by the LCD and the irradiance of the light that is -nally received by the paper is similar to the derivation of the relationship between scene radiance and image irradi-

ance for a camera (see [Horn, 1996 ] Section 10.3) exceptthat the plane of the paper that the light is projected ontois not frontal, unlike the camera image plane. Taking intoaccount this foreshortening, the relationship becomes:

) (

!

%1 0

2 ) 3

5 47 6

%9 8' @ AC BE DE

%9 8' @ AG F

(2)

where(

is the irradiance of the light received at the paper,0

is the area of the projector lens,3

is the distance from

the lens to the paper (via the planar mirror, which can beignored otherwise),

DH

is the angle that the principal raymakes with the optical axis, and

FI

is the angle that the

principal ray makes with the normal of the paper.

2.2 Paper ModelWe assume that the paper can be modeled as Lambertian. If the total ambient incoming irradiance is

Q P

and the albedoof the paper is RI S

, the radiance of the reected light is:U T

!

R S

%V (X W

P

U Y

(3)

2.3 Camera ModelThe model of the camera is similar to the model of the projector. If the radiance of the paper is

T

, the irradiance of the

light captured on the image plane of the camera is:) `

!

T

% 0

`

2 U a

`

4 6

% 8b @ A7 cH D

(4)

where 0`

is the area of the lens,a

`

is the distance betweenthe lens and the image plane, and

D

`

is the angle betweenthe principal ray and the optical axis [Horn, 1996 ]. We alsoassume that the response of the camera is linear:

`

!$ #

`

%

`

(5)

where#

`

is the gain of the camera and

`

d

.

2.4 Complete Imaging ModelPutting together Equations (1)(5), we obtain a relationshipbetween the intensity of the pixel in the projector

andthe intensity of the corresponding pixel in the camera

:

`

!f e

`

%

R S

%V

P

W

eg h %9 b

(6)

wheree

`

!X #`

%p ir qs

tv u

qsx w y

% 8' @ A

c

D`

andeg ! # %

i

s

t

s w y

%v 8b @ A

B

DE V %

8b @ AG F

. Equation (6) says that the image captured by the ith

2


3/8

Light: lReflected

Gain: gProjected Light: l

Captured Light: l

Intensity: in

Paper

Light: lReceived

Light: lAmbient

Albedo: al

Normal

Camera i

Gain: g

TeleGraffiti Site i

Mirror i

Projector iIntensity: in r

ipi p

i

ci

ci

a

mi

i

i

ipi

c

ci

pi

Figure 2: The image formation model. The projector at site projects a pixel intensity with gain resulting in projectedlight radiance . The irradiance of the projected light reachingthe paper after being reected by the mirror is and the anglethat the projected light makes with the paper normal is . The

total irradiance of the ambient light that reaches the paper is and the albedo of the paper is . The radiance of the reectedlight is , the angle the reected light makes with the normal is

" !

, the irradiance of the light reaching the camera is ! , the gain of the camera is ! , and the pixel intensity in the camera is ! .

camera

is the sum of two terms. The rst terme

`

%

R S

%

P

is the image of the paper in the ambient light only; it is theimage that would have been captured if the projector wasnot there. The second term

e

`

%

RI S

%5 eg %x

is the imagecreated by the light originating from the projector.

2.5 Photometric Calibration

The response functions of cameras and projectors are fre-quently not linear, as assumed in Equations (1) and (5).Empirically we found the response function of the SonyDFW-VL500 cameras that we used to be linear [Takao et al. , 2002 ] (with gamma correction set to OFF2.) On theother hand, we found the Panasonic PT-L701U projectorsthat we used to be highly non-linear. We therefore estimatedtheir response function and corrected for the non-linearity insoftware. See [Takao et al. , 2002 ] for the details.

3 The Final Viewed ImagesIf we set up a two site Tele-Grafti system as in Figure 1,

the projected image

V

is set to equal the image captured atthe other sitev

`

B" #

. If either $!& %

or $!

, the other siteis ') (0 $ . If we now suppose time proceeds in a sequence of steps and denote the image captured by the camera in the ithTele-Grafti site at time 1 by

`

2

1

4

, Equation (6) becomes:

`

2

1

4 !f

`

2 3 2

4W

e

`

%

R S

%x eg %x v

`

B" #

2

14 (

%x 4

(7)

where

`

2 5 2

4 ! e

`

%

RI S

%P

is the image of the paper inthe ambient light only. Denoting 6

! e`

%

RI S

% eg

and

expanding the recursive term

`

B" #

2

14 (

% 4

yields:

2

1

4 !

2 5 2

4

W

6

%

`

B7 #

2 5 2

4

W

6

%

6

B" #

%

2

17 (

4 Y

(8)

From Equation (6) half of the initial conditions of this re-currence relation are:

2

4 !

2 5 2

4

W

6

%'

2

4

(9)

where v V

2

4

is the image projected by the projector in theith site at system startup. The other half of the initial condi-tions are:

2

% 4 !

2 5 2

4

W

6

%9

`

B" #

2

4 !

`

2 5 2

4W

6

%'

`

B7 #

2 3 2

4W

6

%

6

B7 #

B" #

2

4 Y

(10)

The solution of the recurrence relation in Equation (8) is apair of fairly complex expressions, one for 1 even and onefor 1 odd. See [Takao et al. , 2002 ] for the full details. Ineither case, in the limit 19 8A @ the expression simplies to:

2

@

4 !

v

2 3 2

4W

6

%'

`

B" #

2 5 2

4

%

(0 6

%

6

B7 #

(11)

assuming that 6

%

6

B" #

C B

%

. If 6

%

6

B7 #

9 D

%

the value of

`

2

1

4

keeps increasing until it saturates the camera and theprojector and

V

2

1

4 !" v

2

1

4 !

.Note that the gain 6

! e`

%

RI S

% e

is not a constant, butdepends on the paper albedo. The gain is therefore differentfor points on the paper that have been written on comparedto points that have not been written on. The terms

e

`

andeg

also vary across the surface of the paper because theangles

D

`

,Dr

, andF

vary, albeit very slowly. Because thevariation is so slow, for simplicity we assume that

e`

andeg

are constant in the remainder of this paper.

3.1 Empirical ValidationIn our empirical validation we use a blank piece of paper.This is sufcient to validate Equation (11). The albedos RI S

and gains 6

are therefore treated as constant in this section.The gain 6

depends on the camera and projector parame-ters through

e`

ande

. It is generally easier to change thecamera parameters, especially with the Sony DFW-VL500ssince their parameters can be changed over the Firewire in-terface. Note that until Section 6 we assume that the cameraAutomatic Gain Control is switched off.

The gain 6

is set in the following way. We place a blank piece of paper under the camera and grab an image with the

projector switched off. We then set the projector to projectintensity

G

!E % G F

and grab a second image. We subtractthe rst image from the second and estimate the averageintensity in a small region in the center of the paper. Wethen perform a binary search over the camera parametersuntil this average value equals

% G FI H

6

. (Either the shutteror aperture can be varied. We use the aperture.) Naturally,for each trial with new camera parameters we grab a newimage with the projector switched off.

3


4/8

0 20 40 60 80 100 1200

20

40

60

80

100

120

in(A)

i n ( i n f )

expected valueexperiment results

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

1

2

3

4

5

6

7

8

9

10

11

Gain

i n ( i n

f ) / i n ( A )

expected ratioexperiment results

(a) Varying the Intensity !

!

(b) Varying the Gain

Figure 3: Empirical validation of Equation (11). (a) We plot !

for various different values of !

!

for xed

. (b) We plot the value of ! " !

!

for various settings of

and !

!

. In both graphs we compare theempirically measured results with the values predicted by Equation (11). We nd close agreement in both cases validating Equation (11).

Our results are presented in Figure 3. We set the gains6$ #

!

6

6 and intensitiesv

`

#

2 3 2

4 !

`

6

2 5 2

4

to be vari-

ous values and compare the nal measured image intensity v 2 @

4

with that predicted by Equation (11). The ambientlight image

`

2 3 2

4

is varied independently of the gain inour experiments by varying the amount of ambient light inthe room using an array of spot-lights. In Figure 3(a) wepresent results varying the ambient light images, keepingthe gains xed. In Figure 3(b) we present results varyingthe gains, and implicitly the ambient light images also. InFigure 3(b) we therefore plot the ratio of the nal viewedimage to the ambient light image, a quantity that should beconstant if Equation (11) is correct. As can be seen fromthegraphs in Figure 3, the predicted values closely match thosemeasured empirically, thereby validating Equation (11).

4 Choosing the Optimal GainEquation (11) describes the nal state that a two site Tele-Grafti system will settle into. It also describes the nalimage captured by the cameras and viewed by the users of the system. How can we set the gains 6

and 6 B" #

to max-imize the image quality of

2

@

4

?Equation (11) implies that should we want

2 5 2

4

and v

`

B" #

2 5 2

4

to contribute equal weight to

`

2

@

4

we shoulduse a gain 6

& %

%

; if 6& '

%

the user will not see the image v

`

B" #

2 5 2

4

from the other site. On the other hand, in the limit6

6

B" #

#8

%

the nal viewed image v

2

@

4

8 @ and

the cameras saturate. Too large a gain also leads to visualechoing. When one of the users waves their hand acrossthe paper they see multiple echoes of it, transmitted back from the image projected at the other site. Another sideeffect of visual echoing is that, because of inevitable errorsin geometric calibration and paper tracking, sketches run.See Figure 4(a-c) for an example of such visual echoing.

In practice we need to nd a compromise between thesetwo extremes. One way to choose the gains is to choose

them to make v

2

@

4

as close as possible to

2 5 2

4

W

`

B7 #

2 3 2

4

. It turns out that it is best to use the relative error

between these two quantities. If we add in a similar expres-sion for

v

`

B" #

2

@

4

, the other site, we aim to minimize:($ )

2 5 2

4

W

6

%9

`

B" #

2 3 2

4

%

( 6

%

6

B" #

(

2 5 2

4

(

`

B" #

2 5 2

41 0

6

W

)

`

B" #

2 5 2

4

W

6

B7 #

%'

2 5 2

4

%

( 6

%

6

B" #

(

2 5 2

4

(

v

`

B" #

2 5 2

41 0

63 2 5 4

6

`

2 5 2

4W

`

B" #

2 5 2

48 7

6

Y

In this expression we have ignored the fact that the gains6

and 6 B7 #

are variables and depend upon the albedo of the paper. Assume that 9

2

RI S

4

denotes the probability thatthe albedo of the paper is R S

. We can then generalize theexpression in the equation above to:

@

#B AD C

E

()

`

2 5 2

4W

6

%'

`

B" #

2 5 2

4

%

( 6

%

6

B7 #

(

2 5 2

4

(

v

`

B" #

2 5 2

41 0

6

W

)

`

B" #

2 5 2

4W

6

B" #

%9

2 5 2

4

%

(0 6

%

6

B7 #

(

2 5 2

4

(

`

B" #

2 5 2

4 0

62

9

2

R S

4

9

2

RI S

B" #

4

6

`

2 5 2

4W

`

B" #

2 5 2

47

6G F

RI S

F

R S

B" #

Y

The ranges of the integrals are G %

4I H

because the maxi-

mum value that albedos can take is%

4P H

[Horn, 1996 ]. Theequation above contains two types of terms, gains 6

andambient light images

2 3 2

4

. Because 6

! e`

%

R S

%I eg

and v

2 3 2

4 ! e

`

%

RI S

%P

these quantities are related by:

`

2 5 2

4 !

6

%

P

e

Y

(12)

In order to proceed we now make the following three sim-plifying assumptions: (1) we assume that

P

is constant

4


5/8

(a (b (c (d (e

Figure 4: (a-c) Visual echoing caused by using too large a gain 5 I

, combined with the inevitable errors in tracking the paper andcalibrating

!

: (a) !

, (b) !

, (c) !

. (d) The nal viewed image !

with optimal gain

and under normalof ce illumination is very dark. (e) The nal viewed image !

with optimal gain

and under strong ambient illumination.

across the paper, (2) we assume thate

is constant acrossthe paper, and (3) we assume that

s

s

is the same at thetwo sites. The rst of these three assumptions just meansthat the ambient light is constant across the paper which isvery reasonable. The second assumption just means that thevarious angles between the projected and reected light, thepaper, the camera the projector are all constant across the

paper. This is also a reasonable approximation. The thirdassumption can easily be generalized, however. The detailsare omitted but it is straightforward to estimate

s

s

for eachof the two sites separately from Equation (12). The con-stant relative values of these expressions can then includedand different optimal gains derived for the two sites.

Even after making these assumptions, the optimal valueclearly still depends on the weighting function 9

2

RI S

4

. Thesketches we work with are mostly simple black on whiteline drawings. The albedo is therefore nearly always either

RI S

, the paper albedo, or . Let us therefore assume that:

9

2

R S

4 !

%

2

RI S

4W

%

2

RI S

( R S

I 4

(13)

where 2

% 4

is the Dirac delta function; i.e. a unit impulse.It is possible to continue the analysis with other choices of the weighting function 9

2

R S

4

and derive the optimal gainin other scenarios. We just need to make a concrete choiceto proceed. Let us denote 6

! e

`

%

RI S

% e

, the gain of the paper. The optimality criterion then simplies to:

()

2

6

4

6

%

6

B7 #

W

2

6

%

6

B" #

W

6

(

%x 4 %

6

B" #

%

(0 6

& %

6

B7 #

0

6

W

)

6

%

2

6

B7 #

4

6

W

2

6

%

6

B" #

W

6

B" #

(

%x 4 %

6

%

(0 6

& %

6

B7 #

0

6

W

2

6

4

6

W

2

6

B" #

4

6

4

6

6

W

6

B7 #

#

7

6

Y

(14)

See [Takao et al. , 2002 ] for more details of the derivation.The minimum value of the expression in Equation (14) iseasily found numerically to be 6

!

6

B" #

! G Y

. Equa-tion (14) is plotted in Figure 5(a) for 6

!

6

B" #

. The opti-mal gain of the paper is therefore

Y

. The gain of the paper6

can be set using the method described in Section 3.1.

5 Illumination AugmentationMost LCD projectors are designed to be very bright becauseof the quadratic falloff in brightness with the distance

3

between the projector and the projection surface. See Equa-tion (2). In Tele-Grafti the value of

3

is smaller than thevalue that most projectors are designed for. As a result

e

is often relatively large. To obtain 6

! Y

, we have touse a small aperture (or a fast shutter) to make

e

`

smallenough. The result is that

2 3 2

4 ! e`

%

R S

%

) P

, the imageof the paper in ambient light, is very dark. As a result, the -nal image viewed

2

@

4

is also very dark. See Figure 4(d)for an example. Ideally we would like

2

@

4

to have itsbrightest pixels to have intensity close to

%

tomaximize the dynamic range of the camera. Assume thatthe paper is the brightest object. Then, to obtain:

2

@

4 !

2 3 2

4W

6

& %'

`

B" #

2 5 2

4

%

(0 6

%

6

B7 #

! !

%

(15)when

`

2 3 2

4 !"

`

B7 #

2 3 2

4

and 6

h !

6

B" #

!X G Y

, we need:

`

2 3 2

4 !

%

(

2

6

4

6

%W

6

H v

!

2

%

(I 6

4 H v

%

%x # " Y V Y

(16)If the brightest pixel in

v

2 3 2

4

'

% $ "

, the brightest pixel in

2

@

4

will be'

and a large fraction of the dynamicrange of the camera (and projector) will be wasted.

The only way to increase

2 3 2

4g ! e

`

%

R S

%P

is toincrease the ambient light

P

because the albedo RI S

is xedand

e`

must to be set to ensure that the gain 6

! Y

. Itis often impossible or inconvenient to increase the ambientillumination in the room. Fortunately, as we now show how,it is possible to use the projector to provide the additionalillumination. Suppose that instead of setting the projectedimage

V

to be the captured image

`

B" #

we set it to be:

!& %

W

!

(

%

!

H

`

B" #

'

(17)

i.e. we effectively add constant additional illumination of intensity

%

. In making this change, we effectively changethe gain of the projector to be smaller by the factor

2

!

(

5


6/8

0

0.5

1

1.5

2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

V a l u e o f

E q u a t

i o n

( 1 4 )

Gain G1 = G2

0

20

40

60

80

100

120

140

0 20 40 60 80 100 120 140

V a l u e o

f b f r o m

E q u a

t i o n

( 1 9 )

In(A)

(a) Derivation of the Optimal Gain (b) The Base Intensity to Project

Figure 5: (a) Choosing the Gain. We choose the optimal gains and to make !

as close as possible to !

!

;i.e. to minimize Equation (14). The minimum value is achieved when

. Here we plot Equation (14) for

.

(b) Choosing the additional ambient light to be projected. Given the brightest pixel in !

and assuming

, the graph above(b) of Equation (19) gives the value of to be projected using Equation (17) to obtain the brightest pixel in !

to be P

% 4

4

v !

. We therefore have to increase the gain of thecamera by the factor

4

2

(

% 4

to keep the gainof the overall system constant. The value of

2 5 2

4

willtherefore increase by this factor because of the change in thegain of the camera and by the amount

!

%

6

% %

4

2

(

% 4

because of the additional ambient light. To choose%

tooptimize the dynamic range of the camera we need to solve:

v !

(

%

H v

`

2 3 2

4W

6

H % !

2

%

( 6

4 H !

(18)

The solution of this equation is:% !

2

%

( 6

4 H v

(

`

2 5 2

4 Y

(19)

Equation (19), plotted in Figure 5(b) for 6

! Y

and v !

!

, can be used to estimate a suitable base in-tensity

%

to project using Equation (17) to ensure that theoverall system does not saturate, and yet the maximum pos-sible dynamic range of the camera and projector are used.Example images of the system running using the value of

%

computed using Equation (19) are included in Figure 6.Note that these images have the image quality that we havebeen striving for; they are bright, have excellent contrast,and there is minimal visual echoing. Also note that theintensity correction in Equation (17) can easily be imple-mented as a table lookup. See [Takao et al. , 2002 ].

6 Automatic Gain Control

We have analyzed the Tele-Grafti feedback loop assum-ing the cameras are set up with xed apertures and shut-ter speeds to give a constant gain, however Tele-Grafti isnormally operated with the automatic gain control (AGC)switched on. How does using AGC affect our analysis?

The analysis itself is not affected. The only thing that isaffected is our ability to set the gain independently. Instead,AGC operates by adjusting the aperture (or shutter speed)to enforce the requirement that the average intensity in a

certain area of the image (normally the center) is a xedvalue, the exposure. When AGC is switched on, the cameragain is adjusted until Equation (11) is satised, with

2

@

4

equal to the exposure. If the ambient light level is low, thisresults in a large gain 6

& %

%

and the system has substantialvisual echoing. If the ambient light level is high, the gain

6

& %

and what is written at the other site cannot be seen.When using automatic gain control, the only parameter

we can use to indirectly control the gain is the amount of ad-ditional ambient light provided by the projector. In partic-ular, the offset is the one parameter we can use to changethe gain and thereby the image quality. As we increasethe gain of the cameras will be reduced and there will beless visual echoing. On the other hand, as we increase , theimage from the other site will be less visible.

Using a similar procedure to that in Section 3.1 we canbuild a lookup table for the gain as a function of . See[Takao et al. , 2002 ] for the details and an example of sucha table. We can then use this table to set the gain to be anydesired value. The results are shown in Figure 7.

In practice we found that the best way to run Tele-Graftiis to use AGC. If there is insufcient light there will be vi-sual echoing. The value of should then be adjusted byhand until the visual echoing stops and the right trade-off between dynamic range and lack of echoing is achieved.

7 Color ImageryWorking with color images simply means that there is a sep-

arate gain for each channel. Assuming the camera and projector are reasonably color balanced these gains are fairlysimilar and the analysis above can be performed with theaverage of the three gains. If there is substantial differencein the gains, obtaining good image quality can be difcult.If we set up the system so that the average gain of the paperis

Y

, the gain of one of the channels may be much larger;in extreme cases the gain may be

%

%I Y

. In such cases oneof the color channels may saturate. An example of such a

6


7/8

(a (b (d(cFigure 6: A 2 site Tele-Graf ti system operating with optimal gain

and additional ambient light provided by the projector:(a) = !

, (b) = !

, (c) = !

, (d) = !

. Note the superior image quality of the images compared to Figure 4(c-d).

Paper ImagesunderAmbient Light

Final Viewed Imageswith Gain=0.05

Final Viewed Imageswith Gain=0.95

with Gain=0.50Final Viewed Images

TeleGraffiti Site 1 TeleGraffiti Site 2

(b=23)

(b=156) (b=148)

(b=35)

(b=3) (b=1)

Figure 7: Final viewed images with automatic gain control turnedon. The nal viewed images ( !

) with gain=0.05, 0.50, 0.95are shown in the second, third and fourth rows respectively. Toobtain these gains, we computed the base intensity to project using the lookup table described in [Takao et al. , 2002 ].

situation is included in [Takao et al. , 2002 ].

8 Image SeparationWhen we switch on Tele-Grafti the system quickly con-verges to the steady state:

v

`

2

@

4 !

`

2 5 2

4

W

6

%9

`

B7 #

2 3 2

4

%

( 6

%

6

B" #

Y

(20)

Is it possible to reverse this process? Can we recover whatis drawn on the paper from what is imaged? Doing this

corresponds to inverting Equation (20). As in Section 4 thedifculty in doing this is the fact that 6

is not constant, butis related to

2 5 2

4

through:

`

2 5 2

4 !

6

%

P

e

Y

(21)

If the ambient lightP

can vary completely independently

across the paper, Equation (20) is not invertible. In Sec-tion 4 we assumed that

P

ande

are constant across thepaper. To invert Equation (20) we now make the same as-sumption. We also assume that the value of

s

s

has beenestimated at system startup. This can easily be performed.We set the gain of the paper 6

to a xed value. We canthen measure the average ambient light image of the paperand use the results to compute

s

s

from Equation (21).

Once

s

s

is known, it is straightforward to invert Equa-tion (20), a pair of simultaneous quadratic equations. Forlack of space, the reader is referred to [Takao et al. , 2002 ]for the details. The result, however are closed form solu-tions for the ambient light images

`

2 3 2

4

and v

`

B7 #

2 5 2

4

interms of the nal viewed images

2

@

4

and

`

B" #

2

@

4

.We have implemented this solution to the image separa-

tion problem, the results of which are shown in Figures 8.We found that where the intensity is roughly constant theoriginal drawn image is restored very well. At points withhigh gradient, however, there are signicant artifacts causedby the defocussing effect of several components in thesystem: the camera, the projector, and the image warping.Defocus effects are not incorporated in our image formationmodel. Potentially a more sophisticated image formationmodel could be derived. In the meantime we process theresults in Figures 8 with a simple algorithm to remove mostof the artifacts. We: (1) detect edges in the nal viewedimages, (2) compare the magnitudes of the edges at the twosites, and (3) smooth the edges at the weaker site basedon the assumption that the edges originate from the side atwhich they are stronger. The results of applying this algo-rithm to the results in Figure 8. Overall the separated im-ages match the actual drawn images reasonable well, (espe-cially considering that the system was effectively set up inSection 4 to make this task as hard as possible.)

7


8/8

(c1) (c2)

With OverlapsPattern 1

Final Viewed Images Improved Separat ion Results Actual Drawn Images

(g1) (g2) (h1) (h2)

(d1) (d2) (e1) (e2)

(i1) (i2)

(f1) (f2)

(a1) (a2) (b1) (b2)

Pattern 2No Overlaps

Pattern 3Thin Lines

Figure 8: The results of image separation. The rst column of each row contains the two nal viewed images of the two sites from whichwe compute the separated images shown in the second column. The third column contains the actual drawn images at the two sites.

9 DiscussionWe have performed a steady-state analysis of Tele-Grafti.We have shown how this analysis can be used to derive theoptimal settings for the system and to separate the steady-state images to derive estimates of what is actually drawn onthe paper. This steady-state analysis requires a number of assumptions, most notably that what appears on the paperdoes not change. One possible direction for future work is to perform a perturbation analysis to determine howstable the system is when something new is drawn on thepaper, or some other change occurs in the system settings.

AcknowledgmentsThe research described in this paper was conducted whilethe authors were all at Carnegie Mellon Robotics Institute.We would like to thank Iain Matthews and Bart Nabbe fortheir help constructing Tele-Grafti [Takao et al. , 2001 ].

References[Hall et al. , 1999 ] D. Hall, C. Le Gal, J. Martin,

T.Kapuscinski, and J. Crowley. MagicBoard: Acontribution to an intelligent ofce environment. InProc. of the 7th Intl. Symp. on Intel. Rob. Sys. , 1999.

[Horn, 1996 ] B. Horn. Robot Vision . McGraw Hill, 1996.[Leibe et al. , 2000 ] B. Leibe, T. Starner, W. Ribarsky,

Z. Wartell, D. Krum, B. Singletary, and L. Hodges. Theperceptive workbench: Towards spontaneous and natu-ral interaction in semi-immersive virtual environments.In Proc. of the IEEE Virtual Reality Conference , 2000.

[Raskar and Beardsley, 2001 ] R. Raskar and P. Beardsley.A self-correcting projector. In Proc. of CVPR , 2001.

[Raskar et al. , 1998 ] R. Raskar, G. Welch, M. Cutts,A. Lake, L. Stesin, and H. Fuchs. The ofce of the fu-ture: A unied approach to image-based modeling andspatially immersive displays. In SIGGRAPH , 1998.

[Sato et al. , 2000 ] Y. Sato, Y. Kobayashi, and H. Koike.Fast tracking of hands and ngertips in infrared imagesfor augmented desk interface. In Proc. of the 4th Intl.Conf. on Automatic Face and Gesture Recognition , 2000.

[Sukthankar et al. , 2001a ] R. Sukthankar, T.-J. Cham, andG. Sukthankar. Dynamic shadow elimination for multi-projector displays. In Proc. of CVPR , 2001.

[Sukthankar et al. , 2001b ] R. Sukthankar, R. Stockton, andM. Mullin. Smarter presentations: Exploiting homogra-phy in camera-projector systems. In ICCV , 2001.

[Takao et al. , 2001 ] N. Takao, J. Shi, S. Baker, I. Matthews,and B. Nabbe. Tele-Grafti: A paper-based remotesketching system. In ICCV , Demo Session, 2001.

[Takao et al. , 2002 ] N. Takao, J. Shi, and S. Baker. Tele-Gratti. Technical Report CMU-RI-TR-02-10, Robotics

Institute, Carnegie Mellon University, March 2002.[Takao et al. , 2003 ] N. Takao, J. Shi, and S. Baker. Tele-

grafti: A camera-projector based remote sketching sys-tem with hand-based user interface and automatic ses-sion summarization. International Journal of Computer Vision , 53(2):115 133, July 2003.

[Wellner, 1993 ] P. Wellner. Interacting with paper on theDigitalDesk. Comm. of the ACM , 36(7):8696, 1993.

8

takao baker shi

Documents