Copyright © 2010 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail permissions@acm.org. ETRA 2010, Austin, TX, March 22 – 24, 2010. © 2010 ACM 978-1-60558-994-7/10/0003 $10.00

Development of Eye-Tracking Pen Display Based on Stereo Bright Pupil Technique

Michiya Yamamoto* School of Science and Technology

Kwansei Gakuin University

Takashi Nagamatsu† Graduate School of Maritime Sciences

Kobe University

Tomio Watanabe‡ Department of Systems Engineering

Okayama Prefectural University

Abstract

The intuitive user interfaces of PCs and PDAs, such as pen dis-play and touch panel, have become widely used in recent times. In this study, we have developed an eye-tracking pen display based on the stereo bright pupil technique. First, the bright pupil camera was developed by examining the arrangement of cam-eras and LEDs for pen display. Next, the gaze estimation method was proposed for the stereo bright pupil camera, which enables one point calibration. Then, the prototype of the eye-tracking pen display was developed. The accuracy of the system was approximately 0.7° on average, which is sufficient for hu-man interaction support. We also developed an eye-tracking tabletop as an application of the proposed stereo bright pupil technique.

CR Categories: H.5.2 [Information Interfaces and Presenta-tion]:User Interfaces—Input devices and strategies; I.4.9 [Image Processing and Computer Vision]: Applications

Keywords: embodied interaction, eye-tracking, pen display, bright pupil technique

1 Introduction

Today, the intuitive user interfaces of PCs and PDAs, such as pen display and touch panel, have bcome widely used. These devices are expected to open up a new embodied interaction and communication as well as interaction between humans and com-puters.

By focusing on the importance of embodied interaction, the authors have developed a CG-embodied communication support system [Yamamoto el al. 2005]. Especially, the importance of timing control in generating embodied motions and actions is made clear for supporting natural, familiar, and polite interaction via CG and robot agent [Yamamoto et al. 2008]. However, for making further uses of embodiment, it is required to analyze the relationships between body motion and attention.

If we could integrate pen display and eye-tracker, it becomes possible to analyze various embodied interactions. For example,

we could analyze how a presenter indicates or emphasizes a slide in presentation by using intuitive pen display. In addition, such an eye-tracking pen display could become a gadget for realizing a new mode of interaction between humans and com-puters.

In this study, we have developed an eye-tracking pen display based on the stereo bright pupil technique, and have made an eye-tracking tabletop as its application.

2 Technical Requirements

There are several eye-trackers which we can be listed as de facto standards such as Tobii X120. However, they are not suitable for use with pen display. The biggest problem of such eye-trackers is that they have cameras and IR LEDs under their displays (Figure 1). When a right handed person use a pen on the display, the right arm may hide the camera or LED.

The tracking distance and gaze angle may also cause a problem when a user draws on a pen display. Because, the tracking dis-tance of existing eye-trackers is approximately 50 cm or more, and the gaze angle is approximately 30° in many cases. If we put an eye-tracker at the left bottom of the display and use a pen on the display, the tracking distance becomes too close and gaze angle becomes too wide.

In addition, easy calibration is required for eye-tracking pen display, so that intuitive interface can be realized. Thus, we can summarize the technical requirements as follows:

• Free arrangement of cameras and LEDs to prevent ob-struction by the right hand

• Robust gaze estimation with short distance & wide gaze angle

• Easy calibration

*michiya.yamamoto@kwansei.ac.jp †nagamatu@kobe-u.ac.jp ‡watanabe@cse.oka-pu.ac.jp

Figure 1: Typical layout of cameras and LEDs.

IR LEDs

Cameras

IR LEDs

Cameras

165

3 Reviews of Previous Studies to Decide Ar-rangement of Cameras and IR LEDs

As the first step of this study, we analyzed the body motions involved in using pen display of a right handed user. For this, we used motion capture system (Vicon Motion Systems, Vicon 512) and measured a subject’s body motion; i.e., movement of head, right shoulder, and arm. As shown in Figure 2, the posture of the subject and the angle of pen display were assumed to be limited to 3 cases to avoid hiding the cameras and IR LEDs.

We developed a software for analyzing the arrangement. Figure 3 shows the screen shot of the software which draws the results of measurement of 10 subjects. It can be seen that there is an unavailable volume for arranging cameras and IR LEDs at the left bottom.

Next, we reviewed previous studies and developed a prototype of the system by considering its technical requirements. The 3D gaze-tracking approach was selected for accuracy [Shih et al. 2004; Guestrin et al. 2007; Nagamatsu et al. 2008a]. This ap-proach involves the use of two cameras and three or four LEDs. Figure 4 (a) shows the arrangement of the system proposed by Nagamatsu et al. In this study, we first developed a prototype of the system by positioning the cameras and LEDs: two cameras are placed to the left of the pen display, and one LED each is placed on the top, left, and bottom frames of the pen display (Figure 4 (b)). However, even with such an arrangement, stable eye-tracking cannot be realized due to the obstructions by the

right hand and the eyelid. Therefore, we reviewed the arrange-ments proposed in previous studies again. Some researchers have proposed camera-LED integrated systems. For example, Ohno developed a system that involved the use of one camera and two LEDs [Ohno 2006]. Chen et al. developed a system that involved the use of two cameras and two LEDs mounted near the camera centers; in this arrangement, the camera and the LED were integrated into one component [Chen 2008]. We can arrange such a system to the left of the pen display; however, such a system would be inadequate if the pen display is to be used at various angles. The two cameras should be separated for the eye tracking pen display system.

4 Stereo Bright Pupil Technique for Pen Dis-play

4.1 Bright Pupil Camera

On the basis of the reviews of previous papers, we decided to use the stereo bright pupil technique. We integrated an IR LED at the center of the camera (POINT GRAY, FFMV-03MTM, 752x480 pixels) lens, as shown in Figure 5; this modified cam-era is called the bright pupil camera. A 35-mm lens and an IR filter are attached. We positioned two bright pupil cameras sepa-rately to the left of the pen display (Figure 4 (c)). When these cameras are used, the light from the LED reflects on the retina and a bright pupil can be observed in the camera image.

4.2 Eye Model

Figure 6 shows the eye model in this study, which is typical in model-based approaches. An eye consists of two balls. It has two axes: one is the optical axis of the eye that is the geometric center line of the eye, and the other is the visual axis that is the line of sight connecting the fovea. These axes intersect at the center of the corneal curvature. The average of horizontal and vertical angles between the optical and visual axes are 5.5° and 1.0°, respectively [OSAKA, 1993].

4.3 Image Processing

By using two bright pupil cameras, the light from one of the LED reflects at the retina and the camera image of the pupil

Figure 5: Bright pupil camera.

Figure 4: Arrangement of cameras and LEDs.

Figure 3: Arrangement volume of cameras and LEDs.

Pen display

Unavailable volumefor arranging cameras and IR LEDs Right arm

Markers

Sitting, Pen display at the angle of 60°

Standing, Pen display at the angle of 60 °

Standing, Pen display at the angle of 15 °

Figure 2: Measurement of body motion while using a pen display.

Figure 6: Eye model.

(a) (b) (c)

LED

Camera

CorneaPupil

Visual AxisOptical Axis

Center of Corneal CurvatureRotation

Center

A

E

B Pupil Center

Fovea

166

becomes bright. In addition, there are two reflections of light sources from the outer surface of the cornea called the first Pur-kinje image (Figure 7, left). First, we carried out edge detection in order to detect the position of the pupil. Next, we fitted an ellipse to the edge, and calculated the pupil center. To detect the position of the Purkinje image, we trimmed the neighborhood of the pupil center, and binarlized the image. We considered the two bright points as a Purkinje image (Figure 7, right). This image processing was performed using Open CV 1.0.

4.4 Estimation of the Optical Axis of the Eye

We estimated the optical axis on the basis of the results of image processing. We initially calculated the relationship between each pixel on the image plane and the corresponding 3D position by calibrating the camera. We assumed that the light source and the camera center are at the same position. Then, we obtained a plane that contains A and B by using the expression ( ) ( ) ( ) 0− × − − =C Β' C P' X Ci , where X is a point on the plane (Figure 8). One bright pupil camera was used to determine one plane that contains the optical axis. Therefore, the optical axis can be obtained as the intersection of two planes obtained using the two cameras. While Chen estimated the optical axis by determining Virtual −B A in Figure 8, we determined the exact optical axis [Nagamatsu et al. 2010]. After that, the user gazes at a point on the pen display for calibration. The difference be-tween optical axis and visual axis is revised by doing this cali-bration [Nagamatsu et al. 2008b]. The cross point of optical axis and pen display is estimated as the gaze point.

5 Evaluation

5.1 Method

We integrated the bright pupil camera and pen display (Wacom, DTI-520, 15 inch (380 mm), 1024 × 768 pixels) and developed a prototype of the eye-tracking pen display. Figure 9 shows a pro-totype of the eye-tracking pen display. Here, the gaze estimation

can be realized while a user is drawing a line while looking at the tip of the pen. The white cross is the estimated point. We can confirm that the center of the white cross and the tip of the pen is almost the same. We developed this system on an HP xw4600 Workstation with MS Windows XP. The frame rate was ap-proximately 10 fps.

We then evaluated the prototype. Figure 10 shows the experi-mental setup. The left part is the eye-tracking pen display and a subject. The minimum distance between the subject and pen display was 30 cm. The angle of pen display was 60°. The right LCD is displaying a captured and processed image.

In the experiment, we asked the user to gaze at the marker at the left side of the pen display for calibration. We next displayed a white cross on the pen display, and asked him to gaze at the center of the white cross for 10 frames. The cross was displayed on each of the 128 pixels. Because of the narrow range of view angle and focus of the cameras, the area where a user can move is limited. 3 students participated in the experiment.

5.2 Results

Figure 11 shows the results. The accuracy was average 17.4 pixels (5.2 mm) on the screen, which means about 0.71°. It was equivalent to Tobii, etc. In other words, the pen display can recognize 22 horizontal lines.

Figure 10: Experimental setup.

0

128

256

384

512

640

768

0 128 256 384 512 640 768 896 1024

Subject1Subject2Subject3

Figure 11: Result of evaluation experiment.

Figure 7: Example of image processing.

Figure 8: Estimation of the Optical Axes.

Figure 9: Prototype of the eye-tracking pen display.

300 mm

60 °

Image Plane

Light / Camera

Optical Axis Pupil Center

Center of Corneal Curvature

B'

Β''

C'P

B

Purkinje Image

ΑP

CornealSurface Virtual B

Purkinje Imageon Image Plage

Edge detection

Binarlize

167

In the case of some subjects, the Purkinje image was reflected on the edge of the cornea, and the gaze point could not be cor-rectly estimated, as shown in Figure 12. However, this problem can be solved by using one or more bright pupil cameras in a layout-free arrangement.

6 Application

The proposed method can be applied to develop various types of eye-tracking systems.

For example, we have developed a prototype of an eye-tracking tabletop interface as shown in Figure 13. We integrated two bright pupil cameras and a projector. The image is projected on the tabletop. This interface is used to realize both eye-gaze inter-action and physical interaction. For example, the red square indicates the gaze point on the tabletop. When a user is looking at a physical pointer on the tabletop, the red square moves ap-propriately, followed by physical movement of the pointer. We can enlarge the tabletop interface and extend the interaction area to include off-surface areas.

In this manner, bright pupil cameras enable flexible arrangement of cameras, which can lead to the developments of various hu-man-computer interfaces such as pen displays and tabletops as well as interaction analysis of laptops.

7 Conclusion

In this study, we have developed an eye-tracking pen display based on the stereo bright pupil technique. First, the bright pupil camera was developed by reviewing and examining the ar-rangement of cameras and LEDs for pen display. Next, the gaze estimation method was proposed for the bright pupil camera, which enables one-point calibration and wide-angle accuracy. Then, the prototype of the eye-tracking pen display was devel-oped. The accuracy of the system was approximately 0.7° on average, which is sufficient for pen display. We also developed

a prototype of an eye-tracking tabletop as an application of the proposed stereo bright pupil technique, and confirmed effective-ness of the system.

Acknowledgement

This work under our project “Embodied Communication Inter-face for Mind Connection” has been supported by “New IT In-frastructure for the Information-explosion Era” of MEXT Grant-in-Aid for Scientific Research on Priority Areas. Also, our pro-ject "Generation and Control Technology of Human-entrained Embodied Media" has been supported by CREST (Core Re-search for Evolution Science and Technology) of JST (Japan Science and Technology Agency).

References

CHEN, J., TONG, Y., GRAY, W., AND JI. Q. 2008. A Robust 3D Eye Gaze Tracking System using Noise Reduction. In Proceed-ings of the 2008 symposium on Eye tracking research & appli-cations, 189–196.

GUESTRIN, E. D., AND EIZENMAN, M. 2007. Remote Point-of-Gaze Estimation with Free Head Movements Requiring a Sin-gle-Point Calibration. In Proceedings of the 29th Annual Inter-national Conference of the IEEE EMBS, 4556–4560.

NAGAMATSU, T., KAMAHARA, J., IKO, T., AND TANAKA, N. 2008. One-Point Calibration Gaze Tracking Based on Eyeball Kine-matics Using Stereo Cameras. In Proceedings of the 2008 sym-posium on Eye tracking research & applications, 95–98.

NAGAMATSU, T., KAMAHARA, J., AND TANAKA, N. 2008. 3D Gaze Tracking with Easy Calibration Using stereo Cameras for Ro-bot and Human Communication. In Proceedings of IEEE RO-MAN 2008, 59–64.

NAGAMATSU, T., IWAMOTO, Y., KAMAHARA, J., TANAKA, N., AND YAMAMOTO, Y. 2010. Gaze Estimation Method based on an Aspherical Model of the Cornea Surface of Revolution about the Optical Axis of the Eye. In Proceedings of Eye Tracking Re-search & Applications Symposium ETRA 2010. (to appear).

OHNO, T. 2006. One-point calibration gaze tracking method. In Proceedings of the 2006 symposium on Eye tracking research & applications, 34.

OSAKA, R. 1993. Experimental Psychology of Eye Movements (in Japanese). The University of Nagoya Press, Nagoya, Japan.

SHIH, S.-W., AND LIU, J. 2004. A Novel Approach to 3-D Gaze Tracking using Stereo Cameras. IEEE Transactions on Systems, Man, and Cybernetics Part B 34, 1, 234–245.

YAMAMOTO, M., AND WATANABE, T. 2005. Development of an Embodied Interaction System with InterActor by Speech and Hand Motion Input. In CD-ROM of the 2005 IEEE International Workshop on Robots and Human Interactive Communication, 323–328.

YAMAMOTO, M., AND WATANABE, T. 2008. Timing Control Ef-fects of Utterance to Communicative Actions on Embodied In-teraction with a Robot and CG Character. International Journal of Human-Computer Interaction 24, 1, 103–112.

Figure 13: Prototype of an eye-tracking tabletop.

Figure 12: Purkinje image on edge of cornea.

Tabletop

Britht Pupil Camera Projector

PhysicalInteraction

Eye-GazeInteraction

User

168