Multimedia Satellite Network System for Communication between

Immersive Indoor and Outdoor Environments

Noritaka OSAWA, Kimio KONDO, Kikuo ASAI National Institute of Multimedia Education, 2-12 Wakaba, Mihama, Chiba, 261-0014 JAPAN

{osawa,kkondo,asai}@nime.ac.jp

Abstract. We developed a multimedia satellite network system based on the Internet protocol (IP) and personal computers. The system enables collaborative distance education between outdoor and indoor environments. Integration of IP technologies and satellite communications enhances the flexibility of the network system. The satellite communication system enables communicating to and from outdoor environments where terrestrial infrastructures do not exist or are poor. Personal-computer-based systems allow use of currently available software and make outdoor systems portable and easy to install outdoors. We developed some software modules for collaborative distance education. Their functions include IP multicast communications and distributed shared focus presentation such as shared pointers, and they work on personal computers. We conducted an experiment for collaborative distance education between a farm and an immersive display environment by using the multimedia satellite communication system and a transportable earth station outdoors. The experiment shows that our system can be used for collaborative distance education between outdoor and indoor environments, and achieves an enhanced collaborative environment.

1. Introduction Video conferencing systems have been widely used in distance education and remote

meetings, but we do not think that they easily and efficiently support collaborative distance education between indoor and outdoor environments. If they did, students in a lecture room could better understand the outdoor environment and be motivated. This type of distance education would be useful in horticulture, environmental studies, natural sciences, and studies that require fieldwork.

We are therefore developing a multimedia satellite network system that supports distributed real-time collaboration between people in indoor environments and people in outdoor environments, for example, such as mountains and fields, where terrestrial communication infrastructures are poor or do not exist.

Network technologies based on the Internet protocol (IP) are used to enhance the flexibility of the satellite network system and make it easier to connect the satellite communication system with LANs or other terrestrial networks. We developed some software modules that work on personal computers. In addition, commercially available items such personal computers and software components are also employed to make the system portable, easy to install and easy to operate outdoors. Our developed software modules include IP multicast sender and receiver filters for audio, video and pointer information, filters for video extraction and composition, and a filter for nonlinear zooming functions. These filters complement functionalities that are implemented in the personal computers.

We conducted an experiment using the developed system, a panoramic camera and an immersive projection display system. Video signals of surroundings obtained with the

panoramic camera set up at a horticultural farm were transmitted through a communication satellite and were displayed in the immersive projection system with multiple screens.

The experiment showed that the system we developed would be useful for collaborative education involving communication between outdoor and indoor environments. The experiment also showed that this type of remote collaborative education could enhance the impression of being present in distance education environments while using a limited satellite communication bandwidth. In other words, the developed system can produce an enhanced collaborative environment.

2. Developed System We have been developing communication support tools based on DirectShow technologies

[5]. With DirectShow, basic software components, or filters can be used. Software codecs and video/audio capture/render functions that are already available can be used with no need to re-implement existing functions.

The functionalities of the DirectShow components currently available, however, are insufficient to support distributed collaboration that uses multiple displays. Therefore we developed some new filters.

2.1. Filters

This subsection briefly describes newly developed filters for multicast senders and receivers, distributed shared pointers, image extraction, image composition, nonlinear zooming, and image correction.

2.1.1. Multicast sender and receiver filters A multicast sender filter can send video, audio, and pointing information as IP/UDP

packets, and a multicast receiver filter can receive the IP/UDP packets of video, audio, and pointing information. Supported video formats include DV (720x480 for NTSC) for high-resolution and high-frame-rate video, and MPEG4-based encoded video (resolution less than 720x480) mainly for low-bit-rate signals. Supported audio formats include PCM, MP3, CCITT µ-Law, and G.723.1. Using a property function of the filters, a multicast address and a TTL (hop count) can be specified. The property of a multicast receiver filter is shown in Figure 1.

Figure 1: Filter property (multicast receiver)

Figure 2: Shared multiple pointers (red rectangle at

the left and blue rectangle at the right)

2.1.2. Distributed shared pointers Shared focus information given by pointing devices such as a mouse is distributed with

multicasting pointers (pointing symbols or cursors). Multiple sites can share information by using multiple pointers and a presenter can thus show a focus point while participants can at the same time show which points they find interesting and about which they want to know more. Figure 2 shows a screenshot that has two shared pointers.

2.1.3. Video extraction filter A video extraction filter extracts a region within a video image. Using some focus

information can dynamically change the extracted region. For example, a remote user can select the extracted region by using a mouse.

2.1.4. Image composition filter The image composition filter combines two video images: one large and the other small.

The shape of the small region can be chosen from a predefined set (rectangle, oval, and checker pattern) and can be moved according to the focus information. One example of the use of this filter is for combining at a receiver a large low-resolution and low-frame-rate video image (contextual video) and a small high-resolution and high-frame-rate video (focused video) (Figure 3). This combination of contextual video and focused video is useful when limited communication bandwidth makes it impossible to transmit the information needed for large high-resolution and high-frame-rate video images.

Figure 3: Combination of large low-resolution and low-frame-rate video image and small high-resolution and high-frame-rate video image. The resolution is high in the region to lower right of red rectangle pointer.

Figure 4: Nonlinear zooming

2.1.5. Nonlinear zooming filter The nonlinear zooming filter distorts a video in order to show a focus region. A simple

extraction filter can show a focus region but cannot show other surrounding regions, whereas the nonlinear zooming filter shows not only a focus region but also other regions as contexts although non-focus regions are distorted. The design of the systems assumes that the output format is smaller than the input format. The resolution of the focus region is the same as that of the input video, but the context region is shrunk or zoomed out. This filter provides a Focus+Context function [3] in information visualization, and we think that this function can

help participants to understand where important regions in the video image are. Figure 4 shows a screenshot of a nonlinearly zoomed image.

2.1.6. Image correction filter The video images from the cameras of a panoramic image system are distorted because

their objective lenses are wide-angle lenses. The image-correction filter generates an accurate image from a distorted image. This filter was used for real-time video image correction in our experiment.

2.2. GraphEdit

GraphEdit is a tool in Microsoft’s DirectShow SDK (Software Development Kit). It enables us to compose a filter graph by simple GUI operations. The flows of video, audio and data streams are represented by a directed graph in the tool, which is a useful tool for experiments because it does not require explicit programming. Figure 5 shows a screenshot of a GraphEdit filter graph. Programs such as those for multicast video senders and receivers are easily generated using our multicast sender/receiver graphs with the existing filters. Examples of such graphs are shown in Figure 6 and Figure 7.

Figure 5: A GraphEdit filter graph

Figure 6: Simple multicast sender graph: Video capture -> video encoder -> multicast sender

Figure 7: Simple multicast receiver graph: multicast receiver -> video decoder -> video renderer

3. Experimental Operation After several preliminary tests at the National Institute of Multimedia Education (NIME),

we conducted an experiment to demonstrate that the system we developed can be used for distance education that requires communication between outdoor and indoor environments. This experiment was held on October 2 in 2002. We chose a horticulture farm of Chiba University as an outdoor environment and used a 360-degree panoramic imaging (recording) system with four cameras in order to enhance the understanding of an outdoor environment to show the usefulness of the system’s multiple video streams, and to demonstrate its flexible resolution and frame-rate control functions.

The panoramic imaging system used the reverse-pyramid type of mirror as shown in Figure 8. Panoramic videos were encoded by using an MPEG4-based software encoder and then were transmitted. The panoramic view was shown in the indoor environment by using a Tele-Existence Environment for Learning eXploration (TEELeX) system [1] implemented at the NIME. Both environments were connected with the Space Collaboration System (SCS) [6], which is a bi-directional satellite communication system. In the following subsections, the system configuration in the experiment, SCS, and TEELeX are briefly described.

(a) panoramic camera (b) Use of panoramic camera in hothouse

Figure 8: Panoramic camera in hothouse

3.1. System Configuration

Figure 9 shows the system configuration for the experiment. In the outdoor environment, we used the SCS transportable earth station (Figure 10), personal computers running the developed software, and the panoramic image system. As mentioned, the TEELeX system was used as a surround display in the indoor environment.

Video signals from the four cameras of the panoramic image system were encoded by using MPEG4-based software compression filter and then were transmitted by the IP-multicast sender filter that we developed. Since the encoded videos were contextual videos, they were low-frame-rate and high-resolution videos.

The encoded video data was transmitted through SCS and received at the NIME. The received multicast data reached the TEELeX through a LAN at the NIME. The data was decoded by two personal computers. Two-video decoding was assigned for each PC, which had a dual-display function. The PC configuration could then be easily changed if the decoded video signals were distributed to the displays in the TEELeX system because the multicasting was independent of the number of PCs. The synchronization of the multiple

displays was based on time synchronization between PCs. A pair of conventional IP videoconference systems was also used to show that the

developed system and existing systems could coexist. In the experiment, IP videoconference systems with a handy camera were used for transmitting questions and answers between the lecturer and the student. This conferencing function could have been provided by the system that we developed though this was not done in our experiment. Additionally, a pair of IP-based telephones were prepared and used for communication by the experimental staff.

All signals in this experiment were transmitted in one SCS channel that had a bandwidth of about 1.5 Mbps. This effective and efficient transmission was made possible by our system’s flexible control of frame rates and video resolution.

Dell Precision 620

Windows 2000 server

Dell Precision 620

Windows 2000 server

LAN

SCS ControlRoom

SCS MobileStation

PanoramicCamera

Hub

i.Link

1 2 34 5 67 8 9* 8 #

IP Telephone

1 2 34 5 67 8 9

* 8 #

IP Telephone

ViaVideo or NetMeeting

on ThinkPad

IP Video ConferenceSystem

Polycom ViewStation

LAN

Communication Satellite

Satellite Antenna Satellite Antenna

VAIO No.1

VAIO No.2

VAIO No.3

VAIO No.6

Horticultural Farm

Satellite ModemSatelliteModem

TEELeX

NIME Router

Media Converter

Media Converter

Media Converter

Media Converter

TEELeX

Figure 9: Experimental system configuration

Figure 10: SCS transportable earth station

3.2. SCS The SCS [6] is a Japanese inter-university learning network system using a geostationary

satellite. The NIME has a hub station and operates the SCS. The SCS consists of a

geostationary satellite (JCSAT-3) and 150 VSAT stations including the hub station and a transportable station. One hundred twenty-three research/educational institutions in Japan belong to the SCS. They are 81 national universities, 11 research institutes, 13 private universities, 15 colleges and 3 other organizations. In the fiscal year 2000, more than 1200 sessions were held and the total number of hours was more than 3000. The SCS is used mainly for inter-university lectures, symposia and research forums.

The existing SCS enables stations at universities to exchange audiovisual signals. Any stations can transmit signals. Thus, SCS makes possible two-way interaction using satellite communications. The information rate of a channel of SCS is 1536 kbps and up to three channels can be used in a session. In the experiment, an IP-based protocol was used to transmit video and audio signals.

3.3. TEELeX

The Tele-Existence Environment for Learning eXploration (TEELeX) [1] is a surround display system with a multi-screen display and it uses the immersive projection technique, originally developed as CAVETM by the University of Illinois [4]. It has a large cubic screen for immersive video. Each face is 3 meters by 3 meters and the resolution of each face is 1000 pixels by 1000 pixels. Circular polarization is used to provide a stereoscopic view to users, who need only to wear lightweight stereo glasses [2]. Two video inputs are used to give a stereoscopic view on each side: one video input is for the right eye and the other for the left eye.

However, four face screens were used in the experiment, and the view was monoscopic rather than stereoscopic. Figure 11 shows a schematic diagram of the TEELeX system in the experiment and Figure 12 shows snapshots of the inside of the TEELeX system in the experimental session.

Figure 11: Schematic diagram of TEELeX system used in experiment

Figure 12: Two snapshots inside of TEELeX in experimental session

3.4. Results A lecturer at the farm gave a brief lecture about the hothouse and cultivated plants

(Spathiphyllum Schott and others). A student participating in the TEELeX system could see the hothouse in all directions, hear the lecture, and communicate with the lecturer. The lecture lasted about 20 minutes.

The student reported that she felt a more vivid scene of presence than she experienced in a usual video lecture. The experiment showed that the system we developed could be used for distance learning and outdoor education. The surrounding immersive projection display enhanced the sense of presence. The system enabled flexible configuration and quality controls. These made possible use of a panoramic view that needs four video streams within the limited bandwidth provided by the communication satellite.

The system we developed enabled a configuration for distance education to be set up more easily than it could be set up using a professional TV system broadcasting from outdoors because the pieces of equipment are small and readily available.

4. Summary We developed a multimedia satellite network system based on the IP and personal

computers. The usability and functionalities of the system were confirmed in an experiment. We will further improve the system’s functions for effective distance education between outdoor and indoor environments, and will make the system easier to use. We believe that the Focus+Context functions enable effective communication between outdoor and indoor environments. Moreover, we think that the enhanced collaborative environment achieved by the multimedia satellite network system will make outdoor distance education easier and more widely used. The system will also be used to motivate students to study environmental subjects.

Acknowledgements This research was partially supported by a Grant-in-Aid for Scientific Research on

Priority Areas (12040241), and by a Grant-in-Aid for Scientific Research (15200057) in Japan.

References [1] K. Asai, N. Osawa, and Y. Sugimoto. Virtual Environment System on Distance Education, Proc. of

EUROMEDIA '99, pp. 242-246, 1999. [2] K. Asai, Y. Y. Sugimoto, and F. Saito. Multi-screen display with liquid crystal projectors. Proceedings of

ISMCR'99, pp. 253-258, 1999. [3] S. K. Card, J. D. Mackinlay, and B. Shneiderman, Readings in Information Visualization - Using Vision to

Think, Morgan Kaufmann Publ., 1999. [4] C. Cruz-Neira, D. J. Sandin, T. A. DeFanti, R. V. Kenyon, and J. C. Hart. The cave automatic virtual

environment. Communications of the ACM, 35(2), pp.64-72, 1992. [5] Microsoft, DirectShow

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/directx9_c/directx/htm/directshow.asp [6] K.Tanaka, K.Kondo. Configuration of Inter-University Satellite Network (Space Collaboration System),

Trans.IEICE D-I, Vol.J82-D-I, No.4, pp.581-588, Apr.1999 (in Japanese).