Design and Implementation of a Wireless Handheld Multimodal

Digital Video Library Client System

Sam K. S. Sze Henry K. P. ChoiFeb 27 2002Group Meeting

Abstract

We have developed a client system for accessing a multimodal digital video library (DVL) namely, iVIEW.

It provides a user interface that meets the challenge of rich multimodal information presentation on wireless handheld device. An XML schema is employed to organize the multimodal metadata.

Furthermore, we investigated a context awareness mechanism complementary to the XML schema to facilitate scalable degradation under restricted resources in wireless application environment.

Presentation Overview

Introduction iVIEW Overview – A Practical Perspective iVIEW Wireless Client Scalable Degradation Conclusion

Introduction

Evolving DVL technologies Common challenges for wireless handheld

device application development Challenges for implementing wireless digital

video library client system

Introduction:Evolving DVL Technologies

Multimodal information extraction and organization. The modals includes: speech, on-screen characters, geographical locations, names, faces, on-screen objects …

Internet access Wireless access at anytime and anywhere

Introduction:Common Challenges of Wireless Handheld Devices

Limitations Inherited from Hardware Screen Size Input Device (Pen device) CPU Power Memory Battery Life

Bandwidth Limitations

Introduction:Challenges for Implementing a Wireless Digital Video Library Client

Intelligent User Interface Content Selection Result Set Refinement Integrate Handheld Platform Operation

Manner Resources Management

iVIEW Overview

iVIEW Overview:If Someone Comes to Room 121 and Ask VIEW Technologies ….

I want my video content to be searchable over Internet and wireless tomorrow.

What kind of stuff can you guys provide? ?

iVIEW Overview:What’s your answer? My Answer is …

Demonstration

iVIEW Client Architecture

iVIEW Client: iVIEW Client System Development

The iVIEW Client has been developed in several approaches: As a Microsoft Windows Native Application using Visual

Basic as the frontend with a number of Visual C++ DLLs. (CMU version, Michael Christel) Further extended to support both English and Chinese. (April 2001, Sam)

As a Java Applet for internet access using internal frames (Jacky FYP). Extended to support Real and Quicktime streaming, XML message parsing (August 2001, Sunny, Sam).

As a web page for internet access using Dynamic HTML and ASP. (December 2001, Sam).

Wireless …

iVIEW Client: A History of iVIEW Wireless Client

April 2001 (1 month) Survey - Sam June 2001, (3 weeks) Wireless Web Version – Gordon, Sam August 2001, (2 weeks) Framework of Multi-windows native

application using Embedded Visual C++ - Sam October 2001 (2 weeks) Bluetooth and GSM compliance

tested – Sam, Mike November 2001 (3 weeks) A variant version with HTML

viewer control – Mike December 2001 (1 month) A multilingual search engine

using ASP – Sam, Gordon January 2002 (1 month) A full version of the multi-windows

application - Henry February 2002 (2 weeks) A performance monitor added -

Henry

iVIEW Wireless Client:Implementation

Platform: iPAQ PocketPC

Wireless connections: GSM HSCSD 2.5G CDMA 802.11b Bluetooth

Multi-windows User Interface

3 Phrases of Operation: Query Result Set Manipulation Presentation

iVIEW Wireless Client:Query

Chinese and English Query

iVIEW Wireless Client:Result Set Manipulations

Segment View, Tree View, Topics View

iVIEW Wireless Client:Presentation

<?xml version="1.0" encoding="big5" ?> <sequence path="/iview/video/"> <time start="0">

  <script> GOVERNMENT HAS RESTORED FULL </script>   <frame file="frame141_00.jpg" />   </time> <time start="2">  <script> DIPLOMATIC RELATIONS WITH LIBYA </script>   </time> <time start="4">  <script> AFTER A 15 YEAR SUSPENSION. CNN'S </script>   <frame file="frame141_01.jpg" />   </time> <time start="6">  <script> MARGARET LOWRIE HAS THE DETAILS FROM LONDON   </script>   <map>LONDON</map>   </time> <time start="8">  <script> BRITISH FOREIGN SECRETARY ROBIN COOK </script>   </time> <time start="11">  <script> ANNOUNCED DIPLOMATIC TIES WITH LIBYA WOULD BE RESTORED BECAUSE IT </script>   </time> <time start="14">  <script>IT NOW ACCEPTS RESPONSIBILITY </script>   <frame file="frame141_02.jpg" />   </time> . . . . .  </sequence>

iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation

iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation

iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation

iVIEW Wireless Client:Embedded Resources Monitor

Resources CPU Usage Memory Utilization Bandwidth Consumption

iVIEW Wireless Client:Gracefully Degradation at Multimodal Presentation

A set modal significance order chain

(i.e. S(m1) > S(m2) > S(m3)> …)

should also be defined for different video types.

If system utilization saturates, the dispatch will be signaled to make a decision to stop dispatching a set of active modal of least significance. Mathematically,

iVIEW Wireless Client:Measurement – A Typical Client Operation Profile

iVIEW Wireless Client:Measurement – Degrade Transition

Conclusion

A wireless DVL client system, that meets the challenges of handling complex multimodal presentations in a handheld device, had been developed. We had shown a user interface design that allow flexible content selection, result set refinement in different views that fits the pen computing operation profile, and it also equips the capability of high complexity multimodal presentation.  

We define a multimodal presentation XML with its corresponding context aware scalable degrade mechanism that optimizes the presentation content under resources limited and fluctuating environment.

Q & A