ipaper@globis - interactive paper research

CSCW Seminar 2007, June 12, 2007

iPaper@GlobIS – Interactive Paper Research

CSCW Seminar

Beat Signer, Nadir Weibel, Adriana Ispas and Moira C. Norrie

Global Information Systems Research Group

Department of Computer Science, ETH Zurich

http://vub.academia.edu/BeatSigner

CSCW Seminar 2007, June 12, 2007 Beat Signer, [email protected]

Problems of Existing Applications

Many projects focus on the input device, paper,

printing and other hardware technologies rather

than on the data integration and information

management aspects isolated solutions

The linking of paper tends to be based on

physical rather than information-centric concepts

difficult to integrate new input / output devices


Our Approach

General interactive framework

different forms of paper-based interactions (enhanced

reading, enhanced writing, …)

rapid prototyping as well as large scale publishing

integration of new types of resources and services

integration of new input devices

The key to a highly integrated interactive paper

solution lies in the introduction of a platform for

general cross-media information management


Active Areas and Virtual Page Layers

image

video clip


iPaper Architecture


iServer Metamodel and Resource Plug-ins

iPaper

Shape Page

Application Database

DB ObjectData/Metadata

Query

Movie Clips

Time Span /

Spatial InfoMovie Clip

HTML Documents

XPointer XHTML

Document

Selectors

Entities

Resources

link


iServer

layer

|Layers|

layer

Active

Layers

resource

Containers

activeComponent

Active Com-

ponents

medium

Media

medium

Movies

medium

Images

medium

Webpages

OnLayer

Contains

(1,1)

(1,1) (0,*) (0,*)

document

Documents(0,*)

shape

Shapes

partition (2,*)

(0,*)

rectangle

Rectangles

ellipse

Ellipses

polygon

Polygons

circle

Circles

complexShape

Complex

Shapes

HasShapes

page

PagesOnPage

Contains

Pages

(0,*)

(1,1)

(1,1)

(0,*)

Resources

parameter

Preferences

link

Links

user

Users

group

Groups

individual

Individuals

HasMembers

HasTarget

(0,*)

(0,*)(0,*)

(0,*)

(0,*)

(0,*) (0,*)

(1,*)

(0,*)

(0,*)

(1,1)

(0,*)

partitionpartition

(1,*)

CreatedBy

RefersTo

Has

Preferences

parameter

Properties

selector

Selectors

resource

Resources

HasSource

(0,*)

AccessibleTo

InaccessibleTo

(0,*)

(0,*)

entity

EntitiesHasProperties

Users

Plug-ins

Links

(0,*)

Layers

iPAPER

iSERVER


iServer Interfaces

Java API

XML import and export

Interface based on XML and XSLT

XHTML

WAP, i-mode

VoiceXML

Web Service

Peer-to-peer (P2P)


iWeb Firefox Extension


iWeb Plug-in

Firefox Web Client

SOAP Request

SOAP Response

World Wide Web

HTTP

Request

HTTP

Response

Link DBiWeb Plug-in

iServExt


Active Content

Execute code (services) on the client and server side

Active Component

A

Active Component

B

Active Component

C

iS

erv

er

AC Manager

Server

R

eq

ue

st S

en

de

rActive Component

Stub B

AC Names

ACS Manager

C

lien

t D

evic

e

AC Names

Client

HT

TP


Active Components

Button

Slider

Slider2D

DateSelector

Browser

MapSelector

Rating

Capture

CaptureAndSpeak

Image

Sound

Movie

SQLClient

XCMRequest

ClientCtrlRequest

PaperPointControl

PaperPointAnnot

...


Active Components …

<?xml version="1.0" encoding="UTF-8" ?>

<iserver>

<activeComponent id="s1a" creator="beat">

<name>Show slide 1</name>

<properties>

<parameter>

<key>org.ximtec.iserver.ac:command</key>

<value>showSlide</value>

</parameter>

<parameter>

<key>org.ximtec.iserver.ac:slide</key>

<value>1</value>

</parameter>

</properties>

<identifier>PAPERPOINT_CONTROL</identifier>

</activeComponent>

...

</iserver>

PAPERPOINT_CONTROL

Show slide 1

command showSlide

slide 1


PaperPoint Presentation Tool


XML-Based Authoring

...

<document id="handout" creator="beat">

<name>Slide Handout</name>

<id>h1</id>

<size>

<width>210</width>

<height>297</height>

</size>

<content>handout.pdf</content>

</document>

<page id="p1" creator="beat" document="h1">

<name>Page 1, Slide Controller</name>

<number>1</number>

</page>

...

<rectangle id="fs" creator="beat"

layer="default" resource="p1">

<name>First Slide</name>

<upperLeft>

<point><x>24</x><y>12</y></point>

</upperLeft>

<size>

<width>25</width><height>8</height>

</size>

</rectangle>

...

...

<activeComponent id="fsa" creator="beat">

<name>First slide action</name>

<properties>

<parameter>

<key>org.ximtec.iserver.ac:command</key>

<value>firstSlide</value>

</parameter>

</properties>

<identifier>PAPERPOINT_CONTROL</identifier>

</activeComponent>

...

<link id="fsl" creator="beat" sources="fs"

targets="fsa">

<name>Show the first slide</name>

</link>

...


Digital Link Authoring Tool


PaperPoint Architecture


EdFest Components


EdFest Architecture Overview

iPaper Client

XCMcontent publishing

platform

Metadata DB Appln DB

iServer/iPapercross-media

link server

Link DB

Context

Engine

Context DB

Client

Controller

Active

Components

Text-to-Speech

Engine ICR Software

handwriting

recognition


Content Publishing Framework

XCMcontent publishing

XML data

of festival

venues

and events

iServercross-media link server

• paper link data

• active areas on paper

• layers

• digital resources

• user modelimport

publish

PDF

XML link

definition

data

iPublishpublishing framework

Structure Style

XSL CSS


Print-n-Link

Citations build the basis

for a scientific web of

publications

Use digital services to

search / retrieve cited

publications

Automatic authoring


Print-n-Link Architecture

Display

Printer

Driver

Document DB

iPAPER

iDOC

Publish

Document

Pattern

Semantic Mapper

Citation

Analyser

Citation

Retrieval

Document

Lookup

Citation DB

Print-n-Link Engine

iSERVER

Active Content

Link DB

iPaper Plug-in

Publish

Positions and

Active Content

Definition

Publish

Citations

Desktop

Print

(x,y)

Coordinates

Text-to-

Speech

Web Search

Services

iPaper

Client

Print-n-Link: Weaving the Paper Web

Moira C. NorrieDept. of Computer Science

ETH Zurich8092 Zurich, Switzerland

[email protected]

Beat SignerDept. of Computer Science


[email protected]

Nadir WeibelDept. of Computer Science


[email protected]

ABSTRACT

Citations form the basis for a web of scientific publications.Search engines, embedded hyperlinks and digital librariesall simplify the task of finding publications of interest onthe web and navigating to cited publications or web sites.

However the actual reading of publications often takes placeon paper and frequently on the move. We present a systemPrint-n-Link that uses technologies for interactive paper toenhance the reading process by enabling users to access dig-ital information and/or searches for cited documents from a

printed version of a publication using a digital pen for inter-action. A special virtual printer driver automatically gener-ates links from paper to digital services during the printingprocess based on an analysis of PDF documents. Depend-ing on the user setting and interaction gesture, the systemmay retrieve metadata about the citation and inform the

user through an audio channel or directly display the citeddocument on the user’s screen.

Categories and Subject Descriptors

D.2.11 [ Software Engineering ]: Software Architectures;H.4.m [ Information Systems Applications ]: Miscella-neous; H.5.m [ Information Interfaces and Presenta-tion ]: Miscellaneous

General Terms

Design, Algorithms, Experimentation

Keywords

Interactive paper, document integration, digital library, ci-tation management

1. INTRODUCTIONNowadays we are all familiar with the notion of hyperlinks

as a means of creating interconnected document spaces thatcan be easily searched and navigated with the help of search

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.DocEng’06, October 10–13, 2006, Amsterdam, The Netherlands.Copyright 2006 ACM 1-59593-515-0/06/0010 ...$5.00.

engines and web browsers. The ease with which we cannot only find documents on the web, but also follow linksto related documents is considered to be one of the mainadvantages of working with digital documents as opposedto paper equivalents. Yet many studies have shown thatpaper is still the preferred med ium when it comes to reading

documents for reasons of comfort, mobility and also ease ofannotation.

In this paper, we show how emerging technologies for in-teractive paper enable a bridge to be built between paperand digital document spaces that allows users to travel backand forth between the two using the medium most appropri-

ate to the current task and sit uation. Users can print doc-uments for reading and, by activating hyperlinks on paperwith a digital pen, they can not only access digital informa-tion about related documents, but also initiate web searchesto retrieve copies of those documents. The retrieved docu-

ments may in turn be printed for reading with automaticallygenerated embedded hyperlinks on the paper. Mobility issupported by providing disconnected users on the move withaudio information about related documents and the optionof adding them to a request list to be processed when recon-nected.

As an application, we take the everyday academic task ofreading scientific publications. Whether the reading activ-ity is for the purpose of knowledge discovery or reviewing,readers regularly look up citations to other publications byflicking to the reference list at the back of the paper. In somecases, a search for the cited publication may follow and, if

found, often it is printed and added to a pile of documents tobe read. Citations are the established meansof providing thelinks that form the basis of the scientific web of publicationsthat spans the digital and paper worlds. Just as citationswithin digital documents can be used to automatically gen-erate hyperlinks to cited documents, we can automatically

generate links within paper documents through an analysisof the PDF file when the document is printed.

Section 2 presents a more detailed look at the activityof reading scientific publications and a discussion of relatedwork. Section 3 describes the digital pen and paper tech-

nologies and how these can be used to support interactionbetween paper and digital services. We then present thegeneral functionality of our enhanced reading system in Sec-tion 4 and the overall architecture in Section 5. Details ofthe virtual printer component used to analyse PDF docu-ments for automatic link generation as well as printing are

given in Section 6. Concluding remarks follow in Section 7.

Interactive Paper Architecture

PrintingSemantic Mapper

Document Retrieval


Printed-Digital Mapping

static physical rendering

dynamic logical document


Printed-Digital Mapping

Source Digital

DocumentPaginated

(Print-Ready)

Document

Enhanced

Virtual Printer

Interactive

Document

(e.g. Anoto)

Augmenting

Technology

(ex. Anoto)

Geometrical Shapes,

Active Components

Logical

RepresentationPhysical

Representation

iServer

(ID, page, x,y)

iDoc

Digital

Elements


Mixed Physical-Digital Model


Interactive Paper Applications

Enhanced Reading interactive paper maps

interactive festival brochures

educational materials

auction brochures

scientific publications

…

Enhanced Writing photo album

researcher's notebook

mammography

search system

…

Paper-Based Interfaces

PaperPoint

digital libraries

video analysis tool

educational games

Artists' Installations

Lost Cosmonaut

Generosa Enterprise

Interactive TableTops

http://www.globis.ethz.ch/research/paper/applications/


Mammography Annotations


The Lost Cosmonaut

Writing as a collaborative

act of memory and story-

telling

Reading, writing and

narrative as an act of

making sense

Handwriting as a tool for

human computer

interaction

In collaboration with Axel Vogelsang,

Artists in Labs Programme


Generosa Enterprise

Interactive drawing installation

Exhibited at the 150 Years ETH Zurich Jubilee (more

than 200'000 visitors)


iGesture Framework

Common interface for

different gesture

recognition algorithms

Tools for implementing

and evaluating new

algorithms

Gesture set management


iGesture Admin Tab


Capture of Samples


Conclusions

Interactive paper framework (iPaper)

information-centric approach supports different

technologies, functionalities and applications

rapid prototyping (reusable active components)

Extensible platform for cross-media information

management (iServer)

Cross-media authoring / publishing tools

manual, content-driven and automatic link authoring


Student Projects

Basic technologies and infrastructure

iServer resource plug-ins, …

Integration of new tracking technologies

Interactive paper authoring tools

iGesture recognition framework (www.igesture.org)

Paper-based applications

PaperPoint, paper-based interface to digital

agenda/todo list (e.g. Outlook), paper-digital

notebook, paper-digital photo album, …

34

We offer Master and Semester projects and are also looking for Hilfsassistenten


References

http://www.ipaper.inf.ethz.ch

http://www.igesture.org

Fundamental Concepts for Interactive Paper and

Cross-Media Information Spaces, B. Signer,

Diss ETH Zurich Nr. 16218, 2006

35

ipaper@globis - interactive paper research

Science

events cscw seminar

edubeatsignercscw seminar

edfest components cscw

peer p2p cscw seminar

active content

active components paperpoint

slide handoutfirst slide

slide controller