Cross-Manipulation in Mixed Reality Based on a Bridge Virtual Marker

Raphael Grasset∗ Julian Stadon † Mark Billinghurst‡

HIT Lab NZUniversity of Canterbury

Private Bag 4800, Christchurch, NZ

ABSTRACT

In this paper we introduce a concept for transferring and manip-ulating objects between different interface contexts (AR and VR),thus extending the capabilities of manipulation in transitional inter-faces, which provide navigation between these different contexts.To transport objects between AR and VR environments, we proposethe use of a virtual marker proxy, which provides a bridge betweenworlds. The concept is demonstrated with a prototype that allowsusers to transfer objects between a Massively Multiplayer OnlineWorld and a desktop Augmented Reality application.

Index Terms: H.5.1 [Information Interfaces and Presentation]:Multimedia Information Systems—Artificial, augmented, and vir-tual realities; H.5.3 [Information Interfaces and Presentation]:Group and Organization Interfaces—Collaborative computing

1 INTRODUCTION

Hybrid collaborative systems supporting a combination of Aug-mented Reality Environments, Virtual Reality Environments orWeb Interfaces have been previously explored. In this type ofMixed Reality collaboration (often highly asymmetric), users canexperience a shared world such as that proposed in [1]. In this con-text, the information and the model are generally similar, the dif-fering element is indeed the interface and the representation of thedata (see the top of Figure 1 for a typical example).

Another range of configuration issues consist in cases where thedata is not shared between both worlds, each world is associatedwith its own data (illustrated at the bottom of Figure 1). In this con-text manipulating and exchanging data and content between worldsis intrinsically more challenging. It can be sometimes even moredifficult when users have no control over the architecture of the sys-tem used; for example with a Massively Multiplayer Online World(MMO). Offering a way to combine a traditional AR system with aVR world like a MMO is therefore attractive; thus there is a needfor techniques to transfer the data between both of these world (seebottom of Figure 1).

Previous solutions have been proposed in the context of multiplevirtual worlds in virtual reality. For example, Kiyokawa et al.[3]suggested the use of a window portal metaphor for remote manipu-lation, using the concept of a tunnel window.

For Augmented Reality, Butz et al. [2] proposed a drag and droptechnique between a 3D AR space to a screen space within the EM-MIE system. Schmalstieg et al. [6] has explored the use of tangibleobject and public locales to migrate applications between remotephysical spaces in a context of a collaborative AR scenario. Fi-nally, MacIntyre has recently modified a client of a MMO for creat-ing mixed reality experiences [4]. However, his purpose was more

∗e-mail:Raphael.Grasset@hitlabnz.org†e-mail:Julian.Staton@hotmail.com‡e-mail:Mark.Billinghurst@hitlabnz.org

AR VR

Shared Data

AR VR

Data 2Data 1

?

User User

UserUser

Figure 1: Cross-Manipulation Problem: In the top figure, two usersin different worlds (Augmented Reality and Virtual Reality Environ-ments) can manipulate similar data, such as the purple element viaa shared data mechanism. However if the data are separated, like inthe bottom figure, how can objects be transferred between worlds?

the creation of a unique mixed environment rather than dealing withtwo separate environments.

This brief literature survey shows that different solutions havebeen proposed for symmetric environments, but hardly any researchhas been conducted with an asymmetric solution (like between anAR Context and a VR Context) where different type of data areassociated with each world. In the next section we introduce ourapproach to the problem. We focused on the scenario of an ARenvironment coupled with a VR environment for a mixed-realitycollaboration.

2 CONCEPT

2.1 Virtual Marker

Our approach has been largely inspired by previous works in Vir-tual Reality and interactive workspaces that use the concept of abridge. A bridge defines a dedicated space where the user can trans-fer things between worlds, spaces, and contexts. Since we considerthe possible inability to modify or redesign the Virtual Reality En-vironment, it is necessary to define this bridge as part of the contentof the virtual world.

To do this, we introduce the notion of a virtual marker. Realfiducial markers are generally used for tracking but they also sup-port association with virtual objects using tangible interface con-cepts in an Augmented Reality Environment. Similarly, equivalentVirtual Markers can be described in the virtual world. However, inour case the presence of a virtual marker in the virtual world willprovide an anchor for an object in the world, and thus a bridge into

Screen (Window of VR)

Virtual Marker

RealMarker

Figure 2: The user can visualize both real markers and virtual mark-ers in the Augmented Reality Environment.

the virtual environment.To support the transfer between both worlds, a window of the

virtual world should be available in the real world. Having thiswindow, a user can observe the content of the virtual world andtherefore visualize virtual markers. In the user’s view, this virtualmarker becomes equivalent to a real marker, integrated in the ARApplication (illustrated in Figure 2).

2.2 Interactive Space

Assuming the availability of a virtual marker, our system uses threecomponents to transfer objects from the virtual world to the ARworld. We use the virtual marker, a transfer tool and a contextualbase. The Contextual Base provides a reference space for manip-ulating the Augmented Reality Content, while the Transfer Toolprovides the interface to transfer things between the Virtual Worldand the Augmented Reality World.

The user can transfer elements using proximity between twomarkers. They can thus realize bidirectional transfers in this or-der: Virtual Marker ↔ Transfer Tool ↔ Base. To help the user tounderstand the relationship between them we augment the markerswith basic visual feedback provided by a virtual torus around eachof the components (shown in Figure 3).

[You might want to put more here about the actual mechanismof how the transfer is done - see what I wrote below]

Window of VRContext

TransferTool

AR ContextBase

Virtual Bridge Marker

Window of VRContext

TransferTool

AR ContextBase

Virtual Bridge Marker

ARVR

Figure 3: A Virtual transfer: the red torus is on the virtual marker, thegreen torus on the transfer tool, and the blue on the AR base marker.

2.3 Object Coherence State

The transfer of an object between both worlds implies the ability todefine the state of the object in both of these worlds. For example, ifwe transfer an object from SecondLife to the AR Base and modifyan object property like scale, what should happen with the object inSecondLife, is it affected by this or is it in undefined state? If wetransfer the object back into SecondLife how will its properties beaffected?

To resolve this issue we consider three cases, mimicking the gen-eral solution implemented in standard 3D Modeling software. Atransferred object can be in three states: (1) a copy, (2) an instanceor (3) the same unique object. The instance will keep a coherentstate when manipulated in both of the worlds. The other two caseswon’t.

3 DESIGN AND IMPLEMENTATION

We have implemented this concept between SecondLife and a ded-icated AR prototype application. This prototype is part of an artinstallation that will be presented for an exhibition in May, in Perth,Australia.

The AR system has been implemented using the osgART library[5]. The user is equipped with an E-Magin Z800 HMD, and a Log-itech Quickcam 5000 USB camera. The VR World is displayedon a standard LCD screen. A 3D mesh corresponding to the vir-tual avatar of SecondLife is used to demonstrate the transfer. Whenthe user moves the transfer tool from the AR context base next tothe virtual marker on screen a 3D mesh is transferred from the ARenvironment into the on-screen SecondLife environment. To showthat we can change object state while doing this, when the user hastransferred the uncoloured 3D mesh from the transfer tool to thebase, the object is coloured and textured (and can be also animated).

We have been confronted with different design issues for the im-plementation of this prototype, including:

Window: the visibility of the virtual marker is highly dependenton the technology used to display the virtual marker. Thus the useof a non-reflective desktop screen provided the best solution. Aprojected image is problematic since the use of the camera for ARtracking requires good ambient lighting.

Markers: the virtual marker can be easily designed with built-intools from SecondLife. Using the navigation mode of SecondLife,placing a vertical marker in front of a clear background will im-prove the robustness of the AR tracking.

Coherence State: it remains challenging to support and maintaina coherent object state with SecondLife. The SecondLife API islimited, so we have only been able to implement a copy mode inour application.

4 CONCLUSION

We have presented an implementation for transferring and manipu-lating objects from a VR MMO to an AR World based on using avirtual marker. Acting like a bridge, the user can have the benefitof both worlds to visualize and modify different objects. We haveidentified design issues with this type of solution and presented ourinitial approach to resolving them. In the future we hope to inves-tigate further the problems related to maintaining a coherent datastate between both worlds.

REFERENCES

[1] B. Brown, I. MacColl, M. Chalmers, A. Galani, C. Randell, andA. Steed. Lessons from the lighthouse: collaboration in a shared mixedreality system. In CHI ’03, pages 577–584, 2003.

[2] A. Butz, T. Hollerer, S. Feiner, B. MacIntyre, and C. Beshers. Envelop-ing users and computers in a collaborative 3d augmented reality. InIWAR ’99: Proceedings of the 2nd IEEE and ACM International Work-shop on Augmented Reality, page 35, Washington, DC, USA, 1999.

[3] K. Kiyokawa and H. Takemura. A tunnel window and its variations:Seamless teleportation techniques in a virtual environment. In HCIInternational 2005, 2005.

[4] T. Lang, B. MacIntyre, and I. J. Zugaza. Massively multiplayer onlineworlds as a platform for augmented reality experiences. In IEEE VR’08, 2008.

[5] J. Looser, R. Grasset, H. Seichter, and M. Billinghurst. Osgart - a prag-matic approach to mr. In Industrial Workshop at ISMAR 2006, 2006.

[6] D. Schmalstieg and G. Hesina. Distributed applications for collabora-tive augmented reality. In IEEE Virtual Reality 2002, 2002.